Here is my use case. A few days ago I've wrote an application to mirror my Flickr accounts. The pics are downloaded on file system and the metadata are stored in CouchDB. Now I use the Flickr interface to upload and tags pics but I got my own data on my own server. That's always cool.

Well done but now it can be useful to have a small web app to see the pics. Right ? That's what I've do. Since I love jQuery and CouchDB is full json compliant I don't want to write a complex app with tones of python code. So the idea is to have a small wsgi app to only serve static javascripts/html/css files and a proxy app to serve json by proxying CouchDB.

For now I'm using WSGIProxy. The code is very simple and look like this:

from wsgiproxy.exactproxy import proxy_exact_request
from webob import Request

class Proxy(object):
    def __init__(self, db=None, **kwargs):
        self.db = db
    def __call__(self, environ, start_response):
        req = Request(environ)
        if req.method == 'GET':
            req.server_name = '127.0.0.1'
            req.server_port = 5984
            req.script_name = ''
            req.path_info = '/%s%s' % (self.db, req.path_info)
            resp = req.get_response(proxy_exact_request)
            resp.content_type = 'text/jasacsript'
        else:
            resp = exc.HTTPForbidden()
        return resp(environ, start_response)


def make_app(global_conf, **local_conf):
    conf = global_conf.copy()
    conf.update(local_conf)
    return Proxy(**conf)

That's cool and simple and ok for small files. But if you want to handle large request WSGIProxy will raise a MemoryError. This is no longer fun. I've already sent a patch but don't want to bother the Paste team with that.

Then I've try to use restkit and thought that it can be the best library to wrote a proxy app. restkit manage a pool of http connection and handle large file request in a clean way.

The result is a small contribution. A set of WSGI applications included in the wsgi_proxy extention.

Here is a simple Paste config file to show own to use the proxies:

[server:main]
use = egg:Paste#http
port = 4969

[app:main]
use = egg:Paste#urlmap
/couchdb = couchdb
/ = proxy

[app:couchdb]
use = egg:restkit#host_proxy
uri = http://localhost:5984/mydb

[app:proxy]
use = egg:restkit#host_proxy
uri = http://benoitc.github.com/restkit/
max_connections=50
allowed_methods = get head post

No code needed. Cheers.

You can also use the Proxy class to proxify clients request and transform the response on the fly:

from webob import Request
from restkit.ext.wsgi_proxy import Proxy

proxy = Proxy()

def application(environ, start_response):
    req = Request(environ)
    req.environ['SERVER_NAME'] = 'example.com'
    req.environ['SERVER_PORT'] = '80'
    # do stuff
    ...
    resp = req.get_response(proxy)

    # do stuff ...
    ...

    return resp(environ, start_response)

I've also tried to replace WSGIProxy in deliverance with an ugly monkey patch:

from restkit.ext.wsgi_proxy import Proxy
from wsgiproxy import exactproxy
print 'Patching exactproxy with restkit proxy'
exactproxy.proxy_exact_request = Proxy(max_connections=4, allowed_methods=['GET', 'HEAD', 'POST']).__call__

It work perfect.

Most (good) web frameworks have a way to avoid Cross Site Request Forgery but in the Ajax world it's not so easy.

Here is my solution. First generate a secret key (sha1 of the current date or whatever) store it in user's session and show an hidden field in the requested webpage:

<input type="hidden" id="_req" name="_req" value="your secret key" />

Here is a snippet for Pylons:

def secure_field(with_id=False):
    value = request.environ.get('_req', None)
    if value is None:
        # generate a key if not already done for the request
        value = sha.new('%s-%s' % (datetime.now(), random.random())).hexdigest()
        session['_req'] = value
        request.environ['_req'] = value
        log.info('setting secure key to %r', value)
    if with_id:
        return hidden('_req', id='_req', value=value)
    return hidden('_req', value=value)

Notice that the field can be render multiple time with the same key (but only one with an id because of XHTML). The key is generated per request. This allow to have multiple _req fields in the same page so you can also add it to non-Ajax forms (and secure them too).

Then you need to POST this key on each Ajax request. Here is a wrapper for jQuery's post method:

post: function(url, data, callback, dataType) {
    // wrap $.post to add _req field
    if (!dataType) dataType = 'html';
    if (typeof(data) == typeof('')) {
        // $(form).serialize() return a string
        data += '&_req='+$('#_req').val();
    } else {
        data['_req'] = $('#_req').val();
    }
    $.post(url, data, callback, dataType);
}

The _req key is added to each POST request.

Last thing. You need to check that the key stored in user's session is also in the POST data. Here two Pylons decorators to avoid illegal requests:

@decorator
def secure_post(func, *args, **kwargs):
    """return html"""
    if request.method == 'POST':
        _req = session.get('_req', None)
        if _req is not None and _req == request.POST.get('_req'):
            del request.POST['_req']
            data = func(*args, **kwargs)
            return data
    if request.environ.get('paste.testing') is True:
        return func(*args, **kwargs)
    return _('Forbidden')

@decorator
def secure_json(func, *args, **kwargs):
    """return json"""
    if request.method == 'POST':
        _req = session.get('_req', None)
        if _req is not None and _req == request.POST.get('_req'):
            del request.POST['_req']
            data = func(*args, **kwargs)
    if request.environ.get('paste.testing') is True:
        data = func(*args, **kwargs)
    else:
        data = dict(error=_('Forbidden'))
    response.content_type = 'application/json'
    return json.dumps(data)

That's it. Now you are sure that all Ajax requests came from the user's web page. Cheers.

Of course the key is valid for more than one Ajax request. You may regenerate it for each main html page. May be this can be improved to change the key for each request including Ajax's but... It's already secure. Right ?

FormAlchemy 1.3 is released. From the website:

FormAlchemy eliminates boilerplate by autogenerating HTML input fields from a
given model. FormAlchemy will try to figure out what kind of HTML code should
be returned by introspecting the model's properties and generate ready-to-use
HTML code that will fit the developer's application.

Why I choose FormAlchemy instead of another form library ? Everybody knows that explicit is better than implicit. So most of form libraries use a schema to define how widgets are render. FormAlchemy avoid that by using the schema of the data model (SQLAlchemy mappers at the origin). So it generate forms implicitly using your explicit data model.

Another reason is that FormAlchemy is independent. This mean that you can use it in Pylons, Django, repoze.bfg, bobo and (put your favorite framework here).

Thats cool. But we can do more. And that's what we tried to do. SQLAlchemy is not the only library to define data model. So let use the others !

>>> from formalchemy.ext import couchdb
>>> from couchdbkit import schema
>>> class Pet(couchdb.Document):
...     name = schema.StringProperty()

>>> fs = FieldSet(Pet)
>>> fs.bind(Pet())
>>> print fs.render()

>>> from zope import interface
>>> from zope import schema

>>> class IPet(interface.Interface):
...     name = schema.TextLine(u'name')

>>> class Pet(object):
...     interface.implements(IPet)

>>> from formalchemy.ext.zope import FieldSet
>>> fs = FieldSet(IPet)
>>> fs = fs.bind(Pet())
>>> print fs.render()

Shabti

Shabti is a set of pylons templates initiated by Graham Higgins. There is now a FormAlchemy template in Shabti to quick initialize a pylons project with FormAlchemy and fa.jquery. The template also initialize a CRUD interface in the admin controller for you.

Here is some screenshots of the admin UI using fa.jquery:

User listing:

User edit form:

Notice that at this time Shabti require pylons-dev.

pip and buildout are two way to deal with python eggs. Both have the same functionality eg. Install python eggs in a isolated environment.

I've learn buildout first because I came from the zope world and it's the standard way to install zope for a while now. But when pip appears I give it a try.

At the first look pip seems interesting because it allow to install a bundle of packages with fixed versions. But the main problem is that pip use virtualenv to isolate packages so each time you need a new environment you need a new virtualenv and fetching all packages again. This can take a lot of time if you are using lxm or python library with C code. And more and more if you have a lot of projects.

Instead, you can share eggs between buildout's directories. You just need to tell where they are. Add this to your ~/.buildout/default.cfg:

[buildout]
eggs-directory=/home/gawel/eggs

That's all. When buildout need a egg he'll try to find it in this directory before fetching. If no version is found the egg is fetched and installed in this directory. Of course, you can have more than one version per package. You can tell buildout which version to use (see bellow).

I'm a Pylons fan so I already have all packages needed in my ~/eggs directory to install a new pylons environment. Let's create a new project in an isolated environment:

gawel:~/tmp% date
Mar 25 aoû 2009 21:49:11 CEST
gawel:~/tmp% mkdir pylons
gawel:~/tmp% cd pylons
gawel:~/tmp/pylons% vi buildout.cfg
gawel:~/tmp/pylons% buildout
Creating directory '/Users/gawel/tmp/pylons/bin'.
Creating directory '/Users/gawel/tmp/pylons/parts'.
Creating directory '/Users/gawel/tmp/pylons/develop-eggs'.
Installing eggs.
Generated script '/Users/gawel/tmp/pylons/bin/paster'.
Generated script '/Users/gawel/tmp/pylons/bin/sphinx-build'.
Generated script '/Users/gawel/tmp/pylons/bin/sphinx-quickstart'.
Generated script '/Users/gawel/tmp/pylons/bin/sphinx-autogen'.
gawel:~/tmp/pylons% ./bin/paster create -t pylons myproject
Selected and implied templates:
  Pylons#pylons  Pylons application template

Variables:
  egg:      myproject
  package:  myproject
  project:  myproject
Enter template_engine (mako/genshi/jinja2/etc: Template language) ['mako']:
(...)
  Copying templates/default_project/test.ini_tmpl to ./myproject/test.ini
Running /Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python setup.py egg_info
gawel:~/tmp/pylons% date
Mar 25 aoû 2009 21:50:24 CEST

This take 1mn13s. Now try with pip. I'll not ;)

That's the main reason why I choose buildout.

buildout also make unit testing easyer. Just put this config file in your package root:

[buildout]
newest = false
parts = eggs
develop = .

[eggs]
recipe = zc.recipe.egg
eggs =
  YourPackageName
  nose

Run buildout. And you'll be able to run ./bin/nosetests in an isolated environment with your package installed in develop mode (develop = .). I have one in all my projects if you need some examples.

Another reason is that buildout can be extended easily. One feature that exist in pip but not in buildout is the ability to fetch eggs from VCS's urls. This is not a builtin feature in buildout. But I've created a buildout extension for that (gp.vcsdevelop). And you know what ? This extension use pip !! ;) By the way, there is no plugin system in pip AFAIK.

Now the last reason. I wonder how pip's users upgrade an existing project. Do they need to install another environment ? With buildout i'ts easy. There is an extension to list all packages versions used by a buildout project. You just need to use the generated file as a buildout's version.cfg and tell buildout to use it.

[buildout]
versions = versions.cfg
...

Then update this file on your production server. Run bin/buildout again. That's it. Your project is up to date and use the correct versions just because buildout create is own sys.path with required eggs.

gawel:~/tmp/pylons% cat bin/paster
#!/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python

import sys
sys.path[0:0] = [
  '/Users/gawel/eggs/Pylons-0.9.7-py2.6.egg',
  '/Users/gawel/eggs/PasteScript-1.7.3-py2.6.egg',
  '/Users/gawel/eggs/setuptools-0.6c9-py2.6.egg',
  (...)
  '/Users/gawel/eggs/WebHelpers-0.6.4-py2.6.egg',
  '/Users/gawel/eggs/Routes-1.10.3-py2.6.egg',
  ]

import paste.script.command

if __name__ == '__main__':
    paste.script.command.run()

For all those reason, I will not use pip for now.

I know that buildout is more complicated than pip. But it's also more powerful. So, are you planning to learn how buildout works ? If so, I've wrote a How To for Pylons. You can also find it on pylonshq. I think this can help you to learn buildout even if you don't plan to use Pylons (but you should too ;).

jQuery work fine in a Firefox extension and it's easy to manipulate the DOM of the current document with it.

If you need to call some web services then it will fail. I've solved the problem and tried to submit a path on the jquery-devel mailing list.

The problem is not solved since it require some too specific Firefox code. So if you need this, check the patch I've submitted and apply the changes in the ajax function by yourself.