Here is my use case. A few days ago I've wrote an application to mirror my Flickr accounts. The pics are downloaded on file system and the metadata are stored in CouchDB. Now I use the Flickr interface to upload and tags pics but I got my own data on my own server. That's always cool.

Well done but now it can be useful to have a small web app to see the pics. Right ? That's what I've do. Since I love jQuery and CouchDB is full json compliant I don't want to write a complex app with tones of python code. So the idea is to have a small wsgi app to only serve static javascripts/html/css files and a proxy app to serve json by proxying CouchDB.

For now I'm using WSGIProxy. The code is very simple and look like this:

from wsgiproxy.exactproxy import proxy_exact_request
from webob import Request

class Proxy(object):
    def __init__(self, db=None, **kwargs):
        self.db = db
    def __call__(self, environ, start_response):
        req = Request(environ)
        if req.method == 'GET':
            req.server_name = '127.0.0.1'
            req.server_port = 5984
            req.script_name = ''
            req.path_info = '/%s%s' % (self.db, req.path_info)
            resp = req.get_response(proxy_exact_request)
            resp.content_type = 'text/jasacsript'
        else:
            resp = exc.HTTPForbidden()
        return resp(environ, start_response)


def make_app(global_conf, **local_conf):
    conf = global_conf.copy()
    conf.update(local_conf)
    return Proxy(**conf)

That's cool and simple and ok for small files. But if you want to handle large request WSGIProxy will raise a MemoryError. This is no longer fun. I've already sent a patch but don't want to bother the Paste team with that.

Then I've try to use restkit and thought that it can be the best library to wrote a proxy app. restkit manage a pool of http connection and handle large file request in a clean way.

The result is a small contribution. A set of WSGI applications included in the wsgi_proxy extention.

Here is a simple Paste config file to show own to use the proxies:

[server:main]
use = egg:Paste#http
port = 4969

[app:main]
use = egg:Paste#urlmap
/couchdb = couchdb
/ = proxy

[app:couchdb]
use = egg:restkit#host_proxy
uri = http://localhost:5984/mydb

[app:proxy]
use = egg:restkit#host_proxy
uri = http://benoitc.github.com/restkit/
max_connections=50
allowed_methods = get head post

No code needed. Cheers.

You can also use the Proxy class to proxify clients request and transform the response on the fly:

from webob import Request
from restkit.ext.wsgi_proxy import Proxy

proxy = Proxy()

def application(environ, start_response):
    req = Request(environ)
    req.environ['SERVER_NAME'] = 'example.com'
    req.environ['SERVER_PORT'] = '80'
    # do stuff
    ...
    resp = req.get_response(proxy)

    # do stuff ...
    ...

    return resp(environ, start_response)

I've also tried to replace WSGIProxy in deliverance with an ugly monkey patch:

from restkit.ext.wsgi_proxy import Proxy
from wsgiproxy import exactproxy
print 'Patching exactproxy with restkit proxy'
exactproxy.proxy_exact_request = Proxy(max_connections=4, allowed_methods=['GET', 'HEAD', 'POST']).__call__

It work perfect.

A few month ago I've started a small project named MercurialApp.

The main goal is to serve my Mercurial repositories as a wsgi application with Paste. It work fine and I use it on https://hg.gawel.org.

What I'm thinking now is that it was cool to serve the project documentations with the same application. I always have a docs/ folder in all my project with the Sphinx documentation in it. So let's use it.

Now you can add a sphinx_docs option in the configuration. Then MercurialApp add a changegroup hook (applied when a push occurs) in all repositories. This hook look for a docs/conf.py in the repository and if it exist try to rebuild the documentation in {sphinx_docs}/html/{reponame}/docs/.

Then if you look at the code in MercurialApp you will see something like this:

self.app = Cascate([StaticURLParser(os.path.join(c.sphinx_docs, 'html')), hgwebdir])

As you can see the newly generated application will try to serve a static file in sphinx_docs and if it does not exist serve the hgwebdir application. This way if you try to fetch /projectname/docs/ you'll see the sphinx documentation. /projectname will serve the hgwebdir application.

Here is a sample config file for MercurialApp:

[server:main]
use = egg:Paste#http
port = 5000

[app:main]
use = egg:MercurialApp

[hg:main]
# this is a public repo served at /
# everybody can read. Only gawel can push
hgwebdir = %(here)s/public_repositories
allow_read = *
allow_push = gawel

[hg:private]
# this is a private repo served at /private
# Only gawel can read and push
hgwebdir = %(here)s/public_repositories
allow_read = gawel
allow_push = gawel

The work still in progress but I like this project because I can browse my code as usual and the docs is always up to date without any wiki and/or WYSIWYG editor.

By the way Sphinx and Mercurial are two great python projects. A bunch of non python projects use them to write documentation or store source code. But what I love is that with python you can combine two great application in one. Python give me the power !

A few weeks ago I've discover Facebook Connect. So I decide to try to use it in a new project. It look like an easy way for new user to register in my application.

The first step is to follow the Quick start guide. Then whe need to take care of Facebook in our application. There is already a cool python library to play with Facebook called pyfacebook. So you need to install it.

In my project I aleady use repoze.what for authentification. It just rocks. So i decide to write an IIDentifier plugin for Facebook:

import sha
import facebook
from webob import Request
from zope.interface import implements
from repoze.who.interfaces import IIdentifier

DEFAULT_FIELDS = ['uid', 'name', 'first_name', 'birthday', 'relationship_status',
                  'proxied_email', 'sex', 'hometown_location',
                  'pic', 'pic_big', 'pic_small', 'pic_square']

class Params(dict):
    def __getattr__(self, attr):
        return self.get(attr, '')
    def __html__(self):
        return repr(self)

class Facebook(object):
    implements(IIdentifier)

    def __init__(self, api_key, secret_key):
        self.api_key = api_key
        self.secret_key = secret_key

    def identify(self, environ):
        req = Request(environ)

        # initialize the api object
        fbapi = facebook.Facebook(self.api_key, self.secret_key) # init pyfacebook instance
        if fbapi.check_session(req):
            environ['repoze.who.fb'] = fbapi
            user = fbapi.users.getInfo([fbapi.uid], DEFAULT_FIELDS)[0]

            # we used the proxied_email and a generated password to retriave
            # the user from our DB
            user.update(
                login=user['proxied_email'],
                email=user['proxied_email'],
                password=sha.new('%s%s' % (fbapi.uid, self.secret_key)).hexdigest(),
                )

            # we store the facebook data in environ
            environ['repoze.who.fbuser'] = Params(user.items())
            return user

    def remember(self, environ, identity): pass
    def forget(self, environ, identity): pass

At the first time there is no user in our DB. So the user isn't really connected to our interface. The trick is to use a javascript callback to show a registration form to the user with fields pre-filled with Facebook's data.

<html xmlns="http://www.w3.org/1999/xhtml" xmlns:fb="http://www.facebook.com/2008/fbml">
  <head></head>
  <body>
   <script src="http://static.ak.connect.facebook.com/js/api_lib/v0.4/FeatureLoader.js.php" type="text/javascript">
   </script>
   <fb:login-button onlogin="facebook_onlogin();"></fb:login-button>
   <script type="text/javascript">
     FB.init("YOUR_API_KEY_HERE", "xd_receiver.htm");
     var facebook_onlogin() function() {
         // here is the trick
         window.location.href = '/register/facebook';
     }
   </script>
</body> </html>

When the form is submitted, you need to create the real user in your application. That's all !

I'm using deliverance since a few month now to skin this blog and afpy.org. I also contribute to pyquery since i like the idea of manipulating xml in python like with jQuery. So the next step is to use a kind of pyquery rule in deliverance.

After looking at the deliverance code it seems that adding new rules is easy. You just need to register it in the deliverance rules. So here is the result:

from deliverance import rules
from pyquery import PyQuery as pq

class PyQuery(rules.AbstractAction):
    """PyQuery rule for deliverance"""
    name = 'pyquery'
    def __init__(self, source_location, callback=None):
        self.source_location = source_location
        self.callback = callback

    def apply(self, content_doc, theme_doc, resource_fetcher, log):
        """apply the rule"""
        self.callback(pq([content_doc]), pq([theme_doc]), resource_fetcher, log)

    @classmethod
    def from_xml(cls, tag, source_location):
        """Parses and instantiates the class from an element"""
        use = tag.attrib['use']
        modname, funcname = use.split(':')
        mod = __import__(modname, globals(), locals(), [''])
        callback = getattr(mod, funcname)
        return cls(source_location, callback)

# register the new rule
rules._actions['pyquery'] = PyQuery

Now i'm able to use my rule. I just need to import the above module before any deliverance one:

from myproject import pyquery_rule
from deliverance.middleware import DeliveranceMiddleware
# initialize your middleware

Then here is the rules:

<ruleset>
  <theme href="/theme.html" />
  <rule class="default">
      <pyquery use="myproject.rules:default" />
  </rule>
</ruleset>

Where myproject/rules.py look like this:

def default(content, theme, resource_fetcher, log):
    """rule used for testing"""
    content('title').text('My site - '+content('title').text())
    content('#footer').remove()
    theme('body').append(content('body'))

content and theme are pyquery objects. Feel free to do whatever you want with them. Enjoy !!

Notice that this work only with deliverance's trunk (or >=0.3).

A few month ago I've created a middleware to generate and cache images thumbnails. I need this to evaluate the image size from the browser's window on the client side and then get the thumbnail at an url based on this size.

The result is iw.thumbs. The package provide a highly configurable middleware to serve images thumbnails.

The principle is to map an uri to a file system directory. Then if an uri match the specified regexp, a thumbnail is served by the middleware.

For example, with the default configuration, the url http://localhost/thumbs/100x100/image.png will render a thumbnail of /var/mapped_dir/image.png.

There is also a "size" uri parser to use named size and prevent potential DoS attacks.

That what i'm using here, for my blog. I have two size: blog (490x490) and large (750x750). Here the Paste configuration:

[app:thumbs]
use = egg:iw.thumbs
url_regexp = ^/(?P<size>%s)(?P<path>/.+)
url_parser = iw.thumbs.url:size_parser
image_dir = %(here)s/var/rst/images
cache_dir = %(here)s/data/thumbs
sizes =
    blog = 490x490
    large = 750x750

As you can see, the package also provide an application factory in case of you just want to math a specific prefix.

Then I just have to map it in my url mapper section:

[app:main]
use = egg:Paste#urlmap
/thumbs = thumbs
/ = pylons # my pylons application

Another middleware exist to generate thumbnail with a different approach. repoze.bitblt allow you to generate thumbnail from the size specified in your image tag. It also add a secure part to the image url to prevent DoS attacks. I think both package have their place.

What I like with iw.thumbs is that I only need to change a configuration parameter instead of all image tags if the width of my blog column change in the future. The other good point is that you can generate image tags on the client side wich seems not really possible with repoze.bitblt.

But well, both middleware are another good reason to use WSGI applications.

And as an example, here is Alain, the AFPy mascot render with the blog size. If you click on it, you'll get the large size in a popup.

This time it's really winter. When it's cold the only thing i like is geeking at home. So I decide to rewrite my website with a set of WSGI applications.

The first thing done was to initialize a Pylons project for my blog. I got something working after a few hours. Pylons really speed developments.

Then I need a skin. I decide to give a try to Deliverance. Skinning with Deliverance is quite easy. After reading the docs, I've create a layout based on my old html code and a set of rules. Add this to my configuration file:

[filter:deliverance]
use = egg:Deliverance
theme_uri = file:///%(here)s/themes/layout.html
rule_uri = file:///%(here)s/themes/rules.xml

That's it. My Pylons app looks good without any style sheet as you can see ;)

Another part of my website is my projects documentations which are generated by Sphinx and served by a static app. Those pages are now skinned with Deliverance and you can't really know that you browse another application.

I also use Trac to browse my svn repository so I've try to put it in my WSGI stack. Trac now support WSGI except that it do something strange with the output and/or stderr so you'd better not using an error middleware with it else you get a beautiful blank page. You can create a WSGI application with a few lines of code:

# -*- coding: utf-8 -*-
import os
from trac.web.main import dispatch_request
def make_trac(conf, trac_env):
    def application(environ, start_request):
        environ['trac.env_path'] = trac_env
        return dispatch_request(environ, start_request)
    return application

and skin it with Deliverance. Notice that in my case, Trac seems inserting different encoding in the same page so I need to hack it a bit.

The last thing done is to use a ProxyApp to serve and skin my buildbot. Of course, this part is also skinned with Deliverance.

I now have only one process to serve all my python apps so I can use repoze.who to manage authentication for all apps. I have a ldap server, so I've write a small plugin for it. This work perfect, and I'm now able to log in Pylons and Trac.

I'm using Deliverance as a WSGI middleware and it seems a bit different than when you use it as a proxy. I encounter a few bugs. I have to only use XPath to find nodes in the DOM. Other expressions just don't work. You also can't remove nodes. But, well, this seems promising.