Integrating Django and MoinMoin with Redis
##########################################

:date: 2012-12-02 14:50
:tags: Django, MoinMoin, Redis
:slug: integrating-django-and-moinmoin-with-redis
:author: Brian Neal
:summary: Here is how I integrated the MoinMoin wiki with Django using Redis!

We want a Wiki!
===============

Over at SurfGuitar101.com_, we decided we'd like to have a wiki to capture
community knowledge. I briefly looked at candidate wiki engines with an eye
towards integrating them with Django_, the framework that powers
SurfGuitar101.com_. And of course I was biased towards a wiki solution that was
written in Python_. I had tried a few wikis in the past, including the behemoth
MediaWiki_. MediaWiki is a very powerful piece of software, but it is also
quite complex, and I didn't want to have to maintain a PHP infrastructure to run
it.

Enter MoinMoin_. This is a mature wiki platform that is actively maintained and
written in Python. It is full featured but did not seem overly complex to me.
It stores its pages in flat files, which seemed appealing for our likely small
wiki needs. It turns out I had been a user of MoinMoin for many years without
really knowing it. The `Python.org wiki`_, `Mercurial wiki`_, and `omniORB
wiki`_ are all powered by MoinMoin_. We'd certainly be in good company.

Single Sign-On
==============

The feature that clinched it was MoinMoin's flexible `authentication system`_.
It would be very desirable if my users did not have to sign into Django and
then sign in again to the wiki with possibly a different username. Managing two
different password databases would be a real headache. The ideal solution would
mean signing into Django would log the user into MoinMoin with the same
username automatically.

MoinMoin supports this with their `external cookie authentication`_ mechanism.
The details are provided in the previous link; basically a Django site needs to
perform the following:

#. Set an external cookie for MoinMoin to read whenever a user logs into Django.
#. To prevent spoofing, the Django application should create a record that the
   cookie was created in some shared storage area accessible to MoinMoin. This
   allows MoinMoin to validate that the cookie is legitimate and not a fake.
#. When the user logs out of the Django application, it should delete this
   external cookie and the entry in the shared storage area.
#. Periodically the Django application should expire entires in the shared
   storage area for clean-up purposes. Otherwise this storage would grow and
   grow if users never logged out. Deleting entries older than the external
   cookie's age should suffice.

My Django Implementation
========================

There are of course many ways to approach this problem. Here is what I came up
with. I created a Django application called *wiki* to hold this integration
code. There is quite a lot of code here, too much to conveniently show in this
blog post. I will post snippets below, but you can refer to the complete code
in my `Bitbucket repository`_. You can also view online the `wiki application in
bitbucket`_ for convenience.

Getting notified of when users log into or out of Django is made easy thanks to
Django's `login and logout signals`_. By creating a signal handler I can be
notified when a user logs in or out. The signal handler code looks like this:

.. sourcecode:: python

   import logging
   from django.contrib.auth.signals import user_logged_in, user_logged_out
   from wiki.constants import SESSION_SET_MEMBER

   logger = logging.getLogger(__name__)

   def login_callback(sender, request, user, **kwargs):
       """Signal callback function for a user logging in.

       Sets a flag for the middleware to create an external cookie.

       """
       logger.info('User login: %s', user.username)

       request.wiki_set_cookie = True

   def logout_callback(sender, request, user, **kwargs):
       """Signal callback function for a user logging in.

       Sets a flag for the middleware to delete the external cookie.

       Since the user is about to logout, her session will be wiped out after
       this function returns. This forces us to set an attribute on the request
       object so that the response middleware can delete the wiki's cookie.

       """
       if user:
           logger.info('User logout: %s', user.username)

           # Remember what Redis set member to delete by adding an attribute to the
           # request object:
           request.wiki_delete_cookie = request.session.get(SESSION_SET_MEMBER)


   user_logged_in.connect(login_callback, dispatch_uid='wiki.signals.login')
   user_logged_out.connect(logout_callback, dispatch_uid='wiki.signals.logout')

When a user logs in I want to create an external cookie for MoinMoin. But
cookies can only be created on HttpResponse_ objects, and all we have access to
here in the signal handler is the request object. The solution here is to set
an attribute on the request object that a later piece of middleware_ will
process. I at first resisted this approach, thinking it was kind of hacky.
I initially decided to set a flag in the session, but then found out that in
some cases the session is not always available. I then reviewed some of the
Django supplied middleware classes and saw that they also set attributes on the
request object, so this must be an acceptable practice.

My middleware looks like this.

.. sourcecode:: python

   class WikiMiddleware(object):
       """
       Check for flags on the request object to determine when to set or delete an
       external cookie for the wiki application. When creating a cookie, also
       set an entry in Redis that the wiki application can validate to prevent
       spoofing.

       """

       def process_response(self, request, response):

           if hasattr(request, 'wiki_set_cookie'):
               create_wiki_session(request, response)
           elif hasattr(request, 'wiki_delete_cookie'):
               destroy_wiki_session(request.wiki_delete_cookie, response)

           return response

The ``create_wiki_session()`` function creates the cookie for MoinMoin and
stores a hash of the cookie in a shared storage area for MoinMoin to validate.
In our case, Redis_ makes an excellent shared storage area. We create a sorted
set in Redis to store our cookie hashes. The score for each hash is the
timestamp of when the cookie was created. This allows us to easily delete
expired cookies by score periodically.

.. sourcecode:: python

   def create_wiki_session(request, response):
       """Sets up the session for the external wiki application.

       Creates the external cookie for the Wiki.
       Updates the Redis set so the Wiki can verify the cookie.

       """
       now = datetime.datetime.utcnow()
       value = cookie_value(request.user, now)
       response.set_cookie(settings.WIKI_COOKIE_NAME,
               value=value,
               max_age=settings.WIKI_COOKIE_AGE,
               domain=settings.WIKI_COOKIE_DOMAIN)

       # Update a sorted set in Redis with a hash of our cookie and a score
       # of the current time as a timestamp. This allows us to delete old
       # entries by score periodically. To verify the cookie, the external wiki
       # application computes a hash of the cookie value and checks to see if
       # it is in our Redis set.

       h = hashlib.sha256()
       h.update(value)
       name = h.hexdigest()
       score = time.mktime(now.utctimetuple())
       conn = get_redis_connection()

       try:
           conn.zadd(settings.WIKI_REDIS_SET, score, name)
       except redis.RedisError:
           logger.error("Error adding wiki cookie key")

       # Store the set member name in the session so we can delete it when the
       # user logs out:
       request.session[SESSION_SET_MEMBER] = name

We store the name of the Redis set member in the user's session so we can
delete it from Redis when the user logs out. During logout, this set member is
retrieved from the session in the logout signal handler and stored on the
request object. This is because the session will be destroyed after the logout
signal handler runs and before the middleware can access it. The middleware
can check for the existence of this attribute as its cue to delete the wiki
session.

.. sourcecode:: python

   def destroy_wiki_session(set_member, response):
       """Destroys the session for the external wiki application.

       Delete the external cookie.
       Deletes the member from the Redis set as this entry is no longer valid.

       """
       response.delete_cookie(settings.WIKI_COOKIE_NAME,
                              domain=settings.WIKI_COOKIE_DOMAIN)

       if set_member:
           conn = get_redis_connection()
           try:
               conn.zrem(settings.WIKI_REDIS_SET, set_member)
           except redis.RedisError:
               logger.error("Error deleting wiki cookie set member")

As suggested in the MoinMoin external cookie documentation, I create a cookie
whose value consists of the username, email address, and a key separated by
the ``#`` character. The key is just a string of stuff that makes it difficult
for a spoofer to recreate.

.. sourcecode:: python

   def cookie_value(user, now):
       """Creates the value for the external wiki cookie."""

       # The key part of the cookie is just a string that would make things
       # difficult for a spoofer; something that can't be easily made up:

       h = hashlib.sha256()
       h.update(user.username + user.email)
       h.update(now.isoformat())
       h.update(''.join(random.sample(string.printable, 64)))
       h.update(settings.SECRET_KEY)
       key = h.hexdigest()

       parts = (user.username, user.email, key)
       return '#'.join(parts)

Finally on the Django side we should periodically delete expired Redis set
members in case users do not log out. Since I am using Celery_ with my Django
application, I created a Celery task that runs periodically to delete old set
members. This function is a bit longer than it probably needs to be, but
I wanted to log how big this set is before and after we cull the expired
entries.

.. sourcecode:: python

   @task
   def expire_cookies():
       """
       Periodically run this task to remove expired cookies from the Redis set
       that is shared between this Django application & the MoinMoin wiki for
       authentication.

       """
       now = datetime.datetime.utcnow()
       cutoff = now - datetime.timedelta(seconds=settings.WIKI_COOKIE_AGE)
       min_score = time.mktime(cutoff.utctimetuple())

       conn = get_redis_connection()

       set_name = settings.WIKI_REDIS_SET
       try:
           count = conn.zcard(set_name)
       except redis.RedisError:
           logger.error("Error getting zcard")
           return

       try:
           removed = conn.zremrangebyscore(set_name, 0.0, min_score)
       except redis.RedisError:
           logger.error("Error removing by score")
           return

       total = count - removed
       logger.info("Expire wiki cookies: removed %d, total is now %d",
               removed, total)

MoinMoin Implementation
=======================

As described in the MoinMoin external cookie documentation, you have to
configure MoinMoin to use your external cookie authentication mechanism.
It is also nice to disable the ability for the MoinMoin user to change their
username and email address since that is being managed by the Django
application. These changes to the MoinMoin ``Config`` class are shown below.

.. sourcecode:: python

   class Config(multiconfig.DefaultConfig):

       # ...

       # Use ExternalCookie method for integration authentication with Django:
       auth = [ExternalCookie(autocreate=True)]

       # remove ability to change username & email, etc.
       user_form_disable = ['name', 'aliasname', 'email',]
       user_form_remove = ['password', 'password2', 'css_url', 'logout', 'create',
               'account_sendmail', 'jid']

Next we create an ``ExternalCookie`` class and associated helper functions to
process the cookie and verify it in Redis. This code is shown in its entirety
below. It is based off the example in the MoinMoin external cookie
documentation, but uses Redis as the shared storage area.

.. sourcecode:: python

   import hashlib
   import Cookie
   import logging

   from MoinMoin.auth import BaseAuth
   from MoinMoin.user import User
   import redis

   COOKIE_NAME = 'YOUR_COOKIE_NAME_HERE'

   # Redis connection and database settings
   REDIS_HOST = 'localhost'
   REDIS_PORT = 6379
   REDIS_DB = 0

   # The name of the set in Redis that holds cookie hashes
   REDIS_SET = 'wiki_cookie_keys'

   logger = logging.getLogger(__name__)


   def get_cookie_value(cookie):
       """Returns the value of the Django cookie from the cookie.
       None is returned if the cookie is invalid or the value cannot be
       determined.

       This function works around an issue with different Python versions.
       In Python 2.5, if you construct a SimpleCookie with a dict, then
           type(cookie[key]) == unicode
       whereas in later versions of Python:
           type(cookie[key]) == Cookie.Morsel
       """
       if cookie:
           try:
               morsel = cookie[COOKIE_NAME]
           except KeyError:
               return None

           if isinstance(morsel, unicode):             # Python 2.5
               return morsel
           elif isinstance(morsel, Cookie.Morsel):     # Python 2.6+
               return morsel.value

       return None


   def get_redis_connection(host=REDIS_HOST, port=REDIS_PORT, db=REDIS_DB):
       """
       Create and return a Redis connection using the supplied parameters.

       """
       return redis.StrictRedis(host=host, port=port, db=db)


   def validate_cookie(value):
       """Determines if cookie was created by Django. Returns True on success,
       False on failure.

       Looks up the hash of the cookie value in Redis. If present, cookie
       is deemed legit.

       """
       h = hashlib.sha256()
       h.update(value)
       set_member = h.hexdigest()

       conn = get_redis_connection()
       success = False
       try:
           score = conn.zscore(REDIS_SET, set_member)
           success = score is not None
       except redis.RedisError:
           logger.error('Could not check Redis for ExternalCookie auth')

       return success


   class ExternalCookie(BaseAuth):
       name = 'external_cookie'

       def __init__(self, autocreate=False):
           self.autocreate = autocreate
           BaseAuth.__init__(self)

       def request(self, request, user_obj, **kwargs):
           user = None
           try_next = True

           try:
               cookie = Cookie.SimpleCookie(request.cookies)
           except Cookie.CookieError:
               cookie = None

           val = get_cookie_value(cookie)
           if val:
               try:
                   username, email, _ = val.split('#')
               except ValueError:
                   return user, try_next

               if validate_cookie(val):
                   user = User(request, name=username, auth_username=username,
                           auth_method=self.name)

                   changed = False
                   if email != user.email:
                       user.email = email
                       changed = True

                   if user:
                       user.create_or_update(changed)
                   if user and user.valid:
                       try_next = False

           return user, try_next

Conclusion
==========

I've been running this setup for a month now and it is working great. My users
and I are enjoying our shiny new MoinMoin wiki integrated with our Django
powered community website. The single sign-on experience is quite seamless and
eliminates the need for separate accounts.


.. _SurfGuitar101.com: http://surfguitar101.com
.. _Django: https://www.djangoproject.com
.. _MediaWiki: http://www.mediawiki.org
.. _MoinMoin: http://moinmo.in/
.. _Python: http://www.python.org
.. _Python.org wiki: http://wiki.python.org/moin/
.. _Mercurial wiki: http://mercurial.selenic.com/wiki/
.. _omniORB wiki: http://www.omniorb-support.com/omniwiki
.. _authentication system: http://moinmo.in/HelpOnAuthentication
.. _external cookie authentication: http://moinmo.in/HelpOnAuthentication/ExternalCookie
.. _login and logout signals: https://docs.djangoproject.com/en/1.4/topics/auth/#login-and-logout-signals
.. _HttpResponse: https://docs.djangoproject.com/en/1.4/ref/request-response/#httpresponse-objects
.. _middleware: https://docs.djangoproject.com/en/1.4/topics/http/middleware/
.. _Redis: http://redis.io/
.. _Bitbucket repository: https://bitbucket.org/bgneal/sg101
.. _wiki application in bitbucket: https://bitbucket.org/bgneal/sg101/src/a5b8f25e1752faf71ed429ec7f22ff6f3b3dc851/wiki?at=default
.. _Celery: http://celeryproject.org/