bgneal@4: Integrating Django and MoinMoin with Redis bgneal@4: ########################################## bgneal@4: bgneal@4: :date: 2012-12-02 14:50 bgneal@4: :tags: Django, MoinMoin, Redis bgneal@4: :slug: integrating-django-and-moinmoin-with-redis bgneal@4: :author: Brian Neal bgneal@7: :summary: Here is how I integrated the MoinMoin wiki with Django using Redis! bgneal@4: bgneal@4: We want a Wiki! bgneal@4: =============== bgneal@4: bgneal@4: Over at SurfGuitar101.com_, we decided we'd like to have a wiki to capture bgneal@4: community knowledge. I briefly looked at candidate wiki engines with an eye bgneal@4: towards integrating them with Django_, the framework that powers bgneal@4: SurfGuitar101.com_. And of course I was biased towards a wiki solution that was bgneal@4: written in Python_. I had tried a few wikis in the past, including the behemoth bgneal@4: MediaWiki_. MediaWiki is a very powerful piece of software, but it is also bgneal@4: quite complex, and I didn't want to have to maintain a PHP infrastructure to run bgneal@4: it. bgneal@4: bgneal@4: Enter MoinMoin_. This is a mature wiki platform that is actively maintained and bgneal@4: written in Python. It is full featured but did not seem overly complex to me. bgneal@4: It stores its pages in flat files, which seemed appealing for our likely small bgneal@4: wiki needs. It turns out I had been a user of MoinMoin for many years without bgneal@4: really knowing it. The `Python.org wiki`_, `Mercurial wiki`_, and `omniORB bgneal@4: wiki`_ are all powered by MoinMoin_. We'd certainly be in good company. bgneal@4: bgneal@4: Single Sign-On bgneal@4: ============== bgneal@4: bgneal@4: The feature that clinched it was MoinMoin's flexible `authentication system`_. bgneal@4: It would be very desirable if my users did not have to sign into Django and bgneal@4: then sign in again to the wiki with possibly a different username. Managing two bgneal@4: different password databases would be a real headache. The ideal solution would bgneal@4: mean signing into Django would log the user into MoinMoin with the same bgneal@4: username automatically. bgneal@4: bgneal@4: MoinMoin supports this with their `external cookie authentication`_ mechanism. bgneal@4: The details are provided in the previous link; basically a Django site needs to bgneal@4: perform the following: bgneal@4: bgneal@4: #. Set an external cookie for MoinMoin to read whenever a user logs into Django. bgneal@4: #. To prevent spoofing, the Django application should create a record that the bgneal@4: cookie was created in some shared storage area accessible to MoinMoin. This bgneal@4: allows MoinMoin to validate that the cookie is legitimate and not a fake. bgneal@4: #. When the user logs out of the Django application, it should delete this bgneal@4: external cookie and the entry in the shared storage area. bgneal@4: #. Periodically the Django application should expire entires in the shared bgneal@4: storage area for clean-up purposes. Otherwise this storage would grow and bgneal@4: grow if users never logged out. Deleting entries older than the external bgneal@4: cookie's age should suffice. bgneal@4: bgneal@4: My Django Implementation bgneal@4: ======================== bgneal@4: bgneal@4: There are of course many ways to approach this problem. Here is what I came up bgneal@4: with. I created a Django application called *wiki* to hold this integration bgneal@4: code. There is quite a lot of code here, too much to conveniently show in this bgneal@4: blog post. I will post snippets below, but you can refer to the complete code bgneal@4: in my `Bitbucket repository`_. You can also view online the `wiki application in bgneal@4: bitbucket`_ for convenience. bgneal@4: bgneal@4: Getting notified of when users log into or out of Django is made easy thanks to bgneal@4: Django's `login and logout signals`_. By creating a signal handler I can be bgneal@4: notified when a user logs in or out. The signal handler code looks like this: bgneal@4: bgneal@4: .. sourcecode:: python bgneal@4: bgneal@4: import logging bgneal@4: from django.contrib.auth.signals import user_logged_in, user_logged_out bgneal@4: from wiki.constants import SESSION_SET_MEMBER bgneal@4: bgneal@4: logger = logging.getLogger(__name__) bgneal@4: bgneal@4: def login_callback(sender, request, user, **kwargs): bgneal@4: """Signal callback function for a user logging in. bgneal@4: bgneal@4: Sets a flag for the middleware to create an external cookie. bgneal@4: bgneal@4: """ bgneal@4: logger.info('User login: %s', user.username) bgneal@4: bgneal@4: request.wiki_set_cookie = True bgneal@4: bgneal@4: def logout_callback(sender, request, user, **kwargs): bgneal@4: """Signal callback function for a user logging in. bgneal@4: bgneal@4: Sets a flag for the middleware to delete the external cookie. bgneal@4: bgneal@4: Since the user is about to logout, her session will be wiped out after bgneal@4: this function returns. This forces us to set an attribute on the request bgneal@4: object so that the response middleware can delete the wiki's cookie. bgneal@4: bgneal@4: """ bgneal@4: if user: bgneal@4: logger.info('User logout: %s', user.username) bgneal@4: bgneal@4: # Remember what Redis set member to delete by adding an attribute to the bgneal@4: # request object: bgneal@4: request.wiki_delete_cookie = request.session.get(SESSION_SET_MEMBER) bgneal@4: bgneal@4: bgneal@4: user_logged_in.connect(login_callback, dispatch_uid='wiki.signals.login') bgneal@4: user_logged_out.connect(logout_callback, dispatch_uid='wiki.signals.logout') bgneal@4: bgneal@4: When a user logs in I want to create an external cookie for MoinMoin. But bgneal@4: cookies can only be created on HttpResponse_ objects, and all we have access to bgneal@4: here in the signal handler is the request object. The solution here is to set bgneal@4: an attribute on the request object that a later piece of middleware_ will bgneal@4: process. I at first resisted this approach, thinking it was kind of hacky. bgneal@4: I initially decided to set a flag in the session, but then found out that in bgneal@4: some cases the session is not always available. I then reviewed some of the bgneal@4: Django supplied middleware classes and saw that they also set attributes on the bgneal@4: request object, so this must be an acceptable practice. bgneal@4: bgneal@4: My middleware looks like this. bgneal@4: bgneal@4: .. sourcecode:: python bgneal@4: bgneal@4: class WikiMiddleware(object): bgneal@4: """ bgneal@4: Check for flags on the request object to determine when to set or delete an bgneal@4: external cookie for the wiki application. When creating a cookie, also bgneal@4: set an entry in Redis that the wiki application can validate to prevent bgneal@4: spoofing. bgneal@4: bgneal@4: """ bgneal@4: bgneal@4: def process_response(self, request, response): bgneal@4: bgneal@4: if hasattr(request, 'wiki_set_cookie'): bgneal@4: create_wiki_session(request, response) bgneal@4: elif hasattr(request, 'wiki_delete_cookie'): bgneal@4: destroy_wiki_session(request.wiki_delete_cookie, response) bgneal@4: bgneal@4: return response bgneal@4: bgneal@4: The ``create_wiki_session()`` function creates the cookie for MoinMoin and bgneal@4: stores a hash of the cookie in a shared storage area for MoinMoin to validate. bgneal@4: In our case, Redis_ makes an excellent shared storage area. We create a sorted bgneal@4: set in Redis to store our cookie hashes. The score for each hash is the bgneal@4: timestamp of when the cookie was created. This allows us to easily delete bgneal@4: expired cookies by score periodically. bgneal@4: bgneal@4: .. sourcecode:: python bgneal@4: bgneal@4: def create_wiki_session(request, response): bgneal@4: """Sets up the session for the external wiki application. bgneal@4: bgneal@4: Creates the external cookie for the Wiki. bgneal@4: Updates the Redis set so the Wiki can verify the cookie. bgneal@4: bgneal@4: """ bgneal@4: now = datetime.datetime.utcnow() bgneal@4: value = cookie_value(request.user, now) bgneal@4: response.set_cookie(settings.WIKI_COOKIE_NAME, bgneal@4: value=value, bgneal@4: max_age=settings.WIKI_COOKIE_AGE, bgneal@4: domain=settings.WIKI_COOKIE_DOMAIN) bgneal@4: bgneal@4: # Update a sorted set in Redis with a hash of our cookie and a score bgneal@4: # of the current time as a timestamp. This allows us to delete old bgneal@4: # entries by score periodically. To verify the cookie, the external wiki bgneal@4: # application computes a hash of the cookie value and checks to see if bgneal@4: # it is in our Redis set. bgneal@4: bgneal@4: h = hashlib.sha256() bgneal@4: h.update(value) bgneal@4: name = h.hexdigest() bgneal@4: score = time.mktime(now.utctimetuple()) bgneal@4: conn = get_redis_connection() bgneal@4: bgneal@4: try: bgneal@4: conn.zadd(settings.WIKI_REDIS_SET, score, name) bgneal@4: except redis.RedisError: bgneal@4: logger.error("Error adding wiki cookie key") bgneal@4: bgneal@4: # Store the set member name in the session so we can delete it when the bgneal@4: # user logs out: bgneal@4: request.session[SESSION_SET_MEMBER] = name bgneal@4: bgneal@4: We store the name of the Redis set member in the user's session so we can bgneal@4: delete it from Redis when the user logs out. During logout, this set member is bgneal@4: retrieved from the session in the logout signal handler and stored on the bgneal@4: request object. This is because the session will be destroyed after the logout bgneal@4: signal handler runs and before the middleware can access it. The middleware bgneal@4: can check for the existence of this attribute as its cue to delete the wiki bgneal@4: session. bgneal@4: bgneal@4: .. sourcecode:: python bgneal@4: bgneal@4: def destroy_wiki_session(set_member, response): bgneal@4: """Destroys the session for the external wiki application. bgneal@4: bgneal@4: Delete the external cookie. bgneal@4: Deletes the member from the Redis set as this entry is no longer valid. bgneal@4: bgneal@4: """ bgneal@4: response.delete_cookie(settings.WIKI_COOKIE_NAME, bgneal@4: domain=settings.WIKI_COOKIE_DOMAIN) bgneal@4: bgneal@4: if set_member: bgneal@4: conn = get_redis_connection() bgneal@4: try: bgneal@4: conn.zrem(settings.WIKI_REDIS_SET, set_member) bgneal@4: except redis.RedisError: bgneal@4: logger.error("Error deleting wiki cookie set member") bgneal@4: bgneal@4: As suggested in the MoinMoin external cookie documentation, I create a cookie bgneal@4: whose value consists of the username, email address, and a key separated by bgneal@4: the ``#`` character. The key is just a string of stuff that makes it difficult bgneal@4: for a spoofer to recreate. bgneal@4: bgneal@4: .. sourcecode:: python bgneal@4: bgneal@4: def cookie_value(user, now): bgneal@4: """Creates the value for the external wiki cookie.""" bgneal@4: bgneal@4: # The key part of the cookie is just a string that would make things bgneal@4: # difficult for a spoofer; something that can't be easily made up: bgneal@4: bgneal@4: h = hashlib.sha256() bgneal@4: h.update(user.username + user.email) bgneal@4: h.update(now.isoformat()) bgneal@4: h.update(''.join(random.sample(string.printable, 64))) bgneal@4: h.update(settings.SECRET_KEY) bgneal@4: key = h.hexdigest() bgneal@4: bgneal@4: parts = (user.username, user.email, key) bgneal@4: return '#'.join(parts) bgneal@4: bgneal@4: Finally on the Django side we should periodically delete expired Redis set bgneal@4: members in case users do not log out. Since I am using Celery_ with my Django bgneal@4: application, I created a Celery task that runs periodically to delete old set bgneal@4: members. This function is a bit longer than it probably needs to be, but bgneal@4: I wanted to log how big this set is before and after we cull the expired bgneal@4: entries. bgneal@4: bgneal@4: .. sourcecode:: python bgneal@4: bgneal@4: @task bgneal@4: def expire_cookies(): bgneal@4: """ bgneal@4: Periodically run this task to remove expired cookies from the Redis set bgneal@4: that is shared between this Django application & the MoinMoin wiki for bgneal@4: authentication. bgneal@4: bgneal@4: """ bgneal@4: now = datetime.datetime.utcnow() bgneal@4: cutoff = now - datetime.timedelta(seconds=settings.WIKI_COOKIE_AGE) bgneal@4: min_score = time.mktime(cutoff.utctimetuple()) bgneal@4: bgneal@4: conn = get_redis_connection() bgneal@4: bgneal@4: set_name = settings.WIKI_REDIS_SET bgneal@4: try: bgneal@4: count = conn.zcard(set_name) bgneal@4: except redis.RedisError: bgneal@4: logger.error("Error getting zcard") bgneal@4: return bgneal@4: bgneal@4: try: bgneal@4: removed = conn.zremrangebyscore(set_name, 0.0, min_score) bgneal@4: except redis.RedisError: bgneal@4: logger.error("Error removing by score") bgneal@4: return bgneal@4: bgneal@4: total = count - removed bgneal@4: logger.info("Expire wiki cookies: removed %d, total is now %d", bgneal@4: removed, total) bgneal@4: bgneal@4: MoinMoin Implementation bgneal@4: ======================= bgneal@4: bgneal@4: As described in the MoinMoin external cookie documentation, you have to bgneal@4: configure MoinMoin to use your external cookie authentication mechanism. bgneal@4: It is also nice to disable the ability for the MoinMoin user to change their bgneal@4: username and email address since that is being managed by the Django bgneal@4: application. These changes to the MoinMoin ``Config`` class are shown below. bgneal@4: bgneal@4: .. sourcecode:: python bgneal@4: bgneal@4: class Config(multiconfig.DefaultConfig): bgneal@4: bgneal@4: # ... bgneal@4: bgneal@4: # Use ExternalCookie method for integration authentication with Django: bgneal@4: auth = [ExternalCookie(autocreate=True)] bgneal@4: bgneal@4: # remove ability to change username & email, etc. bgneal@4: user_form_disable = ['name', 'aliasname', 'email',] bgneal@4: user_form_remove = ['password', 'password2', 'css_url', 'logout', 'create', bgneal@4: 'account_sendmail', 'jid'] bgneal@4: bgneal@4: Next we create an ``ExternalCookie`` class and associated helper functions to bgneal@4: process the cookie and verify it in Redis. This code is shown in its entirety bgneal@4: below. It is based off the example in the MoinMoin external cookie bgneal@4: documentation, but uses Redis as the shared storage area. bgneal@4: bgneal@4: .. sourcecode:: python bgneal@4: bgneal@4: import hashlib bgneal@4: import Cookie bgneal@4: import logging bgneal@4: bgneal@4: from MoinMoin.auth import BaseAuth bgneal@4: from MoinMoin.user import User bgneal@4: import redis bgneal@4: bgneal@4: COOKIE_NAME = 'YOUR_COOKIE_NAME_HERE' bgneal@4: bgneal@4: # Redis connection and database settings bgneal@4: REDIS_HOST = 'localhost' bgneal@4: REDIS_PORT = 6379 bgneal@4: REDIS_DB = 0 bgneal@4: bgneal@4: # The name of the set in Redis that holds cookie hashes bgneal@4: REDIS_SET = 'wiki_cookie_keys' bgneal@4: bgneal@4: logger = logging.getLogger(__name__) bgneal@4: bgneal@4: bgneal@4: def get_cookie_value(cookie): bgneal@4: """Returns the value of the Django cookie from the cookie. bgneal@4: None is returned if the cookie is invalid or the value cannot be bgneal@4: determined. bgneal@4: bgneal@4: This function works around an issue with different Python versions. bgneal@4: In Python 2.5, if you construct a SimpleCookie with a dict, then bgneal@4: type(cookie[key]) == unicode bgneal@4: whereas in later versions of Python: bgneal@4: type(cookie[key]) == Cookie.Morsel bgneal@4: """ bgneal@4: if cookie: bgneal@4: try: bgneal@4: morsel = cookie[COOKIE_NAME] bgneal@4: except KeyError: bgneal@4: return None bgneal@4: bgneal@4: if isinstance(morsel, unicode): # Python 2.5 bgneal@4: return morsel bgneal@4: elif isinstance(morsel, Cookie.Morsel): # Python 2.6+ bgneal@4: return morsel.value bgneal@4: bgneal@4: return None bgneal@4: bgneal@4: bgneal@4: def get_redis_connection(host=REDIS_HOST, port=REDIS_PORT, db=REDIS_DB): bgneal@4: """ bgneal@4: Create and return a Redis connection using the supplied parameters. bgneal@4: bgneal@4: """ bgneal@4: return redis.StrictRedis(host=host, port=port, db=db) bgneal@4: bgneal@4: bgneal@4: def validate_cookie(value): bgneal@4: """Determines if cookie was created by Django. Returns True on success, bgneal@4: False on failure. bgneal@4: bgneal@4: Looks up the hash of the cookie value in Redis. If present, cookie bgneal@4: is deemed legit. bgneal@4: bgneal@4: """ bgneal@4: h = hashlib.sha256() bgneal@4: h.update(value) bgneal@4: set_member = h.hexdigest() bgneal@4: bgneal@4: conn = get_redis_connection() bgneal@4: success = False bgneal@4: try: bgneal@4: score = conn.zscore(REDIS_SET, set_member) bgneal@4: success = score is not None bgneal@4: except redis.RedisError: bgneal@4: logger.error('Could not check Redis for ExternalCookie auth') bgneal@4: bgneal@4: return success bgneal@4: bgneal@4: bgneal@4: class ExternalCookie(BaseAuth): bgneal@4: name = 'external_cookie' bgneal@4: bgneal@4: def __init__(self, autocreate=False): bgneal@4: self.autocreate = autocreate bgneal@4: BaseAuth.__init__(self) bgneal@4: bgneal@4: def request(self, request, user_obj, **kwargs): bgneal@4: user = None bgneal@4: try_next = True bgneal@4: bgneal@4: try: bgneal@4: cookie = Cookie.SimpleCookie(request.cookies) bgneal@4: except Cookie.CookieError: bgneal@4: cookie = None bgneal@4: bgneal@4: val = get_cookie_value(cookie) bgneal@4: if val: bgneal@4: try: bgneal@4: username, email, _ = val.split('#') bgneal@4: except ValueError: bgneal@4: return user, try_next bgneal@4: bgneal@4: if validate_cookie(val): bgneal@4: user = User(request, name=username, auth_username=username, bgneal@4: auth_method=self.name) bgneal@4: bgneal@4: changed = False bgneal@4: if email != user.email: bgneal@4: user.email = email bgneal@4: changed = True bgneal@4: bgneal@4: if user: bgneal@4: user.create_or_update(changed) bgneal@4: if user and user.valid: bgneal@4: try_next = False bgneal@4: bgneal@4: return user, try_next bgneal@4: bgneal@4: Conclusion bgneal@4: ========== bgneal@4: bgneal@4: I've been running this setup for a month now and it is working great. My users bgneal@4: and I are enjoying our shiny new MoinMoin wiki integrated with our Django bgneal@4: powered community website. The single sign-on experience is quite seamless and bgneal@4: eliminates the need for separate accounts. bgneal@4: bgneal@4: bgneal@4: .. _SurfGuitar101.com: http://surfguitar101.com bgneal@4: .. _Django: https://www.djangoproject.com bgneal@4: .. _MediaWiki: http://www.mediawiki.org bgneal@4: .. _MoinMoin: http://moinmo.in/ bgneal@4: .. _Python: http://www.python.org bgneal@4: .. _Python.org wiki: http://wiki.python.org/moin/ bgneal@4: .. _Mercurial wiki: http://mercurial.selenic.com/wiki/ bgneal@4: .. _omniORB wiki: http://www.omniorb-support.com/omniwiki bgneal@4: .. _authentication system: http://moinmo.in/HelpOnAuthentication bgneal@4: .. _external cookie authentication: http://moinmo.in/HelpOnAuthentication/ExternalCookie bgneal@4: .. _login and logout signals: https://docs.djangoproject.com/en/1.4/topics/auth/#login-and-logout-signals bgneal@4: .. _HttpResponse: https://docs.djangoproject.com/en/1.4/ref/request-response/#httpresponse-objects bgneal@4: .. _middleware: https://docs.djangoproject.com/en/1.4/topics/http/middleware/ bgneal@4: .. _Redis: http://redis.io/ bgneal@4: .. _Bitbucket repository: https://bitbucket.org/bgneal/sg101 bgneal@4: .. _wiki application in bitbucket: https://bitbucket.org/bgneal/sg101/src/a5b8f25e1752faf71ed429ec7f22ff6f3b3dc851/wiki?at=default bgneal@4: .. _Celery: http://celeryproject.org/