Mercurial > public > pelican-blog

annotate content/Coding/020-django-moinmoin.rst @ 10:6c03ca07a16d

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Renamed my tools directory to "tools". I named it __bgn because I was worried it would clash with a future Pelican updaet. But it seems like this would only happen if I re-ran the quickstart script. "tools" is a better name. :)
author Brian Neal <bgneal@gmail.com>
date Sun, 02 Feb 2014 11:32:13 -0600
parents 49bebfa6f9d3
children
rev   line source
bgneal@4 1 Integrating Django and MoinMoin with Redis
bgneal@4 2 ##########################################
bgneal@4 3
bgneal@4 4 :date: 2012-12-02 14:50
bgneal@4 5 :tags: Django, MoinMoin, Redis
bgneal@4 6 :slug: integrating-django-and-moinmoin-with-redis
bgneal@4 7 :author: Brian Neal
bgneal@7 8 :summary: Here is how I integrated the MoinMoin wiki with Django using Redis!
bgneal@4 9
bgneal@4 10 We want a Wiki!
bgneal@4 11 ===============
bgneal@4 12
bgneal@4 13 Over at SurfGuitar101.com_, we decided we'd like to have a wiki to capture
bgneal@4 14 community knowledge. I briefly looked at candidate wiki engines with an eye
bgneal@4 15 towards integrating them with Django_, the framework that powers
bgneal@4 16 SurfGuitar101.com_. And of course I was biased towards a wiki solution that was
bgneal@4 17 written in Python_. I had tried a few wikis in the past, including the behemoth
bgneal@4 18 MediaWiki_. MediaWiki is a very powerful piece of software, but it is also
bgneal@4 19 quite complex, and I didn't want to have to maintain a PHP infrastructure to run
bgneal@4 20 it.
bgneal@4 21
bgneal@4 22 Enter MoinMoin_. This is a mature wiki platform that is actively maintained and
bgneal@4 23 written in Python. It is full featured but did not seem overly complex to me.
bgneal@4 24 It stores its pages in flat files, which seemed appealing for our likely small
bgneal@4 25 wiki needs. It turns out I had been a user of MoinMoin for many years without
bgneal@4 26 really knowing it. The `Python.org wiki`_, `Mercurial wiki`_, and `omniORB
bgneal@4 27 wiki`_ are all powered by MoinMoin_. We'd certainly be in good company.
bgneal@4 28
bgneal@4 29 Single Sign-On
bgneal@4 30 ==============
bgneal@4 31
bgneal@4 32 The feature that clinched it was MoinMoin's flexible `authentication system`_.
bgneal@4 33 It would be very desirable if my users did not have to sign into Django and
bgneal@4 34 then sign in again to the wiki with possibly a different username. Managing two
bgneal@4 35 different password databases would be a real headache. The ideal solution would
bgneal@4 36 mean signing into Django would log the user into MoinMoin with the same
bgneal@4 37 username automatically.
bgneal@4 38
bgneal@4 39 MoinMoin supports this with their `external cookie authentication`_ mechanism.
bgneal@4 40 The details are provided in the previous link; basically a Django site needs to
bgneal@4 41 perform the following:
bgneal@4 42
bgneal@4 43 #. Set an external cookie for MoinMoin to read whenever a user logs into Django.
bgneal@4 44 #. To prevent spoofing, the Django application should create a record that the
bgneal@4 45 cookie was created in some shared storage area accessible to MoinMoin. This
bgneal@4 46 allows MoinMoin to validate that the cookie is legitimate and not a fake.
bgneal@4 47 #. When the user logs out of the Django application, it should delete this
bgneal@4 48 external cookie and the entry in the shared storage area.
bgneal@4 49 #. Periodically the Django application should expire entires in the shared
bgneal@4 50 storage area for clean-up purposes. Otherwise this storage would grow and
bgneal@4 51 grow if users never logged out. Deleting entries older than the external
bgneal@4 52 cookie's age should suffice.
bgneal@4 53
bgneal@4 54 My Django Implementation
bgneal@4 55 ========================
bgneal@4 56
bgneal@4 57 There are of course many ways to approach this problem. Here is what I came up
bgneal@4 58 with. I created a Django application called *wiki* to hold this integration
bgneal@4 59 code. There is quite a lot of code here, too much to conveniently show in this
bgneal@4 60 blog post. I will post snippets below, but you can refer to the complete code
bgneal@4 61 in my `Bitbucket repository`_. You can also view online the `wiki application in
bgneal@4 62 bitbucket`_ for convenience.
bgneal@4 63
bgneal@4 64 Getting notified of when users log into or out of Django is made easy thanks to
bgneal@4 65 Django's `login and logout signals`_. By creating a signal handler I can be
bgneal@4 66 notified when a user logs in or out. The signal handler code looks like this:
bgneal@4 67
bgneal@4 68 .. sourcecode:: python
bgneal@4 69
bgneal@4 70 import logging
bgneal@4 71 from django.contrib.auth.signals import user_logged_in, user_logged_out
bgneal@4 72 from wiki.constants import SESSION_SET_MEMBER
bgneal@4 73
bgneal@4 74 logger = logging.getLogger(__name__)
bgneal@4 75
bgneal@4 76 def login_callback(sender, request, user, **kwargs):
bgneal@4 77 """Signal callback function for a user logging in.
bgneal@4 78
bgneal@4 79 Sets a flag for the middleware to create an external cookie.
bgneal@4 80
bgneal@4 81 """
bgneal@4 82 logger.info('User login: %s', user.username)
bgneal@4 83
bgneal@4 84 request.wiki_set_cookie = True
bgneal@4 85
bgneal@4 86 def logout_callback(sender, request, user, **kwargs):
bgneal@4 87 """Signal callback function for a user logging in.
bgneal@4 88
bgneal@4 89 Sets a flag for the middleware to delete the external cookie.
bgneal@4 90
bgneal@4 91 Since the user is about to logout, her session will be wiped out after
bgneal@4 92 this function returns. This forces us to set an attribute on the request
bgneal@4 93 object so that the response middleware can delete the wiki's cookie.
bgneal@4 94
bgneal@4 95 """
bgneal@4 96 if user:
bgneal@4 97 logger.info('User logout: %s', user.username)
bgneal@4 98
bgneal@4 99 # Remember what Redis set member to delete by adding an attribute to the
bgneal@4 100 # request object:
bgneal@4 101 request.wiki_delete_cookie = request.session.get(SESSION_SET_MEMBER)
bgneal@4 102
bgneal@4 103
bgneal@4 104 user_logged_in.connect(login_callback, dispatch_uid='wiki.signals.login')
bgneal@4 105 user_logged_out.connect(logout_callback, dispatch_uid='wiki.signals.logout')
bgneal@4 106
bgneal@4 107 When a user logs in I want to create an external cookie for MoinMoin. But
bgneal@4 108 cookies can only be created on HttpResponse_ objects, and all we have access to
bgneal@4 109 here in the signal handler is the request object. The solution here is to set
bgneal@4 110 an attribute on the request object that a later piece of middleware_ will
bgneal@4 111 process. I at first resisted this approach, thinking it was kind of hacky.
bgneal@4 112 I initially decided to set a flag in the session, but then found out that in
bgneal@4 113 some cases the session is not always available. I then reviewed some of the
bgneal@4 114 Django supplied middleware classes and saw that they also set attributes on the
bgneal@4 115 request object, so this must be an acceptable practice.
bgneal@4 116
bgneal@4 117 My middleware looks like this.
bgneal@4 118
bgneal@4 119 .. sourcecode:: python
bgneal@4 120
bgneal@4 121 class WikiMiddleware(object):
bgneal@4 122 """
bgneal@4 123 Check for flags on the request object to determine when to set or delete an
bgneal@4 124 external cookie for the wiki application. When creating a cookie, also
bgneal@4 125 set an entry in Redis that the wiki application can validate to prevent
bgneal@4 126 spoofing.
bgneal@4 127
bgneal@4 128 """
bgneal@4 129
bgneal@4 130 def process_response(self, request, response):
bgneal@4 131
bgneal@4 132 if hasattr(request, 'wiki_set_cookie'):
bgneal@4 133 create_wiki_session(request, response)
bgneal@4 134 elif hasattr(request, 'wiki_delete_cookie'):
bgneal@4 135 destroy_wiki_session(request.wiki_delete_cookie, response)
bgneal@4 136
bgneal@4 137 return response
bgneal@4 138
bgneal@4 139 The ``create_wiki_session()`` function creates the cookie for MoinMoin and
bgneal@4 140 stores a hash of the cookie in a shared storage area for MoinMoin to validate.
bgneal@4 141 In our case, Redis_ makes an excellent shared storage area. We create a sorted
bgneal@4 142 set in Redis to store our cookie hashes. The score for each hash is the
bgneal@4 143 timestamp of when the cookie was created. This allows us to easily delete
bgneal@4 144 expired cookies by score periodically.
bgneal@4 145
bgneal@4 146 .. sourcecode:: python
bgneal@4 147
bgneal@4 148 def create_wiki_session(request, response):
bgneal@4 149 """Sets up the session for the external wiki application.
bgneal@4 150
bgneal@4 151 Creates the external cookie for the Wiki.
bgneal@4 152 Updates the Redis set so the Wiki can verify the cookie.
bgneal@4 153
bgneal@4 154 """
bgneal@4 155 now = datetime.datetime.utcnow()
bgneal@4 156 value = cookie_value(request.user, now)
bgneal@4 157 response.set_cookie(settings.WIKI_COOKIE_NAME,
bgneal@4 158 value=value,
bgneal@4 159 max_age=settings.WIKI_COOKIE_AGE,
bgneal@4 160 domain=settings.WIKI_COOKIE_DOMAIN)
bgneal@4 161
bgneal@4 162 # Update a sorted set in Redis with a hash of our cookie and a score
bgneal@4 163 # of the current time as a timestamp. This allows us to delete old
bgneal@4 164 # entries by score periodically. To verify the cookie, the external wiki
bgneal@4 165 # application computes a hash of the cookie value and checks to see if
bgneal@4 166 # it is in our Redis set.
bgneal@4 167
bgneal@4 168 h = hashlib.sha256()
bgneal@4 169 h.update(value)
bgneal@4 170 name = h.hexdigest()
bgneal@4 171 score = time.mktime(now.utctimetuple())
bgneal@4 172 conn = get_redis_connection()
bgneal@4 173
bgneal@4 174 try:
bgneal@4 175 conn.zadd(settings.WIKI_REDIS_SET, score, name)
bgneal@4 176 except redis.RedisError:
bgneal@4 177 logger.error("Error adding wiki cookie key")
bgneal@4 178
bgneal@4 179 # Store the set member name in the session so we can delete it when the
bgneal@4 180 # user logs out:
bgneal@4 181 request.session[SESSION_SET_MEMBER] = name
bgneal@4 182
bgneal@4 183 We store the name of the Redis set member in the user's session so we can
bgneal@4 184 delete it from Redis when the user logs out. During logout, this set member is
bgneal@4 185 retrieved from the session in the logout signal handler and stored on the
bgneal@4 186 request object. This is because the session will be destroyed after the logout
bgneal@4 187 signal handler runs and before the middleware can access it. The middleware
bgneal@4 188 can check for the existence of this attribute as its cue to delete the wiki
bgneal@4 189 session.
bgneal@4 190
bgneal@4 191 .. sourcecode:: python
bgneal@4 192
bgneal@4 193 def destroy_wiki_session(set_member, response):
bgneal@4 194 """Destroys the session for the external wiki application.
bgneal@4 195
bgneal@4 196 Delete the external cookie.
bgneal@4 197 Deletes the member from the Redis set as this entry is no longer valid.
bgneal@4 198
bgneal@4 199 """
bgneal@4 200 response.delete_cookie(settings.WIKI_COOKIE_NAME,
bgneal@4 201 domain=settings.WIKI_COOKIE_DOMAIN)
bgneal@4 202
bgneal@4 203 if set_member:
bgneal@4 204 conn = get_redis_connection()
bgneal@4 205 try:
bgneal@4 206 conn.zrem(settings.WIKI_REDIS_SET, set_member)
bgneal@4 207 except redis.RedisError:
bgneal@4 208 logger.error("Error deleting wiki cookie set member")
bgneal@4 209
bgneal@4 210 As suggested in the MoinMoin external cookie documentation, I create a cookie
bgneal@4 211 whose value consists of the username, email address, and a key separated by
bgneal@4 212 the ``#`` character. The key is just a string of stuff that makes it difficult
bgneal@4 213 for a spoofer to recreate.
bgneal@4 214
bgneal@4 215 .. sourcecode:: python
bgneal@4 216
bgneal@4 217 def cookie_value(user, now):
bgneal@4 218 """Creates the value for the external wiki cookie."""
bgneal@4 219
bgneal@4 220 # The key part of the cookie is just a string that would make things
bgneal@4 221 # difficult for a spoofer; something that can't be easily made up:
bgneal@4 222
bgneal@4 223 h = hashlib.sha256()
bgneal@4 224 h.update(user.username + user.email)
bgneal@4 225 h.update(now.isoformat())
bgneal@4 226 h.update(''.join(random.sample(string.printable, 64)))
bgneal@4 227 h.update(settings.SECRET_KEY)
bgneal@4 228 key = h.hexdigest()
bgneal@4 229
bgneal@4 230 parts = (user.username, user.email, key)
bgneal@4 231 return '#'.join(parts)
bgneal@4 232
bgneal@4 233 Finally on the Django side we should periodically delete expired Redis set
bgneal@4 234 members in case users do not log out. Since I am using Celery_ with my Django
bgneal@4 235 application, I created a Celery task that runs periodically to delete old set
bgneal@4 236 members. This function is a bit longer than it probably needs to be, but
bgneal@4 237 I wanted to log how big this set is before and after we cull the expired
bgneal@4 238 entries.
bgneal@4 239
bgneal@4 240 .. sourcecode:: python
bgneal@4 241
bgneal@4 242 @task
bgneal@4 243 def expire_cookies():
bgneal@4 244 """
bgneal@4 245 Periodically run this task to remove expired cookies from the Redis set
bgneal@4 246 that is shared between this Django application & the MoinMoin wiki for
bgneal@4 247 authentication.
bgneal@4 248
bgneal@4 249 """
bgneal@4 250 now = datetime.datetime.utcnow()
bgneal@4 251 cutoff = now - datetime.timedelta(seconds=settings.WIKI_COOKIE_AGE)
bgneal@4 252 min_score = time.mktime(cutoff.utctimetuple())
bgneal@4 253
bgneal@4 254 conn = get_redis_connection()
bgneal@4 255
bgneal@4 256 set_name = settings.WIKI_REDIS_SET
bgneal@4 257 try:
bgneal@4 258 count = conn.zcard(set_name)
bgneal@4 259 except redis.RedisError:
bgneal@4 260 logger.error("Error getting zcard")
bgneal@4 261 return
bgneal@4 262
bgneal@4 263 try:
bgneal@4 264 removed = conn.zremrangebyscore(set_name, 0.0, min_score)
bgneal@4 265 except redis.RedisError:
bgneal@4 266 logger.error("Error removing by score")
bgneal@4 267 return
bgneal@4 268
bgneal@4 269 total = count - removed
bgneal@4 270 logger.info("Expire wiki cookies: removed %d, total is now %d",
bgneal@4 271 removed, total)
bgneal@4 272
bgneal@4 273 MoinMoin Implementation
bgneal@4 274 =======================
bgneal@4 275
bgneal@4 276 As described in the MoinMoin external cookie documentation, you have to
bgneal@4 277 configure MoinMoin to use your external cookie authentication mechanism.
bgneal@4 278 It is also nice to disable the ability for the MoinMoin user to change their
bgneal@4 279 username and email address since that is being managed by the Django
bgneal@4 280 application. These changes to the MoinMoin ``Config`` class are shown below.
bgneal@4 281
bgneal@4 282 .. sourcecode:: python
bgneal@4 283
bgneal@4 284 class Config(multiconfig.DefaultConfig):
bgneal@4 285
bgneal@4 286 # ...
bgneal@4 287
bgneal@4 288 # Use ExternalCookie method for integration authentication with Django:
bgneal@4 289 auth = [ExternalCookie(autocreate=True)]
bgneal@4 290
bgneal@4 291 # remove ability to change username & email, etc.
bgneal@4 292 user_form_disable = ['name', 'aliasname', 'email',]
bgneal@4 293 user_form_remove = ['password', 'password2', 'css_url', 'logout', 'create',
bgneal@4 294 'account_sendmail', 'jid']
bgneal@4 295
bgneal@4 296 Next we create an ``ExternalCookie`` class and associated helper functions to
bgneal@4 297 process the cookie and verify it in Redis. This code is shown in its entirety
bgneal@4 298 below. It is based off the example in the MoinMoin external cookie
bgneal@4 299 documentation, but uses Redis as the shared storage area.
bgneal@4 300
bgneal@4 301 .. sourcecode:: python
bgneal@4 302
bgneal@4 303 import hashlib
bgneal@4 304 import Cookie
bgneal@4 305 import logging
bgneal@4 306
bgneal@4 307 from MoinMoin.auth import BaseAuth
bgneal@4 308 from MoinMoin.user import User
bgneal@4 309 import redis
bgneal@4 310
bgneal@4 311 COOKIE_NAME = 'YOUR_COOKIE_NAME_HERE'
bgneal@4 312
bgneal@4 313 # Redis connection and database settings
bgneal@4 314 REDIS_HOST = 'localhost'
bgneal@4 315 REDIS_PORT = 6379
bgneal@4 316 REDIS_DB = 0
bgneal@4 317
bgneal@4 318 # The name of the set in Redis that holds cookie hashes
bgneal@4 319 REDIS_SET = 'wiki_cookie_keys'
bgneal@4 320
bgneal@4 321 logger = logging.getLogger(__name__)
bgneal@4 322
bgneal@4 323
bgneal@4 324 def get_cookie_value(cookie):
bgneal@4 325 """Returns the value of the Django cookie from the cookie.
bgneal@4 326 None is returned if the cookie is invalid or the value cannot be
bgneal@4 327 determined.
bgneal@4 328
bgneal@4 329 This function works around an issue with different Python versions.
bgneal@4 330 In Python 2.5, if you construct a SimpleCookie with a dict, then
bgneal@4 331 type(cookie[key]) == unicode
bgneal@4 332 whereas in later versions of Python:
bgneal@4 333 type(cookie[key]) == Cookie.Morsel
bgneal@4 334 """
bgneal@4 335 if cookie:
bgneal@4 336 try:
bgneal@4 337 morsel = cookie[COOKIE_NAME]
bgneal@4 338 except KeyError:
bgneal@4 339 return None
bgneal@4 340
bgneal@4 341 if isinstance(morsel, unicode): # Python 2.5
bgneal@4 342 return morsel
bgneal@4 343 elif isinstance(morsel, Cookie.Morsel): # Python 2.6+
bgneal@4 344 return morsel.value
bgneal@4 345
bgneal@4 346 return None
bgneal@4 347
bgneal@4 348
bgneal@4 349 def get_redis_connection(host=REDIS_HOST, port=REDIS_PORT, db=REDIS_DB):
bgneal@4 350 """
bgneal@4 351 Create and return a Redis connection using the supplied parameters.
bgneal@4 352
bgneal@4 353 """
bgneal@4 354 return redis.StrictRedis(host=host, port=port, db=db)
bgneal@4 355
bgneal@4 356
bgneal@4 357 def validate_cookie(value):
bgneal@4 358 """Determines if cookie was created by Django. Returns True on success,
bgneal@4 359 False on failure.
bgneal@4 360
bgneal@4 361 Looks up the hash of the cookie value in Redis. If present, cookie
bgneal@4 362 is deemed legit.
bgneal@4 363
bgneal@4 364 """
bgneal@4 365 h = hashlib.sha256()
bgneal@4 366 h.update(value)
bgneal@4 367 set_member = h.hexdigest()
bgneal@4 368
bgneal@4 369 conn = get_redis_connection()
bgneal@4 370 success = False
bgneal@4 371 try:
bgneal@4 372 score = conn.zscore(REDIS_SET, set_member)
bgneal@4 373 success = score is not None
bgneal@4 374 except redis.RedisError:
bgneal@4 375 logger.error('Could not check Redis for ExternalCookie auth')
bgneal@4 376
bgneal@4 377 return success
bgneal@4 378
bgneal@4 379
bgneal@4 380 class ExternalCookie(BaseAuth):
bgneal@4 381 name = 'external_cookie'
bgneal@4 382
bgneal@4 383 def __init__(self, autocreate=False):
bgneal@4 384 self.autocreate = autocreate
bgneal@4 385 BaseAuth.__init__(self)
bgneal@4 386
bgneal@4 387 def request(self, request, user_obj, **kwargs):
bgneal@4 388 user = None
bgneal@4 389 try_next = True
bgneal@4 390
bgneal@4 391 try:
bgneal@4 392 cookie = Cookie.SimpleCookie(request.cookies)
bgneal@4 393 except Cookie.CookieError:
bgneal@4 394 cookie = None
bgneal@4 395
bgneal@4 396 val = get_cookie_value(cookie)
bgneal@4 397 if val:
bgneal@4 398 try:
bgneal@4 399 username, email, _ = val.split('#')
bgneal@4 400 except ValueError:
bgneal@4 401 return user, try_next
bgneal@4 402
bgneal@4 403 if validate_cookie(val):
bgneal@4 404 user = User(request, name=username, auth_username=username,
bgneal@4 405 auth_method=self.name)
bgneal@4 406
bgneal@4 407 changed = False
bgneal@4 408 if email != user.email:
bgneal@4 409 user.email = email
bgneal@4 410 changed = True
bgneal@4 411
bgneal@4 412 if user:
bgneal@4 413 user.create_or_update(changed)
bgneal@4 414 if user and user.valid:
bgneal@4 415 try_next = False
bgneal@4 416
bgneal@4 417 return user, try_next
bgneal@4 418
bgneal@4 419 Conclusion
bgneal@4 420 ==========
bgneal@4 421
bgneal@4 422 I've been running this setup for a month now and it is working great. My users
bgneal@4 423 and I are enjoying our shiny new MoinMoin wiki integrated with our Django
bgneal@4 424 powered community website. The single sign-on experience is quite seamless and
bgneal@4 425 eliminates the need for separate accounts.
bgneal@4 426
bgneal@4 427
bgneal@4 428 .. _SurfGuitar101.com: http://surfguitar101.com
bgneal@4 429 .. _Django: https://www.djangoproject.com
bgneal@4 430 .. _MediaWiki: http://www.mediawiki.org
bgneal@4 431 .. _MoinMoin: http://moinmo.in/
bgneal@4 432 .. _Python: http://www.python.org
bgneal@4 433 .. _Python.org wiki: http://wiki.python.org/moin/
bgneal@4 434 .. _Mercurial wiki: http://mercurial.selenic.com/wiki/
bgneal@4 435 .. _omniORB wiki: http://www.omniorb-support.com/omniwiki
bgneal@4 436 .. _authentication system: http://moinmo.in/HelpOnAuthentication
bgneal@4 437 .. _external cookie authentication: http://moinmo.in/HelpOnAuthentication/ExternalCookie
bgneal@4 438 .. _login and logout signals: https://docs.djangoproject.com/en/1.4/topics/auth/#login-and-logout-signals
bgneal@4 439 .. _HttpResponse: https://docs.djangoproject.com/en/1.4/ref/request-response/#httpresponse-objects
bgneal@4 440 .. _middleware: https://docs.djangoproject.com/en/1.4/topics/http/middleware/
bgneal@4 441 .. _Redis: http://redis.io/
bgneal@4 442 .. _Bitbucket repository: https://bitbucket.org/bgneal/sg101
bgneal@4 443 .. _wiki application in bitbucket: https://bitbucket.org/bgneal/sg101/src/a5b8f25e1752faf71ed429ec7f22ff6f3b3dc851/wiki?at=default
bgneal@4 444 .. _Celery: http://celeryproject.org/