Mercurial > public > pelican-blog

annotate content/Coding/020-django-moinmoin.rst @ 4:7ce6393e6d30

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Adding converted blog posts from old blog.
author Brian Neal <bgneal@gmail.com>
date Thu, 30 Jan 2014 21:45:03 -0600
parents
children 49bebfa6f9d3
rev   line source
bgneal@4 1 Integrating Django and MoinMoin with Redis
bgneal@4 2 ##########################################
bgneal@4 3
bgneal@4 4 :date: 2012-12-02 14:50
bgneal@4 5 :tags: Django, MoinMoin, Redis
bgneal@4 6 :slug: integrating-django-and-moinmoin-with-redis
bgneal@4 7 :author: Brian Neal
bgneal@4 8
bgneal@4 9 We want a Wiki!
bgneal@4 10 ===============
bgneal@4 11
bgneal@4 12 Over at SurfGuitar101.com_, we decided we'd like to have a wiki to capture
bgneal@4 13 community knowledge. I briefly looked at candidate wiki engines with an eye
bgneal@4 14 towards integrating them with Django_, the framework that powers
bgneal@4 15 SurfGuitar101.com_. And of course I was biased towards a wiki solution that was
bgneal@4 16 written in Python_. I had tried a few wikis in the past, including the behemoth
bgneal@4 17 MediaWiki_. MediaWiki is a very powerful piece of software, but it is also
bgneal@4 18 quite complex, and I didn't want to have to maintain a PHP infrastructure to run
bgneal@4 19 it.
bgneal@4 20
bgneal@4 21 Enter MoinMoin_. This is a mature wiki platform that is actively maintained and
bgneal@4 22 written in Python. It is full featured but did not seem overly complex to me.
bgneal@4 23 It stores its pages in flat files, which seemed appealing for our likely small
bgneal@4 24 wiki needs. It turns out I had been a user of MoinMoin for many years without
bgneal@4 25 really knowing it. The `Python.org wiki`_, `Mercurial wiki`_, and `omniORB
bgneal@4 26 wiki`_ are all powered by MoinMoin_. We'd certainly be in good company.
bgneal@4 27
bgneal@4 28 Single Sign-On
bgneal@4 29 ==============
bgneal@4 30
bgneal@4 31 The feature that clinched it was MoinMoin's flexible `authentication system`_.
bgneal@4 32 It would be very desirable if my users did not have to sign into Django and
bgneal@4 33 then sign in again to the wiki with possibly a different username. Managing two
bgneal@4 34 different password databases would be a real headache. The ideal solution would
bgneal@4 35 mean signing into Django would log the user into MoinMoin with the same
bgneal@4 36 username automatically.
bgneal@4 37
bgneal@4 38 MoinMoin supports this with their `external cookie authentication`_ mechanism.
bgneal@4 39 The details are provided in the previous link; basically a Django site needs to
bgneal@4 40 perform the following:
bgneal@4 41
bgneal@4 42 #. Set an external cookie for MoinMoin to read whenever a user logs into Django.
bgneal@4 43 #. To prevent spoofing, the Django application should create a record that the
bgneal@4 44 cookie was created in some shared storage area accessible to MoinMoin. This
bgneal@4 45 allows MoinMoin to validate that the cookie is legitimate and not a fake.
bgneal@4 46 #. When the user logs out of the Django application, it should delete this
bgneal@4 47 external cookie and the entry in the shared storage area.
bgneal@4 48 #. Periodically the Django application should expire entires in the shared
bgneal@4 49 storage area for clean-up purposes. Otherwise this storage would grow and
bgneal@4 50 grow if users never logged out. Deleting entries older than the external
bgneal@4 51 cookie's age should suffice.
bgneal@4 52
bgneal@4 53 My Django Implementation
bgneal@4 54 ========================
bgneal@4 55
bgneal@4 56 There are of course many ways to approach this problem. Here is what I came up
bgneal@4 57 with. I created a Django application called *wiki* to hold this integration
bgneal@4 58 code. There is quite a lot of code here, too much to conveniently show in this
bgneal@4 59 blog post. I will post snippets below, but you can refer to the complete code
bgneal@4 60 in my `Bitbucket repository`_. You can also view online the `wiki application in
bgneal@4 61 bitbucket`_ for convenience.
bgneal@4 62
bgneal@4 63 Getting notified of when users log into or out of Django is made easy thanks to
bgneal@4 64 Django's `login and logout signals`_. By creating a signal handler I can be
bgneal@4 65 notified when a user logs in or out. The signal handler code looks like this:
bgneal@4 66
bgneal@4 67 .. sourcecode:: python
bgneal@4 68
bgneal@4 69 import logging
bgneal@4 70 from django.contrib.auth.signals import user_logged_in, user_logged_out
bgneal@4 71 from wiki.constants import SESSION_SET_MEMBER
bgneal@4 72
bgneal@4 73 logger = logging.getLogger(__name__)
bgneal@4 74
bgneal@4 75 def login_callback(sender, request, user, **kwargs):
bgneal@4 76 """Signal callback function for a user logging in.
bgneal@4 77
bgneal@4 78 Sets a flag for the middleware to create an external cookie.
bgneal@4 79
bgneal@4 80 """
bgneal@4 81 logger.info('User login: %s', user.username)
bgneal@4 82
bgneal@4 83 request.wiki_set_cookie = True
bgneal@4 84
bgneal@4 85 def logout_callback(sender, request, user, **kwargs):
bgneal@4 86 """Signal callback function for a user logging in.
bgneal@4 87
bgneal@4 88 Sets a flag for the middleware to delete the external cookie.
bgneal@4 89
bgneal@4 90 Since the user is about to logout, her session will be wiped out after
bgneal@4 91 this function returns. This forces us to set an attribute on the request
bgneal@4 92 object so that the response middleware can delete the wiki's cookie.
bgneal@4 93
bgneal@4 94 """
bgneal@4 95 if user:
bgneal@4 96 logger.info('User logout: %s', user.username)
bgneal@4 97
bgneal@4 98 # Remember what Redis set member to delete by adding an attribute to the
bgneal@4 99 # request object:
bgneal@4 100 request.wiki_delete_cookie = request.session.get(SESSION_SET_MEMBER)
bgneal@4 101
bgneal@4 102
bgneal@4 103 user_logged_in.connect(login_callback, dispatch_uid='wiki.signals.login')
bgneal@4 104 user_logged_out.connect(logout_callback, dispatch_uid='wiki.signals.logout')
bgneal@4 105
bgneal@4 106 When a user logs in I want to create an external cookie for MoinMoin. But
bgneal@4 107 cookies can only be created on HttpResponse_ objects, and all we have access to
bgneal@4 108 here in the signal handler is the request object. The solution here is to set
bgneal@4 109 an attribute on the request object that a later piece of middleware_ will
bgneal@4 110 process. I at first resisted this approach, thinking it was kind of hacky.
bgneal@4 111 I initially decided to set a flag in the session, but then found out that in
bgneal@4 112 some cases the session is not always available. I then reviewed some of the
bgneal@4 113 Django supplied middleware classes and saw that they also set attributes on the
bgneal@4 114 request object, so this must be an acceptable practice.
bgneal@4 115
bgneal@4 116 My middleware looks like this.
bgneal@4 117
bgneal@4 118 .. sourcecode:: python
bgneal@4 119
bgneal@4 120 class WikiMiddleware(object):
bgneal@4 121 """
bgneal@4 122 Check for flags on the request object to determine when to set or delete an
bgneal@4 123 external cookie for the wiki application. When creating a cookie, also
bgneal@4 124 set an entry in Redis that the wiki application can validate to prevent
bgneal@4 125 spoofing.
bgneal@4 126
bgneal@4 127 """
bgneal@4 128
bgneal@4 129 def process_response(self, request, response):
bgneal@4 130
bgneal@4 131 if hasattr(request, 'wiki_set_cookie'):
bgneal@4 132 create_wiki_session(request, response)
bgneal@4 133 elif hasattr(request, 'wiki_delete_cookie'):
bgneal@4 134 destroy_wiki_session(request.wiki_delete_cookie, response)
bgneal@4 135
bgneal@4 136 return response
bgneal@4 137
bgneal@4 138 The ``create_wiki_session()`` function creates the cookie for MoinMoin and
bgneal@4 139 stores a hash of the cookie in a shared storage area for MoinMoin to validate.
bgneal@4 140 In our case, Redis_ makes an excellent shared storage area. We create a sorted
bgneal@4 141 set in Redis to store our cookie hashes. The score for each hash is the
bgneal@4 142 timestamp of when the cookie was created. This allows us to easily delete
bgneal@4 143 expired cookies by score periodically.
bgneal@4 144
bgneal@4 145 .. sourcecode:: python
bgneal@4 146
bgneal@4 147 def create_wiki_session(request, response):
bgneal@4 148 """Sets up the session for the external wiki application.
bgneal@4 149
bgneal@4 150 Creates the external cookie for the Wiki.
bgneal@4 151 Updates the Redis set so the Wiki can verify the cookie.
bgneal@4 152
bgneal@4 153 """
bgneal@4 154 now = datetime.datetime.utcnow()
bgneal@4 155 value = cookie_value(request.user, now)
bgneal@4 156 response.set_cookie(settings.WIKI_COOKIE_NAME,
bgneal@4 157 value=value,
bgneal@4 158 max_age=settings.WIKI_COOKIE_AGE,
bgneal@4 159 domain=settings.WIKI_COOKIE_DOMAIN)
bgneal@4 160
bgneal@4 161 # Update a sorted set in Redis with a hash of our cookie and a score
bgneal@4 162 # of the current time as a timestamp. This allows us to delete old
bgneal@4 163 # entries by score periodically. To verify the cookie, the external wiki
bgneal@4 164 # application computes a hash of the cookie value and checks to see if
bgneal@4 165 # it is in our Redis set.
bgneal@4 166
bgneal@4 167 h = hashlib.sha256()
bgneal@4 168 h.update(value)
bgneal@4 169 name = h.hexdigest()
bgneal@4 170 score = time.mktime(now.utctimetuple())
bgneal@4 171 conn = get_redis_connection()
bgneal@4 172
bgneal@4 173 try:
bgneal@4 174 conn.zadd(settings.WIKI_REDIS_SET, score, name)
bgneal@4 175 except redis.RedisError:
bgneal@4 176 logger.error("Error adding wiki cookie key")
bgneal@4 177
bgneal@4 178 # Store the set member name in the session so we can delete it when the
bgneal@4 179 # user logs out:
bgneal@4 180 request.session[SESSION_SET_MEMBER] = name
bgneal@4 181
bgneal@4 182 We store the name of the Redis set member in the user's session so we can
bgneal@4 183 delete it from Redis when the user logs out. During logout, this set member is
bgneal@4 184 retrieved from the session in the logout signal handler and stored on the
bgneal@4 185 request object. This is because the session will be destroyed after the logout
bgneal@4 186 signal handler runs and before the middleware can access it. The middleware
bgneal@4 187 can check for the existence of this attribute as its cue to delete the wiki
bgneal@4 188 session.
bgneal@4 189
bgneal@4 190 .. sourcecode:: python
bgneal@4 191
bgneal@4 192 def destroy_wiki_session(set_member, response):
bgneal@4 193 """Destroys the session for the external wiki application.
bgneal@4 194
bgneal@4 195 Delete the external cookie.
bgneal@4 196 Deletes the member from the Redis set as this entry is no longer valid.
bgneal@4 197
bgneal@4 198 """
bgneal@4 199 response.delete_cookie(settings.WIKI_COOKIE_NAME,
bgneal@4 200 domain=settings.WIKI_COOKIE_DOMAIN)
bgneal@4 201
bgneal@4 202 if set_member:
bgneal@4 203 conn = get_redis_connection()
bgneal@4 204 try:
bgneal@4 205 conn.zrem(settings.WIKI_REDIS_SET, set_member)
bgneal@4 206 except redis.RedisError:
bgneal@4 207 logger.error("Error deleting wiki cookie set member")
bgneal@4 208
bgneal@4 209 As suggested in the MoinMoin external cookie documentation, I create a cookie
bgneal@4 210 whose value consists of the username, email address, and a key separated by
bgneal@4 211 the ``#`` character. The key is just a string of stuff that makes it difficult
bgneal@4 212 for a spoofer to recreate.
bgneal@4 213
bgneal@4 214 .. sourcecode:: python
bgneal@4 215
bgneal@4 216 def cookie_value(user, now):
bgneal@4 217 """Creates the value for the external wiki cookie."""
bgneal@4 218
bgneal@4 219 # The key part of the cookie is just a string that would make things
bgneal@4 220 # difficult for a spoofer; something that can't be easily made up:
bgneal@4 221
bgneal@4 222 h = hashlib.sha256()
bgneal@4 223 h.update(user.username + user.email)
bgneal@4 224 h.update(now.isoformat())
bgneal@4 225 h.update(''.join(random.sample(string.printable, 64)))
bgneal@4 226 h.update(settings.SECRET_KEY)
bgneal@4 227 key = h.hexdigest()
bgneal@4 228
bgneal@4 229 parts = (user.username, user.email, key)
bgneal@4 230 return '#'.join(parts)
bgneal@4 231
bgneal@4 232 Finally on the Django side we should periodically delete expired Redis set
bgneal@4 233 members in case users do not log out. Since I am using Celery_ with my Django
bgneal@4 234 application, I created a Celery task that runs periodically to delete old set
bgneal@4 235 members. This function is a bit longer than it probably needs to be, but
bgneal@4 236 I wanted to log how big this set is before and after we cull the expired
bgneal@4 237 entries.
bgneal@4 238
bgneal@4 239 .. sourcecode:: python
bgneal@4 240
bgneal@4 241 @task
bgneal@4 242 def expire_cookies():
bgneal@4 243 """
bgneal@4 244 Periodically run this task to remove expired cookies from the Redis set
bgneal@4 245 that is shared between this Django application & the MoinMoin wiki for
bgneal@4 246 authentication.
bgneal@4 247
bgneal@4 248 """
bgneal@4 249 now = datetime.datetime.utcnow()
bgneal@4 250 cutoff = now - datetime.timedelta(seconds=settings.WIKI_COOKIE_AGE)
bgneal@4 251 min_score = time.mktime(cutoff.utctimetuple())
bgneal@4 252
bgneal@4 253 conn = get_redis_connection()
bgneal@4 254
bgneal@4 255 set_name = settings.WIKI_REDIS_SET
bgneal@4 256 try:
bgneal@4 257 count = conn.zcard(set_name)
bgneal@4 258 except redis.RedisError:
bgneal@4 259 logger.error("Error getting zcard")
bgneal@4 260 return
bgneal@4 261
bgneal@4 262 try:
bgneal@4 263 removed = conn.zremrangebyscore(set_name, 0.0, min_score)
bgneal@4 264 except redis.RedisError:
bgneal@4 265 logger.error("Error removing by score")
bgneal@4 266 return
bgneal@4 267
bgneal@4 268 total = count - removed
bgneal@4 269 logger.info("Expire wiki cookies: removed %d, total is now %d",
bgneal@4 270 removed, total)
bgneal@4 271
bgneal@4 272 MoinMoin Implementation
bgneal@4 273 =======================
bgneal@4 274
bgneal@4 275 As described in the MoinMoin external cookie documentation, you have to
bgneal@4 276 configure MoinMoin to use your external cookie authentication mechanism.
bgneal@4 277 It is also nice to disable the ability for the MoinMoin user to change their
bgneal@4 278 username and email address since that is being managed by the Django
bgneal@4 279 application. These changes to the MoinMoin ``Config`` class are shown below.
bgneal@4 280
bgneal@4 281 .. sourcecode:: python
bgneal@4 282
bgneal@4 283 class Config(multiconfig.DefaultConfig):
bgneal@4 284
bgneal@4 285 # ...
bgneal@4 286
bgneal@4 287 # Use ExternalCookie method for integration authentication with Django:
bgneal@4 288 auth = [ExternalCookie(autocreate=True)]
bgneal@4 289
bgneal@4 290 # remove ability to change username & email, etc.
bgneal@4 291 user_form_disable = ['name', 'aliasname', 'email',]
bgneal@4 292 user_form_remove = ['password', 'password2', 'css_url', 'logout', 'create',
bgneal@4 293 'account_sendmail', 'jid']
bgneal@4 294
bgneal@4 295 Next we create an ``ExternalCookie`` class and associated helper functions to
bgneal@4 296 process the cookie and verify it in Redis. This code is shown in its entirety
bgneal@4 297 below. It is based off the example in the MoinMoin external cookie
bgneal@4 298 documentation, but uses Redis as the shared storage area.
bgneal@4 299
bgneal@4 300 .. sourcecode:: python
bgneal@4 301
bgneal@4 302 import hashlib
bgneal@4 303 import Cookie
bgneal@4 304 import logging
bgneal@4 305
bgneal@4 306 from MoinMoin.auth import BaseAuth
bgneal@4 307 from MoinMoin.user import User
bgneal@4 308 import redis
bgneal@4 309
bgneal@4 310 COOKIE_NAME = 'YOUR_COOKIE_NAME_HERE'
bgneal@4 311
bgneal@4 312 # Redis connection and database settings
bgneal@4 313 REDIS_HOST = 'localhost'
bgneal@4 314 REDIS_PORT = 6379
bgneal@4 315 REDIS_DB = 0
bgneal@4 316
bgneal@4 317 # The name of the set in Redis that holds cookie hashes
bgneal@4 318 REDIS_SET = 'wiki_cookie_keys'
bgneal@4 319
bgneal@4 320 logger = logging.getLogger(__name__)
bgneal@4 321
bgneal@4 322
bgneal@4 323 def get_cookie_value(cookie):
bgneal@4 324 """Returns the value of the Django cookie from the cookie.
bgneal@4 325 None is returned if the cookie is invalid or the value cannot be
bgneal@4 326 determined.
bgneal@4 327
bgneal@4 328 This function works around an issue with different Python versions.
bgneal@4 329 In Python 2.5, if you construct a SimpleCookie with a dict, then
bgneal@4 330 type(cookie[key]) == unicode
bgneal@4 331 whereas in later versions of Python:
bgneal@4 332 type(cookie[key]) == Cookie.Morsel
bgneal@4 333 """
bgneal@4 334 if cookie:
bgneal@4 335 try:
bgneal@4 336 morsel = cookie[COOKIE_NAME]
bgneal@4 337 except KeyError:
bgneal@4 338 return None
bgneal@4 339
bgneal@4 340 if isinstance(morsel, unicode): # Python 2.5
bgneal@4 341 return morsel
bgneal@4 342 elif isinstance(morsel, Cookie.Morsel): # Python 2.6+
bgneal@4 343 return morsel.value
bgneal@4 344
bgneal@4 345 return None
bgneal@4 346
bgneal@4 347
bgneal@4 348 def get_redis_connection(host=REDIS_HOST, port=REDIS_PORT, db=REDIS_DB):
bgneal@4 349 """
bgneal@4 350 Create and return a Redis connection using the supplied parameters.
bgneal@4 351
bgneal@4 352 """
bgneal@4 353 return redis.StrictRedis(host=host, port=port, db=db)
bgneal@4 354
bgneal@4 355
bgneal@4 356 def validate_cookie(value):
bgneal@4 357 """Determines if cookie was created by Django. Returns True on success,
bgneal@4 358 False on failure.
bgneal@4 359
bgneal@4 360 Looks up the hash of the cookie value in Redis. If present, cookie
bgneal@4 361 is deemed legit.
bgneal@4 362
bgneal@4 363 """
bgneal@4 364 h = hashlib.sha256()
bgneal@4 365 h.update(value)
bgneal@4 366 set_member = h.hexdigest()
bgneal@4 367
bgneal@4 368 conn = get_redis_connection()
bgneal@4 369 success = False
bgneal@4 370 try:
bgneal@4 371 score = conn.zscore(REDIS_SET, set_member)
bgneal@4 372 success = score is not None
bgneal@4 373 except redis.RedisError:
bgneal@4 374 logger.error('Could not check Redis for ExternalCookie auth')
bgneal@4 375
bgneal@4 376 return success
bgneal@4 377
bgneal@4 378
bgneal@4 379 class ExternalCookie(BaseAuth):
bgneal@4 380 name = 'external_cookie'
bgneal@4 381
bgneal@4 382 def __init__(self, autocreate=False):
bgneal@4 383 self.autocreate = autocreate
bgneal@4 384 BaseAuth.__init__(self)
bgneal@4 385
bgneal@4 386 def request(self, request, user_obj, **kwargs):
bgneal@4 387 user = None
bgneal@4 388 try_next = True
bgneal@4 389
bgneal@4 390 try:
bgneal@4 391 cookie = Cookie.SimpleCookie(request.cookies)
bgneal@4 392 except Cookie.CookieError:
bgneal@4 393 cookie = None
bgneal@4 394
bgneal@4 395 val = get_cookie_value(cookie)
bgneal@4 396 if val:
bgneal@4 397 try:
bgneal@4 398 username, email, _ = val.split('#')
bgneal@4 399 except ValueError:
bgneal@4 400 return user, try_next
bgneal@4 401
bgneal@4 402 if validate_cookie(val):
bgneal@4 403 user = User(request, name=username, auth_username=username,
bgneal@4 404 auth_method=self.name)
bgneal@4 405
bgneal@4 406 changed = False
bgneal@4 407 if email != user.email:
bgneal@4 408 user.email = email
bgneal@4 409 changed = True
bgneal@4 410
bgneal@4 411 if user:
bgneal@4 412 user.create_or_update(changed)
bgneal@4 413 if user and user.valid:
bgneal@4 414 try_next = False
bgneal@4 415
bgneal@4 416 return user, try_next
bgneal@4 417
bgneal@4 418 Conclusion
bgneal@4 419 ==========
bgneal@4 420
bgneal@4 421 I've been running this setup for a month now and it is working great. My users
bgneal@4 422 and I are enjoying our shiny new MoinMoin wiki integrated with our Django
bgneal@4 423 powered community website. The single sign-on experience is quite seamless and
bgneal@4 424 eliminates the need for separate accounts.
bgneal@4 425
bgneal@4 426
bgneal@4 427 .. _SurfGuitar101.com: http://surfguitar101.com
bgneal@4 428 .. _Django: https://www.djangoproject.com
bgneal@4 429 .. _MediaWiki: http://www.mediawiki.org
bgneal@4 430 .. _MoinMoin: http://moinmo.in/
bgneal@4 431 .. _Python: http://www.python.org
bgneal@4 432 .. _Python.org wiki: http://wiki.python.org/moin/
bgneal@4 433 .. _Mercurial wiki: http://mercurial.selenic.com/wiki/
bgneal@4 434 .. _omniORB wiki: http://www.omniorb-support.com/omniwiki
bgneal@4 435 .. _authentication system: http://moinmo.in/HelpOnAuthentication
bgneal@4 436 .. _external cookie authentication: http://moinmo.in/HelpOnAuthentication/ExternalCookie
bgneal@4 437 .. _login and logout signals: https://docs.djangoproject.com/en/1.4/topics/auth/#login-and-logout-signals
bgneal@4 438 .. _HttpResponse: https://docs.djangoproject.com/en/1.4/ref/request-response/#httpresponse-objects
bgneal@4 439 .. _middleware: https://docs.djangoproject.com/en/1.4/topics/http/middleware/
bgneal@4 440 .. _Redis: http://redis.io/
bgneal@4 441 .. _Bitbucket repository: https://bitbucket.org/bgneal/sg101
bgneal@4 442 .. _wiki application in bitbucket: https://bitbucket.org/bgneal/sg101/src/a5b8f25e1752faf71ed429ec7f22ff6f3b3dc851/wiki?at=default
bgneal@4 443 .. _Celery: http://celeryproject.org/