view static/robots.txt @ 527:645025056dfa

Adding some management commands to the downloads application to help manage the SG101 MP3 compilations. Added a command to generate a HTML report of all the downloads in a given category. Added a command that generates a bash script that wgets all the files in a category.
author Brian Neal <bgneal@gmail.com>
date Wed, 21 Dec 2011 01:08:21 +0000
parents 46d7f69b61db
children
line wrap: on
line source
User-agent: *
Crawl-delay: 30