Mercurial > public > sg101

annotate static/robots.txt @ 896:0054a4a88c1c

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Remove checking for https availability. This seems to take quite a while, plus Python doesn't validate the cert, so we could end up with dodgy sites.
author Brian Neal <bgneal@gmail.com>
date Wed, 25 Feb 2015 21:09:41 -0600
parents 46d7f69b61db
children
rev   line source
bgneal@335 1 User-agent: *
bgneal@335 2 Crawl-delay: 30