Mercurial > public > sg101

view static/robots.txt @ 980:3ebde23a59d0

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Pass timeout to requests instead of global timeout. Global timeout didn't seem to work once we started using requests.
author Brian Neal <bgneal@gmail.com>
date Thu, 15 Oct 2015 18:35:56 -0500
parents 46d7f69b61db
children
line wrap: on
line source
User-agent: *
Crawl-delay: 30