# Directions for robots. See this URL: # http://info.webcrawler.com/mak/projects/robots/norobots.html # for a description of the file format. # Crawlers are cool, but some areas are off-limits User-agent: * Disallow: /cgi-bin/ Disallow: /ftp/ Disallow: /stats/