# Bot operators will need to contact root@apache.org in order to be explicitly allowed. # Bots that do not respect crawl-delay instructions are not permitted. # To allow a bot access, add a pair of lines of the form: # # User-agent: Name # Disallow: # User-agent: Googlebot Disallow: User-agent: googlebot-image Disallow: User-agent: googlebot-mobile Disallow: User-agent: MSNBot Disallow: User-agent: Slurp Disallow: User-agent: Nutch Disallow: User-agent: ia_archiver Disallow: User-agent: yahoo-mmcrawler Disallow: User-agent: psbot Disallow: User-agent: yahoo-blogs/v3.9 Disallow: # Default action: don't allow. Global crawl-delay is set to two seconds. User-agent: * Disallow: / Crawl-delay: 2