The Ultimate Guide to robots.txt Declarations for 20+ AI Web Scrapers

The Ultimate Guide to robots.txt Declarations for 20+ AI Web Scrapers

As AI companies increasingly scrape web content to train large language models, website administrators face a critical challenge: distinguishing between legitimate search engine crawlers and AI training bots. This technical guide provides comprehensive robots.txt configuration rules to control access for both categories.

Understanding the Difference: Search Crawlers vs AI Scrapers

Traditional search engine crawlers index content to serve search results to users, driving traffic back to your site. AI scrapers, however, extract content for model training without necessarily providing attribution or traffic in return. Your robots.txt strategy should reflect these different use cases.

Basic robots.txt Syntax

Before diving into specific bot configurations, understand these fundamental directives:

• User-agent: Identifies the specific crawler
• Disallow: Blocks access to specified paths
• Allow: Permits access to specific paths (overrides Disallow)
• Crawl-delay: Sets delay between requests in seconds

Major Search Engine Crawlers (Generally Allow)

AI Scrapers: Complete User-Agent List

Here's a comprehensive table of AI web scrapers with their user-agent strings:

Sample robots.txt Configurations

Configuration 1: Block All AI Scrapers

# Block AI scrapers
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: Google-Extended
User-agent: anthropic-ai
User-agent: Claude-Web
User-agent: CCBot
User-agent: cohere-ai
User-agent: FacebookBot
User-agent: Meta-ExternalAgent
User-agent: Applebot-Extended
User-agent: PerplexityBot
User-agent: AI2Bot
User-agent: Bytespider
User-agent: Diffbot
User-agent: omgili
User-agent: YouBot
User-agent: Kangaroo Bot
User-agent: webz.io
Disallow: /

# Allow search engines
User-agent: Googlebot
User-agent: Bingbot
User-agent: Slurp
User-agent: DuckDuckBot
Allow: /

Configuration 2: Selective AI Access

# Allow specific AI bots for partnership/visibility
User-agent: PerplexityBot
User-agent: ChatGPT-User
Allow: /

# Block training-focused scrapers
User-agent: GPTBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: anthropic-ai
User-agent: Bytespider
Disallow: /

# Protect sensitive areas from all bots
User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /api/

Configuration 3: Granular Content Control

# Allow AI access to blog but not documentation
User-agent: GPTBot
User-agent: Claude-Web
Allow: /blog/
Disallow: /docs/
Disallow: /premium/

# Rate limiting for aggressive crawlers
User-agent: CCBot
Crawl-delay: 10
Disallow: /

User-agent: *
Allow: /

Best Practices and Considerations

• Monitor compliance: Not all bots respect robots.txt; monitor server logs for violations
• Regular updates: New AI scrapers emerge frequently; update your robots.txt quarterly
• Test configurations: Use Google Search Console's robots.txt tester before deployment
• Consider alternatives: Implement rate limiting, IP blocking, or authentication for sensitive content
• Legal implications: robots.txt is advisory; consider Terms of Service for legal protection
• SEO impact: Ensure you don't accidentally block legitimate search crawlers

Verification and Monitoring

After implementing your robots.txt configuration:

1. Validate syntax using online robots.txt validators
2. Check accessibility at yourdomain.com/robots.txt
3. Monitor server logs for user-agent patterns
4. Track bandwidth usage for unusual spikes
5. Review and update quarterly as new bots emerge

By implementing these configurations, you maintain control over how AI systems access your content while preserving beneficial search engine visibility. The landscape continues evolving, making ongoing monitoring essential for effective web resource management.