Complete Guide to AI Crawler Bots and How to Block Them
Executive Summary: AI companies deploy different bots for model training and search indexing. You can control bots like GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Google), Applebot-Extended (Apple), and CCBot (Common Crawl) via robots.txt. The table below provides user-agent strings, purposes, and robots.txt compliance for major AI crawlers.
Major AI Bots and Their Purposes
Each AI company may operate multiple bots for different purposes. Some harvest web content for model training, others index content for search results, and still others fetch pages on-demand in response to user queries. The following table summarizes the known major AI bots and their intended uses:
| Company | User-Agent | Purpose | robots.txt Compliance |
|---|---|---|---|
| OpenAI | GPTBot | Model training (GPT-4, GPT-5, etc.) | Yes |
| OpenAI | OAI-SearchBot | ChatGPT search results (not used for training) | Yes |
| OpenAI | ChatGPT-User | User-initiated page fetching | Partial (OpenAI notes that robots.txt rules may not apply uniformly to this agent) |
| Anthropic | ClaudeBot | Model training (Claude models) | Yes |
| Anthropic | Claude-User | User-initiated page fetching | Variable |
| Anthropic | Claude-SearchBot | Search quality improvement | Yes |
| Perplexity | PerplexityBot | Search result indexing | Yes |
| Perplexity | Perplexity-User | User-initiated page fetching | Variable |
| Google-Extended | Gemini Apps and Vertex AI training/grounding control (robots.txt token, not a separate crawler) | Yes | |
| Apple | Applebot-Extended | Foundation model training control (robots.txt token) | Yes |
| Apple | Applebot | Siri and Spotlight search | Yes |
| Common Crawl | CCBot | General web archiving (many AI models use this dataset) | Yes |
| Meta | Meta-ExternalAgent | AI training and product development (verify with official documentation) | Yes (default) |
| Amazon | Amazonbot | Search and product recommendations (verify AI training use with official documentation) | Yes |
| ByteDance | Bytespider | Search and content indexing (verify AI training use with official documentation) | Yes |
Important note: Verify the full details of AI training usage for Meta, Amazon, and ByteDance bots from official documentation. User-agent strings may change over time or new bots may be introduced.
Model Training vs. Search Bots: Key Differences
AI companies typically operate two distinct categories of bots:
Training Bots
Bots such as GPTBot, ClaudeBot, and CCBot harvest web content to train large language models. Blocking these bots prevents your content from being used in future model training. However, data previously collected may already be part of existing training datasets.
Search and User-Initiated Bots
Bots like OAI-SearchBot, PerplexityBot, and Claude-SearchBot index content for real-time search results. User agents such as ChatGPT-User, Claude-User, and Perplexity-User fetch pages when a user makes a specific request. robots.txt rules may not always be uniformly enforced for these types of bots.
Blocking AI Bots with robots.txt
To block specific AI bots, add rules like the following to your robots.txt file:
# Block OpenAI model training
User-agent: GPTBot
Disallow: /
# Block Anthropic model training
User-agent: ClaudeBot
Disallow: /
# Block Perplexity indexing
User-agent: PerplexityBot
Disallow: /
# Block Google Gemini/Vertex AI training (does not affect Search ranking)
User-agent: Google-Extended
Disallow: /
# Block Apple model training (does not affect Siri/Spotlight)
User-agent: Applebot-Extended
Disallow: /
# Block Common Crawl
User-agent: CCBot
Disallow: /
Note: Blocking Google-Extended does not affect Google Search ranking. Google AI Overviews are managed via standard Googlebot and snippet controls (nosnippet, max-snippet, data-nosnippet).
Blocking All AI Bots in Bulk
If you wish to block all known AI bots:
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: Bytespider
Disallow: /
Limitations of robots.txt
robots.txt is a courtesy protocol, not a security mechanism. For non-compliant bots, implement server-level or WAF (Web Application Firewall) blocking. Additionally, user-initiated agents (such as ChatGPT-User, Claude-User) may not always honor robots.txt rules; in these cases, IP-based blocking or rate limiting may be necessary.
About llms.txt
llms.txt is a file format proposed in 2024 that aims to provide structured information about your site to AI models. However, this is not an official standard, and there is no verified evidence that major search or AI providers use it for ranking or attribution. You may experiment with llms.txt, but do not expect transformative results. Follow official documentation for authoritative guidance.
Frequently Asked Questions
Does blocking GPTBot affect previous ChatGPT training?
No. Blocking GPTBot now prevents your content from being used in future model training, but has no effect on data previously collected and incorporated into training datasets. There is no retroactive deletion or "unlearning" mechanism.
Will blocking Google-Extended hurt my Google Search rankings?
No. Google-Extended controls only training and grounding usage for Gemini Apps and Vertex AI. Google Search ranking is determined by standard Googlebot and is independent of Google-Extended. To control AI Overviews, use snippet controls such as nosnippet, max-snippet, or data-nosnippet.
How do I block user-initiated bots (ChatGPT-User, Claude-User)?
Because these bots operate in response to user requests, they may not always honor robots.txt rules. OpenAI notes that robots.txt rules may not apply uniformly to ChatGPT-User. For more precise control, implement server-level IP-based blocking, rate limiting, or WAF rules. Check official documentation for current IP ranges.
How do I test robots.txt changes?
Google Search Console's legacy "robots.txt tester" has been deprecated. Instead, use the "robots.txt report" and "URL Inspection" tools to verify how Googlebot interprets your robots.txt file. For other bots, check official documentation for testing tools or monitor your server logs.