What Cloudflare's AI Bot Blocking Really Blocks (And What It Doesn't)
Executive Summary: Cloudflare's "AI Scrapers and Crawlers" setting (or similar) blocks model training bots (GPTBot, ClaudeBot, etc.) while leaving search engine crawlers (Googlebot, Bingbot) unaffected. However, since setting names and scope can change, verify which User-agents are listed in your Cloudflare dashboard and ensure there are no conflicts with your robots.txt file. To test, examine your server logs or simulate specific User-agents with cURL.
How Cloudflare's AI Bot Blocking Works
Since late 2023, Cloudflare has offered customers an "AI bot blocking" option. This setting typically appears in Security → Bots or Security → Settings under a name like "Block AI Scrapers and Crawlers." Its purpose is to block bots that collect content for large language model training (GPTBot, ClaudeBot, CCBot, etc.) with a single click.
Cloudflare manages this list through its bot management infrastructure; when you enable this setting, Cloudflare's WAF returns HTTP 403 or a challenge to AI training bots it has identified. Critical point: this setting does not affect search engine indexing bots (Googlebot, Bingbot, Yandex). Cloudflare doesn't assume you want to block search traffic to your site; it only targets training-focused crawlers.
Which Bots Are Affected?
The list shown in Cloudflare's official documentation and dashboard may be updated over time. Generally, these User-agents fall into this category:
GPTBot(OpenAI model training)ClaudeBot(Anthropic model training)CCBot(Common Crawl)Google-Extended(Google's token for Gemini/Vertex AI training; however, since this is a robots.txt token, verify how Cloudflare handles it at the WAF level)- Other AI data collection bots like
anthropic-ai,Omgilibot,FacebookBot
Which bots are NOT affected:
Googlebot,Googlebot-Image,Googlebot-News(Google Search indexing)Bingbot(Microsoft Bing indexing)Slurp(Yahoo)YandexBot(Yandex)Applebot(Apple Siri/Spotlight search; note thatApplebot-Extendedis for training and may be blocked)
In Cloudflare's "Bot Management" or "Firewall Rules" section, you may find a list showing which User-agents are blocked. Check the official documentation for the current list: search for "AI bots" or "bot management" at developers.cloudflare.com.
Verifying That Search Bots Aren't Blocked
After enabling the Cloudflare setting, follow these steps to ensure search bots can still access your site:
1. Examine Server Logs
Check the access logs of your origin server behind Cloudflare (Apache, Nginx, IIS, etc.). Search for requests with Googlebot or Bingbot User-agents from the last 24-48 hours:
grep "Googlebot" /var/log/nginx/access.log | tail -20
If you see 200 or 304 status codes from these bots, search bots are not being blocked. If you see 403 or 503, review your Cloudflare settings or Firewall Rules.
2. Google Search Console and Bing Webmaster Tools
In Google Search Console, use Settings → Crawl Stats or the URL Inspection tool. You should see that Googlebot has successfully crawled your pages in recent days. Similarly, in Bing Webmaster Tools, check crawl activity in the "Crawl Control" or "Site Explorer" section.
If you notice a sudden drop in crawl volume or see "Blocked by robots.txt" / "Blocked by firewall" errors, your Cloudflare settings or robots.txt file may be misconfigured.
3. Cloudflare Analytics and Firewall Events
In your Cloudflare dashboard, navigate to Security → Events or Analytics → Security. Here you can see the User-agents of blocked requests. If Googlebot or Bingbot appear in this list, a Firewall Rule or Rate Limiting rule may be inadvertently blocking them. The AI bot setting itself should not block these bots.
Testing Method: Simulation with cURL
You can use cURL to test how Cloudflare responds to specific User-agents. Run these commands from your terminal:
# Send request as Googlebot
curl -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
-I https://yoursite.com/
# Send request as GPTBot
curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" \
-I https://yoursite.com/
Expected results:
- The Googlebot request should return
HTTP/1.1 200 OKor304 Not Modified. - The GPTBot request (if AI bot blocking is enabled) should return
HTTP/1.1 403 Forbiddenor a Cloudflare challenge page.
Important caveat: Cloudflare may verify User-agents by checking IP addresses. Real Googlebot comes from Google's official IP ranges; a Googlebot User-agent sent from your computer may be detected as fake and receive a challenge. This is a normal security measure. For definitive testing, rely on your server logs and Search Console.
Checking for robots.txt Conflicts
Bots blocked at Cloudflare's WAF level will be blocked even if marked as Allow in your robots.txt file. Conversely, a bot you've set to Disallow in robots.txt can still access your content if Cloudflare allows it (robots.txt is a courtesy protocol, not a security mechanism).
For consistency, we recommend this structure:
# robots.txt
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
If AI bot blocking is enabled in Cloudflare, the Disallow rules above are already enforced at the WAF level. However, keeping robots.txt current ensures consistency in environments not using Cloudflare (staging servers, CDN bypass, etc.).
Setting Names and Scope May Change
Cloudflare regularly updates its product interface and bot lists. The "AI Scrapers and Crawlers" setting may appear in the future as "AI Training Bots" or under another name. Additionally, Cloudflare may add new AI bots to the list or remove some from the category.
Recommended approach:
- Before enabling the setting in your Cloudflare dashboard, read the description text and "Learn more" link.
- Search for "bot management" or "AI bots" at
developers.cloudflare.comto check current documentation. - Monitor Search Console, Bing Webmaster, and your server logs for 24-48 hours after enabling the setting.
- If you notice a drop in search traffic, temporarily disable the setting and consult Cloudflare support.
Frequently Asked Questions
Will Google AI Overviews show my content when Cloudflare's AI bot blocking is enabled?
Google AI Overviews use content indexed by regular Googlebot and are managed through snippet controls (max-snippet, data-nosnippet). Since Cloudflare's AI bot blocking doesn't affect Googlebot, you can still appear in AI Overviews. If you don't want to appear in AI Overviews, add Disallow: / for User-agent: Google-Extended in robots.txt and use data-nosnippet on your pages. However, this won't affect your Google Search rankings.
Are ChatGPT user requests (ChatGPT-User) also blocked?
Cloudflare's AI bot list typically targets training crawlers (GPTBot). ChatGPT-User is a user-initiated agent triggered when a user asks ChatGPT to "summarize this page," and OpenAI indicates these requests may not honor robots.txt rules in the same way. Whether Cloudflare blocks this User-agent depends on the current list. If you want to block user-initiated AI requests as well, you can create a Firewall Rule in Cloudflare that manually blocks User-agents like ChatGPT-User, Claude-User, etc.
Do I still need to write to robots.txt if AI bot blocking is enabled in Cloudflare?
Technically, Cloudflare is already blocking at the WAF level, but we recommend also writing to robots.txt. Reasons: (1) If Cloudflare is bypassed (if your origin IP is known), robots.txt still applies. (2) If you don't use Cloudflare in staging or test environments, robots.txt provides protection. (3) Some bots check robots.txt and back off without sending requests to Cloudflare, reducing log clutter. Using both layers is the safest approach.
How frequently is Cloudflare's bot list updated?
Cloudflare updates the list as new AI bots emerge or existing bot behavior changes, but doesn't publish an official update schedule. When a significant bot is added (for example, when a new large language model company launches a crawler), Cloudflare may announce it on their blog or in documentation. If your site security is critical, regularly monitor new User-agents in Cloudflare's "Security Insights" or "Firewall Analytics" section and add manual Firewall Rules as needed.