July 14, 2026 • By Elena Rostova, Lead Dev Engineer • 6 min read

A website’s robots.txt file used to be simple: configure rules for Googlebot, Bingbot, and maybe a few aggressive scrapers. Today, developers must manage a new breed of crawlers: AI search agents and LLM training scrapers. In this guide, we break down how to configure your access rules to maximize visibility in AI search while preventing unauthorized scraping of your proprietary data.

Understanding the Two Types of AI Crawlers

It is crucial to distinguish between the two categories of AI scrapers:

1. AI Search Crawlers (Real-time): These bots crawl pages dynamically to resolve live queries and cite sources. Examples include PerplexityBot, GPTBot (in search mode), and Claude-Search. Blocking these means your site will not be cited as a live source.
2. LLM Training Scrapers: These bots download text data in bulk to train future AI foundation models. Examples include Google-Extended, GPTBot (in training mode), and CCBot. Blocking these prevents your text from being used to train future iterations of models, but does not affect real-time search citations.

How to Allow AI Crawlers (Recommended for AIO)

If you want to be cited in ChatGPT Search, Perplexity, and Claude summaries, insert these configurations in your robots.txt file located at the root of your domain:

# Allow Perplexity AI Live Search
User-agent: PerplexityBot
Allow: /

# Allow OpenAI ChatGPT Live Search
User-agent: GPTBot
Allow: /

# Allow Anthropic Claude Live Search
User-agent: Claude-Search
Allow: /

# Allow Apple Intelligence Search Agent
User-agent: Applebot-Extended
Allow: /

The Strategic Middleware Option

What if you want Perplexity and ChatGPT to cite your blog posts, but you don't want them scraping your proprietary search layouts, API documentation, or internal user directories? You can restrict pathing precisely:

User-agent: GPTBot
Allow: /blog/
Allow: /resources/
Disallow: /api/
Disallow: /search/
Disallow: /admin/

Checking for Cloudflare and CDN Blocks

Often, developers allow AI bots in their robots.txt, but wonder why their site still fails to appear in AI search results. The culprit is typically Web Application Firewall (WAF) rules on CDNs like Cloudflare, AWS CloudFront, or Fastly. Make sure your firewalls are configured to allow legitimate user-agent headers from verified AI bots.