Copy-Paste robots.txt Templates for Every Website Need
Executive Summary: Below you'll find copy-paste robots.txt templates for three fundamental scenarios: (1) full access for all bots, (2) blocking AI model training while permitting search crawlers and user-initiated access, (3) blocking all automated crawlers. We explain which bots each template affects and the practical implications for your site.
Scenario 1: Full Access for All Bots (Default Open)
This template permits all crawlers and AI bots to index your site and use it as training data. Appropriate for new sites, news publishers, or content producers seeking maximum visibility.
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml
Implications:
- Search crawlers like Googlebot and Bingbot will index all pages; you'll appear in Google Search and Bing results.
- Training bots such as GPTBot, ClaudeBot, and CCBot can use your content for model training.
- Your content may appear as a cited source in AI-powered search results including ChatGPT search, Perplexity, and Google AI Overviews.
- User-initiated agents (ChatGPT-User, Claude-User) are permitted; users can reference your site in AI conversations.
Important: This configuration does not protect your content's copyright. Bots that respect the robots exclusion protocol will access your site; for non-compliant crawlers, you'll need server-level blocking (IP filtering, User-Agent filtering).
Scenario 2: Block AI Model Training Only, Allow Search and User Access
This template allows your content to appear in search engines and permits user-initiated AI tool access, while blocking bulk data scraping for commercial model training. This represents a balanced approach for publishers and content creators.
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-User
Allow: /
User-agent: anthropic-ai
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
Implications:
- Blocked: GPTBot (OpenAI training), ClaudeBot (Anthropic training), CCBot (Common Crawl), Google-Extended (Gemini/Vertex AI training), and Applebot-Extended (Apple model training) will not crawl your site. Your content won't be used in these companies' future model training.
- Permitted: Googlebot and Bingbot index normally; your Google Search and Bing rankings remain unaffected. User-initiated agents like ChatGPT-User, Claude-User, and Perplexity-User continue to function; users can reference your site in AI conversations.
- Google AI Overviews: Google-Extended only controls Gemini apps and Vertex AI training; AI Overviews snippets are managed through Googlebot and meta directives like
max-snippetanddata-nosnippet. This template does not block AI Overviews. - Perplexity search: Blocking PerplexityBot stops Perplexity's indexing, but Perplexity-User (user requests) can still access your content. For complete blocking, set both to
Disallow: /.
Critical note: robots.txt is a courtesy protocol. For non-compliant crawlers, you must implement User-Agent-based blocking through server configuration (nginx, Apache) or WAF rules. Additionally, this template only affects listed bots; it should be updated as new or unknown AI crawlers emerge.
Scenario 3: Block All Bots (Maximum Restriction)
This template blocks all automated crawlers. Appropriate for development/staging sites, private membership platforms, or situations where public indexing is undesirable. However, you will not appear in search engines.
User-agent: *
Disallow: /
Implications:
- All crawlers including Googlebot and Bingbot cannot access your site; you won't be indexed in Google Search or Bing.
- All AI training bots (GPTBot, ClaudeBot, CCBot, etc.) and user-initiated agents (ChatGPT-User, Claude-User) are blocked.
- Previously indexed pages are not removed from search results; for that, you need a
noindexmeta tag or HTTP header. - Sitemap declaration is optional; since bots are already blocked it serves no purpose, though you may retain it if you plan to enable selective access in the future.
Alternative (development environment): To block only specific directories:
User-agent: *
Disallow: /admin/
Disallow: /test/
Disallow: /private/
Allow: /
Sitemap: https://example.com/sitemap.xml
In this case, the /admin/, /test/, and /private/ directories are closed to all bots, while the rest of the site remains accessible.
Implementation Considerations
robots.txt Location and Syntax
The file must be located in your site's root directory at https://example.com/robots.txt. It will not function in subdirectories (e.g., /blog/robots.txt). The filename must be lowercase robots.txt; Robots.txt or ROBOTS.TXT may not be recognized by some systems. Include blank lines between User-agent groups; this clarifies rules for different bot groups.
Validation and Testing
The robots.txt report in Google Search Console (Settings > robots.txt) displays your file's syntax and accessibility. Use the URL Inspection tool to test whether specific URLs are blocked; the legacy 'robots.txt tester' has been deprecated. Similar reports are available in Bing Webmaster Tools.
Security and Access Control
robots.txt is not a security mechanism; the file is publicly accessible and readable. Do not hide sensitive directories (admin panels, user data) with Disallow in robots.txt; this reveals their location to attackers. Instead, use server-level authentication (HTTP Basic Auth, OAuth) or IP allowlisting. For non-compliant crawlers, implement User-Agent filtering or rate limiting in nginx/Apache.
About llms.txt
llms.txt is a file format proposed in 2024 that aims to provide AI models with summary information about a site. However, this is not an official standard, and it has not been confirmed that major providers like Google, OpenAI, or Anthropic use it for ranking, attribution, or access control. You may add it as an experimental approach, but it does not replace robots.txt and you should not expect miraculous results. Check official documentation for current status.
Frequently Asked Questions
Does the Scenario 2 template block Google AI Overviews?
No. Google-Extended only controls model training for Gemini apps and Vertex AI; it does not affect AI Overviews ranking or snippet display in Google Search. To control citations in AI Overviews, use max-snippet, data-nosnippet, or nosnippet meta directives. If you block Googlebot, you'll be excluded from both regular search and AI Overviews.
How long after updating robots.txt do changes take effect?
Most search crawlers check robots.txt once daily; Google typically updates within a few hours. You can verify the file has been read through the 'robots.txt report' in Search Console. However, removing previously indexed pages from search results may take weeks. For rapid removal, use the 'Removals' tool in Search Console or add noindex to pages.
Does blocking ChatGPT-User and Claude-User prevent users from using my site in AI conversations?
Partially. ChatGPT-User and Claude-User are agents that fetch pages when a user makes a request. If you block them, when users say "read this URL" in an AI conversation, the bot cannot access it. However, OpenAI indicates that robots.txt rules may not be enforced identically for user-initiated requests; check official documentation for precise behavior. For complete blocking, implement User-Agent filtering at the server level.
Does full blocking in Scenario 3 protect against legal liability?
No. robots.txt is a courtesy protocol, not a legal contract or access control mechanism. It does not provide definitive protection regarding copyright, GDPR, or other legal matters. If you have concerns about your content being used in AI training, seek legal counsel and update your terms of service if necessary. robots.txt guides well-behaved bots; for malicious scrapers, you need server security, rate limiting, and legal recourse.