How to Control AI Bot Access to Your WordPress Site
Executive Summary: WordPress generates a virtual (dynamic) robots.txt file by default. SEO plugins like Yoast SEO or Rank Math allow you to customize this virtual file, or you can upload a physical robots.txt to your root directory (which overrides the virtual file). To block AI bots, add User-agent directives; for non-compliant crawlers, implement server-level blocking via .htaccess or WAF.
How WordPress Handles Robots.txt
Out of the box, WordPress automatically serves a robots.txt file at example.com/robots.txt. This file doesn't physically exist on your server—it's dynamically generated by WordPress's do_robots function. The default output looks like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
If you upload a physical robots.txt file to your root directory (public_html or www), WordPress stops generating the virtual file and defers to the physical one. This is a critical distinction: the two methods don't coexist—a physical file always takes precedence.
Method 1: Editing the Virtual Robots.txt via SEO Plugins
Using Yoast SEO
Yoast SEO provides a built-in editor for customizing the virtual robots.txt:
- Navigate to Yoast SEO → Tools → File editor.
- Click "Create robots.txt file" (first-time setup).
- Add your AI bot rules in the editor. Example:
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-content/uploads/private/
Yoast stores changes in the database and modifies the output via the do_robots filter. It doesn't create a physical file.
Using Rank Math
Rank Math offers a similar interface:
- Go to Rank Math → General Settings → Edit Robots.txt.
- Add your directives above the default rules:
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Rank Math also generates a virtual file; if a physical robots.txt exists, plugin settings won't apply.
All in One SEO (AIOSEO)
In AIOSEO, navigate to General Settings → Robots.txt to edit. The interface mirrors Yoast and Rank Math—add your rules and save.
Method 2: Uploading a Physical Robots.txt File
For greater control or if you're not using an SEO plugin, create a physical file:
- Open a text editor (Notepad++, VS Code, etc.).
- Write your rules (refer to examples above).
- Save as
robots.txt(no extension, lowercase). - Upload via FTP or cPanel File Manager to the WordPress root directory (same folder as
wp-config.php).
After uploading, verify by visiting example.com/robots.txt in your browser. WordPress will no longer generate the virtual file.
Blocking AI Bots: Practical Examples
The following table summarizes common AI crawlers and their control directives:
| Bot Name | Purpose | Robots.txt Directive |
|---|---|---|
| GPTBot | OpenAI model training | User-agent: GPTBot |
| Google-Extended | Gemini/Vertex AI training (does not affect Search ranking) | User-agent: Google-Extended |
| ClaudeBot | Anthropic model training | User-agent: ClaudeBot |
| CCBot | Common Crawl archive | User-agent: CCBot |
| PerplexityBot | Perplexity search indexing | User-agent: PerplexityBot |
| Applebot-Extended | Apple foundation model training | User-agent: Applebot-Extended |
Important: User-initiated agents like ChatGPT-User, Claude-User, and Perplexity-User may not honor robots.txt rules in the same way. OpenAI notes that user requests may be subject to different policies. Blocking these requires server-level controls.
Blocking Non-Compliant Bots via .htaccess
Robots.txt is a courtesy protocol—non-compliant bots require .htaccess (Apache) or server configuration enforcement. Add these rules to your WordPress root .htaccess file:
# Add ABOVE WordPress rules
RewriteEngine On
# Block GPTBot
RewriteCond %{HTTP_USER_AGENT} GPTBot [NC]
RewriteRule .* - [F,L]
# Block CCBot
RewriteCond %{HTTP_USER_AGENT} CCBot [NC]
RewriteRule .* - [F,L]
# Block ClaudeBot
RewriteCond %{HTTP_USER_AGENT} ClaudeBot [NC]
RewriteRule .* - [F,L]
# Block PerplexityBot
RewriteCond %{HTTP_USER_AGENT} PerplexityBot [NC]
RewriteRule .* - [F,L]
These rules return a 403 Forbidden response to specified User-Agents. The [NC] flag enables case-insensitive matching. For Nginx, add similar if blocks in your server configuration (though Nginx if usage requires caution—consult your hosting provider).
Controlling Google AI Overviews and Snippets
Google AI Overviews (formerly SGE) uses content crawled by regular Googlebot and is not controlled by Google-Extended. To limit AI Overviews usage, employ snippet control meta tags:
<meta name="robots" content="max-snippet:50, max-image-preview:standard">
Or mark specific sections with data-nosnippet:
<div data-nosnippet>
This text won't be used in AI Overviews.
</div>
Yoast SEO and Rank Math allow per-page nosnippet configuration. Use these methods if you want to preserve Google Search rankings while limiting AI usage.
Common Mistakes to Avoid
- Mixing physical and virtual files: Once you upload a physical robots.txt, plugin settings won't apply. Check
example.com/robots.txtto verify which is active. - Omitting blank lines between User-agent groups: Leave a blank line between each
User-agentblock; otherwise, some bots may misinterpret rules. - Blocking Googlebot entirely:
User-agent: Googlebot / Disallow: /removes your site from Google. To block only AI training, useGoogle-Extended. - Treating robots.txt as a security tool: Robots.txt doesn't stop malicious bots. Secure sensitive directories with
.htaccess, password protection, or WAF. - Not testing changes: Use Google Search Console's Settings → robots.txt report and URL Inspection tool to validate rules. The legacy "robots.txt tester" has been deprecated.
- Relying on user-initiated agents like ChatGPT-User: These may not honor robots.txt consistently; server-level blocking may be necessary.
The Reality of llms.txt
The llms.txt file, proposed in 2024, aims to provide AI models with summary information about your site. However, it's not an official standard, and there's no verified evidence that major providers like Google, OpenAI, or Anthropic use it for ranking, attribution, or training. You can test it as an experimental approach, but it doesn't replace robots.txt or meta tags.
Frequently Asked Questions
Should I use a physical robots.txt or a plugin-managed virtual file?
If you're using an SEO plugin (Yoast, Rank Math), virtual file management is more practical—you can make quick changes from the WordPress dashboard. If you're not using a plugin or want full server-level control, a physical file is preferable. In either case, verify which is active by checking example.com/robots.txt.
Does blocking Google-Extended affect Google Search rankings?
No. Google-Extended controls only training/grounding usage for Gemini apps and Vertex AI; it doesn't affect Google Search rankings. As long as you don't block regular Googlebot crawling, your site will continue to appear in Google.
How do I block bots that ignore robots.txt?
Robots.txt is a courtesy protocol and can't stop non-compliant bots. Block them via .htaccess (Apache) or Nginx configuration using User-Agent filtering (see .htaccess example above). Alternatively, use WAF services like Cloudflare or Sucuri, or your hosting provider's server-level firewall.
I don't want to appear in AI Overviews but still want Google Search visibility—how?
Google AI Overviews uses content crawled by regular Googlebot and isn't controlled by Google-Extended. To limit AI Overviews usage, employ snippet controls like max-snippet, max-image-preview, or data-nosnippet. Yoast SEO and Rank Math offer per-page nosnippet settings. This reduces AI visibility while preserving rankings. However, Google's AI Overviews policies may evolve—monitor official documentation for updates.