7 Common robots.txt Mistakes That Hide Your Site from AI
Executive Summary: The most frequent robots.txt errors include blocking your entire site with Disallow: /, misusing wildcard characters, conflicting User-agent blocks that override each other, and serving stale files from CDN or staging environments in production. Most of these issues can be detected using Google Search Console's robots.txt report, URL Inspection tool, and direct browser verification.
1. Blocking Your Entire Site with Disallow: /
The most catastrophic robots.txt error is a configuration that blocks all crawlers from your entire site:
User-agent: *
Disallow: /
This configuration prevents all crawlers—including Googlebot, GPTBot, and ClaudeBot—from accessing any page on your site. The result: your site disappears from Google search results, and AI tools like ChatGPT cannot index your content.
Correct usage: If you only want to block specific directories:
User-agent: *
Disallow: /admin/
Disallow: /private/
To leave your entire site open to crawlers, either leave the Disallow: directive empty or omit it entirely:
User-agent: *
Disallow:
Testing method: View your file in Google Search Console → Settings → robots.txt report. Also verify the content directly by opening https://yoursite.com/robots.txt in your browser.
2. Misusing Wildcard Characters
The standard robots.txt syntax recognizes only * (matching any character sequence) and $ (end of line) as wildcards. However, some administrators attempt to use regex-like patterns:
User-agent: *
Disallow: /search?*query=
Disallow: /*.pdf$
The first line is correct; * blocks all search URLs containing "query=". The second line is also valid; $ blocks files ending in ".pdf". However, this is incorrect:
Disallow: /[0-9]+/
Square brackets and the + operator are invalid in robots.txt; this line only blocks the literal path /[0-9]+/, not pages with numeric IDs.
Correct usage: Use wildcards to block pages with numeric IDs:
Disallow: /product/*/reviews
This blocks all paths like /product/123/reviews and /product/abc/reviews.
Testing method: Enter a sample URL (e.g., /product/123/reviews) in Google Search Console's URL Inspection tool and check the "Crawl" tab to see whether it's blocked by robots.txt.
3. Conflicting User-agent Blocks That Override Each Other
In robots.txt, you define separate blocks for each User-agent. However, if you forget to add blank lines between blocks or define the same User-agent twice, crawlers will only apply the first matching block:
User-agent: GPTBot
Disallow: /blog/
User-agent: *
Disallow: /admin/
User-agent: GPTBot
Allow: /blog/public/
In the example above, there are two blocks for GPTBot. Crawlers typically apply only the first block; the second Allow rule is ignored. As a result, GPTBot sees the entire /blog/ directory as blocked, including /blog/public/.
Correct usage: Write a single block for each User-agent and separate blocks with blank lines:
User-agent: GPTBot
Disallow: /blog/
Allow: /blog/public/
User-agent: *
Disallow: /admin/
Also, avoid placing Allow rules before Disallow; while more specific rules take precedence in robots.txt, proper ordering improves readability.
Testing method: Validate your file with a robots.txt checker (e.g., Ryte Robots.txt Validator). You can also test your server's response for GPTBot with curl -A "GPTBot" https://yoursite.com/blog/public/ (though note that robots.txt is a courtesy protocol, not an access control mechanism; even if the server returns 200, the crawler should still respect robots.txt).
4. Stale robots.txt in CDN or Cache
If you use Cloudflare, Fastly, or another CDN, your robots.txt file may be cached on edge servers. Even after updating the file, crawlers may continue to see the old version:
# Old file (cached in CDN)
User-agent: *
Disallow: /
You've uploaded a new file:
User-agent: *
Disallow: /admin/
But Googlebot still sees the old Disallow: / rule and isn't crawling your site.
Solution: Purge the cache for robots.txt from your CDN control panel. In Cloudflare, go to "Caching → Purge Cache → Custom Purge" and enter https://yoursite.com/robots.txt. Also consider reducing the cache TTL for robots.txt or disabling caching entirely (Cache-Control: no-cache).
Testing method: Open https://yoursite.com/robots.txt in your browser and perform a hard refresh (Ctrl+F5). Check the response headers (Developer Tools → Network tab) for X-Cache: HIT or an Age: value, which indicates the file is being served from cache. After purging, you should see X-Cache: MISS.
5. Deploying Staging robots.txt to Production
In development or staging environments, you typically use a robots.txt that blocks all crawlers for testing purposes:
# staging.yoursite.com/robots.txt
User-agent: *
Disallow: /
However, if this file is accidentally deployed to your production site, your entire site becomes hidden from search engines and AI crawlers. This error is particularly common in CI/CD pipelines, file synchronization processes, or manual FTP uploads.
Prevention: Generate robots.txt content dynamically based on environment variables. For example, in Node.js/Express:
app.get('/robots.txt', (req, res) => {
if (process.env.NODE_ENV === 'production') {
res.type('text/plain');
res.send('User-agent: *\nDisallow: /admin/');
} else {
res.type('text/plain');
res.send('User-agent: *\nDisallow: /');
}
});
Alternatively, maintain two separate files (robots.staging.txt and robots.production.txt) in your deployment script and copy the correct one as robots.txt during deployment.
Testing method: After each deployment, verify https://yoursite.com/robots.txt on your live site. Also review the robots.txt report in Google Search Console; when the file changes, Search Console typically updates within a few hours.
6. Incorrect Comment Syntax
In robots.txt, comment lines begin with #. However, some administrators place the comment character mid-line:
User-agent: * # All crawlers
Disallow: /admin/ # Admin panel
This syntax is invalid. Crawlers ignore everything after the # character but still process what comes before it. In the example above, User-agent: * and Disallow: /admin/ work correctly, but for readability and standards compliance, comments should be on separate lines:
# All crawlers
User-agent: *
# Admin panel
Disallow: /admin/
Testing method: Validate your robots.txt file with a syntax checker. Also check for warning messages in Google Search Console's robots.txt report.
7. Incorrect Sitemap Path
When specifying your sitemap URL in robots.txt, using a relative path or omitting the protocol may prevent crawlers from finding your sitemap:
Sitemap: /sitemap.xml
The correct format requires a full URL:
Sitemap: https://yoursite.com/sitemap.xml
If you have multiple sitemaps, specify each on a separate line:
Sitemap: https://yoursite.com/sitemap.xml
Sitemap: https://yoursite.com/sitemap-news.xml
Testing method: Submit your sitemap URL in Google Search Console → Sitemaps and check its status. If you receive a "Couldn't fetch" error, correct the URL in robots.txt and resubmit in Search Console.
Comprehensive Testing Checklist
| Test Step | Tool/Method | Expected Result |
|---|---|---|
| File accessibility | Browser: https://yoursite.com/robots.txt |
200 OK, plain text content |
| Syntax validation | Google Search Console → robots.txt report | No errors or warnings |
| Specific URL testing | Search Console → URL Inspection → Crawl tab | "Allowed" or "Blocked by robots.txt" |
| Cache verification | Developer Tools → Network → Response Headers | X-Cache: MISS or current Last-Modified |
| Sitemap validation | Search Console → Sitemaps | "Success" status |
Frequently Asked Questions
How long does it take for robots.txt changes to take effect?
Google and other search engines check your robots.txt file before each crawl but may cache it (typically for 24 hours). Google Search Console doesn't offer a "re-fetch robots.txt" option; after updating the file, Googlebot will retrieve the new version within a few hours. If you use a CDN, you must manually purge the cache; otherwise, the old file may remain active for days.
I blocked GPTBot but ChatGPT still shows content from my site—why?
GPTBot is only the crawler OpenAI uses for model training. ChatGPT's search feature (ChatGPT Search) uses the OAI-SearchBot crawler, which respects robots.txt. However, when a user asks ChatGPT to "summarize this site," the ChatGPT-User agent (user-initiated requests) comes into play; OpenAI indicates that robots.txt rules may not apply the same way to these requests. To block user-initiated fetches, you'll need server or WAF-level IP/User-Agent filtering, though this is technically challenging and may produce false positives.
I blocked Google-Extended—did my Google Search ranking drop?
No. Google-Extended is a robots.txt control token (not a separate crawler) that only governs training/grounding usage for Gemini Apps and Vertex AI. Google Search ranking is determined by regular Googlebot and is unaffected by Google-Extended. However, Google AI Overviews (AI summaries in search results) are managed through regular Googlebot and snippet controls (nosnippet, max-snippet, data-nosnippet HTML attributes). Blocking Google-Extended does not affect AI Overviews.
Should I use meta robots or X-Robots-Tag instead of robots.txt?
Robots.txt prevents crawlers from fetching (crawling) a page; meta robots or X-Robots-Tag prevents a page from being indexed. If you want to remove a page from both crawling and indexing, blocking via robots.txt alone isn't sufficient—because if the crawler can't see the page, it can't read the meta tag either. For "don't index but allow crawling" scenarios, Google recommends not blocking in robots.txt and using <meta name="robots" content="noindex"> on the page. For sensitive content (e.g., user data), robots.txt is not a security mechanism; use server-level authentication (HTTP Basic Auth, IP whitelisting) or WAF rules instead.