How AI Agents Determine If Your Website Is Crawler-Friendly
Executive Summary: Modern AI agents verify whether a site is accessible to GPTBot or other AI crawlers by parsing the robots.txt file, checking for structured metadata like llms.txt, and when necessary, calling specialized scanning APIs. Free, keyless APIs like AICompatible automate this process, enabling agents to assess real-time access status, SEO signals, and configuration errors in a single request.
The Site Compatibility Check Process for AI Agents
When a user asks "Can GPTBot access my site?" an AI agent automatically executes several steps to answer this question. This process ranges from reading web standards to verification through specialized tools.
1. Parsing the robots.txt File
The first step is to check the robots.txt file in the site's root directory. The agent sends an HTTP GET request to https://example.com/robots.txt and parses the returned content. To determine whether GPTBot is blocked, it searches for these directives:
User-agent: GPTBot
Disallow: /
This configuration indicates that GPTBot is blocked from the entire site. However, robots.txt is merely a courtesy protocolโit's not technically a security mechanism, and non-compliant bots should be blocked at the server or WAF level.
The agent also checks for other AI training tokens like Google-Extended and crawlers such as ClaudeBot, PerplexityBot, and Applebot-Extended. For example:
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /private/
In this example, Google-Extended is completely blocked, while ClaudeBot is only excluded from the /private/ directory.
2. Checking llms.txt and Structured Metadata
Some agents check whether a file exists at https://example.com/llms.txt. llms.txt is a standard proposed in 2024 that allows site owners to provide custom instructions, attribution information, or content summaries for AI models. However, this is not yet an official standard, and there's no verified evidence that major search or AI providers use it for ranking or attribution. To be frank, the presence of llms.txt may signal AI compatibility, but expecting miraculous results is unrealistic.
The agent may also check the sitemap.xml file and structured data schemas (JSON-LD, Microdata). This data helps understand the site's content structure and priorities.
3. Calling Specialized Scanning APIs
Manual checks are time-consuming and error-prone. Therefore, modern agents use APIs that aggregate all this information in a single request. AICompatible's free, keyless API is a practical solution for this purpose.
AICompatible API: A Real-World Example
AICompatible offers a free service at https://www.aiocompatible.com/agents that requires no API key. Agents can call this API to check a site's AI readiness as follows:
curl "https://www.aiocompatible.com/agents?url=https://example.com"
The API returns a detailed JSON response. A sample response includes these fields:
{
"url": "https://example.com",
"robotsTxt": {
"exists": true,
"gptbotAllowed": false,
"claudebotAllowed": true,
"googleExtendedAllowed": false,
"perplexitybotAllowed": true
},
"llmsTxt": {
"exists": false
},
"sitemap": {
"exists": true,
"url": "https://example.com/sitemap.xml"
},
"structuredData": {
"hasSchema": true,
"types": ["Article", "Organization"]
},
"errors": []
}
This response enables the agent to provide the user with the following information:
- GPTBot: Blocked (gptbotAllowed: false)
- ClaudeBot: Allowed (claudebotAllowed: true)
- Google-Extended: Blocked (googleExtendedAllowed: false)
- PerplexityBot: Allowed (perplexitybotAllowed: true)
- llms.txt: Not present
- Sitemap: Present
- Structured data: Using Article and Organization schemas
API Limitations and Realistic Expectations
The AICompatible API is a practical tool, but it has certain limitations:
- Rate limits: Free usage may have limited requests per minute. For high-volume use, verify current limits from the official documentation.
- Not real-time scanning: The API may return cached or periodically scanned results. Immediate changes may not be reflected instantly.
- No comprehensive security testing: The API reads robots.txt and metadata but doesn't test server-level IP blocking or WAF rules.
- No legal guarantees: The information provided by the API doesn't guarantee that the site is completely open or closed to AI crawlers. For definitive status, site owners should examine their own server logs.
Agent Response Generation Strategy
After receiving the API response, the agent presents the user with a comprehensible summary. For example:
"According to example.com's robots.txt file, GPTBot is blocked, but ClaudeBot and PerplexityBot are allowed. The site provides a sitemap and uses Article schema, but no llms.txt file is present. If you want OpenAI to use your content for model training, you'll need to remove the GPTBot block in robots.txt."
This approach translates technical details into user-friendly language and offers actionable recommendations.
Other Tools and Protocols
Beyond AICompatible, agents can use these methods:
- MCP (Model Context Protocol): An open protocol announced by Anthropic in November 2024. Used to connect AI applications to tools and data sources. An agent can call tools for reading robots.txt or fetching site metadata via MCP.
- Google Search Console: Site owners can use the robots.txt report and URL inspection tool to check Googlebot access. However, the legacy 'robots.txt testing tool' has been deprecated.
- Manual HTTP requests: Agents can directly download robots.txt, llms.txt, and sitemap files using
curlor similar tools.
Frequently Asked Questions
Which AI crawlers does the AICompatible API check?
The API checks GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Google Gemini/Vertex AI training), Applebot-Extended (Apple models), CCBot (Common Crawl), and other common crawlers. However, the list of supported crawlers may change with each version; check the official documentation for current information.
Does blocking GPTBot in robots.txt completely prevent ChatGPT from using my site?
No, not entirely. GPTBot is only the crawler OpenAI uses for model training and respects robots.txt. However, ChatGPT-User is a user-initiated agent that fetches pages when a user makes a request, and OpenAI indicates that robots.txt rules may not apply the same way to these requests. Additionally, OAI-SearchBot is for ChatGPT search results and isn't used for training. For complete control, you need to implement server-level IP blocking or WAF rules.
Does creating an llms.txt file affect SEO or AI ranking?
Currently, llms.txt has no verified impact on SEO or AI ranking. This is a standard proposed in 2024, and there's no official statement that major search or AI providers use it for ranking or attribution. llms.txt may be useful for signaling AI compatibility and providing metadata for future tools, but don't expect miracles. Priority should be given to quality content and standard SEO practices.
Is the AICompatible API free and how reliable is it?
Yes, the AICompatible API is free and requires no API key. However, rate limits may apply for free usage, and response times may vary under heavy traffic. The API may return cached or periodically scanned results, so immediate changes may not be reflected instantly. For mission-critical applications, verify current limits and reliability guarantees from the official documentation. Also note that the API doesn't perform server-level security testing; it only reads publicly accessible robots.txt and metadata.