How to Detect Fake Web Crawlers Pretending to Be GPTBot or Googlebot
Executive Summary: Because the User-Agent header is simply a text string in an HTTP request that can be easily modified, any bot can masquerade as GPTBot, Googlebot, or any other crawler. To verify authentic identity, you must check whether the request originates from an IP address within the official provider's range and confirm through reverse DNS lookup that the IP genuinely belongs to that organization. Robots.txt rules only apply to well-behaved bots; spoofed or malicious crawlers must be blocked at the server or firewall level.
Why Is User-Agent Spoofing So Trivial?
In the HTTP protocol, the User-Agent is merely a header line that the client can freely set. Using tools like curl, wget, or Python requests, sending any User-Agent value takes seconds:
curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" https://example.com/
This command appears to the server as GPTBot, yet it could originate from an entirely different source. If you only examine the User-Agent header in server logs, you cannot distinguish which requests actually come from OpenAI's crawler. This is why User-Agent alone is not a reliable proof of identity.
Verification Logic Using Official IP Ranges and Reverse DNS
Major search engines and AI providers publish the IP blocks from which their crawlers operate or standardize their reverse DNS records. Verification follows a two-step process:
1. Reverse DNS Lookup (PTR Record)
Perform a reverse DNS query on the requesting IP address to determine which domain name it points to. For example, Googlebot IPs return PTR records like *.googlebot.com or *.google.com. Then query the A/AAAA record of that domain name to verify it resolves back to the same IP (forward-confirmed reverse DNS).
2. Official IP Range Verification
Some providers publish IP lists in JSON or text format. You verify whether the requesting IP falls within a CIDR block in this list. For instance, OpenAI provides GPTBot IP ranges in a JSON file; Anthropic and Perplexity offer similar lists.
Both methods are highly resistant to IP spoofing because they require TCP handshake completion, making it practically infeasible for an attacker to forge both the IP and network path. This makes IP-based verification far more reliable than User-Agent strings.
Command-Line Verification Examples
Googlebot Verification (Linux/macOS)
Assume you see a request in server logs from IP 66.249.66.1 with a Googlebot User-Agent:
# 1. Reverse DNS lookup
host 66.249.66.1
# Output: 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
# 2. Forward DNS lookup (verification)
host crawl-66-249-66-1.googlebot.com
# Output: crawl-66-249-66-1.googlebot.com has address 66.249.66.1
If the IP address points to a googlebot.com domain and that domain resolves back to the same IP, this is an authentic Googlebot request. If the PTR record shows a different domain or the forward lookup returns a different IP, the User-Agent is spoofed.
GPTBot IP Range Verification
OpenAI publishes GPTBot IPs at: https://openai.com/gptbot.json. Download the file and verify whether an IP falls within these ranges:
# Download the JSON file
curl -s https://openai.com/gptbot.json > gptbot.json
# Example IP: 20.171.210.5
# Verification using Python (ipaddress module)
python3 <<EOF
import json, ipaddress
with open("gptbot.json") as f:
data = json.load(f)
ip = ipaddress.ip_address("20.171.210.5")
ranges = [ipaddress.ip_network(r["ipv4Prefix"]) for r in data["prefixes"] if "ipv4Prefix" in r]
if any(ip in net for net in ranges):
print("✓ IP within authentic GPTBot range")
else:
print("✗ IP not in GPTBot range")
EOF
This script checks whether the given IP belongs to one of the CIDR blocks published by OpenAI. Similar JSON files are available for Anthropic (ClaudeBot) and Perplexity (PerplexityBot).
Bingbot and Other Microsoft Crawlers
For Bingbot, use the reverse DNS method; PTR records return *.search.msn.com:
host 40.77.167.1
# Output: 1.167.77.40.in-addr.arpa domain name pointer msnbot-40-77-167-1.search.msn.com.
host msnbot-40-77-167-1.search.msn.com
# Output: msnbot-40-77-167-1.search.msn.com has address 40.77.167.1
Server or WAF-Level Blocking
Robots.txt is merely a courtesy protocol for well-behaved bots. To block spoofed or malicious crawlers, you must use server configuration or web application firewall (WAF) rules:
Nginx Example
# Allow only verified Googlebot IPs
geo $is_googlebot {
default 0;
66.249.64.0/19 1;
# Additional Google IP blocks...
}
server {
location / {
if ($http_user_agent ~* "Googlebot") {
set $bot_check "A";
}
if ($is_googlebot = 0) {
set $bot_check "${bot_check}B";
}
if ($bot_check = "AB") {
return 403;
}
}
}
This configuration rejects requests with "Googlebot" in the User-Agent that don't originate from Google's IP ranges.
Cloudflare WAF
Cloudflare automatically identifies verified bots using the cf.bot_management.verified_bot field. Create a custom rule to block spoofed User-Agents:
(http.user_agent contains "GPTBot" and not cf.bot_management.verified_bot)
This rule blocks or challenges requests using the GPTBot User-Agent that fail Cloudflare's verification system.
Which Verification Method for Which Bot?
| Bot | Verification Method | Documentation |
|---|---|---|
| Googlebot | Reverse DNS (*.googlebot.com) | Google Documentation |
| Bingbot | Reverse DNS (*.search.msn.com) | Bing Documentation |
| GPTBot | IP range JSON | https://openai.com/gptbot.json |
| ClaudeBot | IP range JSON | https://docs.anthropic.com/en/ip-addresses |
| PerplexityBot | IP range JSON | https://docs.perplexity.ai/docs/perplexitybot |
| Applebot | Reverse DNS (*.applebot.apple.com) | Apple Documentation |
Frequently Asked Questions
Is reverse DNS lookup always reliable?
Reverse DNS is reliable as long as DNS records are properly configured and the attacker hasn't compromised the DNS server. Performing forward-confirmed reverse DNS (checking both PTR and A/AAAA records) increases reliability. However, since DNS records can theoretically be manipulated, the most robust approach for critical applications is to use both IP range lists and reverse DNS verification together.
How frequently are IP range lists updated?
Providers like OpenAI, Anthropic, and Perplexity update their IP lists regularly, but don't guarantee update frequency. We recommend automatically fetching and caching these lists weekly or monthly on your server. For major providers like Google and Bing, which rarely change IP ranges, the reverse DNS method is more practical.
Isn't blocking User-Agents in robots.txt sufficient?
No. Robots.txt is merely a directive for well-behaved bots and contains no technical enforcement mechanism. A malicious bot can completely ignore robots.txt or continue accessing your site using a different User-Agent. For actual blocking, you must use server configuration, firewall rules, or a WAF.
How do I see the real IP when using Cloudflare or a CDN?
Reverse proxies like Cloudflare pass the real visitor IP in headers such as CF-Connecting-IP or X-Forwarded-For. Configure your server logs to read these headers. For Nginx, use the real_ip_header CF-Connecting-IP; directive and define Cloudflare IP ranges with set_real_ip_from. Otherwise, all requests will appear to originate from Cloudflare's own IPs, making bot verification impossible.