November 03, 2026 • By AICompatible Team • 7 min read

How to Detect AI Bots Crawling Your Website Using Server Logs

# Identifying AI Bots in Server Logs: Which Bots Are Crawling Your Site and How Often?

TL;DR: Filter AI bots in your Apache or Nginx access logs using grep 'GPTBot\|ClaudeBot\|PerplexityBot', then analyze IP addresses, timestamps, and request volumes with awk. Verify legitimate bots by comparing User-Agent strings against reverse DNS records and official IP ranges to detect spoofed crawlers.

Basic Grep Commands: Which Bots Have Visited?

Your server logs are typically located at /var/log/apache2/access.log or /var/log/nginx/access.log. Start by searching for known AI bot identifiers:

grep -i 'GPTBot\|ClaudeBot\|PerplexityBot\|Google-Extended\|CCBot\|Applebot-Extended' /var/log/nginx/access.log

The -i flag enables case-insensitive matching. If you have rotated log files (such as access.log.1, access.log.2.gz):

zgrep -i 'GPTBot\|ClaudeBot' /var/log/nginx/access.log*

zgrep reads compressed (.gz) files directly. Each output line represents a single HTTP request, displaying the IP address, timestamp, request path, and User-Agent string.

Request Volume and IP Analysis with Awk

To determine how many requests each bot has sent:

grep -i 'GPTBot' /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn

This command filters GPTBot requests, extracts IP addresses from the first column ($1), sorts them, and counts occurrences. Output appears as:

    142 40.88.21.235
     89 13.66.139.23
     12 52.167.144.199

The left column shows request count; the right shows the IP address. The highest-volume sources appear first.

Filtering by Date Range

To view bot activity for a specific date (e.g., January 15, 2025):

awk '$4 ~ /15\/Jan\/2025/ && /GPTBot/' /var/log/nginx/access.log

In Nginx logs, the fourth field ($4) typically contains timestamps in [15/Jan/2025:14:23:10 +0000] format. Apache logs follow a similar structure.

Which Pages Were Crawled?

To identify which URLs GPTBot accessed:

grep -i 'GPTBot' /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

The seventh field ($7) contains the request path (e.g., /blog/ai-seo-strategies). This output displays the top 20 most-crawled pages. If you see administrative paths like /wp-admin or /login.php, this indicates abnormal bot behavior and potential spoofing.

How to Detect Spoofed Bots

Malicious crawlers can forge User-Agent headers. Verify whether a GPTBot or ClaudeBot is legitimate using two methods:

1. Reverse DNS Verification

First, perform a reverse DNS lookup on the suspicious IP:

host 40.88.21.235

Expected output:

235.21.88.40.in-addr.arpa domain name pointer msnbot-40-88-21-235.search.msn.com.

Then resolve this domain name back to an IP:

host msnbot-40-88-21-235.search.msn.com

If the result doesn't match the original IP, it's a spoofed bot. OpenAI's GPTBot typically originates from *.openai.com or major cloud provider ranges; Anthropic's ClaudeBot comes from *.anthropic.com or AWS ranges. However, these companies don't always publish comprehensive IP documentation, making verification challenging.

2. Verify Against Official IP Ranges

Some organizations publish official IP lists:

OpenAI (GPTBot): No official IP list published; typically originates from AWS, Azure, GCP, and other major cloud providers.
Anthropic (ClaudeBot): Generally uses AWS IP ranges, but no definitive list exists.
Common Crawl (CCBot): Operates on AWS infrastructure; expect commoncrawl.org reverse DNS records.
Google (Googlebot, Google-Extended): Official verification documentation available.

Run a whois query on suspicious IPs:

whois 13.66.139.23 | grep -i 'OrgName\|Organization'

If the organization is a major provider like Microsoft, Amazon, or Google and the User-Agent matches, it's likely legitimate. However, if it originates from an obscure hosting provider while claiming to be GPTBot, treat it as suspicious.

Anomalous Behavior Indicators

Excessive request rates: A "GPTBot" sending hundreds of requests per minute is suspicious; legitimate bots typically respect Crawl-delay directives in robots.txt.
Administrative path scanning: Bots targeting /wp-login.php, /admin, /.env are conducting reconnaissance, not training data collection.
HTTP status patterns: Bots generating numerous 404 or 403 errors may be probing site structure.

Summarizing All AI Bots with a Single Command

To view request counts for all known AI bots in a summary table:

awk '{print $1, $12}' /var/log/nginx/access.log | \
  grep -iE 'GPTBot|ClaudeBot|PerplexityBot|Google-Extended|CCBot|Applebot-Extended|OAI-SearchBot|ChatGPT-User' | \
  awk '{print $2}' | sort | uniq -c | sort -rn

In Nginx logs, the twelfth field ($12) typically contains the User-Agent (this varies by log format; verify your configuration). Output:

   1523 "GPTBot/1.0"
    876 "ClaudeBot/1.0"
    234 "PerplexityBot/1.0"
     89 "CCBot/2.0"

Validating Log Format

If User-Agent data doesn't appear, verify your log format. In Nginx /etc/nginx/nginx.conf:

log_format combined '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

Here $http_user_agent is the final field. Apache uses %{User-Agent}i. Adjust your awk column numbers according to your specific format.

Blocking or Allowing Bots

To block specific bots, server-level blocking is more effective than robots.txt (which is a courtesy protocol, not enforceable). Nginx example:

if ($http_user_agent ~* (GPTBot|ClaudeBot)) {
    return 403;
}

For Apache in .htaccess or httpd.conf:

SetEnvIfNoCase User-Agent "GPTBot" bad_bot
SetEnvIfNoCase User-Agent "ClaudeBot" bad_bot
Deny from env=bad_bot

However, this method relies on User-Agent strings, which spoofed bots can modify. More robust approaches include IP-based blocking or implementing a Web Application Firewall (WAF).

Frequently Asked Questions

Are GPTBot and OAI-SearchBot the same?

No. GPTBot is OpenAI's crawler for model training and respects robots.txt. OAI-SearchBot powers ChatGPT's search feature and isn't used for training. ChatGPT-User is the agent that fetches pages when users make specific requests; OpenAI notes that robots.txt rules may not apply uniformly to these requests. You may see all three in your logs.

Does blocking Google-Extended affect search rankings?

No. Google-Extended is a robots.txt control token (not a separate crawler); it governs training/grounding usage for Gemini apps and Vertex AI but doesn't affect Google Search rankings. Google AI Overviews are managed through standard Googlebot and snippet controls (nosnippet, max-snippet, data-nosnippet). Blocking Google-Extended won't impact your search visibility.

My logs show many "Mozilla/5.0" entries—are these bots?

Most modern bots include "Mozilla/5.0" prefixes in User-Agent strings for browser compatibility. Example: Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot). "Mozilla/5.0" alone doesn't indicate a bot; look for subsequent identifiers like "compatible; GPTBot". If you only see standard browser strings like "Mozilla/5.0 (Windows NT 10.0; Win64; x64)", this could be a genuine user or a spoofed bot. IP and behavioral analysis is required.

I've detected a spoofed bot—what should I do?

First, block the IP at the server level using iptables, firewall rules, or a WAF. If attacks originate from multiple IPs, implement rate limiting: for example, temporarily block IPs exceeding 10 requests per minute. Tools like Fail2ban automate this process. User-Agent-based blocking alone is insufficient since attackers easily modify headers. For large-scale attacks, consider DDoS protection services like Cloudflare or AWS Shield.