November 04, 2026 • By AICompatible Team • 7 min read

How to Block AI Crawlers at the Server Level with Nginx and Apache

# Blocking AI Bots with Nginx and Apache (.htaccess): Ready-to-Use Rules

Executive Summary: robots.txt is a courtesy protocol and is not binding for non-compliant bots. You can implement server-level blocking based on User-Agent headers using the map directive in Nginx or RewriteCond in Apache. Misconfigured rules can take down your entire site; always test with curl -A "Bot-Name" https://yoursite.com before deploying to production.

Why Server-Level Blocking Is Necessary

The robots.txt file is an instruction set that tells crawlers which content they should not access. However, training bots like GPTBot, ClaudeBot, and CCBot, as well as malicious scrapers, may ignore these rules. Server-level blocking provides definitive control by rejecting specific User-Agents before the HTTP request is processed.

Important: This method becomes ineffective if a bot modifies its User-Agent header. However, legitimate bots (GPTBot, ClaudeBot, etc.) typically use identifiable User-Agents, and you can also verify their IP ranges for additional validation.

User-Agent Blocking with Nginx

Method 1: map Directive (Recommended)

In Nginx, the map directive checks the User-Agent header and assigns a variable. This method is performant and provides a clean structure for multiple bots. Add this to the http block in your nginx.conf or site configuration file:

http {
    map $http_user_agent $blocked_bot {
        default 0;
        ~*GPTBot 1;
        ~*ChatGPT-User 1;
        ~*ClaudeBot 1;
        ~*Claude-Web 1;
        ~*CCBot 1;
        ~*PerplexityBot 1;
        ~*Bytespider 1;
        ~*Amazonbot 1;
    }

    server {
        listen 80;
        server_name yoursite.com;

        if ($blocked_bot) {
            return 403;
        }

        # Other configurations...
    }
}

Explanation: ~* performs case-insensitive regex matching. For matching User-Agents, the $blocked_bot variable is set to 1, and the if block returns a 403 Forbidden response. The default 0; line allows all unlisted bots.

Method 2: if Directive (For Simple Cases)

For blocking a single bot, you can use this directly in the server or location block:

server {
    listen 80;
    server_name yoursite.com;

    if ($http_user_agent ~* (GPTBot|ClaudeBot|CCBot)) {
        return 403;
    }

    # Other configurations...
}

Warning: The if directive in Nginx can lead to unexpected behavior in some cases. For complex rules, map should be preferred.

Reloading Configuration

sudo nginx -t  # Syntax check
sudo systemctl reload nginx

User-Agent Blocking with Apache (.htaccess)

Blocking with mod_rewrite

In Apache, add the following rules to your .htaccess file or <Directory> block. The mod_rewrite module must be enabled:

RewriteEngine On

# Block AI training bots
RewriteCond %{HTTP_USER_AGENT} GPTBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ChatGPT-User [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ClaudeBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} CCBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} PerplexityBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Bytespider [NC]
RewriteRule .* - [F,L]

Explanation: [NC] enables case-insensitive matching, [OR] creates an OR condition with the next rule, [F] returns a 403 Forbidden response, and [L] marks this as the last rule (stops processing if matched).

Blocking with mod_authz_core (Apache 2.4+)

A more modern and performant approach:

<IfModule mod_authz_core.c>
    <RequireAll>
        Require all granted
        Require not env blocked_bot
    </RequireAll>
</IfModule>

SetEnvIfNoCase User-Agent "GPTBot" blocked_bot
SetEnvIfNoCase User-Agent "ClaudeBot" blocked_bot
SetEnvIfNoCase User-Agent "CCBot" blocked_bot
SetEnvIfNoCase User-Agent "PerplexityBot" blocked_bot

Risk of Misconfigured Rules Taking Down Your Entire Site

Server-level blocking rules can affect all visitors when they contain syntax or logic errors. Common mistakes include:

Incorrect regex: Spacing errors like if ($http_user_agent ~ *GPTBot) can prevent Nginx from starting.
Overly broad rules: RewriteCond %{HTTP_USER_AGENT} .* blocks all User-Agents, effectively taking down your site for everyone.
Bracket errors: Unclosed <IfModule> tags in Apache or missing curly braces in Nginx.
Infinite loops: Misconfigured RewriteRule directives can cause 500 Internal Server Errors.

Prevention: Always test in a staging environment or local server before deploying to production. The nginx -t command catches syntax errors but won't detect logic errors.

Testing with curl

Before deploying rules to production, test by sending different User-Agents with the curl command:

Testing Blocked Bots

curl -A "GPTBot/1.0" https://yoursite.com -I

Expected output:

HTTP/1.1 403 Forbidden

Testing Normal Browsers

curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" https://yoursite.com -I

Expected output:

HTTP/1.1 200 OK

Testing Empty User-Agent

curl -A "" https://yoursite.com -I

Some scrapers don't send a User-Agent. To block empty User-Agents as well:

Nginx:

if ($http_user_agent = "") {
    return 403;
}

Apache:

RewriteCond %{HTTP_USER_AGENT} ^$
RewriteRule .* - [F,L]

Additional Security Recommendations

Rate limiting: Throttle IPs making excessive requests using limit_req_zone in Nginx or mod_ratelimit in Apache.
IP verification: Legitimate bots like GPTBot publish their IP ranges. Implement reverse DNS checks to prevent spoofing (e.g., verifying OpenAI's IPs).
WAF deployment: Services like Cloudflare and AWS WAF offer more flexible interfaces for managing User-Agent-based rules.
Log monitoring: Track blocked requests to identify which bots are accessing your site most frequently.

Frequently Asked Questions

I accidentally applied a bot block to my entire site—how do I reverse it?

For Nginx, comment out the relevant if or map block in your nginx.conf or site configuration file (prefix lines with #) and run sudo systemctl reload nginx. For Apache, delete or comment out the RewriteCond lines in your .htaccess file. Changes take effect immediately. If you're unsure, restore from your backup configuration file.

Can I block bots that change their User-Agent header?

User-Agent headers are easily modified by clients, so they're not 100% reliable. For more robust protection, use IP-based blocking, CAPTCHA, JavaScript challenges (like Cloudflare), or behavioral analysis (abnormal request rates, focus on specific endpoints). Legitimate bots typically don't modify their User-Agent, but malicious scrapers can masquerade as normal browsers.

A bot claiming to respect robots.txt is still scraping my content—what should I do?

First, verify the bot is actually reading robots.txt (check server logs for /robots.txt requests). If it's reading but still crawling, check your robots.txt rules for syntax errors or incorrect User-agent names. If the problem persists, implement the server-level blocking methods described above. You can also identify the bot's IP address and block it at the firewall level. For legitimate bots (GPTBot, ClaudeBot), you can file complaints through the platform provider's support channels.

Will blocking all AI bots hurt my SEO?

Blocking training bots like GPTBot, ClaudeBot, and CCBot does not directly affect Google Search rankings because they're separate from Googlebot. However, if you block Google-Extended, you'll prevent usage by Gemini apps and Vertex AI; this still doesn't impact Google Search. Never accidentally block Googlebot (User-Agent: Googlebot). If you don't want to appear in platforms like Perplexity or ChatGPT search, you can block their respective bots (PerplexityBot, OAI-SearchBot); this won't reduce your traditional search engine traffic.