How to Configure Cloudflare WAF Rules to Allow Perplexity and ChatGPT while Blocking Aggressive Crawlers

Understanding Cloudflare's AI Bot Management: Why the Global Block Button Is Dangerous

Cloudflare's one-click "Block AI Bots" feature indiscriminately blocks all AI crawlers, including beneficial search engines like Perplexity and ChatGPT that drive significant organic traffic to your site. This nuclear option prevents your content from being indexed by next-generation AI search platforms, effectively making your website invisible to millions of users who rely on AI-powered search tools for information discovery.

The Strategic Importance of Selective AI Bot Management

Modern web traffic increasingly originates from AI-powered search engines and assistants. Perplexity, ChatGPT Search, Google's AI Overviews, and similar platforms represent the future of information retrieval. Blocking these legitimate crawlers while using Cloudflare's blanket AI bot block creates a critical visibility gap that can severely impact your site's discoverability and organic reach.

The solution lies in implementing granular Web Application Firewall (WAF) rules that distinguish between beneficial AI crawlers and aggressive scrapers that consume bandwidth without providing value. This guide provides comprehensive instructions for configuring Cloudflare WAF rules that protect your infrastructure while maintaining visibility in AI search ecosystems.

Major AI Crawler User-Agents and IP Ranges

Before configuring WAF rules, you must understand the identification methods used by legitimate AI crawlers. The following table provides comprehensive information about major AI bot user-agents and their associated IP ranges:

Configuring Custom WAF Rules to Allow Beneficial AI Crawlers

Cloudflare's WAF Custom Rules provide the flexibility to create sophisticated bot management policies. The following configuration allows legitimate AI search crawlers while blocking aggressive scrapers and unauthorized bots.

Step 1: Access Cloudflare WAF Custom Rules

1. Log into your Cloudflare dashboard
2. Select the domain you want to configure
3. Navigate to Security → WAF → Custom rules
4. Click Create rule to begin configuring your custom ruleset

Step 2: Create an Allow Rule for Legitimate AI Crawlers

Create a rule with the following configuration:

Rule Name: Allow Legitimate AI Search Crawlers

Expression:

(http.user_agent contains "PerplexityBot") or
(http.user_agent contains "GPTBot") or
(http.user_agent contains "OAI-SearchBot") or
(http.user_agent contains "Google-Extended") or
(http.user_agent contains "anthropic-ai") or
(http.user_agent contains "Applebot") or
(http.user_agent contains "cohere-ai") or
(http.user_agent contains "FacebookBot" and not http.user_agent contains "facebookexternalhit")

Action: Allow

Priority: Set to 1 (highest priority to ensure this rule executes first)

Step 3: Create a Block Rule for Aggressive Generic Scrapers

After allowing legitimate crawlers, create a secondary rule to block known aggressive scrapers:

Rule Name: Block Aggressive Scrapers and Unauthorized Bots

Expression:

(http.user_agent contains "scrapy") or
(http.user_agent contains "python-requests") or
(http.user_agent contains "curl") or
(http.user_agent contains "wget") or
(http.user_agent contains "go-http-client") or
(http.user_agent contains "axios") or
(http.user_agent contains "node-fetch") or
(http.user_agent eq "") or
(http.user_agent contains "Bytespider") or
(http.user_agent contains "PetalBot") or
(http.user_agent contains "AhrefsBot" and not cf.bot_management.verified_bot) or
(http.user_agent contains "SemrushBot" and not cf.bot_management.verified_bot) or
(http.user_agent contains "DotBot") or
(http.user_agent contains "MJ12bot") or
(http.user_agent contains "BLEXBot")

Action: Block

Priority: Set to 2

Step 4: Implement Challenge Rules for Suspicious Traffic Patterns

For traffic that doesn't match either allow or block criteria but exhibits suspicious patterns, implement a challenge rule:

Rule Name: Challenge Suspicious Bot-Like Behavior

Expression:

(cf.bot_management.score lt 30) and
not (cf.bot_management.verified_bot) and
not (http.user_agent contains "PerplexityBot") and
not (http.user_agent contains "GPTBot") and
not (http.user_agent contains "OAI-SearchBot") and
not (http.user_agent contains "Google-Extended") and
not (http.user_agent contains "Applebot")

Action: Managed Challenge

Priority: Set to 3

Implementing Custom Rate Limiting Rules for API Endpoint Protection

Rate limiting provides an additional layer of protection against scraping abuse, particularly for API endpoints and resource-intensive pages. Follow these steps to configure intelligent rate limiting that doesn't impact legitimate AI crawlers:

Step 1: Navigate to Rate Limiting Rules

1. In your Cloudflare dashboard, go to Security → WAF → Rate limiting rules
2. Click Create rule

Step 2: Configure API Endpoint Rate Limiting

Rule Name: API Endpoint Rate Limiting with AI Crawler Exemption

Expression:

(http.request.uri.path contains "/api/") and
not (http.user_agent contains "PerplexityBot") and
not (http.user_agent contains "GPTBot") and
not (http.user_agent contains "OAI-SearchBot") and
not (http.user_agent contains "Applebot") and
not (cf.bot_management.verified_bot)

Characteristics:

• Requests: 100 requests
• Period: 60 seconds
• Counting method: Count by IP address

Action: Block for 1 hour

Step 3: Configure Content Page Rate Limiting

For content pages, implement more generous rate limits that accommodate legitimate reading patterns:

Rule Name: Content Page Rate Limiting

Expression:

(http.request.uri.path contains "/blog/" or http.request.uri.path contains "/articles/") and
not (http.user_agent contains "PerplexityBot") and
not (http.user_agent contains "GPTBot") and
not (http.user_agent contains "OAI-SearchBot") and
not (http.user_agent contains "Google-Extended") and
not (http.user_agent contains "Applebot") and
not (cf.bot_management.verified_bot)

Characteristics:

• Requests: 300 requests
• Period: 300 seconds (5 minutes)
• Counting method: Count by IP address

Action: Managed Challenge

Step 4: Implement Aggressive Crawler Rate Limiting

Create a strict rate limit specifically for known aggressive crawlers that weren't blocked outright:

Rule Name: Aggressive Crawler Throttling

Expression:

(http.user_agent contains "AhrefsBot") or
(http.user_agent contains "SemrushBot") or
(http.user_agent contains "MJ12bot") or
(cf.bot_management.score lt 20 and not cf.bot_management.verified_bot)

Characteristics:

• Requests: 10 requests
• Period: 60 seconds
• Counting method: Count by IP address

Action: Block for 24 hours

Step 5: Monitor and Adjust Rate Limits

1. Navigate to Security → Events to monitor triggered rules
2. Review the Activity log for false positives affecting legitimate traffic
3. Adjust rate limit thresholds based on your site's typical traffic patterns
4. Use Analytics → Security to identify emerging scraper patterns
5. Update your allow list quarterly as new AI search engines emerge

Advanced Configuration: IP Range Verification

User-agent strings can be spoofed, making IP range verification an essential secondary validation method. Implement these advanced rules for enhanced security:

Verified Bot IP Range Validation

Create a rule that requires both correct user-agent AND IP range verification for high-value content:

(http.request.uri.path contains "/premium/") and
(
  (http.user_agent contains "GPTBot" and not cf.bot_management.verified_bot) or
  (http.user_agent contains "PerplexityBot" and not cf.bot_management.verified_bot)
)

Action: Managed Challenge

This rule challenges crawlers claiming to be legitimate AI bots but not originating from verified IP ranges, protecting premium content from spoofing attempts.

Monitoring and Maintenance Best Practices

Effective WAF rule management requires ongoing monitoring and adjustment. Implement these best practices:

• Weekly Review: Check Security Events for blocked legitimate traffic and adjust rules accordingly
• Monthly Analysis: Review Analytics data to identify new crawler user-agents and emerging scraping patterns
• Quarterly Updates: Update your allow list as new AI search engines launch and existing ones modify their crawlers
• Alert Configuration: Set up Cloudflare notifications for unusual spikes in blocked traffic
• Documentation: Maintain internal documentation of rule changes and the rationale behind each configuration
• Testing Protocol: Before deploying new rules to production, test them in "Log" mode to identify potential issues

Robots.txt Coordination

WAF rules should complement, not replace, your robots.txt directives. Maintain a coordinated approach:

# Allow beneficial AI crawlers
User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot
Allow: /

User-agent: anthropic-ai
Allow: /

# Block aggressive scrapers
User-agent: Bytespider
Disallow: /

User-agent: PetalBot
Disallow: /

User-agent: AhrefsBot
Crawl-delay: 10
Disallow: /api/

User-agent: SemrushBot
Crawl-delay: 10
Disallow: /api/

Performance Impact Considerations

WAF rules execute on every request, so optimization is crucial for maintaining site performance:

• Rule Order: Place most frequently matched rules (allow rules for common crawlers) at higher priority to reduce processing time
• Expression Efficiency: Use simple string matching (contains) rather than complex regex when possible
• Rule Consolidation: Combine related conditions into single rules rather than creating multiple overlapping rules
• Caching Integration: Ensure WAF rules don't interfere with Cloudflare's caching by avoiding unnecessary challenges on cacheable resources
• Geographic Considerations: For region-specific content, add geographic filters to reduce unnecessary rule evaluation

Frequently Asked Questions

Why shouldn't I use Cloudflare's one-click "Block AI Bots" feature?

Cloudflare's global AI bot block is an indiscriminate tool that blocks all AI crawlers, including beneficial ones like Perplexity, ChatGPT Search, and Google's AI features. These platforms drive significant organic traffic and represent the future of search. Blocking them makes your content invisible to millions of users who rely on AI-powered search tools. Custom WAF rules allow you to selectively allow legitimate AI search engines while blocking aggressive scrapers that provide no value.

How do I verify that my WAF rules are working correctly?

Monitor your rules through Cloudflare's Security Events dashboard (Security → Events). This shows real-time data on which rules are triggering and what traffic they're affecting. Start new rules in "Log" mode rather than "Block" mode to observe their behavior without impacting traffic. Review the Activity log daily for the first week after implementing new rules to catch any false positives. You can also use curl commands with different user-agent strings to test your rules manually.

What's the difference between "Block" and "Managed Challenge" actions?

"Block" immediately denies access with no opportunity for the visitor to proceed, returning a 403 Forbidden error. This is appropriate for known malicious bots and aggressive scrapers. "Managed Challenge" presents an intelligent challenge that legitimate browsers can pass automatically while blocking bots. Cloudflare's system determines the appropriate challenge type (invisible, JavaScript challenge, or CAPTCHA) based on the request characteristics. Use Managed Challenge for suspicious traffic that might include legitimate users, and Block for confirmed malicious sources.

How often should I update my AI crawler allow list?

Review and update your allow list quarterly at minimum, as the AI search landscape evolves rapidly. New AI search engines launch regularly, and existing ones may change their crawler user-agents or IP ranges. Subscribe to announcements from major AI companies (OpenAI, Anthropic, Google, Perplexity) to stay informed about crawler changes. Monitor your Security Events log for blocked user-agents that might be legitimate new crawlers. When major AI search products launch (like ChatGPT Search or new Google AI features), update your rules immediately to maintain visibility.

Can legitimate AI crawlers spoof their user-agents to bypass my rules?

While user-agents can be spoofed, legitimate AI companies don't engage in this practice as it would damage their reputation and violate webmaster guidelines. However, malicious actors may spoof legitimate crawler user-agents. This is why IP range verification is crucial. Use Cloudflare's cf.bot_management.verified_bot field, which validates that requests claiming to be from known bots actually originate from verified IP ranges. For high-value content, implement rules that require both correct user-agent AND verified IP range. This prevents spoofing while allowing legitimate crawlers.

What rate limits are appropriate for AI crawlers?

Legitimate AI crawlers typically respect reasonable rate limits and crawl-delay directives. For beneficial crawlers like PerplexityBot and GPTBot, exempt them from rate limiting entirely or set very generous limits (1000+ requests per 5 minutes). These crawlers are already designed to be respectful and won't overwhelm your server. For aggressive SEO crawlers like AhrefsBot or SemrushBot, implement strict limits (10-20 requests per minute) to prevent resource exhaustion. Monitor your server load and adjust limits based on your infrastructure capacity. API endpoints should have stricter limits than content pages.

Should I block all Python and curl user-agents?

Blocking generic user-agents like "python-requests" or "curl" can be effective against unsophisticated scrapers, but may also block legitimate automated tools, monitoring services, and internal scripts. Instead of blanket blocking, use a layered approach: block these user-agents only on sensitive endpoints (APIs, admin panels) while allowing them on public content. Alternatively, use rate limiting rather than outright blocking—legitimate tools will respect rate limits while aggressive scrapers will trigger blocks. Consider implementing API key authentication for programmatic access rather than relying solely on user-agent blocking.

How do I handle false positives where legitimate users get blocked?

False positives are inevitable with aggressive bot management. Implement a clear unblock process: create a dedicated page explaining why the block occurred and providing contact information for legitimate users to request unblocking. Use "Managed Challenge" instead of "Block" for borderline cases, as this allows legitimate users to prove they're human. Monitor your Security Events dashboard daily for patterns indicating false positives (e.g., blocks from corporate IP ranges, specific geographic regions, or during business hours). Maintain a whitelist of known good IP ranges (your office, major corporate networks, cloud providers used by legitimate services).

What's the impact of these WAF rules on my Cloudflare bill?

WAF Custom Rules are included in Cloudflare Pro plans and higher, with limits on the number of rules you can create (typically 10-100 depending on your plan). The rules themselves don't incur per-request charges—they're evaluated as part of Cloudflare's standard request processing. Rate Limiting rules may have separate pricing on some plans. Bot Management features (cf.bot_management.score) require a Business or Enterprise plan. The performance impact is minimal as Cloudflare's edge network processes these rules efficiently. The cost savings from blocking malicious traffic (reduced bandwidth, server load, and scraping-related costs) typically far exceed any additional Cloudflare fees.

How do I balance SEO crawler access with scraper protection?

Distinguish between legitimate SEO crawlers (Googlebot, Bingbot) and aggressive SEO tools (AhrefsBot, SemrushBot). Always allow verified search engine crawlers—use cf.bot_management.verified_bot to ensure they're legitimate. For SEO tool crawlers, implement rate limiting rather than blocking if you find their data useful for competitive analysis. Use robots.txt crawl-delay directives to slow aggressive crawlers without blocking them entirely. For premium or gated content, block all crawlers except verified search engines. Remember that legitimate search engine crawlers (Google, Bing) are essential for traditional SEO, while AI search crawlers (Perplexity, ChatGPT) are essential for AI search visibility—you need both.

Can I use these rules to create a paid API access model?

Yes, WAF rules can enforce a tiered access model. Create rules that block all automated access to specific endpoints, then issue API keys to paying customers and whitelist their IP ranges or require authentication headers. Use Cloudflare Workers in combination with WAF rules for more sophisticated authentication. Implement rate limiting tiers based on subscription level—free tier gets 100 requests/hour, paid tier gets 10,000 requests/hour. For AI companies wanting to train on your content, you can selectively allow their crawlers to specific sections while requiring commercial agreements for premium content. This approach is increasingly common as publishers seek to monetize AI training data access.

What should I do if a new AI search engine launches?

When a new AI search engine launches, research their crawler documentation to identify their user-agent string and IP ranges. Most legitimate AI companies publish this information in their robots.txt documentation or developer pages. Add their user-agent to your allow list within 24-48 hours of launch to ensure your content is indexed early. Monitor Security Events to see if their crawler is being blocked by existing rules. Consider the company's reputation and the potential traffic value before allowing their crawler—not all AI search engines will drive meaningful traffic. Join webmaster communities and forums where new crawler launches are discussed to stay informed about emerging platforms.