How to Get Your Website Indexed by Perplexity AI Search
Executive Summary: PerplexityBot is Perplexity's autonomous crawler that indexes web content for search results; Perplexity-User is a user-initiated agent that fetches page content in real-time when users query Perplexity. If your robots.txt blocks PerplexityBot or your WAF/firewall blocks Perplexity IP ranges, your content won't appear in Perplexity search results. To maintain visibility, you must allow both agents access, ensure JavaScript rendering support, and implement reasonable rate limits.
Understanding PerplexityBot vs. Perplexity-User
Perplexity employs two distinct user agents to power its search experience:
- PerplexityBot: Perplexity's autonomous web crawler that indexes content and builds the search database. It visits pages at regular intervals, respects robots.txt directives, and adds content to Perplexity's search index. This functions similarly to Googlebot or Bingbot—crawling, indexing, and maintaining a searchable corpus of web content.
- Perplexity-User: A user-initiated agent that fetches page content in real-time when someone asks a question on Perplexity. This operates on-demand in response to user queries and represents a different paradigm where robots.txt enforcement may vary (though Perplexity officially states it respects both agents' access controls).
Blocking both agents will completely prevent your site from appearing in Perplexity. Blocking only one creates partial visibility issues: blocking PerplexityBot prevents indexing, while blocking Perplexity-User prevents real-time content retrieval for user queries.
How robots.txt Blocking Affects Perplexity Visibility
Your robots.txt file instructs Perplexity's crawlers which pages they may access. To completely block Perplexity, add these directives:
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
This configuration prevents Perplexity from crawling and indexing your entire site. The consequences:
- Your content won't appear in Perplexity search results.
- When users query Perplexity, your site won't be cited or referenced as a source.
- Perplexity cannot leverage your site's information when generating answers.
Selective blocking is also possible. To restrict access to specific directories only:
User-agent: PerplexityBot
Disallow: /admin/
Disallow: /private/
User-agent: Perplexity-User
Disallow: /admin/
Disallow: /private/
Remember that robots.txt is a courtesy protocol; for non-compliant bots, you'll need server-level or WAF enforcement.
WAF and Firewall Blocking Impact on Perplexity
To block non-compliant or aggressive bots, you can implement IP or user-agent blocking at the Web Application Firewall (WAF) or server level. If you block Perplexity IP ranges or user agents:
- Perplexity's crawlers cannot access your site at all (regardless of robots.txt).
- Your content won't be indexed or visible in Perplexity.
- Real-time content retrieval for user queries becomes impossible.
Example blocking rules for Perplexity in common platforms like Cloudflare, AWS WAF, or Nginx:
# Nginx example
if ($http_user_agent ~* (PerplexityBot|Perplexity-User)) {
return 403;
}
In Cloudflare, create user-agent-based blocking rules under "Firewall Rules." WAF blocking provides more definitive control than robots.txt and effectively stops non-compliant crawlers.
JavaScript Rendering and Rate Limiting Considerations
Perplexity supports JavaScript rendering to properly index modern web applications. However, if your site relies heavily on JavaScript and Perplexity's crawlers exceed rendering timeouts, your content may be incompletely indexed. Additionally, if Perplexity's crawlers send requests too frequently, your server may slow down or trigger automatic rate limiting.
Best practices:
- JavaScript rendering: Serve critical content via server-side rendering (SSR) or static HTML. Don't assume Perplexity will fully render complex JavaScript applications.
- Rate limiting: Implement reasonable server-level rate limits (e.g., 10-20 requests per minute). Whitelist Perplexity's official IP ranges to preserve legitimate traffic while preventing abuse.
- Crawl-delay: The crawl-delay directive in robots.txt is non-standard and Perplexity's compliance is unverified. Enforce rate limits at the server level instead.
Perplexity Visibility Checklist
To ensure your site appears in Perplexity—or to deliberately block it—follow this checklist:
| Checkpoint | To Allow Visibility | To Block |
|---|---|---|
| robots.txt | No Disallow rules for PerplexityBot and Perplexity-User, or explicit Allow rules | Add Disallow: / for both agents |
| WAF/Firewall | Perplexity IPs and user agents whitelisted | Block Perplexity user agents or IP ranges (return 403/444) |
| JavaScript Rendering | Critical content served via SSR or static HTML | — |
| Rate Limiting | Reasonable limits (10-20 requests/minute), Perplexity IPs whitelisted | Aggressive rate limits (1-2 requests/minute) or complete block |
| Sitemap | sitemap.xml current and referenced in robots.txt | — |
| Content Quality | Structured, readable content; clear meta descriptions and headings | — |
Testing Your Perplexity Visibility
To verify whether your site appears in Perplexity:
- Search on Perplexity: Query your site name or unique content snippets on Perplexity. If your site appears in results, it's indexed.
- Examine server logs: Check for requests from PerplexityBot and Perplexity-User user agents. No requests may indicate robots.txt or WAF blocking.
- Validate robots.txt: Open
https://yoursite.com/robots.txtin a browser and confirm Perplexity agents are allowed. - Review WAF logs: In Cloudflare, AWS WAF, or similar tools, check whether Perplexity requests are being blocked.
Perplexity hasn't published official IP ranges or crawl frequency documentation; monitor Perplexity support channels for current information.
Frequently Asked Questions
What happens if I block PerplexityBot but allow Perplexity-User?
Blocking PerplexityBot prevents your content from being added to Perplexity's search index. However, allowing Perplexity-User means Perplexity can still fetch your pages in real-time when users ask direct questions. Your content may not appear in the index but could still surface in response to user queries. For consistent visibility, we recommend allowing both agents, as Perplexity's architecture relies on both.
Perplexity's crawlers are slowing down my server—what should I do?
Implement server-level rate limiting (e.g., Nginx's limit_req_zone or Cloudflare's rate limiting). Whitelist Perplexity's official IP ranges to preserve legitimate traffic, but set reasonable limits like 10-20 requests per minute. If aggressive behavior persists, you can add Disallow: / in robots.txt to block completely, or implement user-agent-based blocking in your WAF.
I've allowed Perplexity in robots.txt but still don't appear—why?
Several factors could be at play: (1) Your WAF or firewall may be blocking Perplexity IPs or user agents; check server and WAF logs. (2) If your site relies heavily on JavaScript and Perplexity can't render it, content may be incompletely indexed; serve critical content via SSR or static HTML. (3) If your site is new or has low authority, Perplexity may not have crawled it yet; keep your sitemap.xml current and wait. (4) Perplexity's crawl frequency may be low, so your content may not yet be indexed.
Does blocking Perplexity provide copyright protection?
robots.txt or WAF blocking technically prevents Perplexity from accessing your content, but this isn't a legal protection mechanism. If your content was previously indexed or sourced elsewhere, it may still appear in Perplexity answers. If you have copyright concerns, review Perplexity's official copyright policy and seek legal counsel if necessary. robots.txt is a courtesy protocol for access control—it doesn't provide legal protection.