PerplexityBot Explained: What It Crawls and How to Manage It

If you have spotted 'PerplexityBot' in your server logs and wondered whether to welcome it or shut the door, you are in good company. It is one of the less-documented AI crawlers, and most of what gets written about it lumps it in with the training bots run by other AI companies. That is a mistake, because PerplexityBot does a different job. This guide covers what it does, how it differs from Perplexity's second user-agent, and how to manage it without accidentally removing your brand from Perplexity's answers.

What is PerplexityBot?

PerplexityBot is the web crawler that builds and refreshes the index behind Perplexity's search. Its user-agent string identifies itself as 'PerplexityBot/1.0' and points to perplexity.ai/perplexitybot. Perplexity describes its purpose as surfacing and linking websites in its search results, and states that it is not used to crawl content for AI foundation models.

That distinction is the core of any sensible allow-or-block decision. Several other AI crawlers collect material for model training, which is a legitimate reason to opt out for some publishers. PerplexityBot is closer in spirit to a classic search crawler: it exists so pages can be found and cited. If you want a Perplexity citation, this is the bot you need to let in.

How is it different from Perplexity-User?

Perplexity runs two documented agents. PerplexityBot crawls in the background to build the search index. Perplexity-User acts at the moment a person asks a question: it may visit a page to help produce an accurate answer and then link to it. Perplexity says that because this is a user-initiated fetch, it generally ignores robots.txt rules. In practice, that means a Disallow line aimed at Perplexity-User is unlikely to do what you expect, and blocking it reliably needs a server or firewall rule instead.

Should you allow PerplexityBot?

For most businesses that want to be recommended, yes. Perplexity answers questions with cited sources, and a page its crawler cannot reach is a page it is far less likely to cite. Our guide on how to rank in Perplexity covers what the engine tends to cite once it can see you; allowing the crawler is the precondition for all of it.

There are sensible reasons to block it: paywalled publishers, sites with licensing restrictions, or content you simply do not want summarised. The key is making that a deliberate choice. Plenty of sites block PerplexityBot by accident, through a blanket rule or a security-plugin default, and only discover it when a competitor starts appearing in answers they used to own.

How to allow or block PerplexityBot in robots.txt

Name the bot explicitly rather than relying on a wildcard. To allow it everywhere, add a block that begins with 'User-agent: PerplexityBot' followed by 'Allow: /'. To block it everywhere, use the same user-agent line followed by 'Disallow: /'. To protect only part of the site, disallow specific directories and leave the rest open.

1. Open yourdomain.com/robots.txt in a browser and read it as the crawler would, top to bottom
2. Look for a 'User-agent: *' block with 'Disallow: /'. If there is no more specific PerplexityBot block beneath it, the bot is blocked
3. Add a named PerplexityBot block with the access you intend
4. Save, redeploy and fetch the live file again to confirm the change is published
5. Check your CDN or firewall dashboard for any bot-protection rule that could still turn the crawler away

Why robots.txt allowing it may not be enough

Robots.txt is a polite request, not an enforcement mechanism. Many hosts, CDNs and security plugins run their own bot-management layer that returns a block or a challenge page to traffic they classify as automated, regardless of what robots.txt says. If PerplexityBot is permitted on paper but your logs show it receiving 403 or challenge responses, the fix lives in your firewall settings, not your robots.txt.

Perplexity publishes the IP ranges both agents use, so you can allowlist genuine traffic or confirm that a request claiming to be PerplexityBot really is one. The user-agent string alone proves nothing, because anyone can send it. Our guide on which AI crawlers to allow or block explains how the major bots compare, and our walkthroughs of GPTBot and ClaudeBot cover their equivalents.

A quick audit routine

• Search logs for 'PerplexityBot' and 'Perplexity-User' to see whether either is reaching you at all
• Check the status codes returned: 200s are healthy, 403s and challenge pages are not
• Compare your robots.txt intent with what your firewall actually enforces
• Repeat after any migration, CDN change or security-plugin update, since defaults are routinely reset
• Then check the outcome: run your key questions through Perplexity and see whether you are named

To see whether letting the crawler in has translated into mentions, our guide on how to track AI visibility explains how to measure it across Perplexity, ChatGPT and the other engines on a repeatable schedule.

Sources: Perplexity's official bot documentation at docs.perplexity.ai/guides/bots, and the AI Crawlers Reference Guide published by Otterly at help.otterly.ai/ai-crawlers.