AI crawler directory
Every AI crawler, and what to do about it
Being crawled is the prerequisite for being cited. Here's every major AI bot — who runs it, what it's for, and the exact robots.txt rule to allow or block it.
New to this? See what AI crawlers are and how llms.txt fits in.
| Bot | Operator | Purpose | Honors robots.txt |
|---|---|---|---|
| GPTBot | OpenAI | GPTBot is the crawler OpenAI uses to collect web content for training and improving its models, including the GPT family that underpins ChatGPT. It is not used for ChatGPT's live search or browsing features — those use a separate, distinct crawler called OAI-SearchBot, and treating the two as interchangeable is the single most common mistake site owners make when deciding what to allow. | Stated |
| OAI-SearchBot | OpenAI | OAI-SearchBot crawls and indexes pages specifically to power ChatGPT's search feature's live, cited answers — a distinct job from training, and the crawler most directly responsible for whether ChatGPT can cite your site in real time. | Stated |
| ChatGPT-User | OpenAI | ChatGPT-User fetches a specific page in real time when a user explicitly asks ChatGPT to open, read or summarize a URL, or via a connected plugin or action — an on-demand fetch rather than a broad, scheduled crawl. | Stated |
| ClaudeBot | Anthropic | ClaudeBot crawls web content to train Anthropic's Claude models. Anthropic has introduced additional, more specific tokens for user-initiated fetches and search indexing as its live-answer features have expanded. | Stated |
| PerplexityBot | Perplexity AI | PerplexityBot crawls and indexes content specifically to power Perplexity's cited, synthesized answers — a live-retrieval crawler rather than a training one, which makes crawl access unusually important for this engine specifically. | Disputed |
| Google-Extended | Google-Extended controls whether your content can be used to train and ground Google's generative AI features (Gemini, AI Overviews), separate from classic Search indexing, which is handled by standard Googlebot instead. | Yes | |
| Googlebot | Googlebot is Google's core Search crawler, in service for decades — now also the primary feed for AI Overviews and AI Mode, since both are extensions of the same underlying Search index rather than a separate system. | Yes | |
| Bingbot | Microsoft | Bingbot is Microsoft's core Search crawler, feeding both classic Bing search and Microsoft Copilot's grounded answers, making it relevant to both traditional search visibility and Copilot-specific AI citations. | Yes |
| Applebot-Extended | Apple | Applebot-Extended controls whether crawled content can be used for Apple's generative AI features (Apple Intelligence), separate from Applebot itself, which powers Siri and Spotlight search indexing rather than generative AI training. | Yes |
| Amazonbot | Amazon | Amazonbot crawls the web for general indexing and to support Amazon's AI features, including product-discovery and shopping-assistant experiences that increasingly draw on content beyond Amazon's own marketplace. | Yes |
| Bytespider | ByteDance (TikTok's parent company) | Bytespider crawls web content to train ByteDance's AI models, including Doubao and other internal systems used across ByteDance's product portfolio. | Disputed |
| CCBot | Common Crawl (nonprofit) | CCBot crawls the web to build the open, freely-downloadable Common Crawl dataset, which many AI labs use as raw pretraining data, either directly or via datasets derived from it. | Yes |
| Meta-ExternalAgent | Meta | Meta-ExternalAgent crawls web content to train Meta's AI models, including the Llama family, and to support Meta's AI-powered search and assistant features across its app portfolio. | Yes |
| cohere-ai | Cohere | cohere-ai crawls web content to train Cohere's enterprise-focused language models, which power a range of business AI products rather than a single consumer-facing chat assistant. | Yes |
| DuckAssistBot | DuckDuckGo | DuckAssistBot crawls content to power DuckAssist, DuckDuckGo's AI-generated instant-answer feature shown above regular search results for privacy-focused users. | Yes |
Check which bots can actually reach your site
Scoutern's free AI bot checker tests your robots.txt against every crawler in this directory.