What is AI Crawlers?
AI crawlers are the bots — GPTBot, ClaudeBot, PerplexityBot and others — that fetch web content either to train an AI model or to power live, retrieval-based answers; identifying and welcoming the ones that matter to you is a prerequisite for GEO.
Every major AI lab and several search engines now operate at least one distinct crawler, separate from their classic search bot, whose sole purpose is feeding AI systems: some train a model's weights (GPTBot, ClaudeBot, Bytespider), some power live retrieval for a chat assistant's search feature (OAI-SearchBot, PerplexityBot), and some fetch a single page in real time because a user asked the assistant to read it (ChatGPT-User, Claude-User).
Each crawler identifies itself with a distinct user-agent token and, in most cases, publishes its behavior toward robots.txt. A site can allow or block each one independently — blocking GPTBot stops your content being used to train future OpenAI models without affecting ChatGPT's live search feature, which uses OAI-SearchBot instead. That granularity is the whole point: a brand can choose to opt out of AI training while still wanting to be found and cited in live AI answers, or vice versa.
See the full AI crawler directory for the exact user-agent tokens, operators, and ready-to-use robots.txt snippets for every major bot — training crawlers, live-retrieval crawlers, and classic search bots that now also feed AI features.