ClaudeBot
Stated to honor robots.txtClaudeBot crawls web content to train Anthropic's Claude models. Anthropic has introduced additional, more specific tokens for user-initiated fetches and search indexing as its live-answer features have expanded.
- User-agent
- ClaudeBot
- Operator
- Anthropic
Allow in robots.txt
User-agent: ClaudeBot
Allow: /Block in robots.txt
User-agent: ClaudeBot
Disallow: /Claude's developer-heavy audience makes technical documentation and API references a disproportionately valuable crawl target for this bot compared to purely consumer-facing marketing pages.
Why ClaudeBot traffic spiked in early 2026
Search interest in ClaudeBot climbed sharply around January and February 2026, coinciding with Anthropic expanding its crawl coverage while scaling up data collection for newer Claude models. For many site owners, this was the first time ClaudeBot appeared meaningfully in server logs at all, which drove a wave of site owners investigating the bot for the first time — often reacting by blocking it outright before understanding what it actually does.
Should you allow or block ClaudeBot?
If being referenced or recommended by Claude matters to your brand — increasingly likely for any company selling to buyers who might ask an AI assistant for a recommendation — allowing ClaudeBot is the straightforward choice. Block it only if you have a specific, considered reason to opt out of AI training data collection, understanding that this removes a pathway to being cited by Claude, not just a defensive privacy measure with no trade-off.