Scoutern

Bytespider

Disputed robots.txt compliance

Bytespider crawls web content to train ByteDance's AI models, including Doubao and other internal systems used across ByteDance's product portfolio.

User-agent
Bytespider
Operator
ByteDance (TikTok's parent company)

Allow in robots.txt

User-agent: Bytespider
Allow: /

Block in robots.txt

User-agent: Bytespider
Disallow: /

Bytespider has been widely reported as one of the most aggressive crawlers by request volume, and is one of the most commonly blocked bots by publishers who otherwise allow other AI crawlers.

Frequently asked questions

It has been repeatedly reported for unusually high crawl volume relative to the traffic or citations it drives back, leading many publishers to block it even while allowing GPTBot or ClaudeBot.

More AI crawlers

OpenAI

GPTBot

OpenAI

OAI-SearchBot

OpenAI

ChatGPT-User

Verify your robots.txt against every AI crawler

Scoutern's free AI bot checker tests exactly which bots can reach your site.

Check my robots.txtSee pricing