Bytespider
ByteDance · Training crawler
What it does
Bytespider collects page content to be used in model training. Collects content for ByteDance model training. Widely blocked for aggressive crawl rates.
It does not execute JavaScript. Whatever your server returns in the initial HTML response is the entire page as far as Bytespider is concerned — if your content is rendered client-side, this crawler reads an empty container.
Should you block Bytespider?
Blocking it keeps your content out of the training corpus. It does not remove you from any answer surface, because this crawler is not what answers are built from.
Your content is not collected for ByteDance model training. There is no user-facing answer surface tied to this crawler that blocking would remove you from.
To allow it
User-agent: Bytespider
Allow: /To block it
User-agent: Bytespider
Disallow: /Put the rule in your robots.txt at the site root. A group that names Bytespider explicitly takes precedence over your User-agent: * group, so naming it is how you make your intent survive later edits to the wildcard rules.
Verifying a real Bytespider request
Bytespider is among the most commonly spoofed user agent strings on the web, so a request claiming to be Bytespider very often is not. If you see sustained aggressive traffic under this name, check the source IPs before concluding anything about who is sending it.
The mistake people make
Blocking it in robots.txt and expecting the traffic to stop. Robots.txt is a request, not a control; the reason Bytespider appears on so many block lists is its crawl rate, and a crawl rate problem is solved at the firewall or CDN, not in a text file.
Can Bytespider reach your site?
The free scan parses your robots.txt, evaluates every AI crawler against it, and tells you how much of a page Bytespider would actually come away with.
Run the free scan