What is Bytespider and should you block it?
Bytespider · ByteDance · Training crawler
Bytespider is the web crawler run by ByteDance, the company behind TikTok and Toutiao. It is not one of the crawlers behind ChatGPT, Perplexity, Claude or Google Search, so blocking it costs no visibility in any of them. Most sites that block it do so because of how much it crawls.
What it does
Bytespider collects page content to be used in model training. Collects content for ByteDance model training. Widely blocked for aggressive crawl rates.
It does not execute JavaScript. Whatever your server returns in the initial HTML response is the entire page as far as Bytespider is concerned — if your content is rendered client-side, this crawler reads an empty container.
Does Bytespider respect robots.txt?
ByteDance's webmaster documentation for Toutiao search says Bytespider follows the Robots Exclusion Protocol. Bot-management firms and site operators have reported otherwise: in 2024, Kasada and Dark Visitors reported it ignoring robots.txt, and others report it stops once it re-reads an updated file. Any client can also send the Bytespider name, so some of that traffic may not be ByteDance at all.
Put the rule in robots.txt first. If requests carrying the Bytespider user agent keep arriving at disallowed paths after a few days, block or rate-limit them at your CDN or firewall.
Should you block Bytespider?
No. Bytespider has no role in Google Search or in ChatGPT, Perplexity or Claude answers. ByteDance has said it is used for search purposes in its own products, so the only reach you give up is ByteDance's.
Your content is not collected for ByteDance model training. There is no user-facing answer surface tied to this crawler that blocking would remove you from.
To allow it
User-agent: Bytespider
Allow: /To block it
User-agent: Bytespider
Disallow: /Put the rule in your robots.txt at the site root. A group that names Bytespider explicitly takes precedence over your User-agent: * group, so naming it is how you make your intent survive later edits to the wildcard rules.
Verifying a real Bytespider request
Bytespider is among the most commonly spoofed user agent strings on the web, so a request claiming to be Bytespider very often is not. If you see sustained aggressive traffic under this name, check the source IPs before concluding anything about who is sending it.
The mistake people make
Blocking it in robots.txt and expecting the traffic to stop. Robots.txt is a request, not a control; the reason Bytespider appears on so many block lists is its crawl rate, and a crawl rate problem is solved at the firewall or CDN, not in a text file.
Frequently asked questions
- What is Bytespider?
- Bytespider is the web crawler run by ByteDance, the company behind TikTok and Toutiao. It is not one of the crawlers behind ChatGPT, Perplexity, Claude or Google Search, so blocking it costs no visibility in any of them. Most sites that block it do so because of how much it crawls.
- Will blocking Bytespider hurt my SEO?
- No. Bytespider has no role in Google Search or in ChatGPT, Perplexity or Claude answers. ByteDance has said it is used for search purposes in its own products, so the only reach you give up is ByteDance's.
- How do I block Bytespider in robots.txt?
- Add a group to the robots.txt at your site root: "User-agent: Bytespider" followed by "Disallow: /". Use "Allow: /" instead to allow it explicitly.
Further reading
- The Complete Guide to robots.txt for AI Crawlers: Which Bots to Allow
GPTBot vs OAI-SearchBot, ClaudeBot vs Claude-User, Googlebot vs Google-Extended: which AI crawlers to allow, which to block, and copy-ready robots.txt templates.
- How to Create and Optimize an llms.txt File for Your Website
The llms.txt format, where the file goes, how to keep it token-efficient, working examples for Next.js and static sites, and an honest account of who uses it.
- What is Generative Engine Optimization (GEO)? The Complete 2026 Guide
How ChatGPT Search, Perplexity and Claude find, read and cite pages, why that differs from Googlebot, and what the research says actually improves AI visibility.
Test your site against 15 AI crawlers now
The free scan parses your robots.txt, evaluates every AI crawler against it, and tells you how much of a page Bytespider would actually come away with.
Run the free scan