What is ClaudeBot?
ClaudeBot · Anthropic · Training crawler
ClaudeBot is Anthropic's crawler. It collects content that may be used to train AI models. Collects content for Anthropic model training.
What it does
ClaudeBot collects page content to be used in model training. Collects content for Anthropic model training.
It does not execute JavaScript. Whatever your server returns in the initial HTML response is the entire page as far as ClaudeBot is concerned — if your content is rendered client-side, this crawler reads an empty container.
Should you block ClaudeBot?
It is a licensing decision, not a visibility one. ClaudeBot collects content that may be used to train AI models; blocking it does not remove you from Google Search or from AI answer engines.
Your content is not collected for Anthropic model training. As with GPTBot, this is a content-licensing position rather than a visibility one, and it has no bearing on whether Claude can read your page when a user asks about it.
To allow it
User-agent: ClaudeBot
Allow: /To block it
User-agent: ClaudeBot
Disallow: /Put the rule in your robots.txt at the site root. A group that names ClaudeBot explicitly takes precedence over your User-agent: * group, so naming it is how you make your intent survive later edits to the wildcard rules.
Verifying a real ClaudeBot request
Anthropic publishes the IP ranges its crawlers use. Verify the source address rather than the user agent; ClaudeBot is among the more frequently spoofed strings precisely because sites allow-list it.
The mistake people make
Blocking ClaudeBot along with Claude-User and Claude-SearchBot in one rule group. They are three different bots doing three different jobs, and only the first is about training.
Anthropic's other crawlers
- Claude-SearchBotIndexes pages to support Claude search results.
- Claude-UserFetches a page on behalf of a Claude user following a link.
Frequently asked questions
- What is ClaudeBot?
- ClaudeBot is Anthropic's crawler. It collects content that may be used to train AI models. Collects content for Anthropic model training.
- Should I block ClaudeBot?
- It is a licensing decision, not a visibility one. ClaudeBot collects content that may be used to train AI models; blocking it does not remove you from Google Search or from AI answer engines.
- How do I block ClaudeBot in robots.txt?
- Add a group to the robots.txt at your site root: "User-agent: ClaudeBot" followed by "Disallow: /". Use "Allow: /" instead to allow it explicitly.
Further reading
- The Complete Guide to robots.txt for AI Crawlers: Which Bots to Allow
GPTBot vs OAI-SearchBot, ClaudeBot vs Claude-User, Googlebot vs Google-Extended: which AI crawlers to allow, which to block, and copy-ready robots.txt templates.
- How to Create and Optimize an llms.txt File for Your Website
The llms.txt format, where the file goes, how to keep it token-efficient, working examples for Next.js and static sites, and an honest account of who uses it.
- What is Generative Engine Optimization (GEO)? The Complete 2026 Guide
How ChatGPT Search, Perplexity and Claude find, read and cite pages, why that differs from Googlebot, and what the research says actually improves AI visibility.
Test your site against 15 AI crawlers now
The free scan parses your robots.txt, evaluates every AI crawler against it, and tells you how much of a page ClaudeBot would actually come away with.
Run the free scan