Skip to content
Crawlable

ClaudeBot

Anthropic · Training crawler

Operator
Anthropic
Runs JavaScript
No
Blocking costs visibility
No

What it does

ClaudeBot collects page content to be used in model training. Collects content for Anthropic model training.

It does not execute JavaScript. Whatever your server returns in the initial HTML response is the entire page as far as ClaudeBot is concerned — if your content is rendered client-side, this crawler reads an empty container.

Should you block ClaudeBot?

Blocking it keeps your content out of the training corpus. It does not remove you from any answer surface, because this crawler is not what answers are built from.

Your content is not collected for Anthropic model training. As with GPTBot, this is a content-licensing position rather than a visibility one, and it has no bearing on whether Claude can read your page when a user asks about it.

To allow it

User-agent: ClaudeBot
Allow: /

To block it

User-agent: ClaudeBot
Disallow: /

Put the rule in your robots.txt at the site root. A group that names ClaudeBot explicitly takes precedence over your User-agent: * group, so naming it is how you make your intent survive later edits to the wildcard rules.

Verifying a real ClaudeBot request

Anthropic publishes the IP ranges its crawlers use. Verify the source address rather than the user agent; ClaudeBot is among the more frequently spoofed strings precisely because sites allow-list it.

The mistake people make

Blocking ClaudeBot along with Claude-User and Claude-SearchBot in one rule group. They are three different bots doing three different jobs, and only the first is about training.

Anthropic's other crawlers

Can ClaudeBot reach your site?

The free scan parses your robots.txt, evaluates every AI crawler against it, and tells you how much of a page ClaudeBot would actually come away with.

Run the free scan