What is Claude-SearchBot?
Claude-SearchBot · Anthropic · Search index crawler
Claude-SearchBot is Anthropic's crawler. It builds the index an AI answer engine retrieves and cites from. Indexes pages to support Claude search results.
What it does
Claude-SearchBot builds the index that answers are retrieved from. Indexes pages to support Claude search results.
It does not execute JavaScript. Whatever your server returns in the initial HTML response is the entire page as far as Claude-SearchBot is concerned — if your content is rendered client-side, this crawler reads an empty container.
Should you block Claude-SearchBot?
Only if you want out of Anthropic's AI answers. Claude-SearchBot builds the index an AI answer engine retrieves and cites from, so blocking it means your pages stop being available as a source there.
Your pages stop being indexed as sources Claude can cite. The effect is the same shape as blocking any search index crawler: you are not down-ranked, you are simply not in the candidate set.
To allow it
User-agent: Claude-SearchBot
Allow: /To block it
User-agent: Claude-SearchBot
Disallow: /Put the rule in your robots.txt at the site root. A group that names Claude-SearchBot explicitly takes precedence over your User-agent: * group, so naming it is how you make your intent survive later edits to the wildcard rules.
Verifying a real Claude-SearchBot request
Check the source IP against Anthropic's published ranges. The search crawler is distinct from ClaudeBot and should be evaluated on its own, not by assuming one Anthropic range covers all of them.
The mistake people make
Assuming a wildcard Allow covers it. It does, until someone adds a blanket Disallow for an unrelated reason — a staging leak, a scraper problem — and the search crawler goes down with everything else because it was never named.
Anthropic's other crawlers
- ClaudeBotCollects content for Anthropic model training.
- Claude-UserFetches a page on behalf of a Claude user following a link.
Frequently asked questions
- What is Claude-SearchBot?
- Claude-SearchBot is Anthropic's crawler. It builds the index an AI answer engine retrieves and cites from. Indexes pages to support Claude search results.
- Should I block Claude-SearchBot?
- Only if you want out of Anthropic's AI answers. Claude-SearchBot builds the index an AI answer engine retrieves and cites from, so blocking it means your pages stop being available as a source there.
- How do I block Claude-SearchBot in robots.txt?
- Add a group to the robots.txt at your site root: "User-agent: Claude-SearchBot" followed by "Disallow: /". Use "Allow: /" instead to allow it explicitly.
Further reading
- The Complete Guide to robots.txt for AI Crawlers: Which Bots to Allow
GPTBot vs OAI-SearchBot, ClaudeBot vs Claude-User, Googlebot vs Google-Extended: which AI crawlers to allow, which to block, and copy-ready robots.txt templates.
- How to Create and Optimize an llms.txt File for Your Website
The llms.txt format, where the file goes, how to keep it token-efficient, working examples for Next.js and static sites, and an honest account of who uses it.
- What is Generative Engine Optimization (GEO)? The Complete 2026 Guide
How ChatGPT Search, Perplexity and Claude find, read and cite pages, why that differs from Googlebot, and what the research says actually improves AI visibility.
Test your site against 15 AI crawlers now
The free scan parses your robots.txt, evaluates every AI crawler against it, and tells you how much of a page Claude-SearchBot would actually come away with.
Run the free scan