CrawlableBot
CrawlableBot is a diagnostic crawler operated by Crawlable (usecrawlable.com) to test website crawlability, server-rendered raw HTML, and bot directive readiness.
If you found this page in your access logs, someone ran a diagnostic scan against your site. Everything below describes exactly what that involved.
The user-agent string
Every request CrawlableBot makes sends this header, unmodified:
CrawlableBot/1.0 (+https://usecrawlable.com/bot; AI-readability auditor)We do not rotate user agents, spoof a browser, or send requests under any other identity. If a request claims to be a browser, it is not us.
How it behaves
- On demand only. There is no schedule and no background crawling. A scan happens when a person enters a domain and asks for one, and stops when that scan finishes.
- Small and bounded. At most 40 pages per audit, four concurrent connections, and a hard 45-second wall-clock budget for the whole crawl. Add your
robots.txt,llms.txtand sitemap and the total is around fifty requests. - Read-only. GET requests for public pages. Nothing is submitted, no forms, no state changed.
- Public content only. We do not attempt to bypass logins, paywalls or access controls, and we do not index private data. A page that requires authentication is simply recorded as unreachable.
- No JavaScript. One HTTP request per page, no headless browser. That is the point: it measures what a non-rendering AI crawler would receive.
Blocking CrawlableBot
Add this to your robots.txt and we will stop, permanently and immediately:
User-agent: CrawlableBot Disallow: /
The rule is checked before any page is fetched, so an opted-out site is never crawled — the only request we make is for robots.txt itself. Anyone who tries to scan the site gets a refusal instead of a report.
Worth knowing
The rule has to name CrawlableBot. A wildcard Disallow: / under User-agent: * will not stop us, because measuring what happens to a site that blocks crawlers is the diagnosis the tool exists to perform — and the report will tell its owner exactly that.
You can also email support@usecrawlable.com and we will block your domain at our end. A real person reads it.
Seeing more traffic than this describes?
Then it probably is not us. Send us the log lines — timestamps, paths and source IPs — at support@usecrawlable.com. Anyone can put our string in their user-agent header, and we would rather know when someone does.