13 of 15 AI crawlers never run your JavaScript
Your site looks fine.AI can't read it.
GPTBot, ClaudeBot and PerplexityBot fetch your raw HTML and never execute a line of your JavaScript. If your content arrives after hydration, it does not exist as far as they are concerned — and nobody tells you.
Crawlable measures exactly how much of your site they can read, then generates the files that fix it.
One page, no signup, about twenty seconds. We fetch your HTML exactly as GPTBot would.
$ curl -A "GPTBot" https://yoursite.com
<!doctype html>
<html lang="en">
<head>
<title>Home</title>
<script src="/_next/static/chunks/main.js" defer></script>
</head>
<body>
<div id="__next"></div>
</body>
</html>
# 0 words of content. 14 scripts.
# Everything you wrote is in the JavaScript.Your visitors never see this. Neither do you — your browser runs the scripts. The crawler does not.
Why this is invisible to you
Every tool you already use renders JavaScript before it looks. Your browser does. Google's crawler does. Your analytics do. So the one audience that does not — the crawlers feeding ChatGPT, Claude and Perplexity — is the one audience you have never actually seen your site through.
We fetch like a bot, not a browser
One HTTP request, no JavaScript engine, no hydration. Exactly what GPTBot gets, measured in words rather than opinions.
We score what survives
Readable words, text-to-markup ratio, empty framework shells, heading structure, structured data, and which crawlers your robots.txt actually lets in.
We hand you the fix
A generated llms.txt built from your real pages, a robots.txt that keeps your existing rules while unblocking retrieval crawlers, and the JSON-LD your pages are missing.
This is not another AI visibility tracker
The monitoring tools run prompts and report which brands got mentioned. Useful, and priced accordingly — $95 to $500 a month, forever.
They all assume the crawler can read your site. That assumption is the thing worth checking first, and it is the one thing none of them check.
| Trackers | Crawlable | |
|---|---|---|
| Answers | What AI says about you | What AI can read of you |
| Price | $95–$500 / month | $39 once |
| Output | Dashboards and trends | Files you paste today |
| Needed when | You already rank | Before anything else works |
What lands in your inbox
llms.txtBuilt from your actual page titles, descriptions and sections — not a blank template. Grouped into docs, products, blog and the rest.
robots.txtExplicit Allow groups for the 8 retrieval crawlers, your existing wildcard rules carried over untouched, and a Sitemap directive.
schema.jsonldOrganization, WebSite and BreadcrumbList blocks filled in with your real name, URL and description. Paste and ship.
FIXES.mdEvery finding ordered by what it costs you, with the affected URLs listed and a concrete remedy per item.
Pay once, keep the files
The monitoring tools charge $95 to $500 a month to tell you what AI says about you. This tells you what AI can read, and hands you the fix.
Single audit
One site, fully audited, with the fix files.
- Up to 40 pages crawled as a non-rendering crawler sees them
- Per-page raw-HTML readability verdict
- AI crawler policy check across 15 bots
- Structured data, heading and metadata audit
- Generated llms.txt, robots.txt and JSON-LD
- FIXES.md prioritised by revenue impact
Agency pack
Five client audits at half the per-site price.
- Everything in Single audit, five times
- Credits never expire
- Re-run an audit to show before and after
- Share a report by link with no login
- Priority email support
Agency
Unlimited audits under your own brand.
- Unlimited audits, unlimited sites
- White-label reports with your name and colour
- Re-audit clients monthly to prove progress
- API access for your own dashboards
- Cancel any time
Paid through Lemon Squeezy, which acts as merchant of record and handles VAT and sales tax worldwide. Your license key arrives by email straight after payment.
Questions
- Is it really true that AI crawlers do not run JavaScript?
- For most of them, yes. Google's crawlers render JavaScript, and Google-Extended and Applebot-Extended inherit that infrastructure. GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot and the rest fetch raw HTML and parse it. That is why a site can rank perfectly well in Google Search and still be absent from AI answers: the two pipelines do not see the same page.
- How is this different from Profound, Peec or Otterly?
- Those tools monitor what AI systems say about your brand across a set of prompts, billed monthly. Crawlable measures whether AI systems can read your site at all, and generates the files to fix it, billed once. They answer different questions, and this one comes first — prompt monitoring on a site a crawler cannot read tells you only that you are absent.
- Does llms.txt actually do anything?
- It depends who you ask, and the audit is honest about that. Google has publicly said llms.txt does not help with its AI features. Anthropic and OpenAI both publish llms.txt files for their own developer sites and recommend them for agent workflows, and Perplexity has been observed using them. It is one small file, it can only help the agent case, and it carries no risk — so the audit flags a missing one as a warning rather than a failure, and generates it for you either way.
- What does the free scan include?
- One page of your choice, fetched as a non-rendering crawler, plus your robots.txt and llms.txt. You get the real score, the real findings and the real per-page numbers. What it does not include is the multi-page crawl and the generated fix files.
- How many pages does a paid audit cover?
- Up to 40, sampled across your sitemap so the audit covers your homepage, docs, product and blog rather than 40 near-identical URLs from one section. On most sites that is enough to tell you everything a full crawl would about AI readability, because rendering problems are template-level, not page-level.
- Will the crawl hurt my site?
- No. It is at most 40 GET requests at four concurrent connections, with a polite user agent and a per-request timeout. That is a fraction of what a single search engine crawl does in an hour.
- Can I put my own branding on the report?
- On the Agency plan, yes — set your name and brand colour once and every report you generate carries them. The one-time plans produce Crawlable-branded reports, which are still shareable by link with no login required.
- Who handles payment and tax?
- Lemon Squeezy, acting as merchant of record. That means they are the seller of record on your invoice and they handle VAT and sales tax in whichever country you are in. Crawlable never sees or stores a card detail, and your license key is stored hashed.
- What if the audit finds nothing wrong?
- Then you have confirmation, which is worth something, and a generated llms.txt and robots.txt you did not have before. Sites built on Astro, Gatsby, WordPress and well-configured Next.js often score in the 80s and 90s. The audit is only surprising when it is bad.
Find out in twenty seconds
The free scan is a real measurement of a real page on your real site.
Scan my site