13 of 15 AI crawlers never run your JavaScript
Is your website invisibleto AI search engines?
Traditional SEO is only half the battle. When someone searches on ChatGPT, Perplexity or Claude, will your site be cited — or ignored? Crawlable audits your raw-HTML extraction bottlenecks, generates ready-to-deploy Fix Kits (robots.txt, sitemap.xml, schema.jsonld, llms.txt), and re-scans to verify your score.
Launch Special: Free Deep Audit (Up to 40 Pages)
Normally limited to single-page audits on the free tier. Full 40-page crawler enabled during our Public Beta.
Up to 40 pages, no signup, about a minute. We fetch your HTML exactly as GPTBot would.
Engineered for web apps, modern websites, e-commerce, and agencies.
$ curl -A "GPTBot" https://yoursite.com
<!doctype html>
<html lang="en">
<head>
<title>Home</title>
<script src="/_next/static/chunks/main.js" defer></script>
</head>
<body>
<div id="__next"></div>
</body>
</html>
# 0 words of content. 14 scripts.
# Everything you wrote is in the JavaScript.Your visitors never see this. Neither do you — your browser runs the scripts. The crawler does not.
Audit, patch, verify
Every tool you already use renders JavaScript before it looks. Your browser does. Google's crawler does. Your analytics do. So the one audience that does not — the crawlers feeding ChatGPT, Claude and Perplexity — is the one audience you have never actually seen your site through.
Deep diagnostic crawl
Up to 40 pages fetched the way a non-rendering crawler fetches them — one HTTP request, no JavaScript engine. We audit crawler directives, bot permissions, structured schema and hydration dependencies, and score what actually survives.
Instant code patches
Download a drop-in .zip: robots.txt with retrieval bots separated from training bots, a sitemap.xml built from your verified 200s, llms.txt written from your real pages, and a JSON-LD entity graph built from your site's real name, URL and description.
Verification re-scan
Deploy the files, press re-scan with the verification scans included in your plan, and watch the readiness score move. A re-scan that shows no change says so plainly — that is the point of measuring twice.
Why both SEO and AEO?
They are not the same job. Google renders your JavaScript and has crawled you for years; the answer engines mostly do neither. Optimising for one and assuming it covers the other is how a site ranks perfectly well and still never gets cited.
For Google and Bing
The classic technical layer, still worth getting right.
- A sitemap.xml you can start from. We read your existing sitemap to choose which pages to crawl, then generate valid sitemaps.org XML from the URLs that came back 200.
- Canonical hygiene. Trailing-slash variants and tracking parameters split one page into several in an index. We collapse them before crawling, so your score and your generated sitemap count each page once.
- JSON-LD schema parsing. Every structured-data block is parsed and validated rather than counted, because malformed JSON-LD is indistinguishable from none at all.
For AI answer engines
Perplexity, ChatGPT Search and Claude — where the rules are different.
- Whitelist the retrieval bots.
OAI-SearchBot,PerplexityBotandClaude-SearchBotfetch pages to answer a question someone is asking right now. Blocking them removes you from the answer. - Separate them from the training bots.
GPTBotandGoogle-Extendedare a content-licensing decision with no effect on visibility. Most robots.txt files block all of them together, by accident. - Serve readable raw HTML. 13 of the 15 crawlers we track never run your JavaScript. Content that arrives after hydration does not exist to them.
- Deploy llms.txt. A short Markdown map of what your site is and where its important pages are, written for a context window rather than a search index.
This is not another recurring SEO subscription
Legacy enterprise SEO suites charge upwards of $129 to $549 every month on a recurring retainer — and what you get for it is a diagnostic report telling you what is broken.
They also focus on keyword ranks while assuming a modern AI crawler can read your client-rendered pages. Crawlable checks what actually decides AI discovery: raw-HTML extraction, retrieval bot permissions, structured data and heading structure. Then, instead of handing you homework, it generates the production-ready files that fix what it found.
| Legacy SEO suites | AI rank trackers | Crawlable (Fix & Verify) | |
|---|---|---|---|
| Pricing | $129–$549 / month | Monthly subscription | $29–$199 one-time |
| What you get | A list of errors, no code | Prompt mentions over time | Automated Fix Kit: robots.txt, sitemap.xml, schema.jsonld, llms.txt |
| Verification | Re-audit while you keep paying | Re-checks while you keep paying | Included re-scans to verify the fix landed |
| What it measures | Keyword ranks and backlinks | What AI says about you | What AI can actually read of you |
The Fix Kit
Five files in a .zip, built from your own crawl rather than a template. Four are drop-in; schema.jsonld leaves two social handles and a search URL for you to fill in. FIXES.md says where each one belongs.
robots.txtRetrieval crawlers allowed explicitly, training crawlers listed separately so blocking them stays a licensing decision rather than an accident. Your existing wildcard Disallow rules are carried over, with a Sitemap directive appended.
sitemap.xmlValid sitemaps.org XML built only from canonical URLs the crawl confirmed returning 200 — no redirects, no 404s, no tracking parameters. It covers the pages this audit crawled, so a larger site gets a correct template to extend rather than a replacement sitemap.
llms.txtBuilt from your actual page titles, descriptions and sections — not a blank template. Grouped into docs, products, blog and the rest, sized for a context window.
schema.jsonldAn entity graph: Organization and WebSite with SearchAction, plus BreadcrumbList when your site has nested pages — filled in with your real name, URL and description, with placement instructions inline.
FIXES.mdEvery finding ordered by what it costs you, with the affected URLs listed, a concrete remedy per item, and the deployment order for the four files above.
Pay once, keep the files
Legacy SEO suites charge $129 to $549 a month, on a recurring retainer, to tell you what is broken. This tells you what AI can actually read of your site, hands you the files that fix it, and re-scans to prove they worked.
Public Beta — the diagnostic is free
Free Beta Audit
- Full 40-page diagnostic
- Raw-HTML extraction analysis
- AI crawler access checks
- No signup, no licence key
Fix Kit plans below
- Automated code generation
- Downloadable .zip Fix Kit
- Re-scan validation credits
- Domain tracking across a portfolio
Starter
Fix & Verify
For one website or web app you want cited, not ignored.
- 1 website or web app audited, up to 40 pages
- Fix Kit .zip: robots.txt, sitemap.xml, llms.txt, schema.jsonld
- 2 verification re-scans included (30-day window)
- Re-scan from your report for a before-and-after score
- FIXES.md with deployment instructions
Growth
For a small portfolio: production, staging and a second brand.
- Up to 3 websites, web apps or client domains tracked
- 10 total scans across those domains (60-day window)
- A complete Fix Kit for each of the 3 sites
- Re-scan any of them to prove a fix landed, without spending a slot
- Every Fix Kit re-downloadable from your dashboard
Agency Pro
For consultants, agencies, and web developers.
- Up to 15 client domains tracked — sites, stores or web apps
- 50 total scans across the portfolio (60-day window)
- A Fix Kit per client site, generated from that site's own crawl
- Enough scans to audit a portfolio and verify every fix
- Report links you can send straight to a client
Your licence key arrives by email straight after payment.
Questions
- Is it really true that AI crawlers do not run JavaScript?
- For most of them, yes. Two exceptions do render JavaScript: Google-Extended, which inherits Google's rendering infrastructure, and Apple's Applebot-Extended. GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot and the rest fetch raw HTML and parse it. That is why a site can rank perfectly well in Google Search and still be absent from AI answers: the two pipelines do not see the same page.
- How is this different from an AI visibility tracker?
- Those tools monitor what AI systems say about your brand across a set of prompts, billed monthly. Crawlable measures whether AI systems can read your site at all, and generates the files to fix it, billed once. They answer different questions, and this one comes first — prompt monitoring on a site a crawler cannot read tells you only that you are absent.
- Does llms.txt actually do anything?
- It depends who you ask, and the audit is honest about that. Google has publicly said llms.txt does not help with its AI features. Anthropic and OpenAI both publish llms.txt files for their own developer sites and recommend them for agent workflows, and Perplexity has been observed using them. It is one small file, it can only help the agent case, and it carries no risk — so the audit flags a missing one as a warning rather than a failure, and generates it for you either way.
- What does the free scan include?
- One page of your choice, fetched as a non-rendering crawler, plus your robots.txt and llms.txt. You get the real score, the real findings and the real per-page numbers. What it does not include is the multi-page crawl and the generated fix files.
- How many pages does a paid audit cover?
- Up to 40, sampled across your sitemap so the audit covers your homepage, docs, product and blog rather than 40 near-identical URLs from one section. On most sites that is enough to tell you everything a full crawl would about AI readability, because rendering problems are template-level, not page-level.
- Will the crawl hurt my site?
- No. It is at most 40 page fetches at four concurrent connections, plus your robots.txt, llms.txt and sitemap — roughly fifty requests in total, with a polite user agent and a per-request timeout. That is a fraction of what a single search engine crawl does in an hour.
- Can I share a report with a client?
- Yes. Every report has its own link that anyone can open with no login and no account. The generated fix files stay private to your licence — a client can read the findings without being able to download the kit.
- How is payment and tax handled?
- Checkout runs on a hosted payment page, so Crawlable never sees or stores a card detail. VAT and sales tax are calculated and remitted for your country automatically, and you get a proper invoice. Your licence key is stored only as a hash, plus its last four characters so you can tell your keys apart.
- What if the audit finds nothing wrong?
- Then you have confirmation, which is worth something, and a generated llms.txt and robots.txt you did not have before. Sites built on Astro, Gatsby, WordPress and well-configured Next.js often score in the 80s and 90s. The audit is only surprising when it is bad.
Find out in twenty seconds
The free scan is a real measurement of a real page on your real site.
Scan my site