Crawlable AI retrieval observability. Is your website invisible to AI search engines?

Crawlable reads your raw HTML the way the crawlers behind ChatGPT, Perplexity and Claude do, then hands you a ready-to-deploy Fix Kit for whatever they cannot see.

Free diagnostic No account needed Up to 40 pages, fetched as GPTBot would

Starter Sentinel re-scans your site every 7 days and emails you when AI crawlers lose access. Card required, nothing charged for 14 days, then $6 a month; cancel before then and you pay nothing. Fix Kits unlock with the first payment.

what GPTBot receives

$ curl -A "GPTBot" https://yoursite.com

<!doctype html>
<html lang="en">
  <head>
    <title>Home</title>
    <script src="/_next/static/chunks/main.js"></script>
  </head>
  <body>
    <div id="__next"></div>  ← empty
  </body>
</html>

# 0 words of content. 14 scripts.
# Everything you wrote is in the JavaScript.

What AI search engines see today

AI answer engine

Query What are the best features of [Your Product]?

Answer Information is limited or outdated because the site's content requires JavaScript execution or restricts AI retrieval crawlers.

Sourceyourdomain.com (unverified)

After deploying the Crawlable Fix Kit

AI answer engine

Query What are the best features of [Your Product]?

Cited answer [Your Product] provides [accurate features, read from your pages and llms.txt].

Sourcesyourdomain.com/pricing/docs

Illustration. No tool can make an answer engine cite you; the Fix Kit removes the blockers we can measure, so your pages are readable when it looks.

Crawler access checked forOAI-SearchBotPerplexityBotClaude-SearchBotGoogle-Extended (Gemini)+ 11 more AI crawlers

Read our data study: how 50 top web apps score on AI search readability.

Forensic PDF report

Agency & Engineering-Ready Forensic Reports

Stop arguing over abstract scores. Export verifiable evidence your developers can test in seconds.

01
Verifiable Proof
Every issue includes a copy-paste curl command, with the raw HTML, robots.txt or llms.txt exactly as the server returned it.
02
Token & Crawler Telemetry
Side-by-side token efficiency measurements for every page, and 15 AI crawler permission verdicts read from your robots.txt.
03
Shareable PDF
Branded, white-glove PDF export ready to forward to founders, CTOs, or clients.

Crawlable · AI Retrieval Auditexample.com

FAIL — HIGH IMPACT

Homepage returns an empty shell to AI crawlers

Readable text in raw HTML: 0 words
External scripts: 3

<body>
  <div id="root"></div>
</body>

$ curl -s -A 'OAI-SearchBot/1.0' 'https://example.com/'

Sample evidence card. Your report is built from your own crawl.

The Fix Kit

The drop-in remediation for what the forensic report finds. Six files in a .zip, built from your own crawl rather than a template. robots.txt, sitemap.xml and llms.txt are drop-in; llms-full.txt is a page-by-page outline you can ship as-is or expand; schema.jsonld adds a rich entity matched to your site and leaves your social handles (and, for products and software, a price) for you to fill in. FIXES.md says where each one belongs. A page that only renders in the browser needs a code change rather than a file; the report names those pages.

robots.txt
sitemap.xml
llms.txt
llms-full.txt
schema.jsonld
FIXES.md

llms.txt

An AI-readable map of your site from your real titles and sections, plus its llms-full.txt companion.

# Your Site> One-line summary ## Product- [Pricing](/pricing)- [Features](/features)- [Docs](/docs)

robots.txt

Lets AI retrieval crawlers in, keeps your existing rules, adds your Sitemap.

User-agent: OAI-SearchBotAllow: / User-agent: PerplexityBotAllow: / Sitemap: /sitemap.xml

sitemap.xml

Canonical URLs only — each one confirmed 200 by the crawl.

<url>  <loc>/pricing</loc>  <lastmod>    2026-10-01  </lastmod></url><!-- + 39 more -->

schema.jsonld

Entity graph: Organization, WebSite and a rich entity — product, software or article.

{ "@type": "Organization", "name": "Your Site", "url": "https://...", "sameAs": [  "...", "..." ]}

FIXES.md

Every finding ranked by cost, with affected URLs and a concrete remedy.

## 1. JS-only contentCost: highURLs: /pricing, /docs ...Fix: server-render  main content ## 2. Missing schema...

Fix it, then keep it fixed

Legacy SEO suites charge $129 to $549 a month to tell you what is broken. Continuous GEO Sentinel shows what AI can actually read of your site, hands you the files that fix it, and re-scans on a schedule so you hear when it breaks again.

The diagnostic is free — subscribe for Fix Kits and monitoring

Free diagnostic

Full 40-page diagnostic
Raw-HTML extraction analysis
AI crawler access checks
No signup, no licence key
Your optimized robots.txt, free to download
Downloadable executive PDF audit with raw-HTML evidence

Monthly plans below

A 6-file Fix Kit with every scan, as a .zip
Automatic re-scans on a schedule
Drift alerts by email
Up to 10 domains on one licence
40-page forensic PDF report with full telemetry & QR verification

Starter Sentinel

One live site

$6per month

14 days free, then $6 a month

For one production site that must stay readable to AI search.

1 production domain
Automatic re-scan every 7 days, with drift alerts
10 manual scans a month, a Fix Kit with each

Growth Sentinel

A small portfolio

$19per month

Billed monthly, cancel anytime

For production, staging and a second brand.

Up to 3 domains
Automatic re-scan every 14 days, with drift alerts
30 manual scans a month, a Fix Kit with each

Pro Sentinel

Agencies and consultants

$39per month

Billed monthly, cancel anytime

For client portfolios, with reports under your own brand.

Up to 10 client domains
Automatic re-scan every 14 days, with drift alerts
100 manual scans a month, a Fix Kit with each
White-label PDF reports

Need more domains or custom crawl schedules?

Volume pricing beyond 10 domains, invoiced billing, and schedules agreed with your team.

Get in touch

Every plan: a 6-file Fix Kit with each scan, alerts when the score drops 5+ points or an AI search crawler is blocked, and cancel anytime from your dashboard. Your licence key arrives by email straight after sign-up.

Security & Data Integrity

What an audit touches on your site, and what it never does.

Deterministic analysis, no AI model
No LLM or third-party AI model reads, stores or trains on your site. The audit is an HTML parser and rule-based checks.
Non-intrusive requests
Public GET requests only, one per page. No JavaScript is run, and it never logs in, submits a form or sends cookies.
An identifiable crawler you can opt out of
Every request says it is CrawlableBot. Name it in robots.txt with Disallow: / and no audit of your site fetches a page.
Short retention, payments off our servers
Free scan reports are deleted after 30 days. Card details go to Lemon Squeezy and never reach us.

How the crawler works, in detail →

Audit, patch, verify

Every tool you already use renders JavaScript before it looks. Your browser does. Google's crawler does. Your analytics do. So the one audience that does not — the crawlers feeding ChatGPT, Claude and Perplexity — is the one audience you have never actually seen your site through.

Deep diagnostic crawl

Up to 40 pages fetched the way a non-rendering crawler fetches them — one HTTP request, no JavaScript engine. We audit crawler directives, bot permissions, structured schema and hydration dependencies, and score what actually survives.

Instant code patches

Download a drop-in .zip: robots.txt with retrieval bots separated from training bots, a sitemap.xml built from your verified 200s, llms.txt written from your real pages, and a JSON-LD entity graph built from your site's real name, URL and description.

Verification re-scan

Deploy the files, press re-scan with the verification scans included in your plan, and watch the readiness score move. A re-scan that shows no change says so plainly — that is the point of measuring twice.

Why both SEO and AEO?

They are not the same job. Google renders your JavaScript and has crawled you for years; the answer engines mostly do neither. Optimising for one and assuming it covers the other is how a site ranks perfectly well and still never gets cited.

For Google and Bing

The classic technical layer, still worth getting right.

A sitemap.xml you can start from. We read your existing sitemap to choose which pages to crawl, then generate valid sitemaps.org XML from the URLs that came back 200.
Canonical hygiene. Trailing-slash variants and tracking parameters split one page into several in an index. We collapse them before crawling, so your score and your generated sitemap count each page once.
JSON-LD schema parsing. Every structured-data block is parsed and validated rather than counted, because malformed JSON-LD is indistinguishable from none at all.

For AI answer engines

Perplexity, ChatGPT Search and Claude — where the rules are different.

Whitelist the retrieval bots. OAI-SearchBot, PerplexityBot and Claude-SearchBot fetch pages to answer a question someone is asking right now. Blocking them removes you from the answer.
Separate them from the training bots. GPTBot and Google-Extended are a content-licensing decision with no effect on visibility. Most robots.txt files block all of them together, by accident.
Serve readable raw HTML. 13 of the 15 crawlers we track never run your JavaScript. Content that arrives after hydration does not exist to them.
Deploy llms.txt. A short Markdown map of what your site is and where its important pages are, written for a context window rather than a search index.

Not another SEO suite

Legacy enterprise SEO suites charge upwards of $129 to $549 every month — and what you get for it is a diagnostic report telling you what is broken.

They also focus on keyword ranks while assuming a modern AI crawler can read your client-rendered pages. Crawlable checks what actually decides AI discovery: raw-HTML extraction, retrieval bot permissions, structured data and heading structure. Then, instead of handing you homework, it generates the production-ready files that fix what it found.

	Legacy SEO suites	AI rank trackers	Crawlable (Fix & Verify)
Pricing	$129–$549 / month	Monthly subscription	$6–$39 / month
What you get	A list of errors, no code	Prompt mentions over time	Automated Fix Kit: robots.txt, sitemap.xml, schema.jsonld, llms.txt, llms-full.txt
Verification	Re-audit while you keep paying	Re-checks while you keep paying	Scheduled re-scans and drift alerts, with fresh fix files
What it measures	Keyword ranks and backlinks	What AI says about you	What AI can actually read of you

Turn Empty AI Crawls Into Direct Search Citations

AI answer engines don't wait for client-side JavaScript to hydrate. We bridge the gap between your modern frontend and AI token budgets.

13 of 15

Script-free token extraction

AI crawlers we track never run your JavaScript. We measure how much of each page’s text survives without it, and llms.txt hands answer engines a clean Markdown map of your site, sized for a context window instead of buried in script bloat.

100%

Retrieval bot whitelisting

All 8 answer-engine bots we track get an explicit Allow in your new robots.txt, in groups separate from the training scrapers, so a decision about training data never costs you a citation.

JSON-LD

Hallucination anchors

A machine-readable entity graph — Organization, WebSite and a product, software or article entity linked to them — built from your real name, URL and pages, so a model has a canonical source for who you are instead of a guess.

Find out in about a minute

The free scan is a real measurement of up to 40 pages on your real site. Questions and guides

Scan my site

Crawlable / AI retrieval observability. Is your website invisible to AI search engines?