# Crawlable > Crawlable audits any website or web app the way non-rendering AI crawlers read it — raw HTML, no JavaScript — then generates a Fix Kit (robots.txt, sitemap.xml, llms.txt, schema.jsonld) and re-scans to verify the fixes landed. Free single-page scan; $29 for a full 40-page audit plus two verification re-scans. Most AI crawlers (GPTBot, ClaudeBot, PerplexityBot and others) do not execute JavaScript. Content that only appears after hydration is invisible to them. Crawlable measures how much of a site they can actually read. ## Start here - [Free AI readability scan](https://usecrawlable.com/#scan): enter a domain, get a real measurement of one page in about twenty seconds. - [Pricing](https://usecrawlable.com/#pricing): $29 one time for 1 domain and 3 scans, $79 for 3 domains and 10 scans, $199 for 15 domains and 50 scans. - [FAQ](https://usecrawlable.com/#faq): how this differs from an AI visibility tracker, whether llms.txt works, what the audit covers. ## Platform guides - [Can AI crawlers read Next.js sites?](https://usecrawlable.com/platforms/nextjs): Next.js is excellent for AI crawlers when you use Server Components or static generation, and invisible to them when you opt into client rendering. - [Can AI crawlers read React (create-react-app / Vite SPA) sites?](https://usecrawlable.com/platforms/react): A plain React SPA is close to invisible to AI crawlers: the served HTML is a near-empty div and everything else arrives via JavaScript they do not run. - [Can AI crawlers read WordPress sites?](https://usecrawlable.com/platforms/wordpress): WordPress serves complete HTML by default, which puts it near the top for AI readability — the usual problems are structured data and bot blocking, not rendering. - [Can AI crawlers read Shopify sites?](https://usecrawlable.com/platforms/shopify): Shopify serves Liquid-rendered HTML, so products are readable — but collection filtering, reviews and recommendations are usually client-side and invisible. - [Can AI crawlers read Webflow sites?](https://usecrawlable.com/platforms/webflow): Webflow publishes static HTML with the content baked in, which makes it genuinely good for AI crawlers — the weak spot is CMS collection pages with thin content. - [Can AI crawlers read Squarespace sites?](https://usecrawlable.com/platforms/squarespace): Squarespace serves most page content in HTML but loads galleries, summary blocks and some newer sections client-side, so parts of the page go missing for AI crawlers. - [Can AI crawlers read Wix sites?](https://usecrawlable.com/platforms/wix): Wix has improved a great deal but still ships a heavily script-driven page, so raw-HTML content is thinner than what a visitor sees. - [Can AI crawlers read Astro sites?](https://usecrawlable.com/platforms/astro): Astro is about as good as it gets for AI readability: zero JavaScript by default means everything ships as HTML. - [Can AI crawlers read Framer sites?](https://usecrawlable.com/platforms/framer): Framer publishes static HTML with text included, so it reads reasonably well — though heavy use of effects and CMS components thins out the raw content. - [Can AI crawlers read Vue & Nuxt sites?](https://usecrawlable.com/platforms/vue-nuxt): Nuxt with SSR on is readable; a plain Vue SPA or Nuxt with ssr:false is not. - [Can AI crawlers read Angular sites?](https://usecrawlable.com/platforms/angular): A default Angular application renders entirely in the browser, so AI crawlers receive an empty and nothing else. - [Can AI crawlers read Gatsby sites?](https://usecrawlable.com/platforms/gatsby): Gatsby builds static HTML per route, so content is readable — the risk is components that only render after hydration. ## AI crawler reference - [All AI crawlers](https://usecrawlable.com/ai-crawlers): which crawlers cost you visibility when blocked, and which are a licensing choice. - [GPTBot](https://usecrawlable.com/ai-crawlers/gptbot): OpenAI. Collects content for OpenAI model training. Blocking it does not remove you from ChatGPT answers. - [OAI-SearchBot](https://usecrawlable.com/ai-crawlers/oai-searchbot): OpenAI. Builds the index behind ChatGPT search results. Blocking it removes you from ChatGPT search. - [ChatGPT-User](https://usecrawlable.com/ai-crawlers/chatgpt-user): OpenAI. Fetches a page when a ChatGPT user follows or asks about a specific link. - [ClaudeBot](https://usecrawlable.com/ai-crawlers/claudebot): Anthropic. Collects content for Anthropic model training. - [Claude-SearchBot](https://usecrawlable.com/ai-crawlers/claude-searchbot): Anthropic. Indexes pages to support Claude search results. - [Claude-User](https://usecrawlable.com/ai-crawlers/claude-user): Anthropic. Fetches a page on behalf of a Claude user following a link. - [PerplexityBot](https://usecrawlable.com/ai-crawlers/perplexitybot): Perplexity. Builds the Perplexity index. Blocking it removes you from Perplexity citations. - [Perplexity-User](https://usecrawlable.com/ai-crawlers/perplexity-user): Perplexity. Fetches a page when a Perplexity user opens or asks about a specific link. - [Google-Extended](https://usecrawlable.com/ai-crawlers/google-extended): Google. Controls use of your content for Gemini training and for grounding answers in Gemini Apps and Vertex AI. Does not affect Google Search or AI Overviews. - [Applebot-Extended](https://usecrawlable.com/ai-crawlers/applebot-extended): Apple. Controls use of your content for Apple foundation model training. - [Amazonbot](https://usecrawlable.com/ai-crawlers/amazonbot): Amazon. Feeds Alexa and Amazon answer surfaces. - [Bytespider](https://usecrawlable.com/ai-crawlers/bytespider): ByteDance. Collects content for ByteDance model training. Widely blocked for aggressive crawl rates. - [meta-externalagent](https://usecrawlable.com/ai-crawlers/meta-externalagent): Meta. Collects content for Meta AI model training. - [cohere-ai](https://usecrawlable.com/ai-crawlers/cohere-ai): Cohere. Collects content for Cohere model training. - [MistralAI-User](https://usecrawlable.com/ai-crawlers/mistralai-user): Mistral. Fetches a page on behalf of a Le Chat user. ## Guides - [Blog](https://usecrawlable.com/blog): practical guides to AI search visibility. - [Google SEO vs. AI Search Optimization: Key Differences Every Founder Must Know](https://usecrawlable.com/blog/google-seo-vs-ai-search-optimization): How AI answer engines choose sources differently from Google, where backlinks still matter, how citations replace rankings, and how to measure AI referral traffic. - [How to Create and Optimize an llms.txt File for Your Website](https://usecrawlable.com/blog/how-to-create-llms-txt): The llms.txt format, where the file goes, how to keep it token-efficient, working examples for Next.js and static sites, and an honest account of who uses it. - [How to Configure robots.txt for AI Crawlers (Without Compromising Security)](https://usecrawlable.com/blog/robots-txt-for-ai-crawlers): Allow the AI crawlers that get you cited, opt out of the ones that only train models, and avoid treating robots.txt as a security control. Includes a ready-made file. - [What is Generative Engine Optimization (GEO)? The Complete 2026 Guide](https://usecrawlable.com/blog/what-is-generative-engine-optimization): How ChatGPT Search, Perplexity and Claude find, read and cite pages, why that differs from Googlebot, and what the research says actually improves AI visibility. - [Why AI Search Engines Can’t Read Client-Side Rendered (CSR) Sites](https://usecrawlable.com/blog/why-ai-search-engines-cant-read-client-side-rendered-sites): Most AI crawlers fetch raw HTML and never run your JavaScript. Here's what they actually see on a client-rendered site, how to test it, and how to fix it. ## Key facts - Is it really true that AI crawlers do not run JavaScript? For most of them, yes. Two exceptions do render JavaScript: Google-Extended, which inherits Google's rendering infrastructure, and Apple's Applebot-Extended. GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot and the rest fetch raw HTML and parse it. That is why a site can rank perfectly well in Google Search and still be absent from AI answers: the two pipelines do not see the same page. - How is this different from an AI visibility tracker? Those tools monitor what AI systems say about your brand across a set of prompts, billed monthly. Crawlable measures whether AI systems can read your site at all, and generates the files to fix it, billed once. They answer different questions, and this one comes first — prompt monitoring on a site a crawler cannot read tells you only that you are absent. - Does llms.txt actually do anything? It depends who you ask, and the audit is honest about that. Google has publicly said llms.txt does not help with its AI features. Anthropic and OpenAI both publish llms.txt files for their own developer sites and recommend them for agent workflows, and Perplexity has been observed using them. It is one small file, it can only help the agent case, and it carries no risk — so the audit flags a missing one as a warning rather than a failure, and generates it for you either way. ## Optional - [Sitemap](https://usecrawlable.com/sitemap.xml): every indexable URL.