Can AI crawlers read Astro sites?
Static HTML
AI crawlers can read this platform by default.
Astro is about as good as it gets for AI readability: zero JavaScript by default means everything ships as HTML.
What is actually happening
Astro renders components to HTML at build time and ships no client JavaScript unless you explicitly opt a component in with a client: directive. The raw response is the finished page, which is the ideal case for a crawler that will not run scripts. The islands architecture means the interactive parts of your page are the exception rather than the default, so the amount of content that depends on hydration is small by construction.
13 of the major AI crawlers — including GPTBot, OAI-SearchBot, ChatGPT-User — fetch your HTML and parse it without running a JavaScript engine. Anything your page adds after hydration is not part of what they read.
What to do on Astro
- 01Keep client: directives on genuinely interactive islands only. A component marked client:only renders nothing at all server-side and is invisible in the HTML.
- 02Use content collections with typed frontmatter to generate per-page titles and descriptions automatically, so no page ships with a missing or duplicated tag.
- 03Add an llms.txt endpoint at src/pages/llms.txt.ts that generates the file from your content collection at build time. It then updates itself whenever you publish, which is the only way a file like this stays accurate.
- 04Emit Organization and WebSite JSON-LD in your base layout so every page inherits it, and add Article schema in the blog layout.
- 05Generate robots.txt from the same source as your sitemap at src/pages/robots.txt.ts, so the Sitemap directive can never point at a stale path.
What trips people up
- client:only renders nothing server-side. It is the one Astro directive that reliably costs you AI visibility, and it is easy to reach for when a component touches window during setup.
- Output mode "server" with no per-route prerendering makes behaviour depend on your adapter and cache configuration rather than on the framework, so the guarantees above no longer hold automatically.
- A high baseline score can make people stop checking. Astro removes the rendering problem; it does nothing about thin content, missing schema or a robots.txt that blocks retrieval crawlers.
Where the files go on Astro
Put llms.txt and robots.txt in /public, or generate them as endpoints at src/pages/llms.txt.ts and src/pages/robots.txt.ts.
Stop guessing where your site sits
The baseline above is the platform. The free scan measures your actual site: how many words a crawler reads, whether your pages come back as empty shells, and which crawlers your robots.txt lets in.
Scan my Astro site