Can AI crawlers read Gatsby sites?
Static HTML
AI crawlers can read this platform by default.
Gatsby builds static HTML per route, so content is readable — the risk is components that only render after hydration.
What is actually happening
Gatsby compiles every page to HTML at build time from its GraphQL data layer, so content sourced through a page query is in the response before any script runs. The exceptions are specific and worth knowing: anything fetched in useEffect, anything gated behind a window check, and any route registered as client-only. Those render nothing at build time and therefore nothing for a crawler.
13 of the major AI crawlers — including GPTBot, OAI-SearchBot, ChatGPT-User — fetch your HTML and parse it without running a JavaScript engine. Anything your page adds after hydration is not part of what they read.
What to do on Gatsby
- 01Source content through page queries rather than runtime fetches, so it is baked into the build output.
- 02Use the Gatsby Head API for metadata so titles and descriptions are emitted at build time rather than set by the browser after load.
- 03Audit every component gated on typeof window !== "undefined". That guard is correct for avoiding build errors and also means the component contributes nothing to the HTML.
- 04Generate llms.txt from your GraphQL data in gatsby-node, so the file is rebuilt from the same source as your pages and cannot drift.
- 05Emit Organization schema from your layout and Article schema from the blog template, both at build time.
What trips people up
- Deferred Static Generation pages are built on first request, so a crawler arriving before a human may receive a placeholder rather than the page.
- Client-only routes defined with matchPath produce no static HTML at all — they are invisible by design, which is fine for app routes and not for content.
- A long build can tempt you toward client-side data fetching for freshness. That trade buys you fresher data and costs you the crawler entirely; incremental builds are the better answer.
Where the files go on Gatsby
Put llms.txt and robots.txt in the /static folder, which is copied to the build root.
Stop guessing where your site sits
The baseline above is the platform. The free scan measures your actual site: how many words a crawler reads, whether your pages come back as empty shells, and which crawlers your robots.txt lets in.
Scan my Gatsby site