Can AI crawlers read Next.js sites?
Hybrid
Readability depends on how the site is configured.
Next.js is excellent for AI crawlers when you use Server Components or static generation, and invisible to them when you opt into client rendering.
What is actually happening
The App Router renders Server Components to HTML on the server, so the text ships in the initial response. The moment a route is marked "use client" and fetches its data in an effect, the served HTML is an empty shell and every non-rendering crawler sees nothing.
13 of the major AI crawlers — including GPTBot, OAI-SearchBot, ChatGPT-User — fetch your HTML and parse it without running a JavaScript engine. Anything your page adds after hydration is not part of what they read.
What to do on Next.js
- 01Audit which routes carry "use client" at the page level — a client boundary around an interactive widget is fine, one around the whole page is not.
- 02Move data fetching into Server Components or route-level async functions rather than useEffect.
- 03Use generateStaticParams for content routes so pages are built ahead of time.
- 04Set generateMetadata per route so every page has a unique title and description in the raw HTML.
- 05Check the output of `curl -s https://yoursite.com | grep -c "your body text"` — if it is zero, the crawler sees zero too.
What trips people up
- Streaming with Suspense still ships the fallback first. Crawlers that read the full response get the streamed content, but a crawler that cuts the connection early may not.
- next/dynamic with ssr:false removes the component from the HTML entirely.
- A page that renders fine in the browser proves nothing — the browser runs the JavaScript the crawler skips.
Where the files go on Next.js
Put llms.txt in /public/llms.txt. Generate robots.txt with an app/robots.ts route, or place a static file in /public.
Stop guessing where your site sits
The baseline above is the platform. The free scan measures your actual site: how many words a crawler reads, whether your pages come back as empty shells, and which crawlers your robots.txt lets in.
Scan my Next.js site