Generative Engine Optimization (GEO) for Next.js

The platform pitfall: content that only exists after hydration

OAI-SearchBot, PerplexityBot and Claude-SearchBot fetch the HTML your server returns and read it without running a JavaScript engine. So do 13 of the 15 AI crawlers Crawlable tracks. Whatever is not in that first response does not exist for them.

'use client' on its own is not the problem. Next.js still renders client components to HTML on the server. The ceiling appears when a component renders its real content only after it mounts: data fetched in useEffect, components loaded with next/dynamic and ssr: false, or markup hidden behind an isMounted flag. The server sends the loading skeleton, and the skeleton is what gets indexed.

'use client';

import { useEffect, useState } from 'react';

export default function PricingPage() {
  const [plans, setPlans] = useState<Plan[]>([]);

  useEffect(() => {
    fetch('/api/plans').then((r) => r.json()).then(setPlans);
  }, []);

  // The server renders this branch. It is all OAI-SearchBot ever receives.
  if (plans.length === 0) return <div className="skeleton" />;

  return <PlanTable plans={plans} />;
}

// No 'use client': a Server Component, rendered to HTML before it is sent.
export default async function PricingPage() {
  const plans = await getPlans(); // your database or CMS call

  // Interactive pieces can still be client components. They get their data
  // as props, so their HTML is rendered on the server too.
  return <PlanTable plans={plans} />;
}

Crawlable scores this as raw-HTML readability, 35 of the 100 points. A page served as an empty shell takes the full penalty for its share of the site, and a single empty shell caps the whole score at 64, however good your schema and llms.txt are.

The quickest check is curl -s https://example.com/pricing | grep -c "a sentence from the page". Zero means a non-rendering crawler gets zero too.

STEP 1 OF 3

Serve llms.txt from an App Router route

A folder named llms.txt with a route.ts inside answers at /llms.txt. Generating the file from the same data as your pages means it cannot drift out of date the way a hand-edited file in public/ does.

Route handlers are dynamic by default in Next.js 15, so export const dynamic = 'force-static' builds the file once at next build and serves it as a static asset.

// app/llms.txt/route.ts
export const dynamic = 'force-static';

const SITE = 'https://example.com';

const PAGES = [
  { section: 'Docs', title: 'Getting started', path: '/docs/getting-started', summary: 'Install, configure and deploy in ten minutes.' },
  { section: 'Docs', title: 'API reference', path: '/docs/api', summary: 'Every endpoint, parameter and error code.' },
  { section: 'Product', title: 'Pricing', path: '/pricing', summary: 'Plans, limits and what each one includes.' },
];

export function GET() {
  const sections = new Map<string, string[]>();
  for (const page of PAGES) {
    const lines = sections.get(page.section) ?? [];
    lines.push(`- [${page.title}](${SITE}${page.path}): ${page.summary}`);
    sections.set(page.section, lines);
  }

  const body = [
    '# Example Co',
    '',
    '> Example Co makes invoicing software for freelancers. These pages explain what it does, how to set it up and what it costs.',
    '',
    ...[...sections].flatMap(([name, lines]) => [`## ${name}`, '', ...lines, '']),
  ].join('\n');

  return new Response(body, {
    headers: { 'Content-Type': 'text/markdown; charset=utf-8' },
  });
}

• ✓Keep the shape Crawlable validates: one # heading, a > summary line, ## sections, and absolute links with a description after each one. A well-formed file earns 15 of the 20 llms.txt points.
• ✓Add app/llms-full.txt/route.ts the same way, with the full text of your key pages, for the remaining 5.
• ✓Pick one source: a route or public/llms.txt, not both.
• ✓If middleware handles locales or auth, make sure its matcher skips /llms.txt. A redirect to /en/llms.txt counts as no file.

STEP 2 OF 3

Separate retrieval bots from training crawlers in app/robots.ts

Each AI company runs different crawlers for different jobs. OAI-SearchBot builds the index ChatGPT Search answers from; ChatGPT-User fetches a page when a person asks about it; GPTBot collects training data. Blocking GPTBot does not remove you from ChatGPT answers. Blocking OAI-SearchBot does.

Name both kinds, in separate groups, so each rule says what you meant. Next.js writes an array userAgent as one group with several User-agent lines.

// app/robots.ts
import type { MetadataRoute } from 'next';

// Search and answer engines: allowed, so your pages can be retrieved and cited.
const RETRIEVAL = ['OAI-SearchBot', 'ChatGPT-User', 'Claude-SearchBot', 'Claude-User', 'PerplexityBot', 'Perplexity-User'];

// Bulk training crawlers: a licensing decision, not a visibility one.
// Disallowed here; set allow: '/' instead if you are happy to be trained on.
// Google-Extended also controls grounding in Gemini Apps and Vertex AI.
const TRAINING = ['GPTBot', 'ClaudeBot', 'Google-Extended', 'Applebot-Extended', 'CCBot', 'Bytespider', 'meta-externalagent'];

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: RETRIEVAL, allow: '/', disallow: ['/api/'] },
      { userAgent: TRAINING, disallow: '/' },
      { userAgent: '*', allow: '/', disallow: ['/api/'] },
    ],
    sitemap: 'https://example.com/sitemap.xml',
  };
}

• ✓This file earns the full 20 crawler-governance points: 10 for retrieval bots named and allowed with none blocked, 10 for training crawlers in groups of their own. A group that lists GPTBot and OAI-SearchBot together earns only the first 10, because one rule cannot treat them differently.
• ✓A named group does not inherit the * group. Repeat any Disallow the retrieval bots should still respect, as /api/ is here.
• ✓Keep one source: app/robots.ts or public/robots.txt, not both.

STEP 3 OF 3

Structure a JSON-LD entity graph in app/layout.tsx

Put the site-wide entities, Organization and WebSite, in the root layout with stable @id values. Every page then has them in its raw HTML, and page-level entities can point at them instead of repeating them.

// app/layout.tsx
const SITE = 'https://example.com';

const graph = {
  '@context': 'https://schema.org',
  '@graph': [
    {
      '@type': 'Organization',
      '@id': `${SITE}/#organization`,
      name: 'Example Co',
      url: SITE,
      logo: `${SITE}/logo.png`,
      sameAs: ['https://www.linkedin.com/company/example-co'],
    },
    {
      '@type': 'WebSite',
      '@id': `${SITE}/#website`,
      url: SITE,
      name: 'Example Co',
      publisher: { '@id': `${SITE}/#organization` },
    },
  ],
};

export default function RootLayout({ children }: { children: React.ReactNode }) {
  return (
    <html lang="en">
      <body>
        <script
          type="application/ld+json"
          // Escaping "<" stops a value containing </script> from closing the tag.
          dangerouslySetInnerHTML={{ __html: JSON.stringify(graph).replace(/</g, '\\u003c') }}
        />
        {children}
      </body>
    </html>
  );
}

// app/blog/[slug]/page.tsx
export default async function PostPage({ params }: { params: Promise<{ slug: string }> }) {
  const { slug } = await params;
  const post = await getPost(slug);

  const article = {
    '@context': 'https://schema.org',
    '@type': 'TechArticle',
    '@id': `https://example.com/blog/${slug}#article`,
    headline: post.title,
    description: post.summary,
    datePublished: post.publishedAt,
    dateModified: post.updatedAt,
    author: { '@type': 'Person', name: post.authorName },
    publisher: { '@id': 'https://example.com/#organization' },
    isPartOf: { '@id': 'https://example.com/#website' },
  };

  return (
    <article>
      <script
        type="application/ld+json"
        dangerouslySetInnerHTML={{ __html: JSON.stringify(article).replace(/</g, '\\u003c') }}
      />
      <h1>{post.title}</h1>
      {/* ... */}
    </article>
  );
}

• ✓Organization and WebSite on their own earn 5 of the 25 schema points. A rich entity on the page itself, such as TechArticle, BlogPosting, Product or SoftwareApplication, earns 15, and 25 once that entity is complete: an author for articles, offers for products.
• ✓Render JSON-LD from Server Components. A tag manager or a useEffect that injects it after load never reaches a non-rendering crawler.

Frequently asked questions

Does 'use client' hide my Next.js page from AI crawlers?

Not on its own. Next.js renders client components to HTML on the server, so their initial output is in the response. What goes missing is content a component renders only after it mounts: data fetched in useEffect, components loaded with next/dynamic and ssr: false, and markup gated behind a mounted flag.

Should llms.txt live in public/ or in an App Router route?

Either works. A file in public/ is simplest for a hand-written list. A route at app/llms.txt/route.ts with export const dynamic = 'force-static' generates the file from the same data as your pages at build time, so it cannot fall out of date.

Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI uses OAI-SearchBot to build the index ChatGPT Search answers from and ChatGPT-User to fetch pages people ask about. GPTBot only collects training data. Blocking GPTBot is a licensing decision; blocking OAI-SearchBot removes you from those answers.

How can I see what PerplexityBot actually reads on my Next.js site?

Fetch the page without a browser: curl -s https://your-site.com/page and search the output for a sentence you can see on screen. If it is missing, a non-rendering crawler does not get it either. Crawlable’s free scan does the same fetch across your pages and reports how many words each one delivers.

More Next.js gotchas

• ◆Streaming with Suspense still ships the fallback first. Crawlers that read the full response get the streamed content, but a crawler that cuts the connection early may not.
• ◆next/dynamic with ssr:false removes the component from the HTML entirely.
• ◆A page that renders fine in the browser proves nothing — the browser runs the JavaScript the crawler skips.

Further reading

• How to Make React Apps Crawlable for AI Search: SSR, Prerendering and Dynamic Rendering

  Google can render React; ChatGPT, Claude and Perplexity crawlers can't. Code for Vite SSR, Next.js and Astro prerendering, dynamic rendering, and a curl check.

• What GPTBot Actually Sees When Your Site Relies on Client-Side JavaScript

  AI crawlers like GPTBot and OAI-SearchBot fetch raw HTML without executing client-side JS. Here is what they actually extract from SPAs.

• Why AI Search Engines Can’t Read Client-Side Rendered (CSR) Sites

  Most AI crawlers fetch raw HTML and never run your JavaScript. Here's what they actually see on a client-rendered site, how to test it, and how to fix it.

• What is Generative Engine Optimization (GEO)? The Complete 2026 Guide

  How ChatGPT Search, Perplexity and Claude find, read and cite pages, why that differs from Googlebot, and what the research says actually improves AI visibility.

All guides on the blog