Next.js SEO Checklist for AI Search: Rendering, Metadata, Schema and Crawlers

A checklist for making Next.js sites readable by Google and AI crawlers: server rendering, the Metadata API, JSON-LD, sitemap and robots files, links, middleware and testing.

Ashruba Pankhade Published 4 min read

A Next.js site is ready for AI search when every public page sends its full text, headings, links, metadata and structured data in the HTML before any JavaScript runs, and when search and AI crawlers are allowed to fetch it. In the App Router that means keeping pages as server components, fetching content on the server, using the Metadata API for titles and descriptions, generating sitemap and robots files from code, and checking that middleware or a CDN does not block AI crawlers. This checklist covers each point with the relevant Next.js feature.

Why Next.js sites need a specific check

Next.js can render pages on the server, at build time or in the browser, and one project often mixes all three. Googlebot renders JavaScript, but most AI crawlers do not. In our render test, a Next.js page that fetched its content in the browser sent a crawler 3 visible words, while the server-rendered version of the same page sent all 218.

1. Rendering

  • Keep page.js files as server components. Put “use client” only on the interactive components inside them, such as forms, filters or tabs.
  • Fetch the page’s main content on the server, in the server component, not in useEffect.
  • Use static generation for pages that rarely change, with generateStaticParams for dynamic routes, and set revalidate for content that updates.
  • Check that content inside tabs, accordions and carousels is present in the HTML on first load, even if it is visually hidden.
  • Avoid counters or charts whose numbers only exist after JavaScript runs. Put the real value in the HTML.

2. Metadata

  • Export metadata or generateMetadata from every page for the title and description. These only work in server components, so a page marked “use client” cannot define its own.
  • Set alternates.canonical on every page.
  • Set alternates.languages for translated pages, so hreflang is output correctly.
  • Add openGraph fields so links shared on LinkedIn and messaging apps show the right title and image.
// app/services/[slug]/page.js
export async function generateMetadata({ params }) {
  const { slug } = await params;
  const service = await getService(slug);
  return {
    title: service.seoTitle,
    description: service.seoDescription,
    alternates: { canonical: `https://www.example.com/services/${slug}/` },
  };
}

3. Structured data

  • Output JSON-LD from the server component in a script tag of type application/ld+json, built from the same data that renders the page.
  • Use one connected graph: Organization, WebSite, and Article or Service nodes linked by @id.
  • Make sure the structured data matches what is visible on the page.

4. Sitemap, robots.txt and llms.txt

  • Create app/sitemap.js to generate the sitemap from your routes and data, with lastModified set from the content’s real update date.
  • Create app/robots.js to generate robots.txt, with rules for AI crawlers that match your policy and a link to the sitemap.
  • If you publish llms.txt, serve it from a route handler at app/llms.txt/route.js so it stays in step with your content.
// app/robots.js: allow AI search, block training
export default function robots() {
  return {
    rules: [
      { userAgent: ["GPTBot", "ClaudeBot", "CCBot"], disallow: "/" },
      { userAgent: "*", allow: "/" },
    ],
    sitemap: "https://www.example.com/sitemap.xml",
  };
}
  • Use the Link component or plain anchor tags with real href values. Buttons that navigate with onClick are invisible to crawlers.
  • Make sure important pages are linked from server-rendered navigation, not only from menus built in the browser.
  • Render pagination as links, not as “load more” buttons that fetch the next items with JavaScript.

6. Middleware, hosting and CDN

  • Check middleware for bot detection or geo redirects that treat crawlers differently from browsers.
  • Check CDN and firewall bot settings. Since July 2025, Cloudflare blocks known AI crawlers by default for new domains.
  • Confirm that error pages return real status codes: 404 for missing pages, not a 200 with “not found” text.

7. Images

  • Give every meaningful image descriptive alt text through the Image component’s alt prop.
  • Do not put important text, prices or data only inside images.

How to test the result

  1. Run curl -A "GPTBot" https://www.yoursite.com/page/ and check the output contains the page’s text.
  2. Run our AI crawler access checker on your key templates: home, service or product, article and a listing page.
  3. Use the URL Inspection tool in Google Search Console to see the HTML Google rendered.
  4. Check your server or hosting logs for visits from OAI-SearchBot, PerplexityBot and Claude-SearchBot, and their status codes.

Frequently asked questions

Does the Pages Router work too?

Yes. Use getStaticProps or getServerSideProps to fetch content on the server, and next/head for metadata. The same rule applies: content must be in the HTML before JavaScript runs.

Is static export fine for AI search?

Yes. A static export produces complete HTML files, which every crawler can read.

We have a React single-page app. Do we need to move to Next.js?

Not necessarily Next.js, but you need server rendering or build-time pre-rendering for public pages. For a marketing site, moving to a framework that renders on the server is usually the simplest path.

Sources

  1. Next.js documentation, “Server and Client Components”, “Metadata”, “sitemap.xml” and “robots.txt” file conventions
  2. Google Search Central, “Understand the JavaScript SEO basics”
  3. Vercel, “The rise of the AI crawler”, December 2024
  4. Cloudflare, announcement on blocking AI crawlers by default, July 2025

Related reading