Headless WordPress With Next.js: SEO and AI Search Setup

How to set up a headless WordPress and Next.js site for search and AI crawlers: server rendering, SEO fields, revalidation, public sitemaps, redirects and keeping the backend out of search.

Ashruba Pankhade Published 4 min read

A headless WordPress site with a Next.js front end is ready for search and AI crawlers when Next.js renders every public page on the server from WordPress data, takes titles, descriptions and canonicals from WordPress’s SEO fields, generates its own sitemap and robots.txt on the public domain, applies redirects managed in WordPress, and keeps the WordPress backend itself out of search results. The most common failures are a CMS subdomain that gets indexed, sitemaps that list backend URLs, and pages that fetch content in the browser.

The architecture

  1. WordPressEditors write content; SEO plugin stores metadata.
  2. APIWPGraphQL or the REST API serves content and SEO fields.
  3. Next.jsRenders pages on the server; revalidates on publish.
  4. Public domainThe only site search engines should see.

1. Render content on the server

Fetch posts and pages from WordPress in server components, with static generation and revalidation, so every crawler receives full HTML:

// app/blog/[slug]/page.js
export const revalidate = 3600;

async function getPost(slug) {
  const res = await fetch(
    `${process.env.WP_API}/wp-json/wp/v2/posts?slug=${encodeURIComponent(slug)}&_embed=1`,
    { next: { tags: [`post-${slug}`] } }
  );
  const [post] = await res.json();
  return post ?? null;
}

export default async function Page({ params }) {
  const { slug } = await params;
  const post = await getPost(slug);
  if (!post) notFound();
  return (
    <article>
      <h1 dangerouslySetInnerHTML={{ __html: post.title.rendered }} />
      <div dangerouslySetInnerHTML={{ __html: post.content.rendered }} />
    </article>
  );
}

Import notFound from next/navigation, so missing posts return a real 404. Content from WordPress is HTML written by your editors; sanitise it if untrusted users can publish.

2. Use WordPress’s SEO fields for metadata

SEO plugins such as Yoast expose their data through the REST API (Yoast adds a yoast_head_json field) and through WPGraphQL extensions. Map those fields in generateMetadata:

export async function generateMetadata({ params }) {
  const { slug } = await params;
  const post = await getPost(slug);
  const seo = post?.yoast_head_json ?? {};
  return {
    title: seo.title,
    description: seo.description,
    alternates: { canonical: `https://www.example.com/blog/${slug}/` },
  };
}

Set the canonical to the public domain yourself. Values stored in WordPress often point to the backend URL.

3. Revalidate on publish

When an editor saves a post, WordPress should call a protected Next.js route that revalidates the page and notifies IndexNow with the public URL. A small WordPress function on the save_post hook can send that request. The Next.js side is covered in setting up IndexNow on Next.js and Laravel. Do not rely on a WordPress IndexNow plugin here, because it would submit the backend URLs.

4. Sitemap and robots.txt on the public domain

  • Generate the sitemap in Next.js with app/sitemap.js from WordPress data, using public URLs and each post’s modified date as lastModified.
  • Generate robots.txt with app/robots.js, pointing to the public sitemap.
  • Do not submit the WordPress sitemap to Search Console.

The WordPress installation, often on a subdomain such as cms.example.com, should not be indexed. Options, in order of strength: put it behind authentication or restrict access by IP, send an X-Robots-Tag: noindex header for all its front-end URLs, and tick “Discourage search engines” in WordPress. Check Google with a site: search for the backend domain after launch.

6. Redirects

If editors manage redirects in WordPress, for example with the Redirection plugin, expose them to Next.js and apply them in middleware or at build time in next.config.js, so changed slugs keep their links and rankings.

7. Previews

Use Next.js draft mode so editors can preview unpublished posts, protected by a secret, without making drafts public or crawlable.

Test before and after launch

  1. Run our AI crawler access checker on a post, a page and a listing.
  2. Check that titles, descriptions and canonicals in the HTML match WordPress and use the public domain.
  3. Check the sitemap contains only public URLs.
  4. Confirm the backend returns noindex or requires login.
  5. Test that a changed slug redirects correctly.

Frequently asked questions

REST API or WPGraphQL?

Both work. WPGraphQL lets you fetch exactly the fields you need in one request, which suits complex pages. The REST API needs no extra plugin.

Is headless better for SEO than normal WordPress?

Not by itself. A well-built traditional WordPress site is fully crawlable. Headless is worth it for performance, design freedom or multiple front ends, if the SEO basics above are handled.

What about structured data?

Output JSON-LD from Next.js server components, built from WordPress data, or pass through the schema generated by your SEO plugin after replacing backend URLs with public ones.

Sources

  1. Next.js documentation, data fetching, generateMetadata, sitemap and robots file conventions, draft mode
  2. WordPress REST API handbook
  3. Yoast developer documentation, REST API
  4. Google Search Central, “Robots meta tag and X-Robots-Tag specifications”

Related reading