How to Add llms.txt and Markdown Page Versions to a Next.js Site

Code for generating llms.txt and Markdown versions of pages in Next.js, keeping them out of Google's index, and an honest note on what llms.txt can and cannot do.

Ashruba Pankhade Published 4 min read

To add llms.txt and Markdown versions of pages to a Next.js site, create a route handler at app/llms.txt/route.js that builds the file from your content source, and a second route that returns each page’s content as Markdown with a text/markdown content type. Keep the Markdown versions out of Google’s index with an X-Robots-Tag header and point them to the HTML page as canonical. Be clear about the limits: no major AI provider has said it uses llms.txt to choose sources, so treat this as a low-cost addition that will not by itself change your visibility.

What llms.txt is and what it is not

llms.txt was proposed in September 2024 by Jeremy Howard of Answer.AI. It is a Markdown file at the root of a site that summarises the site and links to its most useful pages, ideally with clean Markdown versions of those pages, so language models and AI agents can read them without navigation, scripts and layout. It has been adopted by many developer documentation sites. Google has said it does not use the file, and OpenAI, Anthropic and Perplexity have not said their search crawlers use it to choose sources. Our llms.txt guide covers the evidence.

It is still useful in two situations: developer documentation that coding assistants and agents read directly, and sites where AI agents act on behalf of users and benefit from a clean summary.

Step 1: generate llms.txt from your content

// app/llms.txt/route.js
import { getServices, getGuides } from "@/lib/content";

export const revalidate = 3600; // rebuild at most once an hour

export async function GET() {
  const site = "https://www.example.com";
  const [services, guides] = await Promise.all([getServices(), getGuides()]);

  const lines = [
    "# Example Company",
    "",
    "> Example Company provides [what you do] for [whom] in [where].",
    "",
    "## Services",
    ...services.map((s) => `- [${s.title}](${site}/services/${s.slug}.md): ${s.summary}`),
    "",
    "## Guides",
    ...guides.map((g) => `- [${g.title}](${site}/learn/${g.slug}.md): ${g.summary}`),
  ];

  return new Response(lines.join("\n"), {
    headers: { "Content-Type": "text/plain; charset=utf-8" },
  });
}

Generating the file from the same source as your pages keeps it current. A hand-written llms.txt goes out of date quickly.

Step 2: serve Markdown versions of pages

The proposal suggests that a page’s Markdown version be available at the same URL with .md added. In the App Router, a rewrite maps those URLs to a route handler:

// next.config.js
module.exports = {
  async rewrites() {
    return [
      { source: "/services/:slug.md", destination: "/api/md/services/:slug" },
      { source: "/learn/:slug.md", destination: "/api/md/learn/:slug" },
    ];
  },
};

// app/api/md/[section]/[slug]/route.js
import { getMarkdown } from "@/lib/content";

export async function GET(request, { params }) {
  const { section, slug } = await params;
  const md = await getMarkdown(section, slug);
  if (!md) return new Response("Not found", { status: 404 });

  const htmlUrl = `https://www.example.com/${section}/${slug}/`;
  return new Response(md, {
    headers: {
      "Content-Type": "text/markdown; charset=utf-8",
      "X-Robots-Tag": "noindex",
      "Link": `<${htmlUrl}>; rel="canonical"`,
    },
  });
}

If your content is stored as Markdown or MDX, return it directly. If it lives in a CMS as HTML, convert it on the server with a library such as Turndown, and strip navigation, forms and scripts.

Step 3: keep Markdown versions out of search results

  • The X-Robots-Tag noindex header stops search engines from indexing the Markdown copies, so they do not compete with your HTML pages.
  • The canonical Link header points to the HTML page.
  • Do not add .md URLs to your XML sitemap.
  • Do not block them in robots.txt if you want AI agents to be able to fetch them.

Step 4: test

$ curl -I https://www.example.com/llms.txt
# expect 200 and Content-Type: text/plain
$ curl -I https://www.example.com/services/geo.md
# expect 200, text/markdown, X-Robots-Tag: noindex and a canonical Link header

Priorities, honestly

If your site’s HTML is empty before JavaScript runs, fix that before anything in this article. Crawlers that power AI search read your HTML pages, and a server-rendered page does more for AI visibility than any llms.txt file. See the Next.js checklist for AI search and our render test.

Frequently asked questions

Is there an llms-full.txt?

Some sites also publish llms-full.txt with the full content of key pages in one file. It follows the same idea and has the same limits.

Will Markdown versions create duplicate content?

Not if they carry noindex and a canonical link to the HTML page, as above.

Should a small business site bother?

It is optional. It takes an hour with the code above and does no harm, but expect no measurable change in AI answers from it alone.

Sources

  1. Jeremy Howard, “The /llms.txt file”, llmstxt.org, September 2024
  2. Google Search Central, “AI features and your website”
  3. Next.js documentation, route handlers and rewrites
  4. Google Search Central, “Robots meta tag and X-Robots-Tag specifications”

Related reading