Sitemaps That Scale: Index Files, lastmod, Image and Video Sitemaps

A sitemap setup that scales lists only indexable, canonical URLs, splits them into several files by content type, groups those files under a sitemap index, and gives each URL an accurate last-modified date. Each sitemap file can hold up to 50,000 URLs or 50 MB uncompressed. Google ignores the priority and changefreq fields and uses lastmod only when it is consistently accurate. Image and video sitemaps help search engines find media they might otherwise miss.

The limits and fields that matter

Split by type

/sitemap_index.xml
  /sitemaps/pages.xml
  /sitemaps/services.xml
  /sitemaps/posts-1.xml
  /sitemaps/posts-2.xml
  /sitemaps/products-1.xml

Splitting keeps files small, and in Search Console you can see indexing results per sitemap. If products index well but posts do not, you know where to look.

Accurate lastmod

• Use the date the content last changed meaningfully, not the date of any database write.
• Do not set every URL to today’s date on every build. Google stops trusting lastmod that changes without real changes.
• Keep a separate content-updated timestamp in your data model if background jobs touch records.

Generating it in Next.js

For large sites, Next.js can produce several sitemaps with generateSitemaps:

// app/products/sitemap.js
const PER_FILE = 45000;

export async function generateSitemaps() {
  const total = await countPublishedProducts();
  return Array.from({ length: Math.ceil(total / PER_FILE) }, (_, id) => ({ id }));
}

export default async function sitemap({ id }) {
  const products = await getPublishedProducts({ offset: id * PER_FILE, limit: PER_FILE });
  return products.map((p) => ({
    url: `https://www.example.com/products/${p.slug}/`,
    lastModified: p.contentUpdatedAt,
  }));
}

Staying below the 50,000 limit leaves room for growth between builds. List the generated files in a sitemap index or in robots.txt. For Laravel, see the spatie/laravel-sitemap example in the Laravel SEO checklist; WordPress core and SEO plugins generate an index automatically.

Image sitemaps

<url>
  <loc>https://www.example.com/products/steel-flange/</loc>
  <image:image>
    <image:loc>https://www.example.com/images/steel-flange-wnrf.jpg</image:loc>
  </image:image>
</url>

Declare the image namespace on the urlset element. Image sitemaps are most useful for images loaded by JavaScript or hosted on a CDN domain; images in standard img elements are usually found anyway.

Video sitemaps

Video sitemaps list the watch page, thumbnail, title, description and content or player URL for each video. They help Google find videos, but a video is indexed only when it is the main content of its page. See video in AI answers.

Submit and monitor

1. Reference the sitemap index in robots.txt: Sitemap: https://www.example.com/sitemap_index.xml.
2. Submit it in Search Console and Bing Webmaster Tools.
3. Check the Sitemaps report for errors and the Page indexing report, filtered by sitemap, for URLs submitted but not indexed.
4. Use IndexNow for immediate notifications to Bing and others; see IndexNow on Next.js and Laravel.

Common mistakes

• Including redirected, noindexed or parameter URLs
• http URLs in an https site’s sitemap, or URLs on the wrong host
• Staging URLs leaking into production sitemaps
• One huge file that times out or exceeds the limits

Frequently asked questions

Does a sitemap guarantee indexing?

No. It helps discovery. Indexing depends on quality and on how Google assesses each page.

Do small sites need a sitemap?

Google says small, well-linked sites may not need one, but it rarely hurts and helps with new pages.

Do AI crawlers use sitemaps?

Some crawlers read sitemaps to discover URLs, but AI companies have not documented this in detail. Accurate sitemaps help search engines, whose indexes many AI assistants draw on.

Sources

1. Google Search Central, “Build and submit a sitemap” and “Manage your sitemaps with sitemap index files”
2. Google Search Central, image and video sitemap documentation
3. Next.js documentation, generateSitemaps