Infinite Scroll Without Losing Crawlability

Crawlers do not scroll. The pattern that keeps infinite scroll for people and real paginated URLs for everything else, the fragment endpoint mistake, and why a load more button is usually the better choice.

Infinite scroll feels like a design decision and it is a crawlability decision. Content that appears when a person scrolls appears because a script fetched it. A crawler does not scroll, does not trigger the script, and sees exactly what was in the first response: the first batch of items and nothing after.

On a product listing or an article archive, that means most of the content is reachable only by a human with a mouse wheel. We touched on this in our piece on pagination generally. Here is the implementation.

The principle: real URLs underneath, enhancement on top

The only pattern that works reliably is to build pagination that functions without JavaScript, then add the scrolling behaviour as a layer for people.

Concretely, every batch of items corresponds to a real URL. The server renders that URL as a complete page containing those items and ordinary links to the previous and next batches. A crawler follows those links and sees everything.

Then, for visitors, a script intercepts the next link, fetches the following batch, and appends it to the page instead of navigating. The visitor experiences continuous scrolling. The crawler experiences conventional pagination. Both are looking at the same content through different doors.

Update the address as the visitor scrolls

When the script appends the next batch, it should also update the browser’s address to the URL of that batch, using the history interface rather than a full navigation.

This matters for three reasons. A visitor who shares the link shares the position they reached, not the top of the list. The back button behaves as expected rather than dumping them at the start. And the URL in the address bar is a real page, so anything that captures it, from analytics to a bookmark, captures something that works.

Sites that append content without updating the address produce the second most common complaint about infinite scroll, which is that you can never get back to where you were.

The fragment endpoint mistake

The most common technical fault, and it is easy to make.

Developers often build a separate endpoint that returns just the items for batch N, as an HTML fragment or as data, and the script calls that. Two problems follow.

First, if that endpoint is crawlable, it is a thin page containing a list of items with no navigation, no heading and no canonical, and it may get indexed as such. Second, the actual paginated URLs often do not exist at all, because the fragment endpoint was built instead of them rather than alongside them.

The fix is to make the paginated URL itself serve both purposes. A request for batch N returns a full page. The same request, flagged by the script as a partial request through a header or parameter, returns only the items. One URL, two responses, nothing extra to index.

If you must keep a separate fragment endpoint, keep it out of the index and never link to it from anywhere crawlable.

A load more button is usually better

Worth saying plainly, because infinite scroll is often chosen by default.

A load more button uses exactly the same underlying structure, real paginated URLs with a script layered on top, and it has several advantages. The footer is reachable, which matters on any site where the footer carries links people need. The visitor controls when more content loads, which is better on slow connections and metered data. It is easier to make accessible, because focus management is simpler. And it is easier to measure, because each click is an explicit event.

Infinite scroll earns its place on feeds where the content is endless and position does not matter. On a product listing or an archive, where somebody is looking for a specific item, a button is better for people and identical for crawlers.

Performance on the phones people actually use

Appending items indefinitely means the page grows indefinitely. On a mid-range Android device, which is where most Indian traffic arrives, a list that has grown to several hundred items with images becomes slow to scroll and sometimes crashes the tab.

Cap what remains in the document. Once the visitor has scrolled past a certain number of batches, remove the earliest ones from the document while keeping their position in the history, so scrolling back reloads them. This is more engineering than most teams expect, which is a further argument for the button.

Lazy load images within each batch as well, so appending a batch does not fetch forty images at once. The wider discipline is in mobile performance on low-end Android phones.

Accessibility, briefly

Infinite scroll is difficult for keyboard and screen reader users, because content appears without announcement and focus has nowhere sensible to go.

Announce newly loaded content through a live region, keep the loaded items in the normal reading order, and make sure a keyboard user can reach whatever a mouse user can. If that sounds like work, it is, and the button handles most of it for free. The overlap with search is in accessibility and SEO.

Analytics

Without a page load, standard analytics records one view for a session that saw twenty batches. Send a virtual view each time the address updates, so engagement is measured against what people actually saw.

Decide in advance whether a batch load counts as a page view in your reporting, because it inflates view counts and deflates time per page, and somebody will ask why the numbers changed.

Testing it

Four checks, none of which require tools.

  1. Disable JavaScript and load the listing. You should see the first batch with working previous and next links.
  2. Follow those links. Each should return a complete page with its own items and its own links.
  3. Fetch a deep page directly by URL. It should work without having scrolled to it.
  4. Compare what the raw response contains against what the rendered page contains after scrolling, which is the method in rendered versus raw HTML. The gap should be zero for anything you want found.

Then check your crawl for the fragment endpoint. If it appears, it is being discovered, and it should not be.

Each paginated page still has to be a page

Everything in the general pagination guidance applies underneath. Each page is self-canonical, carries real links, is crawlable, and is not blocked. Page one does not exist at two addresses. The series does not run so deep that items at the end are effectively unreachable.

The combination of infinite scroll with filters and sorting is where this gets complicated, and that is covered in faceted navigation and pagination.

How big the first batch should be

A detail that decides how the listing looks to a crawler, and it is rarely thought about.

Whatever is in the first response is the page as far as discovery is concerned. If the script loads ten items at a time and the category holds four hundred, the category page is a ten-item page with a link to page two. That is thin, and on a competitive category it is noticeably thinner than a conventional listing that shows forty.

Make the first batch a proper page. Somewhere between twenty four and forty eight items is typical for product listings, and the number should be the same for the paginated URLs so that page two on the scroll path and page two by direct link contain the same items. If the two paths disagree about where batches begin, the same item appears on different pages depending on how you arrived, and the duplicate and canonical reports start filling up for no good reason.

The batch size also governs how deep the series runs. Four hundred items at ten per batch is forty paginated pages, and the items on page thirty eight are a long way from anything with authority. At forty per batch it is ten pages. Fewer, fuller pages are better for crawling and no worse for people, provided images within each batch are lazy loaded.

One more thing to decide: what happens to the scroll position on return. A visitor who taps into an item from batch six and comes back should land in batch six, not at the top. Because the address was updated as they scrolled, the browser knows which page they were on, and restoring that page with its items is the job of the script. Sites that skip this are the ones people describe as frustrating without being able to say why.

What is a waste of effort

Infinite scroll with no underlying URLs, then trying to fix discovery with a sitemap. The sitemap lists the items. Nothing links to them. They are orphaned.

Pre-rendering a huge number of items into the first response to make them crawlable. You have made the page enormous for every visitor to serve a crawler that would have followed links.

Assuming rendering will handle it. Rendering does not scroll either.

Infinite scroll on a page with a footer people need. They will never reach it, and the complaint will be about the footer rather than the scroll.

Building the fragment endpoint first and the paginated URLs never.

A sensible order of work

  1. Build conventional pagination first: real URLs, real links, self-canonical, each page complete.
  2. Confirm it works with JavaScript disabled before adding anything.
  3. Decide between a button and scrolling. Default to the button unless the content is a genuine feed.
  4. Layer the script on top, intercepting the next link and appending the batch.
  5. Update the address on each batch and send a virtual view.
  6. Serve partial responses from the paginated URL itself rather than from a separate crawlable endpoint.
  7. Cap the document size and lazy load images within batches.
  8. Announce loaded content for assistive technology and test with a keyboard.
  9. Run the four tests, and check your crawl for anything that should not be there.

Related patterns for content hidden until interaction are in making tabs, accordions and read more content crawlable, and the framework-specific mechanics in JavaScript SEO for Angular and Vue.

If your listings are only reachable by scrolling and you want it fixed without a redesign, get in touch.