Rendered vs Raw HTML: A Playwright Script That Shows What AI Crawlers Miss

The quickest way to find pages that AI crawlers cannot read is to compare, for each URL, what a non-rendering crawler receives from your server with what a browser shows after JavaScript runs. Playwright, a browser automation library, makes this easy: the script below fetches each page both ways and reports words, links and headings, flagging pages whose content depends on JavaScript. Googlebot renders JavaScript, but most AI crawlers do not, so a large gap means those crawlers see an incomplete page.

Why compare raw and rendered HTML

Our React and Next.js render test showed that a client-rendered page can send crawlers almost nothing. On a real site, some templates render on the server and others do not, and a single component fetched in the browser can hide prices, reviews or specifications. Comparing every key template catches these gaps.

Setup

$ pip install playwright
$ python -m playwright install chromium

The script

#!/usr/bin/env python3
"""Compare what a non-rendering crawler receives with what a browser renders.
Needs: pip install playwright && python -m playwright install chromium
Usage: python3 render_compare.py urls.txt"""
import re, sys, urllib.request
from playwright.sync_api import sync_playwright

def text_stats(html):
    body = re.sub(r"<script.*?</script>|<style.*?</style>", " ", html, flags=re.S | re.I)
    words = len(re.sub(r"<[^>]+>", " ", body).split())
    links = len(re.findall(r"<a\s[^>]*href=", body, re.I))
    h1 = len(re.findall(r"<h1[\s>]", body, re.I))
    return words, links, h1

urls = [u.strip() for u in open(sys.argv[1]) if u.strip()]
with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    print(f"{'url':45} {'raw words':>9} {'rendered':>9} {'raw links':>9} {'rendered':>9}  raw h1")
    for url in urls:
        raw = urllib.request.urlopen(urllib.request.Request(url, headers={"User-Agent": "render-compare/1.0"}), timeout=20).read().decode("utf-8", "replace")
        page.goto(url, wait_until="networkidle", timeout=30000)
        rendered = page.content()
        rw, rl, rh = text_stats(raw)
        dw, dl, _ = text_stats(rendered)
        flag = "  <- content depends on JavaScript" if dw > 2 * max(rw, 1) else ""
        print(f"{url:45} {rw:9} {dw:9} {rl:9} {dl:9}  {rh}{flag}")
    browser.close()

What our test showed

We ran it against a small test page built like a single-page app, where JavaScript inserts the heading, text and navigation after load:

url                   raw words  rendered  raw links  rendered  raw h1
http://localhost/             2       124          0         1  0  <- content depends on JavaScript

A crawler that does not run JavaScript would see 2 words, no links and no heading. A browser sees 124 words, a link and an H1. On a real site, run it on one URL per template and on your most important pages.

Reading the results

Extending it

• Compare specific elements, such as price or review count selectors, raw versus rendered.
• Save both HTML versions for pages that fail, to share with developers.
• Run it in your deployment pipeline alongside the SEO smoke test.

Frequently asked questions

Does Google see the rendered version?

Googlebot renders JavaScript, sometimes after a delay. Most AI crawlers read only the raw HTML, which is why the raw column matters for AI visibility.

Why use networkidle?

It waits until network requests settle, so content fetched after load is included in the rendered version. Very busy pages may need a fixed wait instead.

Is there a no-code alternative?

Our AI crawler access checker compares crawler responses for a single URL, and many desktop crawlers can crawl with and without JavaScript rendering.

Sources

1. Playwright for Python documentation
2. Google Search Central, “Understand the JavaScript SEO basics”