Machine Readability and Clean HTML for AI Crawlers

How crawlers read your HTML, why semantic elements matter, the truth about text-to-HTML ratio, and a five-step machine readability test.

Machine readability is how easily a crawler or parser can extract the meaning of a page from its code: the main text, the headings, the links and the structure. A page can look perfect in a browser and still be hard for machines to read if the content is loaded late by JavaScript, placed in images, or buried in layers of layout code. Clean, semantic HTML fixes most of these problems.

What crawlers actually receive

Googlebot renders JavaScript, although rendering may happen after the first crawl. Most AI crawlers do not run JavaScript at all and read only the HTML the server sends. Before judging any content, check what that first response contains. You can see it with “View page source” in a browser (not “Inspect”, which shows the page after scripts run), with a command-line tool such as curl, or with the “View crawled page” option in Google Search Console’s URL Inspection tool.

Semantic HTML

Semantic elements tell a parser what each part of the page is:

Element Meaning
main The main content of the page, as opposed to navigation and footer
article A self-contained piece such as a post or guide
h1 to h3 The heading hierarchy
nav, header, footer Site furniture that can be skipped when extracting content
ul, ol, table Lists and tabular data with their structure intact
time A date in a machine-readable format

Page builders often produce div elements styled to look like headings, lists or tables. People see the same thing, but a parser loses the structure.

Common problems

  • Main text inserted by JavaScript after the page loads
  • Important information only inside images, sliders or videos, with no text version
  • Content in tabs or accordions that is fetched only when clicked
  • Headings chosen for font size instead of hierarchy, such as several h1 elements or skipped levels
  • Very deep nesting of wrapper elements from page builders
  • Content inside iframes, which crawlers treat as a separate document

About text-to-HTML ratio

Some SEO tools report a “text-to-HTML ratio” and treat a low value as a problem. Google has said it is not a ranking factor. Heavy markup still has costs: Lighthouse warns when a page has more than about 1,400 DOM nodes, because large DOMs slow down rendering and interaction. Cleaning up bloated markup is worth doing for speed and clarity; the ratio itself can be ignored.

Machine readability and accessibility

Most machine readability fixes are also accessibility fixes. Screen readers rely on the same headings, landmarks, lists, table headers and alt text that parsers use. Following the Web Content Accessibility Guidelines (WCAG) improves both at once.

A quick test

  1. Open the page source and search for a sentence from the middle of the main content. If it is missing, it is loaded by JavaScript.
  2. Check there is one h1 and that h2 and h3 headings follow the outline of the page.
  3. Check that tables use table, tr, th and td elements.
  4. Turn off images and check that no key information disappears.
  5. Run Lighthouse and look at the DOM size warning.

Frequently asked questions

Is a page builder bad for SEO?

Not automatically. Some produce clean markup, others produce very heavy code. Test the output rather than the tool’s name.

Does Google need JavaScript-free pages?

No, Google renders JavaScript. Other search engines and most AI crawlers do not, so server-rendered content reaches more systems.

Related