Technical Architecture for AI Search: Schema, llms.txt, Crawlers

Technical architecture decides whether search engines and AI systems can reach a page, read its text and understand who published it. Content quality only matters after these steps work. This guide covers the page structure, markup and crawler settings we check on every project, and notes where common advice is overstated.

Answer-first structure

Journalists call it the inverted pyramid: the main fact first, supporting detail after. For search and AI retrieval it means each section starts with a sentence or two that answers the section’s question on its own, followed by explanation, examples and exceptions.

This matters because retrieval systems select passages, and the lost-in-the-middle research shows models make most use of information placed at the start of their input. A section that opens with background and reaches the answer in its fourth paragraph gives the system nothing useful to lift from its opening lines.

Content chunking

Retrieval systems split pages into chunks before creating embeddings. Some split by a fixed number of tokens, others by headings or paragraphs. You cannot control the method, but you can make every likely split point produce a sensible chunk:

One topic per section, under a heading that names it
Sections that do not depend on earlier text (“as mentioned above”, “this approach”) to make sense
The subject named in full at the start of a section instead of “it” or “they”

Advice to write in chunks of exactly 256 or 512 tokens is a working guideline taken from how some retrieval systems are configured. It is not a published rule of any search engine, and the right length depends on the question being answered. The content chunking guide covers the methods and a before-and-after example.

Tables and data-dense formatting

Prices, specifications, comparisons and timelines are easier for both people and machines to read as an HTML table with a header row than as prose. Google often uses tables as table-style featured snippets. Keep tables as real HTML: a table saved as an image cannot be read by crawlers or quoted by an assistant. The tables guide has a full checklist.

Machine readability

A crawler that does not run JavaScript sees only the HTML your server sends. We check that the main text, headings, internal links and structured data are all present in that first response, and that the page uses semantic elements (main, article, h1 to h3, ul, table) rather than generic div tags styled to look like headings. The machine readability guide includes a quick test.

You will see “text-to-HTML ratio” or “DOM-to-text ratio” quoted as a ranking factor. Google has said it is not one. A very large DOM does slow pages down, and Lighthouse warns when a page has more than about 1,400 DOM nodes, so page-builder layouts with deeply nested wrappers are still worth cleaning up, for speed rather than for the ratio itself.

JSON-LD structured data

Google recommends JSON-LD for structured data. A frequent problem is conflicting schema: a theme and an SEO plugin each output their own Organization and WebSite nodes, sometimes with different names or logos.

We build one graph per page in which each node has a stable @id and refers to others by it. The Article node points to its author’s Person node, the Person node points to the Organization through worksFor, and the Organization lists its official profiles in sameAs. Search engines and AI systems can then connect the page, the author and the company without guessing.

Entity resolution and disambiguation

Entity resolution is the process by which a search engine decides that mentions on different websites refer to the same real-world company or person. Brands with common names, or names shared with other companies, are the hardest to resolve.

The main tools are consistent naming everywhere, sameAs links in your schema to the profiles you control, and a presence in databases that search engines and language models use as references, such as Wikidata. Wikidata has its own notability rules, so an entry is only possible when there is independent evidence about the company. We check eligibility before recommending it.

Knowledge graphs

A knowledge graph stores facts as nodes (a company, a person, a product) connected by labelled edges (founded by, located in, makes). Google’s Knowledge Graph is the best known. Your own JSON-LD graph is a small knowledge graph about your site, and the more closely it matches what other trusted sources say, the more confidently search engines link the two. See the knowledge graphs guide.

llms.txt and llms-full.txt

llms.txt is a Markdown file at the root of a site that lists the pages most useful to language models, proposed by Jeremy Howard of Answer.AI in September 2024. llms-full.txt is a variant that contains the full text of those pages in one file. Both are easy to generate, and this site publishes an llms.txt.

As of 2026, no major AI provider has confirmed that it uses llms.txt when choosing which sources to cite, and Google has said it does not use it. It is worth adding because it costs little, but there is no evidence yet that it changes rankings or citations.

AI crawler rules in robots.txt

Each AI company publishes the user agents of its crawlers, and most separate training crawlers from search crawlers. A robots.txt that allows AI search but blocks model training looks like this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Google-Extended is left out of this example on purpose: it controls Gemini’s use of your content for grounding as well as training, so blocking it can also reduce visibility in Gemini answers.

robots.txt works as a request to crawlers and does not enforce access. Well-known crawlers follow it; others may not. To block a crawler for certain, block it at the server or firewall by verified IP range. robots.txt also has no way to notify crawlers of changes. For that, see IndexNow below.

Freshness signals

Search systems and AI assistants prefer current information for topics that change. Show a visible “updated” date on articles, keep datePublished and dateModified in your Article schema, and keep the lastmod values in your XML sitemap accurate.

Google’s documentation on byline dates asks publishers to change the date only when the content has meaningfully changed. Bumping dates without real edits can lead search engines to distrust your dates altogether. More detail is in the freshness signals guide.

NAP consistency

NAP stands for name, address and phone number. When these differ between your website, Google Business Profile, directories and social profiles, search engines are less sure the listings belong to one business. This affects local rankings and gives AI assistants conflicting facts to choose from. Pick one exact format and use it everywhere. The NAP consistency guide lists where to check.

Faster indexing

IndexNow is a protocol launched in 2021 by Microsoft Bing and Yandex that lets a site notify search engines the moment a URL is added, changed or deleted. It is supported by Bing, Yandex, Seznam and Naver. Google does not use it. Bing’s index also feeds Microsoft Copilot, so IndexNow can shorten the time before changes appear there. Setup steps are in the IndexNow guide.

Google’s Indexing API only accepts job postings and livestream video pages. For everything else on Google, an accurate sitemap and good internal linking remain the way to get pages crawled quickly.

Technical Architecture for AI Search