Retrieval-augmented generation (RAG) is a method in which a language model is given relevant documents at the moment it answers a question, and writes its answer from those documents instead of relying only on what it learned in training. ChatGPT search, Perplexity, Google’s AI Overviews and most other assistants that show source links use some form of RAG. For a website, RAG is the process that decides whether your page becomes one of the sources.
Where the term comes from
The name comes from a 2020 paper, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, by Patrick Lewis and colleagues at Facebook AI Research, University College London and New York University, presented at NeurIPS 2020. They combined a retriever that searched a Wikipedia index with a text generator, and showed that the combined system answered factual questions more accurately than a generator working from memory alone. It could also be updated by changing the index, without retraining the model.
Those two properties, fresher facts and answers traceable to a source, are why the approach spread to web search assistants.
The stages of a RAG system
1. Indexing
Before any question is asked, documents are collected by a crawler, cleaned of navigation and boilerplate, split into chunks and stored. In most modern systems each chunk is also converted into an embedding, a vector that represents its meaning. Web assistants either maintain their own index (Perplexity, OpenAI’s OAI-SearchBot crawl) or query a search engine’s index (Google’s systems use Google’s own index; Microsoft Copilot uses Bing’s).
2. Retrieval
The user’s question is rewritten into one or more search queries. The system finds candidate chunks using keyword matching, vector similarity, or both. Using both is called hybrid retrieval, and it is common because keywords catch exact names and numbers while vectors catch meaning.
3. Re-ranking
The first retrieval returns a broad shortlist. A second, slower model reads the question alongside each candidate and scores how well it answers. Only the top few chunks go forward.
4. Generation
The selected chunks are placed in the model’s input together with the question and instructions. The model writes the answer and, in systems that show citations, links sentences to the chunks they came from.
Where a page drops out
| Stage | Common reason a page is lost | What to check |
|---|---|---|
| Indexing | Crawler blocked in robots.txt or at the firewall; content only rendered by JavaScript | robots.txt rules per AI crawler; raw HTML of key templates |
| Chunking | Answer split across several sections, or a section that makes no sense without the previous one | Whether each section stands alone |
| Retrieval | Page does not use the words or cover the concepts behind the sub-queries | Which questions the page actually answers |
| Re-ranking | Passage is on topic but reaches the answer slowly | Whether the answer is in the first sentence or two |
| Generation | Passage says what many others say, so the claim is merged and credited elsewhere | Whether the page adds specific facts others lack |
What RAG does not fix
RAG reduces invented answers but does not stop them. The model can still misread a source, combine two sources incorrectly, or attach a citation to a sentence the source does not support. A 2023 Stanford study of generative search engines found that only about half of the generated sentences were fully supported by their citations. For a business, this means an assistant can cite your page and still describe you incorrectly. For that reason, checking what assistants actually say about you belongs in any GEO plan.
How to make a page easier for RAG systems to use
- Let the relevant crawlers in. Check robots.txt for each AI search crawler, and check your CDN or firewall, which may block AI bots by default.
- Serve the main content in the initial HTML, without relying on JavaScript to insert it.
- Give each section one topic, a heading that names it, and an answer in its first sentence.
- Name the subject in full at the start of a section instead of using “it” or “this”.
- Add facts only you can provide: your figures, prices, methods and results.
- Keep company facts identical across your site and your profiles, so the model has nothing conflicting to choose between.
Frequently asked questions
Is RAG the same as AI search?
RAG is the method most AI search products use. AI search is the product. Some assistants also answer from training data alone when they decide a search is not needed, and in that case no retrieval happens.
Does RAG use Google’s index?
It depends on the product. Google’s AI features use Google’s index. Microsoft Copilot uses Bing. Perplexity and OpenAI run their own crawlers and also use third-party search providers. This is why being indexed by more than one search engine matters for AI visibility.
Can I see which of my pages an AI system retrieved?
Only indirectly. Server logs show visits from AI crawlers and user-triggered fetchers such as ChatGPT-User, and citations in answers show which pages were used. No assistant publishes a full list of what it retrieved and discarded.