Dense retrieval finds passages by comparing the meaning of a question with the meaning of stored text, using embeddings. Re-ranking then takes the best candidates and scores them again with a slower, more accurate model. Most AI search systems use both, in that order, and a page has to survive both steps before any of its text reaches the answer.
Sparse and dense retrieval
Traditional search uses sparse retrieval. The best-known method, BM25, scores documents by how often the query’s words appear in them, adjusted for document length and for how rare each word is across the collection. It is fast and very good at exact terms: product codes, names, error messages.
Dense retrieval represents both the question and each passage as an embedding and finds the passages whose embeddings are closest. It matches meaning, so “reduce cloud spend” can find a passage about “lowering AWS bills”. The approach was popularised by Dense Passage Retrieval (DPR), described by Karpukhin and colleagues at Facebook AI in 2020, which beat BM25 by 9 to 19 percentage points on top-20 retrieval accuracy in their question-answering tests.
| Sparse (for example BM25) | Dense (for example DPR) | |
|---|---|---|
| Matches | Exact words | Meaning |
| Strong at | Names, codes, numbers, rare terms | Paraphrases, synonyms, natural questions |
| Weak at | Different wording for the same idea | Exact identifiers, very new terms |
| Speed | Very fast | Fast with approximate nearest neighbour indexes |
Hybrid retrieval
Because each method misses what the other catches, production systems usually run both and merge the result lists. A common merging method is reciprocal rank fusion, described by Cormack and colleagues in 2009, which combines rankings by position rather than by raw scores.
Re-ranking
First-stage retrieval compares a question with pre-computed passage embeddings, so it never reads the question and passage together. A re-ranker does. A cross-encoder takes the question and one passage as a single input and outputs a relevance score. Nogueira and Cho showed in 2019 that a BERT-based cross-encoder greatly improved passage ranking on Microsoft’s MS MARCO benchmark, and the approach has been standard since.
Cross-encoders are too slow to run on a whole index, so they only see the top candidates from the first stage, often a few dozen to a few hundred. Some systems use a middle option, late-interaction models such as ColBERT (Khattab and Zaharia, 2020), which compare question and passage word by word at lower cost.
What each stage rewards
- Sparse retrieval rewards using the exact terms people search for, especially names and technical terms.
- Dense retrieval rewards passages that clearly cover the concept behind the question.
- Re-ranking rewards passages that answer the specific question directly, early and completely.
A passage that is on topic but slow to reach its point can pass the first stage and lose at re-ranking to a shorter, more direct passage from another site.
Practical steps
- Use the exact names of products, standards and tools as people write them.
- Explain concepts in plain language, including the other terms people use for them.
- Answer the question in the first sentence of the relevant section.
- Keep each section to one question, so it scores highly for that question instead of moderately for several.
Frequently asked questions
Does Google use dense retrieval?
Google has described using neural models in Search, including BERT from 2019 and neural matching for understanding concepts. It does not publish the details of its retrieval pipeline.
Do I need to write differently for each stage?
No. A section that uses the right terms, explains the concept and answers the question at its start does well at all three.