Evaluating RAG Quality: Metrics and a Test Harness

Without an evaluation set every change to a retrieval system is a guess that feels like progress. What to measure, how to build the test set from real questions, and the refusal cases nobody includes.

Most teams building retrieval systems evaluate them by trying a few questions and seeing whether the answers look good. That works until the first change, at which point nobody can say whether things improved, because the comparison is a memory of how it felt last week.

An evaluation set is a day of work and it converts the entire project from opinion to measurement. It is the single highest-return thing to build, and it is almost always skipped because it is not the interesting part.

Measure retrieval and generation separately

This is the structural point that makes everything else useful.

A bad answer has two possible causes. The right information was never retrieved, or it was retrieved and the model did not use it properly. Those need completely different fixes, and an end-to-end score cannot distinguish them.

So measure them as two stages, always.

Retrieval: did the passage containing the answer appear in what was retrieved, and how highly was it ranked.

Generation: given what was retrieved, was the answer correct, supported by that context, and complete.

Teams that only measure the end result spend weeks adjusting prompts when the real problem is chunking, or swapping databases when the real problem is a vague system prompt.

The retrieval metrics that matter

Three numbers, and the first one is by far the most important.

Recall at k. In what proportion of questions does the correct passage appear anywhere in the retrieved set. If this is low, nothing downstream can save you, because the model is being asked to answer from material that does not contain the answer. This is your ceiling.

Rank of the correct passage. Where it lands when it is found. If recall at fifty is high but recall at five is low, the information is being retrieved and buried, which is a ranking problem with a known fix.

Precision. How much of what you retrieved was irrelevant. Matters because irrelevant context costs tokens and dilutes attention.

Report recall at several values of k rather than one. The shape of that curve tells you more than any single figure: a system that reaches high recall only at k of fifty has a very different problem from one that never reaches it at all.

The generation metrics that matter

Four, and they catch different failures.

  • Groundedness. Is every claim in the answer supported by the retrieved context. This catches the system inventing plausible detail, which is the failure that destroys trust fastest.
  • Correctness. Does the answer match the expected answer. Related to groundedness and not the same: an answer can be faithfully derived from the wrong passage.
  • Completeness. Did it include everything it should. Partial answers are the most under-measured failure, because they read as confident and correct.
  • Appropriate refusal. When the context does not contain the answer, did it say so. More on this below, because it is the category nobody tests.

Building the test set

Fifty to a hundred questions is enough. The composition matters more than the count.

  1. Take real questions. From your support inbox, your sales call notes, your site search logs, your chat transcripts. Not questions you composed, because invented questions use your vocabulary and real ones use the customer’s, which is precisely the gap retrieval has to bridge.
  2. Include the awkward ones. Questions whose answer spans two documents. Questions using the wrong terminology. Questions with a typo. Questions in the language mix your users actually type.
  3. Label the expected passages. For each question, which chunk or document should be retrieved. This is the tedious part and it is what makes retrieval measurable.
  4. Write the expected answer. Short, factual, the version a knowledgeable colleague would give.
  5. Add the refusal set. Fifteen to twenty questions your system should decline, covered next.
  6. Keep it versioned alongside your code, and add to it whenever a real failure is reported.

That last habit is what makes the set improve rather than rot. Every production failure becomes a permanent test case, so the same thing cannot regress twice.

The refusal set is the part everybody skips

Most evaluation only tests questions the system should answer. In production, a large share of what arrives is not.

Include questions outside your scope entirely. Questions about a competitor. Questions whose answer genuinely is not in your corpus. Questions seeking advice you should not give, which for a business in a regulated field is the important category. Questions attempting to extract the system prompt or to redirect the assistant’s behaviour.

For each, the expected behaviour is a clean refusal or a handover, not an invented answer.

Measuring this matters because the two failure directions trade against each other. Tighten the system to refuse more and it starts declining questions it could have answered. Loosen it and it invents. Without both sets measured, you are tuning one and breaking the other without knowing.

The design side of this sits in our piece on guardrails for business chatbots, and the broader safety question in building AI agents that use your data safely.

Scoring it without a team of reviewers

Retrieval scores itself. You labelled the expected passages, so checking whether they were retrieved is a comparison a script does exactly.

Generation is harder. Three approaches, and the sensible answer is a combination.

Exact or keyword matching against the expected answer. Cheap, crude, and genuinely useful for factual questions with a specific answer such as a number, a date or a policy term.

A model as judge. Give another model the question, the retrieved context, the expected answer and the produced answer, and ask it to score against a written rubric. This scales and it has known biases: judges tend to reward longer and more confident answers, and they are unreliable without a specific rubric. Use a clear scale with described levels rather than asking for a score out of ten.

Human review of a sample. Twenty answers, read properly, once a month. This is what calibrates the automated scores. If your judge and your humans disagree systematically, fix the rubric before trusting the judge again.

Make it a harness, not an exercise

The value only arrives when running it is trivial.

Write a script that takes the question set, runs the full pipeline, records retrieval and generation results per question, and writes a report. Store every run with the configuration that produced it: embedding model, chunk size and strategy, retrieval count, reranker, prompt version.

Then run it on every change and compare against the previous run. Not just the aggregate, which hides everything. You want the per-question diff, because the question that regressed is the interesting one, and an improvement of two points overall can conceal a serious failure on a category of questions that matters.

Keep the run history. Six weeks later somebody will ask why the chunk size is what it is, and the answer should be a measurement rather than a memory.

What the numbers should change

Use them to decide, not to report.

Low recall at any k means the answer is not being retrieved. Look at chunking, at your embedding model, and at whether the content contains the answer at all.

Good recall at fifty and poor recall at five means a ranking problem. Add reranking.

Good retrieval and poor groundedness means a generation problem. Look at the prompt, at how much context you are sending, and at whether the model is being instructed to answer only from the provided material.

Good scores everywhere and unhappy users means your test set does not reflect reality. Go back to real questions.

What is a waste of effort

A single aggregate score. It tells you something changed and never what.

Questions you wrote yourself. They are easier than real ones in ways you cannot see, because you know the corpus.

Public benchmarks. They measure performance on somebody else’s documents.

Evaluating only when something feels wrong. By then you have made twelve changes and cannot attribute the regression.

A hundred questions you never update. A static set stops reflecting what users ask within a quarter.

Perfecting the rubric before running anything. A rough harness running weekly beats an excellent one that is still being designed.

A sensible sequence

  1. Collect fifty real questions from your support inbox or call notes.
  2. Label the expected passage and write the expected answer for each.
  3. Add fifteen refusal cases, including out-of-scope and prompt-manipulation attempts.
  4. Write the script that runs the set and reports retrieval and generation separately.
  5. Record a baseline with your current configuration.
  6. Change one thing, rerun, and compare per question rather than in aggregate.
  7. Review twenty answers by hand monthly to keep the automated scores honest.
  8. Add every production failure to the set permanently.

The components this measures are covered separately: chunking, embedding model choice, reranking and the storage layer each have their own piece on this site. The implementation context is in building a RAG chatbot on your own content.

If you have a system that works sometimes and nobody can say why, that is an evaluation problem before it is anything else. Get in touch, or see what our AI development work covers.