Building a RAG eval harness that catches regressions before your users do (2026 update)

Every team I've worked with that ships a RAG system in production eventually has the same conversation. The retrieval metrics look great — hit-rate is 89%, recall@5 is 94%, the offline eval smiles at you — and then a user asks a question that should be trivial, and the system confidently gives back an answer stitched together from three unrelated documents. What went wrong? Nothing your eval was measuring.

This is a piece about how we build eval harnesses for RAG systems that actually catch the regressions users care about. It's grown a lot in 2026 as agentic retrieval and MCP have moved from experiment to default — so consider this the current-year update to the argument, not a static best-practice list.

Why hit-rate lies

Hit-rate — did we retrieve at least one relevant document in the top-k — is the metric everyone starts with because it's easy to compute and looks like recall. It is a comforting lie. The system can hit every relevant document and still produce a wrong answer, because:

• The relevant document was ranked 5th, and the generator over-weighted the noisier documents at ranks 1–4.
• The relevant document was retrieved but was contradicted by a stale, more prominently-cited document also in the context.
• The relevant document was there but the LLM ignored it in favour of its parametric memory.
• The query was ambiguous and "relevant" was defined too loosely in your golden set.

Hit-rate optimises retrieval as if the generator is a passive component. It is not.

The regression that shipped and taught us this

In mid-2025 we shipped a documentation assistant for an enterprise client. Retrieval quality by every offline metric was better than the previous version. Support tickets went up 30% in the first week. On investigation, the new embedding model was pulling in more "loosely relevant" chunks — architecturally related but factually stale — and the generator was faithfully synthesising them into confident wrong answers. Our hit-rate was up. Our faithfulness was down. Our eval didn't measure faithfulness.

That was the last time I trusted a retrieval-only evaluation.

The four dimensions that matter

We now score every RAG deploy on four independent dimensions, each with its own golden set and its own regression gate:

1. Retrieval quality — did the right documents make it into context? Measured via recall@k and mean reciprocal rank against a curated golden set, per query type.
2. Faithfulness — is the generated answer supported by the retrieved documents, or did the model invent, extrapolate, or contradict? Scored by a stronger LLM judge with explicit rubric, or by span-alignment when we can afford it.
3. Coverage — did the answer address the full question, or only the parts the retrieval surfaced? Especially important for multi-hop queries.
4. Freshness — for time-sensitive queries, is the answer citing current information or 18-month-old cached snapshots? This one blows up in production the fastest.

Each dimension can regress independently. A retrieval improvement can drop faithfulness. A prompt change can improve coverage but hurt freshness. If you only track one metric, you'll ship one of these regressions eventually.

Golden set construction, in practice

The single highest-leverage investment in a RAG eval is the golden set. Everything else is downstream of it. What has worked for us:

• Start from real user queries, not synthetic ones. Synthetic queries have a distinctive shape that lets your system cheat.
• Stratify by intent — factual lookup, reasoning, multi-hop, comparative, temporal. Report per-stratum, not just aggregate. Aggregate hides which slice is regressing.
• Include known-hard cases — the queries that broke you in the past. This is your regression suite. It only grows.
• Have a real human label the ground truth. LLM-generated labels are fine for scaling, but the seed set should be human-authored or your judges will be measuring their own biases.
• Refresh 10% of the set quarterly. Users' questions change. If your golden set is frozen from Q1 2025 and it's now Q3 2026, your eval is measuring irrelevant behaviour.

Automated in CI, not as a Jupyter ritual

The eval is a gate on the deploy pipeline, not a notebook run when someone remembers. Concretely:

• Every PR that touches prompts, retrieval config, or model versions runs a subset (~200 queries) of the golden set against the current and proposed configs. Report a diff.
• Every merge to main runs the full golden set (~3,000 queries) and blocks deploy if any dimension regresses beyond the gate.
• Weekly, we run a larger shadow eval against production traffic samples, to catch drift the golden set misses.
• Every gate failure files a ticket with the specific query, the specific regression, and both answers side by side. The person who caused the regression fixes it or the change is reverted. No exceptions.

The 2026 wrinkle: agentic retrieval and MCP

Through 2025, RAG was a single-step pipeline: query → embed → retrieve → generate. Evaluation was tractable because the shape was fixed. In 2026, the norm is agentic retrieval — the agent decides what to retrieve, from which source, when to re-query, when to stop — and the arrival of MCP has made "which source" mean "any tool the agent has access to." This is much more powerful. It is also much harder to evaluate.

The eval framework has to expand to cover:

• Tool-selection accuracy — did the agent pick the right MCP tool for the query, or did it query the CRM when it should have queried the docs?
• Retrieval trajectory efficiency — how many tool calls did it take to get to the answer? Regressions here show up as latency and cost blowups before they show up as quality issues.
• Cross-tool consistency — when two tools return conflicting information, does the agent resolve it or does it silently pick one?
• Stop-condition correctness — did the agent stop retrieving when it had enough, or did it over-retrieve (expensive) or under-retrieve (wrong answer)?

We do this by logging every tool call in the trajectory and evaluating the trajectory as a whole, not just the final answer. It's more work. It's also the only way to catch the class of bugs where the answer is right but the path was expensive, or the answer is wrong but individual retrieval steps looked fine.

The dashboard we wish existed

There's an obvious product opportunity here — a RAG eval platform that treats faithfulness, coverage, freshness, and trajectory as first-class dimensions and integrates cleanly into CI. RAGAS, LangSmith, Braintrust, and the newer entrants are all moving in this direction but none of them is complete. If you're building in this space, ping me — happy to compare notes.

The short version

If you take one thing from this: an eval harness is not a nice-to-have you add after your RAG system is in production. It is the mechanism that makes the RAG system safe to change. Without it, every improvement is a coin flip. With it, you can move fast because you know when you've broken something.

The teams that ship reliably in this space are not the ones with the fanciest retrieval — they're the ones with the tightest feedback loop between change and measurement. Build the loop first. The system quality follows.