AI engineering

RAG Isn't a Product. Evaluation Is.

Nine out of ten Retrieval-Augmented Generation projects we're asked to review have the same problem, and it isn't retrieval. It's that nobody set up an evaluation pipeline before writing a line of code.

"RAG" has become the default answer whenever someone wants to put an LLM on top of their internal documents. Pipe the docs into a vector store, wire up a retriever, call GPT-4 with the retrieved chunks, ship. Six months later the executive sponsor is asking why customer service tickets aren't going down and the head of engineering is quietly Googling "how to know if my RAG is any good".

The uncomfortable answer: they didn't decide up front how they'd know, and now they're stuck with a black-box system whose quality nobody can quantify.

Why "does it feel right?" isn't good enough

Every RAG system has three moving parts that can each be individually broken:

1. Retrieval — did we pull back the right document chunks?
2. Grounding — did the model actually use them?
3. Generation — is the answer correct, complete and safe?

A vibes-based review can't distinguish between "retrieval is great but the model is hallucinating anyway" and "the model is fine but retrieval is missing 40% of the relevant chunks". Both look like "the AI is a bit rubbish" to the user, but the fixes are entirely different — and if you don't measure each part independently, you're guessing.

The four evaluations every RAG needs

1. Retrieval hit rate

Given a question, does the retriever return at least one chunk that could plausibly answer it? Build a small labelled set (100–500 questions with known-good source documents) and measure recall@k for k = 1, 3, 5, 10. If recall@5 is below ~85%, no downstream cleverness will save you.

2. Grounding rate

Given the answer the model produced, is every substantive claim in it supported by the retrieved chunks? This is the one to catch hallucinations. We use an LLM-as-judge here with a strict rubric — the judge sees the answer and the chunks, and marks each claim as supported / partially supported / unsupported.

3. End-to-end correctness

Forget the internals for a moment — is the final answer correct? For factual questions this is straightforward (match against a gold answer). For open-ended questions ("summarise this policy") use a paired-comparison with a reference answer and an LLM judge.

4. Safety & refusal

Does the system refuse gracefully when a question is out of scope, adversarial, or the retriever returned nothing relevant? An honest "I don't know" is better than a confident hallucination — but you'd be surprised how many RAGs never learn to say so.

Building the eval set (the actual hard bit)

The eval framework is code — a few hundred lines. The bottleneck is always the labelled dataset.

The good news: you don't need thousands of examples. 200 well-chosen questions, with real user language and the correct source documents identified, will produce remarkably stable metrics. The bad news: you have to actually build it, and it can't be outsourced to someone who doesn't understand your domain.

Our sequence, roughly:

1. Mine 500 real user questions from support tickets, search logs, sales enablement etc.
2. Deduplicate to ~200, weighted by frequency and business criticality.
3. For each question, a domain expert identifies which document(s) contain the answer.
4. For 20–40 of them, write a reference "ideal answer" to compare against.
5. Version-control the whole thing in git alongside the code.

This costs a few days of a domain expert's time. It will save you months.

The LLM-as-judge trap (and how to avoid it)

LLM-as-judge evaluation is powerful but easy to get wrong. Two failure modes we see constantly:

Judge alignment drift. Your judge model doesn't measure what you actually care about — it measures a plausible correlate. Fix: calibrate against 30–50 human-labelled examples before trusting the judge on unseen data. If judge agreement with humans is below ~85%, tune the rubric.

Same-family cross-contamination. Never use GPT-4 to judge GPT-4 output; the judge is biased toward the generator's style. Use a different model family — Claude, Gemini, a strong open-source model — or, for numeric grounding checks, deterministic string matching.

What "good" looks like

For most B2B RAG systems, we aim for:

• Retrieval recall@5: ≥ 90%
• Grounding rate: ≥ 95% of substantive claims supported
• End-to-end correctness: ≥ 85% on the eval set
• Refusal on out-of-scope questions: ≥ 90%

These aren't universal — a legal contract assistant needs stricter grounding than an internal marketing knowledge tool. The numbers matter less than having numbers at all, tracked over time, so you can see whether your Tuesday afternoon prompt change made things better or worse.

The bottom line

Every RAG project we've rescued started with the same six-week move: freeze feature work, build a labelled eval set, wire up the four measurements above, and then — and only then — go and fix the things the eval is telling you are broken.

Nine times out of ten, the fixes turn out to be much smaller than the team feared: a chunking-strategy tweak, a re-ranking model, a system prompt clean-up. The evaluation was the missing organ; once it's in, the system can heal.