Evaluating RAG: Faithfulness, Recall, and the Golden Set

Why Your RAG Score Is Probably Lying to You

You shipped a RAG system. Users are getting answers. You ran some spot checks — looked reasonable. You have not measured anything formally, but nothing is obviously on fire. This is the most dangerous state a RAG system can be in.

Here is what you are not seeing. Your retrieval layer is missing relevant chunks for 20% of queries — not catastrophically, just consistently. The generation model, being capable, is partly compensating by synthesizing plausible-sounding answers from the chunks it did retrieve. Your answers are coherent. They are also subtly wrong in ways that are very hard to detect by reading them. The failure is invisible until a user who actually knows the answer catches it.

The fix is measurement. But the measurement trap is just as real: if you measure the wrong things, you get false confidence. RAG evaluation requires measuring two distinct layers separately, because they fail in distinct ways.

The diagram above shows six metrics across two layers. The key discipline is the separation. Retrieval metrics — recall@k, MRR, precision@k — tell you whether the right chunks are being found. Generation metrics — faithfulness, citation accuracy, refusal rate — tell you whether the model is telling the truth about what it found. A retrieval regression can hide perfectly inside a stable generation score if the model compensates by reasoning from whatever it did retrieve.

The Two Layers, Measured Separately

Retrieval quality starts with a concept called recall@k. Given a question where you know which chunks should appear in the top-k results, what fraction of those expected chunks actually did? If your golden set says question Q requires chunks A, B, and C to answer correctly, and your retrieval returns A and B but misses C, your recall@k for that question is 0.67. Average across all your golden questions and you have a retrieval score you can track over time.

MRR — mean reciprocal rank — complements recall@k by telling you where in the ranked results the first relevant chunk appears. Recall@k being 1.0 is less useful if the relevant chunk is consistently at position 10 out of 10. A high MRR means relevant chunks appear near the top, where reranking and generation can use them most effectively.

The boundary between RAG-specific evaluation and broader LLM eval discipline is worth naming explicitly. This track owns the RAG metrics: recall@k and MRR for retrieval, faithfulness and citation accuracy for generation, all measured in the context of a corpus-grounded pipeline. The testing-ai-adjacent-systems track covers LLM-judge design as a general discipline, calibration methods, and eval ops at scale. The distinction matters for where you invest your time: if your faithfulness judge is producing inconsistent scores, that is a judge design problem for that track; if your retrieval recall is below 0.8, that is a retrieval pipeline problem for this one.

Generation quality has one load-bearing metric: faithfulness. Does every claim in the answer appear somewhere in the retrieved chunks? You cannot check this by reading the answer — a hallucination by definition sounds plausible. The standard approach is to use a second model as a judge. You ask claude-haiku-4-5: "Here are the retrieved chunks. Here is the generated answer. Does every claim in the answer appear in the chunks?" The judge responds with a structured verdict and a list of unsupported claims.

const schema = {
  type: "object",
  properties: {
    supported: { type: "boolean" },
    unsupportedClaims: {
      type: "array",
      items: { type: "string" },
    },
  },
  required: ["supported", "unsupportedClaims"],
  additionalProperties: false,
}

Notice: no minItems, no maxItems on the array — the validator rejects those constraints. The schema is intentionally minimal. The judge's response rate matters more than exhaustive constraint specification.

Building the Golden Set

A golden set is a collection of question → expected-chunk-ids pairs that you curate, version, and protect. About 30 questions is the right order of magnitude for a typical production corpus. But the distribution of those questions determines what the golden set actually catches.

Ten easy questions — where a single highly-relevant chunk answers the query directly — give you a floor. If easy questions are failing, something is fundamentally broken. Ten hard questions — multi-hop queries that require synthesizing across multiple chunks, or questions where the vocabulary in the query differs significantly from the vocabulary in the corpus — give you signal about the real working of your retrieval layer. Five adversarial questions — queries designed to probe the boundary between retrieval and hallucination, where a partial answer exists in the corpus but the full answer does not — test whether your faithfulness defense holds under pressure. Five out-of-corpus questions — queries where the answer genuinely does not exist in your corpus — test whether your refusal path works.

The adversarial and out-of-corpus categories are where most teams are underinvested. A golden set with only easy and hard questions produces high scores that do not reflect production behavior. Real users ask questions at the edges. Your golden set should too.

The Eval-on-Every-Change Loop

The loop diagram shows the eight-step eval cycle. The important thing is step 2 — the trigger. Eval does not run on a schedule. It runs on change events: a content merge to the corpus, a chunking parameter change, a prompt revision, an embedding model upgrade. Each of these changes can move scores in any direction, and the only way to know which direction is to run the eval.

This is the distinction from a quarterly benchmark. Quarterly benchmarks catch large drifts. Eval-on-change catches the incremental regressions that compound into large drifts before you notice them.

The regression gate at step 7 is the mechanism. You maintain a baseline — a JSON file versioned in the repo alongside the corpus — and every eval run compares against it. If recall@k drops more than 0.05 from baseline, the pipeline is blocked. If faithfulness drops more than 0.03, the pipeline is blocked. These numbers are not arbitrary: they represent the difference between "this change degraded the system measurably" and "normal variance in sampling."

export interface EvalSummary {
  recallAtKMean: number     // average recall@k across all non-refusal questions
  mrrMean: number           // mean reciprocal rank
  faithfulnessMean: number  // average faithfulness score
  refusalAccuracy: number   // fraction of out-of-corpus questions correctly refused
  questionScores: QuestionScore[]
}

The schema separates refusal questions from retrieval questions because they measure different things. A question that should be refused is not a failure case for recall@k — it is a success case for refusal accuracy. Mixing them into a single score would reward a system that never retrieves anything (perfect refusal accuracy, meaningless recall@k).

What the Metrics Actually Tell You

Recall@k below 0.8 means your retrieval layer is missing too much. Check: is k large enough to include all expected chunks? Is your similarity threshold cutting off relevant results? Has an embedding model change shifted the similarity landscape? Start with the threshold and k before blaming the model.

Faithfulness below 0.9 means the generation layer is synthesizing beyond the evidence. Check: does your system prompt explicitly constrain the model to only claim what the sources say? Is the model receiving enough context — are the chunks long enough to contain the full answer? Is the citation schema enforced in a way the model is respecting?

MRR below 0.7 means relevant chunks are buried. Reranking exists to solve this. If you are already reranking and MRR is still low, check whether the reranker is seeing the right candidates — the initial candidate pool needs to be large enough to contain the relevant chunks before reranking can help.

The academy's own AI tutor — the RAG system answering learner questions about these lessons — runs this eval loop on every lesson merge. When a lesson is updated, the corpus changes, the affected chunks are re-embedded, and the golden set runs to verify that the answers learners were getting about that lesson topic have not degraded. This is not aspirational — it is the system you are learning to build.