Evaluating RAG Systems with a Small Golden Dataset

Retrieval-augmented generation (RAG) is easy to demo and surprisingly hard to evaluate. A fluent answer can hide weak retrieval, while a good retriever can be blamed for an answer model that ignores its evidence.

A useful evaluation process separates those failure modes. You do not need thousands of examples to begin. A carefully maintained golden dataset of 30 to 100 representative questions can catch many regressions before users do.

Define what the system is supposed to do

Start with the product contract rather than a model metric. For a documentation assistant, useful requirements might be:

• retrieve the document that contains the answer;
• answer only from retrieved evidence;
• say when the evidence is insufficient;
• preserve important qualifiers such as versions, limits, and exceptions.

These requirements lead to two different evaluation layers: retrieval quality and answer quality.

Build the golden dataset from real tasks

Each case should contain a question and the evidence that a correct system should find. A compact record can look like this:

{
  "id": "auth-rotation-01",
  "question": "How often should service credentials be rotated?",
  "relevant_documents": ["security/credential-rotation.md"],
  "must_include": ["90 days"],
  "must_not_claim": ["rotation is automatic"]
}

The exact schema is less important than consistency. Include easy lookup questions, questions that require combining two passages, ambiguous wording, and deliberately unanswerable questions.

Avoid synthetic-only test sets

Generated questions are useful for expanding coverage, but they often mirror the structure of the source text too closely. Real user questions contain shorthand, incorrect terminology, and missing context. Seed the dataset from support tickets, search logs, documentation feedback, and common developer tasks when those sources can be used without exposing private data.

Measure retrieval before judging the answer

If a relevant document is known, retrieval can be evaluated deterministically. One simple metric is recall at k: did at least one relevant document appear in the first k results?

For 40 cases, if 35 have a relevant document in the first five results, recall@5 is 87.5%.

That number is not enough by itself. Also inspect:

• rank: relevant evidence at position 1 is more useful than at position 20;
• coverage: multi-document questions may require all relevant sources;
• noise: excessive irrelevant chunks consume context and can distract the model;
• filter correctness: tenant, product, language, and version filters must not leak unrelated material.

When retrieval changes, record the old and new result lists for failed cases. That makes regressions debuggable instead of merely measurable.

Score answers against evidence

Exact string matching is usually too strict for generated text. Prefer a small rubric with observable criteria.

A reviewer can score each answer on:

1. Correctness — are the technical claims correct?
2. Grounding — are important claims supported by retrieved evidence?
3. Completeness — are required constraints and exceptions included?
4. Abstention — does the system decline when the evidence cannot answer the question?

For high-risk domains, human review should remain part of the process. Model-based graders can accelerate evaluation, but they are another model and can introduce their own bias. Calibrate any automated grader against a set of human-scored examples.

Keep retrieval and generation experiments separate

Changing the embedding model, chunk size, prompt, reranker, and answer model at once makes the result impossible to attribute.

A better sequence is:

Experiment 1: retrieval only

Freeze the generation step and compare retrieval metrics. For example, test chunk sizes of 300 and 600 tokens using the same embedding model and query set.

Experiment 2: generation only

Freeze the retrieved context and compare prompts or answer models. This reveals whether a new answer configuration is actually better at using the same evidence.

Experiment 3: end to end

Finally run the complete pipeline to detect interactions between retrieval and generation.

This staged approach turns evaluation into engineering rather than prompt guessing.

Include negative and adversarial cases

A RAG system should be evaluated on what it must not do. Add cases where:

• the corpus does not contain the answer;
• an outdated document conflicts with a current one;
• a retrieved page includes instructions that should be treated as data rather than system instructions;
• two products use the same term with different meanings;
• the question presupposes a false fact.

For an unanswerable case, a confident invented answer is a failure even if it sounds plausible.

Track results as regressions, not a leaderboard

A single aggregate score can improve while important cases get worse. Store per-case results and compare them between revisions.

A useful report includes:

retrieval recall@5:  0.875 -> 0.925
answer pass rate:    0.800 -> 0.825
abstention pass:     0.950 -> 0.900
regressed cases:     3
improved cases:      5

The drop in abstention quality deserves investigation even though the overall answer pass rate increased.

Common pitfalls

Testing only happy-path facts

Simple factual questions overestimate real-world quality. Include multi-step, ambiguous, and missing-answer cases.

Letting the dataset become stale

When documentation changes, golden evidence must change too. Treat evaluation data as versioned code and review updates alongside source changes.

Using production secrets in test fixtures

Evaluation data should contain public or sanitized content. Replace credentials, internal hostnames, account identifiers, and private URLs with obvious placeholders.

Optimizing to the test set

A tiny golden set is a regression suite, not proof of general quality. Keep a separate holdout set or periodically add unseen examples so tuning does not overfit familiar cases.

A practical release gate

For a production RAG change, define thresholds before running the experiment. For example: recall@5 must not decline, no critical golden case may regress, and abstention failures must remain below a fixed limit.

Then review the changed cases manually. Metrics tell you where to look; examples tell you why the system changed.

The most valuable RAG evaluation system is not the most elaborate one. It is the one developers can run repeatedly, understand when it fails, and update as the product evolves.