RAG evals that fail CI: a golden-set harness in about 100 lines of Python

How do you stop a retrieval change from silently making your RAG worse? Put a small golden set in CI and fail the build when metrics drop below a committed baseline.

Result: I built this in four files. On 30 questions it flagged 3 of 4 accidental regressions I injected. The one it passed was a real quality change that stayed inside the tolerance I set (hit@5 fell 3.3 points). One regression (chunks of 200 characters) raised hit@1 and MRR while cutting hit@5 by 13.3 points, so gating on a single metric would have missed it.

The basics of retrieval evaluation are in the free lesson on RAG evaluation. This post is the CI part.

Setup

• Retriever under test: heading-aware chunks (800 characters) with the heading path prepended, ranked by BM25. It is the same chunker as in the chunking benchmark, and the 30 questions are the paraphrase set from the hybrid search post. I used BM25 only so CI needs no model download and runs in a fraction of a second.
• Golden set: golden.jsonl, one line per question: {"question": ..., "expected_doc": ...}. A hit means the expected doc appears in the top-k documents.
• Metrics: hit@1, hit@5, MRR (cut off at rank 10).
• Gate: pytest fails if any metric is more than 0.05 below baseline.json. With 30 questions, one question is 0.033, so one flipped answer passes and two fail.
• Regressions injected: each is a one-field change to the config: smaller chunks, larger chunks, no heading prefix, and a tokenizer bug (lowercasing removed).

I did not test a citation-rate metric here. The harness only scores retrieval.

Results

Numbers are from results.txt. Differences are in percentage points versus baseline, computed from the unrounded values.

Change	hit@1	hit@5	MRR	Gate
Baseline	0.367	0.933	0.583	pass
chunk size 800 to 200	0.500 (+13.3)	0.800 (-13.3)	0.628 (+4.5)	fail (hit@5)
chunk size 800 to 3000	0.367 (0.0)	0.900 (-3.3)	0.601 (+1.8)	pass
heading prefix off	0.300 (-6.7)	0.867 (-6.7)	0.513 (-7.0)	fail (all 3)
lowercase off	0.400 (+3.3)	0.733 (-20.0)	0.553 (-3.0)	fail (hit@5)

What the numbers mean

The 200-character case is why you gate on several metrics. Small chunks make the best chunk more specific, so the right doc more often lands at rank 1 (+13.3 points). But 6 questions dropped out of the top 5 entirely. If your gate were “MRR must not fall”, this change would pass and ship. My reading of the cause: short chunks split the answer away from the words the question uses, so some docs have no chunk that scores at all. I did not test that explanation directly.

The tokenizer bug moved MRR only 3.0 points yet lost 20.0 points of hit@5, which is 6 more questions failing. An average metric hid it; the per-metric threshold did not.

Removing the heading prefix hit every metric by roughly 7 points, so it failed all three tests at once.

3000-character chunks passed the gate with a 3.3-point hit@5 drop. That is one question. It sits inside the tolerance on purpose, and it also shows the cost: a 30-question set cannot separate a real 3-point drop from noise.

Caveats

• 30 questions is a smoke test. Every question is worth 3.3 points, so differences under about 5 points are not reliable. Grow the set from real user queries and from every bug report.
• The questions and the corpus are the same ones I used to tune the earlier posts, so the baseline is optimistic. Your own numbers will differ.
• The expected answer is a document, not a passage. Coarse, but stable while you re-chunk.
• BM25 only. A dense or hybrid retriever adds nondeterminism (model versions, ANN settings), so pin those and expect to widen the tolerance.
• The injected regressions are mine; real ones are less tidy.

Code

The retriever config is a frozen dataclass, so a regression is a one-field diff. Full rag.py is in the Experiment files box; the harness and gate are below.

# harness.py
import json, os
HERE = os.path.dirname(__file__)

def load_golden(path=os.path.join(HERE, "golden.jsonl")):
    return [json.loads(l) for l in open(path) if l.strip()]

def evaluate(retriever, golden):
    rows, h1, h5, rr = [], 0, 0, 0.0
    for g in golden:
        ranked = retriever.search(g["question"], k=10)
        rank = ranked.index(g["expected_doc"]) + 1 if g["expected_doc"] in ranked else None
        h1 += rank == 1; h5 += rank is not None and rank <= 5; rr += 1 / rank if rank else 0
        rows.append({"question": g["question"], "expected": g["expected_doc"], "rank": rank})
    n = len(golden)
    return {"hit@1": h1 / n, "hit@5": h5 / n, "mrr": rr / n, "n": n}, rows

# test_golden.py
Update baseline.json deliberately (in the same PR, reviewed) when you improve the pipeline."""
import json, os, pytest
from rag import Retriever, Config
from harness import load_golden, evaluate

HERE = os.path.dirname(__file__)
TOLERANCE = 0.05   # absolute; 30 questions = 0.033 per question, so one flipped answer passes, two fail
CONFIG = Config(**json.loads(os.environ.get("RAG_CONFIG_OVERRIDE", "{}")))

@pytest.fixture(scope="module")
def result():
    metrics, rows = evaluate(Retriever(CONFIG), load_golden())
    return metrics, rows

@pytest.mark.parametrize("metric", ["hit@1", "hit@5", "mrr"])
def test_metric_not_regressed(result, metric):
    metrics, rows = result
    base = json.load(open(os.path.join(HERE, "baseline.json")))[metric]
    misses = [r["question"] for r in rows if r["rank"] is None or r["rank"] > 5]
    assert metrics[metric] >= base - TOLERANCE, (
        f"{metric} {metrics[metric]:.3f} < baseline {base:.3f} - {TOLERANCE}; "
        f"{len(misses)} questions missed top-5, e.g. {misses[:3]}")

baseline.json is committed and holds {"hit@1": 0.3667, "hit@5": 0.9333, "mrr": 0.5828}. GitHub Actions workflow:

# .github/workflows/rag-eval.yml
name: rag-eval
on:
  pull_request:
    paths: ["rag/**", "golden.jsonl", "baseline.json", "prompts/**"]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install pytest
      - run: bash fetch_corpus.sh
      - run: pytest -q test_golden.py

Rule of thumb

1. Commit a golden set and a baseline; fail the build on any single metric dropping, not on an average.
2. Set tolerance from your set size: at least two questions’ worth, so one flip is not a red build.
3. Print the missed questions in the failure message so the fix starts from a list.
4. Raise the baseline only in the same reviewed PR that improves the pipeline.
5. Trigger the job on changes to chunking, embeddings, prompts and the golden set itself. For a wider checklist see the production checklist.

Experiment files

rag.py, harness.py, test_golden.py, golden.jsonl, baseline.json, run_demo.py, ci.yml and results.txt are in the Experiment files box below. Run bash experiments/fetch_corpus.sh first, then python3 run_demo.py.