Benchmarks/Evals

Golden-set evals in CI: catching retrieval regressions

Can a small golden set in CI catch retrieval regressions?

3 of 4injected regressions caught

Short answer: A 30-question golden set gated in pytest flagged 3 of 4 injected regressions. A hit@1-only gate would have missed one that cut hit@5 from 0.933 to 0.800.

Tested 30 September 2026 · Read the full write-up

Metrics after each change↑ higher is better
0.000.250.500.751.00scoreBaselineBaselinechunk size 800 to 200chunk size 800 to 200chunk size 800 to 3000chunk size 800 to 3000heading prefix offheading prefix offlowercase offlowercase offhit@1, Baseline: 0.367hit@1, chunk size 800 to 200: 0.5hit@1, chunk size 800 to 3000: 0.367hit@1, heading prefix off: 0.3hit@1, lowercase off: 0.4hit@5, Baseline: 0.933hit@5, chunk size 800 to 200: 0.8hit@5, chunk size 800 to 3000: 0.9hit@5, heading prefix off: 0.867hit@5, lowercase off: 0.733MRR, Baseline: 0.583MRR, chunk size 800 to 200: 0.628MRR, chunk size 800 to 3000: 0.601MRR, heading prefix off: 0.513MRR, lowercase off: 0.553
  • hit@1
  • hit@5
  • MRR

What the numbers say

  • Gate on several metrics: shrinking chunks to 200 characters raised hit@1 but dropped hit@5 by 13.3 points.
  • The change that passed (chunks of 3,000) stayed inside the tolerance.
  • No model or API calls: a full eval runs in seconds on CPU.

Results (change vs baseline, in points)

Baseline0.3670.9330.583pass
chunk size 800 to 2000.500 (+13.3)0.800 (-13.3)0.628 (+4.5)fail (hit@5)
chunk size 800 to 30000.367 (0.0)0.900 (-3.3)0.601 (+1.8)pass
heading prefix off0.300 (-6.7)0.867 (-6.7)0.513 (-7.0)fail (all 3)
lowercase off0.400 (+3.3)0.733 (-20.0)0.553 (-3.0)fail (hit@5)

Setup

Python 3.12.14, pytest 9.1.1, stdlib-only BM25 retriever (no model, no API calls). Corpus: FastAPI English docs (151 Markdown files). Golden set: 30 hand-written questions, each mapped to one expected doc. Runner CPU only; a full eval takes about 0.2 s. Scripts and raw output are in the Experiment files box.

Reproduce it

Read the full write-upFree lesson: RAG evaluationShare XLinkedIn

More benchmarks

All benchmarks →