Home/Benchmarks
Measured, not guessed.
Every result here comes from code that was actually run on real data. Each benchmark shows what was tested, on what, and how to reproduce it.
- 7benchmarks
- 67measured configurations
- 6topics
Hybrid search: BM25 + vectors, alpha swept 0 to 1
No single retriever wins everywhere. Pure BM25 beat pure vectors on exact code identifiers (0.772 vs 0.279 MRR). Pure vectors beat BM25 on paraphrased questions (0.768 vs 0.566). An RRF blend at alpha 0.8 kept all three query styles above 0.72.
See the full results- heading MRR
- identifier MRR
- paraphrase MRR
- Chunking
Chunk size sweep: 200 to 2,000 characters
How big should RAG chunks be?
0.877best strict hit@1 (heading-aware, 200 chars) - Evals
Golden-set evals in CI: catching retrieval regressions
Can a small golden set in CI catch retrieval regressions?
3 of 4injected regressions caught - Models · small experiment
Predicting embeddings vs tokens with less data
Does predicting embeddings degrade more gracefully than predicting tokens?
−99.4%token-space score with 10x less data - Chunking
Five chunking strategies on real docs
Which chunking strategy retrieves best?
0.785hit@1, heading-aware + heading path - Retrieval
Hybrid search: BM25 + vectors, alpha swept 0 to 1
Should you fuse BM25 with vector search, and with what weight?
α 0.8balanced blend: 0.782 / 0.753 / 0.722 MRR - Vector search
Exact vector search vs HNSW, 10k to 1M vectors
When do you actually need a vector database?
3.5 msexact search, 100k × 384-dim, 2 vCPUs - Routing · small experiment
A TF-IDF router instead of an LLM call
Can a classic classifier route requests instead of an LLM?
0.628 msmedian latency per decision, $0
How these are made
- Run, then write. Each number comes from a script that was executed before the post was published.
- Setup on the page. Hardware, library versions, dataset and query counts are listed with every result.
- Reproducible. Scripts and raw output are linked, so you can rerun them or disagree with them.
- Dated. Results are tied to the versions tested. When something is re-run, the date changes.
Want something measured? Suggest a benchmark.