Benchmarks/Retrieval

Hybrid search: BM25 + vectors, alpha swept 0 to 1

Should you fuse BM25 with vector search, and with what weight?

α 0.8balanced blend: 0.782 / 0.753 / 0.722 MRR

Short answer: No single retriever wins everywhere. Pure BM25 beat pure vectors on exact code identifiers (0.772 vs 0.279 MRR). Pure vectors beat BM25 on paraphrased questions (0.768 vs 0.566). An RRF blend at alpha 0.8 kept all three query styles above 0.72.

Tested 27 September 2026 · Read the full write-up

MRR by fusion weight (alpha = BM25 share)↑ higher is better
0.000.250.500.751.00MRR0.00.20.40.50.60.80.91.0alphaheading MRR, 0.0: 0.491heading MRR, 0.2: 0.581heading MRR, 0.4: 0.642heading MRR, 0.5: 0.679heading MRR, 0.6: 0.71heading MRR, 0.8: 0.782heading MRR, 0.9: 0.813heading MRR, 1.0: 0.859identifier MRR, 0.0: 0.279identifier MRR, 0.2: 0.471identifier MRR, 0.4: 0.569identifier MRR, 0.5: 0.696identifier MRR, 0.6: 0.735identifier MRR, 0.8: 0.753identifier MRR, 0.9: 0.768identifier MRR, 1.0: 0.772paraphrase MRR, 0.0: 0.768paraphrase MRR, 0.2: 0.794paraphrase MRR, 0.4: 0.806paraphrase MRR, 0.5: 0.751paraphrase MRR, 0.6: 0.731paraphrase MRR, 0.8: 0.722paraphrase MRR, 0.9: 0.678paraphrase MRR, 1.0: 0.566
  • heading MRR
  • identifier MRR
  • paraphrase MRR

What the numbers say

  • alpha = 1.0 is pure BM25, alpha = 0.0 is pure vectors.
  • Identifier queries need BM25: vectors alone found the right chunk first only 20.7% of the time.
  • Paraphrase queries peaked at alpha 0.4 (0.806 MRR).

MRR by alpha

0.00.4910.2790.768
0.20.5810.4710.794
0.40.6420.5690.806
0.50.6790.6960.751
0.60.7100.7350.731
0.80.7820.7530.722
0.90.8130.7680.678
1.00.8590.7720.566

hit@1 and hit@5 at selected alphas

0.00.2070.3970.6670.933
0.40.5150.6560.6670.967
0.80.6620.8890.5330.933
1.00.6920.8890.3670.867

Setup

FastAPI English docs (151 Markdown files, 2,713 heading-aware chunks), sentence-transformers 6.1.0 with all-MiniLM-L6-v2 (384-dim), Python 3.12.14, faiss-cpu 1.15.1. 773 heading queries, 305 exact-identifier queries, 30 hand-written paraphrase queries. Full script and raw output at the end.

Reproduce it

Read the full write-upShare XLinkedIn

More benchmarks

All benchmarks →