BM25 from scratch in Python: 27 lines, score-identical to rank_bm25 and up to 1.7x faster
A from-scratch BM25 matched rank_bm25 on 500 queries (max score difference 1.8e-13) and ran 1.7x faster on short queries. The idf formula changed 14 top-1 results.
Make your RAG say 'I don't know': a score threshold, measured
A cosine threshold of 0.40 refused 24 of 30 unanswerable questions and still answered 29 of 30 answerable ones. BM25 scores separated them worse (AUC 0.91 vs 0.99).
Reciprocal Rank Fusion k=60: what the constant does, measured on real BM25 + vector runs
Swept RRF k from 0 to 1000 on 1,119 FastAPI doc queries: k=1 scored 4.1 MRR points above k=60 (0.733 vs 0.692). Past k=100 nothing changes.
Does chunk overlap help? Measured at 0, 10, 20 and 40 percent
On the FastAPI docs, 40% overlap raised BM25 strict hit@5 by 13 to 20 points for fixed chunks, but cost 1.6x the index. 10% overlap gave 2 to 5 points.
Chunk size sweep: 200 to 2,000 characters on real docs
Fixed-size chunks fell from 0.58 to 0.16 hit@1 as size grew from 200 to 2,000 characters. Heading-aware chunks only fell from 0.88 to 0.76 on the FastAPI docs.
RAG evals that fail CI: a golden-set harness in about 100 lines of Python
A 30-question golden set with pytest caught 3 of 4 retrieval regressions; a hit@1-only gate would have missed one that cut hit@5 from 0.933 to 0.800.
Hybrid search in Python: BM25 + vectors with Reciprocal Rank Fusion, alpha swept 0 to 1
Tested BM25+RRF on 2,713 FastAPI doc chunks: BM25 beat vectors by 49 MRR points on identifiers; vectors beat BM25 by 24 on paraphrases. alpha=0.8 balanced both.
RAG chunking, measured: heading-aware chunks doubled hit@1 on the FastAPI docs
I tested 5 chunking strategies on 151 real docs and 773 queries. Fixed-size chunks crossed section boundaries 59% of the time. Heading-aware chunks never did.