Prompt injection through documents: 12 poisoned chunks, 3 regex detectors, 1 of 8 new attacks caught
A regex detector caught 12 of 12 poisoned RAG chunks I wrote it for, but only 1 of 8 fresh ones. Flagged 4 of 4,643 clean FastAPI doc paragraphs.
Make your RAG say 'I don't know': a score threshold, measured
A cosine threshold of 0.40 refused 24 of 30 unanswerable questions and still answered 29 of 30 answerable ones. BM25 scores separated them worse (AUC 0.91 vs 0.99).
Does chunk overlap help? Measured at 0, 10, 20 and 40 percent
On the FastAPI docs, 40% overlap raised BM25 strict hit@5 by 13 to 20 points for fixed chunks, but cost 1.6x the index. 10% overlap gave 2 to 5 points.
Chunk size sweep: 200 to 2,000 characters on real docs
Fixed-size chunks fell from 0.58 to 0.16 hit@1 as size grew from 200 to 2,000 characters. Heading-aware chunks only fell from 0.88 to 0.76 on the FastAPI docs.
RAG evals that fail CI: a golden-set harness in about 100 lines of Python
A 30-question golden set with pytest caught 3 of 4 retrieval regressions; a hit@1-only gate would have missed one that cut hit@5 from 0.933 to 0.800.
RAG chunking, measured: heading-aware chunks doubled hit@1 on the FastAPI docs
I tested 5 chunking strategies on 151 real docs and 773 queries. Fixed-size chunks crossed section boundaries 59% of the time. Heading-aware chunks never did.
The RAG production checklist: 25 checks before real users see it
25 checks across ingestion, retrieval, generation, evaluation and operations that catch the RAG failures demos hide. Free printable PDF included.