Home/Blog
Blog
Experiments on RAG, agents, MCP and evals. Every number on this site comes from code that was actually run.
25 experiments publishedScript + raw output for every measurementNew posts Mon · Wed · Fri
What one RAG query costs: a token-by-token breakdown (with calculator)
A RAG query with 512-token chunks and top-5 sent 2,339 input tokens: about $1.09 per 1,000 queries on gpt-5-mini. Context is 53% of that bill.
pgvector vs numpy on 100k vectors: exact scan is 5x slower, HNSW is 7x faster
On 100k x 384 vectors, numpy exact search took 7.2 ms, pgvector without an index 37 ms, and pgvector HNSW 1.0 ms at 0.955 recall@10. Measured on 2 vCPUs.
BM25 from scratch in Python: 27 lines, score-identical to rank_bm25 and up to 1.7x faster
A from-scratch BM25 matched rank_bm25 on 500 queries (max score difference 1.8e-13) and ran 1.7x faster on short queries. The idf formula changed 14 top-1 results.
- Newsletter
Get each new experiment by email
One short email per post with the numbers, plus the free 25-point RAG production checklist.
Prompt injection through documents: 12 poisoned chunks, 3 regex detectors, 1 of 8 new attacks caught
A regex detector caught 12 of 12 poisoned RAG chunks I wrote it for, but only 1 of 8 fresh ones. Flagged 4 of 4,643 clean FastAPI doc paragraphs.
Recovering headings from PDFs for RAG chunking: font size alone found 26%
On 5 arXiv PDFs, 'font bigger than body' recovered 26% of headings. Bold plus a numbered pattern hit 77% precision and 53% recall; two PDFs broke every rule.
Embedding dimensions: what 256 vs 1024 vs 3072 costs in memory, latency and recall
Truncating a Matryoshka embedding from 1024 to 256 dims cut memory 4x and lost 1.8 points of recall@10 on 705 FastAPI doc queries. int8 lost nothing.
Multi-tenant RAG without data leaks: 5 designs, one test suite
Unfiltered and prompt-only retrieval leaked on 1,800 of 1,800 test queries. Per-tenant partitions leaked on 0. Canary tests, and the falsy-tenant bug they miss.
Streaming LLM answers from FastAPI with SSE: sources first, then tokens
An SSE endpoint sent sources after 52 ms instead of a 1,664 ms JSON wait. One blocking time.sleep in the generator pushed first byte to 3,280 ms with 5 clients.
Make your RAG say 'I don't know': a score threshold, measured
A cosine threshold of 0.40 refused 24 of 30 unanswerable questions and still answered 29 of 30 answerable ones. BM25 scores separated them worse (AUC 0.91 vs 0.99).
Reciprocal Rank Fusion k=60: what the constant does, measured on real BM25 + vector runs
Swept RRF k from 0 to 1000 on 1,119 FastAPI doc queries: k=1 scored 4.1 MRR points above k=60 (0.733 vs 0.692). Past k=100 nothing changes.
Build an MCP server in Python for your docs: stdio and HTTP, with measured payload sizes
A working MCP docs server (mcp 2.3.0, 2 tools) over stdio and HTTP: a k=5 search returns 1,818 bytes of text, but the full result is 2.19x larger.
Metadata filtering in vector search: post-filter loses results, measured
At 1% selectivity, post-filtering top-100 from HNSW recalled 19% of the true results. In-graph filtering kept 100% but took 17 ms. Measured on 100k vectors.
Databricks control plane vs compute plane (data plane), explained
What runs in the Databricks control plane, the classic compute plane and the serverless compute plane, where your data lives, and how clusters connect.
FastAPI headers: read, validate and set them (and the underscore trap)
Read request headers in FastAPI with Header(), handle repeated headers, group them in a Pydantic model, require API keys and set response headers. All tested.
OpenAI Responses API vs Chat Completions: what changes in your code
Side-by-side Python for the Responses API and Chat Completions: inputs, output_text, state with previous_response_id, tools, structured output and streaming.
Does chunk overlap help? Measured at 0, 10, 20 and 40 percent
On the FastAPI docs, 40% overlap raised BM25 strict hit@5 by 13 to 20 points for fixed chunks, but cost 1.6x the index. 10% overlap gave 2 to 5 points.
Chunk size sweep: 200 to 2,000 characters on real docs
Fixed-size chunks fell from 0.58 to 0.16 hit@1 as size grew from 200 to 2,000 characters. Heading-aware chunks only fell from 0.88 to 0.76 on the FastAPI docs.
RAG evals that fail CI: a golden-set harness in about 100 lines of Python
A 30-question golden set with pytest caught 3 of 4 retrieval regressions; a hit@1-only gate would have missed one that cut hit@5 from 0.933 to 0.800.
JEPA vs LLMs: what LeCun's world models actually change for engineers
Analysis: JEPA predicts embeddings, not tokens. My text-chunk stand-in: token-space prediction collapsed 99.4% with 10x less data, embedding-space only 45.8%.
Do you need a vector database? Brute-force search benchmarked from 10k to 1M vectors
Exact numpy search over 100k embeddings takes 3.5 ms on 2 vCPUs. Measured latency, memory and HNSW recall to show when an index actually pays off.
Hybrid search in Python: BM25 + vectors with Reciprocal Rank Fusion, alpha swept 0 to 1
Tested BM25+RRF on 2,713 FastAPI doc chunks: BM25 beat vectors by 49 MRR points on identifiers; vectors beat BM25 by 24 on paraphrases. alpha=0.8 balanced both.
Jev vs LLMs: when a decision model beats a chat model (and when it doesn't)
Analysis: Jev claims 70-500ms typed decisions (vendor-reported). My runnable stand-in, TF-IDF + logistic regression, routed text at 0.6ms and $0, 60% accuracy.
RAG chunking, measured: heading-aware chunks doubled hit@1 on the FastAPI docs
I tested 5 chunking strategies on 151 real docs and 773 queries. Fixed-size chunks crossed section boundaries 59% of the time. Heading-aware chunks never did.
The RAG production checklist: 25 checks before real users see it
25 checks across ingestion, retrieval, generation, evaluation and operations that catch the RAG failures demos hide. Free printable PDF included.
Stop fake citations in RAG: validate them server-side in 25 lines of Python
LLMs will cite passage [7] when you gave them 5. A tested validator that strips invented citations and returns a grounded flag your UI can trust.