Chunk size sweep: 200 to 2,000 characters on real docs

What chunk size should a RAG pipeline use? I swept seven sizes from 200 to 2,000 characters over the FastAPI docs, with two chunkers. The setup follows the earlier chunking post.

Short answer: smaller is better up to a point, but how you cut matters more than how big. Fixed-size chunks lost most of their accuracy as they grew. Heading-aware chunks barely moved. Once a size-neutral metric is used, everything above about 400 characters is within noise of each other.

If you are new to the topic, the free lesson on ingestion and chunking covers the basics.

Setup

• Corpus: 152 English FastAPI doc files, 1.46M characters. (The earlier post had 151 files and 773 queries; the upstream docs have since changed, so this run has 781 queries. Don’t compare the two runs cell by cell.)
• Queries: every h2/h3 heading with 2+ words and at least 300 characters of body.
• Retriever: BM25, top 5 chunks.
• Chunkers: (1) fixed windows with 10% overlap; (2) heading-aware: split at headings, pack paragraphs up to the size limit, prepend the heading path.
• Strict hit: a retrieved chunk lies at least 50% inside the target section.
• Lenient hit: a retrieved chunk shares at least 100 characters with the target section.

I added the lenient metric because the strict one has a built-in bias. A 2,000-character chunk that contains the whole answer plus a neighbouring section can fail the strict rule. A 200-character chunk almost never does.

Results

Chunker	Size	Chunks	Avg chars	Crosses a section	Strict hit@1	Strict hit@5	MRR	Lenient hit@1	Lenient hit@5
Fixed	200	8,179	198	22.5%	0.583	0.810	0.674	0.583	0.810
Fixed	400	4,119	393	38.9%	0.503	0.758	0.602	0.700	0.908
Fixed	600	2,766	585	50.9%	0.447	0.700	0.548	0.730	0.931
Fixed	800	2,086	774	59.8%	0.391	0.602	0.475	0.736	0.949
Fixed	1,200	1,416	1,139	70.8%	0.292	0.448	0.354	0.736	0.941
Fixed	1,600	1,083	1,486	76.8%	0.197	0.321	0.247	0.734	0.937
Fixed	2,000	884	1,818	81.3%	0.161	0.238	0.193	0.745	0.935
Heading-aware	200	6,367	229	0.0%	0.877	0.981	0.921	0.511	0.862
Heading-aware	400	3,980	367	0.0%	0.822	0.977	0.887	0.744	0.918
Heading-aware	600	3,149	464	0.0%	0.799	0.972	0.872	0.741	0.924
Heading-aware	800	2,739	533	0.0%	0.785	0.964	0.858	0.734	0.922
Heading-aware	1,200	2,398	609	0.0%	0.763	0.960	0.846	0.730	0.932
Heading-aware	1,600	2,250	649	0.0%	0.761	0.956	0.843	0.736	0.936
Heading-aware	2,000	2,176	671	0.0%	0.757	0.956	0.840	0.739	0.942

What the numbers say

Fixed chunks degrade fast with size under the strict rule. Strict hit@1 fell from 0.583 at 200 characters to 0.161 at 2,000, a 3.6x drop. The share of chunks crossing a section boundary rose from 22.5% to 81.3% over the same range. That is the mechanism: a big window mixes several sections.

Heading-aware chunks are nearly flat. Strict hit@1 went from 0.877 to 0.757, a 12-point drop across a 10x change in the size limit. The average chunk only grew from 229 to 671 characters, because the limit only splits sections that are longer than it. Most FastAPI sections are short, so the limit rarely bites at the high end. Your docs may behave differently.

At the same nominal size of 800, heading-aware beat fixed by 39 points of strict hit@1 (0.785 vs 0.391).

The strict metric favours small chunks, and the lenient one disagrees. Under the lenient rule, fixed chunks went from 0.810 hit@5 at 200 characters to 0.931 at 600, then stayed between 0.935 and 0.949 up to 2,000. Heading-aware at 200 characters has the lowest lenient hit@1 in the table (0.511): a 200-character chunk often has under 100 characters of body once the heading path is counted, so it can’t satisfy the rule. From 400 up, lenient hit@1 sits between 0.730 and 0.744 for heading-aware chunks and 0.700 to 0.745 for fixed ones.

My reading: the tiny-chunk win under the strict rule is partly an artefact of how I score. Finding a chunk from the right section is easy at any size above 400. Finding a clean chunk, one that is only about that section, is where small chunks and heading-aware splitting help.

Caveats

• Queries are headings. Real questions share fewer words with the docs. Expect smaller gaps.
• BM25 only. With embeddings, big chunks blur several topics into one vector, which probably hurts further, but I did not measure it.
• Retrieval, not answer quality. Tiny chunks may retrieve well and still give the LLM too little context to answer. This test can’t see that. Measure answer quality on your own set, see the golden-set post.
• Characters, not tokens. Roughly 4 characters per token for English prose, but code tokenises differently.
• Overlap was fixed at 10%. I did not sweep it here.

Copyable code

The sweep loop is the whole experiment. fixed and heading_aware are the chunkers from the chunking post; the full script is in the Experiment files box.

SIZES = [200, 400, 600, 800, 1200, 1600, 2000]
for size in SIZES:
    chunks = [c for t in docs.values() for c in heading_aware(t, size)]
    search = bm25_index([toks(text_of(c)) for c in chunks])
    # for each heading query: search(title, 5), then score strict and lenient hits

Rule of thumb

1. Fix the boundaries before the size. Splitting at headings gave more than any size change did.
2. With heading-aware chunks, 400 to 800 characters is a safe default on documentation like this. Going to 2,000 cost 6.5 points of strict hit@1 against 400 (0.822 to 0.757), in exchange for fewer chunks.
3. With fixed windows, stay at 400 or below if you can’t change the chunker.
4. Score with more than one metric. If your eval rule favours small chunks, it will tell you to use small chunks.