Benchmarks/Chunking

Five chunking strategies on real docs

Which chunking strategy retrieves best?

0.785hit@1, heading-aware + heading path

Short answer: Heading-aware chunks with the heading path added scored 0.785 hit@1, against 0.389 for fixed 800/100 chunks, which crossed a section boundary 58.8% of the time.

Tested 27 September 2026 · Read the full write-up

hit@1 and hit@5 by strategy↑ higher is better
0.000.250.500.751.00scoreFixed 800/100Fixed 800/100Fixed 530/100 (size-matched)Fixed 530/100 (size-m…Paragraph-packed 800Paragraph-packed 800Heading-awareHeading-awareHeading-aware + heading pathHeading-aware + headi…hit@1, Fixed 800/100: 0.389hit@1, Fixed 530/100 (size-matched): 0.481hit@1, Paragraph-packed 800: 0.375hit@1, Heading-aware: 0.712hit@1, Heading-aware + heading path: 0.785hit@5, Fixed 800/100: 0.611hit@5, Fixed 530/100 (size-matched): 0.766hit@5, Paragraph-packed 800: 0.576hit@5, Heading-aware: 0.933hit@5, Heading-aware + heading path: 0.966
  • hit@1
  • hit@5

What the numbers say

  • Splitting at headings first, then packing paragraphs, raised hit@1 from 0.389 to 0.712.
  • Adding the heading path to each chunk raised it again to 0.785.
  • A size-matched fixed chunker (530/100) still trailed at 0.481, so the gain is not just smaller chunks.

All results

Fixed 800/1002,14277058.8%7.3%0.3890.6110.477
Fixed 530/100 (size-matched)3,44651647.5%6.1%0.4810.7660.598
Paragraph-packed 8001,73783557.2%3.6%0.3750.5760.455
Heading-aware2,7135340.0%2.1%0.7120.9330.804
Heading-aware + heading path2,7135340.0%2.1%0.7850.9660.859

Setup

FastAPI English docs (151 Markdown files, 1.45M characters), 773 queries, BM25 retrieval (k1=1.5, b=0.75), Python 3.11. Full script at the end.

Reproduce it

The complete script is at the end of the write-up.

Read the full write-upFree lesson: ingestion and chunkingShare XLinkedIn

More benchmarks

All benchmarks →