Recovering headings from PDFs for RAG chunking: font size alone found 26%

Heading-aware chunking needs headings, and a PDF does not store them. It stores glyphs with fonts and positions. So how well can you rebuild the heading structure from font information alone?

Short answer: not well with the obvious rule. “Any line in a font larger than the body text” found 26% of the real headings across five papers. Adding bold detection and a numbered-heading pattern (“2.1 Models”) gave the best F1 (0.63) and the best precision (77%), but still only 53% recall. A regex on the numbering alone, with no font information, reached 72% recall at 47% precision. Two of the five PDFs defeated every font rule.

Setup

Ground truth is the PDF outline, the bookmark tree that LaTeX’s hyperref writes. I read it with pypdf. It is a convenient oracle because a human or the typesetter created it, but it is not perfect: it can list entries (like “References”) that look different from section headings.

For prediction I used pdfplumber to read words with font name and size, grouped words into lines by vertical position, and found the body size as the font size covering the most characters. Every paper’s body size came out at 10.0 pt. I compared five rules on lines of 14 words or fewer that don’t end in ., , or ;:

• A: font size at least 0.5 pt above body.
• B: A, or the whole line is bold.
• C: B, and the line starts with a capital or a number.
• D: B, and the line matches a numbered-heading pattern (2.1 Title, A Title).
• E: the numbered-heading pattern only, no font information.

A predicted line counts as correct if its normalised text (numbering stripped, lowercase, alphanumeric only) equals an outline title or is a prefix of one, which allows for headings wrapped over two lines. Precision is correct lines over predicted lines. Recall is outline entries found over outline entries.

Results

Pooled over all five PDFs (from results.txt, in the Experiment files box below):

Rule	Lines predicted	Precision	Recall	F1
A. larger than body	107	0.38	0.26	0.31
B. larger or bold	255	0.36	0.53	0.43
C. B + starts with capital/number	227	0.40	0.53	0.46
D. B + numbered pattern	114	0.77	0.53	0.63
E. numbered pattern only	254	0.47	0.72	0.57

Per paper, rule D:

PDF	Precision	Recall
Attention Is All You Need (1706.03762)	0.96	1.00
RAG paper (2005.11401)	0.97	1.00
Chain-of-thought (2201.11903)	0.76	1.00
ReAct (2210.03629)	0.13	0.07
RAG survey (2312.10997)	0.00	0.00

What the numbers mean

Headings are often the same size as body text. Body text is 10 pt in all five PDFs, and rule A recalled at most 59% of the headings in any single paper. LaTeX conference styles mark section headings with a bold face. Rule A finds only titles and the occasional larger heading.

My first run was wrong because of font names. I detected bold with the pattern bold|bd|black. The papers use Nimbus Roman, whose bold face is named NimbusRomNo9L-Medi. With that miss, rule B recalled 36% on the first paper. Adding medi|semib|demi moved it to 100%. Font names are not standardised, so print the distinct (font, size) pairs for your corpus before trusting any bold test.

The numbered pattern is the strongest signal, and font filters it. Rule E alone has the best recall of any rule (0.72) but also picks up numbered list items and table rows: on the chain-of-thought paper it predicted 109 lines for 35 headings. Combining it with bold (rule D) cut predicted lines from 254 to 114 and raised precision from 0.47 to 0.77, at a cost of 0.19 recall.

Two PDFs broke every rule, for different reasons.

• ReAct: 486 of 10,755 extracted words (4.5%) are unreadable (cid:NNN) glyph codes, from fonts with no usable Unicode map. Words in some regions also come out glued together, so the text can’t match anything. Headings in those regions can’t be recovered from the text layer; you would need OCR.
• The RAG survey (IEEE two-column format): sub-headings such as B. are set in an italic face, which my bold rule doesn’t match, and my line grouping merges the two columns. Only 5.5% of words are italic, so an italic rule is possible, but I did not test one.

Caveats

• Five PDFs, all LaTeX, all arXiv. Business PDFs from Word or InDesign have different font conventions and often no outline to score against.
• The oracle is the outline. Entries such as “References” are counted as headings, and a heading missing from the outline counts as a false positive. Precision is therefore a little pessimistic.
• Two of five papers scored near zero, so the pooled figures depend heavily on the three that worked. Rule D’s per-paper results on those three were 0.76 to 0.97 precision and 1.00 recall.
• I did not test whether better headings improve retrieval. For how chunk boundaries affect retrieval, see RAG chunking strategies, measured and chunk size sweep.

Code

The scoring script is bench.py in the Experiment files box. The core of the best rule:

import re, collections, pdfplumber

BOLD = re.compile(r"bold|bd|black|heavy|-b$|bx|medi|semib|demi", re.I)
NUM = re.compile(r"^(?:\d+(?:\.\d+)*\.?|[A-Z](?:\.\d+)*\.?)\s+[A-Z]")

def headings(path):
    with pdfplumber.open(path) as pdf:
        for page in pdf.pages:
            rows = collections.defaultdict(list)
            for w in page.extract_words(extra_attrs=["size", "fontname"]):
                rows[round(w["top"] / 3)].append(w)
            for k in sorted(rows):
                ws = sorted(rows[k], key=lambda w: w["x0"])
                text = " ".join(w["text"] for w in ws)
                bold = all(BOLD.search(w["fontname"]) for w in ws)
                big = max(w["size"] for w in ws) >= 10.5  # body + 0.5; compute per document
                if (bold or big) and len(ws) <= 14 and NUM.match(text) \
                        and not text.endswith((".", ",", ";")):
                    yield page.page_number, text

Rule of thumb

1. Print the distinct (font name, size) pairs of a sample of your PDFs first. Don’t assume headings are larger.
2. Require two signals: a bold or larger font and a numbered pattern. That gave 0.77 precision here.
3. Check each document for unreadable (cid: text and for a built-in outline. If an outline exists, use it, since it is the cheapest way to get headings.
4. Measure per document, not pooled. Pooled numbers hid two total failures.
5. Fall back to fixed-size chunking for documents where heading recovery fails, and log which ones.

For the basics of splitting documents, see the free guide Ingestion & chunking.