Stop fake citations in RAG: validate them server-side in 25 lines of Python

You give the model five retrieved passages and ask it to cite sources as [1] to [5]. Most of the time it does. Sometimes it writes [7]. Sometimes it writes a confident answer with no citations at all. If your UI turns those markers into links, users see a source that doesn’t exist, and they stop trusting everything else.

The fix is simple and deterministic: treat citations as untrusted output and check them on the server before anything reaches the user.

What the validator does

1. Finds every [n] or [n, m] marker.
2. Keeps numbers that point to a passage the model actually saw (1..n_passages).
3. Removes invented numbers from the text and records them.
4. Returns grounded = True only if at least one valid citation remains and the model didn’t refuse.

The code

import re
from dataclasses import dataclass

NO_ANSWER = "I don't know based on the provided documents."
CITE = re.compile(r"\[(\d+(?:\s*,\s*\d+)*)\]")

@dataclass
class Checked:
    answer: str            # answer with invalid citation markers removed
    cited: list[int]       # valid passage numbers, in order of first use
    invalid: list[int]     # numbers the model made up
    grounded: bool         # safe to show without a warning

def check_citations(answer: str, n_passages: int) -> Checked:
    cited, invalid = [], []
    def fix(m: re.Match) -> str:
        keep = []
        for part in m.group(1).split(","):
            n = int(part)
            if 1 <= n <= n_passages:
                keep.append(n)
                if n not in cited: cited.append(n)
            elif n not in invalid:
                invalid.append(n)
        return "[" + ", ".join(map(str, keep)) + "]" if keep else ""
    cleaned = CITE.sub(fix, answer)
    cleaned = re.sub(r"\s+([.,;:!?])", r"\1", cleaned).strip()
    refused = NO_ANSWER.lower() in answer.lower()
    return Checked(cleaned, cited, invalid, grounded=bool(cited) and not refused)

The tests

All eight pass:

from citations import check_citations, NO_ANSWER

def test_valid_citations_kept():
    r = check_citations("Use BM25 for codes [1]. Fuse with RRF [2].", 3)
    assert r.cited == [1, 2] and r.invalid == [] and r.grounded

def test_out_of_range_dropped():
    r = check_citations("Refunds take 5 days [7].", 5)
    assert r.answer == "Refunds take 5 days." and r.invalid == [7] and not r.grounded

def test_mixed_group():
    r = check_citations("See the docs [1, 9, 2].", 3)
    assert r.answer == "See the docs [1, 2]." and r.cited == [1, 2] and r.invalid == [9]

def test_zero_is_invalid():
    r = check_citations("Answer [0].", 3)
    assert r.invalid == [0] and not r.grounded

def test_no_citations_not_grounded():
    assert not check_citations("Probably yes.", 3).grounded

def test_refusal_not_grounded_even_with_marker():
    assert not check_citations(NO_ANSWER + " [1]", 3).grounded

def test_dedup_and_order():
    r = check_citations("A [2]. B [1]. C [2].", 2)
    assert r.cited == [2, 1]

def test_years_in_brackets_are_not_passages():
    r = check_citations("Released in [2024] per [1].", 3)
    assert r.invalid == [2024] and r.cited == [1]

$ pytest -q
........                                                   [100%]
8 passed in 0.02s

Wiring it into FastAPI

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class Source(BaseModel):
    n: int
    title: str
    url: str

class Answer(BaseModel):
    answer: str
    sources: list[Source]
    grounded: bool
    dropped_citations: list[int]

@app.post("/ask", response_model=Answer)
async def ask(q: str):
    hits = retrieve(q, k=5)                       # your retriever
    if not hits:                                  # nothing relevant: don't call the LLM
        return Answer(answer=NO_ANSWER, sources=[], grounded=False, dropped_citations=[])
    raw = await generate(q, hits)                 # your LLM call, passages numbered 1..5
    r = check_citations(raw, len(hits))
    if r.invalid:
        log.warning("invalid citations", extra={"invalid": r.invalid, "q": q})
    return Answer(
        answer=r.answer,
        sources=[Source(n=n, title=hits[n - 1].title, url=hits[n - 1].url) for n in r.cited],
        grounded=r.grounded,
        dropped_citations=r.invalid,
    )

Only the passages the model actually cited are returned as sources, in the order it cited them.

What to do when grounded is false

• Show a warning (“This answer isn’t backed by your documents”) instead of hiding the answer.
• Or retry once with a stricter instruction, then fall back to the warning.
• Log dropped_citations. A rising rate is an early signal that a prompt or model change made things worse. It is a free metric for your eval dashboard.

Two gotchas

Brackets that aren’t citations. Years like [2024] and array indexes in code look like citation markers. The validator removes out-of-range numbers, which is safe for prose but can mangle code. If your documents contain code, use a distinct marker such as [S1] in the prompt and the regex.

Refusals must be exact. The refusal check relies on the model using your exact refusal sentence. Put it in the system prompt verbatim, and test it on questions your documents can’t answer.

Why this matters more than a better prompt

Prompts reduce the rate of bad citations. They don’t make it zero, and the rate changes every time you switch models. A 25-line check makes the rate of invalid citations shown to users zero, whatever model you use. It is one of the cheapest reliability wins in a RAG system.

This is a specific case of a general pattern: let the LLM write prose, then have a separate, non-LLM decision function verify the part of its output that has a mechanically checkable answer. Jev vs LLMs looks at that split when the verifier is a trained decision model instead of a regex.