Make your RAG say 'I don't know': a score threshold, measured

A RAG system always gets something back from the index. Ask it about cooking and it returns the five nearest chunks of API docs, and the LLM then tries to be helpful. So: can a simple score threshold on retrieval tell “the docs cover this” from “they don’t”?

Short answer: yes, mostly. On the FastAPI docs, the top-1 cosine score separated the two groups well (AUC 0.986), and a threshold of 0.40 refused 24 of 30 unanswerable questions while answering 29 of 30 answerable ones. The leftovers are near-domain questions, which no threshold fully fixes. Raw BM25 scores did worse, and BM25 almost never returns an empty result.

For the retrieval basics, see the free lesson Retrieval that actually works. The chunking and BM25 setup here is the same as in hybrid search with BM25 and RRF.

Setup

• Corpus: 152 files of the FastAPI docs, split into 2,808 heading-aware chunks with the heading path prefixed.
• Retrievers: MiniLM-L6-v2 cosine similarity, and BM25 over the same chunks.
• 30 answerable queries: hand-written questions, each matching one docs page (for example “How do I set a cookie on the response?”).
• 30 unanswerable queries: 15 near-domain (pagination, rate limiting, gRPC, Terraform, Prometheus, Selenium…) and 15 unrelated (egg boiling, football, tides…). I checked by keyword search that the near-domain topics are not documented pages, though a few terms are mentioned in passing.
• Score used for the decision: the top-1 chunk’s score. Refuse when it is below the threshold.

Results

Top-1 cosine score by group:

Group	Median	Min	Max
Answerable (30)	0.619	0.393	0.818
Unanswerable, near-domain (15)	0.385	0.256	0.508
Unanswerable, unrelated (15)	0.218	0.175	0.298

How well each score separates answerable from unanswerable (AUC, 1.0 is perfect, 0.5 is chance):

Signal	AUC
Cosine top-1	0.986
BM25 top-1, stopwords removed	0.927
BM25 top-1, raw	0.910
Cosine gap between top-1 and top-2	0.649

Cosine threshold sweep:

Threshold	Answerable kept	Near-domain refused	Unrelated refused	All unanswerable refused
0.30	100%	13%	100%	57%
0.35	100%	20%	100%	60%
0.40	97% (29/30)	60%	100%	80% (24/30)
0.45	93% (28/30)	73%	100%	87% (26/30)
0.50	90% (27/30)	93%	100%	97% (29/30)
0.55	73%	100%	100%	100%

I also split each group in half, picked the threshold on one half and scored the other. Requiring every answerable calibration query to be kept gave a threshold of 0.393. On the held-out half it kept 15 of 15 answerable queries and refused 11 of 14 unanswerable ones (79%).

What the numbers mean

• Unrelated questions are easy. Their best score (0.298) is below the worst answerable score (0.393). Any threshold between 0.30 and 0.39 catches all of them.
• Near-domain questions are the real problem. Their scores run up to 0.508, which overlaps the lowest answerable queries (0.393, 0.405, 0.500, 0.508, 0.511). The highest-scoring one was “How do I expose Prometheus metrics from my service?” at 0.508, matched to the OpenTelemetry page. Arguably that page is a partial answer, so this one is a labeling grey zone as much as a miss.
• The cost of tightening is visible. Going from 0.40 to 0.50 refuses 5 more unanswerable queries but also loses 2 more answerable ones (the logged-in-user and WebSocket questions, the lowest scorers).
• BM25 is a weaker gate. Its raw scores depend on query length and term rarity, so one cutoff does not transfer. It also never returned an empty candidate list for any of the 60 queries, because words like “how” and “do” match something. With stopwords removed, only 5 of the 30 unanswerable queries came back empty, and no answerable ones. “Check for empty results” alone is not a refusal strategy.
• The top-1 vs top-2 gap is nearly useless here (AUC 0.649). Don’t spend time on it.

Caveats

• 60 queries is small. One query moves a percentage by about 3 points, and the held-out test had only 14 unanswerable queries.
• I wrote both query sets myself, and the unanswerable ones are cleaner than real traffic. Real users ask half-covered or ambiguous questions.
• Thresholds are specific to this model and corpus. Cosine scores from another embedding model sit on a different scale, so recalibrate whenever you change the model or re-chunk.
• This tests retrieval only. A model can still hallucinate when the retrieved chunks are on topic but do not contain the answer; that needs an answer-level check such as citation validation.

Copyable code

import numpy as np

def should_refuse(query_vec, chunk_matrix, threshold=0.40):
    """Vectors must be L2-normalised. Returns (refuse, top_score, top_index)."""
    sims = chunk_matrix @ query_vec
    top = int(np.argmax(sims))
    return bool(sims[top] < threshold), float(sims[top]), top

REFUSAL = "I couldn't find this in the documentation."

Pick the threshold from your own labeled set: take the highest value that still answers at least 95-100% of your known-answerable queries, then check what it refuses.

Rule of thumb

1. Build a set of at least 50 answerable and 50 unanswerable queries from your own logs, including near-domain ones.
2. Use the top-1 cosine score, not BM25, as the gate, and set the threshold just under your lowest answerable score.
3. Put the threshold in a test, so a model or chunking change that moves the score scale fails CI. See a golden set in CI.
4. Expect to refuse most, not all, of the near-domain questions. Treat the rest with answer-level checks.

The scripts and raw output are in the Experiment files box below.