What one RAG query costs: a token-by-token breakdown (with calculator)

How much does one RAG answer cost, and which part of the prompt do you pay for?

Short answer: the retrieved context dominates the input. With 512-token chunks and top-5 retrieval, the average prompt was 2,339 tokens, of which the question was about 10 and the system prompt 38. On gpt-5-mini that is about $1.09 per 1,000 queries, and the context alone is 53% of it. The model you pick matters more than any retrieval setting, though: gpt-5.4 cost 8.8 times as much as gpt-5-mini for the identical prompt.

Setup

I counted tokens, not characters, with tiktoken (o200k_base). Other providers use different tokenizers, so counts will differ somewhat for them.

• Corpus: the FastAPI docs, 153 pages.
• Chunks: token windows of 256, 512 or 1024 with 50 tokens of overlap.
• Retrieval: BM25, top-k of 3, 5 or 10.
• Queries: 200 section headings, phrased as “How do I: ?”.
• Prompt: a 38-token system prompt, numbered context chunks ([1] ...), then the question.

Prices, in USD per 1M tokens, as of 2026-11-11 from the OpenAI pricing page: gpt-5-nano $0.05 in / $0.40 out, gpt-5-mini $0.25 / $2.00, gpt-5.4 $2.50 / $15.00, and text-embedding-3-small $0.02 for embeddings. I assumed a 250-token answer. That number is a guess, not a measurement; change it in the script.

Results

Cost per 1,000 queries in USD, including the query embedding:

Chunk tokens	k	Avg input tokens	gpt-5-nano	gpt-5-mini	gpt-5.4	Context share (mini)
256	3	768	0.14	0.69	5.67	25.8%
256	5	1,267	0.16	0.82	6.92	37.1%
256	10	2,445	0.22	1.11	9.86	53.8%
512	3	1,382	0.17	0.85	7.21	39.3%
512	5	2,339	0.22	1.09	9.60	52.7%
512	10	4,783	0.34	1.70	15.71	69.7%
1024	3	2,621	0.23	1.16	10.30	55.5%
1024	5	4,197	0.31	1.55	14.24	66.8%
1024	10	8,566	0.53	2.64	25.17	80.6%

One-off indexing is negligible: the 512-token index was 960 chunks and 454,032 tokens, which costs $0.0091 to embed.

What the numbers mean

• Context grows linearly with chunk size times k. Going from 256×3 to 1024×10 raised the average input from 768 to 8,566 tokens, and the cost 3.8x on gpt-5-nano and gpt-5-mini and 4.4x on gpt-5.4.
• Output is not free. At 512×5 on gpt-5-mini, the assumed 250 output tokens cost about $0.50 per 1,000 queries, 46% of the total. Output tokens cost 8x input on this model, so a short answer format saves real money.
• Model choice beats tuning. At 512×5, gpt-5.4 cost 44 times gpt-5-nano and 8.8 times gpt-5-mini. No chunking change moved the bill that much.
• Actual context was below the nominal size (k=5 at 512 gave 2,285 tokens, not 2,560) because some chunks are the short tail of a page.

Caveats

• Cost is half the story. I did not measure answer quality, so a cheaper setting may lose correctness. Check it with a golden set before changing k. The chunk size sweep covers the retrieval side.
• Prompt caching, batch discounts and long-context tiers change the real bill. The pricing page lists cached input at a fraction of normal input price, which helps most with a long, stable system prompt. Your prompt is likely longer than my 38 tokens.
• Prices change. Re-check them before relying on this.
• Retries, agent loops, reranker calls and query rewriting each add another call. Multiply by calls per user request.

Copyable code

The script counts tokens per stage and prints the cost table; see the Experiment files box below for cost.py and results.txt. The core is:

import tiktoken
enc = tiktoken.get_encoding("o200k_base")
n = lambda s: len(enc.encode(s))

def query_cost(system, chunks, question, out_tokens, in_price, out_price):
    ctx = "\n\n".join(f"[{i+1}] {c}" for i, c in enumerate(chunks))
    prompt_tokens = n(system) + n(ctx) + n(question)
    return (prompt_tokens * in_price + out_tokens * out_price) / 1e6

Log the real usage fields your API returns too, and compare them with this estimate.

Rule of thumb

Budget roughly k × chunk_tokens + 100 input tokens per query, then multiply by your model’s input price and add the expected output. Pick the cheapest model that passes your eval, then reduce k before you shrink chunks. If you want the basics of how retrieval fits together first, see the retrieval lesson.