OpenAI SDK: Cost & latency optimisation

LLM costs scale with tokens × price × requests, and latency scales mostly with output length and model size. So the levers are few and powerful: send fewer tokens, generate fewer tokens, use a smaller model where it suffices, reuse repeated prefixes (prompt caching), and move non-urgent work to the Batch API. None of it matters until you measure, so start with tracking.

1. Measure first: track usage on every call

from collections import defaultdict
from fake_openai import fake_client

PRICES = {"gpt-5.6-luna": (0.20, 1.20)}        # USD per 1M tokens (input, output), kept in config

class CostTracker:
    def __init__(self):
        self.by_feature = defaultdict(lambda: {"calls": 0, "in": 0, "out": 0, "usd": 0.0})

    def record(self, feature, response):
        p_in, p_out = PRICES[response.model]
        u = response.usage
        row = self.by_feature[feature]
        row["calls"] += 1
        row["in"] += u.input_tokens
        row["out"] += u.output_tokens
        row["usd"] += (u.input_tokens * p_in + u.output_tokens * p_out) / 1_000_000

tracker = CostTracker()
client = fake_client(script=[{"text": "Positive", "usage": (220, 3)}] * 3
                            + [{"text": "A long summary...", "usage": (4_000, 900)}] * 2)
for _ in range(3):
    tracker.record("review-sentiment", client.responses.create(model="gpt-5.6-luna", input="review"))
for _ in range(2):
    tracker.record("ticket-summary", client.responses.create(model="gpt-5.6-luna", input="ticket"))

for feature, row in tracker.by_feature.items():
    print(f"{feature:17} calls={row['calls']} in={row['in']:>6} out={row['out']:>5} "
          f"cost=${row['usd']:.5f}  per call=${row['usd'] / row['calls']:.6f}")

Output

review-sentiment  calls=3 in=   660 out=    9 cost=$0.00014  per call=$0.000048
ticket-summary    calls=2 in=  8000 out= 1800 cost=$0.00376  per call=$0.001880

Record usage per feature, not just in total. “Summaries cost 50× more than sentiment labels” is actionable. “We spent $4,000” is not. In production, send these numbers to your metrics system (the AI Engineering guide covers Langfuse and OpenTelemetry for this).

2. The levers, in order of impact

3. Prompt caching

r = client.responses.create(
    model=MODEL,
    instructions=LONG_STABLE_INSTRUCTIONS,          # identical across calls
    tools=TOOL_SCHEMAS,                             # identical across calls
    input=user_message,                             # the only part that changes
    prompt_cache_key="support-bot-v3",              # optional: groups requests that share a prefix
)
print(r.usage.input_tokens, r.usage.input_tokens_details.cached_tokens)

4. The Batch API

build the JSONL input file

import json

reviews = {"r-001": "Great lamp, fast delivery.", "r-002": "Arrived broken.", "r-003": "OK for the price."}
lines = []
for rid, text in reviews.items():
    lines.append(json.dumps({
        "custom_id": rid,                               # how you match results back later
        "method": "POST",
        "url": "/v1/responses",
        "body": {"model": "gpt-5.6-luna",
                 "instructions": "Label the review Positive, Negative or Neutral. One word.",
                 "input": text, "max_output_tokens": 16},
    }))
with open("batch_input.jsonl", "w", encoding="utf-8") as f:
    f.write("\n".join(lines))
print(open("batch_input.jsonl", encoding="utf-8").read().splitlines()[0][:120], "…")
print(len(lines), "requests")

Output

{"custom_id": "r-001", "method": "POST", "url": "/v1/responses", "body": {"model": "gpt-5.6-luna", "instructions": "Labe …
3 requests

submit, poll, collect (real API)

batch_file = client.files.create(file=open("batch_input.jsonl", "rb"), purpose="batch")
batch = client.batches.create(input_file_id=batch_file.id, endpoint="/v1/responses",
                              completion_window="24h")
# ...later (poll every few minutes, or from a scheduled job):
batch = client.batches.retrieve(batch.id)
if batch.status == "completed":
    results = client.files.content(batch.output_file_id).text
    for line in results.splitlines():
        row = json.loads(line)
        print(row["custom_id"], row["response"]["status_code"])

Batch jobs trade latency for price: results arrive within the completion window, usually much sooner, at a discount, and they do not consume your normal rate limits. Nightly classification, backfills, evaluation runs and embedding a whole corpus are ideal. Always check error_file_id for rows that failed.

5. Latency checklist

Recap