OpenAI SDK: Realtime & voice
A voice assistant built from three calls (transcribe → think → speak, part 09) works, but each hop adds latency, and conversation feels stilted.
In this post
What this post covers
A voice assistant built from three calls (transcribe → think → speak, part 09) works, but each hop adds latency, and conversation feels stilted. The Realtime API keeps a persistent session (WebRTC in browsers, WebSocket on servers) with a speech-to-speech model. It hears audio as it streams, detects when the user stops speaking, can be interrupted, calls tools, and answers with audio in fractions of a second.
1. Chained versus realtime
2. The concepts
| Concept | Meaning |
|---|---|
| Session | a long-lived connection with configuration: instructions, voice, tools, audio formats |
| Events | everything is an event in both directions: audio chunks appended, responses created, deltas, tool calls |
| Turn detection (VAD) | the server decides when the user has finished speaking, so the model can reply without a button |
| Interruption | if the user talks over the reply, the reply is cancelled, as in a real conversation |
| Ephemeral keys | browsers get a short-lived token from your server, never your real API key |
3. A server-side session (sketch)
text in, text out over a realtime connection (real API; not executed here)
import asyncio
from openai import AsyncOpenAI
async def main():
client = AsyncOpenAI()
async with client.realtime.connect(model="gpt-realtime-2.1") as conn: # a current realtime model
await conn.session.update(session={
"type": "realtime",
"instructions": "You are a friendly order-status assistant. Keep answers short.",
"output_modalities": ["text"], # use ["audio"] (and stream mic audio in) for voice
})
await conn.conversation.item.create(item={
"type": "message", "role": "user",
"content": [{"type": "input_text", "text": "Is order 1042 delivered?"}],
})
await conn.response.create()
async for event in conn:
if event.type == "response.output_text.delta":
print(event.delta, end="", flush=True)
elif event.type == "response.done":
break
asyncio.run(main())Realtime is the fastest-moving part of the API. Session fields, event names and model IDs have changed across versions, and the models page currently lists several realtime and "live" models. Treat this code as the shape of the solution. Use the official Realtime guide and the Agents SDK's voice and realtime agents for production voice apps, and test on real devices and networks.
4. When to choose which
Realtime
Live voice conversation: phone agents, in-app voice assistants, interpreting, tutoring. Anything where a one-second pause feels broken.
Chained
Voice notes, transcription of recordings, podcasts from text, or when you need a specific text model in the middle and a few seconds of latency is fine.
Recap
- Realtime = one persistent session streaming audio both ways, with turn detection and interruption.
- Everything is an event; tools work as in the Responses API.
- Browsers use ephemeral keys minted by your server.
- Fast-moving API: follow the current guide; use chained calls when latency does not matter.
Get the free RAG production checklist
25 checks to run before real users see your RAG feature: exact-match misses, invented citations, prompt injection through documents, quality that slips after a prompt change. Confirm your email and the PDF comes with the welcome email, plus each new experiment.