OpenAI SDK: Images, files & audio

Current models read images and PDFs as naturally as text: a receipt photo, a chart screenshot, a 40-page contract. Separate endpoints handle speech-to-text, text-to-speech and image generation. The pattern is always the same: a user message’s content becomes a list of parts instead of a single string.

1. Content parts

2. Images: understanding a receipt

import base64
from pydantic import BaseModel
from fake_openai import fake_client

class Receipt(BaseModel):
    merchant: str
    total: float
    currency: str

# a local file becomes a data URL (a tiny 1x1 PNG here, standing in for a real photo)
png = base64.b64decode("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg==")
data_url = "data:image/png;base64," + base64.b64encode(png).decode()

client = fake_client(script=['{"merchant": "Kochi Coffee House", "total": 485.0, "currency": "INR"}'])
receipt = client.responses.parse(
    model="gpt-5.6-luna",
    input=[{"role": "user", "content": [
        {"type": "input_text", "text": "Extract the merchant, total and currency from this receipt."},
        {"type": "input_image", "image_url": data_url, "detail": "low"},
    ]}],
    text_format=Receipt,
).output_parsed
print(receipt)
sent = client.fake.requests[0]["body"]["input"][0]["content"]
print([part["type"] for part in sent], sent[1]["image_url"][:40] + "…")

Output

merchant='Kochi Coffee House' total=485.0 currency='INR'
['input_text', 'input_image'] data:image/png;base64,iVBORw0KGgoAAAANSU…

3. PDFs and other files

real API — ask questions about a PDF

from openai import OpenAI

client = OpenAI()
uploaded = client.files.create(file=open("contract.pdf", "rb"), purpose="user_data")

response = client.responses.create(
    model="gpt-5.6-luna",
    input=[{"role": "user", "content": [
        {"type": "input_file", "file_id": uploaded.id},
        {"type": "input_text", "text": "List every termination clause with its notice period."},
    ]}],
)
print(response.output_text)

The model sees both the extracted text and images of each page, so tables and layouts survive. For large document collections that you search repeatedly, index them in a vector store instead (part 14), so you do not pay to send whole documents on every question.

4. Audio: transcription and speech

Speech → text

real API

with open("support_call.mp3", "rb") as audio:
    transcript = client.audio.transcriptions.create(model="gpt-transcribe", file=audio)
print(transcript.text)

Text → speech

real API

with client.audio.speech.with_streaming_response.create(
    model="gpt-4o-mini-tts", voice="alloy",
    input="Your order has been dispatched and will arrive on Friday.",
    instructions="Warm, clear, unhurried.",
) as speech:
    speech.stream_to_file("dispatch.mp3")

Transcription plus a text model plus speech is the simplest voice pipeline. For live, low-latency conversation, the Realtime API does all three in one session (part 19). Model names for audio change often, so the ones above are from the models page at the time of writing.

5. Generating images

real API

import base64

result = client.images.generate(
    model="gpt-image-2.5-flare",           # check the current image models
    prompt="Flat illustration of a delivery van driving through the Kerala backwaters at sunrise",
    size="1024x1024",
)
with open("banner.png", "wb") as f:
    f.write(base64.b64decode(result.data[0].b64_json))

Recap