In this post
What this post covers
“FastAPI is fast” is true of the framework overhead, which is rarely your bottleneck. Real services are limited by CPU work in one process, by waiting on databases and APIs, by connection pools, and by accidentally repeated work. This part runs real load against real uvicorn processes to show what scales, what does not, and why. Along the way it finds a surprise: the load generator itself can be the bottleneck.
A coffee bar
One barista (a process) can take many orders while the milk steams (async waiting), so waiting-heavy work needs no extra staff. But if every drink needs two minutes of hand-grinding (CPU work), the only fix is more baristas. Adding baristas does nothing if there are only three cups (the connection pool), and the cheapest drink is the one you already made and kept warm (the cache).
1. The model: processes, the event loop, Little’s law
One process ≈ one core
Python's GIL lets one thread run Python code at a time. CPU-bound work in a single uvicorn process tops out at one core, whatever the concurrency.
Waiting is nearly free
An async endpoint that awaits I/O lets the loop serve thousands of other requests meanwhile (part 11).
Little's law
in-flight requests = throughput × latency. 50 clients each waiting 67 ms can produce at most 50 / 0.067 ≈ 750 requests per second.
2. Scaling with processes, measured
This block starts one and then four uvicorn processes, and drives them with a minimal load generator: raw
keep-alive sockets, spread round-robin as a load balancer would. /cpu burns about 30 ms of CPU;
/io waits 50 ms.
1 vs 4 processes, CPU-bound vs I/O-bound
import asyncio
import json
import os
import signal
import socket
import statistics
import subprocess
import sys
import time
from collections import Counter
from pathlib import Path
import httpx
Path("main.py").write_text('''
import asyncio, hashlib, os
from fastapi import FastAPI
app = FastAPI()
@app.get("/cpu")
def cpu(): # ~30 ms of pure CPU work
data = b"x" * 1024
for _ in range(12_000):
data = hashlib.sha256(data).digest() * 32
return {"pid": os.getpid()}
@app.get("/io")
async def io(): # 50 ms of waiting, like a database call
await asyncio.sleep(0.05)
return {"pid": os.getpid()}
''', encoding="utf-8")
def start_replicas(n: int) -> list[tuple[int, subprocess.Popen]]:
replicas = []
for _ in range(n):
with socket.socket() as s:
s.bind(("127.0.0.1", 0))
port = s.getsockname()[1]
flags = subprocess.CREATE_NEW_PROCESS_GROUP if os.name == "nt" else 0
replicas.append((port, subprocess.Popen([sys.executable, "-m", "uvicorn", "main:app", "--port", str(port),
"--log-level", "warning"], creationflags=flags)))
for port, _ in replicas:
for _ in range(100):
try:
httpx.get(f"http://127.0.0.1:{port}/io")
break
except httpx.TransportError:
time.sleep(0.2)
return replicas
def stop(replicas):
for _, server in replicas:
server.send_signal(signal.CTRL_BREAK_EVENT if os.name == "nt" else signal.SIGTERM)
server.wait(timeout=30)
async def connection(port, path, n, latencies, per_process):
"""One keep-alive connection sending n requests: a minimal, fast load generator."""
reader, writer = await asyncio.open_connection("127.0.0.1", port)
for _ in range(n):
t = time.perf_counter()
writer.write(f"GET {path} HTTP/1.1\r\nHost: bench\r\n\r\n".encode())
head = await reader.readuntil(b"\r\n\r\n")
size = int(next(h for h in head.split(b"\r\n") if h.lower().startswith(b"content-length")).split(b":")[1])
per_process[json.loads(await reader.readexactly(size))["pid"]] += 1
latencies.append((time.perf_counter() - t) * 1000)
writer.close()
async def load(ports, path, connections, per_connection):
latencies, per_process = [], Counter()
t = time.perf_counter()
await asyncio.gather(*(connection(ports[i % len(ports)], path, per_connection, latencies, per_process)
for i in range(connections))) # round-robin, like a load balancer
q = statistics.quantiles(latencies, n=100)
return len(latencies) / (time.perf_counter() - t), q[49], q[98]
def report(label, rps, p50, p99):
print(f"{label:<30} {rps:6.0f} req/s p50 {p50:5.0f} ms p99 {p99:5.0f} ms")
for n in (1, 4):
replicas = start_replicas(n)
ports = [port for port, _ in replicas]
asyncio.run(load(ports, "/cpu", 8, 1)) # warm-up
report(f"{n} process{'es' if n > 1 else ''}, /cpu, 16 clients", *asyncio.run(load(ports, "/cpu", 16, 10)))
report(f"{n} process{'es' if n > 1 else ''}, /io, 50 clients", *asyncio.run(load(ports, "/io", 50, 10)))
if n == 1:
async def with_httpx(port):
latencies = []
async with httpx.AsyncClient() as client:
async def one():
t = time.perf_counter()
await client.get(f"http://127.0.0.1:{port}/io")
latencies.append((time.perf_counter() - t) * 1000)
t = time.perf_counter()
for _ in range(10):
await asyncio.gather(*(one() for _ in range(50)))
q = statistics.quantiles(latencies, n=100)
return len(latencies) / (time.perf_counter() - t), q[49], q[98]
report("1 process, /io, httpx as client", *asyncio.run(with_httpx(ports[0])))
stop(replicas)Output
1 process, /cpu, 16 clients 35 req/s p50 456 ms p99 471 ms
1 process, /io, 50 clients 721 req/s p50 67 ms p99 72 ms
1 process, /io, httpx as client 292 req/s p50 134 ms p99 175 ms
4 processes, /cpu, 16 clients 131 req/s p50 118 ms p99 178 ms
4 processes, /io, 50 clients 773 req/s p50 64 ms p99 72 ms| What the numbers show | Why |
|---|---|
/cpu scales almost exactly 4× with 4 processes | CPU work needs cores; each process has its own GIL. |
/io does not change with more processes | One event loop already overlaps all the waiting. The ceiling is Little's law: 50 clients ÷ 67 ms ≈ 750 req/s. |
The same /io test driven by httpx.AsyncClient is two to three times slower (it varied between runs) | The client saturated first. Benchmark with a dedicated tool (k6, oha, wrk, Locust with many workers) and always check that the load generator is not the limit. |
uvicorn --workers 4 runs four processes sharing one socket; on Linux servers that is fine, and
recent uvicorn versions' supervisor health-checks and restarts crashed workers (and restarts all of them on SIGHUP). In containers the usual advice (also FastAPI's)
is one process per container and let Kubernetes or ECS run replicas: the orchestrator then sees each
process's health and memory. gunicorn -k uvicorn_worker.UvicornWorker is the other classic
setup; the old uvicorn.workers module is deprecated in favour of the uvicorn-worker
package. On Windows, multi-worker uvicorn is unreliable (it failed while this part was being written), which
is one more reason the demo uses separate processes.
3. Reuse outbound connections
The most common performance bug in FastAPI services: creating an httpx.AsyncClient (or a
database engine, or an SDK client) per request. Each one builds a TLS context and opens new TCP
connections. Create it once in the lifespan (part 10) and share it:
a client per request vs one shared client (real upstream server)
import asyncio
import os
import signal
import socket
import subprocess
import sys
import time
from contextlib import asynccontextmanager
from pathlib import Path
import httpx
from fastapi import FastAPI, Request
Path("upstream.py").write_text('''
from fastapi import FastAPI
app = FastAPI()
@app.get("/rates")
async def rates():
return {"INR": 1.0, "USD": 0.012}
''', encoding="utf-8")
with socket.socket() as s: # a real upstream service on a real port
s.bind(("127.0.0.1", 0))
port = s.getsockname()[1]
flags = subprocess.CREATE_NEW_PROCESS_GROUP if os.name == "nt" else 0
upstream = subprocess.Popen([sys.executable, "-m", "uvicorn", "upstream:app", "--port", str(port),
"--log-level", "warning"], creationflags=flags)
URL = f"http://127.0.0.1:{port}/rates"
for _ in range(100):
try:
httpx.get(URL)
break
except httpx.TransportError:
time.sleep(0.2)
@asynccontextmanager
async def lifespan(app):
async with httpx.AsyncClient(timeout=httpx.Timeout(5.0, connect=2.0)) as client:
yield {"http": client} # one client for the whole process
app = FastAPI(lifespan=lifespan)
@app.get("/per-request")
async def per_request():
async with httpx.AsyncClient() as client: # new pool, new TLS context, new TCP connection
return (await client.get(URL)).json()
@app.get("/shared")
async def shared(request: Request):
return (await request.state.http.get(URL)).json() # pooled keep-alive connections
async def main():
async with app.router.lifespan_context(app) as state:
async def with_state(scope, receive, send):
scope["state"] = dict(state)
await app(scope, receive, send)
async with httpx.AsyncClient(transport=httpx.ASGITransport(app=with_state), base_url="http://api") as c:
for path in ("/per-request", "/shared"):
await c.get(path)
t = time.perf_counter()
for _ in range(200):
await c.get(path)
print(f"{path:<13} {(time.perf_counter() - t) / 200 * 1000:5.2f} ms per request")
t = time.perf_counter()
for _ in range(50):
httpx.AsyncClient()
print(f"just constructing an AsyncClient: {(time.perf_counter() - t) / 50 * 1000:.2f} ms")
asyncio.run(main())
upstream.send_signal(signal.CTRL_BREAK_EVENT if os.name == "nt" else signal.SIGTERM)
upstream.wait(timeout=30)Output
/per-request 18.25 ms per request
/shared 1.96 ms per request
just constructing an AsyncClient: 12.71 ms4. Caching, and the stampede
A cache in front of a slow query is easy. The trap is expiry under load: the moment the entry expires,
hundreds of concurrent requests all miss and all hit the database at once (a “cache stampede”). The fix is
single-flight: the first miss loads, and everyone else awaits the same result. HTTP caching adds a
second layer: an ETag lets clients revalidate and get 304 Not Modified with no body.
naive cache vs single-flight, plus ETag revalidation
import asyncio
import hashlib
import json
import time
import httpx
from fastapi import FastAPI, Request, Response
db_calls = 0
async def load_menu_from_db() -> dict:
global db_calls
db_calls += 1
await asyncio.sleep(0.05)
return {"version": 7, "items": [{"sku": "latte", "price": 180}, {"sku": "mocha", "price": 210}]}
class TTLCache:
def __init__(self, ttl: float):
self.ttl, self.data = ttl, {}
async def get(self, key, loader):
hit = self.data.get(key)
if hit and hit[0] > time.monotonic():
return hit[1]
value = await loader() # every concurrent miss loads
self.data[key] = (time.monotonic() + self.ttl, value)
return value
class SingleFlightCache(TTLCache):
def __init__(self, ttl: float):
super().__init__(ttl)
self.inflight: dict[str, asyncio.Task] = {}
async def get(self, key, loader):
hit = self.data.get(key)
if hit and hit[0] > time.monotonic():
return hit[1]
task = self.inflight.get(key)
if task is None: # first miss starts the load...
task = self.inflight[key] = asyncio.create_task(loader())
task.add_done_callback(lambda _: self.inflight.pop(key, None))
value = await asyncio.shield(task) # ...everyone else waits for it
self.data[key] = (time.monotonic() + self.ttl, value)
return value
def build(cache) -> FastAPI:
app = FastAPI()
@app.get("/menu")
async def menu(request: Request):
data = await cache.get("menu", load_menu_from_db)
body = json.dumps(data, separators=(",", ":")).encode()
etag = '"' + hashlib.sha256(body).hexdigest()[:16] + '"'
headers = {"ETag": etag, "Cache-Control": "public, max-age=30"}
if request.headers.get("if-none-match") == etag:
return Response(status_code=304, headers=headers) # the client's copy is current
return Response(body, media_type="application/json", headers=headers)
return app
async def stampede(cache) -> tuple[httpx.Response, httpx.Response]:
global db_calls
db_calls = 0
transport = httpx.ASGITransport(app=build(cache))
async with httpx.AsyncClient(transport=transport, base_url="http://api") as client:
t = time.perf_counter()
responses = await asyncio.gather(*(client.get("/menu") for _ in range(200)))
ms = (time.perf_counter() - t) * 1000
print(f"{type(cache).__name__:<18} 200 concurrent cold requests -> {db_calls:3d} database call{'s' if db_calls != 1 else ''} ({ms:.0f} ms)")
again = await client.get("/menu", headers={"If-None-Match": responses[0].headers["etag"]})
return responses[0], again
asyncio.run(stampede(TTLCache(ttl=30)))
first, again = asyncio.run(stampede(SingleFlightCache(ttl=30)))
print(f"\nfirst response : {first.status_code}, {len(first.content)} bytes, ETag {first.headers['etag']}")
print(f"revalidation : {again.status_code}, {len(again.content)} bytes")Output
TTLCache 200 concurrent cold requests -> 200 database calls (107 ms)
SingleFlightCache 200 concurrent cold requests -> 1 database call (143 ms)
first response : 200, 79 bytes, ETag "bd6b3bd731e6df4b"
revalidation : 304, 0 bytesAcross processes: Redis
An in-process cache is per worker. For shared caching use Redis with the same idea: a short lock key (SET key NX PX 5000) so only one process recomputes. Add random jitter to TTLs so entries don't all expire together.
Invalidate on write
Delete or bump a version key when data changes; keep TTLs short where stale data would hurt (prices, stock). Never cache per-user data under a shared key.
5. Pool arithmetic
Connection limits multiply across layers. Do this arithmetic before an incident does it for you:
how many database connections will we open?
def plan(label, pods, workers, pool_size, max_overflow, db_max=100, reserved=10):
peak = pods * workers * (pool_size + max_overflow)
verdict = "OK" if peak <= db_max - reserved else "exceeds max_connections: 'too many clients' at peak"
layout = f"{pods} pods x {workers} worker{'s' if workers > 1 else ''} x ({pool_size}+{max_overflow})"
print(f"{label:<26} {layout:<34} = {peak:4d} {verdict}")
plan("SQLAlchemy defaults", pods=4, workers=4, pool_size=5, max_overflow=10)
plan("'bigger pool is faster'", pods=4, workers=4, pool_size=20, max_overflow=20)
plan("sized from Little's law", pods=4, workers=1, pool_size=5, max_overflow=5)
plan("autoscaled to 12 pods", pods=12, workers=1, pool_size=5, max_overflow=5)
rps, db_seconds = 600, 0.012 # measured: 600 req/s, 12 ms of database time each
print(f"\nconnections busy on average: {rps} x {db_seconds} = {rps * db_seconds:.1f} (spread over all processes)")Output
SQLAlchemy defaults 4 pods x 4 workers x (5+10) = 240 exceeds max_connections: 'too many clients' at peak
'bigger pool is faster' 4 pods x 4 workers x (20+20) = 640 exceeds max_connections: 'too many clients' at peak
sized from Little's law 4 pods x 1 worker x (5+5) = 40 OK
autoscaled to 12 pods 12 pods x 1 worker x (5+5) = 120 exceeds max_connections: 'too many clients' at peak
connections busy on average: 600 x 0.012 = 7.2 (spread over all processes)
Size pools from measured database time, not guesses; remember autoscaling multiplies them; put
PgBouncer (transaction pooling) in front of PostgreSQL when many processes share one database; and keep
transactions short, because a connection held during a slow external call is a connection nobody else can use.
The same arithmetic applies to the thread pool (40 threads by default for def endpoints, part
11) and to outbound HTTP pools (httpx.Limits).
6. Finding the hot spot
| Question | Tool |
|---|---|
| Which endpoint is slow, and since when? | RED metrics and traces (part 21): p99 per route, span breakdown |
| Where does a live process spend CPU? | py-spy: py-spy top --pid 1234, py-spy record -o flame.svg --pid 1234. It samples without restarting or changing code. |
| Why is this one request slow? | pyinstrument as a middleware enabled by a header or query flag in staging |
| Is the event loop blocked? | PYTHONASYNCIODEBUG=1 logs callbacks slower than 100 ms; a canary /livez latency metric |
| How much load can we take? | k6, oha or Locust against a production-like environment, ramping until p99 or the error rate breaks the SLO |
Other levers, in rough order of payoff: fix N+1 queries and add indexes (part 15); declare return types so
serialisation runs in Pydantic’s Rust core (part 05, about 5× faster); paginate and stream large
responses (parts 14 and 16); compress at the proxy rather than in Python; and apply backpressure with
uvicorn --limit-concurrency (extra requests get 503 immediately instead of queueing until
everything times out).
Recap
- CPU-bound: add processes (replicas or workers). I/O-bound: async and enough concurrency; Little's law sets the ceiling.
- Reuse clients created in the lifespan; a client per request was 9× slower here.
- Cache with single-flight, jittered TTLs and ETags; share across processes with Redis.
- Do the pool arithmetic across pods × workers × pool; use PgBouncer; measure with real tools and check the load generator.
Get the free RAG production checklist
25 checks to run before real users see your RAG feature: exact-match misses, invented citations, prompt injection through documents, quality that slips after a prompt change. Confirm your email and the PDF comes with the welcome email, plus each new experiment.