Module 1 · Thinking in scale

Back-of-the-envelope estimation

Foundations 20 min read Get within 10× in five minutes, then decide

Before anyone draws boxes, a senior engineer asks: how much? How many requests per second, how many bytes per year, how much memory for the cache, how many servers. The goal is not precision. It is to be within a factor of ten, fast, so you can tell a design that needs one database from one that needs fifty. This lesson measures real latencies on a real machine, then builds an estimator for Linkly.

🧳

Packing for a trip

You don't weigh every sock before a trip. You know "a week of clothes fits a carry-on" and "a month does not". Estimation is that kind of knowledge for systems: a few reference numbers, rough multiplication, and the confidence to say "this fits on one machine" or "this obviously doesn't" before building anything.

1. Latency numbers, measured

The famous "latency numbers every programmer should know" are worth feeling for yourself. This block measures common operations on the machine that built this page:

import hashlib, json, os, socket, sqlite3, tempfile, threading, time
from simkit import table, human_time

def timeit(fn, n):
    fn()                                        # warm up
    start = time.perf_counter()
    for _ in range(n):
        fn()
    return (time.perf_counter() - start) / n

lookup = {f"key{i}": i for i in range(100_000)}
payload = os.urandom(1024)
record = {"code": "a1B2c3", "url": "https://example.com/" + "x" * 80, "owner": 42, "clicks": 1234}

db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE links (code TEXT PRIMARY KEY, url TEXT)")
db.executemany("INSERT INTO links VALUES (?, ?)", ((f"c{i}", f"https://example.com/{i}") for i in range(100_000)))
megabyte = bytes(1_000_000)

disk = open(os.path.join(tempfile.mkdtemp(), "data.bin"), "wb", buffering=0)
def write_and_fsync():                           # what a database does to make a commit durable
    disk.write(b"x" * 4096)
    os.fsync(disk.fileno())

server = socket.socket(); server.bind(("127.0.0.1", 0)); server.listen(1)
def echo():
    conn, _ = server.accept()
    while data := conn.recv(64):
        conn.sendall(data)
threading.Thread(target=echo, daemon=True).start()
client = socket.create_connection(server.getsockname())
client.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
def round_trip():
    client.sendall(b"ping")
    client.recv(64)

rows = [[label, human_time(timeit(fn, n))] for label, fn, n in [
    ("dict lookup (incl. Python call overhead)", lambda: lookup["key77777"], 1_000_000),
    ("SHA-256 of 1 KB", lambda: hashlib.sha256(payload).digest(), 100_000),
    ("json.dumps of a small record", lambda: json.dumps(record), 100_000),
    ("SQLite primary-key lookup (in memory)",
     lambda: db.execute("SELECT url FROM links WHERE code = ?", ("c77777",)).fetchone(), 50_000),
    ("copy 1 MB in memory", lambda: bytearray(megabyte), 2_000),
    ("TCP round trip over localhost", round_trip, 5_000),
    ("write 4 KB + fsync to disk", write_and_fsync, 200),
]]
table(rows, ["operation (measured here)", "time each"])
operation (measured here) time each ---------------------------------------- --------- dict lookup (incl. Python call overhead) 73.6 ns SHA-256 of 1 KB 2.23 µs json.dumps of a small record 3.42 µs SQLite primary-key lookup (in memory) 3.28 µs copy 1 MB in memory 223 µs TCP round trip over localhost 52.7 µs write 4 KB + fsync to disk 785 µs
1 ns 10 ns 100 ns 1 µs 10 µs 100 µs 1 ms 10 ms 100 ms 1 s CPU cache~1 ns RAM~100 ns localhost round trip~60 µs (measured) SSD read~100 µs same data centre~0.5 ms round trip disk write + fsync~0.1–5 ms across a continent~40–80 ms around the world~150–250 ms Log scale: each tick is 10× the last. Network distance, not CPU, dominates what users feel.

Memory is cheap to touch

Millions of in-memory operations per second per core. That is why caches work.

Durability costs a millisecond

An fsync is hundreds of times slower than hashing. Databases batch commits (group commit) for exactly this reason.

Distance is king

Light in fibre covers about 200 km per millisecond. A user 10,000 km away pays about 100 ms per round trip before your code runs.

Measured vs typical

The table above was measured on this machine; your laptop will differ by a small factor. The network distances on the diagram are typical published figures, not measurements from here. The ratios are what matter: nanoseconds for memory, microseconds for a local hop, milliseconds for disks and data centres, hundreds of milliseconds across the planet.

2. The estimation toolkit

QuantityFormulaRule of thumb
Average QPSdaily actions ÷ 86,400a day ≈ 105 s; 1M actions/day ≈ 12/s
Peak QPSaverage × peak factor2–3× for global products, 5–10× for local or event-driven ones
Storagerecords/day × bytes × days kept × replicas × (1 + index overhead)3 replicas is common; indexes often add 50–100%
BandwidthQPS × response sizeremember bits vs bytes: 100 MB/s = 800 Mbit/s
Cache memoryhot items × bytes per itema small share of items gets most traffic (lesson 08)
Serverspeak QPS ÷ (per-server capacity × target utilisation), + sparesplan for 50–70% utilisation, plus one server that can fail

3. Estimating Linkly at a million users

import math
from simkit import human, human_bytes

def estimate(users, *, dau_share=0.2, writes_per_dau=2, reads_per_dau=100, peak_factor=5,
             link_bytes=500, replicas=3, years=5, redirect_bytes=600,
             hot_share=0.2, per_server_rps=1_000, target_utilisation=0.6):
    dau = users * dau_share
    writes_s = dau * writes_per_dau / 86_400
    reads_s = dau * reads_per_dau / 86_400
    peak = (reads_s + writes_s) * peak_factor
    links_total = dau * writes_per_dau * 365 * years
    storage = links_total * link_bytes * replicas
    # the hot set: links clicked in a day; assume 20% of the last month's links get clicked daily
    hot_links = dau * writes_per_dau * 30 * hot_share
    servers = math.ceil(peak / (per_server_rps * target_utilisation)) + 1          # +1 spare to survive a failure
    return {
        "average QPS (reads / writes)": f"{reads_s:,.0f} / {writes_s:,.1f}",
        "peak QPS": f"{peak:,.0f}",
        f"links after {years} years": human(links_total),
        f"storage after {years} years (x{replicas} replicas)": human_bytes(storage),
        "egress at peak": f"{human_bytes(peak * redirect_bytes)}/s = {peak * redirect_bytes * 8 / 1e6:,.1f} Mbit/s",
        "cache for the hot set": f"{human(hot_links)} links x {link_bytes} B = {human_bytes(hot_links * link_bytes)}",
        "app servers (60% target, +1 spare)": servers,
    }

for key, value in estimate(1_000_000).items():
    print(f"{key:<40} {value}")

print("\nwhat if the peak factor is 10 and each server only manages 300 req/s?")
print("  app servers:", estimate(1_000_000, peak_factor=10, per_server_rps=300)["app servers (60% target, +1 spare)"])
average QPS (reads / writes) 231 / 4.6 peak QPS 1,181 links after 5 years 730.0M storage after 5 years (x3 replicas) 1.1 TB egress at peak 708.3 KB/s = 5.7 Mbit/s cache for the hot set 2.4M links x 500 B = 1.2 GB app servers (60% target, +1 spare) 3 what if the peak factor is 10 and each server only manages 300 req/s? app servers: 15

Reading the answer

All the links for five years, triple-replicated, fit on one ordinary disk. The hot set fits in a few gigabytes of RAM. Peak traffic needs a handful of servers. Nothing here demands sharding: the design at a million users is about availability, caching and the click log, not raw size.

Sensitivity matters more than precision

Changing two assumptions (peak factor, server capacity) moved the server count from 3 to 15. Always ask which inputs the answer is most sensitive to, and go and measure those. Lesson 04 measures a real server's capacity.

4. Mistakes that wreck estimates

MistakeEffect
Designing for the average, not the peak5–10× too little capacity on the busiest hour
Forgetting replication, backups and indexesstorage 3–6× higher than the "raw" number
Mixing bits and bytesan 8× error in bandwidth, and network bills
Using the average object size when sizes are skeweda few huge objects (videos, big JSON) dominate storage and bandwidth
Ignoring growtha design that fits today and runs out of disk in eight months
False precision ("we need 7.34 servers")wasted time; round, then add headroom

Recap

  • Know the orders of magnitude: ns for memory, µs for local hops, ms for disks and data centres, 100 ms across the planet.
  • QPS = daily actions ÷ ~105; multiply by a peak factor; storage = records × bytes × retention × replicas × index overhead.
  • Write the estimate as code with named assumptions so it can be re-run when an assumption changes.
  • Look for the inputs the answer is sensitive to, and measure those.

Checkpoint

1 · A service does 8.64 million requests a day. Roughly what is its average QPS?
8.64M ÷ 86,400 s = 100 requests per second. A day is about 105 seconds.
2 · Raw data is 1 TB. With 3 replicas and indexes adding 50%, roughly how much disk do you need?
1 TB × 1.5 (indexes) × 3 (replicas) = 4.5 TB, before backups and free space for growth.
3 · Why do databases batch commits (group commit)?
One fsync can cover many transactions, turning a thousand 1 ms flushes into a few.