Why LLM apps need observability

A classic web service is healthy if it is fast and returns no errors. An LLM application can be fast, error-free and wrong: it answers from the wrong document, ignores a new prompt rule, or quietly costs five times more after a change. Observability means being able to answer "what happened on this request, and is the system getting better or worse?" from the data your system emits. This lesson explains what to capture and why. The next four build it with OpenTelemetry, Langfuse and Grafana.

1. What is different about LLM applications

2. The signals

3. Averages lie: look at distributions

LLM latency and cost have long tails: a few requests with huge contexts or long agent loops. A simulation of 2,000 requests of a RAG feature, where 2% are multi-step agent runs, shows why dashboards use percentiles and why cost needs a per-request view:

import numpy as np

rng = np.random.default_rng(7)
n = 2_000
input_tokens = rng.lognormal(mean=7.6, sigma=0.5, size=n).astype(int)     # median ≈ 2,000
output_tokens = rng.lognormal(mean=5.3, sigma=0.7, size=n).astype(int)    # median ≈ 200
agent = rng.random(n) < 0.02                                             # 2% are agent runs
input_tokens[agent] *= 8
output_tokens[agent] *= 5

latency = 0.35 + input_tokens / 5_000 + output_tokens / 90 + rng.exponential(0.2, n)
cost = (input_tokens * 0.20 + output_tokens * 1.20) / 1e6                  # gpt-5.6-luna prices

p50, p95, p99 = np.percentile(latency, [50, 95, 99])
print(f"latency  mean {latency.mean():.2f}s | p50 {p50:.2f}s  p95 {p95:.2f}s  p99 {p99:.2f}s  max {latency.max():.1f}s")
top = np.sort(cost)[::-1]
print(f"cost     total ${cost.sum():.2f} | the most expensive 1% of requests = {top[:n // 100].sum() / cost.sum():.0%} of spend")
print(f"agent runs are {agent.mean():.0%} of requests and {cost[agent].sum() / cost.sum():.0%} of spend")

The mean hides the users waiting at p99 (six times the median here), and a total cost figure hides that 2% of requests, the agent runs, account for 14% of the spend. Both show up only when every request records its tokens, cost and latency with labels such as feature, model and user tier that let you slice them.

4. Silent quality drift

Here is the failure observability exists for. On day 19 a prompt edit ships. Nothing errors. Latency is unchanged. But the share of answers that cite a valid source drops. With a quality metric and a simple baseline rule, you catch it the same day:

import numpy as np

rng = np.random.default_rng(3)
days = 30
http_errors = rng.normal(0.004, 0.001, days).clip(0)          # 0.4% errors, unchanged all month
grounded = rng.normal(0.93, 0.008, days)                      # share of answers with valid citations
grounded[18:] -= 0.07                                         # day 19: prompt v7 goes live

for day in range(7, days):
    window = grounded[day - 7:day]
    threshold = window.mean() - 3 * window.std()
    if grounded[day] < threshold:
        print(f"day {day + 1}: grounded {grounded[day]:.1%} < 7-day baseline {window.mean():.1%} - 3σ  → ALERT")
        break
print(f"HTTP error rate the whole month: {http_errors.min():.2%}–{http_errors.max():.2%} (all green)")

5. What to capture on every request

6. The tool landscape, and how it fits together

Alternatives exist in every box: Arize Phoenix, LangSmith and Helicone for LLM tracing; Datadog, Honeycomb and New Relic as all-in-one platforms. Because they all accept OpenTelemetry, instrumenting with OTel keeps your options open.

7. Privacy is part of the design

Recap