AI Engineering: Why LLM apps need observability

A classic web service is healthy if it is fast and returns no errors. An LLM application can be fast, error-free and wrong: it answers from the wrong document, ignores a new prompt rule, or quietly costs five times more after a change. Observability means being able to answer “what happened on this request, and is the system getting better or worse?” from the data your system emits. This part explains what to capture and why. The next four build it with OpenTelemetry, Langfuse and Grafana.

1. What is different about LLM applications

2. The signals

3. Averages lie: look at distributions

LLM latency and cost have long tails: a few requests with huge contexts or long agent loops. A simulation of 2,000 requests of a RAG feature, where 2% are multi-step agent runs, shows why dashboards use percentiles and why cost needs a per-request view:

2,000 requests: mean vs percentiles, and who spends the money

import numpy as np

rng = np.random.default_rng(7)
n = 2_000
input_tokens = rng.lognormal(mean=7.6, sigma=0.5, size=n).astype(int)     # median ≈ 2,000
output_tokens = rng.lognormal(mean=5.3, sigma=0.7, size=n).astype(int)    # median ≈ 200
agent = rng.random(n) < 0.02                                             # 2% are agent runs
input_tokens[agent] *= 8
output_tokens[agent] *= 5

latency = 0.35 + input_tokens / 5_000 + output_tokens / 90 + rng.exponential(0.2, n)
cost = (input_tokens * 0.20 + output_tokens * 1.20) / 1e6                  # gpt-5.6-luna prices

p50, p95, p99 = np.percentile(latency, [50, 95, 99])
print(f"latency  mean {latency.mean():.2f}s | p50 {p50:.2f}s  p95 {p95:.2f}s  p99 {p99:.2f}s  max {latency.max():.1f}s")
top = np.sort(cost)[::-1]
print(f"cost     total ${cost.sum():.2f} | the most expensive 1% of requests = {top[:n // 100].sum() / cost.sum():.0%} of spend")
print(f"agent runs are {agent.mean():.0%} of requests and {cost[agent].sum() / cost.sum():.0%} of spend")

Output

latency  mean 4.21s | p50 3.26s  p95 9.52s  p99 20.78s  max 63.5s
cost     total $1.71 | the most expensive 1% of requests = 8% of spend
agent runs are 2% of requests and 14% of spend

The mean hides the users waiting at p99 (six times the median here), and a total cost figure hides that 2% of requests, the agent runs, account for 14% of the spend. Both show up only when every request records its tokens, cost and latency with labels such as feature, model and user tier that let you slice them.

4. Silent quality drift

Here is the failure observability exists for. On day 19 a prompt edit ships. Nothing errors. Latency is unchanged. But the share of answers that cite a valid source drops. With a quality metric and a simple baseline rule, you catch it the same day:

catching a quality drop that no error dashboard shows

import numpy as np

rng = np.random.default_rng(3)
days = 30
http_errors = rng.normal(0.004, 0.001, days).clip(0)          # 0.4% errors, unchanged all month
grounded = rng.normal(0.93, 0.008, days)                      # share of answers with valid citations
grounded[18:] -= 0.07                                         # day 19: prompt v7 goes live

for day in range(7, days):
    window = grounded[day - 7:day]
    threshold = window.mean() - 3 * window.std()
    if grounded[day] < threshold:
        print(f"day {day + 1}: grounded {grounded[day]:.1%} < 7-day baseline {window.mean():.1%} - 3σ  → ALERT")
        break
print(f"HTTP error rate the whole month: {http_errors.min():.2%}–{http_errors.max():.2%} (all green)")

Output

day 19: grounded 86.2% < 7-day baseline 93.1% - 3σ  → ALERT
HTTP error rate the whole month: 0.14%–0.73% (all green)

5. What to capture on every request

6. The tool landscape, and how it fits together

Alternatives exist in every box: Arize Phoenix, LangSmith and Helicone for LLM tracing; Datadog, Honeycomb and New Relic as all-in-one platforms. Because they all accept OpenTelemetry, instrumenting with OTel keeps your options open.

7. Privacy is part of the design

Recap