AI System Observability

Observability beyond HTTP 200

Generic Observability covers SLIs, traces, and dashboards. AI systems need request-scoped LLM telemetry:

• Which model, prompt version, retrieval hits
• Token counts and dollar cost
• Quality scores and policy outcomes
• End-to-end trace across compound pipeline stages

Without this, you cannot debug "answers got worse Tuesday" or "bill doubled overnight."

Trace schema (recommended)

trace: ai.completion ├─ span: retrieve (hit_count, top_score, latency_ms) ├─ span: policy.input (blocked: bool, reason) ├─ span: model (model_id, tokens_in, tokens_out, ttft_ms, decode_ms) ├─ span: policy.output └─ span: tools.* (tool_name, success, latency_ms)

Correlate with deployment, prompt_version, tenant_id — cardinality managed via sampling (see Observability track).

Metrics that matter

Metric	Type	Alert on
llm.tokens.input	Counter	Cost anomaly
llm.latency.ttft	Histogram	p95 regression
llm.request.errors	Counter	Vendor outage
rag.retrieval.hit_rate	Gauge	Index drift
policy.block_rate	Gauge	Spike (attack or misconfig)
eval.quality_score	Gauge	Week-over-week drop

RED method still applies — Rate, Errors, Duration — plus cost and quality dimensions.

Logging prompts safely

Full prompt logging helps debug but risks PII/secrets.

Patterns:

• Hash + truncate for default logs
• Full payload only in restricted retention tier
• Opt-in debug mode per session with consent

Langfuse/Arize: purpose-built for LLM trace UI with redaction hooks.

Quality drift detection

Automated monitors:

• Sample 1% traffic → async LLM-judge or human label
• Compare weekly score distribution
• Alert if KL divergence vs baseline

Embedding drift: monitor average retrieval scores on canonical query set.

Cost attribution

Tag every span with feature, team, tenant. Export to FinOps dashboard:

• Cost per DAU per feature
• Top 10 expensive prompt templates
• Anomaly: single tenant 50× normal tokens (loop bug?)

Incident triage playbook

1. Latency — prefill vs decode split; queue depth; vendor status
2. Errors — 429 vs 5xx; circuit breaker state
3. Quality — recent prompt deploy? model version? retrieval freshness?
4. Cost — new feature flag? missing max_tokens?

Interview framing

"Production got worse after deploy — how do you debug?"

1. Correlate quality metric with deploy timeline
2. Diff prompt version and config
3. Slice traces by tenant/cohort
4. Run offline eval on suspect version vs baseline

Senior signal: Propose exemplar traces linked from dashboards — same pattern as exemplars in Prometheus, but for bad answers.

Link forward

Cost & Capacity turns token metrics into budgets and optimization levers.