Dashboards, Exemplars & Triage

The debugging hierarchy

When p99 spikes or errors climb, triage in order:

1. User impact — which SLO, which cohort, how many users? 2. Recent changes — deploys, feature flags, config, traffic shift 3. Golden signals — RED on affected service slice 4. Dependencies — downstream error rate, saturation 5. Traces — exemplars or tail-sampled errors 6. Logs — trace_id filter for stack traces 7. Runtime — profiling, eBPF if still unexplained

Don't start at step 7. Senior engineers resist "let me strace prod" before checking deploy history.

Dashboard design

Hierarchy:

• L0 — company/product SLOs (executive)
• L1 — service RED + saturation (on-call landing)
• L2 — dependency breakdown, per-endpoint, per-AZ
• L3 — JVM/runtime, queue depths, custom business metrics

USE-method panels for nodes; RED for services. Template variables: service, env, version.

Anti-pattern: 47-panel dashboard nobody can read during an incident. Prefer linked dashboards with drill-down.

Exemplars: metrics → traces

Prometheus histograms can attach exemplars — sample trace IDs on specific observations:

http_request_duration_seconds_bucket{le="0.5"} 450 # exemplar: trace_id=abc123 latency=0.48s

Grafana: click heatmap dot → open Tempo/Jaeger trace.

This closes the loop: "p99 is bad" → "here's a real slow request" without querying all traces.

Requires: OTel SDK exemplar support, Mimir/Prometheus exemplar storage, Grafana 7.4+.

Triage shortcuts

Symptom	First look
Single AZ hot	AZ label on RED metrics; networking partition?
Post-deploy spike	version label; rollback candidate
One dependency red	Service map / trace dependency view
Gradual climb	Saturation (queue depth, connection pool)
Sudden cliff	Deploy, cert expiry, DNS (Networking: dig)

Measuring resilience during triage

From Distributed Systems resilience patterns — observe, don't reimplement:

• circuit_breaker_state — is payment breaker open?
• retry_attempts_total — retry storm amplifying load?
• bulkhead_rejected_total — thread pool isolation kicking in?

These explain why errors cluster without reading application code under pressure.

High-cardinality debugging

Charity Majors' approach: ask arbitrary questions of production:

• "Errors for payment_provider=stripe in region=eu-west-1 during deploy v2.3.1"

Requires: trace/log backends with rich attributes — not pre-built dashboards alone.

Anti-patterns

• Dashboard rot — monitors nobody owns
• Alert-only debugging — no dashboards for investigation
• Log grep as first step — too slow at scale; metrics first
• Blaming the network without evidence — use Networking track tools to confirm

Interview workflow

"SLO burn page fires → I open L1 dashboard, confirm checkout service, check deploy in last 30 min, spike in payment dependency p99, exemplar trace shows Stripe timeout, logs confirm connection reset, rollback mitigates."

Link forward

Profiling & eBPF covers when traces and logs aren't enough — CPU hot spots and kernel-level visibility.