Why averages lie
Service handles 100 req/s: 99 at 50ms, 1 at 5s.
- Average ≈ 99ms — looks fine
- p99 ≈ 5s — users are angry
SLOs are defined on percentiles or success-rate windows, not means. Interviewers will push on this.
Histogram mechanics
Prometheus histograms expose:
http_request_duration_seconds_bucket{le="0.1"} 450
http_request_duration_seconds_bucket{le="0.5"} 490
http_request_duration_seconds_bucket{le="1.0"} 498
http_request_duration_seconds_bucket{le="+Inf"} 500
http_request_duration_seconds_sum 125.3
http_request_duration_seconds_count 500
histogram_quantile(0.99, rate(..._bucket[5m])) estimates p99 from bucket counts.
Cumulative buckets — each le includes all observations ≤ that bound.
Bucket design
Choose buckets for your SLO threshold:
- SLO: 99% of requests < 200ms → buckets around 50ms, 100ms, 200ms, 500ms, 1s, 2s
- Too few buckets → quantile interpolation error
- Too many → marginal benefit, slightly more storage
Default buckets (0.005, 0.01, 0.025, ...) suit many APIs; customize for your latency profile.
Summaries vs histograms
Summary computes quantiles client-side — you can't aggregate across pods.
Histogram aggregates server-side — sum by (le) across replicas gives cluster-wide quantiles.
Always prefer histograms in microservices unless you have a specific reason not to.
Mapping metrics to SLIs
| SLI type | Metric source | Example |
|---|---|---|
| Availability | good events / total | 1 - (5xx rate / total rate) |
| Latency | fraction under threshold | histogram_quantile or native histogram SLO recording rules |
| Throughput | rate counter | sanity check, not usually an SLI |
Google SRE: SLI = proportion of good events in a window. Define "good" from the user's perspective (successful response < 300ms).
Recording rules and SLO metrics
Pre-compute expensive queries:
# Pseudo: ratio of requests under 300ms
- record: checkout:request_latency:p99
expr: histogram_quantile(0.99, sum(rate(...)) by (le))Burn-rate alerts (SLOs topic) consume these pre-aggregated series.
Grafana visualization
- Heatmap — bucket distribution over time
- Stat panel — current p50/p95/p99
- Exemplars — dot on histogram → jump to trace (Production Debugging topic)
Interview pitfalls
- "We alert on average latency" — wrong for tail-sensitive services
- "Summary gives accurate p99 across shards" — false
- "histogram_quantile is exact" — it's interpolated; extreme tails need care
Senior signal
Name your SLO threshold, show bucket boundaries around it, and explain how you'd validate quantile accuracy with exemplar traces on slow requests.
Link forward
SLI & SLO Definition formalizes good events; Burn-Rate Alerting turns these metrics into pages.