Why static alerts fail
error_rate > 1% for 5 minutes:
- Too sensitive during low traffic (1 error = 100%)
- Too slow during high traffic (budget already gone)
- Doesn't account for SLO target (99.9% vs 99%)
Burn-rate alerting pages when you're consuming error budget too fast — proportional to SLO severity.
Burn rate formula
burn_rate = (current_error_rate) / (1 - SLO_target)
For 99.9% SLO (0.1% error budget):
- Error rate 0.1% → burn rate 1× (on track)
- Error rate 1% → burn rate 10× (budget gone in 3 days instead of 30)
- Error rate 14.4% → burn rate 144× (budget gone in ~2 hours)
Multi-window, multi-burn-rate (Google SRE)
| Alert | Window | Burn rate | Meaning |
|---|---|---|---|
| Fast page | 1h | 14.4× | Critical — budget exhausted in hours |
| Slow page | 6h | 6× | Serious sustained degradation |
| Ticket | 3d | 1× | Budget tracking — investigate |
Fast window catches sudden outages; slow window catches gradual leaks. Both must fire for high-confidence pages (configurable).
PromQL pattern (conceptual)
# Error ratio over window
1 - (
sum(rate(http_requests_total{status!~"5.."}[1h]))
/ sum(rate(http_requests_total[1h]))
)
# Compare to SLO target and compute burn rate
# (actual implementation uses recording rules — see SRE Workbook)Use recording rules for slo:good_events and slo:total_events — don't compute raw ratios on every alert evaluation.
Paging vs ticketing
| Severity | Criteria | Action |
|---|---|---|
| Page | Fast burn, user-facing SLO | Wake on-call |
| Ticket | Slow burn, internal SLO, non-customer | Next business day |
| Log/dashboard | Informational trend | No notification |
Google SRE rule: every page must be actionable. If on-call can't do something, it's not a page.
Alert fatigue antipatterns
- Paging on causes (single pod CPU) instead of symptoms (SLO burn)
- Duplicate alerts (same incident, 5 monitors)
- Flapping thresholds without hysteresis
- "Warning" pages that train on-call to ignore phones
Inhibition and routing
When SLO burn fires:
- Inhibit lower-severity dependency alerts (DB CPU) — you're already responding
- Route to service owner, not platform team (unless platform owns SLO)
Interview answer
"We alert on multi-window burn rates against our 99.95% latency SLO. 14.4× over 1h pages; 6× over 6h pages with different severity. Static error-rate alerts were removed — they didn't correlate with budget consumption."
Cross-reference
- Alert Design & Fatigue — human factors of on-call
- Metrics topic — histogram SLI recording rules feed burn math
Link forward
Incident Response covers what on-call does when the burn-rate page fires.