Alert Design & Fatigue

The on-call UX problem

Alerts are notifications to sleepy humans. Bad alerts cause:

• Fatigue — ignored pages, slow response
• Burnout — unsustainable rotations
• Blame culture — fear of deploying

Google SRE: optimize for on-call quality of life, not coverage metrics.

Symptom vs cause

Type	Example	Page?
Symptom	Checkout SLO burn rate 14.4×	Yes
Symptom	p99 latency > 2s for 10 min	Maybe (if SLO-linked)
Cause	Single pod CPU > 90%	No (investigate via dashboard)
Cause	Disk 80% full	Ticket (unless imminently critical)

Page on user pain, investigate causes via dashboards and traces.

The three filters (Google SRE)

Every page must be:

1. Actionable — on-call can do something (rollback, scale, failover)
2. Urgent — can't wait until morning
3. Novel — not a known flaky monitor (fix or delete it)

"If I got this at 3am, would I thank the alert?" — if no, fix or remove.

Severity tiers

Tier	Response	Example
SEV1	Immediate all-hands	Complete outage, data loss
SEV2	On-call + escalation	Major feature degraded
SEV3	Business hours	Minor degradation, workaround exists
SEV4	Backlog	Cosmetic, tech debt

Tie severity to customer impact, not internal inconvenience.

Measuring resilience patterns

Circuit breakers, bulkheads, and retry budgets (Distributed Systems topic) need observability, not reimplementation here:

circuit_breaker_state{service="payment", downstream="stripe"} # 0=closed, 1=open, 2=half-open retry_attempts_total{service="payment", result="exhausted"}

Page when SLO burns, not when breaker opens once (that might be correct behavior). Dashboard breaker state during triage.

Alert routing and noise reduction

• Ownership — service team gets their SLO alerts
• Inhibition — parent SLO alert suppresses child dependency noise
• Maintenance windows — silence deploy-related expected blips (carefully)
• Aggregation — one alert per incident, not per replica

Runbooks

Every page links to a runbook:

1. What this alert means
2. First checks (dashboard links, recent deploys)
3. Escalation path
4. Known false positives

Runbook in the alert annotation — not a wiki link nobody can find at 3am.

Interview answer

"We page only on SLO burn-rate and explicit capacity emergencies. Cause-level alerts go to Slack. Every monitor has an owner and quarterly audit — delete what we ignore."

Link forward

Incident Response & Postmortems covers what happens after the page fires.