Incident lifecycle
Detect → Triage → Mitigate → Resolve → Postmortem → Follow-up
Detect — alert, customer report, or synthetic monitor. Triage — confirm impact, assign severity, open incident channel. Mitigate — stop the bleeding (rollback, failover, scale, feature flag). Resolve — root cause fixed, SLO recovering. Postmortem — learn without blame.
Speed matters most in mitigate — root cause can wait.
Incident roles
| Role | Responsibility |
|---|---|
| Incident Commander (IC) | Coordinates, makes decisions, shields responders |
| Operations Lead | Executes technical mitigation |
| Communications Lead | Status page, stakeholder updates |
| Subject Matter Experts | Called in as needed |
IC does not debug — they run the room. Critical at FAANG scale.
Communication
- Dedicated channel (
#inc-2026-07-10-checkout) - Regular updates: "Investigating", "Identified", "Monitoring"
- Status page for external users — honest ETAs, not "we're looking into it" for 4 hours
Template: what we know, what we're doing, next update time.
Reconstructing timeline with observability
Build the timeline from data, not memory:
- Alert fired — burn-rate timestamp
- Deploy correlation —
deployment.versionattribute on spans - Trace exemplars — slow/error traces from the window
- Logs filtered by trace_id — error details
- Metrics — when did error rate inflect?
Networking track adds wire evidence (dig for DNS, curl for endpoint health) when app telemetry points to network.
Blameless postmortems
Google SRE culture:
- Focus on system failures, not individual mistakes
- "What allowed this to happen?" not "who broke it?"
- Psychological safety → honest reports → real fixes
Structure:
- Summary (customer impact, duration)
- Timeline (UTC, data-sourced)
- Root cause(s) — often multiple contributing factors
- What went well / what didn't
- Action items with owners and dates
Action items that stick
Bad: "Be more careful", "Add monitoring" (vague). Good: "Add SLO burn alert for payment dependency", "Integration test for retry storm scenario".
Track action items in the same system as sprint work — not a Google Doc graveyard.
Measuring incident health
- MTTR — mean time to restore (mitigation, not root cause)
- MTTD — mean time to detect (alert vs customer report)
- Incident frequency by service
- Repeat incidents — same root cause = process failure
Interview answer
"I'm IC for SEV2+. First action: confirm customer impact and assign roles. Mitigate via rollback before deep debugging. Postmortem within 5 business days, blameless, action items in Jira with owners."
Cross-reference
- Production Debugging — triage workflow and exemplars
- Distributed Systems — cascading failure patterns in root cause analysis
Senior signal
Distinguish mitigation (rollback stopped pain) from resolution (bug fixed). Incidents can close mitigated with follow-up tickets for root cause.