Bulkheads, Circuit Breakers & Chaos

Bulkheads

Isolate resources so failure in one pool doesn't drain all:

• Separate thread pools per downstream
• Separate connection pools per dependency
• Cell-based architecture — shard fleet by tenant/region (Amazon/Stripe pattern)

Circuit breaker states

Closed → (failures exceed threshold) → Open → (timeout) → Half-open → probe

Open — fail fast, don't hammer sick dependency.

Implement in: app library, service mesh, API gateway (per route).

Coordinate policies — triple circuit breaking causes false positives.

Timeout budgets

Total user deadline = sum of child calls (serial) or max (parallel). Propagate grpc-timeout / Deadline context.

Rule: child timeout < parent remaining budget.

Communication covers deadline propagation; Concurrency covers thread pool exhaustion.

Cascading failure prevention

1. Timeouts everywhere
2. Bulkheads
3. Circuit breakers
4. Load shedding (503 + Retry-After)
5. Graceful degradation — serve cached/stale data

Chaos engineering

Hypothesis: system survives X failure.

Experiments: kill random pods, inject latency, partition network, drain AZ.

Game day prerequisites: observability, runbooks, abort button, off-hours.

Tools: Chaos Mesh, Litmus, AWS FIS, Gremlin.

Distributed Systems link

Resilience patterns in Distributed Deep Cuts overlap — this topic focuses on platform operations and fleet experiments, not algorithmic consensus.

Interview scenario

"Payment provider down" — answer: circuit open, queue async retries, show degraded checkout message, alert on-call, don't block catalog browsing (bulkhead product flows).