Resilience Patterns

Partial failure becomes total failure without design

Distributed systems don't just fail — they cascade. A slow dependency becomes an exhausted thread pool, retry storms amplify load, and healthy nodes get dragged down. Resilience patterns bound blast radius.

Circuit breaker

Track failure rate to a dependency. States: closed (normal), open (fail fast), half-open (probe).

if failures > threshold: open circuit → return error immediately after cooldown: allow one trial request

Prevents threads blocked on a dead service. Fowler's pattern; implemented in Hystrix, resilience4j, Envoy outlier detection.

Bulkhead

Isolate resource pools per dependency — like ship compartments. A flood in payments API threads must not drain catalog API threads.

Kubernetes: separate deployments + rate limits. Thread pools: dedicated executors per downstream.

Retry budget

Blind exponential backoff isn't enough. Retry budgets cap retry traffic as a fraction of total requests (Google SRE). Prevents a 10% error rate from becoming 200% load via retries.

Rules of thumb:

• Retry only idempotent operations (or with idempotency keys)
• Jitter backoff — sleep = random(0, base * 2^attempt)
• Cap max attempts; surface failure to caller

Load shedding

When overloaded, drop work deliberately (HTTP 503, drop low-priority queue) instead of dying entirely. Prefer shedding at the edge before deep call chains.

Timeout hierarchy

Each layer's timeout must be less than the caller's timeout. If A→B→C, timeout_A > timeout_B > timeout_C. Otherwise parent gives up while child still runs — resource leak.

Interview scenario

"Checkout depends on inventory and fraud. Fraud is slow." — Circuit breaker on fraud path, fallback to async review, bulkhead thread pools, 200ms timeout with cached risk score.

Not in scope

Service mesh config tutorials — know the patterns and where they sit in a call graph.