Failure Modes & Degradation

LLM-specific failure modes

Failure	Symptom	User impact
Timeout	Partial stream, 504	Broken UX, duplicate retries
Rate limit	429 burst	Feature unavailable
Hallucination	Confident wrong answer	Trust damage, liability
Retrieval miss	Answer without grounding	Fabricated citations
Tool failure	Side effect partial	Data inconsistency
Policy block	Empty or generic refusal	Frustration if over-tuned

Traditional circuit breakers (Distributed Systems) apply to vendor APIs — open circuit → degradation ladder, not infinite retry.

Degradation ladders

Define explicit rungs before incident:

Rung 0: Full AI (retrieve + frontier model) Rung 1: Smaller/faster model (quality ↓, latency ↓) Rung 2: Cached / templated responses for FAQ intents Rung 3: Non-AI fallback (search results, static help) Rung 4: Graceful "temporarily unavailable"

Each rung needs:

• Trigger (latency SLO breach, error rate, cost cap)
• User-visible copy (honest, not "an error occurred")
• Metric (degradation_level on traces)

Google SRE overload handling: shed load ** deliberately**, don't serve garbage slowly.

Bulkheads

Isolate failure domains:

• Separate rate-limit pools per feature/tenant
• Dedicated model capacity for payment-critical flows
• Retrieval timeout independent of generation timeout

One viral feature shouldn't exhaust tokens for core product.

Retry discipline

Call type	Retry?
Read-only completion	Yes, with backoff + idempotency key
Tool with side effects	No blind retry — use idempotent tool design
Streaming	Retry only if no bytes sent

Retry storms during vendor outage make it worse — coordinate with global circuit breaker.

Handling wrong answers

You can't circuit-break hallucinations. Mitigations:

• Confidence thresholds → "I'm not sure"
• Citation-required mode — show sources or refuse
• User feedback loop → eval regression

Quality incidents are SLO violations — treat like latency incidents.

Chaos and game days

Test:

• Vendor 100% 503 for 10 minutes
• Retrieval index empty
• 10× latency injection on model
• Policy service down

Verify ladder activates and on-call runbook matches reality.

Interview framing

"Model API is down — what happens?"

1. Circuit opens after N failures
2. Serve rung 3 fallback within 1s
3. Alert on-call; status page if user-facing
4. Post-incident: did we retry into bankruptcy?

Senior signal: Mention duplicate submission — users double-click send; idempotency keys prevent double charges on tool calls.

Link forward

Evaluation & Quality measures whether degradation rungs still meet minimum quality bars.