Sync, Async & Streaming Request Paths

Three ways users wait for AI

Path	UX	When to use	Risks
Sync (blocking)	Spinner until full response	Short answers, low variance latency	Timeouts, bad tail latency
Streaming (SSE)	Tokens appear incrementally	Chat, copilot, long generation	Client complexity, partial failure
Async (job queue)	Poll or push when done	Batch, heavy RAG, multi-doc	Stale status, duplicate jobs

The choice is a product + SLO decision, not a framework default.

Sync path

POST /complete → orchestrator blocks → model → response

• Set hard timeouts (client + server) — models hang
• Use connection pooling to vendor APIs (see Networking)
• Return structured errors: rate limit, policy block, timeout — not generic 500

Sync works when p95 E2E < 3s and users expect a single atomic answer.

Streaming path

Server-Sent Events (SSE) or WebSocket deliver token chunks. Martin Kleppmann's streaming insight: treat the UI as a materialized view of an event stream — render partial state, finalize on done.

Production concerns:

• Backpressure — slow clients shouldn't block GPU decode
• Reconnect — mid-stream failure needs resume or graceful abort UX
• Tracing — one trace ID spans chunks; don't log every token
• Billing — charge on completed tokens, handle client disconnect

Streaming improves perceived latency even when total time is unchanged.

Async path

POST /jobs → 202 + job_id → worker pipeline → webhook/poll → result

Use when:

• Input exceeds sync timeout (100-page PDF)
• Pipeline has 5+ stages with minutes of wall time
• You need retry without user waiting

Patterns from Distributed Systems:

• Idempotent job_id (client-supplied UUID)
• Exactly-once side effects via outbox/saga
• Dead-letter queue for poison prompts

Hybrid: "sync with escape hatch"

Start sync; if retrieval or model exceeds 2s, flip to async job and notify. Google products often use this for search generative answers.

Timeout cascade

Client timeout (30s) > Gateway (25s) > Orchestrator (20s) > Model (15s) > Retrieval (3s)

Each layer must fail fast and return actionable errors. Never let the client hang longer than the server.

Interview framing

"Design autocomplete vs document Q&A":

• Autocomplete: streaming, 200ms TTFT budget, cancel in-flight on keystroke
• Doc Q&A: async job, email/push on complete, progress events optional

Senior signal: Mention cancellation — abort upstream model calls when the user navigates away; otherwise you pay for unused tokens.

Link forward

Compound AI Pipelines composes multiple stages across these paths — retrieval blocking inside a streaming response is a common pattern.