Scaling Latency & Throughput

Inference is two phases

LLM serving latency splits into:

Phase	What happens	Scales with
Prefill	Process entire prompt (parallel attention)	Input token count
Decode	Generate one token at a time (autoregressive)	Output token count

Long prompts hurt prefill; long answers hurt decode. Profiling often shows decode dominates for chat, prefill for RAG with huge context.

Continuous batching

Naive batching waits for N requests — bad for tail latency. Continuous batching (vLLM, TGI) adds/removes requests mid-flight as sequences complete.

Result: higher GPU utilization and better p99 than static batching — but queue depth still spikes under overload.

KV-cache memory

Each sequence stores key-value tensors per layer — memory grows with concurrent sequences × context length. This caps:

• Max batch size
• Max concurrent users per GPU
• Practical context window on self-hosted hardware

OOM kills are production incidents — set admission control before the GPU.

Throughput vs latency trade-off

Knob	Throughput ↑	Latency ↑
Larger batch	✓	p99 worse
Higher GPU util (85%+)	✓	Queueing delay
Smaller model	✓	Quality ↓
Shorter max_tokens	✓	UX ↓

Target 70–80% GPU util with headroom for burst — not 99%.

Autoscaling signals

Scale on:

• Queue depth (best leading indicator)
• Time-in-queue p95
• GPU memory pressure

Don't scale only on CPU — inference is GPU-bound.

Cold start: model load to GPU can take minutes — use warm pools, pre-provisioned nodes, or over-provision during known peaks.

Multi-model routing

At Google-scale products, a router sends:

• Simple queries → small fast model (on-device or edge)
• Complex queries → frontier model

Router itself can be a tiny classifier — latency budget ~10–50ms.

Tail latency debugging

p99 spike checklist:

1. Batch queue backed up?
2. New deploy with longer default context?
3. Vendor rate-limit throttling?
4. Prefill spike from 100k-token prompts?

Use Observability traces with phase=prefill|decode attributes.

Interview framing

"Serve 1000 RPS with 4k context":

1. Estimate tokens/sec per GPU from vendor benchmarks
2. Add 30% headroom
3. Describe admission control when saturated (429, degrade to smaller model)
4. Mention streaming to improve TTFT without shortening total decode

Senior signal: Distinguish capacity (max sustainable RPS) from latency SLO (max RPS while meeting p95) — they're different numbers.

Link forward

Knowledge Pipelines adds retrieval latency before prefill — end-to-end budgets must include the full compound path.