vLLM & Continuous Batching

Why naive serving fails

A naive LLM server processes one request per GPU forward pass. GPUs sit idle while waiting for the next request, and KV cache memory is reserved contiguously per sequence — fragmentation kills throughput when prompts vary from 100 to 8k tokens.

Continuous batching (iteration-level scheduling) adds new requests to an in-flight batch between decode steps instead of waiting for the whole batch to finish.

PagedAttention (vLLM)

Inspired by OS virtual memory: KV cache is stored in fixed-size blocks (e.g., 16 tokens), mapped via a block table per sequence.

Approach	Memory	Flexibility
Contiguous KV	Simple	Wastes RAM on short sequences sharing a max-length buffer
Paged blocks	Higher utilization	Variable lengths, prefix sharing, less fragmentation

Interview signal: "PagedAttention lets us batch requests with different sequence lengths without padding to max context."

Prefill vs decode

Two phases with different compute profiles:

Phase	Work	Bottleneck
Prefill	Process full prompt (parallel over tokens)	Compute (FLOPs)
Decode	One token at a time per sequence	Memory bandwidth (KV read/write)

Schedulers often prioritize or chunk prefill to avoid starving short decode-only requests (head-of-line blocking).

Runtime comparison (names to know)

Engine	Notes
vLLM	PagedAttention, wide model support, Ray distributed
TGI	Hugging Face ecosystem, Rust core, good hub integration
TensorRT-LLM	NVIDIA-optimized, strong on H100 FP8
sglang	RadixAttention prefix sharing, structured output focus

Topic 3 covered when to self-host. Here: how the engine schedules work on silicon.

Link to Topic 3 (Model Serving)

Hosted APIs hide this layer. When you self-host or negotiate dedicated capacity, you own:

• Batch size vs latency trade-off
• Max concurrent sequences (max_num_seqs)
• GPU memory reservation for KV blocks

Link to Concurrency

The inference scheduler is a multi-tenant batch queue — similar mental model to thread pools and work stealing, but the "threads" are GPU kernel launches.

Interview answer template

"Under the hood we'd run vLLM with continuous batching. Long prefills get chunked so decode latency doesn't spike. We autoscale on queue wait time, not CPU — GPU util can be high while p95 blows up if prefill dominates."