Disaggregated Inference

Monolithic vs disaggregated serving

Monolithic: one GPU pool runs prefill + decode for every request.

Disaggregated: separate prefill workers and decode workers — optimize each pool for its bottleneck (compute vs memory bandwidth).

Papers to know: DistServe, Splitwise — prefill-heavy and decode-heavy phases scale independently.

Why disaggregate

Workload shape	Dominant phase	Disaggregation benefit
Long RAG context + short answer	Prefill	Scale prefill GPUs without over-provisioning decode
Chat with long multi-turn history	Mixed	Pool sizing per phase
Batch summarization (long docs)	Prefill	Burst prefill without blocking chat decode

Goodput (tokens delivered per GPU-second at SLO) improves when you right-size each phase instead of one pool sized for worst-case combined load.

Architecture sketch

Request → Router → Prefill pool (compute-heavy, batch long prompts) ↓ KV transfer Decode pool (bandwidth-heavy, many concurrent short decodes) ↓ Response

KV transfer between pools is the engineering cost — network bandwidth, serialization, colocation in same AZ.

Dedicated embedding and rerank tiers

Disaggregation isn't only prefill/decode. At RAG scale (Topic 4), teams often run:

Tier	Model type	SLA
Embedding	Bi-encoder (e.g., e5, voyage)	High QPS, low latency, cheap GPUs
Rerank	Cross-encoder	Lower QPS, runs on top-k only
Generation	LLM	Highest cost, strict p95

Separating embedders from LLMs prevents embedding batch jobs from starving chat latency.

Link to Topic 4 (Knowledge Pipelines) — ingestion uses embedders offline; query path uses them online. Same model family, different scaling profile.

Link to Distributed Systems

Disaggregated serving is microservices for GPU phases — load balancing, backpressure, and partial failure between prefill and decode pools.

When to stay monolithic

• Low traffic — ops complexity isn't worth it
• Small models on single GPU — no phase bottleneck split
• Hosted API — vendor handles internally

Interview answer template

"Our RAG workload has 4k-token prefill bursts but 100-token answers — I'd trial disaggregated prefill/decode pools. Embedders on a separate CPU/GPU tier with autoscale on embedding queue depth. LLM pool autoscales on decode wait time."