Speculative Decoding & Prefix Caching

The decode latency problem

Autoregressive generation runs one forward pass per output token. For a 500-token answer, that's 500 serial GPU steps — memory-bandwidth bound, not FLOP bound.

Speculative decoding amortizes this: a small draft model proposes K tokens; the large target model verifies them in one parallel forward pass.

How speculation works

Draft (small): t1, t2, t3, t4 (fast, cheap) Target (large): verify all 4 in one pass Accept prefix where draft matched → skip 3–4 serial steps

Acceptance rate drives speedup. If draft agrees 70% of the time on 4-token blocks, you approach ~2–3× decode throughput.

Factor	Effect
Draft/target distribution match	Higher acceptance
Domain shift (draft not trained on your data)	Low acceptance → overhead
Draft model on same GPU	Memory contention

When not to speculate

• Draft model doesn't fit alongside target on GPU
• Acceptance rate < ~50% — verification cost exceeds savings
• Very short outputs (< 20 tokens) — setup overhead dominates
• Hosted API where vendor already optimizes internally

Prefix caching (KV reuse)

Many production prompts share a static prefix (system prompt, RAG context, tool definitions).

Prefix caching stores KV blocks for the shared prefix so new requests skip recomputing those tokens.

Layer	Who does it
Application	Hash prefix, route to warm replica
Inference engine	vLLM automatic prefix caching, SGLang RadixAttention
Vendor API	OpenAI/Anthropic prompt caching (billing discount)

Link to Topic 7 (Cost & Capacity): prefix caching directly cuts input-token compute cost.

Semantic vs exact cache

Topic 7 covered semantic caching at the response level. Prefix caching is exact — same token sequence, same KV blocks. Different layers of the caching stack.

Interview answer template

"For long system prompts plus RAG context we'd enable prefix caching — the 2k-token context block is identical across users querying the same corpus partition. For decode we'd trial speculative decoding only if eval shows >60% acceptance on our traffic slice."