Quantization for Inference

Why quantize

FP16/BF16 weights are memory-heavy. Quantization reduces bit width → fit larger models on fewer GPUs, higher batch size, lower cost per token.

This is inference compression, not training — aligns with our scope (we're not pretraining transformers).

Formats to name

Method	Idea	Trade-off
INT8 / INT4 (GPTQ, AWQ)	Post-training weight quantization	2–4× memory savings; possible quality loss on reasoning
AWQ	Protect salient weights using activation stats	Better quality than naive INT4
FP8	Hardware-native on H100/Hopper	Good speed + quality on supported kernels
BF16 baseline	No quantization	Best quality, highest cost

Rule of thumb: quantize smaller models and high-volume classification/extraction first; be cautious quantizing frontier reasoning models without rigorous eval regression.

Quality validation (non-negotiable)

Before production cutover:

1. Golden-set eval — same suite as Topic 6 (offline evals)
2. Slice analysis — legal, code, multilingual subsets often degrade differently
3. Latency A/B — confirm speedup is real on your context lengths (sometimes dequant overhead eats gains on short prompts)

Link to Topic 6 (Evaluation & Quality) — quantization is a deployment change that needs eval gates like prompt changes.

Vendor vs self-hosted quantization

Path	Notes
Hosted API	Vendor picks precision; you pick model tier
Self-hosted	You export AWQ/GPTQ artifacts, load in vLLM/TensorRT-LLM
Custom distillation	Train small model to mimic large — different technique, higher effort

Distillation is a Deep Cuts adjacent topic: train a 7B student on teacher outputs. Mention in interviews but it's more ML-research-leaning than AWQ deployment.

Link to Topic 7 (Cost & Capacity)

Quantization is often the first lever after semantic caching when GPU cost dominates unit economics.

Interview answer template

"We'd trial AWQ INT4 on the 13B summarizer — expect ~2× throughput. Gate on offline F1 within 1% of FP16 and a human spot-check on 200 legal docs before routing 10% canary traffic."