GPU Parallelism & Fleet Design

When one GPU isn't enough

Models exceed single-GPU VRAM (70B FP16 ≈ 140GB). Even when they fit, throughput may require replication or sharding.

Three parallelism modes (names interviewers expect):

Mode	Splits	Communication	Typical use
Tensor Parallel (TP)	Layers across GPUs on one node	All-reduce every layer	Low latency, single-node 70B
Pipeline Parallel (PP)	Layers in stages across GPUs	Activations between stages	Very deep models, multi-node
Data Parallel (DP)	Full model copies	None during inference	Throughput scaling

Inference usually combines TP within a replica + DP across replicas behind a load balancer.

Fleet sizing (whiteboard math)

Inputs: model size, context length distribution, target RPS, p95 latency budget.

Rough steps:

1. Profile tokens/sec per GPU at your context percentiles (not average — long tails matter)
2. Account for batching — continuous batching improves tokens/sec but adds queue wait
3. Add replicas until queue wait + generation time < p95 SLO at peak RPS
4. Headroom — 30–40% spare for bursts and rolling deploys

Autoscale signals (ranked):

1. Queue depth / wait time — user-visible
2. Time-per-output-token (TPOT) degradation
3. GPU utilization — necessary but not sufficient (can be 95% and still missing SLO)

Multi-tenant inference clusters

Platform teams run shared GPU pools serving multiple products:

• Namespace isolation — quotas per team, noisy-neighbor protection
• Priority classes — interactive chat > batch embedding jobs
• Preemption — batch jobs yield to latency-sensitive prefill

Link to Distributed Systems: this is queueing + load balancing + capacity planning at datacenter scale.

MIG and fractional GPUs

NVIDIA MIG splits one A100 into isolated instances — useful for small models (classifiers, embedders) but not for large LLM TP groups that need full GPU memory bandwidth.

Link to Topic 3

Topic 3 compared hosted vs self-hosted. This subtopic is for the ML platform engineer designing the cluster behind self-hosted or hybrid paths.

Interview answer template

"70B at 2k context — I'd start TP=4 on one node per replica, benchmark tokens/sec, then horizontally scale replicas. Autoscale on p95 queue wait, not GPU %. Keep one warm standby replica per AZ for failover."