Hosted vs Self-Hosted Serving

The serving decision tree

Option	Best for	Ops burden	Latency floor
Shared API (OpenAI, Anthropic, Google)	Fast ship, frontier quality	Low	Vendor-dependent
Dedicated / provisioned throughput	Predictable capacity, SLA	Medium	Better tails at scale
Self-hosted (vLLM, TGI)	Data residency, cost at huge volume, custom models	High	You own the stack

Most FAANG-adjacent products start hosted and only self-host when unit economics or compliance force it.

Hosted APIs: what you get and don't

Get: latest weights, safety tuning, global CDN, billing per token Don't get: weight-level control, guaranteed capacity on shared tiers, sub-100ms tails under load

Production practices:

• Retry with exponential backoff on 429/503 (respect Retry-After)
• Multiple API keys / regions for blast-radius isolation
• Pin model version strings in config — silent vendor updates happen

Self-hosted: vLLM and TGI

vLLM — PagedAttention, continuous batching, high GPU utilization TGI (Hugging Face) — Rust runtime, good for HuggingFace model hub integration

You operate:

• GPU fleet provisioning and autoscaling
• Model artifact storage and version rollout
• Health checks, load balancing, queue management
• CUDA/driver hell

Break-even is often hundreds of millions of tokens/month — do the math before buying GPUs.

Hybrid architectures

Common at scale:

• Frontier model for hard queries (routing classifier decides)
• Small self-hosted model for classification, extraction, routing
• Vendor API fallback when self-hosted queue depth exceeds SLO

RouteLLM and similar routers optimize cost-quality Pareto fronts (Cost topic).

Data residency and compliance

EU healthcare, financial services, government — may require:

• VPC/private endpoints to vendor
• Self-hosted in-region
• Zero-retention API contracts

Legal picks the topology; engineering implements air gaps.

Interview framing

"When would you self-host?"

1. Quantify monthly token spend vs GPU TCO
2. Name compliance constraints
3. Acknowledge ops cost (SRE headcount, on-call for GPU nodes)
4. Propose hybrid if uncertain

Senior signal: Don't reflexively say "we'll run vLLM" — say "I'd model cost at 12-month forecast traffic and include a 6-week ops ramp."

Link forward

Scaling Latency & Throughput covers what happens inside the serving tier once you've chosen where it runs.