The Production AI Systems Lens

What "production AI" means

A production AI system is not a prompt in a playground. It is a distributed system where one component — the LLM — is probabilistic, expensive, slow, and externally hosted. Everything else (gateway, retrieval, policy, cache, UI) must be engineered assuming the model will occasionally lie, timeout, or cost 10× on a bad prompt.

Chip Huyen's framing: treat AI features like data products with a feedback loop, not one-shot integrations.

The production stack (typical)

Client → API Gateway → Orchestrator → [Retrieve | Tools | Policy] → Model API → Post-process → Client ↓ Traces, eval logs, cost attribution

Layer	Role	Failure mode if ignored
Gateway	Auth, rate limits, routing	Cost explosion, abuse
Orchestrator	Multi-step pipeline, retries	Partial answers, duplicate side effects
Retrieval	Grounding context	Hallucinations at scale
Policy	Input/output safety	Incidents, compliance breach
Model	Generation	Latency spikes, vendor outage

Assumes AI Engineering covered prompting and tool schemas. Here the question is where each box lives and what SLO it inherits.

Probabilistic core vs deterministic shell

Traditional services: same input → same output (modulo bugs). LLM services: same input → distribution of outputs. Implications:

• Testing needs eval suites, not only unit tests
• Idempotency matters when tools have side effects
• Caching is approximate (semantic, not exact)
• Rollbacks are prompt/version rollbacks, not binary deploys

Senior signal: say "I'd wrap the model in a deterministic validation layer" not "I'd fine-tune until it's perfect."

What sibling tracks already cover

Track	You use it for
AI Engineering	Prompting, RAG chunking basics, MCP tools
Distributed Systems	Queues, sagas, circuit breakers, idempotency
Databases	Corpus storage, metadata, transactional updates
Observability	Generic SLOs, tracing, alerting
Networking	SSE streaming, timeouts, connection pooling

This track sits above those: AI-specific trade-offs at the intersection.

The data flywheel

Production systems improve from signals:

1. Implicit — thumbs down, edit-before-send, abandonment
2. Explicit — human review queues, CSAT
3. Automated — eval regression on golden sets

Without instrumentation, you're flying blind on quality drift when the vendor silently updates weights.

Interview framing

"Design an AI feature for 10M DAU":

1. Draw the pipeline boxes and label sync vs async
2. Name the degradation path when the model is down
3. State cost per session and p95 latency targets
4. Describe how you'd detect quality regression

Link forward

Non-Functional Requirements turns this lens into measurable SLIs — latency, quality, cost, safety — the table stakeholders actually negotiate.