Compound AI Pipelines

Beyond single-shot completion

Berkeley's compound AI systems thesis: frontier quality comes from orchestrating models, retrievers, tools, and verifiers — not from one bigger model call. Production stacks are pipelines, not prompts.

Input → Classify → Retrieve → Rerank → Generate → Verify → Format → Output (LLM) (vector) (cross) (LLM) (rules) (template)

Each stage is a failure domain with its own latency and quality profile.

Orchestration patterns

Pattern	Description	Trade-off
Sequential	Stage N waits for N-1	Simple; latency sums
Parallel fan-out	Retrieve + classify concurrently	Lower latency; merge complexity
Branching	Router picks sub-pipeline	Cost-efficient; router errors costly
Loop	Agent re-calls until done	Powerful; runaway token risk

Use AI Engineering for agent patterns — here focus on production guarantees: timeouts per stage, partial results, and idempotent tool calls.

Deterministic vs probabilistic stages

Stage type	Examples	Testing
Deterministic	JSON schema validation, PII regex, template render	Unit tests
Probabilistic	Classification, generation, reranking	Eval suites

Put deterministic checks after generation when possible — catch format errors before users see them.

Partial failure handling

What if retrieval returns zero hits but the model still answers?

• Fail closed — "I don't have enough context" (support bots)
• Fail open with disclaimer — answer + low-confidence badge (search)
• Escalate — human handoff (high-stakes domains)

Document the policy per product — "always answer" causes hallucinations at scale.

Idempotency and side effects

Tool calls (send email, create ticket, charge card) inside pipelines need Distributed Systems discipline:

• Idempotency keys on every tool invocation
• Saga compensation if step 4 fails after step 3 succeeded
• Never let the LLM choose arbitrary SQL — parameterized tools only

Observability per stage

Emit spans: pipeline.classify, pipeline.retrieve, pipeline.generate. Attributes:

• retrieval.hit_count, rerank.top_score
• model.name, tokens.in, tokens.out
• verify.passed (boolean)

Enables "quality dropped 5% — reranker regression or model change?" triage.

Caching in pipelines

• Cache retrieval results by query hash (short TTL)
• Cache embeddings of corpus docs (invalidate on update)
• Avoid caching final answers unless semantic dedup is acceptable (Cost topic)

Interview framing

"Design a code-review bot pipeline":

1. Parse diff (deterministic) → retrieve style guide (RAG) → generate comments (LLM) → filter policy violations → post to API
2. Name failure at each stage and fallback
3. Estimate latency budget per stage

Senior signal: Propose a quality gate — if verifier fails, retry once with constrained prompt, then escalate to human; don't infinite-loop the agent.

Link forward

Model Serving addresses the bottleneck stage (usually generate); Knowledge Pipelines owns retrieve at scale.