Model Rollouts & Feature Flags

AI deploys are config deploys

Changing a prompt is a production deploy — same blast radius as code. Model version bumps, system prompt edits, retrieval top_k, and temperature defaults all need:

• Version control (git, not a dashboard-only edit)
• CI eval gate
• Canary rollout
• Rollback in < 5 minutes

Version everything

Artifact	Version as
System prompt	prompts/support_v14.txt + hash
Model ID	Pinned string, not latest
RAG index	index_alias → physical_index_v7
Policy rules	Config with changelog

Store resolved config in trace attributes for post-incident replay.

Rollout patterns

Canary by cohort

1% internal → 5% new users → 25% → 100%

Gate on: offline eval pass, error rate, quality score, cost/session.

Shadow mode

Run new pipeline parallel; log diff; don't show user. Costs 2× — use for high-risk changes only.

Blue/green prompts

Two active versions; load balancer or feature flag routes traffic. Instant rollback = flip flag.

LaunchDarkly / internal flag systems — same patterns as Distributed Systems feature flags.

Rollback triggers

Automated rollback when:

• Eval score drops > X% vs baseline (1-hour window)
• Error rate > 2× normal
• Policy block rate anomaly
• p95 latency > SLO for 10 min

Human override always available — false positive rollbacks happen.

Prompt registry

Central service:

• Lists approved prompt versions per feature
• Enforces eval CI status before promotion
• Audit log: who promoted what when

Prevents "engineer hot-fixed prod prompt in Langfuse UI" untracked changes.

Model vendor rollouts

When vendor deprecates a model:

1. Run golden eval on replacement
2. Dual-run shadow comparison for 1 week
3. Ramp with feature flag
4. Keep old version callable for 30-day rollback window

Interview framing

"Ship a new system prompt to 10M users":

1. PR with diff + offline eval results
2. Canary 1% with quality/latency/cost dashboards
3. Auto-rollback criteria documented
4. On-call notified; runbook linked

Senior signal: Treat prompt injection regression tests as part of deploy gate — new prompt may be more jailbreakable.

Link forward

Privacy & Compliance covers contractual and regulatory constraints on what you can log, retain, and send to vendors during rollouts.