Offline & Online Evaluations

Why offline eval is necessary but insufficient

LLMs are non-deterministic — you cannot unit-test "correctness" with fixed assertions alone. Offline evals (golden datasets + scorers) catch regressions before deploy. Online evals (A/B tests, production metrics) catch what lab sets miss: real user phrasing, drift, adversarial usage.

Offline evaluation harness

Golden dataset → Pipeline (prompt v3, model X) → Scorer → Pass/fail + score

Component	Best practice
Dataset	100–500 cases minimum; stratified by intent, edge, adversarial
Scorers	Mix: exact match, regex, LLM-as-judge, human rubric
CI gate	Block deploy if score drops > 2% vs baseline
Versioning	Pin dataset + prompt + model in eval run metadata

Tools: Braintrust, Langfuse, custom pytest + JSON fixtures.

LLM-as-judge pitfalls

Cheap but dangerous:

• Judge favors verbose answers
• Position bias (prefers first option)
• Correlated errors (same model family)

Mitigate: chain-of-thought rubric, swap answer order, calibrate against human labels monthly.

Online evaluation

Method	Measures	Cost
A/B test	Business + quality metrics on live traffic	Risk to cohort B
Shadow traffic	New pipeline, don't show output	2× inference cost
Implicit signals	Thumbs, copy rate, task completion	Noisy
Human review sample	Gold-standard quality	$$$, slow

Ship prompt changes canary → ramp with online quality dashboard — same as feature flags.

What to measure

Product-dependent, but common:

• Task success rate (did user accomplish goal?)
• Citation accuracy (RAG products)
• Edit distance (user changed answer before sending)
• Escalation rate (bot → human)

Avoid optimizing only perplexity or automated scores users don't feel.

Eval-driven development workflow

1. Define failure cases from production incidents → add to golden set
2. Fix prompt/pipeline → offline score improves
3. Canary deploy → online metrics confirm
4. Weekly eval report for stakeholders

Eugene Yan: evaluators are products, not one-off scripts.

Regression when vendor updates model

gpt-4o-2024-08-06 → gpt-4o-2024-11-20 can shift behavior silently.

• Run full golden set on version change
• Pin versions in production config
• Subscribe to vendor changelogs

Interview framing

"How do you test an LLM feature?"

1. Offline golden set with CI gate
2. Online canary with rollback triggers
3. Human review for high-stakes paths
4. Continuous harvest of production failures into dataset

Senior signal: Admit offline/online divergence — "our lab set under-represents multilingual queries; we supplement with stratified online sampling."

Link forward

AI System Observability instruments the metrics that feed online eval loops.