Offline eval isn't enough
A frozen golden set can't see what real users do or how the world drifts. Production evaluation closes the loop:
- Online eval / sampling — score a sample of live traffic on your key metrics (faithfulness, relevance, task success) with the same LLM-judge you validated offline. Watch the trend, not just point values.
- Human feedback signals — thumbs up/down, edits, escalations, thumbs-to-copy — cheap, high-signal labels. Feed disagreements back into the golden set.
- Guardrail/quality checks inline — a fast faithfulness or safety pass before returning the answer (block or flag unsupported claims) for high-stakes apps.
Monitoring & drift
Log per request: inputs, retrieved context, tools used, tokens, cost, latency (p50/p95/p99), citations, and the eval scores. Track drift — new query types your system wasn't built for, a stale index, or a model/provider change silently shifting behavior. Alert on metric regressions, not just errors.
Regression gating & experimentation
- CI eval gate — run the golden set on every change; block merges that drop the score (like a test suite for quality).
- A/B tests — for prompt/model changes, split live traffic and compare metrics + business outcomes (conversion, resolution) — pairwise LLM-judge + user signals. Don't ship on offline score alone; confirm it holds in production.
- Cost of eval — LLM-judge over large samples is expensive. Sample for continuous monitoring; run the full set only at release gates.
The mindset
Evaluation is not a one-time gate — it's a continuous loop: offline gate → ship → online monitor → mine failures → grow the golden set → repeat. Teams that win treat their eval set as a first-class, ever-growing asset.