You can't improve what you don't measure
LLM systems are non-deterministic and fail in ways unit tests don't catch — a subtly wrong answer that looks confident. Evaluation is how you turn "it feels better" into a number you can defend. Without it, every prompt tweak or model swap is a guess.
Offline vs online eval
- Offline eval runs your system against a fixed golden dataset (question → expected answer / expected behavior) before shipping. It gates changes — a prompt edit that drops your score doesn't merge. Fast, repeatable, cheap.
- Online eval measures live production traffic: sample real requests and score them (quality, faithfulness, user feedback). Catches what your golden set missed and detects drift over time.
You need both: offline to gate changes, online to catch reality.
The most important skill: locate the failure
For any LLM system with retrieval or tools, a bad output is either a retrieval/context problem (the system didn't get the right information) or a generation problem (it had the right information and still answered wrong). Great evaluation separates these axes so you know which half to fix. This "read the metric split to route the fix" skill is the single most valuable thing an interviewer probes.
What "good" means is task-specific
There's no universal metric. Define success per task: a summarizer cares about faithfulness + coverage; a classifier cares about accuracy/F1; a RAG bot cares about faithfulness + answer relevance + context recall; an agent cares about task-completion rate. Start by writing down, in one sentence, what a correct output looks like — then pick metrics that measure that.