LLM Evaluation Interview Questions: What AI Companies Actually Ask (2026)
Real LLM evaluation interview questions for AI engineers in 2026: reference-free metrics, LLM-as-a-judge bias, faithfulness vs answer relevance, golden sets, and offline vs online eval. With what strong answers look like.
For a while, "how do you evaluate an LLM system?" was a question you could skate past with a mention of accuracy and a wave at BLEU. That era is over. In 2026, frontier models saturate the public benchmarks, so knowing a model's MMLU score tells an interviewer almost nothing about whether you can prove your system works on your task. Evaluation has quietly become the thing that separates engineers who ship LLM features from engineers who ship LLM features that silently regress.
This guide covers the LLM evaluation questions that are actually being asked, organized from fundamentals to metrics to LLM-as-a-judge to production. For each one, here is what a strong answer looks like and what most candidates get wrong.
Why Evaluation Is Now a Core Interview Topic
Evaluation gets tested heavily because it is where LLM engineering stops being a demo and starts being a discipline. Anyone can prompt a model and eyeball the output. The hard part, the part companies actually pay for, is a repeatable and trustworthy signal that tells you whether a change made the system better or worse before your users find out.
That signal is hard to build because LLM outputs are open-ended. There is rarely one correct answer, the failure modes are subtle, and the obvious metrics lie. Interviewers push on evaluation because a candidate's answers reveal, very quickly, whether they have ever had to defend a "this is better, ship it" decision with real evidence. The good news is that the questions are learnable, because good evaluation rests on a small set of concepts and traps.
Fundamentals Questions
"Your team wants to measure the quality of a summarization feature. Why not just use accuracy?"
What most candidates say: Compute accuracy against a reference summary, or have someone check if the summaries are right.
What strong candidates say: Accuracy assumes a single correct output, and that does not exist for open-ended generation. Two summaries can share almost no words and both be excellent, or share most of their words and differ on the one fact that matters. The right move is to break "quality" into properties you can measure separately: faithfulness (are the claims supported by the source?), coverage (did it capture the key points?), and a constraint like length or format. You measure each one with a method that fits it, instead of collapsing everything into a single number that means nothing.
"Explain the difference between reference-based and reference-free evaluation. When would you reach for each?"
What most candidates say: Reference-based compares to a correct answer and reference-free does not. Reference-based is more reliable.
What strong candidates say: Reference-based metrics compare output to a ground-truth answer: exact match and F1 for extraction or classification, BLEU and ROUGE for translation and summarization, embedding similarity for meaning-level comparison. They are precise when a correct answer genuinely exists, and brittle when many outputs are equally valid. Reference-free evaluation (usually LLM-graded) judges a property of the output directly, with no gold answer: faithfulness, answer relevance, tone, format validity. For open-ended generation, reference-free has become the default, because writing a single gold answer for every input is both expensive and misleading. The senior instinct is to match the method to whether "correct" is singular or plural for the task in front of you.
Metrics Questions
"Your summarizer scores poorly on ROUGE, but the summaries read great. What is happening?"
What most candidates say: The summaries must have a subtle quality problem the eye is missing.
What strong candidates say: This is a metric mismatch, not a quality problem. ROUGE measures n-gram overlap with a reference summary, so a summary that is genuinely good but phrased differently from your reference scores low. BLEU and ROUGE systematically punish valid paraphrase, which is exactly what a good abstractive summarizer produces. The fix is to switch to metrics that reward meaning over wording: an LLM-graded faithfulness check (are all claims supported by the source doc?), a coverage check, and a length constraint. Treat n-gram overlap as, at best, a weak signal for free-text tasks.
"A candidate lists faithfulness and answer relevance as if they are interchangeable. Why is that a mistake?"
What most candidates say: They both measure whether the answer is good.
What strong candidates say: They measure different things and can move in opposite directions. Faithfulness asks whether the answer's claims are grounded in the provided context, so it is your hallucination metric. Answer relevance asks whether the answer actually addresses the question the user asked. A response can be perfectly faithful (every claim traceable to the context) while quietly answering a different question than the one posed, which scores high on faithfulness and low on relevance. The reverse also happens: a fluent answer to the right question can be entirely fabricated. You need both, plus context precision and recall on the retrieval side for RAG. The candidates who conflate them have usually never watched a "0.9 faithfulness, users still unhappy" dashboard in production.
"When you have a ranked retrieval step, which metrics do you report and why?"
What most candidates say: Precision and recall.
What strong candidates say: For ranked retrieval you want rank-aware metrics. Recall@k is the fraction of relevant items in the top k, precision@k is the inverse, and MRR captures how high the first relevant result lands. Lead with nDCG@k, because it is both rank-aware and grade-aware, which makes it the standard for comparing retrieval or reranking configurations. Some teams report retrieval quality as a failure rate instead, for example one minus recall@20, because "how often did we miss" tends to focus a team better than a raw score. Naming these without prompting signals that you have tuned a retriever, not just read about one.
LLM-as-a-Judge Questions
"You want to use an LLM to grade another model's outputs at scale. What are the biases you have to control for?"
What most candidates say: The judge might just be wrong sometimes.
What strong candidates say: LLM-as-a-judge is the engine behind most reference-free metrics, and it correlates reasonably with humans. Reported agreement with human reviewers commonly lands around 85%, which is higher than two humans often agree with each other on the same task. But it is a calibrated instrument, not ground truth, and it has measurable, named biases. Position bias: judges favor the first option in pairwise comparisons, so run both orderings and average. Verbosity bias: judges prefer longer answers, so tell the judge that length is not quality. Self-preference bias: a judge favors its own model family, so use a different strong model as the judge where you can. Leniency, or narrow-range bias: pointwise judges cluster their scores, which a clear rubric with a few well-defined levels helps fix. A candidate who can name these has run judge evals. One who says "it might hallucinate" has not.
"Pointwise or pairwise judging: which do you trust more, and for what?"
What most candidates say: Pointwise, because a numeric score is more precise.
What strong candidates say: Models are more reliable at comparing two outputs than at assigning a calibrated absolute number, so pairwise (A vs B, pick the better) is the stronger tool for preference decisions and A/B comparisons. Pointwise (score 1 to 5 on a rubric) is better for tracking a single stable property over time, like faithfulness across releases. Use both deliberately: pairwise when you are choosing between candidates, pointwise when you are monitoring a metric. In either mode, ask the judge to reason before it scores, because chain-of-thought grading is measurably more reliable than a bare number. Then validate against a human-labeled sample and pin the judge model and prompt, because quietly changing the judge quietly changes your metric.
Building Eval Sets Questions
"How do you build the dataset you evaluate against when you have no labeled data?"
What most candidates say: Write a bunch of test questions the team can think of.
What strong candidates say: A golden set is your source of truth: a versioned collection of (input, expected output or criteria) cases, and for RAG (question, ground-truth answer, ground-truth source chunks). The mistake to avoid is filling it with happy-path questions written from memory, because that set will pass while production fails. Sample the real query distribution from your logs, then deliberately add edge cases: out-of-scope questions that test whether the system abstains and says "I don't know," ambiguous and multi-hop queries, and every past production failure as a regression case. To bootstrap with no labels, generate synthetic Q&A from your own corpus with an LLM, but human-review every item, because synthetic sets amplify the generator's blind spots. A focused, reviewed set of 50 to 200 cases beats 5,000 noisy ones, and you freeze and version it so a change to the set does not quietly move your baseline. And never leak the eval set into prompts or training, or it is worthless.
Production Evaluation Questions
"A prompt change passes your offline eval suite, ships, and support tickets rise. What happened, and what should have caught it?"
What most candidates say: The offline eval must have been run incorrectly.
What strong candidates say: A frozen golden set cannot see everything real users do, or how the world drifts. Almost always, a query type spiked after the change, maybe a marketing push or a new user segment, that the offline set did not represent, so faithfulness for that segment cratered in production while the offline number still looked fine. What catches this is online evaluation: score a sample of live traffic with the same judge you validated offline, watch the trend per segment, and pair it with cheap human signals like thumbs up/down, edits, and escalations. The loop is offline gate, ship, online monitor, mine failures, add them to the golden set, repeat. You sample rather than score every request because running a judge over all your traffic is expensive, so you reserve full golden-set runs for release gates.
"How would you stop a quality regression from ever reaching production twice?"
What most candidates say: Test more carefully before shipping.
What strong candidates say: You make evaluation a gate, not a ritual. Every PR that touches a prompt, a model version, or a retrieval config triggers an eval run against the golden set in CI, and a change that regresses past a threshold does not merge. It is a test suite for quality. When a novel failure escapes to production and online monitoring catches it, that failure becomes a new golden-set entry, so CI now blocks its return. That feedback loop, where production discovers new failure modes, the golden set absorbs them, and CI prevents recurrence, is what makes a system permanently more robust instead of endless whack-a-mole. For model or prompt swaps, confirm the offline win holds with a live A/B test on real business outcomes rather than shipping on the offline score alone.
The One Thing Candidates Get Wrong About Evaluation Interviews
The most common mistake is treating evaluation as a metrics quiz, memorizing that BLEU is n-gram overlap and faithfulness is a hallucination metric, and stopping there.
Every evaluation question is really a judgment question in disguise. "Why not accuracy?" asks whether you understand that quality is plural. "Faithfulness versus answer relevance" asks whether you have watched a good-looking dashboard hide a broken product. "Offline versus online" asks whether you know a frozen test set cannot see the future. The engineers who get offers are not the ones who recite the RAGAS metric list. They are the ones who can say "here is the scenario where the metric looks great and the system is still broken, and here is how I would catch it."
That judgment comes from building an eval that failed you, not from reading about metrics in the abstract. If you want to practice exactly that, we have scored problems that put you in those seats: building a lightweight eval harness for an agent, resolving contradictory answers from conflicting sources, and diagnosing a support agent that got slow and expensive after launch. There is also a full LLM Evaluation course that walks from metrics to LLM-as-judge to production eval, and the LLM Evaluation topic hub with every question in one place.