The idea
Use a strong LLM to grade another model's output against a rubric or a reference. It's the engine behind most reference-free metrics (faithfulness, relevance). Cheaper and faster than human eval, and it correlates reasonably with human judgment — which is why it's the modern default. But it's noisy and biased, so treat it as a calibrated instrument, not ground truth.
Two modes
- Pointwise (absolute) — score a single output on a rubric (e.g. faithfulness 1–5, or pass/fail). Simple; good for tracking a metric over time.
- Pairwise (relative) — given two outputs (A vs B), pick the better one. More reliable than absolute scores (models are better at comparing than assigning calibrated numbers) — the basis of preference eval and A/B tests.
The biases you must control for
- Position bias — judges favor the first (or a specific) option in pairwise. Mitigate by running both orderings and averaging, or randomizing.
- Verbosity bias — judges prefer longer answers. Control by instructing "length is not quality" and/or normalizing.
- Self-preference — a judge favors outputs from its own model family. Use a different, strong model as judge where possible.
- Leniency / narrow score range — pointwise judges cluster scores. Use a clear rubric with explicit criteria and few, well-defined levels.
Making it trustworthy
- Write an explicit rubric — define each score level with concrete criteria; ask the judge to reason then score (chain-of-thought improves reliability).
- Validate against humans — hand-label a sample and check the judge's agreement (correlation / Cohen's kappa). If it doesn't correlate, fix the rubric or the judge model.
- Pin the judge — fix the model + prompt version, because changing the judge silently changes your metric.
- Prefer pairwise for preference decisions; pointwise for tracking a stable property over time.