Three families of metrics
1. Reference-based (compare to a ground-truth answer).
- Exact match / accuracy / F1 — for classification, extraction, structured tasks with one right answer. Precise but brittle for free text.
- BLEU / ROUGE — n-gram overlap with a reference; classic for translation/summarization. Cheap but penalizes valid paraphrases — treat as a weak signal for open-ended text.
- Semantic similarity — embed the answer and the reference, compare (cosine). Rewards meaning over wording; better than n-gram overlap for free text, but blind to factual errors that are semantically close.
2. Reference-free / LLM-graded (no ground truth needed). The modern default for open-ended generation — an LLM judges a property directly:
- Faithfulness — are the answer's claims supported by the provided context? (hallucination metric)
- Answer relevance — does the answer address the question?
- Context precision / recall — is the retrieved context relevant / sufficient? (the RAGAS retrieval axis)
3. Task/behavioral metrics.
- Task-completion rate (agents), tool-call success, latency (p50/p95), cost per request, refusal/deflection rate, format-validity rate. These measure the system, not just the text.
Retrieval (IR) metrics — know these by name
When there's a ranked retrieval step: Recall@k (fraction of relevant items in top-k), Precision@k, MRR (1/rank of first relevant), nDCG@k (rank- and grade-aware — the standard for comparing retrieval configs). Anthropic reports quality as 1 − recall@20, i.e. a "retrieval failure rate."
Choosing metrics
Match the metric to what "correct" means. Prefer a small set of metrics you trust over a dashboard of 15 you don't. For open-ended tasks, reference-free LLM-graded metrics + a task metric (latency/cost) usually beat trying to force a single reference answer.