Measure the two halves separately
The single most important production skill is telling a retrieval problem from a generation problem. That's why you measure both axes.
Retrieval metrics (information retrieval):
- Recall@k — fraction of all relevant chunks retrieved in the top-k. Anthropic reports quality as 1 − recall@20 ("retrieval failure rate").
- Precision@k — fraction of retrieved chunks that are relevant.
- MRR — 1 / rank of the first relevant result.
- nDCG@k — rank-aware, graded relevance; the standard for comparing hybrid/rerank configs.
Generation metrics (the RAGAS framework — LLM-graded, reference-free):
- Faithfulness — fraction of the answer's claims supported (entailed) by the retrieved context. The primary hallucination metric.
- Answer / Response Relevancy — is the answer on-topic for the question.
- Context Precision — how much of the retrieved context is actually relevant (signal vs dilution).
- Context Recall — fraction of the info needed to answer that's present in the retrieved context (needs a reference answer). Below 100% = a retrieval gap.
The core diagnostic: read the metric split to route the fix
- Low Context Recall/Precision → retrieval problem → fix chunking, embeddings, hybrid, rerank (Chapters 2–3).
- Good context but low Faithfulness → generation problem → fix grounding/prompt (Chapter 4).
- Good context, faithful, but low Answer Relevancy → task-framing/prompt problem.
Build a golden eval set
Curate (question, ground-truth answer, ground-truth source chunks) triples covering your real query distribution plus edge cases — out-of-corpus questions to test abstention, exact-term queries, multi-hop. You can bootstrap synthetic Q&A from your corpus with an LLM, then human-review. This golden set gates every pipeline change — the same "golden answer" idea Velocode scores you against.