The golden set is your source of truth
An eval set is a versioned collection of test cases: at minimum (input, expected_output_or_criteria), and for RAG (question, ground_truth_answer, ground_truth_source_chunks). Every pipeline change is scored against it. A team without one is tuning on vibes.
What to include
- The real query distribution — sample actual (or realistic) user inputs, not just easy cases.
- Edge cases on purpose — out-of-scope questions (to test abstention/"I don't know"), ambiguous queries, exact-term lookups, multi-hop questions, adversarial inputs, and known past failures (every production bug becomes a regression test).
- Coverage across your task's dimensions — categories, difficulties, input lengths.
Size & synthetic generation
You don't need thousands — a focused 50–200 high-quality cases beats 5,000 noisy ones. To bootstrap, generate synthetic Q&A from your own corpus with an LLM (RAGAS and similar can do this), then human-review every item — synthetic sets amplify the generator's blind spots if unreviewed. Grow the set over time by mining production failures.
Keep it honest
- Freeze and version it (a change to the set changes your baseline).
- Don't leak it into training/prompts — a test set the system has "seen" is worthless.
- Include the abstention cases — a system that always answers will confidently hallucinate on out-of-corpus questions; your eval set must reward "I don't know."
This mirrors Velocode's own model: your submissions are scored against a golden answer. Same principle — a trusted reference the system is measured against.