Did the fine-tune actually help?
Fine-tuning without evaluation is how teams ship a model that's different but not better. Before and after tuning, measure on a held-out test set the model never trained on:
- Compare the fine-tuned model against the baseline (the un-tuned model with good prompting) on the metric that matters for your task. If it doesn't clearly beat prompting, don't ship it — you added cost and lock-in for nothing.
- Check for regressions — a model tuned for one behavior often gets worse at others (catastrophic forgetting). Test general capability, not just the target task.
Data curation is the real work
Most fine-tune quality comes from the dataset, so evaluate the data too: dedupe, fix label errors, ensure diversity and coverage, check for leakage (test examples sneaking into training inflates scores), and balance classes. A small, clean, well-covered dataset beats a large noisy one.
Common pitfalls
- Overfitting to the fine-tune set — great on your examples, worse on real inputs. Hold out a test set; use early stopping.
- No baseline comparison — "it works" without checking it beats prompting.
- Catastrophic forgetting — narrow tuning degrades general ability. Test broadly, or mix in general data.
- Fine-tuning for facts — it won't reliably learn them; you'll get confident wrong answers.
- Not re-evaluating after a base-model upgrade — a better base model may beat your fine-tune, making it obsolete.
The loop
Curate data → fine-tune → evaluate on held-out data vs baseline → check regressions → ship only if it clearly wins → monitor in production and re-evaluate when the base model improves. Same evaluation discipline as everything else — a fine-tune is a model change, gated on evals.