The pattern
Retrieval-Augmented Generation (RAG) does three things at query time: retrieve relevant chunks from an external knowledge base via search, augment the prompt by injecting those chunks as context, and generate a grounded answer. The model's weights never change — the knowledge lives outside the model, in an index you control.
The four problems RAG solves
- Grounding — answers are tied to source documents, which reduces hallucination and lets you cite sources.
- Freshness — you update the index, not the weights, so knowledge stays current.
- Private data — you can inject proprietary data the base model never saw.
- Cost & scale — far cheaper than fine-tuning, and it scales beyond what could ever fit in a single prompt.
The mental model that wins interviews
Chip Huyen's framing in AI Engineering is the one to memorize: "RAG is for facts; fine-tuning is for form." If the failure is information-based (missing or outdated knowledge), reach for RAG. If it's behavior-based (won't follow instructions, wrong tone or format), that's a fine-tuning or prompting problem — more retrieved context won't fix it.
The honest decision framework
The recommended progression is prompt engineering → few-shot examples → RAG → fine-tuning as a last resort. But the most important recent shift is around long context:
- If your knowledge base is under ~200k tokens (~500 pages), skip RAG. Put the whole corpus in the prompt and use prompt caching — Anthropic reports that cuts latency more than 2× and cost up to 90%. For small, static corpora, long-context + caching is now the simpler, often more accurate default.
- But long context does not kill RAG. A model that can take a long context can't necessarily use all of it well — it's slower, costs more per query, and suffers "lost in the middle" (Chapter 4). Data grows faster than context windows, so RAG stays necessary at scale.
Quick heuristic: large or changing corpus → ; small static corpus → ; wrong behavior/format → ; needs multi-step iterative lookup → . These aren't mutually exclusive — "fine-tune for form RAG for facts" is common.