Input is where the tokens hide
For most production apps, input dwarfs output — a long system prompt + few-shot examples + retrieved context, sent on every call. Trimming input is the highest-ROI cost work.
Techniques
- Trim the system prompt — remove redundancy; state rules once. Long boilerplate repeated every call adds up fast.
- Right-size few-shot — 3 well-chosen examples usually match 10; each example is pure input cost on every request.
- Retrieve less, better — the biggest RAG cost lever: rerank and send the top few chunks, not top-20. Precision beats recall for both quality (lost-in-the-middle) and cost. Contextual/parent-child retrieval lets smaller chunks carry more signal.
- Summarize/compress context — for long histories or documents, summarize older turns instead of resending the full transcript. Conversation memory should be managed, not append-only.
- Concise output contract — ask for exactly the format you need ("one sentence," a JSON field), not a discursive essay you'll trim.
Prompt caching (the big one)
Providers cache a stable prompt prefix server-side so repeated calls don't re-pay for it. If your system prompt + few-shot + a fixed corpus are identical across requests, caching them can cut input cost up to ~90% and latency 2×+ (Anthropic). Structure prompts with the static part first (system, examples, fixed context) and the variable part last (the user's question) so the prefix is cacheable. This is also why, for small corpora, "long context + prompt caching" can beat RAG (see the RAG course).
Watch conversation growth
In chat/agent apps, context grows every turn — old messages, tool calls, results all accumulate. Left unmanaged it silently balloons cost and latency and eventually overflows the window. Summarize, truncate, and keep only what the next turn needs.