Right-size and route your models
Not every request needs your best model. The highest-leverage cost/latency decision is model routing:
- Match the model to the task — a small/fast model for classification, extraction, routing, and simple Q&A; a frontier model only for the hard reasoning. Measure quality per task (Evaluation course) to find the smallest model that passes.
- Cascade / router pattern — try a cheap model first; escalate to a stronger one only when the cheap one is uncertain or fails a check. Handles the easy majority cheaply, reserves spend for the hard minority.
- Task-specific small models — sometimes a fine-tuned small model (or an embedding classifier) beats a general frontier model at a narrow task, far cheaper (see the Fine-Tuning course).
Caching layers
- Prompt caching (provider-side) — reuse a stable prefix; up to ~90% input savings (covered in Ch. 2).
- Exact-match response cache — Redis in front of the model; identical request → cached answer, no model call. Set TTLs.
- Semantic cache — cache keyed by embedding similarity of the prompt, so a near-duplicate question hits the cache. Big savings for FAQ-like traffic; tune the similarity threshold so you don't return a wrong-but-similar answer.
Combine them
A mature setup layers all of it: semantic cache first (skip the model on repeats), then a router (cheap model for easy, frontier for hard), with prompt caching on the static prefix and reranked/minimal context. Each layer compounds.
Quality is the constraint
Every optimization trades against quality — a cheaper model, fewer chunks, an aggressive cache can all degrade output. So optimize against your eval set: cut cost/latency, confirm quality holds, keep the win. Blind cost-cutting that tanks quality is a false economy.