From "it works" to "it works affordably at scale"
At volume, cost and reliability become the product. The controls:
- Batching — for non-real-time workloads (bulk classification, offline eval, embedding a corpus), provider batch APIs process large jobs asynchronously at a discount (often ~50%). Use them for anything that doesn't need an instant answer.
- Embedding efficiency — embeddings add up at scale. Use Matryoshka truncation (shorten dimensions) and quantization (lower precision) to cut vector storage and search cost with small quality loss (see the RAG course).
- Inference efficiency (self-hosted) — if you serve your own models, quantization (e.g. 8-bit/4-bit) and batched inference cut GPU cost substantially. Managed APIs handle this for you; it matters when you own the serving.
- Rate limits & quotas — per-user budgets and rate limits (token bucket) prevent a runaway agent loop or an abusive user from running up a huge bill — a cost and security control.
- Guard the loops — bounded agent loops, retry caps with backoff, and circuit breakers stop pathological token burn (Agents/Foundations courses).
Observability for cost
You can't control what you don't see. Track tokens and cost per request, per user, per feature, and per model, with dashboards and alerts on anomalies (a spike = a bug, a loop, or abuse). Attribute cost to features so you know where the money goes and where optimization pays off. Set a budget and alert before you blow it.
The engineering mindset
Cost efficiency isn't a one-time cleanup — it's an ongoing discipline: measure per-request economics, find the expensive tail, apply the right lever (model routing, caching, context trimming, batching), verify quality holds, and monitor continuously. The engineers who ship AI products that survive contact with scale treat cost as a first-class metric alongside latency and quality.