Tokens are the unit of everything
LLMs don't see words — they see tokens (sub-word chunks). A rough rule: ~4 characters ≈ 1 token, or ~0.75 words per token in English. Code, numbers, and non-English text tokenize less efficiently. Tokens drive three things: cost (you pay per token), latency (more tokens = slower), and the context limit (a hard cap on tokens per request).
How pricing works
Providers charge per token, and input is cheaper than output (output tokens often cost 3–5× input, because generation is sequential and expensive). So a request's cost = (input tokens × input price) + (output tokens × output price). Two levers immediately follow: shrink the input (prompt/context) and shrink the output (don't ask for more than you need).
Model tier matters enormously: a frontier model can be 10–50× the price of a small model. Using a big model for a task a small one handles is the most common source of waste.
The cost drivers to internalize
- Input size — long system prompts, few-shot examples, and (especially) RAG context dominate input cost. Retrieval that stuffs 20 chunks when 4 would do is a silent cost multiplier.
- Output size — verbose responses; asking for explanations you'll discard.
- Number of calls — agents and chains make many calls per task; each is billed.
- Model choice — the single biggest lever.
- Retries & loops — failed calls and unbounded agent loops burn tokens (see the Agents course).
Measure before optimizing
You can't optimize what you don't measure. Log tokens (in/out) and cost per request (Foundations/Evaluation courses). Most teams are shocked which 5% of requests drive most of the bill — usually long-context RAG or agent loops. Find those first.