What makes LLM calls slow
- Time to first token (TTFT) — the wait before the first token appears; driven by prompt processing (input length) + queueing.
- Time per output token — generation is sequential, so long outputs are slow. Output length is often the dominant latency factor.
- Model size — bigger models are slower per token.
- Serial dependencies — chains and agents that call the model/tools one after another stack their latencies.
The levers
- Stream the response. Streaming (SSE) doesn't make total time shorter but slashes perceived latency — the user sees tokens immediately (low TTFT) instead of staring at a spinner. Every chat UI streams for this reason.
- Shrink input (faster TTFT) and shrink output (fewer sequential tokens) — the same trims that cut cost cut latency.
- Parallelize independent work. If two retrievals or two tool calls don't depend on each other, run them concurrently instead of serially — this is the #1 fix for slow agents that call tools one at a time.
- Use a smaller/faster model where quality allows — often a cheaper model is also markedly faster.
- Prompt caching cuts TTFT by skipping reprocessing of the cached prefix.
- Cache results (standard + semantic) so repeat/near-repeat requests skip the model entirely.
Design for variable latency
LLM latency is high-variance — the same call can take 1s or 15s. Design around it: set timeouts, stream, show progress, and for slow multi-step work, do it async (kick off a job, return "processing," notify when done) rather than blocking a request. Never leave a user on a spinner with no feedback.
Measure p95/p99, not the average
A good median hides a painful tail. Track p95/p99 latency and TTFT — the tail is what users actually feel and what times out.