Provider rate limits
LLM APIs cap you on RPM (requests/min) and TPM (tokens/min), per model and tier. Exceed them → 429 Too Many Requests (with a Retry-After). At any real scale you will hit limits, so design for them:
- Client-side throttling — a token bucket / concurrency limit to stay under RPM/TPM.
- Exponential backoff + jitter on 429/5xx (Foundations reliability) — the official SDKs do this; configure the caps.
- Request larger tiers as volume grows, and batch non-urgent work.
Error handling
Know the categories:
- 4xx (your fault) — 400 bad request (malformed), 401 auth, 429 rate limit, context-length errors. Fix the request; don't blindly retry a 400.
- 5xx / network (their fault or transient) — 500/502/503, timeouts. Retry with backoff.
- Content/refusal — the model declines or returns a safety refusal; handle gracefully.
Wrap every call in a timeout (LLM calls are slow and variable — never wait forever) and distinguish retryable (5xx, 429, network) from non-retryable (400, 401) errors.
Reliability patterns (from Foundations, applied)
- Timeouts + retries (backoff, idempotent only) + circuit breaker around every provider call.
- Fallbacks / graceful degradation — on provider outage, fall back to a secondary model/provider, a cached result, or a "try again shortly" message + queue. Providers will have incidents.
- Idempotency — for anything that mutates state off an LLM call, use an idempotency key so a retry can't double-execute.
Quotas & cost control
Add per-user rate limits/quotas (matched to plan) — an authenticated but abusive user or a runaway agent loop can otherwise run up a huge bill (a cost and security control; OWASP LLM10). Track token usage per user and alert on anomalies.