Don't hard-couple to one provider
Models and prices change monthly, and providers have outages, so production LLM integrations abstract the provider behind your own interface (a thin llm.complete() layer — Velocode's @/lib/llm.ts pattern). Benefits: swap models/providers without touching feature code, add fallbacks, and centralize retries, logging, and cost tracking in one place.
Model routing & fallbacks
- Routing — send each request to the right model by task/difficulty (cheap model for simple, frontier for hard) — the Cost course's biggest lever, implemented at the integration layer.
- Fallback chain — if the primary provider errors or times out, fall back to a secondary model/provider so an outage degrades quality instead of breaking the feature. (Confirm the fallback's prompt behaves — prompts don't transfer perfectly across models; re-eval.)
- Gateways — LLM gateways/proxies (e.g. LiteLLM, provider-agnostic layers) give one interface across providers with built-in routing, retries, caching, and spend tracking.
Production checklist for an LLM integration
- Keys server-side, per-environment, rotated (Foundations/AI Security).
- Timeouts + retries (backoff) + circuit breaker + fallback on every call.
- Rate limits/quotas per user; global spend budget + alerts.
- Caching — prompt + semantic + exact (Cost course).
- Observability — log tokens, cost, latency/TTFT, model, and outcome per request; trace it (Langfuse/OTel).
- Evals gating prompt/model changes (Evaluation course).
- Pinned model versions; re-eval on upgrade.
- Structured outputs + validation for machine-consumed responses.
The mindset
A production LLM integration isn't "call the API" — it's a resilient, observable, cost-controlled, provider-abstracted layer with evals and fallbacks. That layer is what separates a demo from a product that survives real traffic, price changes, and provider incidents.