Shipping the service
Separate environments (dev / staging / prod), each with isolated config and secrets — the 12-Factor rule: config in the environment, never point staging at prod data. CI/CD runs tests/lint/typecheck/build on every push (GitHub Actions) and promotes green builds through environments, with rollbacks and, for risky changes, canary/blue-green or feature flags. Docker packages app + deps into a reproducible image; a PaaS (Vercel/Fly) or orchestrator (Kubernetes/ECS) runs it — noting serverless's cold-start and connection-limit caveats.
Reliability patterns (the core)
- Timeouts — every network/LLM call gets a bounded deadline. Never wait forever; LLM calls are slow, so set generous-but-finite timeouts and stream to mask latency.
- Retries with exponential backoff + jitter — on transient failures (
429,502/503/504, network) retry with delaysbase·2^nplus random jitter to avoid synchronized retry storms. Cap attempts and total time. Only retry idempotent operations — which is why idempotency keys matter. - Idempotency — make operations safe to repeat (idempotency keys for writes, dedupe on event IDs). This is the through-line from Chapter 1 (APIs) and Chapter 2 (webhooks).
- Circuit breaker — after a threshold of failures to a dependency, open the circuit and fail fast; after a cooldown go half-open to test recovery, then close. States: closed → open → half-open.
- Rate limiting — token-bucket/sliding-window per user; return
429+Retry-After. Also respect providers' RPM/TPM limits with client-side throttling. - Graceful degradation — when a dependency fails, degrade instead of erroring: fall back to a cheaper/secondary model, serve cached/partial results, or queue for later.
Observability
Three pillars: logs (structured events), metrics (latency p50/p95/p99, error rate, throughput — the "four golden signals"), (a request's path across services, via ). For , also track domain telemetry: , and eval scores (Langfuse, LangSmith, Helicone, OTel GenAI conventions).