Two serving modes
- Batch (offline) — run predictions on a schedule over a dataset, store results (e.g. nightly churn scores in a table). Simple, cheap, high-throughput; not fresh. Use when predictions don't need to be real-time.
- Real-time (online) — a model behind an API returns a prediction per request, low-latency. Use when freshness matters (fraud check at checkout, a recommendation on page load). More infra, tighter latency/scaling needs.
Choose by the freshness requirement: if a prediction can be a day old, batch; if it must reflect the current request, real-time.
Serving infrastructure
A real-time model is a service: package the model (often a container), load it behind an inference server (TorchServe, Triton, BentoML, or a managed endpoint — SageMaker, Vertex), and expose an API. Concerns:
- Scaling — autoscale replicas to traffic; load-balance across them. GPU models are expensive, so scale carefully.
- Latency — batching requests (dynamic batching) improves GPU throughput; quantization speeds inference (see Fine-Tuning course).
- Cold starts — large models are slow to load; keep warm replicas.
- Versioning & rollout — deploy new model versions with canary / blue-green and instant rollback (a bad model is a production incident).
For LLMs specifically
Most teams don't self-host — they call a provider API (OpenAI/Anthropic), so "serving" is API integration + the reliability patterns from Foundations (timeouts, retries, circuit breakers, fallbacks). Self-hosting an open model (vLLM, TGI) makes sense at high volume or for privacy/control, and then GPU serving, batching, and quantization become your problem.
Shadow & A/B deployment
De-risk a new model: shadow deployment (run it alongside prod on live traffic without serving its results) to compare safely, then A/B to measure real impact before full rollout.