LLMOps: the same discipline, different artifacts
With hosted LLMs you usually don't train a model — you operate a system around a model API. The MLOps discipline still applies, but the artifacts you version, test, and monitor change:
- Prompts are code — version them, test them against an eval set, gate changes, and roll back. A prompt change is a deploy (Prompt Engineering course).
- Evals replace the metric dashboard — offline eval gates changes, online eval + user feedback catches drift (Evaluation course). Without evals, LLMOps is flying blind.
- The model is a versioned dependency — pin the provider model version; when it updates, re-run your evals (behavior shifts across versions).
- Retrieval/index are operational assets — a stale or poisoned RAG index degrades quality; monitor and refresh it.
What to monitor (the LLM telemetry)
Per request: tokens & cost, latency/TTFT, tool-call success, faithfulness/quality on sampled traffic, refusal/error rates, and user feedback (thumbs, edits, escalations). Trace the full request (LangSmith, Langfuse, Helicone, OpenTelemetry GenAI conventions). Alert on cost spikes (a loop or abuse) and quality regressions.
The feedback loop
The LLMOps loop: ship → monitor quality + cost on live traffic → mine failures → add them to the eval set → improve the prompt/retrieval/model → gate on eval → ship again. Your eval set is the ever-growing asset that makes this loop trustworthy — the same golden-answer principle Velocode scores you against.
Cost & reliability are first-class
Because every call costs money and providers have outages, LLMOps folds in the Cost course (routing, caching, budgets) and the Foundations reliability patterns (timeouts, retries, circuit breakers, fallbacks). "Production-ready LLM app" = evals + observability (incl. cost) + reliability + prompt/version management + a feedback loop.