Before you build anything, you need a map of the terrain. "AI engineering" is not one tool — it is a stack of layers, each solving a specific problem, that together turn a raw model API into a reliable product. This concept gives you that map so the rest of the track has a place to hang each new idea.
Why this appears in interviews
System-design interviews for AI roles almost always start broad: "Design a customer-support assistant over our docs." A strong answer names the layers — model, orchestration, retrieval, evaluation, observability — and explains what goes wrong at each. Candidates who only know "call the OpenAI API" stall the moment the interviewer asks "how do you know it's working?" or "how do you keep costs down?" Knowing the stack lets you reason about any AI product, not just the one you happened to build.
The mental model: model in the middle, scaffolding around it
The model itself is a small part of a production system. Picture concentric layers around it:
- The model layer — the LLM you call.
- The context layer — everything you put into the prompt: retrieval, memory, tools.
- The orchestration layer — the control flow that chains calls, routes, and retries.
- The evaluation layer — how you measure quality offline and in production.
- The observability + serving layer — logging, tracing, caching, cost control, deployment.
Most of an AI engineer's job lives in layers 2–5. The model is a commodity you rent; the value you add is the scaffolding.
Layer 1 — Models and providers
You will choose between hosted APIs (OpenAI, Anthropic, Google) and open-weight models you host yourself (Llama, Mistral, Qwen, often via vLLM, Ollama, or a provider like Together/Fireworks/Bedrock).
The tradeoffs you should be able to state:
- Hosted API — fastest to build, best frontier quality, no infra. Costs scale per token; data leaves your perimeter (though enterprise tiers offer no-training guarantees); you cannot fine-tune freely.
- Self-hosted open model — full control, data stays in your VPC, cheaper at very high volume, fine-tune freely. But you own GPUs, latency tuning, and the quality gap to frontier models.
A common production pattern is model routing: a cheap small model handles easy requests and classification, and only hard requests escalate to an expensive frontier model. This alone can cut cost 5–10x.
Layer 2 — Context: retrieval, memory, and tools
This is where most engineering happens, and it has a name interviewers now use: context engineering — deciding what information to place in the limited context windowContext windowMaximum text an LLM can process at once, in tokens. Exceeding it causes earlier content to be forgotten.Learn more → on each call.
- Retrieval (RAGRAGRetrieval-Augmented Generation — gives LLMs access to external knowledge by retrieving relevant documents before generating a response.Learn more →) — fetch relevant documents from a vector databaseVector databaseDatabase optimised for storing and searching embeddings by similarity using ANN algorithms.Learn more → (Pinecone, Weaviate, Qdrant, pgvector, Chroma) and inject them. The whole of Stage 2.
- Memory — carry conversation history and durable user facts across turns without blowing the context budget (summarize, store, re-inject).
- Tools / function calling — let the model call your code: search, database queries, calculators, external APIs. Increasingly standardized by MCP (Model Context Protocol), an open standard for exposing tools and data to models. The core of Stage 3.
Layer 3 — Orchestration
Real features are rarely a single model call. Orchestration frameworks manage multi-step control flow: chaining, branching, retries, parallel tool calls, and agent loops.
- LangChain / LlamaIndex — batteries-included libraries for RAGRAGRetrieval-Augmented Generation — gives LLMs access to external knowledge by retrieving relevant documents before generating a response.Learn more → and chains.
- LangGraph / CrewAI / the OpenAI AgentsAgent systemsAI systems that take actions, use tools, and complete multi-step tasks by reasoning through a sequence of decisions. SDK — graph- and role-based frameworks for agentsAgent systemsAI systems that take actions, use tools, and complete multi-step tasks by reasoning through a sequence of decisions. that loop, plan, and hand off.
A word of judgment interviewers reward: you do not always need a framework. For a straight RAGRAGRetrieval-Augmented Generation — gives LLMs access to external knowledge by retrieving relevant documents before generating a response.Learn more → endpoint, a few functions and the raw model SDK are clearer and easier to debug than a heavy abstraction. Reach for orchestration frameworks when control flow genuinely branches.
Layer 4 — Evaluation
The layer beginners skip and seniors obsess over. Because LLM output is probabilistic, you cannot ship on vibes. Evaluation splits into:
- Offline evals — a fixed dataset of inputs with known-good outputs (or an LLM-as-judge scoring rubric) that you run on every prompt or model change, exactly like a test suite. Frameworks: RAGAS (for RAGRAGRetrieval-Augmented Generation — gives LLMs access to external knowledge by retrieving relevant documents before generating a response.Learn more →), plus general harnesses.
- Online evals — measuring quality in production via user feedback, judge models on live traffic, and guardrail hit-rates.
If you remember one sentence: evals are the unit tests of AI engineering. No serious system ships without them. Stage 4 covers this deeply.
Layer 5 — Observability, serving, and cost
The operational layer that keeps a live system healthy:
- Observability / tracing — tools like LangSmith, Langfuse, or Arize capture every prompt, retrieved chunk, tool call, token count, latency, and cost, so you can debug why a specific answer went wrong.
- Caching — exact-match and semantic caches cut cost and latency for repeated queries (Stage 3).
- Guardrails — input/output filters for PII, moderation, and prompt-injection (Stage 4).
- Serving + deployment — streaming responses, autoscaling, rate-limit handling, and fallbacks when a provider degrades. This whole discipline is often called LLMOps.
How the layers fail together
The reason to hold the whole stack in your head is that failures cross layers. A user reports "the assistant gave a wrong answer." The cause could be:
- Retrieval (layer 2) returned the wrong chunk,
- Orchestration (layer 3) passed the wrong step's output forward,
- The prompt (layer 2) lacked an "only use context" instruction,
- Or the model (layer 1) simply hallucinated — which only your evals and traces (layers 4–5) can prove.
Debugging AI systems is the skill of knowing which layer to inspect. Every later concept in this track deepens one of these layers; keep this map in mind and each will click into place.
Common interview mistakes
Mistake 1: Equating "AI engineering" with the model. The model is one layer; the differentiated work is context, evaluation, and operations.
Mistake 2: Reaching for a heavy framework by default. Strong candidates justify why they'd use LangGraph vs. plain functions, and often choose the simpler path.
Mistake 3: No evaluation or observability story. "How do you know it works / how would you debug a bad answer?" is a near-guaranteed follow-up. Name the eval and tracing layers.
Mistake 4: Ignoring cost and routing. Frontier-model-for-everything is a red flag; mention small-model routing and caching.
Key vocabulary
- Context engineering — Deliberately choosing what to place in the limited context windowContext windowMaximum text an LLM can process at once, in tokens. Exceeding it causes earlier content to be forgotten.Learn more → each call (retrieval + memory + tools).
- Model routing — Sending easy requests to a cheap model and only hard ones to a frontier model to cut cost.
- Orchestration — Control flow that chains, branches, retries, and loops model + tool calls.
- MCP (Model Context Protocol) — An open standard for exposing tools and data sources to models.
- LLM-as-judge — Using a model to score another model's outputs against a rubric during evaluation.
- Observability / tracing — Capturing prompts, retrievals, tool calls, tokens, latency, and cost to debug behavior.
- LLMOps — The operational discipline of deploying, serving, monitoring, and cost-managing LLM systems.