An AI Engineer builds production systems that use large language models and other AI capabilities to solve real problems. You are a software engineer whose primary building block is a model you call rather than train — and whose hardest problems are reliability, evaluation, cost, and context, not model architecture.
Why this appears in interviews
Almost every AI engineering interview starts here. Interviewers want to calibrate whether you understand the shape of the job. The most common failure mode is candidates who describe themselves as ML researchers — answering "how would you improve this?" with "I'd retrain the model" — when the role is about shipping AI-powered products around a model someone else trained. Getting this distinction right signals domain clarity before you've answered a single technical question.
The mental model — three concentric circles
Data Scientist — innermost circle. Builds models, runs experiments, cares about statistical significance. Often does not ship to production. Measures success in accuracy percentages.
ML Engineer — middle circle. Trains and deploys models: data pipelines, feature engineering, training jobs, serving latency, retraining, model drift. The output is a model. Measures success in throughput, uptime, and offline metrics.
AI Engineer — outermost circle. Does not train foundation models. Takes models that already exist (GPT-4o, Claude, Llama) and builds applications on top of them: RAGRAGRetrieval-Augmented Generation — gives LLMs access to external knowledge by retrieving relevant documents before generating a response.Learn more → pipelines, prompt design, agentsAgent systemsAI systems that take actions, use tools, and complete multi-step tasks by reasoning through a sequence of decisions., evaluation, guardrails. The output is a product feature. Measures success in user outcomes.
The critical difference: AI Engineers are software engineers who happen to work with LLMs. They are not researchers. You use a model the way a backend engineer uses a database — you need a deep working model of how it behaves, but you did not build it.
What an AI engineer actually does
Day to day, the work is:
- Wiring models into products — turning "call GPT-4o" into a feature that is fast, reliable, observable, and cheap enough to run at scale.
- Context engineering — deciding what to put into the model's limited context windowContext windowMaximum text an LLM can process at once, in tokens. Exceeding it causes earlier content to be forgotten.Learn more → each call: retrieved documents (RAGRAGRetrieval-Augmented Generation — gives LLMs access to external knowledge by retrieving relevant documents before generating a response.Learn more →), conversation memory, tool results.
- Building agentsAgent systemsAI systems that take actions, use tools, and complete multi-step tasks by reasoning through a sequence of decisions. and pipelines — chaining model calls with tools, retries, and control flow so the system can take multi-step actions.
- Evaluation — proving the system works with datasets and judges, because you cannot ship probabilistic output on vibes.
- Production operations — monitoring quality, latency, and cost; catching regressions when a provider silently updates a model.
What is not on that list for most roles: training foundation models, deriving loss functions, or writing CUDA kernels. Those belong to ML engineers and researchers.
The mindset shift from deterministic software
If you're coming from software engineering, the biggest adjustment is giving up determinism. In normal software, add(2, 2) returns 4 every time and a passing unit test means the feature works. With LLMs:
- The same prompt can produce different outputs unless you control sampling.
- "Correct" is often a spectrum, not a boolean — an answer can be 80% right.
- Failures are silent: the model returns a confident, well-formatted, wrong answer instead of throwing an exception.
This is why evaluation is the defining discipline of the role. You replace "does the test pass?" with "what fraction of a representative dataset does the system handle well, and did my last change make that better or worse?" Engineers who internalize this ship reliable AI; engineers who don't ship demos that fall over in production.
What you actually need to know
You do not need a PhD or heavy math. You do need working fluency in:
- LLM mechanics — tokens, context windows, inference, sampling (the next few concepts).
- Retrieval / RAGRAGRetrieval-Augmented Generation — gives LLMs access to external knowledge by retrieving relevant documents before generating a response.Learn more → — giving models external knowledge (Stage 2).
- AgentsAgent systemsAI systems that take actions, use tools, and complete multi-step tasks by reasoning through a sequence of decisions. and tools — letting models take actions (Stage 3).
- Evaluation and guardrails — measuring quality and keeping systems safe (Stage 4).
- Production concerns — cost, latency, caching, monitoring (throughout).
Plus solid general software engineering: APIs, data modeling, async I/O, and system design. In practice AI engineers work in Python and TypeScript with vector databases, cloud model APIs, and observability tools — and almost never write CUDA or fine-tune from scratch.
A concrete example of the role
Suppose you're asked to build "an assistant that answers customer questions from our help center." An AI engineer's work is:
- Ingest and chunk the articles, embed them, and store them in a vector databaseVector databaseDatabase optimised for storing and searching embeddings by similarity using ANN algorithms.Learn more → (retrieval).
- On each question, retrieve relevant chunks and build a prompt: "answer only from this context; if it's not here, escalate" (context engineering + prompting).
- Add guardrails so it won't leak PII or answer off-topic, and a fallback to a human when confidence is low.
- Build an eval set of real questions with correct answers, and measure accuracy on every prompt or model change.
- Add tracing so when a bad answer is reported, you can see exactly which chunks were retrieved.
- Watch cost and latency, and route easy questions to a cheaper model.
Every one of those is a concept in this track. By the end you'll be able to design that system — and defend each decision in an interview.
Common interview mistakes
Mistake 1: Describing ML research as AI engineering. Answering "how would you improve accuracy?" with "retrain the model," when the real levers are retrieval, prompting, and evaluation. Frame your experience around systems you shipped.
Mistake 2: Thinking the role is just prompt engineering. Prompting is one piece. Production systems need retrieval, evaluation, cost monitoring, guardrails, and robust error handling.
Mistake 3: Treating the LLM as deterministic. Designing as if the same input always gives the same output, and being surprised by flakiness in production.
Mistake 4: Not knowing model tradeoffs. When to use a frontier model vs a small open model — cost, latency, context windowContext windowMaximum text an LLM can process at once, in tokens. Exceeding it causes earlier content to be forgotten.Learn more →, and capability — is fair game.
Key vocabulary
- Foundation model — A large model pre-trained on broad data (GPT-4o, Claude, Llama) that AI engineers build on rather than train.
- Inference — Running a trained model to get output (what you do on every request), as opposed to training.
- Context engineering — Deliberately choosing what to place in the model's limited context windowContext windowMaximum text an LLM can process at once, in tokens. Exceeding it causes earlier content to be forgotten.Learn more → each call.
- Evaluation (evals) — Measuring system quality against datasets or judge models; the core discipline of the role.
- Guardrails — Input/output checks that keep a system safe, on-topic, and compliant.
- RAGRAGRetrieval-Augmented Generation — gives LLMs access to external knowledge by retrieving relevant documents before generating a response.Learn more → (Retrieval-Augmented GenerationRAGRetrieval-Augmented Generation — gives LLMs access to external knowledge by retrieving relevant documents before generating a response.Learn more →) — Giving LLMs external knowledge at query time. Covered in depth in Stage 2.