An AI Engineer builds production systems that use large language models and AI capabilities to solve real business problems — they sit at the intersection of software engineering and applied AI.
Why this appears in interviews
Almost every AI engineering interview starts here. Interviewers want to know whether you understand the role you are interviewing for. The most common failure mode is candidates who describe themselves as ML researchers or data scientists when the role is about shipping AI-powered products. Getting this distinction right signals self-awareness and domain clarity before you have answered a single technical question.
The mental model — three concentric circles
Think of three concentric circles:
Data Scientist — innermost circle. Builds models, runs experiments, cares about statistical significance. Rarely ships to production. Measures success in accuracy percentages.
ML Engineer — middle circle. Takes the data scientist's model and makes it run reliably in production. Cares about serving latency, retraining pipelines, model drift. Measures success in uptime and throughput.
AI Engineer — outermost circle. Does not train models from scratch. Takes foundation models (GPT-4, Claude, Llama) that already exist and builds applications on top of them. Cares about RAGRAGRetrieval-Augmented Generation — gives LLMs access to external knowledge by retrieving relevant documents before generating a response.Learn more → pipelines, prompt design, agent orchestration, and shipping products users actually use. Measures success in user outcomes.
The critical difference: AI Engineers are software engineers who happen to work with LLMs. They are not researchers.
How the role works in practice
An AI Engineer at a startup might spend their week:
- Building a RAGRAGRetrieval-Augmented Generation — gives LLMs access to external knowledge by retrieving relevant documents before generating a response.Learn more → pipeline that lets a customer support tool answer questions from internal documentation
- Optimizing prompt templates to reduce hallucination rates
- Instrumenting LLM calls to track cost and latency
- Building an agent that can take actions based on user requests
- Evaluating which embeddingEmbeddingNumerical representation of text capturing semantic meaning. Similar texts produce similar vectors, enabling similarity search.Learn more → model gives the best retrieval results for their use case
They use Python, TypeScript, vector databases, cloud APIs, and monitoring tools. They almost never write CUDA kernels or fine-tune models from scratch — those tasks belong to ML Engineers.
Common interview mistakes
Mistake 1: Describing ML research as AI engineering. If you say "I built a transformer from scratch" you are describing ML research. Frame your experience around systems you shipped.
Mistake 2: Thinking the role is just prompt engineering. Prompt engineering is one small piece. Production AI systems require evaluation frameworks, retrieval systems, cost monitoring, safety guardrails, and robust error handling.
Mistake 3: Not knowing model tradeoffs. An AI Engineer should know when to use GPT-4o vs Claude vs a smaller open-source model — cost, latency, context windowContext windowMaximum text an LLM can process at once, in tokens. Exceeding it causes earlier content to be forgotten.Learn more →, and capability tradeoffs are fair game.
Key vocabulary
- Foundation model — A large pre-trained model (GPT-4, Claude, Llama) that serves as the base for AI applications.
- Inference — Running a trained model to get predictions or generate output. Contrast with training.
- Prompt — The input you send to an LLM. Designing prompts well is a core AI engineering skill.
- RAGRAGRetrieval-Augmented Generation — gives LLMs access to external knowledge by retrieving relevant documents before generating a response.Learn more → (Retrieval-Augmented GenerationRAGRetrieval-Augmented Generation — gives LLMs access to external knowledge by retrieving relevant documents before generating a response.Learn more →) — A technique for giving LLMs access to external knowledge. Covered in depth in Concept 5.