A Large Language Model is a neural network trained on massive amounts of text to do one deceptively simple thing: predict the next token in a sequence. That single objective, applied at enormous scale, produces systems that reason, write, and answer questions. You don't need to build a transformer — but you do need a working model of tokens, context windows, attention, and inference, because every architectural decision downstream depends on them.
Why this appears in interviews
Every AI engineering problem eventually comes back to LLM mechanics. Interviewers aren't checking whether you can implement attention; they're checking whether you understand the consequences of how LLMs work — why context windows are finite, why long prompts cost more and get slower, why the model has no memory between calls. A candidate who doesn't understand why context size matters cannot design a sensible RAGRAGRetrieval-Augmented Generation — gives LLMs access to external knowledge by retrieving relevant documents before generating a response.Learn more → system.
The mental model
Picture an extremely sophisticated autocomplete. Give it "The capital of France is" and it predicts "Paris." Scale that to hundreds of billions of parameters trained on much of the internet, and the autocomplete becomes capable enough to reason through problems, write code, and hold a conversation. Everything an LLM does is next-token prediction — that's not a simplification to grow out of; it's literally the mechanism.
Tokens are not words
LLMs process tokens, chunks of roughly 3–4 characters produced by a tokenizer. "Hello world" is 2 tokens; "Supercalifragilistic" is several. This matters concretely: you're billed per token, context limits are measured in tokens, and rare words or code cost more tokens than common prose. Rule of thumb: English word count × ~1.3 ≈ tokens. Tokenization also explains quirks — models miscount letters in a word or struggle with exact string manipulation because they see tokens, not characters.
The context window is everything the model can see
The context windowContext windowMaximum text an LLM can process at once, in tokens. Exceeding it causes earlier content to be forgotten.Learn more → is the maximum number of tokens the model can consider at once — input and output combined. Modern models range from ~128k to 1M+ tokens. The iron rule: whatever is not in the context windowContext windowMaximum text an LLM can process at once, in tokens. Exceeding it causes earlier content to be forgotten.Learn more → does not exist to the model. It cannot access prior conversations, your internal docs, or today's news unless you put them in the context. This single fact is why RAGRAGRetrieval-Augmented Generation — gives LLMs access to external knowledge by retrieving relevant documents before generating a response.Learn more → exists (Stage 2) and why context engineering is a core skill (Stage 4).
A peek under the hood: transformers and attention
You won't implement it, but the intuition sharpens every later decision. LLMs are transformers, and their key mechanism is attention: when processing each token, the model computes how much every other token in the context should influence it — dynamically "attending" to the relevant words. "It" in "the trophy didn't fit in the suitcase because it was too big" attends strongly to "trophy," which is how the model resolves meaning.
Two consequences that matter to engineers:
- Attention is (roughly) quadratic in context length — doubling the context can quadruple the attention work. That's why context windows are finite and why very long prompts are slow and expensive, not just a policy limit.
- Attention is uneven across position — models attend most reliably to the start and end of a long context and can under-weight the middle. This is the root of the "lost in the middle" problem (Stage 4) and a reason to retrieve few, well-placed chunks rather than stuffing the prompt.
How inference works, step by step
- Your prompt (input tokens) is processed in parallel — fast.
- The model generates output one token at a time (autoregressive generation); each new token is conditioned on your input plus everything generated so far.
- Generation stops at an end-of-sequence token or your
max_tokenscap.
This is why first-token latency can be quick while total latency grows with output length — a critical fact for the latency work in Stage 4. (How the model picks each token — temperature, top-p — is the next concept.)
The KV cache
Generating a long response, the model would otherwise re-process the entire context for every new token. The KV cacheKV cacheKey-Value cache — stores intermediate attention computations to speed up token generation, reducing inference latency.Learn more → stores the intermediate attention computations so each prior token is processed once — a big speedup, at the cost of GPU memory that grows with context length. It's also the basis of prompt caching (Stages 3–4), where a provider reuses the cached prefix of a long, stable prompt to cut cost and latency.
Inference is stateless
Every API call starts fresh. The model remembers nothing between calls unless you resend the history. The "conversation" you experience is an illusion you maintain by including prior turns in each new request — which is why longer chats cost more and eventually must be trimmed or summarized (Stage 3 memory).
Training vs inference, and the model landscape
Training is the one-time, hugely expensive process of learning from data; inference is running the trained model on each request (what you pay for and optimize). As an AI engineer you almost always do inference and rarely train.
You'll choose among closed/hosted frontier models (OpenAI's GPT, Anthropic's Claude, Google's Gemini) — best quality, simplest to use, per-token cost, data leaves your perimeter — and open-weight models (Llama, Mistral, Qwen) you can self-host for control, data residency, and cost at high volume, at the price of running the infrastructure. Knowing this tradeoff (and that a cheap small model can handle easy requests while a frontier model handles hard ones — model routing) is fair game in interviews.
Common interview mistakes
Mistake 1: Treating tokens as words. A 10,000-word document is ~13,000+ tokens; always estimate with the ~1.3× multiplier.
Mistake 2: Ignoring context-window implications. "I'll just put the whole knowledge base in the context" is impossible or ruinously expensive at scale — hence RAGRAGRetrieval-Augmented Generation — gives LLMs access to external knowledge by retrieving relevant documents before generating a response.Learn more →.
Mistake 3: Confusing training and inference. Training happens once and is enormously expensive; inference runs on every request.
Mistake 4: Assuming memory. Forgetting that state is your responsibility — the model is stateless between calls.
Key vocabulary
- Token — The basic unit of LLM processing (~3–4 characters); cost and context limits are measured in tokens.
- Context windowContext windowMaximum text an LLM can process at once, in tokens. Exceeding it causes earlier content to be forgotten.Learn more → — Max tokens (input + output) the model can consider at once; anything outside it is invisible to the model.
- Transformer / attention — The architecture and the mechanism by which each token weighs the influence of others; attention cost grows ~quadratically with length.
- Autoregressive generation — Producing output one token at a time, each conditioned on all prior tokens.
- KV cacheKV cacheKey-Value cache — stores intermediate attention computations to speed up token generation, reducing inference latency.Learn more → — Cached attention computations that speed up generation and underpin prompt caching.
- Inference vs training — Running a trained model per request vs the one-time process of learning from data.