How LLMs Generate Text — Sampling, Temperature, and Determinism

An LLM does not "write an answer." It predicts a probability for every possible next token, and then a sampling step picks one. Understanding that two-part process — the model proposes a distribution, the sampler disposes — is what lets you control cost, reliability, and creativity in a real system.

Why this appears in interviews

The single most common confusion in AI engineering interviews is treating an LLM as deterministic software. It is not. If you ask "why did my agent return valid JSON yesterday and broken JSON today with the same prompt?", the answer is almost always in the sampling parameters. Interviewers use questions about temperature and determinism to separate people who have called an LLM API from people who understand what happens inside it. You want to be the second kind.

Step 1 — The model outputs a probability distribution

At every step of generation, the model looks at all the tokens so far and produces a score (a "logit") for every token in its vocabulary — typically 50,000 to 200,000 possible tokens. Those scores are converted, via a function called softmax, into probabilities that sum to 1.

For the prompt The capital of France is, the distribution might look like:

| Token | Probability | |-------|-------------| | Paris | 0.91 | | the | 0.03 | | located | 0.02 | | a | 0.01 | | … (49,996 more) | … |

The model has not "chosen" anything yet. It has only expressed how likely each continuation is. This is the crucial mental shift: the model's output is a distribution, not a token.

Step 2 — The sampler picks one token

A separate, cheap, non-neural step now selects a single token from that distribution. Which selection strategy you use is controlled by API parameters. Then the chosen token is appended to the input and the whole process repeats — one token at a time — until an end-of-sequence token or your max_tokens limit. This loop is called autoregressive generation.

The three parameters you must understand are temperature, top_p, and top_k.

Temperature — how much the distribution is flattened

Temperature rescales the logits before softmax. Intuitively, it controls how much attention the sampler pays to lower-probability tokens.

• temperature = 0 — Always pick the single highest-probability token (this is called greedy decoding). Given the same prompt and model version, output is as close to deterministic as the API offers. Use this for extraction, classification, routing, function calling, and anything where you want the same input to give the same output.
• temperature = 0.7 — A moderate amount of randomness. The default for most chat use-cases. The model will usually pick likely tokens but sometimes explores.
• temperature = 1.0+ — High randomness. The distribution is flattened so unlikely tokens get a real chance. Use for brainstorming, creative writing, or generating diverse candidates. Above ~1.5 output often becomes incoherent.

# Deterministic — use for anything a downstream system parses
resp = client.chat.completions.create(
    model="gpt-4o",
    temperature=0,
    messages=[{"role": "user", "content": "Classify sentiment: 'the food was cold'"}],
)

# Creative — use when you WANT variety
resp = client.chat.completions.create(
    model="gpt-4o",
    temperature=1.1,
    messages=[{"role": "user", "content": "Give me 10 startup name ideas"}],
)

Top-p (nucleus sampling) — how much of the distribution is eligible

top_p = 0.9 means: sort tokens by probability, keep adding them until their cumulative probability reaches 90%, and sample only from that set. It dynamically shrinks the candidate pool — when the model is confident (one token has 0.95 probability) the pool is tiny; when it is unsure the pool is large. Most teams tune either temperature or top_p, not both.

Top-k — a fixed-size candidate pool

top_k = 40 means only the 40 highest-probability tokens are eligible. Simpler than top_p but less adaptive. Less commonly exposed by hosted APIs.

Why "temperature 0" is not truly deterministic

This trips up almost everyone. Even at temperature = 0, you can get different outputs across calls. Reasons:

1. Floating-point non-determinism. On GPUs, the order in which numbers are summed can vary between runs, and two tokens with nearly identical logits can swap places.
2. Model version changes. gpt-4o is a moving pointer; the provider updates the underlying weights. Pin a dated snapshot (e.g. gpt-4o-2024-08-06) when you need reproducibility.
3. Mixture-of-Experts routing. Modern models route tokens to different sub-networks depending on what else is in the batch, which can subtly change outputs.

The practical takeaway: temperature 0 gives you low-variance, not guaranteed-identical, output. If your system's correctness depends on byte-for-byte identical responses, you have a design bug — validate and constrain the output instead (see structured outputs, Stage 2).

Stop sequences and max_tokens

Two more controls shape generation:

• max_tokens caps the length of the output. It does not make the model "answer briefly" — it just truncates. To get short answers, instruct the model and set a sane cap as a backstop.
• Stop sequences are strings that halt generation the moment they appear (e.g. stop at "\n\n" or "</answer>"). Useful for carving one field out of a structured generation.

What this means for system design

Every design decision downstream flows from these mechanics:

Set temperature by job, not by taste. A retrieval router, a JSON extractor, and an SQL generator should run at temperature 0. A "write me three subject-line options" feature should run hot. Mixing these up is a top cause of flaky agentsAgent systemsAI systems that take actions, use tools, and complete multi-step tasks by reasoning through a sequence of decisions..

Latency scales with output tokens, not input tokens. Because generation is one-token-at-a-time, a 20-token answer returns far faster than a 2,000-token answer, even with a huge prompt. Prompt processing is parallel and fast; generation is sequential. This is why "ask for less output" is one of the highest-leverage latency and cost wins.

Streaming exists because of autoregression. Since tokens are produced one at a time, the API can send each to the client as it is generated. That is why chat UIs "type" — you are watching autoregressive generation live. Time-to-first-token and inter-token latency become the metrics users actually feel.

Common interview mistakes

Mistake 1: "I set temperature to 0 so it is deterministic." Say "low variance" instead, and mention floating-point and version drift. This single nuance signals real depth.

Mistake 2: Using a high temperature for structured output. If a function-calling or JSON step is flaky, the first thing to check is that temperature is 0. High temperature is a common, invisible cause of malformed JSON.

Mistake 3: Confusing max_tokens with "be concise." max_tokens truncates mid-sentence; it does not shorten reasoning. Control length with instructions plus a cap.

Mistake 4: Thinking the model "decides" a whole answer. It commits one token at a time with no lookahead. This is why models can paint themselves into a corner and why techniques like chain-of-thought (giving them room to generate intermediate tokens) improve accuracy.

Key vocabulary

• Logits — Raw per-token scores the model outputs before they are turned into probabilities.
• Softmax — The function that converts logits into a probability distribution summing to 1.
• Greedy decoding — Always taking the highest-probability token; what temperature = 0 approximates.
• Temperature — Scales the distribution's sharpness; 0 = focused/repeatable, high = diverse/creative.
• Top-p (nucleus sampling) — Sample only from the smallest set of tokens whose probabilities sum to p.
• Autoregressive generation — Producing output one token at a time, each conditioned on all prior tokens.
• Time-to-first-token (TTFT) — Latency until the first output token; distinct from total latency, which grows with output length.