The bugs that recur in AI apps
LLM apps fail in a handful of characteristic ways — mostly in the plumbing around the model, not the model itself. Recognizing the pattern from the symptom is the whole skill.
- Lost conversation state — the bot "forgets" earlier turns. Cause: the app isn't persisting/resending the message history (the API is stateless — you hold state). Symptom: each turn behaves like the first.
- Embedding-space mismatch — semantic search returns garbage or identical results. Cause: query and documents embedded with different models/versions/normalization, so they live in incompatible vector spaces. (RAG course.)
- Prompt-assembly bugs — the model ignores instructions or misbehaves. Cause: the system prompt is in the wrong role, variables aren't interpolated, delimiters are missing, or context is malformed. (Prompt Engineering course.)
- Truncated output — answers cut off. Cause:
max_tokenstoo low;finish_reason: length. (LLM APIs course.) - Context overflow / lost-in-the-middle — quality degrades on long inputs. Cause: too much context, key info buried, or window exceeded. (RAG/Cost courses.)
- Non-idempotent retries — double-charges, duplicate actions. Cause: retrying a non-idempotent call without an idempotency key. (Foundations course.)
- Caching bugs — stale or wrong answers, or a cache that never hits. Cause: bad cache keys or invalidation. (Cost course.)
- Parsing model output — crashes on unexpected output. Cause: regex-parsing free text instead of using structured outputs + validation.
Debugging them
The method (previous chapter) plus LLM-specific evidence: log the exact prompt the model received, the raw response, the finish reason, and token counts, and use traces. Most "the model is dumb" complaints are a deterministic plumbing bug — wrong prompt, lost state, mismatched embeddings — hiding behind the model's non-determinism. Check the plumbing before blaming the model.