AI systems have vulnerabilities that come from how they learn and process information — not from bugs in code. That single fact makes them resistant to traditional defenses (a firewall doesn't stop a jailbreak) while exposing them to entirely new attack classes. If you understand why AI is structurally different, every specific attack in this track becomes logical rather than a name to memorize. This is the conceptual bedrock of the role.
Why this appears in interviews
Interviewers open here because it separates people who understand AI security from people who've read a list of attacks. If you can explain why prompt injectionPrompt injectionAttack where malicious input overrides an AI system's instructions, causing unintended behaviour. has no clean fix, or why a model leaks its training data, you can reason about novel attacks; if you can't, you're pattern-matching. This concept is the "first principles" the rest builds on.
The mental model — rules vs learning
Traditional software does exactly what the code says. If the code rejects input containing DROP TABLE, that exact SQL injection fails — security is defining and enforcing explicit rules over a bounded input space. AI systems instead learn behavior from data and generalize over an effectively infinite input space. You cannot fully enumerate what a model will do, and inputs that look completely normal can trigger abnormal behavior. Security shifts from "enforce these rules" to "reason about a system whose behavior you can't fully specify" — a fundamentally harder problem.
The properties that make AI uniquely vulnerable
1. No hard boundary between instructions and data. The deepest one. A CPU separates code from data; an LLM receives everything — system prompt, user input, retrieved documents, tool output — as one stream of tokens it was trained to follow. So data can become instructions (prompt injectionPrompt injectionAttack where malicious input overrides an AI system's instructions, causing unintended behaviour.), and there's no architectural fix the way parameterized queries fix SQL injection. This property alone explains most LLM attacks.
2. Non-determinism. The same input can yield different outputs, so security testing is statistical, not conclusive — you can't prove a vulnerability is absent by testing once, and an exploit may fire only intermittently, making it harder to detect and reproduce.
3. Emergent behavior. Large models exhibit capabilities never explicitly trained — including following complex, multi-step instructions hidden in innocent-looking text. You're securing behavior the builders didn't deliberately create and can't fully predict.
4. The instruction-following paradox. The more capable a model is at following instructions, the more capable it is at following malicious instructions. Capability and vulnerability grow together, so you can't simply "make it smarter" to make it safer.
5. Memorization → data leakage. Models memorize parts of their training data and can reveal it — verbatim or via inference — even when not asked directly. The model's outputs are a channel to its training data (membership inference, inversion, extraction).
6. Expanded, opaque supply chain. AI systems stitch together third-party pre-trained models (Hugging Face), datasets, and embeddingsEmbeddingNumerical representation of text capturing semantic meaning. Similar texts produce similar vectors, enabling similarity search.Learn more → you can't fully inspect. A poisoned open-source model or dataset is a supply-chain attack that ships a backdoor into your system.
Safety vs security (a distinction interviewers probe)
- Safety — preventing unintended harm (the model gives dangerous advice by accident).
- Security — preventing intentional harm by an adversary (an attacker makes it give that advice).
They overlap (a jailbreak defeats safety training as an attack) but the threat model differs: security assumes a motivated adversary optimizing against your defenses. Conflating them is a common miss.
What traditional security misses
A classic pentester tests SQL injection, XSS, auth, and cloud misconfig — all still necessary. They will not test for prompt injectionPrompt injectionAttack where malicious input overrides an AI system's instructions, causing unintended behaviour., training-data extraction, jailbreaks, adversarial examples, or retrieval-corpus poisoning. AI security adds a model-and-data threat surface on top of traditional AppSec; it doesn't replace it. That's why the role needs both skill sets.
Why this makes defense hard (and shapes the whole track)
Because these properties are inherent to how models work, the corresponding attacks are mitigated, not eliminated — which is why the rest of the track emphasizes defense in depth, least-privilege architecture, monitoring, and residual-risk thinking over any single silver-bullet control. Every later concept is an instance of defending one of these properties.
Common interview mistakes
Mistake 1: Treating non-determinism as only a reliability issue. It's also a security issue — intermittent exploits are harder to detect and prove absent.
Mistake 2: Conflating safety and security. Unintended harm vs a motivated adversary; the defenses and threat models differ.
Mistake 3: "RAGRAGRetrieval-Augmented Generation — gives LLMs access to external knowledge by retrieving relevant documents before generating a response.Learn more → fixes the data risks." RAGRAGRetrieval-Augmented Generation — gives LLMs access to external knowledge by retrieving relevant documents before generating a response.Learn more → changes the attack surface (adds retrieval poisoning, indirect injection); it doesn't remove memorization/leakage risk.
Mistake 4: Expecting an architectural fix for injection. There's no parameterized-query equivalent because instructions and data share one channel — you layer mitigations instead.
Key vocabulary
- Instruction/data conflation — Models treat all input as potentially-instruction tokens; the root of prompt injectionPrompt injectionAttack where malicious input overrides an AI system's instructions, causing unintended behaviour. with no clean fix.
- Non-determinism — Same input, possibly different output; makes security testing statistical, not conclusive.
- Emergent capability — Unplanned model behavior not predictable from training; part of the attack surface.
- Memorization / leakage — Models reproducing or revealing training data through their outputs.
- Supply-chain attack — Compromising a system via its (model/data/dependency) supply chain rather than the system directly.
- Safety vs security — Preventing unintended harm vs preventing intentional adversarial harm.