A vector databaseVector databaseDatabase optimised for storing and searching embeddings by similarity using ANN algorithms.Learn more → stores embeddingsEmbeddingNumerical representation of text capturing semantic meaning. Similar texts produce similar vectors, enabling similarity search.Learn more → and answers one question extremely fast: "which stored vectors are most similar to this query vector?" It is the storage-and-search layer under every RAGRAGRetrieval-Augmented Generation — gives LLMs access to external knowledge by retrieving relevant documents before generating a response.Learn more → system — the piece that turns a pile of embeddingsEmbeddingNumerical representation of text capturing semantic meaning. Similar texts produce similar vectors, enabling similarity search.Learn more → into a searchable knowledge base at scale.
Why this appears in interviews
Every RAGRAGRetrieval-Augmented Generation — gives LLMs access to external knowledge by retrieving relevant documents before generating a response.Learn more → system needs somewhere to store and search embeddingsEmbeddingNumerical representation of text capturing semantic meaning. Similar texts produce similar vectors, enabling similarity search.Learn more →, so "which vector databaseVector databaseDatabase optimised for storing and searching embeddings by similarity using ANN algorithms.Learn more → and why?" is a near-guaranteed system-design question. Interviewers use it to check three things: do you understand why brute-force search doesn't scale, do you know the approximate-nearest-neighbor tradeoff, and can you pick a solution for the actual scale in front of you rather than reciting a favorite tool.
The mental model
Imagine a library of millions of books where you want the ones most similar in topic to the book in your hand. A regular database searches by title, author, or keyword — exact matches. A vector databaseVector databaseDatabase optimised for storing and searching embeddings by similarity using ANN algorithms.Learn more → searches by meaning: "find the books whose topic-vectors are closest to this one's."
The hard part is scale. With 10 million stored vectors, comparing your query against all 10 million ("brute-force" or exact search) is accurate but too slow for real-time use — it's O(n) per query. Vector databases fix this with Approximate Nearest Neighbor (ANN) algorithms that trade a tiny amount of recall for a massive speedup, turning seconds into milliseconds.
Exact vs approximate search — the core tradeoff
- Exact (brute-force / flat) search checks every vector. Guaranteed to find the true top-k, but slow and memory-heavy at scale. Perfectly fine for a few thousand vectors.
- Approximate (ANN) search uses a pre-built index to skip most comparisons. Occasionally misses the true nearest neighbor, but is orders of magnitude faster. This is what production uses.
The dial between them is recall — the fraction of the true top-k that the approximate search actually returns. You tune parameters to trade recall for speed and memory. Naming this tradeoff explicitly is what separates a strong answer from "I'd use Pinecone."
How HNSW works
HNSW (Hierarchical Navigable Small World) is the most widely used ANN index. Picture navigating a map at multiple zoom levels:
- The top layer is a sparse graph connecting distant points ("cities").
- Each lower layer adds detail ("neighborhoods," then "streets").
- To find neighbors of a query, you enter at the top, greedily hop toward the query, then descend a layer and repeat — zooming in until you reach the approximate nearest neighbors, without ever comparing against every point.
Two parameters you should be able to name:
ef_construction(build time) — how thoroughly the graph is built. Higher = better recall and a bigger, slower-to-build index.ef_search(query time) — how many candidates to explore per query. Higher = better recall, slower queries. This is the runtime speed/accuracy dial.
Other index families exist (IVF partitions vectors into clusters; product quantization compresses vectors to save memory), but HNSW is the default you should reach for and reason about.
Metadata filtering
Real queries are rarely "search all of my data." They're "search documents from this customer, written after January, in the policy collection." Vector databases let you attach metadata to each vector and filter on it alongside similarity search. The subtlety interviewers love:
- Pre-filtering narrows to the matching subset then runs similarity search — accurate, and the modern default.
- Post-filtering runs similarity search first then drops non-matching results — which can return too few results if the filter is selective (you asked for top-10 but 9 got filtered out).
Good multi-tenant design (per-customer isolation) usually rides on metadata filters, so this comes up in enterprise-flavored questions.
Choosing between solutions
| Option | Type | Best for | |--------|------|----------| | pgvector | Postgres extension | You already run Postgres and have under ~1M vectors; keeps data in one place | | Pinecone | Managed cloud | Fast to ship, zero infra; cost grows at scale | | Weaviate | Open-source / hosted | Native hybrid search; production systems wanting control | | Qdrant | Open-source (Rust) | Excellent performance-per-dollar, self-host | | Chroma | Embedded/local | Prototypes and local dev; not for large-scale prod |
The decision axes: scale (millions of vectors push you off pgvector/Chroma), hybrid search support, filtering performance, managed vs self-hosted (data residency and ops burden), and cost at your query volume. A senior-sounding answer says: "Under ~1M vectors and already on Postgres, I'd start with pgvector to avoid a new system; past that, or if I need first-class hybrid search and filtering, I'd move to a dedicated store like Qdrant or Weaviate."
When you might not need one
If your entire knowledge base fits in the model's context windowContext windowMaximum text an LLM can process at once, in tokens. Exceeding it causes earlier content to be forgotten.Learn more → and rarely changes, you may not need retrieval at all — just put it in the prompt (or use prompt caching). And for a few thousand vectors, an in-memory library (FAISS) or even NumPy is simpler than standing up a database. Reaching for infrastructure you don't need is itself a red flag; matching the tool to the scale is the signal.
Common interview mistakes
Mistake 1: Defaulting to pgvector for everything — or, conversely, reaching for a heavyweight vector DBVector databaseDatabase optimised for storing and searching embeddings by similarity using ANN algorithms.Learn more → when 5,000 vectors would fit in memory. Match the tool to the scale.
Mistake 2: Not knowing what ANN trades off. ANN is approximate; recall is the knob, tuned via ef_search/ef_construction.
Mistake 3: Forgetting hybrid search. Pure vector search misses exact keyword and rare-term matches (IDs, error codes); production often needs vector + BM25 fused together (Stage 2).
Mistake 4: Ignoring filtering semantics. Post-filtering can silently return too few results; pre-filtering is usually what you want.
Key vocabulary
- ANN (Approximate Nearest Neighbor) — Search that finds approximately the closest vectors, trading a little recall for large speed gains.
- Recall — The fraction of the true nearest neighbors an approximate search actually returns; the accuracy dial.
- HNSW — The dominant graph-based ANN index; tuned via
ef_constructionandef_search. - Index — The data structure a vector DBVector databaseDatabase optimised for storing and searching embeddings by similarity using ANN algorithms.Learn more → builds so it can search without scanning every vector.
- Metadata filtering — Restricting similarity search to vectors matching structured conditions (customer, date, collection).
- Hybrid search — Combining vector similarity with keyword search (BM25), usually fused with reciprocal rank fusion.