Dense (vector) search
Embed the query, find the nearest chunk vectors. Exact nearest-neighbor is O(N); production uses ANN (Approximate Nearest Neighbor). The dominant index is HNSW — a multi-layer proximity graph. Its query-time knob efSearch trades recall for latency: higher = better recall, slower.
Similarity metric: cosine is standard for text embeddings (dot product is equivalent for normalized vectors). Match the metric the embedding model was trained with.
top-k: how many chunks to feed the model. More k → higher recall but more noise, cost, and "lost in the middle." There's no universal k — tune it. Anthropic found k=20 best when paired with reranking from a large candidate pool; without a reranker, 3–8 is common.
Hybrid search is the current default
Combine dense embeddings with BM25 (lexical/keyword). BM25 nails the exact matches embeddings miss — error codes ("TS-999"), SKUs, names, rare jargon — while dense handles paraphrase and concepts. Pure vector search is dated; hybrid is the baseline.
Fuse the two ranked lists with Reciprocal Rank Fusion (RRF): combine by rank, not raw score, because BM25 and cosine scores aren't comparable. score(d) = Σ 1/(k + rank_i(d)), k≈60.
The #1 hybrid bug: fusing two top-10 lists — over a large corpus their intersection is often empty. Fetch 50–200 candidates per retriever, fuse, then truncate. Give sparse more candidates for jargon-heavy corpora.
Reranking — now a default, not optional
Retrieve a large candidate set cheaply (bi-encoder + BM25), then a cross-encoder jointly encodes each (query, chunk) pair for a precise relevance score and keeps the top few. Anthropic's recipe: retrieve ~150 → rerank → keep 20. Cross-encoders are far more accurate but too slow to run over the whole corpus — that's why the two-stage architecture exists. Models: Cohere rerank-v3.5, bge-reranker-v2-m3 (open), Voyage/Jina rerankers. Adds ~100–300 ms; typically +10–25% precision and fewer hallucinations.