Why we chunk
Documents are split before indexing for two reasons: embeddings over-compress long text (semantics blur when too much is packed into one vector), and you want to retrieve the relevant passage, not a whole document. Chunks are usually no more than a few hundred tokens.
The chunking ladder
- Fixed-size (N tokens, fixed overlap) — simplest, ignores structure, splits mid-sentence. Baseline only.
- Recursive / structural (e.g. LangChain's
RecursiveCharacterTextSplitter) — splits on a hierarchy of separators (paragraph → sentence → word) to respect natural boundaries. The pragmatic default. - Semantic — splits where embedding similarity between adjacent sentences drops. More compute, and benchmarks are mixed — it doesn't reliably beat good recursive chunking, so A/B it on your corpus.
- Parent-child ("small-to-big") — embed small child chunks for precise retrieval, but return the larger parent chunk to the LLM for context. A standard production pattern.
- Late chunking (Jina, 2024) — embed the whole document first, then chunk the token embeddings. Each chunk's vector retains context from the surrounding document. Needs a long-context embedding model.
A reasonable starting point is 256–512 tokens with 10–20% overlap, then tune. Overlap stops facts being severed at boundaries.
Metadata is under-used leverage
Attach source, title, section, date, author, permissions, and doc-type to every chunk. That's what powers filtering (Chapter 3), citations (Chapter 4), freshness, and multi-tenant access control.
Choosing an embedding model
- Use the MTEB Retrieval sub-tasks, not the overall MTEB score — overall rank ≠ retrieval rank.
- Dimensions & cost matter. Models like OpenAI's
text-embedding-3support Matryoshka truncation — shorten dimensions (e.g. 3072→1024) for big storage/latency savings at small quality cost. - Domain fit beats leaderboard rank. Code, legal, biomedical, and multilingual corpora often need specialized embeddings. Test on data.