Search broke after we rebuilt the index
Context
Since we rebuilt the search index last sprint, analysts say the results are
worse. Searches that used to return the obvious patent family now return
unrelated publications, and a patent attorney reported that a publication she had
just pushed into the system did not come back at all. Search used to compare the
query against every document in the corpus (brute_force_search). That got too
slow, so we moved search onto an IVF index (IvfIndex), where documents are
grouped into clusters and a search only looks inside the nearest few clusters.
It is the same shape as the FAISS IVF index we run in production. Brute force
search is still in the repo, so the quality of the new index is measurable rather
than a matter of opinion.
Your task
eval_recall.py is already written, so you can see the collapse before you touch
anything.
- Run
python3 eval_recall.py and write the numbers down: mean recall@10 and
mean vectors scanned per query. It also prints the worst queries with the exact
and the approximate result lists side by side, which is usually the fastest way
in.
- Find out why the new index disagrees with brute force, and fix what you find.
There is more than one cause.
- Re-run it, pick an
nprobe on the evidence from your eval rather than by feel
(python3 eval_recall.py --nprobe 8 lets you try a setting without editing
anything), and write a short NOTES.md, three or four lines: what you found,
what you changed, the before and after numbers, and what you traded away.
Making a document that arrives through IvfIndex.add searchable without waiting
for the next rebuild is a bonus, not a requirement.
Start here
index/ivf.py the approximate index: build, add and search. Most of it is here.
index/embedder.py turns text into a vector
index/brute.py the exact search, and your ground truth. This one is correct.
tests/ describes how the index behaves today, which is not the same as how it
should behave.
What's here
index/embedder.py simulated patent embedder, deterministic, 40 dimensions
index/corpus.py 420 synthetic patent documents generated from a fixed seed
index/brute.py brute_force_search: checks every document, the ground truth
index/ivf.py IvfIndex: the clustered index, plus build_index(docs)
eval_recall.py the recall harness, already written and working
tests/ tests that describe the current behavior
The embedder turns text into a list of numbers (a vector) so that two documents
about the same thing end up with vectors pointing in a similar direction. It is a
simulated stand-in for the hosted embedding model. There is no network and no
third-party package here, so embed just hashes the words of the text into a
fixed set of numbers, the same way every time. It behaves like the production
model in the ways that matter for this problem: it returns raw vectors, vector
length grows with the length of the text, and similarity is cosine (the angle
between two vectors, which ignores how long they are). It knows nothing about
meaning beyond which words two texts share.
Running it
From the repo root:
python3 -m unittest discover -s tests -v
python3 eval_recall.py # the recall harness, works as shipped
python3 eval_recall.py --nprobe 8 # try a different nprobe
Both brute_force_search and IvfIndex.search accept either query text or an
already embedded query vector, and both return [(doc_id, score)] sorted best
first. Keep those names and signatures working: index/embedder.py embed(text),
index/brute.py brute_force_search(query, k), and index/ivf.py IvfIndex
with build(docs), add(doc), search(query, k) plus module-level
build_index(docs).
Time
Aim for about 30 minutes. You do not need to finish everything. We care more
about how you approach it than about how much you get through.