Search broke after we rebuilt the index
Context
Since we rebuilt the search index last sprint, analysts say the results are
worse. Searches that used to return the obvious patent family now return
unrelated publications, and a patent attorney reported that a publication she had
just pushed into the system did not come back at all. Search used to compare the
query against every document in the corpus (brute_force_search). That got too
slow, so we moved search onto an IVF index (IvfIndex), where documents are
grouped into clusters and a search only looks inside the nearest few clusters.
It is the same shape as the FAISS IVF index we run in production. Brute force
search is still in the repo, so the quality of the new index is measurable rather
than a matter of opinion.
Your task
- Write
eval_recall.py so it reports, for a set of queries, recall@10 (how
often the right patent appears in the top 10) of IvfIndex.search against
brute_force_search (treat brute force as ground truth) and the average number
of vectors scanned per query. Expose the metric as
recall_at_k(approx_ids, exact_ids, k). Run it and write down the numbers for
the current code before you change anything.
- Find out why the new index disagrees with brute force and fix the causes you
find. There is more than one.
- Make a publication that arrives through
IvfIndex.add after the nightly build
searchable without waiting for the next rebuild.
- Choose an
nprobe setting (nprobe is how many clusters each search looks in)
on evidence from your eval, not by feel, and say what you are trading away.
- Write a short
NOTES.md (5 to 10 lines): what you found, what you changed,
the before and after recall@10 and vectors scanned per query, and anything you
would do next with more time.
What's here
index/embedder.py simulated patent embedder, deterministic, 40 dimensions
index/corpus.py 420 synthetic patent documents generated from a fixed seed
index/brute.py brute_force_search: checks every document, the ground truth
index/ivf.py IvfIndex: the clustered index, plus build_index(docs)
eval_recall.py unfinished eval harness, for you to complete
tests/ tests that describe the current behavior
The embedder turns text into a list of numbers (a vector) so that two documents
about the same thing end up with vectors pointing in a similar direction. It is a
simulated stand-in for the hosted embedding model. There is no network and no
third-party package here, so embed just hashes the words of the text into a
fixed set of numbers, the same way every time. It behaves like the production
model in the ways that matter for this problem: it returns raw vectors, vector
length grows with the length of the text, and similarity is cosine (the angle
between two vectors, which ignores how long they are). It knows nothing about
meaning beyond which words two texts share.
Running it
From the repo root:
python3 -m unittest discover -s tests -v
python3 eval_recall.py
Both brute_force_search and IvfIndex.search accept either query text or an
already embedded query vector, and both return [(doc_id, score)] sorted best
first. Keep those names and signatures working: index/embedder.py embed(text),
index/brute.py brute_force_search(query, k), and index/ivf.py IvfIndex
with build(docs), add(doc), search(query, k) plus module-level
build_index(docs).
Time
Aim for about 25-30 minutes. You don't need to finish everything; we care more
about how you approach it than about completeness.