Results got worse after we added a second pass
Context
Searchers tell us the patent they want used to come back near the top and now it
sits further down the list, and that searches feel slow. Search here runs in two
stages. First a cheap keyword search pulls a pool of candidate patents out of the
corpus. Then a cross-encoder (a slower model that reads the query and one
document together and scores how well they match) re-reads the top of that pool
and puts it in a new order. We shipped that second stage last sprint and expected
a clear quality win. Instead nDCG@10 (a ranking score: it rewards putting the
most relevant patents nearest the top) went down compared with the keyword stage
on its own, and p95 latency (the time the slowest 5 percent of queries take) went
from well under a millisecond per query to about 640 ms. Nobody has had time to
work out which part of that is the model and which part is us.
Your task
- Fill in
ndcg_at_k and recall_at_k in eval_rank.py (the exact convention
is written out in the module docstring), run the harness, and write the before
numbers down before you change any pipeline code. You need mean nDCG@10, mean
recall@50 (how often the right patent is somewhere in the pool of 50
candidates) and p95 latency on eval/queries.json.
- Get a baseline for the same query set with the second stage bypassed, so you
know what the keyword stage alone was scoring. That is the number the second
stage has to beat.
- Find and fix what is wrong in
pipeline/. Quality and latency are both in
scope. Treat the cross-encoder as a fixed external service: you can change
what you send it, how much of it you send, how many calls you make and what
you do with the scores, but not the scorer itself.
- Re-run the harness and record the after numbers next to the before numbers.
- Write a short
NOTES.md (5 to 10 lines): what you found, what you changed,
what you traded off, and the before and after numbers.
If you run out of time, a clear note about what you would do next is worth more
than a half finished change.
What's here
pipeline/corpus.py loads data/patents.json, builds full document text
pipeline/retrieve.py first_stage(query, pool_size): keyword scorer in the
BM25 style (BM25 is a standard formula that scores a
document on how often the query words appear in it)
pipeline/cross_encoder.py score(query, text) and score_batch(query, texts): the
simulated cross-encoder, including its latency model
pipeline/rerank.py rerank(query, candidates, k): the second stage
pipeline/search.py search(query, k) and search_timed(query, k)
eval_rank.py eval harness, two metric functions left for you
eval/queries.json 13 queries with graded judgments (2, 1 or 0)
data/patents.json 129 patent records: id, title, abstract, claims
tests/ tests that describe how the pipeline behaves today
The cross-encoder is a deterministic simulation, not a network call. It reads the
query and the passage together, so it can match wording it was not given
literally. It is slow in the same shape as the real service: a fixed cost per
call plus a cost per character of passage text. Its docstring spells this out.
It is also imperfect, so do not expect a perfect ranking out of it.
data/patents.json is a small synthetic slice with the same row shape as the
production table. The judgments in eval/queries.json come from a review pass,
where a person rated each document against the query: 2 means highly relevant,
1 means partially relevant, 0 means judged and not relevant. A document with no
entry counts as 0.
Running it
From the repo root:
python3 -m unittest discover -s tests -v # 23 tests, all passing on the starter
python3 eval_rank.py # the harness, once your metrics are in
python3 eval_rank.py --k 5 # same, at a different cut-off
The harness takes about 10 seconds on the starter code because of the simulated
cross-encoder latency.
Time
Aim for about 25-30 minutes. You don't need to finish everything; we care more
about how you approach it than about completeness.