Results got worse after we added a second pass
Context
Searchers tell us the patent they want used to come back near the top and now it
sits further down the list, and that searches feel slow. Search here runs in two
stages. First a cheap keyword search pulls a pool of candidate patents out of the
corpus. Then a cross-encoder (a slower model that reads the query and one
document together and scores how well they match) re-reads the top of that pool
and puts it in a new order. We shipped that second stage last sprint and expected
a clear quality win. Instead nDCG@10 (a ranking score: it rewards putting the
most relevant patents nearest the top) went down compared with the keyword stage
on its own, and p95 latency (the time the slowest 5 percent of queries take) went
from well under a millisecond per query to about 640 ms. Nobody has had time to
work out which part of that is the model and which part is us.
Your task
The eval harness is already written and the tests already pass, so you can see
the problem in numbers before you touch anything.
- Run
python3 eval_rank.py, then python3 eval_rank.py --first-stage-only.
The second one is the keyword stage on its own, which is the number the second
stage was supposed to beat. Write both sets of numbers down.
- Find and fix what is wrong in
pipeline/. Quality and latency are both in
scope. Treat the cross-encoder as a fixed external service: you can change
what you send it, how much of it you send, how many calls you make and what
you do with the scores, but not the scorer itself.
- Re-run the harness and write a short
NOTES.md, three or four lines: what you
found, what you changed, what you traded off, and the before and after numbers.
You are not expected to fix everything. One fix you understand and can explain is
worth more than three you guessed at.
Start here
Three files carry the second stage. Read them in this order:
pipeline/rerank.py the second stage itself, and the shortest file here
pipeline/retrieve.py the keyword stage that builds the candidate pool
pipeline/cross_encoder.py the scorer, including how its latency is charged
Everything else in the tree is loading and plumbing. tests/ describes how the
pipeline behaves today, which is not the same as how it should behave.
What's here
pipeline/corpus.py loads data/patents.json, builds full document text
pipeline/retrieve.py first_stage(query, pool_size): keyword scorer in the
BM25 style (BM25 is a standard formula that scores a
document on how often the query words appear in it)
pipeline/cross_encoder.py score(query, text) and score_batch(query, texts): the
simulated cross-encoder, including its latency model
pipeline/rerank.py rerank(query, candidates, k): the second stage
pipeline/search.py search(query, k) and search_timed(query, k)
eval_rank.py the eval harness, already written and working
eval/queries.json 13 queries with graded judgments (2, 1 or 0)
data/patents.json 129 patent records: id, title, abstract, claims
tests/ tests that describe how the pipeline behaves today
The cross-encoder is a deterministic simulation, not a network call. It reads the
query and the passage together, so it can match wording it was not given
literally. It is slow in the same shape as the real service: a fixed cost per
call plus a cost per character of passage text. Its docstring spells this out.
It is also imperfect, so do not expect a perfect ranking out of it.
data/patents.json is a small synthetic slice with the same row shape as the
production table. The judgments in eval/queries.json come from a review pass,
where a person rated each document against the query: 2 means highly relevant,
1 means partially relevant, 0 means judged and not relevant. A document with no
entry counts as 0.
Running it
From the repo root:
python3 -m unittest discover -s tests -v # 23 tests, all passing on the starter
python3 eval_rank.py # the eval harness, works as shipped
python3 eval_rank.py --first-stage-only # the keyword stage on its own
python3 eval_rank.py --k 5 # same, at a different cut-off
The harness takes about 10 seconds on the starter code because of the simulated
cross-encoder latency.
Time
Aim for about 30 minutes. You do not need to finish everything. We care more
about how you approach it than about how much you get through.