Did that prompt change actually help?
Context
We pull five fields off patent front pages with an LLM, and nobody can tell whether the last prompt edit made it better or worse. The fields are application_number, filing_date, assignee, inventor (the first named inventor) and jurisdiction (the two-letter office code, for example US, EP or JP). There is a half-finished evaluate.py and a small labeled set. Your job is to turn them into an eval you would trust, then use it to improve the extractor.
An eval here means: run the extractor over documents whose correct answers we know, and count how often each field comes out right. Accuracy is the share of fields that match the gold label.
Your task
- Finish the labeled set.
data/labels.json has gold labels for 12 of the 15 front pages in data/docs/. Label the other 3 by reading them. Use the same format as the existing labels: ISO dates (YYYY-MM-DD), application numbers as the document prints them, two-letter office codes, names without addresses.
- Make the scorer sensible. Right now
evaluate.py compares raw strings, so 03/04/2018 and 2018-04-03 count as different. Normalize values before comparing: dates to ISO (a numeric date like 03/04/2018 must be read using the date format the document states, and the date_convention() helper already reads it), case and whitespace for text, and office codes uppercased. Do not let the scorer turn an office name into a code, because that would hide a real extraction error. A missing prediction is always wrong.
- Report per field and list failures.
evaluate() must return a dict that includes:
per_field: {field: {"correct": int, "total": int, "accuracy": float}} for all five fieldsfailures: a list of {"doc", "field", "predicted", "expected"} dicts
Keep the existing keys (accuracy etc.) and keep the names and signatures of normalize, field_matches and evaluate, since other tooling imports them. Make python3 evaluate.py print something a person can act on.
- Find the top failure modes and improve the extractor. Change the prompt and/or add post-processing in
extractor.py (or new modules). We will run your extractor on front pages you have not seen, so fixes must generalize: no special cases for particular documents or file names. extract(text) must keep working with just the document text.
- Write
NOTES.md (5-10 lines): the before and after numbers (overall and per field), the top failure modes you found, what you changed for each, and anything you think the eval still gets wrong or does not cover.
What's here
llm.py model client (simulated model, see below). Treat it as an external model: don't edit it.
extractor.py prompt + model call + JSON parsing. This is what you improve.
evaluate.py scorer and eval runner. Currently exact string match and one overall number.
data/docs/ 15 patent front pages as text
data/labels.json gold labels for 12 of them
tests/ existing unit tests (they pass now; keep them passing)
llm.py is a simulated stand-in for the model we use in production. It makes the same kinds of mistakes, and it is deterministic, so your numbers are the same on every run.
Running it
python3 evaluate.py # run the eval
python3 extractor.py data/docs/fp_02.txt # extract one document
python3 -m unittest discover -s tests -v # run the tests
Python 3.11, standard library only.
Time
Aim for about 25-30 minutes. You don't need to finish everything; we care more about how you approach it than about completeness.