The model's JSON keeps breaking the app
Context
One bad reply from the model takes down the whole batch. When an office action comes in we split
it into items and ask a model to classify each one as JSON, and the result drives the response
workflow: category picks the response template, priority sets the order attorneys work through
the items, and needs_human sends an item to an attorney before any response text is drafted.
A few percent of the time the model's output breaks json.loads or has the wrong shape, and the
run stops on the first bad item. The model here is a deterministic stand-in that replays recorded
outputs. It behaves like the real one, including the bad replies.
The expected output for each item:
| field | rule |
|---|---|
| category | one of claim_rejection, formal_objection, prior_art, procedural, other |
| priority | integer 1 to 4 (1 is handled first) |
| summary | string, at most 200 characters |
| needs_human | boolean |
Your task
- Run
python3 triage.py before you change anything and note how many of the 20
items get a usable result today. That is your before number.
- Validate the model's output against the schema above, hand written, no
libraries.
triage.triage_item must always return something schema valid, with
exactly those four keys. Repair only the cases that are safe to repair without
guessing at meaning, and for everything else retry with a prompt that tells the
model what was wrong, using no more than 3 model calls per item. If the output
is still invalid after the last attempt, or the model call itself fails, return
a safe default with needs_human=True instead of crashing or guessing. No
exception should escape triage_item.
- Re-run it and write a short
NOTES.md, three or four lines: what you decided
was safe to repair versus retry, what your fallback does and why, and the
before and after numbers.
Adding tests for the failure types you find in data/model_fixtures.json is
welcome but not required.
Start here
triage.py triage_item, the function that has to stop crashing
llm.py the model call and the ways it goes wrong
data/model_fixtures.json the malformed outputs it has to survive
What's here
triage.py builds the prompt, calls the model, parses with json.loads
llm.py model client (deterministic stand-in; read its docstring)
data/office_action_items.json 20 items taken from incoming office actions
data/model_fixtures.json the model's recorded outputs per item, first attempt and corrected retry
tests/test_triage.py current tests (all passing, they only cover clean items)
The stand-in model is deterministic. Asking the same question again returns the same output. It
only returns a corrected answer when the retry prompt says what was wrong (it looks for the words
"invalid" or "error" outside the item text). model.prompts records every prompt it received, and
model.calls_for("OA-006") counts calls per item.
Running it
python3 -m unittest discover -s tests -v
python3 triage.py # classify every item (crashes today)
python3 triage.py OA-003 OA-006 # classify specific items
Python 3.11, standard library only. No installs needed.
Time
Aim for about 30 minutes. You do not need to finish everything. We care more
about how you approach it than about how much you get through.