The model's JSON keeps breaking the app
Context
One bad reply from the model takes down the whole batch. When an office action comes in we split
it into items and ask a model to classify each one as JSON, and the result drives the response
workflow: category picks the response template, priority sets the order attorneys work through
the items, and needs_human sends an item to an attorney before any response text is drafted.
A few percent of the time the model's output breaks json.loads or has the wrong shape, and the
run stops on the first bad item. The model here is a deterministic stand-in that replays recorded
outputs. It behaves like the real one, including the bad replies.
The expected output for each item:
| field | rule |
|---|---|
| category | one of claim_rejection, formal_objection, prior_art, procedural, other |
| priority | integer 1 to 4 (1 is handled first) |
| summary | string, at most 200 characters |
| needs_human | boolean |
Your task
- Validate the model's output against this schema (hand-written, no libraries). The result of
triage.triage_item must always be schema-valid, with exactly these four keys.
- Repair only the cases that are safe to repair without guessing at meaning. For everything
else, retry with a prompt that tells the model what was wrong. Use no more than 3 model calls
per item.
- If the output is still invalid after the last attempt, or the model call itself fails, return
a safe default with
needs_human=True instead of crashing or guessing. No exception should
escape triage_item.
- Add tests covering each failure type you find in
data/model_fixtures.json.
- Create a short
NOTES.md (5-10 lines): what you decided was safe to repair versus retry, and
what your fallback does and why. Run python3 triage.py first and record how many of the 20
items get a usable result today, so you can give a before and after number.
What's here
triage.py builds the prompt, calls the model, parses with json.loads
llm.py model client (deterministic stand-in; read its docstring)
data/office_action_items.json 20 items taken from incoming office actions
data/model_fixtures.json the model's recorded outputs per item, first attempt and corrected retry
tests/test_triage.py current tests (all passing, they only cover clean items)
The stand-in model is deterministic. Asking the same question again returns the same output. It
only returns a corrected answer when the retry prompt says what was wrong (it looks for the words
"invalid" or "error" outside the item text). model.prompts records every prompt it received, and
model.calls_for("OA-006") counts calls per item.
Running it
python3 -m unittest discover -s tests -v
python3 triage.py # classify every item (crashes today)
python3 triage.py OA-003 OA-006 # classify specific items
Python 3.11, standard library only. No installs needed.
Time
Aim for about 25-30 minutes. You don't need to finish everything; we care more about how you approach it than about completeness.