Drafts cut off halfway
Context
Attorneys tell us that a generated draft sometimes stops in the middle of a sentence, that
words show up glued together or scrambled, and that refreshing the page loses everything the
model had written so far. When an attorney asks for a draft, the text arrives in the editor
while the model writes it, over server-sent events (a long-lived HTTP response where the
server pushes text to the browser as it is produced, instead of making the browser wait for
the whole draft). The endpoint and the browser side of that stream are in this repo. Nobody
has found a pattern in the complaints yet, and the editor always ends up saying the draft is
complete, even the times the attorney can see it is not.
Your task
measure.js is already written. It replays two captured responses through the
client, one whole draft and one cut off part way, and prints how many events the
client applied, how many characters of the expected draft it ended up with,
whether the text matched exactly, and the status it finished in.
- Run
node measure.js and write those numbers down.
- Make the client read the stream safely. One read from the network gives
whatever bytes happened to arrive, so a single read can hold several events,
or half an event, or the end of one and the start of the next.
parseEvents(buffer) should return only the complete events (an event ends
with a blank line) plus a remainder string with the text left over, and the
caller feeds that remainder into the next read. Nothing on the way from the
wire to the editor may trim or reflow a piece: a piece can be a single space
or half of a long word.
- Make a finished draft tell itself apart from a dropped connection, and let a
client pick up where it left off. Both are specified under "The streaming
contract". Then re-run
measure.js and write a short NOTES.md, three or
four lines: what you changed, the before and after numbers, and what you
would do next.
Adding tests of your own is welcome but not required.
Start here
web/stream_client.js parseEvents, applyChunk, createState, finishStream
stream_server.py the endpoint that sends the events
store.py where a draft is saved
model_stream.py produces the pieces. You should not need to change it.
The streaming contract
Keep the names the rest of the app already uses: create_app(options) and
GET /api/drafts/<id>/stream in stream_server.py, stream_tokens(prompt, options) in model_stream.py, append_chunk(draft_id, text) and
get_draft_text(draft_id) in store.py, and parseEvents, applyChunk,
createState and finishStream in web/stream_client.js.
Finishing cleanly:
- The server sends one last event with
event: done and a data payload of
{"tokens": <number of pieces in the whole draft>, "sha256": "<sha256 hex of the whole draft text>"}.
createState() starts at status: "streaming". A done event puts it in
status: "complete". A stream that ends any other way must not come out as
"complete", and must carry an error the editor can show.
Saving and resuming:
append_chunk(draft_id, text) in store.py, called as pieces arrive, so a
refresh or a dropped connection does not lose text that was already generated.- A request carrying the id of the last event the client received, either in the
Last-Event-ID request header or in a last_event_id query parameter,
continues the same draft. The stored draft ends up exactly what an
uninterrupted run would have stored.
What's here
stream_server.py the streaming endpoint (a WSGI app) and the event format on the wire
model_stream.py drafting model client (simulated, see below). Treat it as external: don't edit it.
store.py where a generated draft is kept, as ordered pieces of text
web/stream_client.js browser side: parse events, fold them into editor state
measure.js replays the captured responses in data/wire through the client
data/drafts.json the draft texts the simulated model writes
data/wire/ two responses captured from the endpoint, plus the draft text they should produce
tests/ existing tests (they pass now; keep them passing)
One event on the wire looks like this, and ends with a blank line:
id: 12
event: token
data: electrode
There is exactly one space after data:, and a reader removes that one space and keeps
everything after it, so a piece that begins with a space (" electrode") survives the trip.
model_stream.py is a deterministic stand-in for the drafting model we call in production.
The same prompt always produces the same pieces, the pieces do not line up with words, and
{"disconnect_after": 12} makes the upstream connection drop part way through a draft, so
both the good path and the bad path can be tested without a network.
Running it
node measure.js # replay the captured responses, print what the editor gets
python3 -m unittest discover -s tests -v # python tests
node --test --no-warnings 'tests/*.test.js' # frontend tests (Node 22.18+)
python3 stream_server.py serve # serve on http://127.0.0.1:8000
curl -N http://127.0.0.1:8000/api/drafts/d-1001/stream # watch the raw events
python3 stream_server.py capture # re-capture data/wire after changing the server
Python 3.11, standard library only. No npm install, no pip install.
Time
Aim for about 30 minutes. You do not need to finish everything. We care more
about how you approach it than about how much you get through.