Streaming responses
LLM generation is slow and sequential, so most apps stream tokens to the user (recap from Foundations): set stream: true and receive incremental chunks over SSE. OpenAI streams delta chunks ending in [DONE]; Anthropic emits typed events (content_block_delta, etc.). Streaming doesn't reduce total time but slashes perceived latency — the user sees output immediately (low TTFT).
Implementation notes:
- Accumulate deltas into the full message on your side (you'll want the complete text + usage at the end).
- Handle interruptions — a client can disconnect mid-stream; clean up.
- Stream through your backend — the client streams from your server (which streams from the provider), so the API key never touches the browser.
- Parsing streamed structured output — streaming JSON is partial until complete; buffer, or use the provider's streamed-object support.
Long-running & async work
Some LLM work is too slow for a request/response cycle (a multi-step agent, a big batch, a report). Don't block an HTTP request for 60s — go async:
- Kick off a job, return
202 Accepted+ a job id immediately, process in the background (a worker/queue), and let the client poll or receive a webhook when done (Foundations webhooks). - Batch APIs — for bulk, non-real-time work (classify a million rows), use the provider batch API: submit a job, it processes asynchronously at a discount, you fetch results later (Cost course).
Choosing the interaction model
- Interactive chat/answer → stream for perceived speed.
- A single quick call → plain request/response.
- Slow multi-step work → async job + poll/webhook, show progress.
- Bulk offline → batch API.
Match the pattern to the latency and never leave a user staring at a spinner with no feedback.