Swap in an open-source model
Context
Our model bill is growing faster than the number of patents we process, and most of the spend
goes on two small jobs. Two places in the product call a hosted model API directly, with
provider-specific request settings written into the app code: one summarizes a patent document
for search results and the drafting sidebar, and one suggests CPC subclass codes for a draft.
We want to run a self-hosted open-source model instead, behind an OpenAI-compatible endpoint
(a local server that speaks the same HTTP API as the hosted provider, the way vLLM and Ollama
do), and keep the hosted model as a fallback. Both models here are deterministic stand-ins that
run offline. They behave like the real ones, including their mistakes.
Your task
- Introduce a small model interface that app code depends on, with two
implementations: an adapter for the hosted client and an adapter for any
OpenAI-compatible
/v1/chat/completions server. After your change, nothing
under app/ should know which provider it is talking to.
- Choose the provider from config read from environment variables:
MODEL_PROVIDER (hosted or local), LOCAL_MODEL_URL (for example
http://127.0.0.1:8089/v1) and MODEL_TIMEOUT_SECONDS. Keep
app.summarize.summarize and app.classify.suggest_codes working with the
same signatures.
- Add a timeout and a fallback: if the local model fails or is too slow, use the
hosted model. Think about which failures should fall back and which should be
surfaced to the caller. Then write a short
NOTES.md, three or four lines: the
decisions you made, and what you would check before switching production
traffic to the local model.
Optional, if you have time: a compare.py that runs the documents in
data/sample_patents.json through both providers and prints latency and how
often the two agree.
Start here
app/summarize.py calls the hosted client directly today
app/classify.py same
hosted_client.py the hosted provider's own request and response shape
local_model_server.py the OpenAI-compatible stand-in, with real rough edges
simulated_models.py is the deterministic behavior behind both stand-ins. You
should not need to edit it.
What's here
hosted_client.py client for the hosted provider (its own request and response shape)
local_model_server.py OpenAI-compatible /v1/chat/completions server standing in for the self-hosted model
simulated_models.py deterministic behavior behind both stand-ins (you should not need to edit it)
app/summarize.py patent document summaries, calls the hosted client directly
app/classify.py CPC subclass code suggestions, calls the hosted client directly
data/sample_patents.json 10 short patent documents to try things against
tests/ current tests (all passing)
The local server has the rough edges of a real deployment: the first request is slow while the
model warms up, it can return 503 when overloaded (fail_every), and it has a small context
window. In tests you can start it on a free port:
import local_model_server
server = local_model_server.start_in_thread(latency=0.0, first_request_latency=0.0)
print(server.url)
local_model_server.stop(server)
Options: latency, first_request_latency, fail_every (503 on every Nth request),
force_status (always return this HTTP status), max_context_chars.
Running it
python3 -m unittest discover -s tests -v
python3 local_model_server.py --port 8089 # run the local model stand-in by hand
Python 3.11, standard library only. No installs needed.
Time
Aim for about 30 minutes. You do not need to finish everything. We care more
about how you approach it than about how much you get through.