Swap in an open-source model
Context
Our model bill is growing faster than the number of patents we process, and most of the spend
goes on two small jobs. Two places in the product call a hosted model API directly, with
provider-specific request settings written into the app code: one summarizes a patent document
for search results and the drafting sidebar, and one suggests CPC subclass codes for a draft.
We want to run a self-hosted open-source model instead, behind an OpenAI-compatible endpoint
(a local server that speaks the same HTTP API as the hosted provider, the way vLLM and Ollama
do), and keep the hosted model as a fallback. Both models here are deterministic stand-ins that
run offline. They behave like the real ones, including their mistakes.
Your task
- Introduce a small model interface that app code depends on, with two implementations: an
adapter for the hosted client and an adapter for any OpenAI-compatible
/v1/chat/completions server. After your change, nothing under app/ should know which
provider it is talking to.
- Choose the provider from config read from environment variables:
MODEL_PROVIDER
(hosted or local), LOCAL_MODEL_URL (for example http://127.0.0.1:8089/v1) and
MODEL_TIMEOUT_SECONDS. Keep app.summarize.summarize and app.classify.suggest_codes
working with the same signatures.
- Add a timeout and a fallback: if the local model fails or is too slow, use the hosted model.
Think about which failures should fall back and which should be surfaced to the caller.
- Write
compare.py that runs the documents in data/sample_patents.json through both
providers and prints, for each one, latency p50 (the middle call), latency p95 (the time the
slowest 5 percent of calls take) and how often the two providers agree. Record the numbers
you get before you change anything, so you can say what moved.
- Create a short
NOTES.md (5-10 lines): the decisions you made, and what you would check
before switching production traffic to the local model.
What's here
hosted_client.py client for the hosted provider (its own request and response shape)
local_model_server.py OpenAI-compatible /v1/chat/completions server standing in for the self-hosted model
simulated_models.py deterministic behavior behind both stand-ins (you should not need to edit it)
app/summarize.py patent document summaries, calls the hosted client directly
app/classify.py CPC subclass code suggestions, calls the hosted client directly
data/sample_patents.json 10 short patent documents to use in compare.py
tests/ current tests (all passing)
The local server has the rough edges of a real deployment: the first request is slow while the
model warms up, it can return 503 when overloaded (fail_every), and it has a small context
window. In tests you can start it on a free port:
import local_model_server
server = local_model_server.start_in_thread(latency=0.0, first_request_latency=0.0)
print(server.url)
local_model_server.stop(server)
Options: latency, first_request_latency, fail_every (503 on every Nth request),
force_status (always return this HTTP status), max_context_chars.
Running it
python3 -m unittest discover -s tests -v
python3 local_model_server.py --port 8089 # run the local model stand-in by hand
Python 3.11, standard library only. No installs needed.
Time
Aim for about 25-30 minutes. You don't need to finish everything; we care more about how you approach it than about completeness.