Step 5: Make it production-shaped

Put the local assistant behind a production-shaped API with an exact and a semantic cache, a token-based rate limit, per-request traces and cost, and an eval gate for CI. Measure cache hit rates on realistic traffic, then watch the gate block a smaller model.

Senior40 min read

Read first: Evals and guardrails How LLM inference works

The assistant works. Before other people depend on it, it needs four more things:

Answers that don’t cost a model call when the question was just asked.
A limit that stops one caller from using all the capacity.
A trace of every request, with its cost.
A gate that stops a change from shipping if it makes answers worse.

You’ll need: step 2 working (this step builds on the RAG assistant, not the agent), then cd ai-assistant/step-5, pip install -r requirements.txt, python -m assistant.index ../docs, and for the gate exercise ollama pull qwen2.5:0.5b.

What you’re adding

service.py: Assistant.ask(), the one path every request takes. It embeds the question, checks the cache, retrieves, generates, and stores the answer in the cache. Every stage is a timed span in a trace.
cache.py: two caches.
- An exact cache, keyed on the normalized question.
- A semantic cache, which also matches questions whose embeddings are close enough.
Every key includes a version: the model, the prompt, and a hash of the indexed text. So an answer from old docs or an old model can never be served.
ratelimit.py: a token bucket per API key. It counts model tokens, not requests. It checks an estimate before the call, and charges the real count after.
api.py: FastAPI with an API key header. It returns 429 Too Many Requests with Retry-After, and puts the trace id and cost in each response.
gate.py: runs the 20-question eval, and exits non-zero below a bar. CI uses it.

def ask(self, question):
    trace = Trace(self.trace_file, model=self.llm.model, cache_version=self.cache.version)
    with trace.span("embed_question"):
        vector = self.llm.embed([f"search_query: {question}"])[0]
    with trace.span("cache") as span:
        hit = self.cache.get(question, vector)   # exact first, then nearest cached question
        span["hit"] = hit is not None
    if hit:
        return Answer(hit.answer, hit.sources, cached=True, ...)
    with trace.span("retrieve"):
        hits = [c for c, _ in self.store.nearest(vector, config.TOP_K)]
    with trace.span("generate") as span:
        reply = self.llm.chat(build_prompt(question, hits), options={"temperature": 0})
        span.update(prompt_tokens=reply.stats.prompt_tokens, output_tokens=reply.stats.output_tokens)
    self.cache.put(question, Entry(reply.text, sources, time.time(), vector))
    ...

uvicorn assistant.api:app --port 8000
curl localhost:8000/ask -H 'x-api-key: team-a' -H 'content-type: application/json' \
     -d '{"question": "How long are audit logs kept?"}'

Measure it

Measure

Replay realistic traffic through four caches

python -m assistant.experiment ../docs

The experiment builds 100 requests that look like real traffic. A few questions are asked over and over, and a long tail is asked once. (This shape is called a Zipf distribution.) The requests are drawn from three groups:

The 20 eval questions.
5 paraphrases of them, like “Can we ship on a Friday?”
5 look-alikes. These share words with an eval question but ask something else, like “How long are application logs kept in the hot store?”

It replays the requests four ways: with no cache, with an exact cache, and with semantic caches at similarity thresholds of 0.90 and 0.95. Cost uses $0.15 per million input tokens and $0.60 per million output tokens. These prices are typical for a small hosted model.

These are the real numbers, on 2 CPU cores with qwen2.5:3b:

Setup	Cache hits	Answers made for another question	Median time	95th percentile	Tokens per request	Cost per 1,000 requests
No cache	0	0	6,622 ms	57,886 ms	961	$0.162
Exact cache	77	0	73 ms	11,950 ms	216	$0.037
Semantic 0.90	79	6	70 ms	13,454 ms	199	$0.034
Semantic 0.95	77	0	73 ms	10,486 ms	216	$0.037

Of the 100 requests, 32 were paraphrases and 16 were look-alikes. A second run on another machine had the same hit counts. Only the times changed: its median with no cache was 3.4 seconds.

The exact cache does almost all the work. Real traffic repeats itself. 77 of 100 requests were answered from the cache, so the cost fell by 77%, from $0.162 to $0.037 per 1,000 requests. The median request went from 6.6 seconds to 73 milliseconds. A hit skips retrieval and the model, and only embeds the question. This is the cheapest win in the whole lab, and it is a dictionary.

The slowest requests are still slow. The 95th percentile went from 58 seconds to about 12. But that is still more than 150 times the median. The slow requests are the misses, and a cache does nothing for a miss. If your slowest users matter, you still need a faster model or more capacity. A cache improves the average, not the worst case.

The semantic cache added little here. At 0.90, it found 2 more hits than the exact cache. And 6 of its hits were answers made for a different question. None of those 6 went to a look-alike, so no wrong answers were served. That sounds like a safe threshold. It isn’t, and the next measurement shows why.

The experiment embeds every paraphrase and look-alike, and finds its nearest eval question. The score is the cosine similarity, where 1.0 means the same direction:

Kind	Question	Nearest eval question	Score
Paraphrase	What rate limit does the public API have by default?	What is the default public API rate limit per API key?	0.949
Paraphrase	SSH key rotation interval?	How often are SSH keys rotated?	0.908
Look-alike	What is the recovery time objective for a SEV1?	What is the recovery point objective for a SEV1…?	0.896
Paraphrase	What’s the retention period for audit logs?	How long are audit logs kept?	0.882
Look-alike	How quickly must the secondary acknowledge?	How quickly must the on-call primary acknowledge?	0.873
Look-alike	…a change to the search service need?	…a change to the billing service need?	0.871
Look-alike	How long are app logs kept in the hot store?	How long are app logs kept in total…?	0.821
Paraphrase	Can we ship on a Friday?	Are deploys allowed on Fridays?	0.728
Paraphrase	how fast do I need to ack a page	How quickly must the on-call primary acknowledge a page?	0.703

The look-alikes and the paraphrases are mixed together. One look-alike changes a single important word: “time” instead of “point”. It scores 0.896. That is higher than a true paraphrase of the audit log question (0.882). So no threshold catches the paraphrases and also blocks the look-alikes.

At 0.90, the cache was 0.004 away from a wrong answer. It nearly served the recovery point answer to someone who asked about recovery time. And only two of the five paraphrases scored above 0.90. At 0.95, it behaved exactly like the exact cache, because in the replay no paraphrase came that close.

So for this kind of traffic, the semantic cache adds risk for a tiny gain: 2 extra hits out of 100. The safe choice is the exact cache. Or use a semantic cache with a second check, such as a cheap model that confirms the two questions really ask the same thing.

Measure

A burst against the rate limit

One API key with 6,000 tokens a minute sends 30 requests at once. Each is estimated at 900 tokens.

Requests sent at once	Allowed	Refused with `429`
30	6	24

The bucket holds 6,000 tokens. Each request is estimated at 900, so 6 fit (5,400), and the seventh does not. The bucket refills at 100 tokens a second. So a refused caller is told to retry after about 3 seconds. From then on, it gets about one request every 9 seconds.

Counting tokens instead of requests matters. One caller sending 30 short questions and one sending 30 long documents are not the same load.

Break it

Swap in a smaller, faster model

A smaller model is tempting. qwen2.5:0.5b is a sixth of the size, and much faster. Point the assistant at it, and run the gate the way CI would:

CHAT_MODEL=qwen2.5:0.5b python -m assistant.gate --min-accuracy 0.85

We ran the gate on both models, in two runs. The results were the same both times:

Model	Retrieval hit rate	Answer accuracy	Gate
`qwen2.5:3b`	20 / 20	19 / 20 (95%)	Passed
`qwen2.5:0.5b`	20 / 20	15 / 20 (75%)	Failed

Retrieval is identical, because the search doesn’t use the chat model. The right documents were in the prompt every time. The smaller model just read them worse:

“Are deploys allowed on Fridays?” It answered “Yes, deploys are allowed on Fridays.” The handbook says only for fixes to an open SEV1 or SEV2.
“What share of traffic does the first deploy wave get, and for how long?” It answered “10% of traffic, for 10 minutes.” Both numbers are wrong.
“How long are application logs kept in total?” It answered 400 days, the number for audit logs.
“What is the default public API rate limit per API key?” It answered with only a file name.

Each answer sounds confident. If you tried the small model on three or four questions by hand, you could easily miss this.

Fix it

Let the gate decide

There is nothing to fix in the code. The fix is the gate itself. It exits with code 1, so CI fails and the change does not ship. That is what happened here, with no human in the loop.

The 3B model passed every time. In two runs, its one miss was the same SEV2 question it missed in step 2. In the third, it got all 20.

The bar is a choice. We set 85%, because the 3B model scores 95%. That leaves room for one or two unlucky answers without blocking every change. Set the bar from your own baseline, and raise it as the eval set grows. Twenty questions is a small set: one question is 5 percentage points.

- run: python -m assistant.index ../docs
- run: python -m assistant.gate --min-accuracy 0.85 # a non-zero exit fails the build

Checkpoint

You now have

A service with an exact and a semantic cache, versioned so stale answers can’t be served
A token-based rate limit with Retry-After
A trace and a cost for every request
An eval gate that fails CI when answers get worse

The finished code is in ai-assistant/step-5. This step puts into practice: Evaluation and guardrails, LLM inference, and from the System Design track, caching and rate limiting.

That’s the whole project. You built a local model, answers from your documents, tools over MCP, an orchestrator that survives crashes, and the production pieces around it. Each concept page in Foundations and Agents points back to the step where you saw it work.