Step 1: Run a local model

Run an open model on your laptop with Ollama, put it behind a streaming Python API, and measure time to first token, decode speed, concurrent users and context overflow. Then break the context window on purpose and fix it with a token budget.

Foundation35 min read

Read first: How LLM inference works

Every AI feature starts with a model answering a request. In this step you run one on your own machine. You put it behind an API that streams its answer, the way chat products do. Then you measure the numbers that the inference page talks about:

How long until the first word.
How fast the words arrive.
What happens when several people ask at once.

You’ll need: the lab code (a zip, about 140 KB). Unzip it and follow the setup in its README (Ollama, two models, Python 3.11, about 10 minutes). Then cd ai-assistant/step-1.

What you’re adding

Three small pieces, all in assistant/:

llm.py, a client for Ollama’s HTTP API. It streams the answer token by token. It also records what each call cost: prompt tokens, output tokens, time to first token, and decode speed.
server.py, a FastAPI endpoint that streams the answer to the caller as server-sent events.
budget.py, which keeps a conversation inside the context window on purpose.

The client is the core. While Ollama generates, it sends one JSON object per line. The last one carries its own timings.

def stream_chat(self, messages, options=None):
    body = {
        "model": self.model,
        "messages": messages,
        "stream": True,
        # Set the context window explicitly: the default has changed between Ollama releases,
        # and every number in this lab depends on it.
        "options": {"num_ctx": config.NUM_CTX, **(options or {})},
    }
    start = time.perf_counter()
    stats = Stats()
    with self.http.stream("POST", "/api/chat", json=body) as response:
        for line in response.iter_lines():
            part = json.loads(line)
            piece = part.get("message", {}).get("content", "")
            if piece:
                # Time to first token is measured on our side, because that's what a user feels:
                # it includes the network, the queue and the prefill.
                if not stats.ttft:
                    stats.ttft = time.perf_counter() - start
                yield piece
            if part.get("done"):
                _fill(stats, part)  # Ollama's own prefill and decode timings
    yield stats

Try it in the terminal, then as an API:

python -m assistant.chat
uvicorn assistant.server:app --port 8000
curl -N localhost:8000/chat -H 'content-type: application/json' -d '{"message": "What is a KV cache?"}'

After every answer, the chat prints three numbers:

How many prompt tokens the model read.
The time to the first token.
The decode speed, in tokens per second.

The rest of this step is about those three numbers.

Measure it

Measure

Run the benchmark

python -m assistant.bench

It runs four experiments and prints each as a table. Our numbers come from a GitHub-hosted machine running qwen2.5:3b. It has 2 CPU cores, 7 GB of RAM and no GPU. A laptop with Apple silicon or a GPU will be several times faster. So compare the shape, not the values.

Prompt length. We sent the same one-sentence summary request three times. In front of it was 100, 1,000 or 3,000 tokens of text.

We ran the benchmark twice, on two machines of the same type. Both runs are shown, because they differ more than you might expect.

Prompt tokens	Time to first token (run 1 / run 2)	Prefill speed	Decode speed
about 130	7 s / 5 s	19 / 28 tokens/s	8.6 / 4.3 tokens/s
about 925	60 s / 39 s	15 / 24 tokens/s	7.5 / 4.0 tokens/s
about 2,745	194 s / 127 s	14 / 22 tokens/s	5.7 / 3.3 tokens/s

The same prompt twice. Ollama keeps the KV cache of recent prompts. Send an identical 3,000-token prompt a second time:

Request	Prompt tokens	Time to first token (run 1 / run 2)
First time	2,744	194.5 s / 126.9 s
Same prompt again	2,744	0.2 s / 0.3 s

Several users at once. 1, 2, 4 and 8 people each ask for about 100 words at the same moment. Ollama is allowed to serve 4 requests in parallel.

Users	Median time to first token (run 1 / run 2)	Worst	Speed each user sees	Total speed
1	2.4 s / 1.8 s	2.4 s / 1.8 s	8.7 / 4.3 tokens/s	7.0 / 3.9 tokens/s
2	4.1 s / 2.9 s	4.1 s / 2.9 s	4.5 / 2.5 tokens/s	7.3 / 4.5 tokens/s
4	8.0 s / 5.8 s	8.0 s / 5.8 s	2.4 / 2.5 tokens/s	7.8 / 7.5 tokens/s
8	21.5 s / 22.2 s	52 s / 51 s	2.3 / 2.5 tokens/s	7.6 / 8.1 tokens/s

These tables hold three lessons. They are the same lessons the inference page explains, now with your own numbers. First, a warning about the numbers themselves.

Same hardware type, different numbers. Both runs used “2 CPU cores, 7 GB”. Yet run 2 read prompts about 50% faster, and wrote answers about half as fast. Cloud machines of one type are not identical, and laptops vary even more. So never compare your numbers with ours. Compare the shape: how a number changes when the prompt grows, or when users are added.

The prompt is the slow part. The prompt grew from about 130 to about 2,745 tokens. On both machines, the wait for the first word grew from a few seconds to two or three minutes. The model must read every prompt token before it writes anything. That reading is called prefill. On a CPU it is slow, and it grows with the prompt.

Decode also slows down as the prompt grows. Each new token looks back at everything before it. This is why prompt size is the first thing to watch in every later step. The retrieved chunks, the tool results and the chat history all go into the prompt.

A repeated prefix is almost free. The same 2,744-token prompt took two to three minutes the first time. The second time it took well under a second. Ollama kept the work from the first request (the KV cache) and reused it. Hosted APIs do the same thing, and charge less for cached tokens. To get this for free, put the parts that don’t change at the start of the prompt: system instructions, tool lists, documents. Put the parts that change at the end.

More users share a fixed total. On both machines, the total speed topped out at about 8 tokens a second.

On run 1, one user already used almost all of it. So more users only split it: two users got half each.
On run 2, one user used only half. So serving 4 at once filled the spare capacity, and the total doubled.

Either way, past the ceiling, each user gets slower. With 8 users and only 4 served at once, the other 4 waited in a queue. On both machines, the worst one waited about 52 seconds for the first word. A GPU has a much higher ceiling, because it can batch many users into one pass. Your laptop’s numbers tell you where your ceiling is.

Break it

Talk past the context window

Start the chat with no budget, tell it something, then keep talking:

python -m assistant.chat --no-budget

The benchmark’s overflow experiment does this automatically. It tells the model a code word, adds long turns of conversation, then asks for the code word back.

Turns	Tokens sent (our estimate)	Tokens the model read	Remembered the code word?
2	874	790	Yes
6	2,542	2,242	Yes
12	5,044	4,030	No

At 12 turns, the conversation was about 5,000 tokens, but the model read only 4,030. The window is 4,096 tokens here. The first thousand tokens were gone, and the code word was in them.

(Our estimate is a little high. It counts about 4 characters per token, and the real tokenizer packs this text slightly tighter. That is why “sent” and “read” differ even at 2 turns.)

When a prompt doesn’t fit, Ollama drops the oldest part of it to make room. It says nothing. The model then answers as if your first messages were never sent. Nothing errors. The answer is simply wrong.

Fix it

Trim on purpose, and pin what matters

budget.py makes the same choice Ollama made, but explicitly:

Keep the system prompt, and any message marked as pinned.
Keep the newest turns that fit.
Drop the rest, and say what was dropped.

def fit(messages, limit, reserve=512):
    # Leave room for the answer. A prompt that fills the window exactly leaves the model no
    # space to reply, and the answer is cut off mid-sentence.
    budget = limit - reserve
    keep = [m.get("role") == "system" or m.get("pinned") for m in messages]
    used = sum(estimate_tokens(m["content"]) for m, k in zip(messages, keep) if k)
    # Walk backwards: the newest turns are what the user is asking about now.
    for i in range(len(messages) - 1, -1, -1):
        if keep[i]:
            continue
        cost = estimate_tokens(messages[i]["content"])
        if used + cost > budget:
            break
        keep[i] = True
        used += cost
    return [m for m, k in zip(messages, keep) if k]

In the chat, start a message with /pin to keep it. With the budget, and the code word pinned:

Turns	Tokens the model read	Remembered the code word?
12, no budget	4,030	No
12, budget and pin	2,960	Yes

With the budget, the model read fewer tokens and remembered more. The budget dropped old turns in the middle of the conversation. It kept the newest ones, and the pinned code word. You decide what is lost, not the server. Real products do the same. Some keep a summary of the dropped turns. Others store facts outside the prompt, as step 2 does with documents.

Checkpoint

You now have

A local model behind a streaming API, with no API keys and no cloud bill
A benchmark that measures time to first token, decode speed and concurrency on your machine
A token budget that drops old turns on purpose and keeps pinned ones

The finished code is in ai-assistant/step-1. This step puts into practice: LLM inference.

Next, in step 2, the assistant stops answering from memory. It starts answering from your documents.