Step 2: Answer from your own docs

1. ✓Run a local model
2. 2Answer from your own docs
3. 3Add tools with MCP
4. 4Orchestrate and resume
5. 5Make it production-shaped

In step 1, the assistant answered from what the model already knew. Ask it about your company’s on-call rules, and all it can do is guess. In this step, it reads your documents. It answers from a folder of files, and cites the file each answer came from. It also gets a score you can track as you change it.

You’ll need: step 1 working, then cd ai-assistant/step-2 and pip install -r requirements.txt. The sample documents are a 14-file engineering handbook for a made-up company, in ai-assistant/docs. To try it on your own docs, point the indexer at your own folder of Markdown files.

What you’re adding

1. index.py splits each document into chunks of about 400 tokens. It embeds each chunk with the local nomic-embed-text model, and saves the vectors with their source file.
2. store.py is the vector store: a NumPy matrix searched by exact dot product. At a few thousand chunks, that takes well under a millisecond. Vector search covers what changes at millions.
3. retrieve.py embeds the question, takes the 4 nearest chunks, and builds the prompt.

assistant/retrieve.py

INSTRUCTIONS = (
    "Answer the question using only the sources below. Cite the [file] for each claim. "
    "If the sources don't contain the answer, say you don't know."
)

def search(question, store, llm, k=config.TOP_K):
    # Embed the question with the SAME model that embedded the chunks. Vectors from two
    # different models live in different spaces, and their distances mean nothing.
    query = llm.embed([f"search_query: {question}"])[0]
    return [chunk for chunk, _ in store.nearest(query, k)]

def build_prompt(question, hits):
    # Each chunk carries its source file, so the model can cite it and the reader can check it.
    context = "\n\n".join(f"[{h.source}]\n{h.text}" for h in hits)
    return [
        {"role": "system", "content": INSTRUCTIONS},
        {"role": "user", "content": f"Sources:\n\n{context}\n\nQuestion: {question}"},
    ]

Index the docs, then ask something only they know:

python -m assistant.index ../docs
python -m assistant.ask "How long are audit logs kept?"

Measure it

Measure

Score it on 20 questions

ai-assistant/evals/questions.json holds 20 questions about the handbook. Each question comes with the file that holds the answer. It also lists the facts a correct answer must contain, as patterns such as 400.

python -m assistant.evals --show

It reports two numbers, because they fail for different reasons:

• Retrieval hit rate: was the file with the answer among the 4 chunks in the prompt?
• Answer accuracy: did the answer contain the required facts?

python -m assistant.experiment ../docs runs every setup on this page and prints the results. These are the real numbers, on 2 CPU cores with qwen2.5:3b and nomic-embed-text:

Setup	Chunks	Retrieval hit rate	Answer accuracy
As shipped: 400 tokens, no overlap	14	20 / 20	19 / 20

Retrieval is perfect: the right file was in the prompt for every question. The one miss is a generation error. The question asked how long a team has to finish SEV2 postmortem action items. The model wrote that “the source does not specify the completion time for SEV2”. The 60-day rule was in the prompt. The model read the SEV1 line next to it and stopped.

We ran the whole experiment twice, on two machines. Every count on this page, and the stale-index answers below, came out the same both times.

That split is why you track two numbers. A retrieval miss means you fix chunking or search. A generation miss with a perfect retrieval means the facts were there, and you fix the prompt or the model. This one is the model.

The handbook is small: 14 files and about 2,200 words. At 400 tokens, almost every file fits in one chunk. That is why there are only 14 chunks, and why retrieval has an easy job here.

Break it

Break it

Shrink the chunks to 100 tokens

python -m assistant.index ../docs --chunk-tokens 100
python -m assistant.evals --show

Setup	Chunks	Retrieval hit rate	Answer accuracy
As shipped: 400 tokens	14	20 / 20	19 / 20
Small: 100 tokens	37	20 / 20	17 / 20

The hit rate didn’t move. The right file was still in the prompt every time. But accuracy dropped from 19 to 17, with three new misses. The hit rate counts files, not facts, so it can’t see this. Here is the clearest miss:

Q: How long are application logs kept in total before they are deleted?

A: Application logs are kept for 30 days in the hot store… After 30 days, they are deleted.

The handbook says 30 days in the hot store, then 150 more days in cold storage, 180 in total. At 100 tokens, the cut fell between those two sentences. The model got the first half of the rule and answered confidently with it.

The other two misses look like the same problem, though we didn’t check where each cut fell. Asked who gets paged if nobody acknowledges, the model answered with a nearby rule about the alert channel, not the engineering manager. Asked if a rollback needs approval during the December freeze, it said yes. The freeze rule says deploys need a director’s approval. The next section says rollbacks never do. The model got the rule without its exception.

Break it

Change a document and don't re-index

The experiment copies the handbook and changes one fact in logging-and-retention.md: audit logs are now kept for 120 days, not 400. Then it asks “How long are audit logs kept?” before and after re-indexing that one file.

	Answer
Before re-indexing	“Audit logs are kept for 400 days. This information is found in the [backups.md] file.”
After re-indexing	“Audit logs are kept for 120 days. This information is directly stated in the [access-and-security.md] source.”

Before re-indexing, the answer is wrong, and nothing tells you so. The file says 120, but the index still holds the old chunk, and that is all the model sees.

Look at the citations too. Both answers name the wrong file. The fact is in logging-and-retention.md. The number was right after re-indexing, but the citation was still wrong. A small model often cites a file that was in the prompt, not the one it used. If citations matter to your users, check them in your eval, not just the facts.

Fix it

Fix it

Overlap the chunks, and re-index on change

Small chunks broke because a rule was cut in half. Overlap fixes that. Each chunk repeats the last 30 tokens of the one before it. So a sentence near a cut appears whole in at least one chunk.

Setup	Chunks	Retrieval hit rate	Answer accuracy
Small: 100 tokens	37	20 / 20	17 / 20
Small with overlap: 100 tokens, 30 overlap	44	20 / 20	20 / 20
Large with overlap: 400 tokens, 60 overlap	14	20 / 20	19 / 20

Small chunks with overlap got all 20 right, the best result on this page. Overlap made no difference at 400 tokens. At that size, almost every file is already one chunk, so there are almost no cuts to overlap. It is the same 19 out of 20, with the same SEV2 generation miss.

So the measured choice for this handbook is 100 tokens with 30 of overlap. It costs 44 chunks instead of 14, which is nothing here. For a big corpus, more chunks means a bigger index and more embedding calls, which is the trade you measure there.

For freshness, run the indexer in watch mode. It re-embeds a file when its modified time changes, replacing that file’s old chunks:

python -m assistant.index ../docs --chunk-tokens 100 --overlap 30 --watch

A production indexer listens for change events from the source system, instead of polling. But the rule is the same: re-index the document that changed, not everything.

Common mistake

Judging a RAG system by trying a few questions by hand

After the chunk-size change above, most answers read almost the same, and the hit rate didn’t move at all. The damage only shows up as a count of right answers on a fixed set. So keep the eval set, and run it after every change. Then read the misses. Each one tells you whether to fix retrieval or generation.

Checkpoint

You now have

• An assistant that answers from your docs and cites its sources
• An eval set and two numbers to track: retrieval hit rate and answer accuracy
• Chunking chosen by measurement, and an index that stays fresh when files change

The finished code is in ai-assistant/step-2. This step puts into practice: Embeddings and vector search, Retrieval-augmented generation, Evaluation and guardrails.

Next, in step 3, the assistant stops only answering and starts doing things, through tools you expose with your own MCP server.