Retrieval-augmented generation

Retrieval-augmented generation (RAG) answers a question in two moves:

1. Find the few passages in your own documents that matter.
2. Give them to the model with the question, and ask it to answer from them.

The model supplies the language and the reasoning. Your documents supply the facts. Almost every “chat with your docs” product is RAG. And almost every one that gives bad answers is failing at the first move, not the second.

The problem it solves

A model knows what was in its training data, up to a cutoff date. It knows nothing about your company’s on-call policy, last week’s pricing change, or a customer’s order history. When asked, it often invents an answer that sounds right, instead of saying it doesn’t know.

There are three ways to give it that knowledge:

	Fine-tune on your docs	Put all your docs in the prompt	Retrieve the relevant passages (RAG)
Facts stay current	No: retrain for every change	Yes	Yes: re-index the changed file
Cites sources	No	Weakly	Yes: every passage carries its source
Cost per question	Low	Very high: every token of every doc, every time	Low: a few thousand tokens
Works for 10 GB of docs	Partly	No: far past any context window	Yes
Respects who may see what	No	No	Yes: filter at retrieval

Fine-tuning teaches behavior and style. It does not teach facts the model can reliably recall. Long context works when the material is small and stable. But say you have a large, changing set of documents, and different users may see different parts. Then retrieval is the answer.

How it works

RAG is two pipelines. An offline one prepares the documents. An online one answers each question.

The two halves of a RAG system

Scroll or pinch to zoom, drag to pan

• Color key
• Service
• Database

Step 1: Load and clean

Documents arrive as HTML pages, PDFs, slides, tickets and chat threads. Each one has to become clean text:

• Remove navigation and footers.
• Keep tables as tables, and headings as headings.
• Run scanned PDFs through OCR (text recognition).

Every passage also carries metadata: the source URL, title, section, last-modified time, and who may read it. The metadata is what makes citations, freshness and access control possible later. Once thrown away, it can’t be recovered.

This step is dull, but it decides a large share of answer quality. Say a table was flattened into a run of numbers. Or a PDF’s two columns were mixed together line by line. Nothing later in the pipeline can retrieve or read that text correctly.

Step 2: Chunk

Documents are split into chunks. Each chunk gets one embedding (see vector search), and one vector can only represent one topic well. Chunk size is a real tradeoff:

• Too small (a sentence or two): matches are precise, but facts get cut off from their context. “They are kept for 400 days” lands in one chunk, and “Audit logs are different” in the one before.
• Too large (several pages): the right section is in there, but everything else blurs its vector. It ranks lower, and it fills the prompt with text that isn’t the answer.

Common defaults:

• 200 to 800 tokens per chunk.
• Split on structure (headings, paragraphs), not at a fixed character count.
• A small overlap between chunks, so a sentence near a boundary appears whole in at least one chunk.

Also start each chunk with its document title and section heading, like “Logging and data retention

Retention periods”. That helps both retrieval and the model’s reading.

Try it below. Ask about audit logs with 40-word chunks and 2 chunks in the prompt, then shrink the chunks to 15 words, then add overlap.

Chunk size 40 wordsOverlap 0 wordsChunks in the prompt (top k) 2

The document, split into chunks. Retrieved chunks are highlighted, numbered by rank.

What the model receives

At 15 words with no overlap, “Audit logs are different” and “They are kept for 400 days” fall into separate chunks. Only one of them may be retrieved, so the model sees a number with no subject.

Overlap, a larger size, or more chunks in the prompt each fix it. Each costs some index size or prompt size. No setting is right everywhere, so you choose it by measuring (Step 6).

Step 3: Embed and index

Each chunk is embedded and stored with its metadata in a vector index. Usually a keyword index of the same chunks sits next to it. Store two more things per chunk:

• A content hash, so loading an unchanged document again skips the embedding cost.
• The embedding model’s name, so you can tell when the model changes.

Step 4: Retrieve

At question time:

1. Rewrite the question when it depends on the conversation. “What about for EU customers?” means nothing on its own. A small model call turns it into “What is the audit log retention period for EU customers?” before searching.
2. Search with the user’s permissions. Apply the access filter inside the search (see vector search). Then text the user can’t read never reaches the prompt.
3. Use hybrid search. Vectors find meaning. Keywords find exact terms, like error codes and names. Merge the two lists with reciprocal rank fusion, which combines them by rank.
4. Rerank. A cross-encoder reads the question and each of the top 50 candidates together, scores them, and keeps the best 5 or so.

Step 5: Build the prompt and generate

The prompt is instructions, then sources, then the question:

Answer the question using only the sources below. Cite the [source] for each claim.
If the sources don't contain the answer, say you don't know.

[on-call.md > Paging] Alerts page through PagerDuty. The primary must acknowledge ...
[incident-severity.md > SEV1] Customer-facing outage or data loss affecting more than 10% ...

Question: How fast must the primary acknowledge a SEV1 page?

The two sentences of instructions do most of the work. “Only the sources below” stops the model from mixing in what it remembers. “Say you don’t know” gives it a way out other than guessing. Test that second one, because some models still guess.

Order and amount matter. Models pay most attention to the start and end of a long context, and can miss material in the middle. So put the best-ranked passages first. And send 3 to 8 good passages, not 30 mediocre ones. Every extra passage costs tokens and can distract the model.

Step 6: Cite, and measure

Each claim in the answer should point to its source, shown as a link the user can check. Then a wrong answer is no longer a silent failure. Users can catch it and report it.

Then measure the two halves separately, because they fail for different reasons:

• Retrieval: take real questions whose answering passages you know. How often does the right passage reach the prompt? This is the hit rate, or recall at k.
• Generation: given the right passages, is the answer correct? Is every claim backed by the sources? The second check is called faithfulness.

A low retrieval score means fixing chunking, embeddings, hybrid search or reranking. A good retrieval score with wrong answers means fixing the prompt or the model. Without this split, every bad answer looks like “the model is dumb”. See evaluation for how to build the question set.

Options and tradeoffs

Retrieve once, or let the model search

The pipeline above retrieves once per question. The alternative is to give the model a search tool and let it decide what to search for, maybe several times. That makes it an agent.

TradeoffSingle-shot retrieval as the default, agentic search for multi-part questions

We gain

Single-shot is one search and one model call. Latency (1 to 3 seconds) and cost are predictable, and it is easy to evaluate.

We give up

It can’t answer questions that need two lookups, where the second depends on the first: “Who owns the service that had last week’s SEV1?” Agentic search handles those, at several times the latency and cost.

Long context instead of retrieval

Say all your documents together are a few hundred pages, and they rarely change. Then putting them all in the prompt, with prefix caching, can beat RAG on quality. Retrieval can’t miss anything, because there is no retrieval.

As the documents grow, this loses on cost per question, latency and access control. Many systems combine the two. They retrieve whole sections or documents instead of small chunks, and let a long-context model read them.

How it fails

Most RAG failures happen before the model sees anything. Here they are, roughly from most to least common:

Failure	What the user sees	Fix
The right passage isn’t retrieved	“I don’t know”, or a guess	Hybrid search, reranking, query rewriting, better chunking
The fact is split across chunks	A confident answer about the wrong subject	Structure-aware chunks, overlap, section headers in chunks
The index is stale	A confident, cited, out-of-date answer	Re-index on change; show “last updated” with citations
Two versions of a document disagree	Either answer, depending on ranking	Index only the current version; prefer newer by metadata
The model ignores the sources	An answer from memory, no citation	Firmer instructions, check every answer has a citation
Text the user may not see is retrieved	A data leak	Permission filter inside the search, never in the prompt
A document contains instructions	The model obeys them	Treat retrieved text as data; see guardrails

Common mistake

Asking the model to respect access control

“Only use documents the user is allowed to see” in a prompt is not access control. The model sees everything in the prompt. It can quote it, and it can be talked into it. Enforce permissions in retrieval, before any text reaches the prompt. That is exactly how a database query filters rows before returning them.

At scale

Take a company with 5 million documents, about 50 million chunks, and 20,000 questions an hour at peak.

Ingestion is a pipeline, not a script. Sources publish change events: webhooks, change data capture, or a crawler’s diff. A queue feeds embedding workers. The workers upsert each chunk (insert it, or replace it if it exists), keyed by document id and chunk number. So a retry or a re-run does no harm (see idempotency). Deleting a document deletes its chunks. A freshness target, like “changes searchable within 10 minutes”, is monitored like any other latency.

Re-embedding is a migration. 50 million chunks of 400 tokens is 20 billion tokens. A hosted embedding model costs a few cents per million tokens. So that’s hundreds of dollars, and a day or more of work if rate limits allow. Plan it as a migration: write to a new index alongside the old one, then switch over. Don’t update in place.

The latency budget for an interactive answer is 2 to 4 seconds. It splits roughly like this:

• Query rewrite: 200 to 400 ms (a small model).
• Hybrid search: 30 to 100 ms.
• Reranking: 50 to 200 ms.
• Time to first token: 300 to 1,000 ms, then streaming.

Skip the rewrite for the first question in a conversation. Cache retrieval results for repeated questions.

Multi-tenancy. With thousands of customers, give each customer its own namespace in the vector store. That gives hard isolation and simple deletes (“remove all our data”). It also avoids filter starvation, where a strict filter leaves the search too few results. Very large customers get their own shards.

In the interview

Interviewer asksOur RAG bot gives wrong answers. How do you find out why?

Split the pipeline and measure each half. Build a set of 50 to 200 real questions, with the passages that answer them. Check retrieval first. If the right passage isn’t in the top k, it’s a retrieval problem: chunking, embeddings, no keyword search, no reranker, or a stale index. If it is there and the answer is still wrong, it’s generation: the prompt, the model, or too much context. Look at 20 failures by hand before changing anything. The fix is usually obvious once you’ve seen them.

Interviewer asksDocuments change every few minutes. How do you keep answers fresh?

Event-driven ingestion. The source system emits a change event. A worker chunks and embeds only that document again, upserts its chunks by id, and deletes chunks that no longer exist. Track the lag from change to searchable as a metric, with an alert. Show the source’s last-updated time next to each citation, so a user can see when an answer is based on an older version.

Interviewer asksWhy not just fine-tune the model on all our documents?

Fine-tuning changes how a model behaves, not reliably what it can recall. Facts learned that way can’t be cited, and they blend together. They go stale with every document change. And they can’t be limited per user. Fine-tuning can help RAG, for example by teaching a small model your answer format or how to cite. But the facts should come from retrieval.

Mid-level covers

Describes chunk, embed, retrieve, and prompt with sources. Asks the model to cite, and to say when it doesn’t know.

Senior adds

Designs ingestion as an idempotent, event-driven pipeline with metadata and permissions. Chooses chunking by measurement. Uses query rewriting, hybrid search and reranking. Enforces access control in retrieval. Evaluates retrieval and generation separately. Plans freshness and re-embedding. Names prompt injection through documents as a risk.

Try it on your laptop

Step 2: Answer from your own docs →

Build RAG over a folder of docs on your laptop, score it on 20 questions, then watch a chunk size change move the score and a stale index give a confident wrong answer.

Recap

The idea. Find the relevant passages, put them in the prompt, answer only from them, cite them.

Offline. Load and clean, keeping metadata (source, time, permissions). Chunk by structure at 200 to 800 tokens, with some overlap. Embed, and index with a content hash and the model version.

Online. Rewrite follow-up questions. Run hybrid search with the permission filter inside. Rerank to the best few, and put the best first. Tell the model to cite, and to say “I don’t know”.

Measure. Retrieval (did the right passage arrive?) separately from generation (was the answer right and supported?).

Biggest risks. Retrieval misses. Facts split across chunks. Stale indexes. Permission leaks through the prompt. And instructions hidden in documents.

Further reading

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, Lewis et al. 2020: the paper that named the technique.
• Lost in the Middle: How Language Models Use Long Contexts: why passage order and count matter.
• Embeddings and vector search for the index, evaluation and guardrails for measuring it, and agents for search the model drives itself.