Embeddings and vector search

Vector search finds text by meaning, not by matching words. An embedding model turns each piece of text into a list of numbers, called a vector. Texts that mean similar things get vectors that sit close together.

Searching is then a geometry problem. Given the question’s point, find the nearest points. Doing that exactly is easy, but slow. Doing it fast means accepting that the search will sometimes miss.

The problem it solves

A support site has an article titled “Refund policy”. A user types “how do I get my money back for a cancelled order”. A keyword index finds nothing useful. The only shared words are “how”, “do”, “I” and “for”. The user leaves.

Keyword search, built on an inverted index, matches the words people type. But people describe the same thing with different words, so many real queries miss. Retrieval for an LLM makes it worse. In RAG, a question that retrieves the wrong passage gets a confident wrong answer.

What we want is a search that knows “money back” and “refund” mean the same thing. That’s what embeddings provide.

How it works

Step 1: Text becomes a vector

An embedding model reads a piece of text and outputs a vector of fixed length: 384, 768, 1,024 or 3,072 numbers, depending on the model. Nobody chooses what each number means. The model was trained on huge numbers of pairs of related texts, like a question and its answer, or a title and its article. Training taught it to place related texts close together, and unrelated ones far apart.

The result is a space where distance roughly tracks meaning. “Refund policy” and “get my money back” end up near each other. “Refund policy” and “deploy freeze” end up far apart.

Three rules follow. Breaking any of them breaks the search:

1. Use the same model for documents and questions. Two models produce two unrelated spaces, and distances between them mean nothing. Changing the model means embedding everything again.
2. Embed passages, not whole documents. One vector for a 50-page manual blurs every topic in it into one point. Split documents into chunks first (see RAG).
3. Follow the model’s instructions. Some models expect a prefix, such as search_query: on questions and search_document: on passages. Without it, they retrieve measurably worse.

Step 2: Measure closeness

Three measures are common:

Measure	Formula	Notes
Cosine similarity	a·b / (|a| |b|)	The angle between vectors. 1 is identical direction, 0 unrelated
Dot product	a·b	Same as cosine when vectors have length 1
Euclidean distance	|a − b|	Same ranking as cosine when vectors have length 1

Most systems normalize every vector to length 1 when it is stored. Then all three give the same ranking, so systems use the cheapest one, the dot product.

Step 3: Exact search

The simplest search compares the question’s vector with every stored vector, and keeps the top k. For 100,000 chunks of 768 numbers, that’s 77 million multiply-adds. That takes a few milliseconds on one CPU core with a fast math library. Exact search is always right. It needs no index, and it handles deletes and filters easily.

It stops working as the collection grows. At 100 million vectors, it’s 77 billion multiply-adds per query, and the vectors alone take 307 GB as 32-bit floats. So large collections use approximate search.

Step 4: Approximate search with a graph (HNSW)

HNSW (Hierarchical Navigable Small World) is the most widely used index. It links each vector to a handful of its nearest neighbors, which forms a graph. It searches by walking the graph toward the question:

• Layers. Every vector is in the bottom layer. A random few are also in a sparser layer above, and fewer still in the layer above that. The upper layers work like express lanes.
• Search. Start at the top layer. Keep moving to whichever neighbor is closer to the question, until none is. Then drop a layer and continue from there. On the bottom layer, keep a list of the ef best candidates found so far. Keep checking their neighbors until no new candidate beats the worst one on the list.
• Knobs. M is the number of links per vector, often 16 to 64. More links give better recall, but use more memory. ef_search is the width of the list (the beam). A higher value finds more of the true neighbors, and costs more distance checks. It’s set per query, so you can trade recall for speed at run time.

Click to move the query, and compare exact search with the graph. Then drop the beam width to 1 and watch recall fall. Raise it until every true neighbor is found.

Exact: compare allHNSW graph

Beam width (ef) 8

Distance computations

-

True 5 nearest found

-

Share of points touched

-

Click or tap to move the query (the large dot). Blue points were compared, green are the 5 results, a red ring is a true nearest neighbor the search missed. Bigger gray dots are the upper layers.

With a reasonable beam, the graph finds all five true neighbors while checking only a fraction of the points. With a beam of 1, it walks greedily into the nearest cluster, and often stops one cluster short.

At millions of points, the saving is far larger than in this small example. HNSW typically reaches 95 to 99% recall while computing a few thousand distances instead of millions.

TradeoffHNSW over exact search

We gain

Queries in single-digit milliseconds at millions of vectors, with recall you can tune per query.

We give up

Some true neighbors are missed. The graph needs memory on top of the vectors, and building it is slow. Deletes leave holes that make it worse until a rebuild.

Step 5: Clusters and compression (IVF and PQ)

Two other techniques matter once memory becomes the limit.

IVF (inverted file) groups the vectors into clusters with k-means, say 4,096 groups. A query is first compared with the center of each group. Then it is compared only with the vectors in the closest nprobe groups. IVF uses little memory beyond the vectors. But it misses neighbors that sit just across a cluster boundary, unless you raise nprobe.

Product quantization (PQ) compresses each vector:

1. Split a 768-number vector into, say, 96 pieces of 8 numbers.
2. Replace each piece with the id of the closest of 256 learned prototypes. That id fits in one byte.
3. So 3,072 bytes become 96, a 32 times saving. Distances are then computed from lookup tables.

Precision drops. So systems often search the compressed vectors to get 100 candidates, then score those again with the full vectors.

IVF with PQ is how collections of a billion vectors fit in memory. Disk-based indexes such as DiskANN keep a compressed copy in RAM and the full graph on SSD.

Step 6: Filters, keywords and reranking

Real queries rarely want “the nearest vectors in the whole collection”. They want the nearest ones that this user may see, in this product, from the last year. That’s a filter. Filters and approximate indexes fight each other:

• Filter after searching. Find the top 10, then drop the ones that fail the filter. With a strict filter, all 10 may fail, and the user gets nothing.
• Filter during the search. Skip disallowed vectors while walking the graph. Good vector databases do this. But a very selective filter cuts the graph into pieces, and recall drops.
• Partition by the filter. Keep one index per tenant or per access group. Isolation is exact, but there are many small indexes to manage.

Vectors are also bad at exact tokens: an error code like ERR_4012, a product SKU, a person’s name. The embedding of ERR_4012 sits near every other error code. Keyword search finds these easily.

So most production retrieval is hybrid. Run a keyword search (BM25) and a vector search, then merge the two ranked lists. Reciprocal rank fusion is the usual merge. Each document scores the sum of 1 / (60 + its rank) across the lists. That needs no tuning of the two score scales.

Finally, a reranker scores the best candidates more carefully. It is a cross-encoder: it reads the question and each candidate passage together, and scores how well the passage answers it. That’s far more accurate than comparing two vectors made separately. It is also far too slow to run on a whole collection. So it runs only on the top 50 to 100 candidates from the first stage.

Retrieve broadly and cheaply, then rerank precisely. That is the same two-stage shape as recommendation systems.

Options and tradeoffs

Option	Examples	Pick it when
A library in your process	FAISS, hnswlib, ScaNN	One service owns the data, and you can rebuild and reload the index yourself
An extension to a database you run	pgvector for PostgreSQL, OpenSearch and Elasticsearch k-NN	Up to tens of millions of vectors, and you want vectors next to the rows, filters and transactions you already have
A dedicated vector database	Milvus, Qdrant, Weaviate, Pinecone	Hundreds of millions of vectors and more, heavy filtered search, or a team that doesn’t want to run the index

TradeoffStart with pgvector or your search engine's k-NN

We gain

One fewer system to run. Vectors are updated in the same transaction as the rows they describe. SQL filters and joins come for free.

We give up

Less tuning control, and lower limits than a dedicated engine. At hundreds of millions of vectors, or very high query rates, a dedicated system is worth its running cost.

Choosing the embedding model matters more than choosing the database. Compare candidate models on your own queries, not only on public leaderboards such as MTEB. A model that tops a general benchmark can lose on your domain’s vocabulary.

How it fails

Recall drops and nobody notices

An approximate index returns 10 results whether or not they’re the right 10. Maybe an index rebuild used a lower ef, or deletes left the graph sparse. Either way, answers get quietly worse. So measure recall regularly. Take a sample of real queries, run them both exactly and through the index, and compare.

The model changes

After an embedding model upgrade, the old vectors and the new question vectors live in different spaces. Every vector must be embedded again. During the migration, you need two indexes and a switch-over, like a database migration. Store the model name and version with every vector.

Exact matches fail

Names, codes, numbers and rare jargon embed poorly. So a pure vector search answers “What does error ERR_4012 mean?” with an article about some other error. Hybrid search fixes this.

Filters starve the results

Say a tenant has 50 documents out of 50 million. With filter-after-search, it gets zero results for most queries. Use filtered search or per-tenant partitions, and test with your smallest tenant.

Common mistake

Treating similarity as relevance

The nearest vector is the most similar text. That is not always the passage that answers the question. “What is our refund policy?” is very similar to another FAQ entry asking the same thing, with no answer in it. Similarity gets you candidates. A reranker or the model decides relevance.

At scale

Take 200 million chunks of 768 dimensions.

Item	Size
Vectors at 32-bit	200M x 3 KB = 614 GB
HNSW links, M = 16 (32 links on the bottom layer, 4 bytes each)	about 26 GB
Same vectors with PQ to 96 bytes	about 19 GB

Full-precision vectors with HNSW need a cluster:

• Shard the vectors across nodes by document id, so writes spread evenly.
• Search every shard in parallel, and merge the top k from each. This is called scatter-gather.
• The slowest shard sets the latency. So keep shards equal in size, and replicate each one for throughput and failover.

With PQ, the whole collection fits in one large machine’s memory. It costs some precision, and a rescoring step wins that back.

Writes need a plan too. Most graph indexes take in new vectors well. But they handle deletes and updates by marking them and filtering at query time, and the graph gets worse until it is rebuilt. So real systems keep a small fresh segment for recent writes. They merge it into the main index in the background. That is the same pattern as an LSM tree.

In the interview

Interviewer asksWhy not just use Elasticsearch keyword search for the support bot?

Users phrase questions differently from how articles are written, so keyword search misses many relevant articles. But keyword search is excellent at exact terms like error codes and product names, where vector search is weak. So I’d use both. Hybrid retrieval with BM25 and vectors, merged by reciprocal rank fusion, then a reranker on the top 50. Elasticsearch and OpenSearch can do the vector half too, which keeps it one system.

Interviewer asksHow do you know your vector index is returning the right results?

Two separate checks. First, index recall: run a sample of real queries through exact search and through the index, and measure how much the top 10 overlap. Alert if it drops. Second, retrieval quality: a labelled set of questions with the passages that answer them, scored as hit rate or recall at k. The first catches index problems. The second catches model and chunking problems.

Interviewer asksEach customer must only see their own documents. How do you enforce that in vector search?

Never by asking the model to ignore other customers’ text. I enforce it in retrieval. Either a separate index or namespace per tenant, or a tenant id on every vector, applied as a filter during the search, not after it. Per-tenant indexes give hard isolation and easy deletes. A shared index with filters is cheaper for many small tenants. I’d test with the smallest tenant, where filter-after-search returns nothing.

Mid-level covers

Explains embeddings and cosine similarity. Uses the same model for documents and queries. Names a vector database with an approximate index.

Senior adds

Explains how HNSW trades recall for speed, and which knobs control it. Sizes memory with and without PQ. Plans shards and scatter-gather. Uses hybrid search and reranking for exact terms. Enforces tenant isolation in the search itself. Measures index recall separately from retrieval quality. Plans re-embedding as a migration.

Try it on your laptop

Step 2: Answer from your own docs →

Build a small exact vector store over your own docs, then score how often the right file comes back for 20 questions.

Recap

Embeddings. A model maps text to a vector, so similar meanings land close. Use the same model for documents and questions. Embed chunks, not whole documents. Store the model version.

Similarity. Normalize vectors to length 1. Then the dot product is the cosine similarity.

Exact search. Compare against everything. Right every time, and fine up to about a million vectors.

Approximate search. HNSW walks a layered graph, and ef trades recall for speed per query. IVF searches a few clusters. PQ compresses vectors 32 times, and rescoring restores precision.

Production retrieval. Filters applied during the search. Hybrid BM25 plus vectors, merged with reciprocal rank fusion. Then a cross-encoder reranker on the top candidates.

Biggest risks. Silent recall loss. Model changes that need a full re-embed. Codes and names that vectors miss. And strict filters that return nothing.

Further reading

• Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs: the HNSW paper.
• Product quantization for nearest neighbor search, Jégou, Douze and Schmid: the compression behind IVF-PQ.
• The FAISS library: a tour of index types and when to use each, from the team that built it.
• Reciprocal rank fusion, Cormack et al.: the simple merge most hybrid search uses.
• Search and inverted indexes for the keyword half of hybrid search, and RAG for what retrieval feeds.