How vector search works: embeddings, similarity, and vector databases explained

Every RAG system has the same hidden ceiling: the LLM can only answer with what retrieval hands it. If the right passage never comes back, no prompt engineering will save the answer. And in most RAG systems, the thing doing the retrieving is vector search.

This chapter builds the mental model from the ground up — no maths degree required. By the end you’ll know what an embedding is, how “close” is measured, how indexes find neighbours among millions of vectors in milliseconds, and when a vector database is worth it.

Why keyword search isn’t enough

Imagine a customer types this into your support bot:

how do I cancel my plan?

And the article that answers it is titled “Terminating your subscription.”

Keyword search compares words. These two share none — not “cancel”, not “plan”. The perfect answer scores zero and never shows up.

People describe the same thing in endless ways: “cancel”, “end”, “stop paying”, “terminate”, “get out of my contract”. Your documents use one of them. Vector search is how you bridge that gap.

Embeddings: GPS coordinates for meaning

An embedding model reads a piece of text and outputs a list of numbers — a vector. The model is trained so that texts with similar meaning get similar numbers.

Think of it like GPS. Every place on Earth has coordinates, and places that are near each other have similar coordinates. An embedding gives every sentence coordinates in a “map of meaning”, and sentences that mean similar things land near each other.

Figure

Real embedding models don’t use two dimensions — they use hundreds or thousands (384, 768, and 1,536 are common). No single number means “about money”; meaning is spread across all of them. But the rule holds: close together = similar meaning.

Here’s what that looks like in code, using a small open-source model you can run on a laptop:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")   # 384 dimensions, free, runs locally

docs = ["Terminating your subscription", "Tracking where your order is"]
query = "how do I cancel my plan?"

doc_vecs = model.encode(docs, normalize_embeddings=True)
q_vec = model.encode(query, normalize_embeddings=True)

print(doc_vecs.shape)   # (2, 384) — each sentence is now 384 numbers
print(doc_vecs @ q_vec) # similarity of the query to each doc

With this model, “how do I cancel my plan?” scores 0.44 against “Terminating your subscription” — the highest of all 46 articles in the demo below, despite sharing zero words.

One hard rule: documents and queries must be embedded by the same model. Vectors from different models live on different maps. Switching models means re-embedding everything — treat it as a migration, not a config change.

Measuring “close”: cosine similarity

To find the nearest neighbours, we need a way to score how close two vectors are. The most common is cosine similarity: it measures the angle between two vectors and ignores their length.

Figure

Metric	What it measures	When to use it
Cosine similarity	The angle between vectors	The default for text
Dot product	Angle × length	Identical to cosine when vectors are normalised — and cheaper to compute
Euclidean (L2)	Straight-line distance	Gives the same ranking as cosine on normalised vectors

In practice: normalise your vectors once (most libraries have a flag for it), then use whatever metric your embedding model’s documentation recommends. The ranking will be the same.

Play with it — drag the arrow tips:

Interactive · drag the arrow tips

SimilarUnrelatedOppositeSame angle, longer B

Cosine similarity

0.98

−10+1

Angle

10°

Dot product

0.98

Distance (L2)

0.17

Length |A| · |B|

1.00 · 1.00

Normalise (force both lengths to 1)

Adjust with sliders

Angle of A Angle of B Length of B

See it for yourself

Below are 46 help-center articles from a fictional online store, embedded with the same model as above. Each point is one article; its shape shows the topic. The 384 dimensions are flattened to 3D so you can see them — drag to rotate, pick a question, and watch where it lands.

Interactive · drag to rotate · click a point

Auto-rotateCluster labels+−Reset

• Billing
• Shipping
• Account
• Returns
• Products
• Off-topic
• Your query

1 · Ask something

how do I cancel my plan?my package never showed upI can't get into my accountcan I send these sneakers back?my card got billed two timeswhat should I cook for dinner tonight?

2 · Or explore an articlePick an article…Terminating your subscriptionHow to downgrade to the free tierUpdating the card on fileWhy was I charged twice this month?Switching from monthly to annual billingPausing your membership for a few monthsUnderstanding the invoice line itemsApplying a promo code at checkoutTracking where your order isEstimated delivery times by regionParcel marked delivered but not receivedChanging the delivery address after orderingExpress and overnight delivery optionsShipping to international destinationsWhat happens if the courier misses youOrders delayed due to weatherResetting a forgotten passwordTurning on two-factor authenticationLocked out after too many login attemptsChanging the email address on your profileSigning in with Google or AppleDeleting your account and personal dataSpotting phishing emails pretending to be usManaging devices that are logged inOur 30-day return policyGetting money back for a damaged itemPrinting a prepaid return labelExchanging an item for a different sizeHow long refunds take to appearItems that cannot be returnedReturning a gift without a receiptStore credit versus refund to cardChoosing the right running shoe for flat feetSizing guide for sneakers and trainersWaterproof hiking boots comparedHow to clean white leather shoesBreaking in new boots without blistersCushioned shoes for long-distance runningKids' shoe sizes by ageVegan and recycled material footwearA quick weeknight pasta recipeHow long to boil an eggBest houseplants for low lightTips for brewing pour-over coffeeTraining a puppy to sitPlanning a weekend road trip3 · Neighbours (k) =

3

Pick a question above. Its nearest neighbours light up in the plot and are ranked here.

Brute force: exact, simple, and fine more often than you think

The simplest way to find the nearest neighbours is to compare the query with every stored vector and keep the best:

import numpy as np

def search(query_vec: np.ndarray, doc_vecs: np.ndarray, k: int = 5):
    scores = doc_vecs @ query_vec            # one dot product per document
    top = np.argpartition(-scores, k)[:k]    # the k best, unordered
    return top[np.argsort(-scores[top])]     # ...sorted best first

That’s it — perfectly accurate, no index, no database. The catch is that the work grows with every document you add:

Documents	Memory (1,536-dim float32)	Multiply-adds per query
100,000	~0.6 GB	~150 million
1,000,000	~6 GB	~1.5 billion
10,000,000	~61 GB	~15 billion

A modern machine chews through the first row in milliseconds. By the last row, every single query scans 61 GB of memory.

ANN indexes: trading a little accuracy for a lot of speed

Approximate nearest neighbour (ANN) indexes skip most of the vectors and return the probably best matches. A well-tuned index typically finds the large majority of the true top results, in a fraction of the time.

The way to measure “a large majority” is recall@k: of the true top 10, how many did the index return? You’ll meet recall again in Chapter 4 — it’s the number that keeps ANN honest.

There are two families of ANN index worth knowing.

IVF: only search the nearest sections

IVF (inverted file index) groups all vectors into clusters ahead of time. At query time it finds the few clusters whose centres are closest to the query and searches only those.

Figure

IVF is memory-friendly and fast to build, but its clusters are learned from your data — when the data drifts, they need retraining.

HNSW: motorways, then main roads, then streets

HNSW (hierarchical navigable small world) is the most common default today. It builds a graph where each vector links to its near neighbours, stacked in layers: the top layers have few vectors and long links; the bottom layer has every vector.

Figure

HNSW gives excellent recall and speed, and you can add vectors at any time without retraining. The cost is memory — the graph lives alongside the vectors — and slower index builds. Its key dial is ef_search: explore more of the graph per query for higher recall, at the cost of latency.

See the trade-off for yourself

Here are brute force, IVF, and HNSW running for real on the same 600 points. Every distance calculation is counted, and recall is measured against the exact answer.

Interactive · click to move the query

Brute forceIVFHNSW

nprobe — clusters searched:

1

of 16Random query

Vectors checked

0 / 600

Recall@10

0 / 10

• Query
• Found (in true top 10)
• Missed (true top 10, not found)
• Checked
• ✕ Cluster centre

Quantization: compressing vectors

At large scale, the vectors themselves become the problem. Quantization compresses them — think JPEG for vectors:

• Scalar (int8): store each number in 1 byte instead of 4. About 4× smaller with little quality loss.
• Binary: 1 bit per dimension. About 32× smaller; search the compressed version first, then re-score the top candidates at full precision.
• Product quantization (PQ): chop each vector into pieces and replace each piece with a short code. Very compact, more quality loss.

Quantization is what makes searching hundreds of millions of vectors affordable.

So what does a vector database add?

An ANN index like FAISS or hnswlib is a data structure. A vector database wraps that index with everything a production application needs:

You need…	Because…
Metadata filtering	“Similar articles — but only for this customer, in English, from this year.”
Inserts, updates, deletes	Your documents change, and the index must stay correct as they do.
Hybrid search	Exact terms like SKUs and error codes need keyword search alongside vectors (Chapter 7).
Persistence and replication	It’s a database. It shouldn’t lose data when a machine dies.
Multi-tenancy and access control	Customer A must never retrieve customer B’s documents.
Scale-out	Eventually, the index outgrows one machine’s memory.

The landscape, by category

Category	Examples	Fits when
Index libraries	FAISS, hnswlib	Research, offline batch jobs, full control
Extensions to a database you already run	pgvector (Postgres), Elasticsearch / OpenSearch, Redis	You already operate it; you want vectors next to your relational data, in the same transactions
Dedicated vector databases	Pinecone, Qdrant, Weaviate, Milvus	Very large scale, heavy filtering, high query rates, or you want it fully managed
Embedded / local	Chroma, LanceDB	Prototypes, notebooks, local-first apps

Here’s how little it takes with pgvector, if you already run Postgres:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
  id         bigserial PRIMARY KEY,
  tenant_id  int  NOT NULL,
  content    text NOT NULL,
  embedding  vector(384)
);

CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

-- The 5 chunks closest in meaning to the query, for one tenant.
-- <=> is cosine distance (0 = identical), so smaller is better.
SELECT id, content
FROM chunks
WHERE tenant_id = 42
ORDER BY embedding <=> $1
LIMIT 5;

Do you need one? Rules of thumb by use case

Use case	Typical shape	Starting point
Support bot over a help center	Thousands of articles, few filters	Brute force or pgvector — almost certainly enough
Internal docs for one company	Tens of thousands of chunks, permission filters	pgvector with an HNSW index and a team_id filter
Multi-tenant SaaS	Millions of chunks, strict per-customer isolation	A store with strong filtered search; test isolation explicitly
E-commerce catalogue	Millions of items, heavy filters (price, size, stock), exact SKUs	Hybrid search with filtering — dedicated store or a search engine like OpenSearch

These are starting points, not verdicts. Chapter 5 puts pgvector and dedicated databases side by side with numbers.

Where this fits in RAG

Vector search is the retrieve step. Everything before it — how you chunk and embed — decides what’s findable. Everything after it — reranking and generation — can only work with what it returns.

Key takeaways

1. Retrieval quality is the ceiling on answer quality. Many “the LLM is wrong” bugs are retrieval bugs.
2. An embedding is coordinates for meaning. Same model for documents and queries, always.
3. Compare scores by rank, not threshold. Absolute similarity values vary by model.
4. Brute force is fine below ~100K chunks. Reach for an ANN index when latency or memory says so.
5. ANN is a dial, not a switch. nprobe and ef_search trade recall for speed — measure recall, don’t assume it.
6. A vector database is an index plus database features. Filtering, updates, and isolation are what you’re really choosing between.

Next, in Chapter 2, we look at the embedding models themselves: what makes one better than another, open-source vs API models, and why the “sneakers” query went wrong.