Lesson 60 gave you RAG. This lesson gives you the database underneath it. A vector database stores embeddings, which are lists of numbers that capture what a piece of text means, and it finds the nearest ones to a question, fast, without scanning the whole store. One support question turns into numbers, the database finds the ten closest help articles, and those ten go to the model as context. I learned vector search the way I learn most tools in this job. I watched a search return confident wrong answers, checked the recall number until it confessed, and then rebuilt the index the search was actually using instead of the one I thought it was using. This lesson walks one question through a vector database, hop by hop, from a first embedding to the deep end: ANN indexes, metadata filters, hybrid search, sharding, and how to measure whether the answers are any good.
Difficulty: Advanced · Know this first: Lesson 60
Every hop charges rent in milliseconds. The bar below is a worked example for one support question, not a measurement from a live system, but the shape is honest: in most slow searches I have chased, the embedding and the rerank are most of the bill, and the search itself is a small slice. Step through the lesson or switch the scenario and watch the bar change. The sharded index does not change the embedding. It changes how long the search waits for the slowest shard. The missing index does not change the question. It changes the search from a walk on a graph to a scan of everything, and the bill follows.
The budget bar above shows three fixed stories. This simulator lets you write your own. It models one question searched against an index, with the index size, the top-k, the recall target, and the query rate as knobs you can turn. Move a slider and watch the search hop, the fetch, the rerank, the index memory, and the total change in the budget bar, in the hop timings, and in the status line. The numbers are a teaching example, not live measurements. The shape is the lesson: recall, speed, and memory trade against each other, and every index choice is a point on that triangle.
A bigger index is a longer search and more memory, in a straight line. Default: search 15 ms, index memory about 8 GB.
Every extra result is more bytes to fetch and one more rerank call. Default: fetch 6 ms, rerank 18 ms.
Higher recall means the search walks further on the graph, and every point of recall past 90 costs a millisecond. Default: search 15 ms.
One node answers 1,000 divided by the search time in queries a second, and the nodes needed are the query rate divided by that, rounded up. Default: 2 nodes.
Try this: (1) Leave the defaults: 1M vectors, top-k 10, recall 95, 100 queries a second. The total is about 86 ms and 2 nodes are needed. (2) Push the index to 10M vectors, top-k 100, recall 99. The total is about 342 ms. Same database, same code. Only the knobs changed. (3) Set it back, then drop the recall target to 80 and watch the search hop fall to 10 ms. That is what recall costs, and the answers get a little worse for it.
Three questions. No score is saved anywhere — this is just you against the material.
A vector database looks like a database with a strange index. It is not. It is four layers stacked on each other, and each layer only makes sense after the one below it. I learned it in this order, and I teach it in this order: turn text into numbers and find the nearest ones, then pick the index that fits the trade, then filter and combine the search, then run it at scale and measure it. Skip a layer and the next one reads like magic. Build them in order and nothing in this lesson is magic.
Definition: a vector database at the basic level stores embeddings, which are lists of numbers that capture what a piece of text means, and it answers a nearest-neighbor query: given a question vector, find the stored vectors closest to it. Closeness is measured by a distance, usually cosine similarity, which is the angle between two vectors, or Euclidean distance, which is the straight-line gap. The query is embedded with the same model that embedded the stored text, because the numbers only match when the model matches. The answer is the top-k nearest vectors, and top-k is the knob that decides how many come back.
Types:
Uses: when a system has to search by meaning, not by spelling. A support question matched to the help articles that answer it. A document matched to the documents like it. The first win is that the search stops needing keywords, because the numbers capture the meaning the words were pointing at.
Scale: holds up to a few thousand vectors with a flat scan, and no more, because a scan compares the question to every vector.
Failure modes: a query embedded with a different model than the index is a search on numbers that do not match, and the answers are wrong without an error. A top-k of one is a single guess, and a single guess is often wrong.
Cost: the embedding model call, per question, and the memory for the vectors. A million vectors at 1,536 dimensions is about 6 GB of numbers, before any index.
Trade-offs: you get search by meaning, and you give up exactness the moment the store grows past a few thousand vectors, because an exact scan is too slow, and the approximate index is the only way forward.
Definition: the approximate nearest neighbor index, the structure that makes the search fast. HNSW builds a graph: each vector links to its nearest neighbors, and the search walks the graph, hopping from neighbor to neighbor, getting closer each time. IVF splits the vectors into clusters, usually a few hundred of them, and the search looks only at the clusters nearest the question. Both skip most of the vectors, both are fast, and both can miss the true nearest vector. The miss rate is the recall, and the recall is the number that decides whether the answers are any good. The index settings are the trade: more graph links or more clusters searched means higher recall and a slower search.
Types:
Uses: when the store is too big to scan. HNSW for a search that has to be fast and right, IVF for a store that has to fit in less memory. The index is a trade, and the trade is measured in recall, not in speed alone. A search that is fast and wrong is not a fast search. It is a wrong answer delivered quickly.
Scale: holds to a few million vectors on one node, with the index in memory.
Failure modes: an index that was never built, so the search falls back to a flat scan and takes hundreds of milliseconds. A recall target set too low, so the search skips the true nearest vectors to stay fast. An index built with one model and searched with another, which is a silent mismatch.
Cost: memory, for the graph or the clusters, on top of the vectors themselves. HNSW costs about 30 percent more memory than the vectors alone.
Trade-offs: HNSW buys speed and recall and costs memory. IVF buys less memory and costs build time and a recall that depends on the cluster count. Quantization buys a smaller index and costs recall, because compressed vectors are rougher numbers.
Definition: the search with rules on top of it. A metadata filter is a condition on the side of the search: only this product, only current articles, only documents after last year. The filter runs inside the search, so the index keeps looking until it has enough results that pass. Hybrid search combines the vector search with a keyword search, and merges the two lists, because some questions are answered by meaning and some by an exact term, like a product code. The store that holds the vectors is the other choice: a dedicated vector database, a Postgres with the pgvector extension, or a document store with a vector index. Each one is a different trade between the search and the rest of the system.
Types:
Uses: when the search has to be right, not just fast. A filter that keeps only current articles, so the model never cites an outdated one. A hybrid search for a question that has both a meaning and an exact term. A rerank when the answers matter enough to pay for a second model call per article. The search is a system, not a query, and the filter, the hybrid merge, and the rerank are all part of the answer.
Scale: holds at tens of millions of vectors, but the filter changes the cost, because a filter that matches almost nothing forces the search to walk further to fill the top-k.
Failure modes: a filter after the search leaves the top ten empty. A hybrid merge that is badly weighted returns keyword matches for a meaning question. A rerank that is skipped to save time leaves the first search's mistakes in place.
Cost: the second model call per article for the rerank, and the cost of the store that fits the team. pgvector buys a store the team already knows and costs search speed at scale. A dedicated store buys speed and costs a second database to run.
Trade-offs: filtering inside the search buys correct results and costs search time when the filter is strict. Hybrid buys both kinds of questions and costs a merge that has to be tuned. A dedicated store buys the fastest search and costs a second system to own.
Definition: a vector database at a size one node cannot hold, and a search that has to be measured, not trusted. Sharding splits the index across nodes, and the search fans out to the shards and merges the results, waiting for the slowest shard. Evaluation is the measurement: a fixed set of test questions with known right answers, run against the index on a schedule, reporting recall and the share of answers a human judge calls right. In production the retrieval layer gets the same treatment as any other service: a latency budget, a recall alert, a rebuild pipeline, and a fallback when the search is down, usually a keyword search or a cached answer, because a RAG system with no retrieval is a model answering from memory.
Types:
Uses: when the index is too big for one node and the answers are too important to trust blindly. A sharded index for ten million vectors, with the query fanning out and the merge waiting for the slowest shard. An evaluation set that runs every week, so a model change or an index rebuild that hurts recall is caught before the users feel it. A vector search is a system with a quality bar, and the bar is measured, not assumed.
Scale: holds at hundreds of millions of vectors across dozens of shards, but the fan-out is the cost, because a query waits for the slowest shard, and one slow shard is the whole trip.
Failure modes: a shard that falls behind on the index build serves stale vectors, and the search returns old articles. An evaluation set that is never updated measures a system that no longer exists. A fallback that was never tested fails when the search is down, and the model answers from memory without anyone noticing.
Cost: the nodes, the evaluation pipeline, and the human time to judge answers. The index memory is the biggest line: ten million vectors at 1,536 dimensions is about 80 GB with the graph overhead.
Trade-offs: sharding buys more vectors and costs fan-out latency. Evaluation buys honesty and costs a pipeline and human judges. A fallback buys uptime and costs a second search path that has to be maintained.
HNSW is a graph, and the search is a walk. Each vector is a node, and each node links to its nearest neighbors, about sixteen of them in a usual setup. The search starts at a node near the top of the graph, looks at its neighbors, moves to the closest one, and repeats, getting closer to the question each hop. The walk stops when no neighbor is closer than the current node. That is the whole search, and it is fast because it never looks at most of the vectors. The settings are the trade: more links per node means a better walk and more memory, and a bigger search budget means the walk looks further and costs more time. I tune it by measuring recall on a fixed set of test questions, not by guessing. A recall of 0.95 at 12 ms is the number I want. A recall of 0.80 at 8 ms is a search that is fast and wrong.
IVF is a different shape. It splits the vectors into clusters, usually a few hundred, and the search looks only at the clusters nearest the question. The build is a clustering pass over the whole store, and the search is a lookup into a few clusters. IVF uses less memory than HNSW, because there is no graph, only the cluster assignment. It loses when the clusters are uneven, because a question that lands in a huge cluster searches a huge cluster, and the time follows the cluster size. I use IVF when memory is the constraint and the vectors cluster well, which text embeddings usually do. I use HNSW when the search has to be fast and the memory is there. The choice is not a preference. It is a measurement: build both, measure recall and latency on the same test questions, and keep the one that wins.
A filter has to run inside the search, and the reason is arithmetic. If the search finds ten neighbors and the filter throws out nine, the answer has one article. The index has to keep looking until it has ten that pass, and that means the search walks further when the filter is strict. A filter that matches one percent of the index makes the search walk a hundred times further, and the time follows. The fix is not to drop the filter. The fix is to shard by the filter field, so the query goes to the shard that holds the matching articles, and the search never looks at the rest. A product filter that sends the query to one shard is faster than a product filter that searches all of them. The layout is the optimization, not the index.
Some questions are answered by meaning, and some by an exact term. A question about a product code, or an error code, is a keyword question, and a vector search on it returns articles that are about the same kind of thing, not the thing itself. Hybrid search runs both: the vector search for meaning, the keyword search for the exact term, and a merge that combines the two lists by rank. The merge is the part that needs tuning, because a merge that weights keywords too heavily returns exact matches that miss the meaning, and a merge that weights vectors too heavily returns meaning that misses the code. I tune the merge on the same test questions I use for recall, and I check the failures by hand, because a merge that is wrong is wrong in a way no metric catches cleanly.
At 10x the vectors, the index moves to its own nodes, and the memory becomes the budget line. Ten million vectors at 1,536 dimensions is about 80 GB with the graph overhead, and that is a node, not a sidecar. The recall gets an alert, because a recall that falls after a rebuild is a wrong answer delivered quickly, and the users feel it before the dashboards do. At 100x, the index is sharded, the evaluation runs weekly, and the rebuild is a pipeline, not a job someone runs by hand. The model change and the index rebuild are one deploy, because a model without a rebuild is a mismatch, and a mismatch is silent. The retrieval layer stops being a database and becomes a service, with a latency budget, a recall bar, and a fallback, and it gets treated like one.
import hnswlib
import numpy as np
DIM = 1536 # numbers per vector, set by the embedding model
N = 1_000_000 # articles in the index
# vectors: float32 array of article embeddings, one row per article
# meta: list of dicts, meta[i] = {"product": ..., "status": ...}
index = hnswlib.Index(space="cosine", dim=DIM)
index.init_index(max_elements=N, M=16, ef_construction=200)
index.add_items(vectors, ids)
index.set_ef(64) # search budget: bigger walks further, costs time
def search(query_vec, k=10):
# the filter runs inside the search: keep asking the index
# until ten current checkout articles have passed it
got, fetch = [], k * 4
while len(got) < k:
labels, _ = index.knn_query(query_vec, k=fetch)
got = [i for i in labels[0]
if meta[i]["product"] == "checkout"
and meta[i]["status"] == "current"][:k]
if len(got) == k or fetch >= N:
break
fetch *= 4
return got
def recall_at_10(query_vec, got):
# the flat scan is the exact truth: too slow for production,
# and the only honest way to measure the index
sims = vectors @ query_vec # vectors are normalized
truth = set(np.argsort(-sims)[:10].tolist())
return len(truth & set(got)) / 10.0
Three pieces, one measurement. The index is HNSW with 16 links per node and a search budget of 64, and those two numbers are the recall trade: more links and a bigger budget mean higher recall and a slower search. The filter runs inside the search, keeping only current checkout articles, and the index keeps walking until it has ten that pass, so the top ten is never empty because of the filter. The recall check at the end is the honest part: the flat scan is the exact truth, too slow to run in production, and the only way to measure whether the index is finding the right vectors. A recall of 0.95 at 12 ms is the number I want. A recall of 0.80 is a search that is fast and wrong, and the test questions catch it before the users do.
This one stayed with me because the search was fast, the dashboards were green, and the answers were wrong for a week. The index was working. It was working on the wrong numbers.
A support assistant answered questions from a help-article index. The team changed the embedding model to a newer one, because the new model was better on the benchmarks. They changed the model for the queries, and they did not rebuild the index. The new question numbers were measured against the old article numbers, and the distances meant nothing. The search returned articles, the reranker ordered them, the model answered from them, and the answers were confident and wrong. The recall metric did not catch it, because the recall test ran against the old index with old question numbers, and the old numbers matched the old index fine. The users caught it, one support ticket at a time, and each ticket looked like a different problem, because the wrong article was different every time.
Three things. First, the model and the index ship together. A model change without a rebuild is not a deploy, it is a mismatch, and the pipeline blocks it. Second, the recall test uses fresh question numbers from the current model, so a mismatch shows up in the test, not in the users' tickets. Third, the answer quality gets a human check: a sample of answers every week, judged by a person, because recall measures the search and only a person measures whether the answer is right. The search did not fail in this incident. The deploy failed, and the deploy was two changes that should have been one. The lesson is that an embedding model and its index are one unit, and changing half of a unit is a bug that no dashboard catches.
Design the vector search for a support assistant with 10 million help articles. Questions arrive at 100 a second. The answers must cite current articles only, the recall bar is 0.95 at top-10, and the latency budget is 150 ms at p99 for the search alone. The embedding model changes twice a year. Sketch the index, the shards, the filter, the hybrid search, the rerank, the evaluation, and the rebuild pipeline, and say what you would measure.
Start with the memory. 10 million vectors at 1,536 dimensions times 4 bytes is about 61 GB of numbers, and the HNSW graph adds about 30 percent, so the index is about 80 GB. That is several nodes, not one, so the index is sharded, and the shard key is the product line, because the question carries a product filter and a filter that narrows the shards is the fastest optimization. The search at 15 ms per node gives about 66 queries a second per node, so 100 queries a second needs 2 nodes per shard, and the fan-out waits for the slowest shard. The recall bar is 0.95 at top-10, measured on a fixed set of test questions with fresh numbers from the current model, run weekly. The rerank costs 18 ms for ten articles, and the embedding costs 35 ms, so the search budget of 150 ms leaves about 80 ms for the model to answer, and the rerank stays, because the answers matter. The rebuild: ten million vectors re-embedded at about 35 ms a question is far too slow one by one, so the rebuild runs as a batch job on GPUs, and the old index serves while the new one builds, then the traffic flips. The model change and the rebuild ship as one deploy, because half a unit is a bug.
HNSW buys speed and recall and costs memory, about 80 GB for ten million vectors, so the index gets its own nodes. Sharding by product line buys a narrow fan-out and costs a hot shard when one product gets all the questions. A recall bar of 0.95 buys right answers and costs search time, because the walk goes further. A rerank buys accuracy and costs a model call per article, so top-k stays at ten. A rebuild pipeline buys a fresh index and costs a batch job and a traffic flip. And the fallback is a keyword search, because a RAG system with no retrieval is a model answering from memory, and the fallback has to be tested, not assumed.