System Design · Lesson 60

RAG: Retrieval-Augmented Generation

Here is the problem in plain words. A language model knows what it was trained on, and nothing after that. It knows nothing about your runbooks, your tickets, or your internal docs. So when someone asks it how to restart the payment service safely, it guesses. Sometimes the guess sounds right. Sometimes it is wrong in a way that pages someone at 2 AM. Retrieval-augmented generation fixes this by cheating, in the honest way. Before the model answers, the system fetches the pages that matter and hands them to the model along with the question. The model reads your actual runbook and answers from that. That is the whole idea. Fetch first, then generate. This lesson follows one question through nine hops: the question gets rewritten, turned into numbers, matched against an index of your docs, ranked, packed into a prompt, answered by the model, checked against the sources, and sent back. I run a small version of this at home on my own machine with local models, and the lessons are the same at any size. The model is only as good as the pages you hand it.

Difficulty: Advanced · Know this first: Lesson 56, Lesson 11
Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one question, not a measurement from a live system, but the shape is honest: the model does most of the waiting, about 890 ms of the 1,105 ms normal trip, while the retrieval hops that decide whether the answer is right cost about 175 ms between them. The stale index costs an extra 100 ms and the answer is wrong. The stuffed prompt costs an extra 655 ms and the answer is thinner. Step through the lesson or switch the scenario and watch the bar change. The lesson is in the bar. Cheap retrieval decides the answer. Expensive generation only writes it down.

Change the retrieval and watch the answer bill

The budget above is one fixed trip. This box is yours. Change the chunk size, raise top-k, turn the reranker off, turn hybrid search on, and watch the totals move. Then ask the question I ask before every deploy: how much of this answer is retrieval, and how much is the model?

How big each piece of a doc is when it gets fingerprinted and stored. Bigger chunks keep more context in one piece, so recall goes up, but each chunk carries more filler, so precision goes down. Prompt tokens also grow, because prompt tokens = top-k x chunk size, and model time grows with prompt tokens.

How many chunks the search hands to the reranker and the prompt. More chunks raise the chance the right page is in the pile, and raise prompt tokens, model time, and the risk of lost in the middle.

On adds 35 ms at hop 5 and raises precision by 8 points, because the reranker drops the chunks that look close but do not help. Off saves the 35 ms and loses the 8 points.

On runs a keyword search alongside the vector search and merges the two shortlists. It adds 25 ms at hop 4 and raises recall by 6 points, because keyword search catches exact names, error codes, and IDs that meaning alone misses.

Prompt tokens2,560 tokenstop-k x chunk size, packed into the context window
Model time890 ms250 ms fixed + 0.25 ms per prompt token
Total for one answer1,105 msrewrite + embed + search + rerank + prompt + model + check + send
Retrieval qualityRecall 65% · Precision 92%bigger chunks help recall, hurt precision

Try this: (1) Set top-k to 20 at the defaults. Prompt tokens jump to 10,240, model time jumps to 2,810 ms, and the total is 3,045 ms. Same question, same docs. Only the retrieval settings changed. (2) Now set chunk size to 128 with top-k at 5. Recall drops to 58% and the answer gets riskier, even though the prompt is tiny. Small chunks are precise and blind at the same time.

Check yourself

Three questions. No score is saved anywhere. This is just you against the material.

Basic to master

I read this topic four times here. First as a model plus my documents, then as retrieval done well, then as knowing whether the answers are right, then as the index run as a service.

Basic — The model plus your documents

Definition: Retrieval-augmented generation is a simple deal. The model does not know my runbooks, so I fetch the right pages first and hand them to the model with the question. The model reads the pages and writes an answer from them. If the pages are missing, the answer is a guess.

Types:

  • Keyword search, matching the exact words in the question.
  • Vector search, matching the meaning of the question.
  • Fixed-size chunks, cutting each doc every 512 tokens with a little overlap.
  • A top-k shortlist, the handful of chunks the model actually gets to read.

Uses: I use the basic level for a support bot that answers from runbooks. A user asks how to restart the payment service safely. The system finds the restart page, hands it to the model, and the model writes the steps in plain words. Every claim in the answer traces to a page.

Dimensions: At low load this level is cheap and honest. What breaks is retrieval. The wrong chunk is fetched, or the right page was never indexed, and the model writes a confident answer from the wrong page. I notice it when the answer has no citation, or the citation points at the wrong page. The cost is an embedding model call per question and an index to maintain. What I give up is the model's general knowledge. The answer is only as good as the docs, and stale docs give stale answers.

Intermediate — Making retrieval good

Definition: At this level I stop accepting the first five chunks and start earning them. Two searches beat one. A reranker beats a raw shortlist. And the chunking decides whether the model ever sees the procedure whole.

Types:

  • Hybrid search, keyword plus vector, merged by rank.
  • A reranker, keeping the best three chunks and dropping the rest.
  • Parent-child chunking, small chunks for the search and the whole section for the prompt.
  • Semantic chunking, cutting at topic changes instead of token counts.

Uses: I use this level the first time a split procedure causes a bad answer, which is usually within the first month. I turn hybrid on for anything with IDs, error codes, or product names in it, because meaning alone misses exact words. I keep top-k small and rerank hard, and I put the best chunk first, because models read the start and the end of a long prompt carefully and skim the middle.

Dimensions: As the doc set grows, retrieval quality decides everything. What breaks is the middle of a long prompt, and the stale index. Docs change every day, and an index is a snapshot of the day it was built. I notice staleness on one metric: the age of the oldest chunk in the index. The cost is an extra search pass, about 25 ms, plus the reranker. What I give up is simplicity. Every extra stage is one more thing to tune, and a rewrite that wanders is three piles of noise instead of one answer.

Advanced — Knowing whether the answers are right

Definition: At this level I stop judging the system by reading a few answers and feeling good. I keep a set of questions with known right answers, and I run the whole set every night. Three numbers tell me what is wrong, and each one points at a different hop.

Types:

  • Faithfulness, whether every claim in the answer traces to a chunk.
  • Context precision, how many of the retrieved chunks were actually useful.
  • Context recall, how many of the chunks that would have answered the question were actually retrieved.
  • Query rewriting, splitting one question into two or three searches and merging the results.

Uses: I use this level when the answers look right and I do not trust my own eyes. If faithfulness drops, the model is inventing. If precision drops, the prompt is filler. If recall drops, the index or the chunking is wrong. Three numbers, three different fixes.

Dimensions: As traffic grows, the embedding hop breaks first, because every question pays for a model call before the search even starts. I cache embeddings for repeat questions and batch the calls. What breaks is the eval set itself. A golden set that never changes goes stale, and a stale eval is a green light that means nothing. I notice it when the evals are green and the real answers are quietly getting worse. The cost is the eval set and the nightly run. What I give up is the comfort of a quick manual check. Manual checks lie. The numbers do not.

Master — The index as a service

Definition: At this level the index is the design decision. One index for everything is simple until one team's docs drown out another team's. I split indexes by team or by doc type, route the question to the right index before I search, and treat re-embedding like a deploy.

Types:

  • Split indexes, one per team or doc type, with routing before the search.
  • A re-embed pipeline, where every doc publish triggers a re-embed of the changed pages, with a timestamp on every chunk.
  • Eval in CI, so a retrieval change runs the golden set before it ships.
  • A freshness alert, on the age of the oldest chunk, so a stuck pipeline pages someone at 10 AM instead of being found at 2 AM.

Uses: I use this level when the RAG system serves a whole company. The eval set grows with the traffic, because the questions users actually ask are the only honest test set.

Dimensions: At 100x the questions, the index is a service with its own capacity and its own alerts. What breaks is the pipeline nobody watches: the re-embed job that stopped last Tuesday, the doc publish that never triggered a re-embed. I notice it on the freshness alert. The cost is a pipeline, an eval set, and an on-call rotation for a system that used to be a script. What I give up is the small, simple setup. At this scale the retrieval system is a product, and products need owners.

Go deeper

The lesson above is one question through nine hops. Production is the rest of it. Here is what I actually spend my time on once the retrieval works and the answers look right.

Hybrid search

Vector search understands meaning. Keyword search understands exact words. A question with an error code, a service name, or a ticket ID needs the exact word, and meaning alone will miss it. A question phrased in plain words needs meaning, and keywords alone will miss it. Hybrid search runs both and merges the two shortlists. I score each chunk by its rank in both lists and keep the best of the merge. The cost is one extra search pass, about 25 ms here. The gain is the questions that have both a name and a meaning, which is most real questions. I turn hybrid on by default for anything with IDs, codes, or product names in it.

Query rewriting and multi-query

One question is often three questions wearing a trench coat. 'The payment service is down, how do I bring it back' is a diagnosis question, a restart question, and a verification question. I rewrite it into all three and search with each one. The merge gives me chunks for the whole job, not just the restart steps. The risk is drift. Each rewrite can wander a little further from what the user meant, and three wandering queries fetch three piles of noise. I cap it at three rewrites, I log all of them, and I check the logs when the answers drift.

Lost in the middle

Models read the start and the end of a long prompt carefully and skim the middle. This is not a bug in one model. It is how attention works when the prompt gets long. The fix is not a better model. The fix is a shorter prompt with better chunks. Keep top-k small, rerank hard, and put the best chunk first. If the answer needs a long doc, I retrieve the doc in pieces and let the model read one piece at a time, instead of stuffing the whole doc into one prompt and hoping the middle survives.

Chunking strategies

Fixed-size chunks are the default: cut the doc every 512 tokens, with 50 tokens of overlap so a sentence is not cut in half. It is simple and it works until a procedure or a table gets split across the cut. Semantic chunking cuts at topic changes instead of token counts, so a section stays whole, but it costs an extra pass over the doc. Parent-child chunking keeps both: small child chunks for the search, and the whole parent section for the prompt, so the model gets the full procedure even though the search matched one small piece. I start with fixed chunks and overlap, and I move to parent-child the first time a split procedure causes a bad answer. That is usually within the first month.

The stale index problem

An index is a snapshot of your docs at the moment you built it. Docs change every day. Runbooks get rewritten. Procedures get retired. If the index does not change with them, the system answers from the past. I treat re-embedding like a deploy. Every doc publish triggers a re-embed of the changed pages. Every re-embed writes a timestamp. And I watch one metric: the age of the oldest chunk in the index. If that age passes a day, something in the pipeline is stuck, and I would rather know at 10 AM than find out from a wrong answer at 2 AM.

RAG evals

I do not judge a RAG system by reading a few answers and feeling good. I keep a set of questions with known right answers, and I run the whole set every night. I measure three things. Faithfulness: does every claim in the answer trace to a chunk. Context precision: of the chunks retrieved, how many were actually useful. Context recall: of the chunks that would have answered the question, how many did we retrieve. If faithfulness drops, the model is inventing. If precision drops, the prompt is filler. If recall drops, the index or the chunking is wrong. Three numbers, and each one points at a different hop.

What changes at 10x and 100x

At 10x the questions, the embedding hop is the first thing that breaks, because every question pays for a model call before the search even starts. I cache embeddings for repeat questions, and I batch the embedding calls. At 100x, the index is the design decision. One index for everything is simple until one team's docs drown out another team's. I split indexes by team or by doc type, and I route the question to the right index before I search. The budget stops being a target and becomes a contract, and every hop owns its share of it. And the eval set grows with the traffic, because the questions users actually ask are the only honest test set.

The retrieval config

One file, checked into git, reviewed like code. No magic numbers in the app.

# retrieval.yaml — the retrieval settings for this lesson's system
# One file, checked into git, reviewed like code. No magic numbers in the app.
chunking:
  size_tokens: 512        # one chunk is about half a page
  overlap_tokens: 50      # so a sentence is never cut in half
  strategy: fixed         # move to parent-child when a split procedure bites
embedding:
  model: local-embed-v1   # SAME model for questions and docs, always
  dimensions: 768
retrieval:
  top_k: 5                # candidates from the search
  hybrid: false           # turn on when questions carry IDs or error codes
  reranker: true          # keep the best 3, drop the rest
  keep_after_rerank: 3
grounding:
  require_citation: true  # no citation, no answer
  refuse_message: "I could not find this in the runbooks. Here are the closest pages."
# nightly eval check — one question from the eval set, three numbers out
result = rag_answer("How do I restart the payment service safely?")
claims = result.claims
faithful = sum(1 for c in claims if c.has_citation) / len(claims)
print(f"faithfulness: {faithful:.2f}  (want 1.00, investigate below 0.95)")
print(f"chunks retrieved: {len(result.chunks)}  useful: {result.useful_chunks}")

One time it broke

This one stayed with me because the system never errored. Every dashboard was green, every hop answered on time, and the answers were wrong for two weeks before anyone noticed.

What broke

A support assistant answered questions from internal docs. A team migrated their runbooks to a new docs site and rewrote the restart procedures in the move. The old pages were deleted. The index was built once, on the day the assistant launched, and nobody rebuilt it. So the assistant kept answering from the old runbooks. It cited them by name. The citations looked right. The procedures were retired. One night an engineer followed the assistant's steps for a service restart, and the steps told him to stop a service that the new procedure says must stay running during the restart. He caught it because he knew the service. The next engineer might not have.

Timeline

  • Day 0The docs team migrates the runbooks to the new site and rewrites the restart procedures. The old pages are deleted the same day.
  • Day 1The first question about the restart gets the old answer. Nobody notices, because the old answer reads well and cites a real page name.
  • Day 4A second engineer gets the same old answer and follows it halfway before a teammate stops him. He mentions it in chat. Nobody files a ticket.
  • Day 9I am asked to look at the assistant because 'the answers feel stale.' I check the index. The newest chunk in it is 41 days old.
  • Day 9, one hour laterI check the re-embed job. It ran once, on launch day, and then silently stopped when the docs site changed its URL pattern. No alert fired, because nobody alerted on it.
  • Day 10I rebuild the index from the new docs site and re-embed everything. The restart question now returns the new procedure.
  • Day 12I confirm from the eval set that the stale answers are gone, and the grounding check now refuses the one question the new docs still do not cover.

What changed after

Three things. First, every doc publish now triggers a re-embed of the changed pages, the same way a code merge triggers a deploy. The index is a build artifact, not a one-time setup. Second, I watch one metric: the age of the oldest chunk in the index, with an alert at 24 hours. If the pipeline stops, I know the same day, not six weeks later. Third, the grounding check now compares the citation against the live doc, not just the chunk. If the doc the chunk came from no longer exists, the answer is refused instead of served. A refusal is embarrassing. A confident wrong answer at 2 AM is dangerous.

If you remember nothing else

Design exercise

This is the kind of question I would ask in an interview, and the kind I ask myself before I sign off on a real retrieval system. Take a minute with it before you read my approach.

The prompt

Design a RAG assistant for 50,000 internal runbooks and docs. It must answer 2,000 questions a day, answer in under 3 seconds at p95, cite its sources, and never invent a procedure. You have an embedding model, a vector store, a language model, and a docs site that publishes changes every day. Sketch the pipeline, the index, and the budget.

Clarifying questions to ask first

I would ask five things before I sized anything. Who asks the questions, and what do they ask about: runbooks, tickets, or both, because the chunking is different for each. How fresh must the answers be: is a one-hour-old index fine, or does a doc publish need to show up in minutes. What happens when the docs do not cover the question: refuse, or show the closest pages and say so. Is 3 seconds the target for the first token or the full answer, because streaming changes the budget completely. And what is the failure we design for: the vector store down, the embedding model slow, or the docs site publishing a bad page.

Rough sizing, worked

Start with the index. 50,000 docs, about 10 chunks each, is 500,000 chunks. Each chunk gets a 768-number fingerprint at 4 bytes a number, which is about 3 KB per chunk, so the vectors alone are about 1.5 GB. The chunk text on top of that is roughly another 1 GB. That fits on one machine with room to spare, so I start with one index and split it later if a team's docs drown out the rest. Now the traffic. 2,000 questions a day is about one question every 43 seconds on average, and maybe 5 questions a second at the morning peak. Little's Law for the sizing: 5 questions a second at 1.1 seconds in the system is about 6 questions inside at once, so the model and the vector store barely notice the load. The budget from this lesson fits: rewrite 25, embed 40, search 60, rerank 35, prompt 15, model 890, check 30, send 10, for 1,105 ms total, well inside 3 seconds, with room for a slower model or a bigger prompt. The freshness math is the real sizing. If the docs site publishes 200 changed pages a day, that is 2,000 changed chunks a day, and re-embedding them takes minutes, not hours, so I re-embed on every publish and keep the index under an hour old.

Trade-offs to name

A bigger top-k buys recall and costs model time and lost-in-the-middle risk, so I keep it at 5 and let the reranker earn its keep. Hybrid search buys exact matches on error codes and costs a second search pass, so I turn it on for runbooks and off for plain prose. A strict grounding check buys safety and costs refusals on good questions the docs almost cover, so I tune it on the eval set and let it show the closest pages instead of a bare refusal. One big index buys simplicity and costs noisy results across teams, so I split by team when the eval precision drops. And the 3-second target is p95, not the average. An average of 1.1 seconds with a p95 of 6 seconds fails this design, and the tail is usually the model, not the search.

Saved in this browser, so the Lessons index can show your progress across the course.

Next in this series
Lesson 61 — LLM Pipelines

Retrieval hands the model the right pages. The next lesson puts the whole pipeline together: how prompts, tools, retries, and fallbacks chain into a system that survives a bad model call at 2 AM.