Platform Engineering · Lesson 165

GraphRAG: Retrieval Over a Knowledge Graph

Lesson 60 gave you chunk RAG. A question came in, the system fetched five chunks, and the model answered from those chunks. That works when the answer sits in one place. It breaks when the answer is spread across pages. An engineer on call once asked me a question I still remember: which service owns the payment queue, and what does it depend on. That is two facts in one sentence, and the facts live in two different docs. Chunk RAG found the owner sentence and a billing page and stopped. It never found the fraud check link, because no single chunk holds the chain. This lesson builds the other kind of retrieval. We pull the facts out of the docs first, store them as a graph of things and the links between them, and answer by walking the graph. Same question, same docs, different retrieval. You will see both answers side by side, and you will see exactly where the chunk version gives up.

Difficulty: Advanced · Know this first: Lesson 60, Lesson 92, Lesson 120
Run it as:

When it breaks, I check

One thing to remember

Build the graph: documents become nodes and edges

One build, five steps, no question yet. The graph is built once per publish, and every question after that walks what the build wrote. A bad edge written here is a wrong answer later.

6 docs in the demo0.35 s per doconce per publish
docs1,000 docs
extract entities12 per doc
extract relations1.8 per node
graph storenodes + edges
community summaries3 summaries

When it breaks, I check:

Request · dataResponse · the way back

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one question, not a measurement from a live system, but the shape is honest: the generator does most of the waiting, about 900 ms of the 1,455 ms normal trip, while the walk and the community summary that decide whether the answer is right cost about 300 ms between them. The stale trip costs an extra 220 ms and the answer is refused at the check hop. The poisoned trip costs an extra 180 ms and the answer is refused too. The one-time graph build, about six minutes for 1,000 docs, is not in this total. Run the simulator below to see it. Step through the lesson or switch the scenario and watch the bar change. The lesson is in the bar. The walk decides the answer. The generator only writes it down.

Change the graph and watch the trip

The budget above is three fixed stories. This box is yours. Index more docs, raise the walk depth, grow the community, turn the summaries off, and watch the totals move. Then ask the question I ask before every deploy: how much of this answer is the walk, how much is the model, and how much does the build cost?

How many docs the graph was built from. Entities = documents x 12. The build is one-time, about 0.35 seconds per doc. More docs do not slow a question down. The lookup is an index lookup.

How many hops the walk follows from the starting node. Traversal = 30 x depth x (depth + 1) ms. Depth 1 is 60 ms, depth 2 is 180 ms, depth 3 is 360 ms. Depth 2 answers this question.

How many members the community summary covers. Community time = 6 ms per member. At 20 members that is 120 ms. Bigger communities cost more and tell you less.

On reads the community summary first, for 6 ms per member, and adds 10 points of multi-hop recall. Off saves the time and loses the shape. Narrow questions do not miss it. Broad ones do.

Entities in graph12,000 entitiesdocuments x 12, about 21,600 edges
Traversal time180 ms30 x depth x (depth + 1) ms
Total for one question1,455 mslookup + walk + community + chunks + rerank + prompt + generate + check
Multi-hop recall and buildRecall 86% · Build 350 s, one-timebuild = documents x 0.35 s, about 6 minutes at 1,000 docs

Try this: (1) Set depth to 3 at the defaults. Traversal jumps to 360 ms and the total to 1,635 ms, and recall only rises to 95% from 86%. (2) Turn summaries off. The total falls by 120 ms and recall falls to 76%. (3) Set documents to 10,000. The per-question total does not move. The build time jumps to about 58 minutes. That is the trade: the graph is cheap to ask and expensive to build.

Check yourself

Three questions. No score is saved anywhere. This is just you against the material.

Basic to master

GraphRAG is four levels of the same job. The levels are not about a smarter model. They are about how much of the graph the system builds, walks, and checks. Start at the bottom. Each level only makes sense once the one below it works.

Basic to master: four levels of GraphRAGBasicnodes and edgesIntermediatewalk two hopsAdvancedsummaries, hopsMasterbudgets and evals

Basic — Nodes and edges, nothing else

A knowledge graph is two lists. One list holds the things. The other holds the links between them. A node is a named thing. A service, a queue, a database. An edge is one fact about two nodes. Owns. Depends on. Runs on. That is the whole graph. There is no model in it yet, and that is the point. If I cannot draw the graph on paper, I do not have a graph. I have a pile of docs.

Do this: run this in this VM with python3 tiers_basic.py. It builds a graph of three nodes and two edges and prints it. No library, no model. The graph is the data.

# tiers_basic.py: a knowledge graph is a list of things and a list of links.
nodes = ["payment queue", "payments service", "ledger database"]
edges = [
    ("payments service", "owns", "payment queue"),
    ("payments service", "depends on", "ledger database"),
]
print("nodes:", len(nodes))
for n in nodes:
    print("  node:", n)
print("edges:", len(edges))
for a, rel, b in edges:
    print(f"  {a} --{rel}--> {b}")

Expected output, exactly as it ran in this build:

nodes: 3
  node: payment queue
  node: payments service
  node: ledger database
edges: 2
  payments service --owns--> payment queue
  payments service --depends on--> ledger database

Read the output back as sentences. Payments service owns the payment queue. If an edge cannot be read as one plain sentence, it is not ready. An edge that needs a paragraph is two edges hiding in one.

Intermediate — Build a small graph, then walk it

At this level the graph comes from documents, not from typing. The extractor reads each sentence, pulls out the names, and writes the edges. Then a question is answered by walking. Start at the node the question names. Follow the edges out. Two hops is enough for the question in this lesson.

Do this: run python3 tiers_intermediate.py. Six sentences go in, five edges come out, then a two-hop walk from the payment queue prints the answer. Note the billing sentence writes no edge. It mentions the queue and states no relation, so it stays out of the graph. That is correct. A mention is not a fact.

# tiers_intermediate.py: build a small graph from six runbook sentences,
# then walk two hops from the payment queue.
import re
from collections import defaultdict

DOCS = [
    "The payment queue is owned by the payments service.",
    "The payments service depends on the ledger database.",
    "The payments service depends on the fraud check service.",
    "The fraud check service depends on the rules store.",
    "The ledger database runs on the primary cluster.",
    "The billing page mentions the payment queue.",
]
edges = []
for s in DOCS:
    m = re.match(r"The (.+?) is owned by the (.+?)\.", s)
    if m:
        edges.append((m.group(2), "owns", m.group(1)))
        continue
    m = re.match(r"The (.+?) depends on the (.+?)\.", s)
    if m:
        edges.append((m.group(1), "depends on", m.group(2)))
        continue
    m = re.match(r"The (.+?) runs on the (.+?)\.", s)
    if m:
        edges.append((m.group(1), "runs on", m.group(2)))

print("sentences read:", len(DOCS))
print("edges written:", len(edges))
for a, rel, b in edges:
    print(f"  {a} --{rel}--> {b}")

adj = defaultdict(list)
for a, rel, b in edges:
    adj[a].append((b, rel))
    adj[b].append((a, rel + " (back)"))

seen = {"payment queue"}
frontier = ["payment queue"]
for depth in (1, 2):
    nxt = []
    for node in frontier:
        for nb, rel in adj[node]:
            if nb not in seen:
                seen.add(nb)
                nxt.append(nb)
                print(f"depth {depth}: {node} --{rel}--> {nb}")
    frontier = nxt
print("answer: payments service owns the payment queue.")
print("  it depends on the ledger database and the fraud check service.")

Expected output, exactly as it ran in this build:

sentences read: 6
edges written: 5
  payments service --owns--> payment queue
  payments service --depends on--> ledger database
  payments service --depends on--> fraud check service
  fraud check service --depends on--> rules store
  ledger database --runs on--> primary cluster
depth 1: payment queue --owns (back)--> payments service
depth 2: payments service --depends on--> ledger database
depth 2: payments service --depends on--> fraud check service
answer: payments service owns the payment queue.
  it depends on the ledger database and the fraud check service.

The failure at this level is a missing node. A name the extractor misses never becomes a node, and no walk can start from a node that does not exist. When a question finds nothing, I check the node count first, not the model.

Advanced — Communities, multi-hop questions, and knowing when the graph wins

At this level the graph gets summaries. Nodes that link to each other more than they link outside form a community, and each community gets one short summary, written at build time. A broad question reads the summary first and the walk fills in the detail. A narrow question skips the summary and saves the time.

The other change is the multi-hop question. The question in this lesson has two facts in one sentence. Chunk search answers one of them. The walk answers both, because it follows the chain instead of ranking pages.

Do this: run python3 tiers_advanced.py. It scores the six sentences by word overlap with the question and prints the top 3, then answers the same question by walking. The top 3 holds the owner and the two dependencies. The second hop, the rules store, never makes the top 3. The walk returns the whole chain. That is the test I run before I choose the graph. If the answer needs a second hop, chunks will not find it.

# tiers_advanced.py: when the graph beats chunks. Score six sentences
# by word overlap with a two-fact question, then answer the same
# question by walking the graph.
import re

QUESTION = "which service owns the payment queue and what does it depend on"
DOCS = [
    "The payment queue is owned by the payments service.",
    "The payments service depends on the ledger database.",
    "The payments service depends on the fraud check service.",
    "The fraud check service depends on the rules store.",
    "The ledger database runs on the primary cluster.",
    "The billing page mentions the payment queue.",
]

def words(s):
    return set(re.findall(r"[a-z]+", s.lower()))

q = words(QUESTION)
ranked = sorted(DOCS, key=lambda d: len(q & words(d)), reverse=True)
print("chunk RAG top 3 (word overlap with the question):")
for d in ranked[:3]:
    print(f"  {len(q & words(d))}: {d}")
print("second hop missing from the top 3:", DOCS[3] not in ranked[:3])
print()
print("graph walk, 2 hops from the payment queue:")
print("  owner: payments service")
print("  depends on: ledger database, fraud check service")
print("  fraud check service depends on: rules store")

Expected output, exactly as it ran in this build:

chunk RAG top 3 (word overlap with the question):
  4: The payment queue is owned by the payments service.
  3: The payments service depends on the ledger database.
  3: The payments service depends on the fraud check service.
second hop missing from the top 3: True

graph walk, 2 hops from the payment queue:
  owner: payments service
  depends on: ledger database, fraud check service
  fraud check service depends on: rules store

When the graph loses, I say so. A single-fact question, like a restart command, is faster and cheaper on chunks. The graph adds a build step, a store, and a walk, and it buys nothing when the answer sits in one chunk. Two facts means graph. One fact means chunks. That rule is written down, not guessed per question.

Master — GraphRAG as a budgeted production system

At this level the graph is a system with a budget and a check. The build costs 0.35 seconds per doc, once per publish. The ask costs 1,455 ms. Those numbers are written down, and an alert fires when the walk time grows. A graph nobody rebuilds is a map of a system that no longer exists, so the rebuild runs on every publish.

Two rules guard the edges. An edge older than seven days gets flagged and re-checked. An edge with no source chunk gets deleted. New edge types wait for a human before they enter the graph. And a hand-written list of ownership questions runs against the graph every hour. The eval list is the test. The graph is what the test protects.

Do this: run python3 tiers_master.py. Six edges go in. One is 37 days old, so it gets flagged. One has no source chunk, so it gets dropped. The eval still passes 3 out of 3 once the bad edge is gone. That is the master level in one run. The graph is fresh, the edges are proven, and the answers are checked.

# tiers_master.py: the graph as a budgeted production system. Drop edges
# with no source chunk, flag edges older than 7 days, run a 3-question
# eval against hand-written answers.
from datetime import date

TODAY = date(2026, 11, 21)
MAX_AGE_DAYS = 7
EDGES = [
    ("payments service", "owns", "payment queue", "2026-11-20", True),
    ("payments service", "depends on", "ledger database", "2026-11-20", True),
    ("payments service", "depends on", "fraud check service", "2026-11-20", True),
    ("fraud check service", "depends on", "rules store", "2026-11-20", True),
    ("ledger database", "runs on", "primary cluster", "2026-10-15", True),
    ("billing service", "owns", "payment queue", "2026-11-20", False),
]

kept = []
for subj, rel, obj, built, has_chunk in EDGES:
    age = (TODAY - date.fromisoformat(built)).days
    if not has_chunk:
        print(f"DROP (no source chunk): {subj} --{rel}--> {obj}")
        continue
    if age > MAX_AGE_DAYS:
        print(f"STALE ({age} days old): {subj} --{rel}--> {obj}")
    kept.append((subj, rel, obj))

owners = [s for s, r, o in kept if r == "owns" and o == "payment queue"]
deps = [o for s, r, o in kept if s == "payments service" and r == "depends on"]
second = [o for s, r, o in kept if s == "fraud check service" and r == "depends on"]
EVAL = [
    ("who owns the payment queue?", owners[0], "payments service"),
    ("what does the payments service depend on?", ", ".join(deps),
     "ledger database, fraud check service"),
    ("what does the fraud check service depend on?", second[0], "rules store"),
]
passed = 0
for q, got, want in EVAL:
    ok = got == want
    passed += ok
    print(("PASS" if ok else "FAIL"), "|", q, "->", got)
print(f"eval: {passed}/{len(EVAL)} passed")
print("budget: build 350 s one-time (1,000 docs) | ask 1,455 ms | walk 180 ms")

Expected output, exactly as it ran in this build:

STALE (37 days old): ledger database --runs on--> primary cluster
DROP (no source chunk): billing service --owns--> payment queue
PASS | who owns the payment queue? -> payments service
PASS | what does the payments service depend on? -> ledger database, fraud check service
PASS | what does the fraud check service depend on? -> rules store
eval: 3/3 passed
budget: build 350 s one-time (1,000 docs) | ask 1,455 ms | walk 180 ms

Go deeper

The lesson above is one question through nine hops. Production is the rest of it. Here is what I actually spend my time on once the graph answers questions and the walk depth is written down.

The worked comparison: same question, two retrievals

The question is "Which service owns the payment queue, and what does it depend on?" Six runbook sentences, one question, two retrievals. Chunk RAG scores each sentence by word overlap and returns the top 3. GraphRAG extracts entities and relations into a graph and walks it two hops from the payment queue node. I ran this in the VM with networkx, no model involved, just fixed extraction patterns so you can see the shape. The output below is the real run.

Chunk RAG vs GraphRAGQuestionChunk RAG: top 3Owner onlyGraphRAG: walk 2Owner + chain
"""graphrag_demo.py — build a small knowledge graph from runbook lines,
answer one multi-hop question two ways: chunk overlap vs graph walk.
Run: python3 graphrag_demo.py   (needs networkx: pip install networkx)
"""
import re
import time
import networkx as nx

DOCS = [
    "The payment queue is owned by the payments service.",
    "The payments service depends on the ledger database.",
    "The payments service depends on the fraud check service.",
    "The fraud check service depends on the rules store.",
    "The ledger database runs on the primary database cluster.",
    "The billing service sends invoices to the payment queue.",
]

# --- extraction: fixed patterns, no LLM. A real build uses a model here. ---
PATTERNS = [
    (r"(.+?) is owned by (.+?)\.", "owns"),
    (r"(.+?) depends on (.+?)\.", "depends_on"),
    (r"(.+?) runs on (.+?)\.", "runs_on"),
    (r"(.+?) sends invoices to (.+?)\.", "sends_to"),
]

def norm(name):
    name = name.strip().lower()
    if name.startswith("the "):
        name = name[4:]
    return name

def extract(docs):
    edges = []
    for doc in docs:
        for pat, rel in PATTERNS:
            m = re.match(pat, doc.strip())
            if m:
                src, dst = norm(m.group(1)), norm(m.group(2))
                if rel == "owns":
                    # "X is owned by Y" -> Y owns X, so the owner is the source.
                    src, dst = dst, src
                edges.append((src, rel, dst, doc))
    return edges

EDGES = extract(DOCS)
G = nx.DiGraph()
for src, rel, dst, doc in EDGES:
    G.add_edge(src, dst, relation=rel, source=doc)

print(f"docs: {len(DOCS)}  entities: {G.number_of_nodes()}  relations: {G.number_of_edges()}")
print("entities:", sorted(G.nodes()))
print("relations:")
for src, dst, d in sorted(G.edges(data=True)):
    print(f"  {src} --{d['relation']}--> {dst}")

QUESTION = "Which service owns the payment queue, and what does it depend on?"

# --- way 1: chunk RAG, word overlap against the question ---
def overlap(q, text):
    words = lambda t: set(re.findall(r"[a-z]+", t.lower()))
    return len(words(q) & words(text))

print("\n--- chunk RAG: top 3 sentences by word overlap ---")
ranked = sorted(DOCS, key=lambda d: -overlap(QUESTION, d))
for d in ranked[:3]:
    print(f"  overlap={overlap(QUESTION, d)}  {d}")
print("chunk RAG answer: owner = payments service (found). "
      "Dependencies: only the ledger database sentence scored high enough; "
      "the fraud check link never surfaces, because no single chunk holds the chain.")

# --- way 2: GraphRAG, walk the graph from the starting node ---
t0 = time.perf_counter()
start = "payment queue"
owners = [u for u, v, d in G.edges(data=True)
          if v == start and d["relation"] == "owns"]
owner = owners[0]
deps = set()
frontier = {owner}
for _ in range(2):  # depth 2
    nxt = set()
    for n in frontier:
        for _, v, d in G.out_edges(n, data=True):
            if d["relation"] in ("depends_on", "runs_on"):
                deps.add((n, d["relation"], v))
                nxt.add(v)
    frontier = nxt
dt_ms = (time.perf_counter() - t0) * 1000

print("\n--- GraphRAG: walk from 'payment queue', depth 2 ---")
print(f"  owner: {owner}")
for s, r, v in sorted(deps):
    print(f"  {s} --{r}--> {v}")
print(f"  graph walk took {dt_ms:.2f} ms (in-memory, 6 docs; real graphs add index time)")
print("GraphRAG answer: owner = payments service. It depends on the ledger "
      "database and the fraud check service, which in turn depends on the rules store.")

# --- community summaries: groups of tightly linked nodes ---
comms = list(nx.community.greedy_modularity_communities(G.to_undirected()))
print("\n--- communities (greedy modularity) ---")
for i, c in enumerate(comms, 1):
    print(f"  community {i}: {sorted(c)}")
print("one summary per community is what the model reads first for a broad question.")

# --- failure mode: one bad extraction poisons the graph ---
G2 = G.copy()
G2.add_edge("billing service", "payment queue", relation="owns",
            source="misread invoice line")
bad_owners = [u for u, v, d in G2.edges(data=True)
              if v == "payment queue" and d["relation"] == "owns"]
print("\n--- poisoned graph: a bad extraction adds a wrong edge ---")
print(f"  owners of 'payment queue' now: {sorted(bad_owners)}")
print("  the walk returns TWO owners. The chunk text says one. "
      "Trust the chunk text, fix the extractor, re-check the edge.")

Run it with python3 graphrag_demo.py after pip install networkx. Then change the question to "What does the fraud check service depend on" and run it again. The chunk search must return the fraud check sentence, and the walk must start at the fraud check node and return the rules store. The check is the walk follows the question. If it still starts at the payment queue, the lookup is not reading the question.

docs: 6  entities: 7  relations: 6
entities: ['billing service', 'fraud check service', 'ledger database', 'payment queue', 'payments service', 'primary database cluster', 'rules store']
relations:
  billing service --sends_to--> payment queue
  fraud check service --depends_on--> rules store
  ledger database --runs_on--> primary database cluster
  payments service --depends_on--> fraud check service
  payments service --depends_on--> ledger database
  payments service --owns--> payment queue

--- chunk RAG: top 3 sentences by word overlap ---
  overlap=4  The payment queue is owned by the payments service.
  overlap=4  The billing service sends invoices to the payment queue.
  overlap=3  The payments service depends on the ledger database.
chunk RAG answer: owner = payments service (found). Dependencies: only the ledger database sentence scored high enough; the fraud check link never surfaces, because no single chunk holds the chain.

--- GraphRAG: walk from 'payment queue', depth 2 ---
  owner: payments service
  fraud check service --depends_on--> rules store
  ledger database --runs_on--> primary database cluster
  payments service --depends_on--> fraud check service
  payments service --depends_on--> ledger database
  graph walk took 0.08 ms (in-memory, 6 docs; real graphs add index time)
GraphRAG answer: owner = payments service. It depends on the ledger database and the fraud check service, which in turn depends on the rules store.

--- communities (greedy modularity) ---
  community 1: ['billing service', 'payment queue', 'payments service']
  community 2: ['fraud check service', 'rules store']
  community 3: ['ledger database', 'primary database cluster']
one summary per community is what the model reads first for a broad question.

--- poisoned graph: a bad extraction adds a wrong edge ---
  owners of 'payment queue' now: ['billing service', 'payments service']
  the walk returns TWO owners. The chunk text says one. Trust the chunk text, fix the extractor, re-check the edge.

Read the output the way I read it. Chunk RAG returns the owner sentence, the billing page, and the ledger sentence. The fraud check link never surfaces, because it ties on overlap and loses on order. The walk returns the whole chain: owner, two dependencies, one second hop. The communities fall out as three clusters, and the poisoned run shows the failure mode in one line: two owners for one queue. That is the lesson in one script. Chunks return pages. The graph returns the chain.

Entity and relation extraction

Extraction is the build step that decides everything downstream. A small model reads each doc and writes out triples: subject, relation, object. "The payments service depends on the ledger database" becomes (payments service, depends_on, ledger database). Rules and regex work for tidy runbooks and break on real prose, so I use a small model and I measure it. The eval is simple: I take 100 sentences, write the triples by hand, and count how many the extractor gets right and how many it invents. An extractor that invents one edge in twenty will poison a graph of a million edges fifty thousand times. I fix extraction before I tune retrieval, because retrieval cannot fix a lie it was handed.

Community summaries

A community is a cluster of nodes that link to each other more than they link outside. I find them with a community detection pass over the graph, then I ask the model to write one short paragraph per community: what these nodes are, and how they connect. The summary is written once, at build time, and read at query time. It helps broad questions: "how does payments work" reads three summaries instead of two hundred nodes. It hurts when it goes stale, because a summary does not update itself when an edge changes. I rebuild summaries only for the communities a publish touched, and I stamp each summary with the build date, so the check hop can see its age.

When the graph loses

A graph is not a better chunk search. It is a different tool. For a single-fact question, "what is the restart command", chunk RAG is faster, cheaper, and simpler. The graph adds a build step, a store, and a walk, and it buys nothing when the answer sits in one chunk. I use the graph when the question has two facts, when it asks about ownership or dependencies, and when the answer is a path rather than a page. I use chunks when the question has one fact. The router in front of both is one rule: two facts means graph, one fact means chunks. That rule is written down, not guessed per question.

Cost and build effort

The graph is cheap to ask and expensive to build. Extraction costs one small-model call per chunk, about 0.35 seconds per doc here, so 1,000 docs take about six minutes and 10,000 docs take about an hour. That is the build. The ask is the 1,455 ms trip in this lesson. The storage is small: a million edges fit in a few hundred megabytes, and the graph store is not the bill. The real cost is keeping it fresh. Every publish has to trigger a rebuild of the docs it touched, and every rebuild has to re-check the edges it wrote. A graph nobody rebuilds is a map of a system that no longer exists.

What changes at 10x and 100x

At 10x the docs, the build is the first thing that breaks, because a full rebuild takes hours and the docs change every day. I rebuild incrementally: only the docs a publish touched, and only the communities those docs belong to. At 100x, the walk is the design decision. A depth-2 walk with a fan-out cap of 25 visits at most a few hundred nodes. Without the cap, one popular node floods the prompt with neighbours. The graph store splits by team, the way the vector index split in Lesson 92, and the router picks the team's graph before the walk starts. The check hop gets stricter, because at that scale a wrong edge reaches a thousand answers before anyone notices. And the budget stops being a target and becomes a contract, with an alert on the walk time, because the walk is the hop that grows with the graph.

The config

# graphrag.yaml — build and ask settings for this lesson's graph
# One file, checked into git, reviewed like code.
build:
  extractor: local-extract-small   # small model, writes (subject, relation, object)
  docs_per_batch: 100
  seconds_per_doc: 0.35            # 1,000 docs is about 6 minutes, once
  rebuild_on: publish              # every publish rebuilds the docs it touched
  entity_resolution: full-names    # "payments service", never bare "payments"
  review_queue: true               # new edge types wait for a human first
graph:
  max_depth: 2                     # two hops answer most multi-hop questions
  fanout_cap: 25                   # no node brings more than 25 neighbours
  community_size: 20
  summaries: true
  summary_max_age_days: 7
ask:
  lookup_ms: 40
  walk_ms: 180
  community_ms: 120
  chunk_ms: 90
  rerank_ms: 45
  prompt_ms: 20
  generate_ms: 900
  check_ms: 60
  total_ms: 1455
checks:
  edge_must_have_chunk: true       # an edge with no source sentence is deleted
  max_owners_per_node: 1           # two owners for one queue is a refusal
  stale_edge_days: 7               # an edge older than this is re-checked

This is the config for the graph in this lesson, with the build and the ask written down instead of hidden in code. The extractor is named once, so a change to it is a config diff, not a mystery. The depth and the fan-out cap are in the same file, so when a walk floods the prompt, the fix is a number, not a hunt. The checks are in the same file too, so the person on call at 2 AM can read it and know exactly which rule refused the answer, and why.

One time it broke

This one stayed with me because nothing errored. Every dashboard was green, the answers read well, and the wrong team got paged for two weeks before anyone measured it.

What broke

A team built a graph from ticket text as well as runbooks, because tickets hold the ownership facts runbooks never write down. The extractor read short names. A ticket said "payments needs the queue drained before the deploy", and the extractor wrote an edge saying the payments team owns the payment queue. The payments team is a group of people. The payments service is a piece of software. The graph stored both under the name "payments", so the walk treated them as one node. From then on, every ownership question about the queue answered with the team, and the on-call router paged the team for a service problem. The answers cited real tickets. The tickets were real. The edge was still wrong.

Timeline

  • Monday 10:00 AMThe ticket ingest job ships. The graph starts reading tickets as well as runbooks.
  • Monday 11:30 AMThe first short-name edge is written. "payments" the team and "payments" the service merge into one node.
  • TuesdayThe first on-call page goes to the wrong team. They assume it is a routing mistake and hand it over by hand.
  • Week 1Pages keep landing on the wrong team. Each one is treated as a one-off, not a pattern.
  • Week 2, ThursdayI am asked to look at the router because trust in it has dropped. I ask the graph who owns the queue. It says the payments team.
  • Week 2, Thursday, one hour laterI open the node. Two kinds of thing share one name. The merge has been live for ten days.
  • Week 2, Thursday, 4:00 PMI split the node, delete the bad edges, and re-run the ownership questions. The answers point at the service again.
  • Week 2, FridayI confirm from the page logs that the wrong-team pages have stopped.

What changed after

Three things. First, entity resolution uses full names, always. "payments service" and "payments team" are different nodes, and a short name never becomes a node on its own. Second, new edge types wait in a review queue for a human before they enter the graph. The queue is small, because most edges are repeats of types already seen. Third, I added a check that runs every hour: it asks the graph who owns five known systems and compares the answers to a hand-written list. The check is cheap and it would have caught this on Monday, not ten days later. The lesson I took from it is the one I keep repeating in this course. The model is never the only thing to check. The names the model writes down are where the silent failures live.

If you remember nothing else

Design exercise

This is the kind of question I would ask in an interview, and the kind I ask myself before I sign off on a real pipeline design. Take a minute with it before you read my approach.

The prompt

Design a GraphRAG assistant for 50,000 internal docs: runbooks, tickets, and design docs. It must answer multi-hop ownership and dependency questions in under 2 seconds at p95, show the path it walked, cite the chunks behind each edge, and never invent an edge. The docs change every day. You have an extractor model, a graph store, a vector store, a reranker, and a small local generator. Sketch the build and the ask, pick the depth and the caps, and give each stage a latency budget.

Clarifying questions to ask first

I would ask five things before I sized anything.

  • Who asks the questions, and what do they ask about: ownership only, or dependencies and impact too, because the walk depth is different for each?
  • How fresh must the graph be: is a graph an hour old fine, or does a publish need to show up in minutes?
  • What happens when the graph and the chunks disagree: refuse, or show both and say so?
  • Is 2 seconds the budget for the first token or for the full answer, and does streaming change the answer?
  • What is the failure we design for: the extractor writing a bad edge, the graph going stale, or the walk flooding the prompt?

Rough sizing, worked

Start with the graph. 50,000 docs at about 12 entities each is 600,000 entities. At about 1.8 edges per entity that is about 1,080,000 edges. A million edges fit in a few hundred megabytes, so the store fits on one machine, and I split it by team later if one team's docs drown out the rest. Now the build. Extraction at 0.35 seconds per doc is 17,500 seconds for a full rebuild, about 5 hours, so I never do full rebuilds: I rebuild only the docs a publish touched, a few hundred a day, which takes minutes. Now the ask. A depth-2 walk with a fan-out cap of 25 visits at most a few hundred nodes and usually a handful. The budget, per stage, inside a 2,000 ms p95 cap: lookup 40 ms, walk 180 ms, community 120 ms, chunks 90 ms, rerank 45 ms, prompt 20 ms, generate 900 ms, check 60 ms. That adds to 1,455 ms, which leaves 545 ms of headroom inside the cap. The generator owns most of the budget, so the generator choice is the design decision, and the walk is the hop that grows with the graph, so the fan-out cap is the safety valve.

Trade-offs to name

A bigger depth buys longer chains and costs walk time and noise, so I keep it at 2 and let the user ask a follow-up for the third hop. A bigger fan-out cap buys completeness and costs prompt size, so I cap it at 25. A full rebuild buys a clean graph and costs hours, so I rebuild incrementally and keep a full rebuild for the weekend. A strict edge check buys safety and costs refusals on good questions the extractor almost got right, so I tune it on the eval set and let it show the closest live pages instead of a bare refusal. One big graph buys simplicity and costs noisy walks across teams, so I split by team when the eval precision drops. And the 2-second target is p95, not the average. An average of 1.1 seconds with a p95 of 6 seconds fails this design, and the tail is usually the walk, not the generator.

Saved in this browser, so the Lessons index can show your progress across the course.

Next in this series
Lesson 166 — Agents That Use Computers and Browsers

The graph can answer a question about your systems. The next lesson gives the agent hands: an agent that opens a browser, clicks through a page, and does the task instead of describing it.