Platform Engineering · Lesson 170

LangChain: Chains, Prompts, and Agents

Lesson 75 named LangChain in one line. It said chaining means retrieve, build the prompt, call the model, check the answer, with one check between each step. That line was a promise. This lesson keeps it. LangChain is the framework that turns those five steps into code you can read. A prompt template holds the words. A loader reads the runbook. A retriever finds the pages. A parser checks the shape of the answer. An agent decides which tool to use next. I built the chain in this lesson on my own machine, with a fake model and no API key, because the wiring is the lesson. The model is the part you swap later.

Difficulty: Advanced · Know this first: Lesson 74, Lesson 63

What you will build

Four files, and you keep all four. First, chain.py, a chain that loads a runbook, retrieves the pages that fit, and answers with sources. Second, memory.py, the same chain with a memory that holds the last question. Third, agent.py, a ReAct agent with two tools: one reads the runbook, one checks the threshold. Fourth, eval.py, a 3-case eval set that catches one real bug. Everything runs on this machine. The model is fake. No call leaves the machine. No API key.

The story

Arjun is a fresher. He had a script that called a model. It pasted the question into a string, sent the string, and printed what came back. It worked once. Then a teammate asked it why checkout p95 was high, and it answered with a runbook page that did not exist. It cited RB-04. The runbook has three pages.

He did not buy a bigger model. He rebuilt the script as a chain. The prompt became a template, so the words were written once. The runbook became loaded pages, so the model read the real text. The retriever picked the pages, so the model could only use what was retrieved. The parser checked the shape, so a broken answer failed loudly instead of going out. The evals caught the one bug left: the chain recommended a rollback for a reading under the threshold.

He changed five lines. He added a guard. The evals went from 2 of 3 to 3 of 3. The model never changed. The wiring did. That is this lesson.

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one question, not a measurement from a live system, but the shape is honest. On the normal path, the model call is more than half of the trip, and retrieval is the next biggest slice. The template and the verdict are the cheapest hops. Step through the lesson or switch the scenario and watch the bar change. The slow path is the slowest, because it scans every chunk and the agent calls each tool twice. The broken path is almost as slow, because the model writes a longer answer to cover for the missing parser.

Tune the chain, and watch the time move

The budget bar above shows three fixed stories. This simulator lets you write your own. It answers one question: how healthy is your chain. You set the chunks retrieved, the answer length, the memory turns kept, and the tool calls the agent makes. Move a slider and watch the health score, the total time, and the verdict change. The numbers are a teaching example, built from the formulas below, not live measurements. The shape is the lesson: retrieval is the cheapest proof you can buy, and a long answer costs more time than a short one.

Prompt template words + holes Retriever RB-01, RB-03 Model one prompt, once Parser shape checked Memory 4 messages kept Agent decides Thought, Action, Observation, repeat read_runbook check_threshold One question in. One answer out. Every claim with a tool behind it.
5 chunks

Pages fetched at hop 4, 400 tokens each. Zero means the answer comes from the question alone, with no runbook behind it. Ten is a long context and a slow read.

150 words

Words in the written answer. Fifty is terse. One hundred fifty is one paragraph. Three hundred is hedging, and the time shows it.

3 turns

Turns held in the message list at hop 7. Zero means every question is the first question. Three is a short thread. Ten is a long history, and the rewrite reads all of it.

2 calls

Tool calls at hop 8. Zero means the agent answers from the pages alone, with no threshold check. Two is the lab: read the runbook, check the threshold. Four is careful, and slow.

Try this: (1) Set the chunks to 0. The retrieve hop falls to 60 ms, and the health score falls from 80 to 50. The chain answers from the question alone, with no runbook behind it. (2) Push the answer to 300 words. The model hop goes from 950 ms to 1,599 ms, and the trip takes 2,171 ms. Same facts, 649 ms slower. The extra words bought nothing. (3) Set the chunks to 10 and the turns to 10. The score hits 100 of 100, and the trip takes 1,742 ms. Full marks on paper. Now ask whether ten chunks measure anything new, or the same two pages written five ways.

Formulas, in plain terms. Retrieve ms = 60 + chunks x 30. Model ms = 300 + words x 4.33. Memory ms = 15 + turns x 10. Agent ms = 40 + tools x 40. Total time = template 12 + load 85 + split 40 + retrieve + model + parse 60 + memory + agent. Chain health score = chunks x 6 (capped at 50) + turns x 10 (capped at 30) + 10 when the answer stays under 200 words + 10 when the agent calls at least one tool. Model cost is $0.00. The demo runs on this machine with a fake model, so the bill is the lesson: the wiring is free, the waiting is not. Moving a slider updates the hop timings, the budget bar, and the step panel above.

Lab: build the chain and the agent

Do this now, in your own terminal. You will write five files, run the chain, run the memory, run the agent, watch the evals fail once, and end with a grounding check that passes. The model is fake. No call leaves the machine. No API key. The shape is what matters, and the shape is production shape.

Step 1: make the folder and install LangChain

LangChain ships as several packages. Install the four this lab needs. The install takes about a minute.

mkdir langchain-lab && cd langchain-lab
python3 -m venv .venv
source .venv/bin/activate
pip install langchain langchain-core langchain-community langchain-text-splitters

Check the install:

python -c "import langchain; print(langchain.__version__)"

Expected output, exactly as it ran in this build:

1.5.2

Two notes. First, pip prints a deprecation warning for langchain-community. It still installs. The loader in this lab comes from that package. Second, the venv stays in the lab folder. Delete the folder and the install goes with it.

Step 2: write the runbook

Save this as runbook.txt. Three pages. Each page has an id, a title, and the action it tells you to take.

RB-01: Checkout latency is high.
Check the checkout queue depth first. If p95 stays above 800 ms for 5 minutes, roll back the last deploy.

RB-02: Checkout error rate is up.
Check the payment dependency first, then read the checkout service logs for 5xx errors.

RB-03: A deploy just finished.
Compare the deploy diff with the previous version before any rollback. Never roll back blind.

Step 3: write chain.py and run it

Save this as chain.py. It loads the runbook, splits it, stores it, retrieves the pages that fit, builds the prompt, calls the fake model, and parses the answer into three fields. Read the KeywordEmbeddings class before you run it. It counts shared words. That is a stand-in for a real embedding model, and the file says so at the top. A real embedding model reads meaning, not words. Swap it in and nothing else in this file changes.

"""The chain from Lesson 170: load, split, retrieve, prompt, call, parse.

Runs with no model call and no API key. The model is FakeListLLM, so the
answer below is a canned string. The wiring is real. Swap FakeListLLM for
a real chat model and nothing else in this file changes.

The embeddings are a teaching stand-in. A real embedding model turns
meaning into numbers. This one counts shared words, so the demo runs with
no key and no download. Swap it for a real embeddings class and nothing
else in this file changes either.
"""

import re

from langchain_community.document_loaders import TextLoader
from langchain_core.embeddings import Embeddings
from langchain_core.language_models.fake import FakeListLLM
from langchain_core.output_parsers import PydanticOutputParser
from langchain_core.prompts import PromptTemplate
from langchain_core.runnables import (
    RunnableBranch,
    RunnableLambda,
    RunnablePassthrough,
)
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_text_splitters import RecursiveCharacterTextSplitter
from pydantic import BaseModel, Field

WORDS = [
    "checkout", "latency", "p95", "error", "deploy",
    "rollback", "queue", "payment", "logs", "diff",
]


class KeywordEmbeddings(Embeddings):
    """Count shared words. A stand-in for a real embedding model."""

    def _vec(self, text):
        low = text.lower()
        return [float(len(re.findall(r"\b" + w + r"\b", low))) for w in WORDS]

    def embed_documents(self, texts):
        return [self._vec(t) for t in texts]

    def embed_query(self, text):
        return self._vec(text)


class TriageAnswer(BaseModel):
    answer: str = Field(description="One paragraph answer, plain words")
    first_step: str = Field(description="The single first step to take")
    sources: list[str] = Field(
        description="Runbook page ids used, like RB-01"
    )


PARSER = PydanticOutputParser(pydantic_object=TriageAnswer)

PROMPT = PromptTemplate.from_template(
    "You are an on-call assistant. Answer only from the runbook pages "
    "below. If the pages do not cover the question, say so plainly. "
    "Do not guess.\n\n"
    "Runbook pages:\n{context}\n\n"
    "Question: {question}\n\n"
    "{format_instructions}"
)


def load_retriever():
    docs = TextLoader("runbook.txt").load()
    chunks = RecursiveCharacterTextSplitter(
        chunk_size=200, chunk_overlap=20
    ).split_documents(docs)
    store = InMemoryVectorStore.from_documents(chunks, KeywordEmbeddings())
    return store.as_retriever(search_kwargs={"k": 2})


def format_docs(docs):
    return "\n\n".join(d.page_content for d in docs)


def _num(text):
    return int(text.replace(",", ""))


def under_threshold(question):
    p95 = re.search(r"p95 is ([\d,]+) ms", question)
    thr = re.search(r"([\d,]+) ms threshold", question)
    if p95 and thr:
        return _num(p95.group(1)) < _num(thr.group(1))
    m = re.search(r"p95 ([\d,]+) ms, threshold ([\d,]+) ms", question)
    return bool(m) and _num(m.group(1)) < _num(m.group(2))


def build_chain(responses, guard=False):
    """Build the chain. guard=True adds the threshold short-circuit."""
    retriever = load_retriever()
    llm = FakeListLLM(responses=responses)
    plain = (
        {
            "context": retriever | RunnableLambda(format_docs),
            "question": RunnablePassthrough(),
        }
        | PROMPT.partial(format_instructions=PARSER.get_format_instructions())
        | llm
        | PARSER
    )
    if not guard:
        return plain, retriever
    guarded = RunnableBranch(
        (
            under_threshold,
            RunnableLambda(
                lambda _: TriageAnswer(
                    answer="p95 is under the threshold. "
                    "No action. Log it and move on.",
                    first_step="No action. Log the reading.",
                    sources=[],
                )
            ),
        ),
        plain,
    )
    return guarded, retriever


QUESTION = (
    "Checkout p95 is 3,100 ms against an 800 ms threshold, "
    "and a deploy finished two minutes ago. What do I check first?"
)

RESPONSE = (
    '{"answer": "Check the checkout queue depth first. If p95 stays '
    'above 800 ms for 5 minutes, roll back the last deploy, after '
    'comparing the deploy diff with the previous version.", '
    '"first_step": "Check the checkout queue depth first.", '
    '"sources": ["RB-01", "RB-03"]}'
)


if __name__ == "__main__":
    chain, retriever = build_chain([RESPONSE])
    print("RETRIEVED:")
    for doc in retriever.invoke(QUESTION):
        print(" -", doc.page_content.splitlines()[0])
    print()
    print("PROMPT SENT TO THE MODEL:")
    context = format_docs(retriever.invoke(QUESTION))
    print(
        PROMPT.partial(
            format_instructions=PARSER.get_format_instructions()
        ).format(context=context, question=QUESTION)
    )
    print("PARSED ANSWER:")
    print(chain.invoke(QUESTION))
python3 chain.py

Expected output, exactly as it ran in this build. The prompt is shown trimmed: the parser appends its JSON schema instructions after the question, about twenty lines. The words below are verbatim.

RETRIEVED:
 - RB-01: Checkout latency is high.
 - RB-03: A deploy just finished.

PROMPT SENT TO THE MODEL:
You are an on-call assistant. Answer only from the runbook pages below. If the pages do not cover the question, say so plainly. Do not guess.

Runbook pages:
RB-01: Checkout latency is high.
Check the checkout queue depth first. If p95 stays above 800 ms for 5 minutes, roll back the last deploy.

RB-03: A deploy just finished.
Compare the deploy diff with the previous version before any rollback. Never roll back blind.

Question: Checkout p95 is 3,100 ms against an 800 ms threshold, and a deploy finished two minutes ago. What do I check first?
[the parser appends its JSON schema instructions here]

PARSED ANSWER:
answer='Check the checkout queue depth first. If p95 stays above 800 ms for 5 minutes, roll back the last deploy, after comparing the deploy diff with the previous version.' first_step='Check the checkout queue depth first.' sources=['RB-01', 'RB-03']

Read the last line first. The answer cites its sources. That line is the whole lesson. Everything else in this file exists to make that line true.

Step 4: write memory.py and run it

Save this as memory.py. Same chain, wrapped in RunnableWithMessageHistory. Two turns. The second question asks what the threshold was, and the answer comes from the history, not from the runbook.

"""Memory from Lesson 170: the chain remembers the last question.

Same chain as chain.py, wrapped in RunnableWithMessageHistory. The
history lives in memory on this machine. No model call, no API key.
The fake model returns canned strings. The wiring is real.
"""

import re

from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.language_models.fake import FakeListLLM
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.runnables import RunnableLambda, RunnablePassthrough
from langchain_core.runnables.history import RunnableWithMessageHistory

from chain import WORDS, format_docs, load_retriever

HISTORY = InMemoryChatMessageHistory()


def get_history(session_id):
    return HISTORY


PROMPT = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are an on-call assistant. Answer only from the "
            "runbook pages below. Keep answers to one sentence.\n\n"
            "{context}",
        ),
        MessagesPlaceholder("history"),
        ("human", "{question}"),
    ]
)

RESPONSES = [
    "Check the checkout queue depth first. Sources: RB-01, RB-03.",
    "The threshold in your last question was 800 ms. "
    "p95 was 3,100 ms, over the threshold.",
]

def _question(d):
    return d["question"]


def _contextualize(d):
    """A follow-up like "what was the threshold" has no runbook words
    in it, so the retriever gets nothing to match. Rewrite it with the
    last question attached. Real apps do this with a small LLM call.
    Here the history is the rewrite."""
    q = d["question"]
    words = set(re.findall(r"[a-z]+", q.lower()))
    if not (words & set(WORDS)) and d.get("history"):
        last_human = [
            m for m in d["history"] if type(m).__name__ == "HumanMessage"
        ]
        if last_human:
            return last_human[-1].content + " " + q
    return q


if __name__ == "__main__":
    retriever = load_retriever()
    chain = (
        {
            "context": RunnableLambda(_contextualize)
            | retriever
            | RunnableLambda(format_docs),
            "question": RunnableLambda(_question),
            "history": RunnableLambda(lambda d: d.get("history", [])),
        }
        | PROMPT
        | FakeListLLM(responses=RESPONSES)
        | StrOutputParser()
    )
    chatty = RunnableWithMessageHistory(
        chain,
        get_history,
        input_messages_key="question",
        history_messages_key="history",
    )
    print("TURN 1:")
    print(
        chatty.invoke(
            {
                "question": "Checkout p95 is 3,100 ms against an "
                "800 ms threshold. What do I check first?"
            },
            config={"configurable": {"session_id": "oncall-1"}},
        )
    )
    print()
    print("TURN 2:")
    print(
        chatty.invoke(
            {"question": "What was the threshold in my last question?"},
            config={"configurable": {"session_id": "oncall-1"}},
        )
    )
    print()
    print("HISTORY HELD IN MEMORY:")
    for m in HISTORY.messages:
        print(" -", type(m).__name__, ":", m.content[:80])
python3 memory.py

Expected output, exactly as it ran in this build. Two deprecation warnings print above the output. They do not stop the run. They say RunnableWithMessageHistory and InMemoryChatMessageHistory are deprecated in favour of LangGraph persistence. Lesson 171 covers that. The output below starts after the warnings.

TURN 1:
Check the checkout queue depth first. Sources: RB-01, RB-03.

TURN 2:
The threshold in your last question was 800 ms. p95 was 3,100 ms, over the threshold.

HISTORY HELD IN MEMORY:
 - HumanMessage : Checkout p95 is 3,100 ms against an 800 ms threshold. What do I check first?
 - AIMessage : Check the checkout queue depth first. Sources: RB-01, RB-03.
 - HumanMessage : What was the threshold in my last question?
 - AIMessage : The threshold in your last question was 800 ms. p95 was 3,100 ms, over the threshold.

Four messages after two turns. Two human, two AI. That list is the memory. Nothing else. Now read the _contextualize function in the file. The follow-up has no runbook words in it, so the retriever gets nothing to match. Without the rewrite, the query vector is all zeros and the similarity search raises a ValueError. I found that by running the file, not by reading the docs. The rewrite attaches the last question before the retriever sees it. That is the fix, and it has a real name: contextualization.

Step 5: write agent.py and run it

Save this as agent.py. The agent has two tools. read_runbook searches the pages. check_threshold compares p95 with the threshold. The model is fake, so every Thought below is a canned string. The tool calls are real. The runbook read and the threshold check really run.

"""The agent from Lesson 170: a ReAct agent with two tools.

The agent decides what to do. It reads the runbook, checks the
threshold, then answers. The model is FakeListLLM, so every Thought
below is a canned string. The tool calls are real. The runbook read
and the threshold check really run. No model call, no API key.
"""

import re

from langchain_classic.agents import AgentExecutor, create_react_agent
from langchain_core.language_models.fake import FakeListLLM
from langchain_core.prompts import PromptTemplate
from langchain_core.tools import tool


@tool
def read_runbook(query: str) -> str:
    """Search the runbook for pages matching the query.
    Input is a few keywords, for example "checkout latency deploy"."""
    words = set(query.lower().split())
    hits = []
    current = []
    for line in open("runbook.txt"):
        line = line.rstrip("\n")
        if re.match(r"RB-\d\d:", line) and current:
            hits.append("\n".join(current))
            current = [line]
        elif line.strip():
            current.append(line)
    if current:
        hits.append("\n".join(current))
    found = [h for h in hits if words & set(h.lower().split())]
    return "\n\n".join(found) if found else "NO MATCH"


@tool
def check_threshold(reading: str) -> str:
    """Check a p95 reading against its threshold.
    Input is "p95, threshold", for example "3100, 800".
    Returns OVER or UNDER with the numbers."""
    p95, threshold = [float(x) for x in reading.split(",")]
    verdict = "OVER" if p95 >= threshold else "UNDER"
    return f"p95 {p95:g} ms against threshold {threshold:g} ms: {verdict}"


REACT_TEMPLATE = """Answer the following question using the tools below. You may use a tool more than once.

Tools:
{tools}

Use this format:
Question: the question
Thought: what to do next
Action: the tool name, one of [{tool_names}]
Action Input: the input to the tool
Observation: the tool result
... (repeat Thought/Action/Action Input/Observation as needed)
Thought: I have what I need
Final Answer: the answer

Question: {input}
Thought: {agent_scratchpad}"""

RESPONSES = [
    "Thought: I need the runbook pages for latency and the "
    "deploy.\nAction: read_runbook\n"
    "Action Input: latency deploy",
    "Thought: The pages say to check queue depth and compare the "
    "deploy diff. I need to know if p95 is over the threshold before "
    "I recommend a rollback.\nAction: check_threshold\n"
    "Action Input: 3100, 800",
    "Thought: p95 is OVER the threshold and the runbook pages back "
    "the steps. I can answer now.\nFinal Answer: Check the checkout "
    "queue depth first. If p95 stays above 800 ms for 5 minutes, "
    "roll back the last deploy, after comparing the deploy diff. "
    "Sources: RB-01, RB-03.",
]

QUESTION = (
    "Checkout p95 is 3,100 ms against an 800 ms threshold, and a "
    "deploy finished two minutes ago. What do I check first?"
)


def build_agent():
    tools = [read_runbook, check_threshold]
    prompt = PromptTemplate.from_template(REACT_TEMPLATE)
    agent = create_react_agent(FakeListLLM(responses=RESPONSES), tools, prompt)
    return AgentExecutor(
        agent=agent, tools=tools, verbose=True, max_iterations=5
    )


if __name__ == "__main__":
    result = build_agent().invoke({"input": QUESTION})
    print("FINAL ANSWER:")
    print(result["output"])
python3 agent.py

Expected output, exactly as it ran in this build. Line breaks are added where the terminal changes colour. The words are exactly as printed.

> Entering new AgentExecutor chain...
Thought: I need the runbook pages for latency and the deploy.
Action: read_runbook
Action Input: latency deploy
RB-01: Checkout latency is high.
Check the checkout queue depth first. If p95 stays above 800 ms for 5 minutes, roll back the last deploy.

RB-03: A deploy just finished.
Compare the deploy diff with the previous version before any rollback. Never roll back blind.
Thought: The pages say to check queue depth and compare the deploy diff. I need to know if p95 is over the threshold before I recommend a rollback.
Action: check_threshold
Action Input: 3100, 800
p95 3100 ms against threshold 800 ms: OVER
Thought: p95 is OVER the threshold and the runbook pages back the steps. I can answer now.
Final Answer: Check the checkout queue depth first. If p95 stays above 800 ms for 5 minutes, roll back the last deploy, after comparing the deploy diff. Sources: RB-01, RB-03.

> Finished chain.
FINAL ANSWER:
Check the checkout queue depth first. If p95 stays above 800 ms for 5 minutes, roll back the last deploy, after comparing the deploy diff. Sources: RB-01, RB-03.

Count the tool calls. Two. read_runbook, then check_threshold. The Final Answer comes after both. An agent that answers before the tools run is a chain with good formatting. This one earns its answer.

Step 6: write eval.py and watch one case fail

Save this as eval.py. Three cases. The first is the happy path. The second checks a different runbook page. The third is the trap: the keywords match, but p95 is under the threshold, so the right answer is no action at all.

"""Evals from Lesson 170: three cases, one real bug.

Case three is the trap. The keywords match and the runbook page is
right, but p95 is under the threshold. Version one of the chain has
no threshold guard, so it recommends a rollback for an alert that
needs no action. The evals catch it before any reviewer does.
Run: python3 eval.py
"""

from chain import RESPONSE, build_chain

CASES = [
    {
        "name": "latency spike after deploy",
        "question": (
            "Checkout p95 is 3,100 ms against an 800 ms threshold, "
            "and a deploy finished two minutes ago. "
            "What do I check first?"
        ),
        "response": RESPONSE,
        "want_sources": ["RB-01", "RB-03"],
    },
    {
        "name": "error rate with no deploy",
        "question": "Checkout error rate is up. Where do I look first?",
        "response": (
            '{"answer": "Check the payment dependency first, then '
            'read the checkout service logs for 5xx errors.", '
            '"first_step": "Check the payment dependency first.", '
            '"sources": ["RB-02"]}'
        ),
        "want_sources": ["RB-02"],
    },
    {
        "name": "latency under threshold",
        "question": (
            "Checkout p95 is 700 ms against an 800 ms threshold. "
            "What do I do?"
        ),
        "response": (
            '{"answer": "Check the checkout queue depth first. If '
            'p95 stays above 800 ms for 5 minutes, roll back the '
            'last deploy.", "first_step": "Check the checkout '
            'queue depth first.", "sources": ["RB-01"]}'
        ),
        "want_sources": [],
    },
]


def run_eval(guard):
    chain, retriever = build_chain(
        [c["response"] for c in CASES], guard=guard
    )
    passed = 0
    for case in CASES:
        result = chain.invoke(case["question"])
        retrieved = [
            d.page_content.splitlines()[0][:5]
            for d in retriever.invoke(case["question"])
        ]
        grounded = all(s in retrieved for s in result.sources)
        if case["name"] == "latency under threshold":
            ok = grounded and "roll back" not in result.answer.lower()
            detail = (
                "expected no rollback | got "
                + repr(result.answer[:60])
                if not ok
                else "expected no rollback | got no rollback"
            )
        else:
            ok = grounded and result.sources == case["want_sources"]
            detail = (
                f"expected {case['want_sources']} | "
                f"got {result.sources}"
            )
        print(("PASS" if ok else "FAIL"), "-", case["name"], "|", detail)
        passed += ok
    print(f"{passed}/{len(CASES)} eval cases passed")
    return passed


if __name__ == "__main__":
    print("CHAIN V1: no threshold guard")
    run_eval(guard=False)
    print()
    print("CHAIN V2: threshold guard added (chain.py, build_chain)")
    run_eval(guard=True)
python3 eval.py

Expected output, exactly as it ran in this build:

CHAIN V1: no threshold guard
PASS - latency spike after deploy | expected ['RB-01', 'RB-03'] | got ['RB-01', 'RB-03']
PASS - error rate with no deploy | expected ['RB-02'] | got ['RB-02']
FAIL - latency under threshold | expected no rollback | got 'Check the checkout queue depth first. If p95 stays above 800'
2/3 eval cases passed

CHAIN V2: threshold guard added (chain.py, build_chain)
PASS - latency spike after deploy | expected ['RB-01', 'RB-03'] | got ['RB-01', 'RB-03']
PASS - error rate with no deploy | expected ['RB-02'] | got ['RB-02']
PASS - latency under threshold | expected no rollback | got no rollback
3/3 eval cases passed

The failure is honest, and it is the most useful line in this lab. The chain matched the keywords and recommended a rollback for a reading under the threshold. A keyword match cannot tell a real incident from a sentence that happens to share the words. The eval set caught it before any reviewer did. The guard is five lines in chain.py: under_threshold plus the RunnableBranch. It runs before the model, and it cannot be argued with. That is what evals are for.

Check yourself

Four outcomes, all checkable in your terminal. python3 chain.py prints sources=['RB-01', 'RB-03']. python3 memory.py prints four messages under HISTORY HELD IN MEMORY. python3 agent.py prints FINAL ANSWER with sources RB-01 and RB-03, after two tool calls. python3 eval.py prints 3/3 eval cases passed. If any of the four is missing, the chain is not ready to show.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

LangChain is not one thing. It is four levels of the same job, and every chain I have seen sits on one of them. The levels are not about how many components you use. They are about how much of the answer you can check without trusting the model. Read them in order. Each level fixes the failure that hurt the most at the level below it.

Basic: One prompt, one model call, fake model, runs on your machine

At the basic level, you have a prompt template, a fake model, and a parser. No retriever. No memory. The chain takes a question and returns an answer in a shape the code can use. It runs with no key and no network call. It works when you watch it, and it breaks the moment the question needs a fact the model was never given.

Uses: A first chain. A prompt you want to test without paying for a model call.

Failure modes: The model answers from memory, and the memory is wrong. The answer has no source, so nobody can check it. The prompt drifts, because it lives in five places.

Do this: Write one prompt template with two holes: the question and the format. Call it with FakeListLLM. Print the filled prompt before you print the answer. If you cannot read the prompt, the chain is too clever already.

Intermediate: Retriever, parser, evals with a pass rate

At the intermediate level, the chain retrieves before it answers, and the parser checks the shape. The eval set has ten cases or more, each with an expected answer, and the pass rate is written down. The reviewer runs the evals herself. She does not have to trust your demo. She can read the failures.

Uses: A chain you send with an application, or a chain a teammate runs without you in the room.

Failure modes: The retriever returns the wrong pages, and the answer follows them. The eval set measures the easy cases only. The pass rate is 100 percent, and the reviewer trusts it less, not more.

Do this: Build the chain from the lab. Add the threshold guard. Write ten eval cases before you add the next feature. Include one case the chain gets wrong, and write down why. That case is the most honest line in the repo.

Advanced: This lesson: memory, agents, guards, grounded answers

At the advanced level, the chain remembers the thread, the agent picks its own tools, and a guard stops the bad action before the model runs. The answers are grounded: every source in the answer was retrieved first. This is the lesson you just walked. The chain is small. The checks around it are complete.

Uses: The chain and the agent from this lesson. A triage assistant a team can run on call, with evals that prove it works.

Failure modes: The agent calls a tool and ignores the result. The history grows and the rewrite costs more than the answer. The guard is written for one threshold and nobody updates it when the threshold changes.

Do this: Build the agent from the lab. Add the threshold guard. Run the evals until they print 3 of 3. Then ask the chain the follow-up question from the lab, and read the history list out loud. Four messages. If the count is wrong, the memory is wrong.

Master: The chain as a system, and the second one reuses the harness

At the master level, the chain owns budgets in time and in tokens, and the budgets are in version control. The eval set grows with every bug. The traces go to LangSmith, so a bad answer can be replayed hop by hop. The second chain reuses the same harness: same eval runner, same grounding check, same guard shape. At 10x the traffic, the retriever gets a real index and a real embedding model. At 100x, the eval set runs in CI on every push, and a failing eval blocks the merge.

Uses: A set of chains that share one harness, with evals in CI and a cost line on every README.

Failure modes: The eval set grows and nobody prunes it. The harness becomes the project, and the project becomes a demo of the harness. The second chain copies the first one's guard without thinking, and the guard is wrong for the new threshold.

Do this: Put the eval set in CI so a failing case blocks the merge. Then start the second chain with the same harness, and write down the one guard that changes between the two.

Go deeper

Read the docs that matter

Start at the LangChain docs home: https://python.langchain.com/docs/. Read the LCEL page first. LCEL is the pipe operator you used in this lab: prompt, then model, then parser, joined with |. The old chain classes, LLMChain and SequentialChain, are deprecated. Do not learn them. The lab in this lesson uses LCEL throughout, so the docs will look familiar. Then read the retrievers page and the agents page. The retriever page shows the vector stores. The agents page shows the tool calling that Lesson 171 builds on.

Swap the fake model for a real one

One import changes, and one line changes. Install the partner package, put the key in the environment, never in the source. I installed the package in this build and checked the import. It loads. It needs a key to run, and I did not run it, because there is no key in this machine.

pip install langchain-openai
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(model="gpt-4o-mini")

Then pass that llm into build_chain in place of FakeListLLM. The prompt, the retriever, and the parser do not change. The answers stop being canned. The evals stop checking exact words and start checking shape and sources. That is the only change that matters. A real model writes fresh text each time. A fake model repeats the script. The chain is the same either way.

The guard is a pattern, not a trick

RunnableBranch is a router with conditions. Each condition is a function that takes the input and returns true or false. The first true condition wins. The rest never run. The threshold guard in chain.py is one condition and one branch. The condition reads the numbers out of the question. The branch returns a fixed answer with no sources. The plain chain is the last branch, the one that runs when no condition matches. The pattern scales. A second condition can route billing questions to a different chain. A third can route unknown questions to a human. The router is the same shape at every size. The conditions are the only part that grows.

Price it yourself: the grounding check

Write the script below as chain_check.py and run it with python3 chain_check.py. It runs the chain once and checks that every source the answer cites was actually retrieved. A parser checks shape. This checks truth. No model is called.

"""Check from Lesson 170: every source the answer cites was retrieved.

Run: python3 chain_check.py
Prints GROUNDED when every cited page id was in the retrieved set,
and fails loudly otherwise. A parser checks shape. This checks truth.
"""

from chain import QUESTION, RESPONSE, build_chain

if __name__ == "__main__":
    chain, retriever = build_chain([RESPONSE])
    result = chain.invoke(QUESTION)
    retrieved = [
        d.page_content.splitlines()[0][:5]
        for d in retriever.invoke(QUESTION)
    ]
    bad = [s for s in result.sources if s not in retrieved]
    if bad:
        print(f"NOT GROUNDED: cited {bad}, retrieved {retrieved}")
        raise SystemExit(1)
    print(
        f"GROUNDED: cited {result.sources}, retrieved {retrieved}"
    )

Expected output, exactly as it ran in this build:

GROUNDED: cited ['RB-01', 'RB-03'], retrieved ['RB-01', 'RB-03']

Now change the canned response in chain.py so it cites RB-04, and run the check again. It prints NOT GROUNDED and exits with code 1. That is the check doing its job. A citation the retriever never returned is a citation that lies, and the check is the only hop that says so.

Do it yourself: run it from a clean copy

Copy the lab folder to a new directory, the way a teammate would clone it. Then run the four commands. No edits, no fixes, no files from your laptop.

cp -r langchain-lab /tmp/langchain-clean
cd /tmp/langchain-clean
python3 chain.py
python3 eval.py
python3 chain_check.py

Expected output, exactly as it ran in this build:

PARSED ANSWER:
answer='Check the checkout queue depth first. If p95 stays above 800 ms for 5 minutes, roll back the last deploy, after comparing the deploy diff with the previous version.' first_step='Check the checkout queue depth first.' sources=['RB-01', 'RB-03']

CHAIN V2: threshold guard added (chain.py, build_chain)
PASS - latency spike after deploy | expected ['RB-01', 'RB-03'] | got ['RB-01', 'RB-03']
PASS - error rate with no deploy | expected ['RB-02'] | got ['RB-02']
PASS - latency under threshold | expected no rollback | got no rollback
3/3 eval cases passed
GROUNDED: cited ['RB-01', 'RB-03'], retrieved ['RB-01', 'RB-03']

If any of the three commands fails in the clean copy, the lab is not ready. Fix the repo, not the clean copy. The clean copy is the teammate. The teammate does not fix your repo for you.

One time it broke

This one stayed with me because the chain worked in rehearsal and died in the room. Twenty rehearsals, and not one of them rendered the prompt from a clean copy of the file.

What broke

It was a Thursday night. I was showing the chain to a friend on a video call. That afternoon I had renamed the template variable from context to pages, because pages sounded clearer. I changed the template. I did not change the chain. The chain still passed the retrieved text under the key context. The model never saw the runbook. The demo I had rehearsed twenty times had never run from the file as it stood that night.

Timeline

  • 21:02I share my screen and type the command. python3 chain.py.
  • 21:03KeyError: pages. The template wants a variable the chain never passed. I say I will fix the name. I change the template back to context on the call. It works, and my friend has learned nothing.
  • 21:05I read the error out loud. It says the template wants pages and the chain passed context. The error was right. The rename was half done.
  • 21:09I change the chain mapping to pages, the name I actually wanted, and the demo runs. The answer cites RB-01 and RB-03. The demo ends six minutes late.
  • Next dayI add one rule to my own checklist. Every chain gets a test that renders the prompt with dummy values before any model call. A rename that breaks the prompt breaks the test, not the demo.

What changed after

Three things. First, the template and the chain mapping live in the same file, so a rename touches both. Second, the prompt render test runs before the evals, not after them. A prompt that cannot render is a chain that cannot run, and the evals cannot catch it, because the evals never start. Third, I rehearse the failure, not just the demo. I delete a variable, run the chain, and read the error out loud. A chain that has never failed in rehearsal fails in the room. A chain that has failed in rehearsal gets fixed in the room.

If you remember nothing else

Design exercise

Add a third tool to the agent and a fourth case to the evals. The tool is read_metric. It reads a file called metrics.txt with one line: the current checkout p95. It takes no input and returns the number. Wire it into agent.py so the agent calls it before check_threshold, and add a fourth eval case: p95 900 ms against an 800 ms threshold, deploy finished, expected sources RB-01 and RB-03, expected first step "Check the checkout queue depth first." The checkable outcome is eval.py printing 4/4, and agent.py printing three tool calls in the trace, in order: read_runbook, read_metric, check_threshold.

Clarifying questions to ask

  • Who writes metrics.txt: a script that reads the real dashboard, or a human who pastes the number by hand, and how stale can it be before the answer is wrong?
  • What counts as a tool call in the trace: the Action line, or the Observation that comes back, and what happens when the tool returns nothing?
  • What is the one guard the fourth case needs: the threshold guard from the lab, or a new one that checks the metric file exists?
  • How long can the agent loop run: three tools, or three tools plus retries, and who stops it when a tool fails twice?
  • What happens when the metric file is missing: does the agent say so plainly, or does it answer from the pages alone?

Rough sizing, worked

Start with metrics.txt. One line, one number. Then the tool. Ten lines, following the read_runbook shape. Then the agent. One line to add the tool to the list, and one canned Thought in the fake responses so the agent calls it in order. Then the eval case. One dict in the CASES list, with the question, the canned response, and the expected sources. Run the evals. Four cases, four passes. Run the agent. Three tool calls in the trace. Score the work the same way as the lab: one point for the tool, one point for the eval case, one point for the trace order. Three points is done. Two points means the wiring is right and the order is wrong, and the order is the lesson.

Trade-offs to name

Three tools buy a checked answer and cost a slower loop, so I keep three and I do not add a fourth until an eval asks for one. A metric file buys a live number and costs a file to keep fresh, so I write the number by hand in the lab and I say so in the README. A fourth eval case buys proof and costs maintenance, so I keep four and I prune one every time I add one. A canned agent trace buys a repeatable demo and costs a demo that never surprises me, so I rehearse the trace and I also rehearse one tool failing, because the failure is the part the demo never shows.

Saved in this browser, so the Lessons index can show your progress across the course.

Next in this series
Lesson 171: LangGraph

LangChain chains run one way. LangGraph adds loops, branches, and state that survives a restart. The memory in this lesson lives in a list on one machine. Lesson 171 makes it durable, and it gives the agent a graph it can walk more than once.