Platform Engineering · Lesson 85

LLM Evaluation & Benchmarks

An eval is a test for answers. A language model writes answers, and an eval is how you find out whether those answers are any good. That is the whole idea. You give the model a fixed list of questions. You already know the right answers. You score what the model writes against them. The number you get is the eval score. Here is the concrete example I will use all the way through. A support site answers user questions with a language model. A user asks how to reset a password. The eval asks the same question to the model, reads the answer, and checks three things. Did it give the right steps. Did it invent anything. Did it stay short and clear. One question, one score. Two hundred questions, one trend line. I have watched a team ship a worse model because the score went up, and I have watched a good model get blocked because the test was broken. The score is never the truth. The score is a measurement, and a measurement is only as good as the test behind it. This lesson is about designing tests that mean something, and spotting the ones that do not.

Difficulty: Advanced · Know this first: Lesson 64, Lesson 61
Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one question in the eval run, not a measurement from a live system, but the shape is honest: on a clean run, the model call is 1,800 ms and the judge call is 900 ms, and together they are most of the 2,970 ms trip. The checks, the aggregation, and the gate are the rest, and they are cheap. Step through the lesson or switch the scenario and watch the bar change. The contaminated run is faster and the score is higher, which is exactly why it is dangerous. The regression run is slower and the score is lower, which is exactly why the gate exists.

Extra diagram — The eval rig — who grades the grader

A second picture of the same system. I start with one box and build to the full Master picture. The dot is one golden-set question.

10 stepsMoving: one golden-set questionBasic to Master
Golden set200 QsHarness2Candidate model3Exact checks4 CHECKSJudge model5Contamination check6Aggregate7Baseline storeLAST WEEKGate±0.05Branches:contamination trip2 TRIPSone golden-set question
Extra diagram

    Legend: orange dot = one golden-set question  ·  hexagon = a service or model  ·  shaded box = a store, log, or file  ·  dashed line = the wire between boxes. Progression: Starts from scratch with one question and one gold answer — no eval vocabulary assumed. Adds the harness, the checks, the judge and the baseline one at a time. Ends at the Master-level picture: eval-driven iteration, regression evals on every model update, and the judge itself being judged.  ·  Traffic: orange = request forward, green = response back.

    Change the eval and watch the score lie

    The budget above is one fixed trip. This box is yours. Push the leak up, shrink the golden set, turn the judge off, switch to the big model, and watch the score, the run time, and the cost move. Then ask the question I ask before every deploy: is this number a measurement, or is it a memory?

    How many questions are in the golden set. Fifty is a quick smoke test. A thousand is a serious eval. More questions means a longer run and a tighter margin of error, because a small sample wobbles more than a big one.

    What share of questions the LLM judge scores, and what share the exact checks score alone. Eighty percent is the usual split. Zero means no judge at all, which is cheap, but open answers cannot be scored by a script. One hundred means the judge scores everything, which is slow and expensive.

    What share of the golden set leaked into the training data. Zero is clean. Fifty means half the test was seen before it was taken. Leaked questions score near perfect, because the model remembers them, and the reported score climbs while the true score stays flat.

    The fast model answers in about 1,800 ms and costs $0.002 per 1,000 tokens. The big model answers in about 2,600 ms and costs $0.010 per 1,000 tokens. The big model usually writes a better answer. The fast model is cheaper and quicker. There is no free upgrade.

    Total run time9.3 min200 questions at 2,790 ms each
    Total eval cost$0.85200 questions, fast model, judge on 80 in 100
    Reported score0.82true score 0.82, no leak. The reported score is the true score when the set is clean
    Margin of error±0.07at 200 questions, a move smaller than 0.07 is noise, not signal

    Try this: (1) Push the contamination slider to 50% at the defaults. The reported score climbs from 0.82 to 0.91, and the run time and cost do not move at all. Nothing improved. The test leaked. (2) Now set contamination back to 0% and push the golden set to 1,000 questions. The run time grows from 9.3 min to 46.5 min and the cost grows from $0.85 to $4.24, but the margin of error tightens from 0.07 to 0.03. (3) Now set the judge share to 0%. The run time falls to 6.9 min and the cost falls to $0.56, but the eval can only score what a script can check, and a support answer is not a script check.

    Check yourself

    Three questions. No score is saved anywhere — this is just you against the material.

    Basic to master

    The same topic four times, from the first plain idea to the level where I run it in production and debug it at 2 AM. Each level adds one thing the level below cannot do.

    Basic — What it is

    Definition: An eval is a test for answers. A language model writes answers, and an eval is how you find out whether those answers are any good. That is the whole idea. You give the model a fixed list of questions. You already know the right answers. You score what the model writes against them. The number you get is the eval score.

    Types:

    • The golden set: fixed questions with known good answers.
    • Exact checks: link present, length cap, required fact.
    • The judge: a model scoring answers against a rubric.
    • The gate: a score that blocks a deploy.

    Uses: Use it before you trust any model with real users. The concrete example from this lesson is a support site asking "how do I reset a password". The eval asks the same question to the model, reads the answer, and checks three things. Did it give the right steps. Did it invent anything. Did it stay short and clear. One question, one score. Two hundred questions, one trend line.

    Dimensions: At low scale a 200-question golden set runs in minutes. At 10x the traffic the eval run gets slow, because every question costs two model calls. What breaks is trusting the score: I have watched a team ship a worse model because the score went up, and a good model get blocked because the test was broken. It costs model calls for every question, judge calls on top, and human calibration time. You give up trusting a single number for the work of designing a test that means something.

    Intermediate — Measuring your task

    Definition: The most important decision in this whole lesson is what to measure. A benchmark measures what its authors cared about. Your eval must measure what your users care about. I start from the user, not from the benchmark. I pull the last month of real questions, sort them by how often they come up, and build the golden set from the top of that list.

    Types:

    • Public benchmarks: a smoke test, nothing more. They tell me a model is broken. They never tell me a model is good enough for my users.
    • Your golden set: built from real user questions, sorted by frequency.
    • Scoring rules in plain words: what a right answer must contain, what it must not contain, how long it may be.
    • Held-out fresh questions: 50 the model has never seen, the leak detector.

    Uses: Use it before writing any eval code. A good scoring rule says what a right answer must contain. A bad scoring rule says "good answer" and leaves the judge to guess. The rule is the test. If the rule is vague, the score is vague, and no amount of judge tuning will fix it.

    Dimensions: At 10x the traffic, shrink the CI golden set to the fifty questions that matter most and run the full set every night. What breaks is a benchmark's questions: they are old, because the benchmark was written years ago and the world moved on; narrow, because a test that tries to cover everything covers nothing well; and leaked, because every model trainer in the world has seen the public list. It costs building the golden set from real questions, which takes a human's time, and keeping it fresh with a monthly refresh. You give up the comfort of a public scoreboard for a private test that measures your users' actual questions.

    Advanced — Judging the judge

    Definition: A judge has four biases I check for, every time. First, verbosity bias: the judge prefers long answers, even when the extra words add nothing. Second, self-preference: the judge prefers answers written in its own style, which means it grades its own model family higher. Third, position bias: when the judge compares two answers, it prefers the first one it reads, so I swap the order and run it again. Fourth, name bias: when the judge knows which model wrote the answer, it scores the famous one higher, so I strip the names.

    Types:

    • Blind judging: strip the names before scoring.
    • Order swapping: run the comparison twice, swapped.
    • Length caps: stop rewarding long answers.
    • Weekly calibration against humans on a small sample.

    Uses: Use it every week, not once. A judge I have not calibrated is a guess with a decimal point on it. The config in this lesson sends 80 in 100 questions to the judge, blind to names and with the order swapped, and fails the run if the gap against fresh questions is more than 0.10.

    Dimensions: At 100x the traffic the judge becomes the bottleneck, so I score most questions with exact checks and send only the hard ones to the judge. What breaks is contamination: a model that answers golden-set questions much faster than fresh ones is reciting, not reasoning. The paraphrase test catches it: rewrite the golden set questions in different words and run them again. A model that truly understands the task scores the same. A model that memorised the exact wording falls apart. It costs judge calls, human calibration, and the fresh held-out set. You give up judging everything with the model for a mix: exact checks where they work, the judge where only judgment works, humans where only humans work.

    The same eval on the same model gives a different score each run, because sampling is random and the judge is a model. A gain of 0.02 between two runs can be noise. I run each eval five times and I report the mean with a confidence interval, not a single number.

    The interval is the verdict. If the intervals overlap, the models are tied, and I say so. Shipping a model because one lucky run scored higher is a decision made on noise.

    Master — Eval-driven iteration

    Definition: Eval-driven iteration means the eval decides what to fix next, not the loudest person in the room. The loop is short. Run the eval. Read the failures by category. Fix the worst category. Run the eval again. I keep a list of the last ten failures in the open, with the fix each one got, because the list is the memory of the team.

    Types:

    • Failure categories: billing, length, judge disagreement, each with its own fix.
    • Regression evals: the new version must score within 0.05 of the old version on the same golden set, with the same judge, or it does not roll out.
    • Nightly evals: because the model can change behind the same name.
    • The golden set as a service: versioned, with its own on-call.

    Uses: Use it when the team argues about what to fix. If the billing category fails, the prompt needs the billing pages, not a bigger model. If the answers are too long, the prompt needs a length rule, not a new judge. If the judge disagrees with humans, the rubric needs work, not the model.

    Dimensions: At 100x the traffic the golden set is a service of its own, because every model change in the company is gated on it. What breaks is a team that skipped the regression eval and paid three days of bad answers for a model update the provider called minor. A minor update is a deploy I did not ask for. It costs the harness, the judge, the nightly runs, and the calibration. You give up fixing what someone guessed for fixing what the eval shows. Slower in the meeting, faster in the month.

    An eval set that anyone can edit is an eval set nobody trusts. I freeze the release set, and changes go through review, with a reason written next to each case added or removed.

    The frozen set is versioned, and the release decision names the version. A model that passed version 12 did not pass version 13, and the difference is a matter of record. Governance sounds boring until a bad model ships on a gamed eval.

    Go deeper

    The steps above are one question through nine hops. Production is the rest of it. Here is what I actually spend my time on once the golden set is built and the gate is written down.

    Benchmarks and their limits

    A benchmark is a public test with a public scoreboard. The well known ones test different things. One tests general knowledge across many subjects. One tests writing code that runs. One tests math word problems. One tests reading comprehension. The names matter less than the shape. Every benchmark is a fixed list of questions, and every fixed list has the same three limits. First, the questions are old, because the benchmark was written years ago and the world moved on. Second, the questions are narrow, because a test that tries to cover everything covers nothing well. Third, the questions leak, because every model trainer in the world has seen the public list. I use benchmarks the way I use a smoke test. They tell me a model is broken. They never tell me a model is good enough for my users. My users do not ask benchmark questions. They ask how to reset a password at 2 AM.

    Eval design: measure YOUR task

    The most important decision in this whole lesson is what to measure. A benchmark measures what its authors cared about. Your eval must measure what your users care about. I start from the user, not from the benchmark. I pull the last month of real questions, sort them by how often they come up, and build the golden set from the top of that list. Then I write the scoring rule for each question in plain words before I write any code. A good scoring rule says what a right answer must contain, what it must not contain, and how long it may be. A bad scoring rule says good answer and leaves the judge to guess. The rule is the test. If the rule is vague, the score is vague, and no amount of judge tuning will fix it.

    LLM-as-judge pitfalls

    A judge has four biases I check for, every time. First, verbosity bias: the judge prefers long answers, even when the extra words add nothing. Second, self-preference: the judge prefers answers written in its own style, which means it grades its own model family higher. Third, position bias: when the judge compares two answers, it prefers the first one it reads, so I swap the order and run it again. Fourth, name bias: when the judge knows which model wrote the answer, it scores the famous one higher, so I strip the names. The fix for all four is the same. Calibrate against humans on a small sample every week, blind the judge to names, swap the order, and cap the length the judge rewards. A judge I have not calibrated is a guess with a decimal point on it.

    Contamination detection

    A leak is hard to spot because the score looks great. I use four checks. First, the fresh question test: hold out a set of questions the model has never seen, and compare its score on those with its score on the golden set. A big gap is the leak. Second, the overlap check: search the training data for the golden set questions, word for word and near word for word. Third, the speed check: a model that answers golden set questions much faster than fresh ones is reciting, not reasoning. Fourth, the paraphrase test: rewrite the golden set questions in different words and run them again. A model that truly understands the task scores the same. A model that memorised the exact wording falls apart. I run the paraphrase test before every big eval, because it is the cheapest leak detector I have.

    Eval-driven iteration

    Eval-driven iteration means the eval decides what to fix next, not the loudest person in the room. The loop is short. Run the eval. Read the failures by category. Fix the worst category. Run the eval again. The failures tell me which fix to make. If the billing category fails, the prompt needs the billing pages, not a bigger model. If the answers are too long, the prompt needs a length rule, not a new judge. If the judge disagrees with humans, the rubric needs work, not the model. I keep a list of the last ten failures in the open, with the fix each one got, because the list is the memory of the team. A team that fixes what the eval shows gets better every week. A team that fixes what someone guessed gets better by accident.

    Regression evals on model updates

    A regression eval is the eval from Lesson 64 run against a new model version before it ships. The rule is the same as any deploy gate. The new version must score within 0.05 of the old version on the same golden set, with the same judge, or it does not roll out. I run it on every model update, even the ones the provider calls minor, because a minor update is a deploy I did not ask for. I also run it on a schedule, every night, because the model can change behind the same name and only a nightly eval will catch it. The score is split by model version on the dashboard, so the drop shows up as a step, not a mystery. The incident in the next section is a team that skipped this step, and the price was three days of bad answers.

    What changes at 10x and 100x

    At 10x the traffic, the eval run is the first thing that gets slow, because every question costs two model calls. I shrink the golden set for CI to the fifty questions that matter most, and run the full set every night. At 100x, the judge becomes the bottleneck, so I score most questions with exact checks and send only the hard ones to the judge. The golden set becomes a service of its own, with its own versioning and its own on-call, because every model change in the company is gated on it. The budget stops being a target and becomes a contract, and every eval run owns its share of it, in minutes and in dollars.

    The eval, as config

    This is the config shape I would actually write, with plain names and no vendor details:

    eval:
      name: support-answers-eval
      golden_set:
        name: support-golden-v3
        size: 200
        held_out_fresh: 50
      candidate:
        model: fast
        version: pinned-2026-11
        prompt_version: v14
      scoring:
        exact_checks: [link_present, max_length_400, required_fact]
        judge:
          model: fast
          share: 0.80
          rubric: support-rubric-v2
          blind: true
          swap_order: true
      contamination:
        check_training_overlap: true
        paraphrase_test: true
        max_gap_vs_fresh: 0.10
      gate:
        min_score: 0.82
        max_drop_vs_baseline: 0.05
        on_fail: block_deploy

    Read it top to bottom and it is the whole lesson in config form. The golden set is version 3 with 200 questions, plus 50 fresh held-out questions the model has never seen. The candidate is the pinned fast model with prompt version 14. The scoring runs exact checks first, then sends 80 in 100 questions to the judge, blind to names and with the order swapped to kill position bias. The contamination checks look for training overlap, run the paraphrase test, and fail the run if the gap against fresh questions is more than 0.10. The gate blocks any deploy that scores under 0.82 or drops more than 0.05 against the baseline. The config is the eval. The rest is just code.

    Is a 0.02 gain real?

    Two models, five eval runs each. Model B scores 0.02 higher on average. The script computes the mean and the 95 percent confidence interval for each, and then it checks whether the intervals overlap.

    Run it with python3 is_the_gain_real.py. Then change one of B's runs from 0.84 to 0.90 and run it again. Watch the interval move, and watch the verdict.

    Reading the run command

    python3 is_the_gain_real.py has two parts and no flags. python3 is the program. It is the Python 3 interpreter, the thing that reads the file and runs it. is_the_gain_real.py is the file to run. It is a plain argument, not an option. Python reads it top to bottom and runs it. Leave the filename off and Python waits for you to type code by hand. That is the interactive prompt, not a run.

    Three options I reach for all the time. --version prints the Python version and stops. I run it first on a new machine, so I know what I am running against before I trust the output. -u forces unbuffered output. Prints show up the moment they happen instead of sitting in a buffer. That matters when I pipe output into a log file and watch it live. Without it a crashed run can look like it printed nothing. -m runs a library module as a script. python3 -m json.tool runs the JSON tool that ships with Python, with no file of my own. Try them now.

    python3 --version
    python3 -u is_the_gain_real.py
    python3 -m json.tool --help

    Run the script once plain and once with -u. The output is the same here. The script is short and finishes fast. The difference shows up on a long run. With -u each line appears as it is printed. Without it the lines arrive in a burst at the end. That is the check. If both runs print the same lines, the flag did its job.

    def is_the_gain_real():
        import math, statistics
        runs_a = [0.81, 0.79, 0.82, 0.80, 0.78]
        runs_b = [0.83, 0.81, 0.84, 0.82, 0.80]
        def ci(runs):
            mean = statistics.mean(runs)
            se = statistics.stdev(runs) / math.sqrt(len(runs))
            return mean, mean - 1.96 * se, mean + 1.96 * se
        for label, runs in (("A", runs_a), ("B", runs_b)):
            mean, lo, hi = ci(runs)
            print(f"model {label}: mean {mean:.3f}, 95% CI [{lo:.3f}, {hi:.3f}]")
        ma, loa, hia = ci(runs_a)
        mb, lob, hib = ci(runs_b)
        print("CIs overlap:", not (lob > hia or loa > hib), "-> a 0.02 gain is noise, not a win")
        assert not (lob > hia or loa > hib)
    
    
    is_the_gain_real()
    

    Model A: mean 0.800, 95 percent CI [0.786, 0.814]. Model B: mean 0.820, 95 percent CI [0.806, 0.834]. The intervals overlap, so the 0.02 gain is noise, not a win. Five runs is the minimum. One run is a coin flip with extra steps.

    One time it broke

    This one stayed with me because the score went up and the product got worse, and for three days nobody believed the users.

    What broke

    A team refreshed its training data from a folder that also held the golden set. Two hundred test questions and their gold answers went into training. The next eval scored 0.98, the best number the team had ever seen. The team shipped a new model version on the strength of it. In production the answers were worse, not better. The new model was smaller and cheaper, and it had been chosen because the leaked score said it was the best. Users reported wrong answers for three days. The dashboards for latency and errors were green, because nothing had failed. The answers were just wrong, and wrong does not page anyone unless you measure it.

    Timeline

    • Week 1, MondayThe training data refresh runs. A folder copy pulls the golden set into the training data. Nobody notices.
    • Week 2, MondayThe eval runs. The score jumps from 0.86 to 0.98. The team celebrates and moves the new model version to rollout.
    • Week 2, TuesdayThe new model version goes live. Support gets the first wrong answer report by noon. It is answered by hand and closed.
    • Week 2, WednesdayThree more reports land, same pattern. Short answers, missing steps. The team blames the prompt and edits it.
    • Week 2, ThursdayThe prompt edit changes nothing. I open a trace and see the model answers the golden set questions in half the usual time. That is the clue.
    • Week 2, Thursday, 15:00I split the score between the golden set and fresh questions. Golden set 0.98. Fresh questions 0.71. The model memorised the test.
    • Week 2, Thursday, 17:00I find the golden set file in the training data manifest. The leak is confirmed. The model is rolled back to the old version.
    • Week 2, FridayThe eval reruns on a rebuilt golden set with fresh questions. Score 0.84. The new model version is rejected properly this time.

    What changed after

    Three things. First, the golden set lives in its own folder with its own access rule, and no training job can read it. The folder is the fence. Second, every eval run now scores fifty fresh held-out questions alongside the golden set, and a gap bigger than 0.10 fails the run automatically. Third, the paraphrase test runs before every big eval: the golden set questions are rewritten in different words, and a model that memorised the exact wording falls apart on the rewrite. The lesson I took home is simple: a test the model has seen is not a test. Guard the test like you guard the production database, because it is the only thing standing between a good score and a bad deploy.

    If you remember nothing else

    Design exercise

    This is the kind of question I would ask in an interview, and the kind I ask myself before I sign off on a real eval system. Take a minute with it before you read my approach.

    The prompt

    Design the evaluation system for a support assistant that answers questions from 50,000 support documents. It must handle 500 questions a minute in production, and the team ships a prompt change every week and gets a new model version every month. You have the pipeline from Lesson 61, the eval harness from Lesson 64, a fast model at $0.002 per 1,000 tokens, and a big model at $0.010 per 1,000 tokens. Sketch the eval: the golden set, the judge, the gates, the schedule, and what you watch.

    Clarifying questions to ask first

    I would ask five things before I sized anything. What counts as a good answer, and who decides: the user vote, a human review, or a written rubric? How fresh must the golden set be: can it stay fixed for a quarter, or do user questions change faster than that? What is the failure we design for: a bad prompt version, a bad model version, or a leaked test? How fast must the gate be: does a prompt change wait an hour for the eval, or ten minutes? And what is the budget: how many dollars and minutes is one eval run allowed to cost?

    Rough sizing, worked

    Start with the simulator math from this lesson. A golden set of 200 questions at 2,790 ms each runs in 9.3 minutes and costs $0.85 on the fast model with an 80 percent judge share. That is cheap enough to run on every prompt change. So I keep two sets. The CI set is 50 questions, the ones users ask most, and it runs in about 2.3 minutes for $0.21. The full set is 200 questions plus 50 fresh held-out ones, and it runs every night for about $1.06. The judge scores 80 in 100 questions, and the other 20 are scored by exact checks alone. The gate is three rules: the CI score must not fall more than 0.05 below baseline, the nightly score must stay above 0.82, and the gap between the golden set and the fresh questions must stay under 0.10, or the run is void and the deploy is blocked. The big model never runs the eval on its own. It answers the same 50 CI questions once a week, and only if it beats the fast model by more than 0.05 does anyone talk about paying its price. Latency: the CI gate at 2.3 minutes fits inside a normal deploy pipeline, and the nightly run at 11.6 minutes fits inside the night. The margin of error on 50 questions is about 0.14, so the CI gate only blocks on big drops, and the 200 question nightly run with a margin of about 0.07 is the one that decides close calls.

    Trade-offs to name

    A bigger golden set buys confidence and costs time and money, so I keep the CI set small and the nightly set big. A judge on every question buys coverage and costs latency and dollars, so I send only the hard questions to the judge. A strict gate buys safety and costs shipping speed, so the CI gate blocks only on big drops and the nightly run catches the small ones. A fresh golden set buys honesty and costs human review time, so I refresh it monthly with real questions users actually asked, including the ones the pipeline got wrong. And the target is a split score, not a blended one. A blended 0.86 with a billing category at 0.58 fails this design.

    Saved in this browser, so the Lessons index can show your progress across the course.

    Next in this series
    Lesson 86 — Synthetic Data & Dataset Curation

    This lesson taught you how to test a model honestly: the golden set, the judge, the leak checks, and the gate. The next lesson opens up where the test data itself comes from: how to build good training and eval data when real examples are scarce, and how to curate a dataset without letting the test leak into it. That is synthetic data and dataset curation, and it is the next hop in this module.