Platform Engineering · Lesson 79

LLMOps

LLMOps is the job of running a language model in production, day after day, after the demo is over. A language model is a program that writes answers, and LLMOps is everything around it that keeps those answers right, fast, and affordable. It is the same job I do for any other service: version the change, test it before it ships, watch it while it runs, and roll it back when it breaks. The difference is that the model itself keeps changing under you, and a prompt is not a config file. A prompt is the instructions the model reads, and changing one word in it can change every answer the system gives. Here is the concrete example I will use all the way through. A support site answers user questions with a language model. A user asks how to reset a password. The pipeline from Lesson 61 finds the help pages, builds the prompt, and calls the model. LLMOps is what happens around that call. The prompt is pinned to version 14, the one that passed the tests, not whatever someone edited this morning. The answer is checked before the user sees it. Every token is counted and priced. A sample of answers is scored every day. When the model provider ships a new version, it goes through the same gate as a code deploy. That is the whole lesson: the model is the famous part, and the operations around it are the part that decides whether the system survives contact with real users.

Difficulty: Advanced · Know this first: Lesson 61, Lesson 65
Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one question, not a measurement from a live system, but the shape is honest: on a normal question, the model call is 1,800 ms of the 1,925 ms trip. The prompt pin, the guardrails, the trace, and the eval are the rest, and they are the whole point of this lesson. Step through the lesson or switch the scenario and watch the bar change. The regression adds almost a second without the question changing at all. The spike more than doubles the trip, and most of that is a timeout, not work.

Extra diagram — One question through the LLMOps control plane

A second picture of the same system. I start with one box and build to the full Master picture. The dot is the question.

11 stepsMoving: the questionBasic to Master
User1API2Prompt registryv14Input guardrails1STModel5Output guardrails2NDFallback ladder7Trace and tokenledger8Eval sampler1 IN 20Answer plus uservote, feeding10Branches: regressiontrip2 TRIPSthe question
Extra diagram

    Legend: orange dot = the question  ·  hexagon = a service or model  ·  shaded box = a store, log, or file  ·  dashed line = the wire between boxes. Progression: Starts from scratch with one question and one pinned prompt — no LLMOps vocabulary assumed. Adds the guardrails, the fallback, the trace and the eval sample one at a time, then the feedback loop back to the prompt registry. Ends at the Master-level picture: prompts as versioned code, evals in CI, fallback tiers, and model updates that arrive under you.  ·  Traffic: orange = request forward, green = response back.

    Change the traffic and watch the bill

    The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the latency, the model calls, and the monthly bill move. The numbers are a teaching example, not live measurements. The shape is the lesson: the cache and the model choice decide the bill, and the prompt size decides it a second time.

    How many questions arrive every minute. Ten is a quiet internal tool. Five hundred is a busy public support site. More questions means more model calls, unless the cache catches them first.

    How big the prompt is, in input tokens. A token is roughly a word piece. The prompt grows when someone adds more retrieved passages, more instructions, or more history to it. A bigger prompt costs more per call and makes the model slower, because the model has to read it all before it writes.

    What share of questions get answered from the cache instead of the model. Thirty percent is a decent start. Ninety percent means most questions are repeats, and the model rarely runs. A cached answer costs nothing and takes 90 ms.

    The fast model answers in about 1,800 ms and costs $0.002 per 1,000 tokens. The big model answers in about 2,600 ms and costs $0.010 per 1,000 tokens. The big model usually writes a better answer. The fast model is cheaper and quicker. There is no free upgrade.

    Average answer latency1,375 mscached answers at 90 ms, the rest at the full 1,925 ms
    Model calls per minute70100 questions a minute, 30 in 100 answered from cache
    Monthly model cost$8,46770 calls a minute, 1,400 tokens each, fast model. Budget: $10,000 a month
    Cost per 1,000 answers$1.96what 1,000 user questions cost after the cache takes its share

    The user feels the wait. The model is doing the damage, not the checks. The cache is the lever.

    Inside the $10,000 monthly budget. The cache and the fast model are earning their keep.

    Try this: (1) Push the cache hit rate to 90% at the defaults. The average latency falls from 1,375 ms to 274 ms and the monthly cost falls from $8,467 to $1,210, and nothing else changed. (2) Now switch to the big model. The average latency climbs to 1,935 ms and the monthly cost jumps to $42,336. Same questions, same answers wanted. Only the model changed. (3) Now set the prompt size to 4,000 tokens. The monthly cost jumps to $26,611 even on the fast model, because every extra token in the prompt is paid for on every single call.

    Check yourself

    Three questions. No score is saved anywhere — this is just you against the material.

    Basic to master

    The lesson above walks one question through nine hops, from the request to the answer on the screen. This section covers the same pipeline four times, from the first pinned prompt to the level where the model changes under me and I catch it the same day.

    Basic — One question, one trace

    Definition: LLMOps is the job of running the model after the demo is over. The model is a service someone else runs, and the weights are not mine to touch. The thing I version is the prompt. A prompt is a few hundred words of instructions, and changing one sentence in it changes every answer the system gives.

    Types:

    • The prompt version — version 14, loaded by name and version, never by latest.
    • The model version — pinned in config, recorded on every trace, so a change shows up on day one.
    • The guardrail — the check at the entrance and the check at the exit, standing before the model and before the user.
    • The trace — one record per question: prompt version, model version, tokens, latency, and cost.

    Uses: I use this level on every bad answer. I open the trace and read it top to bottom before I touch anything. If I cannot name the prompt version that served it, I cannot roll it back, and I cannot find what changed.

    Dimensions:

    • Load: At ten questions a minute the pipeline is quiet. At five hundred a minute the model calls queue up, and the cache decides whether the bill stays sane.
    • Breaks: It breaks when the prompt is loaded by latest. The answer a user gets depends on what time it is, and a 2 AM debug becomes a guessing game about which text the model actually read.
    • Cost: The cost is the tokens. The model charges by the token, and every extra token in the prompt is paid for on every single call.
    • Trade-off: I give up retraining to get a prompt I can rewrite and ship in an afternoon. I lose control of the weights. I gain a change I can roll back by flipping a flag.

    Intermediate — Prompts are the new weights

    Definition: A prompt registry is a versioned store for prompts, the same way a container registry is a versioned store for images. Each prompt has a name, a version number, the text, and the eval score it got. The pipeline loads a prompt by name and version. A change goes through review, through the eval gate, and out behind a flag to a small slice of traffic first.

    Types:

    • The prompt registry — prompts in version control, reviewed like code, shipped behind a flag.
    • The eval gate — the golden set of about fifty questions, scored before a change ships. The change ships only if the score holds.
    • The flag rollout — 5 percent first, then more, with the rollback a flag flip, not a redeploy.
    • The golden set — refreshed every month with real questions users actually asked, including the ones the pipeline got wrong.

    Uses: I use this level when I change a prompt. I run the evals, I watch the score on the small slice, and I flip the flag back if the score drops. I have rolled a prompt back at 2 AM, and the rollback took seconds because the old version was still there.

    Dimensions:

    • Load: At 10x the traffic the eval harness becomes the bottleneck. I score a smaller sample, run the full golden set on a schedule, and split evals across workers.
    • Breaks: It breaks when the golden set goes stale. A golden set that never changes gives a green light that means nothing, and a bad prompt ships behind it.
    • Cost: The cost is the eval runs. Scoring every answer would cost too much, so I score about one in twenty and record the score next to the versions that produced it.
    • Trade-off: I give up shipping fast to get a gate that blocks a bad prompt. A strict gate buys safety and costs shipping speed, so I keep the golden set small and the gate fast.

    Advanced — Evals, cost, and the fallback ladder

    Definition: An eval is a test for answers. The score is the only number that says whether the answers are good. Latency says they are fast. Cost says they are cheap. Neither one says they are right. The fallback ladder is the plan B written down before plan A fails. The trace prices every answer, and the price tags add up to the monthly bill.

    Types:

    • The eval score, split by version — prompt version and model version separately, because a blended score hides a bad version behind a good one.
    • The cost per answer — on the same dashboard as latency, alerting on a 50 percent jump the way I alert on latency.
    • The fallback ladder — retry with a stricter prompt, smaller model, cached answer, safe answer with links. Each rung cheaper and worse than the one above.
    • The token accounting — tokens in and out, counted on every request, because a prompt that doubled in size overnight explains a bill that doubled overnight.

    Uses: I use this level when the answers get worse and nothing shipped. I split the eval score by model version. If the old version scores 0.86 and the new one scores 0.71, the model is the change and the prompt is innocent.

    Dimensions:

    • Load: At 10x the traffic the token bill is the first thing that breaks. The cache becomes a design decision, and I add a semantic cache that matches questions by meaning, not by exact text.
    • Breaks: It breaks when the model provider ships a new version behind the same name. The answers get shorter and miss steps, the dashboards stay green, and worse does not page anyone unless I measure it.
    • Cost: The cost is the model choice. The fast model answers in about 1,800 ms at $0.002 per 1,000 tokens. The big model takes about 2,600 ms at $0.010. There is no free upgrade.
    • Trade-off: I give up freshness to get a cache that catches most questions. A higher hit rate buys cost and speed and costs freshness, so I set a 24 hour TTL and rebuild the cache on every docs deploy.

    A prompt is code, and I version it like code. Every prompt has a version number, and every eval run records which prompt version produced the score. When the answers get worse, I split the eval score by prompt version first, because a prompt change is the most common cause and the cheapest to revert. I also keep the old prompt available for a fast rollback, the same way I keep the old model version. A prompt edited in a dashboard with no version is a change nobody can undo, and the answers change without a single commit from my team.

    A fallback ladder is the plan B written down before plan A fails. If the main model is down, the system tries the smaller model. If the smaller model is down, it serves the cached answer. If there is no cached answer, it says so plainly instead of inventing one. Each step down the ladder is a quality drop, and I say what the drop is in the runbook, so the team knows what degraded means before the incident. I test the ladder by turning the main model off on purpose and watching the system step down. A ladder that has never been tested is a hope, not a plan.

    Master — The model changes under you

    Definition: A model update is a deploy I did not ask for. The provider ships a new version behind the same name, and my answers change without a single commit from my team. I treat every model update like a deploy I control. I pin the version, I canary the new one to 5 percent, I split the eval score by version, and I keep the old version available for a fast rollback.

    Types:

    • The pinned model version — in config, recorded on every trace, never floating to latest.
    • The canary rollout — 5 percent of traffic behind a flag, rolling out further only if the eval score holds.
    • The nightly eval — run every night, not just on change, paging the on-call when the score drops by more than 0.05.
    • The prompt registry as a service — its own uptime and its own on-call at 100x, because every model call in the company depends on it.

    Uses: I use this level on the day the answers change and nobody shipped anything. I flip the flag back to the pinned version, and the answers return to the old quality within minutes. Then I confirm from the trace that the next twenty answers all score above 0.82.

    Dimensions:

    • Load: At 100x the prompt registry becomes a service of its own. The budget stops being a target and becomes a contract, and every prompt version owns its share of it, in milliseconds and in dollars.
    • Breaks: It breaks when the nightly eval alert goes to a channel nobody watches at night. The score falls at 10 PM, the first user report lands the next morning, and the bad answers have been live for a day.
    • Cost: The cost is the whole token bill. Five hundred questions a minute at 1,400 tokens on the fast model is $60,480 a month without a cache. The budget is $15,000, so the cache has to catch about two in three questions.
    • Trade-off: I give up the newest model to get answers I can trust. A new version goes to 5 percent first, and it only earns the rest if the eval score holds. The newest answer is not always the best one.

    A provider model update is a deploy I did not ask for. The provider ships a new version behind the same name, and my answers change without a single commit from my team. I pin the version, I canary the new one to 5 percent of traffic, I split the eval score by version, and I keep the old version available for a fast rollback. The canary runs for a day, because a model that looks fine in an hour can fail on the traffic that only arrives at night. If the eval score drops, the flag flips back, and the answers return to the old quality within minutes.

    Cost per answer is the number that keeps the LLM bill honest. I compute it from the tokens: input tokens at the input price plus output tokens at the output price, per answer. Then I multiply by the answers per day, and I alert when the daily cost crosses the budget. The trap is the average: a few long answers cost more than a thousand short ones, so I watch the cost of the most expensive 1 percent separately. A prompt that asks the model to reason for ten paragraphs is a prompt that costs ten times the short one, and the eval has to say whether the long answer is ten times better.

    Go deeper

    The lesson above is one question through nine hops. Production is the rest of it. Here is what I actually spend my time on once the model is answering and the budget is written down.

    LLMOps vs MLOps

    MLOps and LLMOps rhyme, and they are not the same job. In MLOps the model is a file you trained, and the work is retraining it, versioning the weights, and watching for drift in the data. In LLMOps the model is a service someone else runs, and the weights are not yours to touch. The thing you version is the prompt, not the model. A prompt is a few hundred words of instructions, and changing one sentence in it changes every answer the system gives. That is why the line in this lesson is that prompts are the new weights. You do not retrain. You rewrite the instructions, run the evals, and ship the new version behind a flag. The eval harness replaces the accuracy metric. The token bill replaces the training bill. And the model can change under you without anyone on your team shipping anything, which never happens with a model file sitting in your own registry.

    Prompt registry and versioning

    A prompt registry is a versioned store for prompts, the same way a container registry is a versioned store for images. Each prompt has a name, a version number, the text, and the eval score it got. The pipeline loads a prompt by name and version, never by latest. Latest is a trap. Latest means the answer a user gets depends on what time it is, and a 2 AM debug becomes a guessing game about which text the model actually read. I keep the prompt in version control next to the code, review changes the same way, and ship them behind a flag to a small slice of traffic first. The rollback is a flag flip, not a redeploy, and I have used it at 2 AM more than once.

    Evals in CI

    An eval gate in CI is a test suite that runs before a prompt or model change ships. The golden set is a fixed list of questions with known good answers, and the gate scores the change against it. The change ships only if the score does not drop, the cost per answer does not jump, and the p95 latency stays under the budget. That is three gates, not one, because a prompt that is better and twice as expensive is not a free upgrade. I keep the golden set small, about fifty questions, and I refresh it every month with real questions users actually asked, including the ones the pipeline got wrong. A golden set that never changes goes stale, and a stale eval is a green light that means nothing. I also run the evals on a schedule, every night, not just on change, because the model can drift under me and only a nightly eval will catch it.

    Tracing and token accounting

    Tracing an LLM request means recording everything about one question in one place: the prompt version, the model version, the retrieved passages, the tokens in and out, the latency of every hop, and the cost. Token accounting is the cost column of that trace. I put cost on the same dashboard as latency, because a pipeline that is fast and broke is still broken. The metric I watch most is cost per answer, split by prompt version and by model version. A blended cost per answer hides a bad version behind a good one, the same way a blended latency hides a slow hop. And I alert on cost the way I alert on latency: a 50 percent jump in cost per answer pages the on-call, because a prompt that doubled in size overnight is a bug with a dollar sign on it.

    Fallback and degradation tiers

    A fallback is not one thing. It is a ladder. The first rung is a retry with a stricter prompt, for an answer that failed the output check. The second rung is the smaller model, for a call that timed out on the big one. The third rung is the cached answer, for a question someone asked before. The fourth rung is the safe answer: links to the help pages and an honest note that the assistant could not answer. Each rung is cheaper and worse than the one above it, and the pipeline walks down the ladder only as far as it has to. I write the ladder down in the config, with the timeout and the condition for each rung, because a fallback decided in the middle of an incident is a guess. The rule I follow is simple: never serve a confident wrong answer. A safe answer is boring. A wrong answer is a support ticket, or worse.

    Model updates and regressions

    A model update is a deploy you did not ask for. The provider ships a new version behind the same name, and your answers change without a single commit from your team. I treat every model update like a deploy I control. First, the model version is pinned in config, and the trace records it on every request. Second, a new model version goes to a small slice of traffic behind a flag, the same as a canary deploy from Lesson 34. Third, the eval score is split by model version, and a drop blocks the rollout. Fourth, the old version stays available for a fast rollback. I have rolled a model version back the same way I roll back a bad deploy, by flipping the flag, and I was glad the old version was still there. The incident in the next section is exactly this shape, and it is the reason I pin versions.

    What changes at 10x / 100x

    At 10x the traffic, the token bill is the first thing that breaks, so the cache becomes a design decision instead of an optimization. I add a semantic cache that matches questions by meaning, not by exact text, and I watch the hit rate like a budget line. At 100x, the eval harness becomes the bottleneck, so I score a smaller sample, run the full golden set on a schedule, and split evals across workers. The prompt registry becomes a service of its own, with its own uptime and its own on-call, because every model call in the company depends on it. The budget stops being a target and becomes a contract, and every prompt version owns its share of it, in milliseconds and in dollars.

    An LLMOps deploy config

    This is the config shape I would actually write, with plain names and no vendor details. It lays out the pinned prompt and model versions, the eval gate, the cost and latency budgets, and the fallback ladder.

    llmops:
      service: support-answers
      prompt:
        name: support-answer
        version: v14
        rollout: 100
      model:
        name: fast
        version: pinned-2026-11
        timeout_ms: 5000
        max_retries: 1
      budgets:
        max_p95_ms: 3000
        max_cost_per_answer_usd: 0.003
        monthly_budget_usd: 10000
      eval_gate:
        golden_set: support-golden-v3
        min_score: 0.82
        on_fail: block_deploy
      fallback:
        - retry_stricter_prompt
        - smaller_model
        - cached_answer
        - safe_answer_with_links
      tracing:
        record: [prompt_version, model_version, tokens_in, tokens_out, cost_usd, latency_ms]

    Read it top to bottom and it is the whole lesson in config form. The prompt is pinned to version 14 at 100 percent of traffic, and the model is pinned to a named version with a five second timeout and one retry. The budgets say the p95 must stay under three seconds, each answer must cost under three tenths of a cent, and the month must stay under ten thousand dollars. The eval gate blocks any deploy that drops the golden set score below 0.82. The fallback ladder is written down in order, so the pipeline never has to guess what to do when the model fails. And the trace records the four things the 2 AM debug needs: prompt version, model version, tokens, and cost. The config is the operations. The rest is just code.

    Eval regression detection is a comparison, not a single number. I run the eval suite on every prompt and model change, and I compare the new score with the last good score on the same questions. A drop of more than a few points blocks the change. The questions matter as much as the score: an eval set built from last month's tickets never saw the new product, so the new version looks good on paper and fails on live data. I refresh the eval questions every month with real failures from production, and I keep the old questions too, so a regression on the old behavior shows up as well.

    in_tokens, out_tokens = 800, 200
    price_in, price_out = 3.00, 15.00  # dollars per million tokens
    cost = in_tokens / 1e6 * price_in + out_tokens / 1e6 * price_out
    print(f"one answer: ${cost:.4f}")
    print(f"10,000 answers a day: ${cost * 10000:.2f} per day")

    Expected output: one answer costs $0.0054, and 10,000 answers a day is $54.00 a day. The per-answer number is the one I carry into design reviews. A feature that answers 10,000 times a day at half a cent each is a $54 a day feature, and the question is whether it earns that every day.

    One time it broke

    This one stayed with me because nobody shipped anything. No deploy ran, no prompt changed, no config was edited, and the answers got worse anyway.

    What broke

    The model provider shipped a new version of the model behind the same name. The support assistant started giving shorter answers that missed steps it used to include. The prompt was the same. The code was the same. The retrieval hop was the same. Only the model had changed, and it had changed without telling us. The eval score fell from 0.86 to 0.71, and the output guardrail started rejecting one in five answers, so users got the safe reply with links instead of a real answer. The dashboards for latency and errors were green, because nothing had failed. The answers were just worse, and worse does not page anyone unless you measure it.

    Timeline

    • Day 1, 10:00The provider ships the new model version behind the same name. Nothing on our side changes.
    • Day 1, 22:00The nightly eval runs and the score falls from 0.86 to 0.71. The alert fires, but it goes to a channel nobody watches at night.
    • Day 2, 09:15First user report: the answer missed a step. Support answers it by hand and moves on. Nobody connects it to the model.
    • Day 3, 11:30Third and fourth reports land, same pattern. Shorter answers, missing steps. I open a trace and see the model version is different from last week.
    • Day 3, 12:00I check the eval score split by model version. The old version scores 0.86. The new version scores 0.71. The model is the change, and the prompt is innocent.
    • Day 3, 13:20I flip the flag back to the pinned model version. Answers return to the old quality within minutes.
    • Day 3, 15:00I confirm from the trace that the next twenty answers all score above 0.82, and I close the reports with the fixed steps.

    What changed after

    Three things. First, the model version is pinned in config and recorded on every trace, so the next change shows up in the trace on day one, not day three. Second, the nightly eval alert pages the on-call when the score drops by more than 0.05, because a score that falls at night is still a regression. Third, a new model version goes to 5 percent of traffic behind a flag first, the same as a canary deploy, and it only rolls out further if the eval score holds. The lesson I took home is simple: a model you do not version is a deploy you do not control, and a deploy you do not control will change your answers on a day you did not choose.

    If you remember nothing else

    Design exercise

    This is the kind of question I would ask in an interview, and the kind I ask myself before I sign off on a real LLM service. Take a minute with it before you read my approach.

    The prompt

    Design the LLMOps setup for a support assistant that answers questions from 50,000 support documents. It must handle 500 questions a minute, answer 95 out of 100 questions in under 3 seconds, and keep the monthly model cost under $15,000. The team ships a prompt change every week and the model provider ships a new model version every month. You have the pipeline from Lesson 61, the guardrails from Lesson 65, a fast model at $0.002 per 1,000 tokens, and a big model at $0.010 per 1,000 tokens. Sketch the operations: versions, gates, budgets, fallbacks, and what you watch.

    Clarifying questions to ask first

    I would ask five things before I sized anything. How fresh must the answers be: can a cached answer from yesterday serve, or must it reflect today's docs? What counts as a good answer, and who judges it: the user vote, a human review, or an eval set? What happens when the model does not know: does it say so, or guess? How spiky is the traffic: is 500 a minute the average or the peak, and what happens at twice that? And what is the failure we design for: a bad prompt version, a bad model version, or the provider being slow?

    Rough sizing, worked

    Start with the simulator math from this lesson. Five hundred questions a minute is 21,600,000 questions a month. At 1,400 tokens and the fast model, every model call costs $0.0028, so serving all of them costs $60,480 a month. The budget is $15,000, which buys about 5.36 million calls, or about 124 calls a minute. That means the cache must catch about 75 out of every 100 questions, which is a lot to ask. So I cut the tokens first: shorten the prompt to 600 tokens and cap the answer at 400, which is 1,000 tokens a call and $0.002 a call. Now $15,000 buys 7.5 million calls a month, or about 174 a minute, and the cache needs to catch about 65 out of 100. I set the target at a 65 percent semantic cache hit rate, which leaves 175 model calls a minute. That is 7,560,000 calls a month at $0.002, which is $15,120, just over the budget, so I trim the prompt a little more or accept a 66 percent hit rate and land at $14,688. Latency: 65 out of 100 answers come from the cache at 90 ms, and the other 35 take the full path at 1,925 ms. The 95th percentile sits in the slow group, at 1,925 ms, which is inside the 3 second target. The big model stays out of the hot path entirely. It runs only on the eval set and on 5 percent of traffic to check whether its answers are actually better, and only then would I pay its price.

    Trade-offs to name

    A higher cache hit rate buys cost and speed and costs freshness, so I set a 24 hour TTL and rebuild the cache on every docs deploy. A shorter prompt buys cost and costs completeness, so I cap the prompt and link the source page for the rest. The fast model buys cost and speed and costs some answer quality, so I measure the quality gap with the eval harness instead of guessing. A strict eval gate buys safety and costs shipping speed, so I keep the golden set small and the gate fast. Weekly prompt changes buy improvement and cost stability, so every change rolls out to 5 percent first and the rollback is a flag flip. And the target is p95, not the average. An average of 1 second with a p95 of 6 seconds fails this design.

    Saved in this browser, so the Lessons index can show your progress across the course.

    Next in this series
    Lesson 80 — DataOps

    This lesson was about operating one model in production. The next lesson opens up the other side of the operations discipline: how the data that feeds every system stays fresh, tested, and trustworthy as it moves through pipelines. That is DataOps, and it is the next hop in this module.