System Design · Lesson 76

MLOps

MLOps is what happens after a model works once. A model is a piece of software that learns from data. The first time you train one, you run a notebook on your laptop and it answers a few questions. That feels like the job is done. It is not. The job is keeping that model right, fast, and safe for months, while the data under it keeps changing. Here is the concrete example I will use all the way through. A support site uses a model to sort incoming tickets. A ticket arrives, the model reads it, and it picks the team that should handle it: billing, login problems, or delivery. The model was trained on last year's tickets. This year the tickets look different. New products, new words, new kinds of problems. If nobody watches, the model keeps answering with last year's knowledge and sends tickets to the wrong teams, quietly, for weeks. MLOps is the system around the model that stops that. It retrains the model on fresh data. It keeps every version, so a bad one can be rolled back. It watches the live data and rings a bell when the data moves. I have debugged these systems at 2 AM the same way I debug any other system: one hop at a time, with a trace in one hand and a dashboard in the other. The model is the famous part. The operations around it decide whether it stays right.

Difficulty: Advanced · Know this first: Lesson 61, Lesson 49
Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one ticket, not a measurement from a live system, but the shape is honest: on a normal ticket, the model call is 180 ms of the 370 ms trip, and the watching hops, logging and drift scoring, are the next biggest slice. Step through the lesson or switch the scenario and watch the bar change. The drifted path nearly doubles the trip without the ticket changing at all. The canary path costs more per ticket, but only one ticket in ten pays it, because the rest still ride version 23.

Move the data and watch the model go stale

The budget above is one fixed ticket. This box is yours. Push the traffic up, drop the feature hit rate, grow the canary, push the drift score past the line, and watch the latency, the flagged count, and the retrain bill move. Then ask the question I ask before every deploy: how much of this system is answering, and how much is watching for the day the data moves?

How many tickets the model sorts every minute. One hundred is a quiet internal queue. Ten thousand is a busy public support site. More predictions means more chances for drift to show up, and more flagged tickets to look at.

What share of tickets find their features already computed in the store. Ninety percent is a healthy store. Fifty percent means half the tickets wait while the numbers are computed fresh, and every miss costs about 140 ms extra.

What share of tickets the new model version serves. Ten percent is a careful rollout. Fifty percent is half the traffic on an unproven version. The new version answers in about 240 ms against the old version's 180 ms, so this slider moves the average latency directly.

How far the live tickets have moved from the training data. Zero means the live data looks exactly like training. Past 0.20 the model is answering about a world it never trained on. This slider moves the flagged count and the retraining schedule.

Average prediction latency392 msfeatures at 56 ms average, model at 186 ms average, the rest fixed at 150 ms
Flagged predictions per day57,6001,440,000 predictions a day, about 4 in 100 flagged low confidence
Retrains per week1drift 0.08 is under the 0.10 line, so the weekly schedule holds
Monthly retrain cost$7791 retrain a week at $180 a run

The data is steady. The model is answering about the world it trained on, and the weekly retrain is enough.

Try this: (1) Push the drift score to 0.30 at the defaults. Flagged predictions jump from 57,600 a day to 216,000 a day, retrains go from 1 a week to 4, and the monthly retrain cost climbs from $779 to $3,118, and the tickets never changed. (2) Now drop the feature hit rate to 50%. The average latency climbs from 392 ms to 456 ms. Same model, same data. Only the feature store changed.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Go deeper

The lesson above is one ticket through nine hops. Production is the rest of it. Here is what I actually spend my time on once the model is answering and the budget is written down.

Data drift vs concept drift

Drift comes in two shapes, and they need different fixes. Data drift means the inputs changed. The tickets use new words, the amounts are bigger, a field that used to be full is now empty half the time. The model sees data it never trained on. Concept drift means the meaning changed while the inputs stayed the same. The same words now mean a different team, because the company reorganized who handles what. The tickets look identical. The right answer moved. Data drift is caught by comparing the live numbers with the training numbers, field by field. Concept drift is caught by watching the correction rate, because the inputs look fine and only the verdicts say the model is wrong. I have seen teams retrain for concept drift and wonder why nothing improved. Retraining on the same labels teaches the model the same old meaning. Concept drift needs new labels, from the teams doing the corrections, before retraining helps at all.

How the drift score is measured

The drift score in this lesson is a plain number from 0 to 1, and behind it are two standard tests. One is called PSI, population stability index. It splits a field into buckets, counts how many live tickets fall in each bucket, counts how many training tickets fell in each bucket, and adds up the differences. A PSI under 0.10 means the field is steady. Over 0.25 means it moved. The other is the KS test, which compares the two distributions without buckets and reports the biggest gap between them. I use PSI for fields I can bucket, like ticket length or account age, and KS for continuous numbers. The rule I follow is simple: alert on one field moving, not just on the total. A total score hides which field moved, and the field is the clue that tells me what changed in the real world.

Retraining triggers

A model should be retrained for a reason, not on a feeling. I use four triggers, and the team writes them down before the model ships. A schedule trigger: retrain every week, whether it needs it or not, because fresh data is cheap insurance. A drift trigger: retrain when the drift score passes 0.20 on any important field. A performance trigger: retrain when the correction rate climbs past 5 percent, or the accuracy on recent tickets drops below the line we set. And a data trigger: retrain when a new product launches or a big batch of corrected labels lands. The worst trigger is the one nobody wrote down, because then retraining happens when someone notices the model is wrong, which is always weeks too late. Every trigger fires into the same pipeline: fresh data, test, registry, rollout. A retrain that skips testing is not a retrain. It is a gamble.

Reproducibility: data plus code plus model

Reproducibility means I can rebuild the exact model version that is serving right now, from scratch, and get the same model. That takes three things, pinned down together: the data snapshot, the training code version, and the model artifact itself. The data snapshot is a frozen copy of the training tickets, with a hash, so I know exactly which tickets trained this version. The code version is a commit, so I know exactly which training script ran. The model artifact is the trained file in the registry, with its own hash. If any one of the three is missing, I cannot reproduce the model, and I cannot explain a bad answer with confidence. I have been in the room where a team could not say which data trained the model that was live. That is not a small gap. It is the whole audit trail missing, and it means the model cannot be trusted, fixed, or defended.

MLOps vs DevOps

MLOps is DevOps with a third thing to version. DevOps versions code and config. MLOps versions code, config, and data, and the model that comes out of all three. That one difference changes almost everything. A code deploy is deterministic: the same code gives the same binary. A model retrain is not: the same code on new data gives a new model, with new behavior, and the only way to know whether it is better is to test it on recent data and compare it with the live version. Rollback is different too. Rolling back code means deploying the old version. Rolling back a model means pointing the registry at the old version, which is fast, and that is why the registry matters. Testing is different as well. Code has unit tests. A model has evals, drift checks, and a canary. I run both disciplines the same way, with pipelines, gates, and rollbacks, but I never pretend a model is just code. The data is the part that moves, and the operations exist to catch it moving.

Feature stores, online vs offline, and skew

A feature store has two sides, and the split causes most of the skew bugs I have seen. The offline side serves the training job. It can take its time, read old data, and compute features in big batches. The online side serves live tickets. It must answer in milliseconds, so it serves numbers that were computed ahead of time and stored ready. Skew happens when the two sides compute the same feature differently: one rounds, one does not, one counts the last 30 days, the other counts the last 7. The model trained on one version and serves on another, and the answers quietly get worse. The fix is boring and it works: one definition, written once, used by both sides. I keep the feature definitions in version control, next to the code, and the offline and online sides both read the same file. If a feature definition changes, it goes through review like a code change, because it is one.

Shadow vs canary

Shadow and canary are the two safe ways to ship a model, and they answer different questions. Shadow means the new version answers every ticket in the background, and nobody sees its answers. I compare them with the live version and learn whether the new model is better, with zero risk to users. Shadow answers one question: is the new model right. Canary means the new version serves real tickets, a small slice first, and users see its answers. I watch the latency, the errors, and the corrections, and I grow the slice if it behaves. Canary answers a different question: is the new model safe in production. I run shadow first, then canary. Shadow is free learning. Canary is controlled risk. Skipping shadow to save time is how a model that scored well in testing meets live data for the first time on real users.

What changes at 10x / 100x

At 10x the traffic, the feature store is the first thing that breaks. I move the hot features to a fast store with a short TTL, precompute the expensive ones on a schedule, and watch the hit rate like a budget line. The prediction log becomes a storage problem too, so I sample it: every prediction's verdict, but only one in ten full feature rows, and I say so in the runbook. At 100x, retraining becomes a pipeline of its own, with its own queue, its own machines, and its own alerts, because a retrain that fights the serving path for GPUs will make live tickets slow. The drift watch moves from hourly to near real time, and the registry gets approval gates, so no version ships without a human signing the eval report. The budget stops being a target and becomes a contract, and every hop owns its share of it, in milliseconds and in dollars.

A drift monitor and retrain trigger config

This is the config shape I would actually write, with plain names and no vendor details.

model:
  name: ticket-router
  live_version: v23
  registry: model-registry
  features:
    store: ticket-features
    definitions_version: v11
drift_monitor:
  run_every_minutes: 60
  fields:
    - ticket_length
    - product_name
    - account_age_days
  method: psi
  warn_at: 0.10
  alert_at: 0.20
  alert_to: ml-oncall
retrain_triggers:
  schedule: weekly
  drift_above: 0.20
  correction_rate_above: 0.05
  on_new_product_launch: true
rollout:
  strategy: canary
  start_share: 0.10
  step_share: 0.10
  step_wait_minutes: 60
  rollback_on:
    correction_rate_above: 0.08
    p95_latency_ms_above: 500

Read it top to bottom and it is the whole lesson in config form. The model is version 23, and its features come from one store with one definitions version, so training and serving cannot drift apart silently. The drift monitor runs every hour, watches three fields with PSI, warns at 0.10, and pages the on-call at 0.20. The retrain triggers are written down, not felt: weekly, drift above 0.20, correction rate above 5 percent, or a new product launch. The rollout is a canary that starts at 10 percent, grows by 10 percent every hour, and rolls back on its own if the correction rate passes 8 percent or the p95 latency passes 500 ms. The rollback rules are the part I care about most. A rollout that cannot roll itself back is a rollout I have to babysit, and I have better things to do at 2 AM.

One time it broke

This one stayed with me because nothing failed. Every dashboard was green, every pod was healthy, and the model was quietly wrong for three weeks.

What broke

A model sorted incoming support tickets into teams. One Monday, an upstream team changed a field in the ticket form. A dropdown that used to send the product name started sending a product code instead. The tickets looked the same on the surface, same length, same words in the body, but the product field, the one the model leaned on most, was now a code the model had never seen in training. The model kept answering fast, with high confidence, and it sent a growing share of tickets to the wrong team. The latency dashboard was green. The error rate was zero. The only number that moved was the correction rate, and nobody had an alert on it. Teams quietly re-routed tickets by hand for three weeks and assumed the other teams were just slow.

Timeline

  • Week 1, Monday, 10:00The upstream form change ships. The product field starts carrying codes instead of names. No alert fires, because no check watches that field.
  • Week 1, Wednesday, 14:20First team mentions in a standup that tickets feel misrouted lately. Nobody opens a ticket about it. It sounds like a people problem.
  • Week 2, Tuesday, 09:30I notice the correction rate in the weekly report. It climbed from 3 percent to 11 percent in ten days. That is not noise. That is a model going wrong.
  • Week 2, Tuesday, 11:00I open the prediction log and split the corrections by field. Every bad routing has the same pattern: the product field holds a code, and the model treats it as an unknown product.
  • Week 2, Tuesday, 13:40I check the drift score. It has been above 0.20 on the product field since the Monday of the form change. The watch was measuring it. Nobody was reading it.
  • Week 2, Wednesday, 10:00I fix the feature definition to translate codes back to names, retrain on the fresh tickets, and ship the new version as a canary at 10 percent.
  • Week 2, Friday, 16:00The canary correction rate is back under 3 percent. I roll the new version out to everyone and close the loop with the upstream team.

What changed after

Three things. First, the drift monitor got an alert on any important field passing 0.20, wired to the same on-call rotation as the latency alerts, because a stale model is an outage that does not page itself. Second, the feature definitions got a contract test: every upstream change to a ticket field runs a check that the feature store still sees the values it expects, and the change does not ship if the check fails. Third, the correction rate got a dashboard panel next to the latency panel, with an alert at 5 percent, because the teams doing the corrections are the fastest sensor in the whole system. The lesson I took home is simple: a model that is fast, confident, and wrong is the most dangerous system I run, because every dashboard I trust says it is fine.

If you remember nothing else

Design exercise

This is the kind of question I would ask in an interview, and the kind I ask myself before I sign off on a real model. Take a minute with it before you read my approach.

The prompt

Design the MLOps system for a model that predicts which deliveries will arrive late. It must sort 50,000 deliveries a day, answer each prediction in under 500 ms at p95, retrain every week on fresh delivery data, and keep a bad model from ever reaching all users at once. You have a feature store, a model registry, a training pipeline, and a monitoring dashboard. Sketch the system and the budget.

Clarifying questions to ask first

I would ask five things before I sized anything. How fresh must the prediction be: does it use live tracking data, or is yesterday's data enough? What counts as late, and who judges it: the courier scan, the customer report, or both? What happens when the model is unsure: does a human look, or does the system just send the standard message? How spiky is the traffic: is 50,000 a day the average or the peak, and what happens on a holiday? And what is the failure we design for: the data drifting, the model down, or a retrain that ships a worse version?

Rough sizing, worked

Start with the traffic. 50,000 deliveries a day is about 0.58 predictions a second on average, and maybe 5 a second at the evening peak. Little's Law for the sizing: 5 predictions a second at 0.37 seconds in the system is about 2 predictions inside at once, so the serving path barely notices the load, and I plan the model servers for 5 a second with headroom to 20. The budget from this lesson fits: features 40, registry 15, model 180, check 25, return 10, logging 35, drift 45, retrain decision 20, for 370 ms total, inside the 500 ms target with 130 ms of headroom. The storage math is the real sizing. One prediction record is about 2 KB, so 50,000 a day is 100 MB a day, and a year is about 36 GB, which fits on one disk with room to spare. The training data is bigger: a year of deliveries with features is roughly 500 GB, so I keep the last 90 days hot for retraining and archive the rest cold. Retraining weekly on 90 days of data takes about 2 hours on the training machines, so I schedule it overnight and it never touches the serving path. The canary math is simple: 10 percent of 50,000 is 5,000 predictions a day on the new version, which is enough to compare correction rates in a day and small enough to roll back without a bad week.

Trade-offs to name

A weekly retrain buys freshness and costs compute, so I retrain on 90 days, not a year, and archive the rest. A canary buys safety and costs speed on the canary slice, so I start at 10 percent and grow by 10 percent an hour. Strict confidence floors buy safety and cost human reviews, so I set the floor at 0.70 and watch the human queue size. A shared feature store buys consistency and costs coordination, so the feature definitions live in version control and change through review. And the target is p95, not the average. An average of 300 ms with a p95 of 900 ms fails this design.

Saved in this browser, so the Lessons index can show your progress across the course.

Next in this series
Lesson 77 — AIOps

The model you just operated sorts tickets and predicts late deliveries. The next lesson turns the operations themselves over to the machine: systems that watch, decide, and act on their own. That is AIOps, and it is the next hop in this module.

Lesson 77 — AIOps →