Here is the plain version first. A model is a file. Training makes the file. That is what Lesson 88 taught. But a file sitting on a disk helps nobody. Somebody has to wrap it up, check it, put it in front of real traffic a little at a time, watch it, and decide whether it stays or goes back. And then do the whole thing again next week, because the data moved. That loop, from trained file to live traffic and back to training again, is the model lifecycle. Here is the concrete example I will use all the way through, the same one from Lesson 76 and Lesson 88. A support site sorts incoming tickets into teams with a small model. The model is called the ticket router. Version 23 is live right now. Version 24 just finished training on fresh tickets, including the corrections the teams made. This lesson walks version 24 from the training machine to full production, hop by hop, and shows the two other ways that trip can go: a slow batch rollout instead of a live one, and a version that fails its checks and never ships at all. I have debugged these trips at 2 AM the same way I debug any other deploy: one hop at a time, with a trace in one hand and a dashboard in the other. The model is the famous part. The lifecycle around it decides whether a good model reaches users safely, and whether a bad one ever does.
Difficulty: Advanced · Know this first: Lesson 76, Lesson 88, Lesson 61
Every hop charges rent in minutes. The bar below is a worked example for one model version, not a measurement from a live system, but the shape is honest: on the canary path, watching costs more than moving. The soak and the first live hour are 120 of the 175 minutes, and that is the point. The deploy itself is 15 minutes. The proof that the deploy was safe is two hours. Step through the lesson or switch the scenario and watch the bar change. The batch path is slower, because scoring a whole day of tickets takes a night, and the failed path is the cheapest of all, because the gate stopped the trip before any user saw the new version.
The budget bar above shows three fixed stories. This simulator lets you write your own. It answers one question: should version 24 take the title from version 23? The champion, version 23, makes a mistake on 4.0 percent of tickets. That number is fixed. You set how many tickets the system sorts a day, how big the canary slice is, how often the challenger gets it wrong, and how fast the rollout climbs. Move a slider and watch the canary count, the mistakes avoided, the time to full rollout, and the verdict change. The numbers are a teaching example, not live measurements. The shape is the lesson: a better challenger is worth promoting, a worse one is worth sending back, and the canary slice decides how much a wrong call costs.
How many tickets the model sorts a day. Ten thousand is a quiet internal queue. Five million is a busy public support site. More tickets means a canary learns faster, and a bad rollout hurts more.
What share of tickets the challenger serves. Ten percent is a careful rollout. Fifty percent is half the traffic on an unproven version. A bigger slice learns faster and fails louder.
How often the challenger gets it wrong. Below 4.0 percent it beats the champion. Above 4.0 percent it loses. The simulator does not care what you hope. It cares what the number says.
How fast the rollout climbs after the canary. Ten percent a step is careful. Fifty percent a step is a sprint. Every step soaks for 6 hours, so bigger steps mean fewer steps and a faster full rollout, with less time to catch a problem at each step.
Try this: (1) Push the challenger error rate to 5.0 percent at the defaults. Bad predictions avoided flips to minus 5,000 a day, and the verdict flips to roll back. Same model, same traffic. Only the error rate changed. (2) Now push the canary share to 50 percent. The canary predictions jump from 50,000 a day to 250,000 a day, and the bad predictions on the canary jump from 1,600 to 8,000. A bigger canary learns faster, and it fails louder. (3) Set the rollout step size to 50 percent. Hours to full rollout fall from 54 to 12. Ask yourself what you lose with that speed: three fewer soak windows, three fewer chances to catch the problem before everyone has it.
Three questions. No score is saved anywhere — this is just you against the material.
The lifecycle is not one thing. It is four levels of the same job, and every team I have worked with sits on one of them. The levels are not about how smart the model is. They are about how much of the trip from training to production is written down, versioned, checked, and automated. Read them in order. Each level fixes the failure that hurt the most at the level below it.
At the basic level, the lifecycle is a file and a script. Somebody trains a model, saves the file to a disk, and a script loads the file and answers questions. There is no registry, no version number, no gate, and no rollback. The model works, and the person who trained it knows where the file lives. That is the whole system. It is how every model starts, and it is fine for exactly one job: proving the model can work at all.
Types:
Uses: A demo for one stakeholder. A one-off analysis that runs once and is never served again. A first version built to answer one question: does this model do the job at all. I use this level to learn, never to serve real users.
Scale: One model, one machine, one user at a time. It breaks the moment a second person needs the same model, because there is only one copy of the file and no record of where it came from.
Failure modes: The file gets overwritten and the old version is gone forever. The model goes stale and nobody notices, because nothing watches it. The person who trained it leaves, and the file becomes a mystery nobody dares to touch.
Cost: Almost nothing in infrastructure. All of the cost is human time: somebody babysits the file, restarts the script, and answers questions about what the model does and why.
Trade-offs: Fast to start, impossible to trust. You get a working model in an afternoon, and you pay for it the first time the model is wrong and nobody can say which version is live, what data trained it, or how to put the old one back.
At the intermediate level, the model becomes a versioned artifact and the trip becomes a pipeline. The trained file goes into a model registry with a version number and a record of its data and code. A CI/CD pipeline, the same shape from Lesson 33, packages the model, runs a test, and deploys it. Batch scoring appears here too: a job that scores a pile of tickets on a schedule, instead of answering live. The model is still deployed by a person pressing a button, but the button runs a pipeline, not a script typed from memory.
Types:
Uses: A small team serving one model that retrains once a month. An internal tool where a bad answer is an annoyance, not an outage. A batch job that scores yesterday's tickets overnight, where a mistake is found the next morning and fixed the next day.
Scale: One model, a few versions, one team. It holds up to a handful of deploys a month. It starts to strain when a second model arrives, because the pipeline was written for one model and the registry holds versions nobody cleans up.
Failure modes: The eval runs on old data, so a bad model passes the gate and ships. The deploy is still a button press, so a tired person ships the wrong version. The batch job fails overnight and nobody notices until the morning, because the job has no alert.
Cost: A registry, a pipeline, and a little storage for old versions. Cheap in money, and the first real cost in discipline: somebody has to write the eval, keep the registry clean, and answer the page when the batch job fails.
Trade-offs: You gain versions and a repeatable deploy, and you pay with process. The pipeline only helps if the team actually uses it, and the first time somebody ships a model by hand to save time, the versions stop meaning anything.
At the advanced level, the lifecycle stops trusting people to catch mistakes and starts checking itself. The eval gate runs on recent data, not old data. The deploy is a canary: 10 percent of traffic first, compared against the champion on the same traffic, with promotion and rollback rules written down before the canary starts. The drift watch from Lesson 76 runs beside the serving path, and the correction rate has an alert on it. This is the level this lesson teaches in full, and it is the level a production model needs before it serves real users.
Types:
Uses: A production service where a bad model costs real users: support routing, fraud checks, recommendations. A team that ships a new version every week or two and needs the rollout to be safe without a person watching every step. Any model where the data moves, which is every model that matters.
Scale: A few models, weekly deploys, thousands to millions of predictions a day. The canary needs enough traffic to compare versions honestly, so a quiet service needs a longer soak to learn the same thing a busy service learns in an hour.
Failure modes: The gate runs on old data and passes a stale model. The canary slice is too small to tell the versions apart, so a bad challenger gets promoted on noise. The rollback rule exists but nobody tested it, and it fails the one night it is needed.
Cost: More compute for the canary slice and the shadow run, more storage for prediction logs, and real engineering time to write the gates and the rollback rules. The cost is visible on the bill, and it is smaller than the cost of one bad rollout.
Trade-offs: You gain safety and speed together, and you pay with complexity. The pipeline has more moving parts, and each part can break. A gate that is too strict blocks good models. A gate that is too loose is decoration. Tuning the gates is the job, and it never ends.
At the master level, the lifecycle is a platform, not a pipeline. Retraining triggers fire the training job on their own. The eval gates run in the pipeline with no human in the loop. Promotion and rollback happen on rules, and a human only signs off on the exceptions. Every version carries full lineage, from the raw data to the live pointer, so an auditor can rebuild any version that ever served. Many models share the same platform, and a team that brings a new model gets the registry, the gates, the canary, and the rollback for free, because the platform already has them.
Types:
Uses: A company running many models across many teams, where each team cannot be trusted to build its own lifecycle safely. A regulated setting where an auditor can ask which data trained the model that made a decision last Tuesday, and the answer has to be exact. A high-stakes model, like fraud or safety, where a bad version reaching all users is a real loss, not a bad afternoon.
Scale: Tens or hundreds of models, daily deploys, billions of predictions. The platform scales by making the lifecycle boring: the same gates, the same canary, the same rollback, for every model, so adding a model costs almost nothing.
Failure modes: The automation itself becomes the risk. A misconfigured gate blocks every model at once. A trigger fires on a bad data batch and retrains the fleet on garbage. A silent data change passes every check because the checks measure the wrong fields. The platform team is the single point of failure, and its on-call rotation is the busiest in the company.
Cost: A platform team, a fleet of training machines, and storage for every version and every prediction log. The per-model cost falls as models share the platform, but the fixed cost is the highest in this lesson, and it only pays off past a handful of models.
Trade-offs: You gain speed, safety, and auditability at a scale no team can do by hand, and you pay with platform complexity and a new kind of risk: the automation is right until the day it is wrong everywhere at once. The master level is not the advanced level done harder. It is a different job, and a team should only take it on when the number of models makes the hand-built lifecycle more expensive than the platform.
The registry is not a folder with version numbers. It is the source of truth for every question that matters after a deploy. Which version is live right now. Which version was live last Tuesday when the bad answers started. Which data trained that version, which code trained it, and who approved it. I treat the registry like I treat a database of record: nothing writes to it except the pipeline, nothing serves traffic unless the registry says so, and every change to it is logged with a name and a time. The rule that makes all of this work is simple. A version that is not in the registry does not exist. A deploy that skips the registry is not a deploy. It is a person copying a file to a server, and the next person on call will not know it happened.
Champion and challenger only works if the comparison is fair, and fair has four parts. Same traffic: the challenger must see the same kind of tickets as the champion, not the easy ones. Same features: both versions read the same feature definitions from the same store, the rule from Lesson 76, or the comparison measures the features, not the models. Same floor: both versions are judged by the same confidence floor, so a challenger that answers everything with low confidence does not win by answering more. And enough traffic: ten tickets cannot tell two versions apart. I size the canary by the question, not by habit. A 10 percent slice on 500,000 tickets a day is 50,000 tickets, and that is enough to compare correction rates in a day. The same 10 percent on 1,000 tickets a day is 100 tickets, and that is noise. A quiet service needs a longer soak, not a bigger slice.
An eval gate is a test suite for a model, and it belongs in the pipeline the same way unit tests belong in a code pipeline. The gate I run has four checks, in order. First, data checks: the eval data is recent, it has no missing labels, and none of it leaked into the training data, the leak from Lesson 88. Second, the eval suite: the new version scores better than the champion on recent tickets, on the metric the team agreed on, not on the metric that makes the new version look good. Third, the latency check: the new version answers inside the budget at the traffic it will actually serve, not on an empty test machine. Fourth, the output-shape check: every answer has the fields the router needs, in the shape the router expects. All four must pass. A gate that warns and ships anyway is not a gate. It is a dashboard with a deploy button next to it.
Batch and online are not two versions of the same thing. They are two different promises. Online promises an answer in milliseconds, for one ticket, right now, and it pays for that promise with always-on machines and a canary that can hurt real users. Batch promises an answer for every ticket, by tomorrow morning, and it pays for that promise with a night of compute and a comparison so clean it cannot be beaten: both versions scored the exact same tickets. I pick batch when the answer is not needed now: nightly reports, next-day routing, anything where a user will not wait on the answer. I pick online when the user is waiting. The mistake I have seen is picking online for a batch-shaped problem, because online feels more serious, and then paying for always-on machines and canary risk to answer a question nobody asked in real time.
Retraining should fire for a reason, and the reasons should be written down before the model ships. I use the four triggers from Lesson 76: a schedule, a drift score past 0.20, a correction rate past 5 percent, and a new product launch. Continuous training means the triggers fire the pipeline on their own, and the pipeline runs the same gate and the same canary as a hand-triggered retrain. The human moves from the loop to the edge of the loop: they review the report, they approve the exceptions, they do not press the button for every version. The warning is the same as with any automation. A trigger that fires on a bad data batch retrains the model on garbage, and the gate is the only thing standing between that garbage and production. I keep a human approval on the promotion step for as long as the team is small, and I only automate promotion when the gates have caught three bad versions in a row without a human noticing first. Trust in automation is earned, one caught failure at a time.
At 10x the traffic, the prediction log becomes the first problem. A million predictions a day at 2 KB a record is 2 GB a day, and a year of it is a storage line item, not a rounding error. I sample it: every verdict, one in ten full feature rows, and I write the sampling rule in the runbook so the next person knows what is missing. The canary also changes. At 10x, a 10 percent slice is a city of tickets, and the comparison is done in hours, not days. At 100x, the lifecycle becomes a platform, and the registry gets approval gates: no version ships without its lineage, its eval report, and a human signing the report. The budget stops being a target and becomes a contract, and every hop owns its share of it, in minutes and in dollars. The shape of the work does not change. The number of models does, and the platform is what keeps that number from becoming a headcount.
model:
name: ticket-router
live_version: v23
candidate_version: v24
registry: model-registry
artifact:
image: ticket-router:v24
data_snapshot: tickets-2026-11-12
training_code: a1b2c3d
eval_gate:
data:
recent_days: 14
no_leak_check: true
checks:
- accuracy_above_champion: true
- p95_latency_ms_below: 500
- output_shape: valid
fail_action: stop_pipeline
rollout:
strategy: canary
start_share: 0.10
step_share: 0.10
step_wait_minutes: 360
promote_if:
correction_rate_below_champion: true
p95_latency_ms_below: 500
rollback_on:
correction_rate_above: 0.08
p95_latency_ms_above: 500
retrain_triggers:
schedule: weekly
drift_above: 0.20
correction_rate_above: 0.05
on_new_product_launch: true
Read it top to bottom and it is the whole lesson in config form. The model is the ticket router. Version 23 is live, version 24 is the candidate, and the artifact carries its data snapshot and its training code commit, so the version can be reproduced. The eval gate runs on the last 14 days of tickets, checks for a leak first, and then checks three things: the new version beats the champion, its p95 latency stays under 500 ms, and its output has the right shape. If any check fails, the pipeline stops. The rollout is a canary that starts at 10 percent, grows by 10 percent every 6 hours, promotes only if the correction rate beats the champion and the latency stays inside budget, and rolls back on its own if the correction rate passes 8 percent or the p95 passes 500 ms. The retrain triggers are the same four from Lesson 76. The rollback rules are the part I care about most. A rollout that cannot roll itself back is a rollout I have to babysit, and I have better things to do at 2 AM.
This one stayed with me because the lifecycle had no gate for the thing that broke it. The model did exactly what it was built to do. That was the problem.
In March 2016, Microsoft launched Tay, a chatbot on Twitter that was designed to learn from the people it talked to. The idea was simple: the more people chatted with Tay, the more natural its replies would become. The model kept learning from live conversations, with no gate between what users taught it and what it said next. Within hours, users were feeding it offensive language, and Tay was repeating it back, in its own voice, to everyone. The model was not hacked. It was not broken. It was learning from live data with no filter, no eval gate, and no human in the loop, and the live data was exactly what the system rewarded. Microsoft took Tay offline within about 16 hours of launch. The lesson is not that learning from users is wrong. The lesson is that a model that learns from live traffic without a gate is a model whose behavior is decided by whoever talks to it loudest.
Three things, and they are the same three things this lesson teaches. First, a gate between live data and the model. A model that learns from users needs a filter on what it learns, an eval on what it says, and a human reviewing both, before the learning reaches production. Unfiltered live data is not training data. It is whatever the loudest users decided to teach it. Second, a staged rollout with a kill switch. Tay went to everyone at once, with no canary, no comparison, and no rollback rule. A 10 percent rollout with a human watching the replies would have caught the problem in an hour, with a tenth of the audience. Third, a rollback that is a switch, not a project. Taking Tay offline took hours of decisions. A lifecycle with a registry pointer and a rollback rule takes the model down in minutes. The incident is public history, and the pattern is still live in every system that retrains on user data today. The data moved. The model followed it. Nobody was watching the gate, because there was no gate.
Design the model lifecycle for a fraud-detection model at a payments company. It must score 2,000,000 transactions a day, answer each score in under 200 ms at p99, retrain every week on fresh transaction data, and keep a bad model from ever reaching all users at once. You have a feature store, a model registry, a training pipeline, and a monitoring dashboard. Sketch the lifecycle: the artifact, the gate, the rollout, and the retraining loop.
Start with the traffic. 2,000,000 transactions a day is about 23 scores a second on average. The evening peak is about 5 times the average, so plan for 120 scores a second at peak. Little's Law for the sizing: 120 scores a second at 0.2 seconds in the system is 24 scores inside at once, so I plan the model workers for 30 concurrent, plus headroom to 50, and I alert at 80 percent. The model file is about 800 MB, and the registry keeps the last 5 versions, so the registry holds about 4 GB of model files. The eval gate runs on 20,000 recent labelled transactions and takes about 15 minutes. The canary is 10 percent, which is 200,000 transactions a day, enough to compare fraud-catch rates in two days. The rollout climbs 10 percent, then 25, then 50, then 100, with 6-hour soaks, so full rollout takes about 18 hours. Retraining runs weekly, overnight, for about 4 hours on the training machines, and it never touches the serving path. The storage math is the real sizing. One prediction record is about 2 KB, so 2,000,000 a day is 4 GB a day, and a year is about 1.5 TB, so I keep 90 days hot and archive the rest cold, and I sample the full feature rows one in ten and say so in the runbook.
A weekly retrain buys freshness and costs compute, so I retrain on 90 days, not a year, and archive the rest. A canary buys safety and costs speed on the canary slice, so I start at 10 percent and grow in steps with a latency condition at each one. A strict confidence floor buys safety and costs human reviews, so I set the floor where the review queue stays under 100 a day and watch it. A shared feature store buys consistency and costs coordination, so the feature definitions live in version control and change through review. And the target is p99, not the average. An average of 100 ms with a p99 of 900 ms fails this design, and the p99 is where the bad model lives.
Saved in this browser, so the Lessons index can show your progress across the course.