I got tired of the 2 AM page for a deploy nobody saw coming. The deploy looked fine. The change was small. The dashboard was green. Then the errors started. I wanted a warning before the deploy shipped, not a page after it broke. That is what this lesson builds. A model that reads past deploys and flags the risky ones. It is not magic. It is counting, done carefully. You label the past, you split the data, you train a model, you read four numbers that tell you if the model is any good. The four numbers matter more than the model. A model with a good accuracy and a bad recall is a model that waves every incident through.
Difficulty: Intermediate · Know this first: Lesson 179, Lesson 75
By the end of this lesson you will have trained a random forest on 2,000 past deploys, read its confusion matrix, and proved with a one-line dummy model that accuracy alone lies. Second, you will move the decision threshold and watch precision trade against recall. Third, you will break the model on purpose twice: once by shrinking the forest, once by training on the test set, so you can see what leakage looks like when it is perfect. You will also know where scikit-learn ends and TensorFlow and PyTorch begin.
What you need: Python 3.12 and scikit-learn, installed into a virtual environment with pip install scikit-learn. Nothing in this lesson runs against a live cloud or a live model API. The data is synthetic, the scripts run on your own machine, and the outputs shown are the outputs as they ran.
Every deploy leaves a trail. Lines changed. Tests run. Files touched. Hour of day. Whether it caused an incident. I collect that trail into rows, one row per deploy, and I label each row with what happened next. That table is the whole lesson. Everything after it is plumbing.
Three kinds of learning live in that table. Supervised learning learns from labeled rows: this deploy caused an incident, that one did not. It answers a question with a known answer. Unsupervised learning learns from unlabeled rows: it groups alerts that look alike, or flags the one alert that looks like nothing else. It answers a question nobody wrote down. Ensembles learn by voting: a random forest grows a hundred small decision trees and lets them vote. Neural networks learn by stacking layers that each find a bigger pattern. They are the deep-learning frameworks you will meet next, TensorFlow and PyTorch. This lesson trains the forest. The survey paragraph in Go deeper says where the other two fit.
| Learning type | The question it answers | The ops example | The algorithm you meet here |
|---|---|---|---|
| Supervised | Given the past, what happens next | Will this deploy cause an incident | Logistic regression, random forest, gradient boosting |
| Unsupervised | What groups or odd ones live in this data | Which alerts belong together, which one is strange | Clustering, anomaly detection |
| Ensembles | What do many small models agree on | A hundred trees vote on each deploy | Random forest, gradient boosting |
| Neural networks | What pattern needs layers to see | Images, text, and signals too tangled for trees | TensorFlow and PyTorch, surveyed in Go deeper |
The maths behind the four metric numbers lives in Lesson 179. If precision and recall feel like new words, read that lesson first. The Python in the lab lives in Lesson 75. If the code feels new, start there.
Write down your own four numbers for the last incident you worked. How many alerts fired. How many were real. How many real ones the alert missed. How many pages were noise. That is a confusion matrix, and you already know how to read one. The rest of this lesson only gives the cells their names.
The diagram above follows one nightly run through the pipeline. These four paths are the same idea drawn four ways. One honest run. One leaky run. One confusion matrix, counted out. One rare-incident night. Run each path. One run takes 7.2 seconds across the path, so you can watch each hop light up in turn.
The return rule: the green dot only starts moving after the orange dot reaches the data hop. A return with no request is a lie. The diagram never draws one. The failure replay at the end of Run all four shows the other side of the rule. Split the data after the features are built, and the test rows leak into training, and the packet never comes back.
One nightly run over 20,000 past deploys. Every hop earns its time, and the metrics hop reads the honest verdict: precision 1.000, recall 0.381, F1 0.552.
Not run yet.
Same run, same rows. The ticket number is in the features and the split happened too late. The metrics hop reads 1.000 everywhere, and that is how you know the run is lying.
Not run yet.
The simulator defaults, counted out. The four cells add to 4,000 test rows. Precision is 160 of 236 flagged, 67.8 percent. Recall is 160 of 200 incidents, 80.0 percent. Accuracy is 97.1 percent, and it is the number that lies.
Not run yet.
One night of alerts at the simulator defaults. The model catches 800 of 1,000 real incidents and cries wolf 380 times. The on-call engineer gets 1,180 pages, and 380 of them are noise.
Not run yet.
Every hop charges rent in milliseconds. The bar below is a worked example for one nightly run over 20,000 past deploys, not a measurement from a live system, but the shape is honest. The train hop is the big one. It costs 240 ms on 16,000 training rows with 100 trees. The drift hop is the small one that matters most: 8 ms when the world holds still, 55 ms when a leak meets live traffic. Move a simulator slider and watch the train hop and the total move with it. Switch the scenario and watch the metrics hop or the drift hop change instead.
The budget bar above shows three fixed stories. This simulator lets you write your own. It answers the trade-off at the heart of this lesson: more rows and more trees cost training time, and a rarer incident class costs precision. You set how many past deploys the table holds, how rare the incidents are, how many trees vote, and how much of the data is held out as the test set. Move a slider and watch the confusion matrix, the four metric numbers, and the budget total change. The numbers are a teaching example, built from the formulas below, not live measurements. The shape is the lesson: accuracy stays high while precision falls, and the rare class is the one that decides the verdict.
How many past deploys the table holds. More rows cost training time and make the verdict steadier.
How rare the incidents are. At 2 percent the false alarms outnumber the true catches, and precision falls below half.
How many trees vote. More trees cost training time. They do not fix a rare class or a leak.
How much of the data is locked away as the test set. A bigger test set steadies the verdict and shrinks the training set.
Try this: (1) At the defaults, 20,000 rows, 5 percent incidents, 100 trees, 20 percent test, the train hop is 240 ms and the budget total is 386 ms. The confusion matrix reads 160 true positives, 76 false positives, 40 false negatives, 3,724 true negatives, of 4,000 test rows. Precision is 67.8 percent, recall is 80.0 percent, F1 is 73.4 percent, accuracy is 97.1 percent. (2) Drop to 5,000 rows, 5 percent incidents, 50 trees, 20 percent test. The train hop falls to 30 ms and the budget total to 106 ms. The matrix reads 40 true positives, 19 false positives, 10 false negatives, 931 true negatives, of 1,000 test rows. The rates are the same: precision 67.8 percent, recall 80.0 percent, F1 73.4 percent, accuracy 97.1 percent. Smaller data, same shape. (3) Push to 100,000 rows, 2 percent incidents, 200 trees, 30 percent test. The train hop grows to 2,400 ms and the budget total to 3,026 ms. The matrix reads 480 true positives, 588 false positives, 120 false negatives, 28,812 true negatives, of 30,000 test rows. Precision falls to 44.9 percent, recall holds at 80.0 percent, F1 is 57.6 percent, accuracy is 97.6 percent. The false alarms, 588, now outnumber the true catches, 480, and accuracy still reads 97.6 percent. That is the accuracy trap, priced.
Formulas, in plain terms. Test rows = rows times test share over 100. Positives = test rows times incident rate over 100. The teaching model catches 80 percent of the incidents and falsely flags 2 percent of the safe deploys. True positives = positives times 0.80, rounded. False positives = safe rows times 0.02, rounded. Precision = true positives over true positives plus false positives. Recall = true positives over true positives plus false negatives. F1 = 2 times precision times recall over precision plus recall. Accuracy = true positives plus true negatives over test rows. Label hop = rows over 1,111, rounded. Features hop = rows over 476, rounded. Train hop = rows times trees over 8,333, rounded. Score hop = test rows over 114, rounded. Drift hop = 40 over the incident rate, rounded. Split is 6 ms, metrics 12 ms, ship 25 ms. The budget total is the nine hops added. These are teaching formulas, not a real forest. The lab runs the real one.
Do this now, on your own machine. You will train a random forest on 2,000 past deploys, read its confusion matrix, and prove with a one-line dummy model that accuracy alone lies. The data is synthetic, so you can run it anywhere. The scripts run on this machine and their outputs are shown exactly as they ran. Nothing here calls a cloud or a model API.
Make a virtual environment and install scikit-learn. The install pulls numpy and scipy with it. That is the whole setup.
python3 -m venv ml-venv ml-venv/bin/pip install scikit-learn
Save this file as ml_lab.py. Each row is one past deploy: 8 numbers describing it, and a label that is 1 when that deploy caused an incident and 0 when it did not. Only 5 percent of the rows are incidents, the same shape as real alert data.
"""Classic ML lab: train a random forest that flags risky deploys.
The data is synthetic, so you can run this anywhere. Each row is one
past deploy: 8 numbers describing it (lines changed, tests run, files
touched, hour of day, and 4 more). The label is 1 when that deploy
caused an incident, 0 when it did not. Only 5 percent of the rows are
incidents, the same shape as real alert data: the rare class is the
one that matters.
Run it: python3 ml_lab.py
"""
import numpy as np
from sklearn.datasets import make_classification
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (accuracy_score, confusion_matrix, f1_score,
precision_score, recall_score, roc_auc_score)
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=2000, n_features=8, n_informative=5,
n_redundant=1, weights=[0.95, 0.05],
random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y)
print(f"Train rows: {len(y_train)}, test rows: {len(y_test)}")
print(f"Incidents in train: {int(y_train.sum())} of {len(y_train)}")
print(f"Incidents in test: {int(y_test.sum())} of {len(y_test)}")
forest = RandomForestClassifier(n_estimators=100, random_state=42)
forest.fit(X_train, y_train)
pred = forest.predict(X_test)
proba = forest.predict_proba(X_test)[:, 1]
print("Confusion matrix, rows are the truth, columns are the prediction:")
print(confusion_matrix(y_test, pred))
print(f"Precision: {precision_score(y_test, pred):.3f}")
print(f"Recall: {recall_score(y_test, pred):.3f}")
print(f"F1 score: {f1_score(y_test, pred):.3f}")
print(f"Accuracy: {accuracy_score(y_test, pred):.3f}")
print(f"ROC AUC: {roc_auc_score(y_test, proba):.3f}")
dummy = DummyClassifier(strategy="most_frequent")
dummy.fit(X_train, y_train)
print(f"Dummy, always predict normal, accuracy: "
f"{accuracy_score(y_test, dummy.predict(X_test)):.3f}")
print(f"Dummy, always predict normal, recall: "
f"{recall_score(y_test, dummy.predict(X_test)):.3f}")
Run it:
python3 ml_lab.py
Expected output, exactly as it ran on this machine:
Train rows: 1600, test rows: 400 Incidents in train: 83 of 1600 Incidents in test: 21 of 400 Confusion matrix, rows are the truth, columns are the prediction: [[379 0] [ 13 8]] Precision: 1.000 Recall: 0.381 F1 score: 0.552 Accuracy: 0.968 ROC AUC: 0.930 Dummy, always predict normal, accuracy: 0.948 Dummy, always predict normal, recall: 0.000
Read it in order. The test set holds 21 incidents out of 400 rows. The forest catches 8 of them and never flags a safe deploy, so precision is 1.000 and recall is 0.381. The dummy model predicts normal every time. It scores 0.948 accuracy and catches nothing. That one line is the whole lesson about accuracy: 94.8 percent, and zero incidents caught.
Change n_estimators from 100 to 10 and run it again. The forest is ten trees now instead of a hundred.
python3 ml_lab.py
Expected output, exactly as it ran on this machine with 10 trees:
Train rows: 1600, test rows: 400 Incidents in train: 83 of 1600 Incidents in test: 21 of 400 Confusion matrix, rows are the truth, columns are the prediction: [[379 0] [ 12 9]] Precision: 1.000 Recall: 0.429 F1 score: 0.600 Accuracy: 0.970 ROC AUC: 0.893 Dummy, always predict normal, accuracy: 0.948 Dummy, always predict normal, recall: 0.000
Recall moved from 0.381 to 0.429, and F1 moved from 0.552 to 0.600. Ten trees got lucky on this small test set. That is the point. One split and 400 test rows is a noisy verdict. A metric that moves when the forest shrinks is a metric you have not understood yet. Change it back to 100 when you are done.
Change the fit line so the forest trains on the test set instead of the train set. This is the leak from the leaky scenario, in miniature.
python3 ml_lab.py
Expected output, exactly as it ran on this machine with the leak:
Train rows: 1600, test rows: 400 Incidents in train: 83 of 1600 Incidents in test: 21 of 400 Confusion matrix, rows are the truth, columns are the prediction: [[379 0] [ 0 21]] Precision: 1.000 Recall: 1.000 F1 score: 1.000 Accuracy: 1.000 ROC AUC: 1.000 Dummy, always predict normal, accuracy: 0.948 Dummy, always predict normal, recall: 0.000
Precision 1.000, recall 1.000, F1 1.000, accuracy 1.000, ROC AUC 1.000. Perfect numbers. A perfect score is the leak announcing itself. The model memorised the test rows and reported them back. Change the fit line back when you are done.
Change test_size from 0.2 to 0.5 and run it again. The test set doubles to 1,000 rows. Watch recall move. A metric that moves when the split moves is a metric you have not understood yet. Then change it back to 0.2.
Nothing to destroy. The scripts and the synthetic data live on your machine, and you can run them again on the next model.
Three questions. No score is saved anywhere. This is just you against the material.
Classic machine learning is not one skill. It is four levels of the same job, and every team I have worked with sits on one of them. The levels are not about how many algorithms you can name. They are about whether the verdict is honest. Read them in order. Each level fixes the failure that hurt the most at the level below it.
At the basic level, you train one model on one split and read the four cells. The forest from the lab, the confusion matrix, and the dummy line. The failure that ends this level is accuracy. A model that flags nothing scores 95 percent, and a reader who trusts that number has learned nothing yet.
Do this: run ml_lab.py and write down the four cells and the dummy line. Say in one sentence what the model actually did.
At the intermediate level, the metric is a choice and the threshold is a price. Run threshold.py from Go deeper. At 0.3 the model flags 17 deploys and catches 11 of 21 incidents. At 0.9 it flags 1 and catches 1. Pick the threshold by the cost of a miss against the cost of a false alarm, and write the pick down.
Do this: run the sweep and write one sentence for each threshold: what it costs and what it buys. Then pick one threshold and say why.
At the advanced level, you compare models on the same split instead of trusting one. The forest against gradient boosting against logistic regression. You handle the rare class with class weights and resampling, and you measure what each one costs. You audit every feature with one question: could I know this before the deploy ships. And you split by time, not at random, because a random split lets the future leak into the past.
Do this: add a time-based split to the lab: train on the first ten months, test on the last two. Watch the metrics move. A model that only works on a random split is a model that never met the future.
At the master level, the model is watched like a service. The live feature distribution is compared with training, week by week, and the retrain line is set in advance. The model and the data it trained on are versioned together, so a bad deploy can be traced to the exact model that scored it. A wrong prediction is priced: a missed incident costs a page and a rollback, a false alarm costs an engineer twenty minutes. The model ships when the price is written down, not before.
Do this: write the retrain rule for your own system, one paragraph, with the number that triggers it. If you cannot name the number, you do not have a rule yet.
Supervised learning needs labels, and labels are the expensive part. Someone has to say what happened, and the label join is where most real projects quietly go wrong. Unsupervised learning needs no labels: clustering groups alerts that look alike, anomaly detection flags the one deploy that looks like nothing else. It is the first tool when nobody knows what the groups are. Ensembles win by voting or by fixing: a random forest votes, gradient boosting fixes the last tree's mistakes one tree at a time. Gradient boosting usually wins on tables, and it usually costs more time to tune. Neural networks stack layers, and each layer finds a bigger pattern. They win on images, text, and sound, and they lose to a forest on a small table of numbers.
TensorFlow and PyTorch are the two frameworks you will meet for neural networks. TensorFlow ships with a full serving stack and a long history in production. PyTorch is the framework most research and most new teams pick first, and its training loop reads like plain Python. Neither is taught in full here. They are the next step after this lesson, not a part of it.
One random split is a noisy verdict. The lab's 400 test rows move recall by whole points when ten trees change. Cross-validation splits the data five or ten times, trains on each, and averages the verdict. It costs five or ten times the training time and it is worth it. For ops data, split by time. A random split puts next week's deploys in training and last week's in testing, and the model learns the future. A time split trains on the past and tests on the future, which is what the model will actually do.
The lab's default 0.5 flags 8 deploys and catches 8 of 21 incidents. The sweep below moves it. Save this file as threshold.py. It trains the same forest as the lab and moves the decision threshold from 0.3 to 0.9. Strictly above the threshold, the deploy is flagged risky. Below it, the deploy ships.
"""Move the threshold, move the trade-off.
The same forest as ml_lab.py, but the decision threshold moves.
The model outputs a probability. Strictly above the threshold, the deploy is
flagged risky. Below it, the deploy ships. Every threshold is a
different deal between catching incidents and crying wolf.
Run it: python3 threshold.py
"""
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import f1_score, precision_score, recall_score
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=2000, n_features=8, n_informative=5,
n_redundant=1, weights=[0.95, 0.05],
random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y)
forest = RandomForestClassifier(n_estimators=100, random_state=42)
forest.fit(X_train, y_train)
proba = forest.predict_proba(X_test)[:, 1]
for t in [0.3, 0.5, 0.7, 0.9]:
pred = (proba > t).astype(int)
flagged = int(pred.sum())
print(f"Threshold {t}: flagged {flagged} of {len(y_test)}, "
f"precision {precision_score(y_test, pred):.3f}, "
f"recall {recall_score(y_test, pred):.3f}, "
f"F1 {f1_score(y_test, pred):.3f}")
Run it:
python3 threshold.py
Expected output, exactly as it ran on this machine:
Threshold 0.3: flagged 17 of 400, precision 0.647, recall 0.524, F1 0.579 Threshold 0.5: flagged 8 of 400, precision 1.000, recall 0.381, F1 0.552 Threshold 0.7: flagged 5 of 400, precision 1.000, recall 0.238, F1 0.385 Threshold 0.9: flagged 1 of 400, precision 1.000, recall 0.048, F1 0.091
At 0.3 the model catches 11 of 21 and cries wolf 6 times. At 0.9 it catches 1 and never cries wolf. The best F1 here is 0.579 at 0.3, but F1 is not the price. The price is what a missed incident costs your team against what a false alarm costs. That price picks the threshold. The default 0.5 is a guess, and this lesson is about replacing guesses with prices.
Precision, recall, F1, ROC, and AUC as classifier metrics live here. Lesson 85, LLM Evaluation and Benchmarks, owns the same words as LLM-eval metrics: assertions, exact match, Pass at k, and benchmark scores for language models. Same vocabulary, different job. A classifier metric scores a yes or no decision. An LLM-eval metric scores generated text. Do not carry one job's numbers into the other.
At 10x the rows, training stops being a script you run by hand and becomes a pipeline: a feature store holds the eight numbers, the training job runs on a schedule, the model lands in a registry with a version. Lesson 93, The Model Lifecycle, teaches the registry and the pipeline. At 100x, the cost moves to the features, not the model. Feature computation on a million rows is the slow hop, and the fix is to compute features once and reuse them. Lesson 76, MLOps, owns the deployment half: serving, monitoring, and the retrain trigger. This lesson owns the first model. The pipeline owns the tenth.
A team shipped a deploy-risk model with a test accuracy of 99.8 percent. The first week it flagged nothing, and three deploys caused incidents anyway. The second week it flagged a deploy that was fine, and the team stopped looking at its scores. The model was right on paper and useless in production.
Two rules. First, the feature audit: every feature must answer one question, could I know this before the deploy ships. A feature that fails the question is deleted, not fixed. Second, the time split: training data ends one month before the test data starts, so the future cannot leak into the past. The rebuilt model scored 93 percent accuracy and a recall of 0.38, and the team trusted it, because the numbers were honest. A model with honest numbers beats a model with perfect ones.
Design a classifier that flags risky deploys before they ship. You have 20,000 past deploys. Five percent caused incidents. The on-call team will look at every flagged deploy, and they will ignore the model if it cries wolf more than half the time. The model must run in the deploy pipeline and answer in under a second.
Split by time, not at random. Train on the first ten months, test on the last two, so the verdict measures the future. Start with the dummy: always predict safe. It scores 95 percent accuracy and catches nothing, and that is the floor every real model must beat. Train logistic regression first, because its weights say which features matter. Then train the forest.
Then the threshold. Read the confusion matrix at 0.5, then sweep the threshold from 0.3 to 0.9. The on-call rule is precision above 0.5, so the threshold must keep false alarms under half the flags. At the lab's numbers, 0.3 gives precision 0.647 and recall 0.524, which clears the bar and catches 11 of 21 incidents. Ship that threshold, with the number written down beside the model.
Then the watch. Compare the live feature averages with the training averages every week, and retrain on fresh data when the drift line is crossed. Size it: 20,000 rows and 100 trees train in 240 ms in the teaching model, and scoring one deploy is under a millisecond. The pipeline budget is the feature computation, not the model.
Trade-offs to name. Precision against recall is the first trade, and the on-call rule picks the side. A simple model against a strong one is the second: logistic regression explains itself, the forest wins on F1, gradient boosting wins a little more and costs tuning time. A threshold that catches more against a threshold the team trusts is the third: a model that flags everything gets ignored, and an ignored model catches nothing.
Saved in this browser, so the Lessons index can show your progress across the course.