Here is the plain version first. Training a model means teaching it by example. You show it a pile of examples of the job done right, it guesses an answer for each one, and it adjusts itself a little every time it guesses wrong. Do that thousands of times and it starts getting the job right on examples it has never seen. That is all training is. Here is the concrete example I will use all the way through. A support site gets tickets all day. Each ticket needs to go to the right team: billing, login problems, or delivery. A person used to read every ticket and pick the team. We trained a small model to do the picking instead. Our examples were old tickets with the team written on each one. The ticket text is the question. The team name is the answer. The model reads the text, guesses a team, and learns from its mistakes. I have debugged training runs the same way I debug a deploy at 2 AM. I look at one step at a time, I check what went in, and I trust the numbers I can measure over the story I want to believe. This lesson is that whole trip: where the examples come from, how they get cleaned and split, what happens inside one training step, and how you know when the model has learned enough or learned too much.
Difficulty: Advanced · Know this first: Lesson 76, Lesson 64
Every step charges rent in minutes, not milliseconds. The bar below is a worked example for one small training run: 50,000 labelled tickets, three epochs, one local GPU box. It is not a measurement from a live system, and your numbers will differ. But the shape is honest. The people work, labelling and cleaning, costs more time than the GPU work. The training loop is the middle of the bar, not the whole bar. Step through the lesson or switch the scenario and watch the bar change. The overfit run adds its time in extra epochs that make the model worse. The leaky run is faster than the clean run, and the time it saved is the time the team should have spent on the split.
A second picture of the same system. I start with one box and build to the full Master picture. The dot is one batch of tickets.
Legend: orange dot = one batch of tickets · hexagon = a service or model · shaded box = a store, log, or file · dashed line = the wire between boxes. Progression: Starts from scratch with one ticket, one guess and one loss number — no training vocabulary assumed. Shows the loop going round once, then labels the repetition (thousands of batches, three epochs). Ends at the Master-level picture: loss curves read for overfitting, regularization applied on purpose, and training run as a service. · Traffic: orange = request forward, green = response back.
A second picture of the same system. I start with one box and build to the full Master picture. The dot is one ticket.
Legend: orange dot = one ticket · hexagon = a service or model · shaded box = a store, log, or file · dashed line = the wire between boxes. Progression: Starts from scratch with one raw ticket and one label written on it. Adds cleaning, then the three-way split with the test set visibly locked away. Ends at the Master-level picture: leakage hunted wherever it hides, and labeling — not the GPU — named as the bottleneck at scale. · Traffic: orange = request forward, green = response back.
The budget above is one fixed run. This box is yours. Add more examples, run more epochs, change the batch size, let more bad labels in, and watch the training time, the total time, and the held-back accuracy move. Then ask the question I ask before every training run: what am I paying for, and what is it buying me?
How many labelled examples the model trains on. More examples cost more labelling and cleaning time, and they make the model harder to memorize its way through.
How many full passes through the data. One epoch means each example seen once. More epochs mean more training time and more chances to memorize.
How many examples the model sees before one optimizer update. A bigger batch means fewer updates per epoch. Each update is steadier, but there are fewer of them.
What share of the labels are wrong. Five percent is a normal dirty dataset. Twenty percent means one label in five teaches the wrong answer.
Try this: (1) Set epochs to 20 at the defaults. The risk goes to High and the accuracy falls, and the checkpoints alone cost 800 minutes. (2) Now set examples to 200,000 and epochs back to 3. The risk is Low again and the accuracy holds at 88 percent, but the total is 1,795 minutes. More data is the honest fix, and it is not free. (3) Now set bad labels to 20 percent. The total never moves, but the accuracy falls to 70 percent. Same run, same time, worse model. The labels are the cheapest lever in this lesson.
Three questions. No score is saved anywhere — this is just you against the material.
The same topic four times, from the first plain idea to the level where I run it in production and debug it at 2 AM. Each level adds one thing the level below cannot do.
Definition: Training a model means teaching it by example. You show it a pile of examples of the job done right, it guesses an answer for each one, and it adjusts itself a little every time it guesses wrong. Do that thousands of times and it starts getting the job right on examples it has never seen. That is all training is.
Types:
Uses: Use it the first time you train anything. The ticket router from this lesson is the example. Old tickets with the team written on each one: the ticket text is the question, the team name is the answer. The model reads the text, guesses a team, and learns from its mistakes.
Dimensions: At low scale a few thousand examples and three epochs on a laptop is fine. At 10x the data, labelling is the bottleneck, not the GPU. What breaks is forgetting to clear the gradients: they pile up and the model jumps around. A classic bug, one line. It costs the GPU hours, the labelled examples, and the human time to label them. You give up hand-written rules for a model that learns from examples. The rules are exact, the model is approximate, and the examples decide which one wins.
Definition: The loop is one training step repeated: forward pass, loss, backward pass, optimizer step, clear the gradients. Around it sit the curves. Training loss is how wrong the model is on data it has seen. Validation loss is how wrong it is on data it has not seen. Both should fall together early.
Types:
Uses: Use it to judge a run. Both losses level off together, and the run is done. Training loss keeps falling while validation loss climbs, and the model is overfitting. Neither falls, and the learning rate is wrong or the data is inconsistent.
Dimensions: At 10x the users the run has to be reproducible: dataset version, split, learning rate, batch size, checkpoint hash, all logged. What breaks is a validation loss that jumps up and down between epochs. That usually means the validation set is too small, so one hard batch swings the average. A bigger validation set steadies the curve. It costs the validation check once per epoch, not once per batch, because running it more often costs time and tells me little extra. You give up judging a run by one number at the end for judging it by the shape of two curves across the whole run.
Definition: Overfitting is the model memorizing instead of learning. Leakage is test information sneaking into training. Regularization is anything that stops the model from memorizing: dropout, weight decay, early stopping.
Types:
Uses: Use it when the model aces training and fails on new data. The sneaky leak is future information: a field that only exists after the answer is known, like the name of the engineer who fixed the ticket, which quietly tells the model the team. And the quiet one is preprocessing: if I compute averages or vocabularies on the whole dataset before I split it, the training set has already seen the test set's statistics.
Dimensions: At 10x the data, labelling gets automated: rules for the easy tickets, a model that pre-labels and a human who corrects it, with a written rule for every disagreement. What breaks is a duplicate ticket in both sets, which makes the score a lie, or one user in both sets, which means the model may be recognizing the user, not learning the job. It costs the labelling pipeline, the human sample, and the re-runs. You give up training on everything for training on a clean split. More data beats a bigger model, up to the point where the new data stops adding new patterns.
Training runs for days, and a crash on day three must not cost three days. I save a checkpoint every hour, with the weights, the optimizer state, and the data position. A resume loads all three. Weights alone are not a resume. Without the optimizer state, the training restarts the learning rate schedule and the loss jumps.
I test the resume before I need it. I kill a short training run on purpose, resume it, and check the loss continues from where it stopped. A resume path that has never been tested will fail on the night it matters.
Definition: At 100x, training is a service with versions, and the data is versioned like code. A dataset change goes through review the same way a code change does, because a dataset change is a code change that nobody can read. The order I try things is: more clean data first, then more epochs with early stopping, then a bigger model. The expensive lever goes last.
Types:
Uses: Use it when the team asks for a bigger model. I have seen a small model on 200,000 clean examples beat a big model on 20,000 examples on every test that mattered. The bigger model has more room to learn patterns, and also more room to memorize.
Dimensions: At 100x the training pipeline, the labelling pipeline, and the dataset versions are each their own system with their own owner. What breaks is ten thousand more tickets that all say the same thing in the same words: they teach nothing new, and at that point the lever is a bigger model or better features. It costs the labelling, the GPU hours, and the versioning. You give up the excitement of the big model for the boring order: clean data first, early stopping second, size last.
The order of the data changes the result. Sorted data teaches the model one thing at a time, and it forgets the first thing by the end. Shuffled data mixes everything, and the model learns all of it. I shuffle with a fixed seed, and I log the seed, so the run is reproducible.
A seed that is not logged is a run that cannot be repeated. When a run goes wrong, the first question is whether the data order changed, and the log must answer it.
The lesson above is one small run from tickets to a shipped model. Production is the rest of it. Here is what I actually spend my time on once the first model ships and the second run starts.
Two curves, one chart. Training loss is how wrong the model is on data it has seen. Validation loss is how wrong it is on data it has not seen. Both should fall together early. Then three endings are possible. Both level off together, and the run is done. Training loss keeps falling while validation loss climbs, and the model is overfitting. Neither falls, and the learning rate is wrong or the data is inconsistent. I also watch for a validation loss that jumps up and down between epochs. That usually means the validation set is too small, so one hard batch swings the average. A bigger validation set steadies the curve. I never judge a run by one number at the end. I judge it by the shape of the two curves across the whole run.
Leakage is test information sneaking into training, and it hides in places that look innocent. Duplicates are the common one: the same ticket in both sets. Future information is the sneaky one: a field that only exists after the answer is known, like the name of the engineer who fixed the ticket, which quietly tells the model the team. Preprocessing is the quiet one: if I compute averages or vocabularies on the whole dataset before I split it, the training set has already seen the test set's statistics. The rule that catches all three is simple. Split first, then clean and compute everything separately per split. And split by the real unit. For tickets, the unit is the ticket thread, not the row. For users, the unit is the user, not the session. If one user appears in both sets, the model may be recognizing the user, not learning the job.
A bigger model has more room to learn patterns, and also more room to memorize. On a small dataset, a big model memorizes faster and fails harder on new data. More data helps first, because it gives the model more patterns to learn and fewer chances to memorize any single one. I have seen a small model on 200,000 clean examples beat a big model on 20,000 examples on every test that mattered. The point where that stops is when the data stops adding new patterns. Ten thousand more tickets that all say the same thing in the same words teach nothing new. At that point, a bigger model, or better features, is the next lever. The order I try things is: more clean data first, then more epochs with early stopping, then a bigger model. The expensive lever goes last.
Regularization is anything that stops the model from memorizing. Dropout means the model randomly ignores a little of itself while training, so it cannot lean on one memorized shortcut. Weight decay means the model's numbers are gently pulled toward zero, so no single number grows huge to memorize one example. Early stopping means I watch the validation loss and stop training when it stops improving, even if epochs are left. I use early stopping first, because it costs nothing and it uses the checkpoint I already save. Dropout and weight decay come next, when early stopping alone is not enough. All three do the same job in different ways: they make memorizing harder and learning easier.
At 10x the data, labelling is the bottleneck, not the GPU. I automate what I can: rules for the easy tickets, a model that pre-labels and a human who corrects it, and a written rule for every disagreement. A human still reads a random sample, but the sample is random and the rules are written down. At 10x the users, the training run has to be reproducible. Every run logs the dataset version, the split, the learning rate, the batch size, and the checkpoint hash, so I can answer a year later which data made this model. At 100x, training is a service with versions, and the data is versioned like code. A dataset change goes through review the same way a code change does, because a dataset change is a code change that nobody can read.
This is the loop from this lesson, written as plain Python-style pseudo-code. It is generic, not tied to one vendor's library. The ideas are the same everywhere.
# training loop: ticket router, 3 epochs, batch size 64
model = SmallClassifier(num_teams=3)
optimizer = Adam(model.parameters(), lr=0.001)
best_val_loss = float("inf")
for epoch in range(3):
model.train()
for batch in train_loader: # 64 tickets at a time
predictions = model(batch.text) # forward pass
loss = loss_fn(predictions, batch.team) # how wrong was it
loss.backward() # backward pass
optimizer.step() # small nudge
optimizer.zero_grad() # clear for next batch
val_loss = evaluate(model, val_loader) # held-back check
save_checkpoint(model, optimizer, epoch) # so a crash costs an hour
if val_loss < best_val_loss:
best_val_loss = val_loss
save_checkpoint(model, optimizer, epoch, tag="best")
# the test set is touched once, at the very end
test_accuracy = evaluate(model, test_loader)
What the snippet means: the inner loop is one training step. Forward pass, loss, backward pass, optimizer step, then clear the gradients so the next batch starts clean. Forgetting the clear step is a classic bug: the gradients pile up and the model jumps around. The validation check runs once per epoch, not once per batch, because running it more often costs time and tells me little extra. The best checkpoint is saved by validation loss, not training loss. The test set appears once, at the end, because the moment it is used to make a decision, it stops being a test.
Twelve loss values from a training run. The checker looks for two things: a spike, where the loss jumps more than 20 percent in one step, and a plateau, where the last five steps barely move.
Run it with python3 read_the_loss_curve.py. Then change step 5 from 2.86 to 1.47 and run it again. The spike must disappear and only the plateau must remain.
python3 read_the_loss_curve.py has two parts and no flags. python3 is the program. It is the Python 3 interpreter, the thing that reads the file and runs it. read_the_loss_curve.py is the file to run. It is a plain argument, not an option. Python reads it top to bottom and runs it. Leave the filename off and Python waits for you to type code by hand. That is the interactive prompt, not a run.
Three options I reach for all the time. --version prints the Python version and stops. I run it first on a new machine, so I know what I am running against before I trust the output. -u forces unbuffered output. Prints show up the moment they happen instead of sitting in a buffer. That matters when I pipe output into a log file and watch it live. Without it a crashed run can look like it printed nothing. -m runs a library module as a script. python3 -m json.tool runs the JSON tool that ships with Python, with no file of my own. Try them now.
python3 --version python3 -u read_the_loss_curve.py python3 -m json.tool --help
Run the script once plain and once with -u. The output is the same here. The script is short and finishes fast. The difference shows up on a long run. With -u each line appears as it is printed. Without it the lines arrive in a burst at the end. That is the check. If both runs print the same lines, the flag did its job.
def read_the_loss_curve():
losses = [2.31, 1.94, 1.71, 1.58, 1.49, 2.86, 1.44, 1.40, 1.40, 1.39, 1.39, 1.39]
for i in range(1, len(losses)):
if losses[i] > losses[i - 1] * 1.20:
print(f"SPIKE at step {i}: {losses[i-1]} -> {losses[i]} (check the batch, then the LR)")
tail = losses[-5:]
if max(tail) - min(tail) < 0.02:
print(f"PLATEAU in last 5 steps ({tail[0]} -> {tail[-1]}): stop or change something")
assert losses[5] > losses[4] * 1.20
assert max(tail) - min(tail) < 0.02
read_the_loss_curve()
SPIKE at step 5: 1.49 to 2.86. The runbook answer is to check the batch first, then the learning rate. PLATEAU in the last 5 steps, 1.4 to 1.39. The run has stopped learning, so I stop the run or change something. A loss curve is a log, and I read it the same way.
This one stayed with me because the eval said we were great and the users said we were wrong. Both were right, in their own way. The eval was measuring the wrong thing.
We shipped a ticket router that scored 96 percent on our test set. On Monday, tickets started landing on the wrong teams. Billing tickets went to delivery. Login tickets went to billing. The model was serving fine. The GPU was fine. Latency was normal. The test score was the lie, and we had believed it. The router had learned to read the name of the engineer who handled the ticket, because that field was still in the training data. The engineer name told the model the team, so it never learned to read the ticket text at all. In production, new engineers and reassigned tickets broke the shortcut, and the model guessed almost at random.
Three things. First, the training data now goes through a field review before any run. Every field is listed, and someone has to say out loud whether it would exist at prediction time. A field that only exists after the answer is known is dropped. Second, the test set is built by a different person than the training data, and a script checks overlap between the two before any eval runs. If a ticket appears in both, the run stops. Third, every ship now includes a weekend of real tickets graded by a human before the canary grows past five percent. The test score is a signal. The human read is the decision.
This is the kind of question I would ask in an interview, and the kind I ask myself before I sign off on a real training run. Take a minute with it before you read my approach.
Design training for an alert triage model. It reads incoming alerts and picks the team that owns the problem: network, database, or app. You have 100,000 past alerts with the owning team written on each one. The alerts arrive 24 hours a day, the team names change when teams reorganize, and the model must never route a database outage to the wrong team for an hour. Sketch the data, the training loop, the eval, and the ship plan.
Start with the simulator math from this lesson. 100,000 raw alerts, and I assume a fifth are duplicates, test pages, or one-line alerts with no teaching value. That leaves about 80,000 good examples. I hold back 8,000 for validation and 8,000 for the test, so about 64,000 train the model. I use the lesson rates: labelling is already done, because the owning team is written on each alert, so the people work is cleaning, about 64,000 times 0.0018 minutes, which is about 115 minutes. I run 3 epochs at batch size 64. The training loop is 3 times 45 times 64,000 over 50,000, which is about 173 minutes. Checkpoints add 3 times 40, which is 120 minutes. The total run is about 115 plus 15 plus 173 plus 120 plus 40, which is about 463 minutes, or just under 8 hours. I pick a small model, not a big one, because the job is three teams and the data is plentiful. I split by time, so the test is the newest alerts. The ship plan is a canary to one shift first, then the whole on-call rotation. The guardrail is a rule, not a hope: if confidence is under 0.6, the alert goes to a human instead of the model. A wrong route on a database outage costs an hour. A human page costs a minute.
More data buys a better model and costs cleaning time, and a fifth of this dataset is junk, so the cleaning step earns its keep. Three epochs buy learning and cost memorization risk, so I stop at the best checkpoint by validation loss. A small model buys cheap serving and fast retraining, and costs some accuracy a big model might have, but the gap is small on a three-team job. Splitting by time buys an honest test and costs some training examples from the newest weeks. Routing low-confidence alerts to a human buys safety and costs some good answers that a human has to read. And the target is not a score. It is a month of pages where no database outage sat with the wrong team for an hour. The eval measures the proxy. The on-call month measures the thing.
Saved in this browser, so the Lessons index can show your progress across the course.