Lesson 74 taught you to automate ops with Python. This lab is where you build the deploy check: a script that reads a pod table, flags the flapping pods, groups the restarts by namespace, and answers one question with an exit code. Ready means deploy. Not ready means stop. By the end you have flapping.csv with exactly two rows, and a script a pipeline can call.
Difficulty: Advanced · Know this first: Lesson 74 — Python for DevOps; Lab 153; Lab 154
Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: on a cache hit the trip is almost free, on a miss the kitchen eats the budget. Step through the lesson or switch the scenario and watch the bar change.
Extra diagram · from scratch to master
Start with five pods and one bad row. Add one box at a time until the deploy stops itself.
The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the flagging, the grouping, and the total change in the budget bar, in the hop timings, and in the total. The numbers are a teaching example, not live measurements. The shape is the lesson: more pods means more grouping, and a skipped check costs a failed rollout.
How many pods the table holds. Every pod is read, flagged, and grouped, so more pods means a longer grouping hop.
How many pods have 5 or more restarts. More flapping pods means a longer list to explain, and a gate that stays shut longer.
How many deploys run without calling the check. Each skipped check costs about 3 min of failed rollout to clean up.
Try this: (1) Set pods to 5, flapping to 2, skips to 0. The flag is about 1 min, the group about 1 min, the gate about 1 min, and the total is about 8 min. (2) Set all three to the worst end: 50 pods, 20 flapping, 3 skips. The flag is about 5 min, the group about 4 min, the gate about 10 min, and the total is about 24 min. Same script. Only the inputs changed.
Three questions. No score is saved anywhere — this is just you against the material.
Here is the rollout script from this lab, ready to edit. Change it, then run the check. The check looks for the four things a working gate needs: read_csv, a flapping flag at 5 restarts, a groupby, and a ready boolean.
Try breaking it on purpose: change >= 5 to >= 50, or delete the groupby. Run the check and watch it fail. Then press Reset and try a change that keeps it passing, like adding a max restarts print.
Expected output when it passes: The script reads the table, flags flapping at 5 restarts, groups by namespace, and writes flapping.csv with exactly two rows. Rollout ready is False.
I take the same pods four times here. First I read them. Then I flag them. Then I group them. Last I gate on them.
Flags in the next block, letter by letter. I read every flag out loud before I run a command I copied. A flag I cannot explain is a flag I do not run yet.
wc -l pods.csv counts the lines. The -l means lines. The answer should be 6, one header row and five pods. I run it before I trust any answer from the file.python3 rollout.py runs the rollout check. No flags. The file name is the argument. The output prints the mean restarts, the max restarts, and whether the rollout is ready.python3 -c "import pandas as pd; print(pd.read_csv('pods.csv').to_string(index=False))" prints the table from the command line. The -c means command string. The index=False inside is a Python argument, not a shell flag. I use the one liner to peek at a file without writing a script for it.python3 --version prints the version. I run it first on a new box, because the version decides what the code can use.mkdir -p ~/lab-rollout && cd ~/lab-rollout makes the lab folder and moves into it. The -p is the parents flag. The && means run the next command only if this one worked.Check it: run wc -l pods.csv and see 6. Then run python3 rollout.py and read the last line: rollout ready should be False, because two pods are flapping.
Definition: A pod table is a CSV file with one pod per row: name, namespace, restarts, CPU. pandas read_csv turns it into a table. The header row becomes the column names. Print the table before you automate over it.
Types:
Uses: I use this for any new table: read it, print it, count it. Try it now: print pods.csv and point at the pod with 12 restarts.
Dimensions: At low volume everything is instant. What breaks first is the file path: a script run from the wrong folder cannot find its file.
Flags in the next block, letter by letter. I read every flag out loud before I run a command I copied. A flag I cannot explain is a flag I do not run yet.
cat flapping.csv prints the flapping pods the script wrote. No flags. I read it back after writing it, because a file I never reread is a file I never checked.wc -l flapping.csv counts the lines. The -l is the lines flag. The answer should be 3, one header row and two flapping pods. If it is not 3, the filter picked the wrong rows.Check it: run cat flapping.csv and wc -l flapping.csv. The file should hold worker-1 and worker-2, and the count should be 3.
Definition: A flapping pod is one that restarts again and again. The rule in this lab is 5 or more restarts. The flag is a boolean column, so the row stays and the judgement is visible. A threshold is a decision, not a measurement.
Types:
Uses: I use flags when the answer is a list of bad rows, not a single number. Try it now: run rollout.py and check the flapping column is True for exactly two pods.
Dimensions: As the table grows, the flag stays one line. What breaks is a threshold nobody wrote down: a report that says flapping without saying 5 is a number nobody can reproduce.
Flags in the next block, letter by letter. I read every flag out loud before I run a command I copied. A flag I cannot explain is a flag I do not run yet.
python3 rollout.py runs the check again after a change. No flags. The output should be the same every time, because the script reads the same file and does the same math.Check it: run the script twice and compare the outputs. They should match line for line.
Definition: groupby splits the table by namespace and summarises each group with a count, a sum, and a mean. The mean says typical, the max says worst, and the two together are the report. Batch is flapping. Prod is quiet.
Types:
Uses: I use groupby for the summary and flags for the judgement. Try it now: group the lab table by namespace and check batch has 21 restarts.
Dimensions: At ten times the pods, groupby still works, and the time grows with the rows. What breaks is a group with one pod: its mean is just that pod, and it looks like a fact about a group.
Flags in the next block, letter by letter. I read every flag out loud before I run a command I copied. A flag I cannot explain is a flag I do not run yet.
./deploy.sh runs the deploy gate. The ./ means run the file in this directory. The script reads the rollout output and blocks the deploy when the rollout is not ready. The exit code is the gate: 0 opens, 1 shuts.chmod +x deploy.sh && bash -n deploy.sh makes the script executable and checks its syntax. The +x adds the execute bit. The -n on bash means no execution, parse only. Silence from bash means the syntax is clean.grep -q "rollout ready: False" checks for the blocking line without printing anything. The -q means quiet, exit code only. The deploy script uses that exit code to decide whether to block. The quotes around the pattern matter, because the pattern has spaces in it.Check it: run ./deploy.sh; echo $?. The script should print DEPLOY BLOCKED and the exit code should be 1, because the rollout is not ready.
Definition: At this level the check is not a report. It is a gate. Ready is a boolean, len(flapping) == 0, and the deploy reads it. The deploy script calls the check, and it stops when the check says False. A gate that the pipeline calls every time is a gate that cannot be forgotten.
Types:
Uses: I use this before every deploy to a cluster I do not own. Try it now: run ./deploy.sh and check it prints DEPLOY BLOCKED and exits 1.
Dimensions: At a hundred pods, the script is the same shape, and the gate is the same. What breaks is a gate nobody runs: a check that exists but is never called protects nothing.
Flags in the next block, letter by letter. I read every flag out loud before I run a command I copied. A flag I cannot explain is a flag I do not run yet.
wc -l pods.csv counts the lines. The -l is the lines flag. The answer should be 6.python3 rollout.py runs the check. No flags. The last line should say rollout ready is False.bash -n deploy.sh && chmod +x deploy.sh checks the syntax and then makes the script executable. The -n is the parse only flag. The && means run the chmod only if the check passed. I check before I make it executable, because an executable script with a syntax error fails at the worst moment.cat flapping.csv prints the flapping pods. No flags. I read it to check the filter picked the right pods.Check it: run wc -l pods.csv and see 6. Then run the rollout, the deploy gate, and cat flapping.csv. The gate should block, and the file should hold the two flapping pods.
The pod table, all five rows. Do this: save it as pods.csv in your lab folder, then run wc -l pods.csv. Expected output: 6 pods.csv, five pods plus the header.
pod,namespace,restarts,cpu
web-1,prod,1,40
worker-1,batch,12,80
worker-2,batch,9,75
api-1,prod,0,35
db-1,prod,2,60
The rollout script, the whole file. Do this: save it as rollout.py and run python3 rollout.py. Expected output: the flapping column is True for worker-1 and worker-2, the groups read batch 21 restarts and prod 3, the last numbers read mean restarts: 4.8 max restarts: 12, and the last line reads flapping pods: worker-1, worker-2 | rollout ready: False.
import pandas as pd
df = pd.read_csv("pods.csv")
df["flapping"] = df["restarts"] >= 5
print(df.to_string(index=False))
print(df.groupby("namespace")["restarts"].agg(["count", "sum", "mean"]).to_string())
print("mean restarts:", df["restarts"].mean(), "max restarts:", df["restarts"].max())
flap = df[df["flapping"]]
flap.to_csv("flapping.csv", index=False)
print("flapping pods:", ", ".join(flap["pod"]), "| rollout ready:", len(flap) == 0)
The deploy wrapper, the gate itself. Do this: save it as deploy.sh, run chmod +x deploy.sh && bash -n deploy.sh, then ./deploy.sh. Expected output: DEPLOY BLOCKED: rollout not ready, and exit 1. The wrapper runs the check first, and the deploy only runs when the check says ready.
#!/bin/bash
set -euo pipefail
out=$(python3 rollout.py)
if echo "$out" | grep -q "rollout ready: False"; then echo "DEPLOY BLOCKED: rollout not ready"; exit 1; fi
echo "DEPLOY READY"
At 10x the pods, the shape holds and the numbers grow. At 100x, the design changes: the table becomes a live query against the cluster, and the check runs on a schedule instead of on demand. The boolean never changes, though. Ready is still len(flapping) == 0.
A deploy went out on a Friday to a cluster with two flapping pods. The deploy script had a rollout check, but the check was a separate step that someone had to run by hand, and the person deploying that day did not know it existed. The flapping pods took the new version down within minutes, and the rollback took an hour, because the rollback also had to wait for the pods to stop flapping.
Two things. First, the deploy script calls the check, and it stops when the check says False. There is no flag to forget. Second, the check prints the flapping pods by name, so the person deploying knows what to fix, not just that something is wrong. Try it now: run ./deploy.sh and confirm it blocks, then fix the table so no pod is flapping and confirm it would pass.
Build a deploy check for a new pod table you make yourself with ten pods, three namespaces, and three flapping pods. It passes when it flags exactly the flapping pods, writes them to a file, and blocks a deploy wrapper with exit 1.
Success looks like this: the output file holds exactly the three flapping pods, the group mean for one namespace matches your hand calculation, and the wrapper prints DEPLOY BLOCKED and exits 1. If the wrapper deploys anyway, it is not reading the ready value.
If the flag marks too many pods, the threshold is too low. If the group mean looks wrong, check the namespace column for a typo. If the wrapper exits 0 with flapping pods, it is not checking the output. If the script cannot find the file, check the folder.