Python for DevOps means using Python to do the small jobs that keep a system running, instead of doing them by hand. I mean jobs like checking whether a deploy finished, restarting a stuck service, or reading a config file and comparing it with what is actually live. Python exists in this role because it is quick to write, easy to read at 2 AM, and it has a library for almost every API I need to talk to. Here is one concrete example. A deploy finishes and I want to know if the new version is really serving traffic. I run one Python script. It asks Kubernetes which pods are running, it checks that each pod says ready, and it prints one line: ready, or not ready. That is the whole lesson in one script. Nine hops, from typing the command to a printed answer I can trust.
Every hop charges rent in seconds. The bar below is a worked example for one script run, not a measurement from a live system, but the shape is honest: on a clean run, waiting for the rollout costs more than everything else put together. The patch itself is 2 seconds. The wait is 6. Step through the lesson or switch the scenario and watch the bar change. The stuck rollout spends 45 seconds waiting and still fails. The run where nothing needed changing finishes in under 6 seconds, because the two most expensive hops were skipped.
The budget above is one fixed run. This box is yours. Add pods, turn the API flaky, change the wait per pod, and watch the total move. Then ask the question I ask before I hand a script to a teammate: how long does this take on a bad day, and what does the second run cost?
List time is 0.3 seconds plus 0.1 seconds per pod. Wait time is pods times wait per pod. At 12 pods the list is 1.5 seconds and the wait is 6.0 seconds.
Wait time is pods times wait per pod. At 12 pods and 0.5 seconds each, the wait is 6.0 seconds.
Retry cost is retries used times 2 seconds: 1 second of backoff plus 1 second for the call. It only costs time when the API flakes toggle is on.
Off means the patch succeeds on the first try and retries cost nothing. On means the patch fails and retries until the retry budget is spent, then the run fails at the patch hop.
Try this: (1) Set pods to 50 and watch the wait hop grow from 6.0 seconds to 25.0 seconds at the default wait per pod. (2) Turn API flakes on with max retries at 5. The patch hop costs 12.0 seconds and still fails. (3) Turn API flakes off again and set max retries to 0. The total drops back to 13.3 seconds, because retries only cost time when something actually fails.
Three questions. No score is saved anywhere — this is just you against the material.
Same topic, four times. First one script doing one job, then talking to APIs properly, then retries, async, and CLIs other people can run, then the level where the automation is a small service a dozen teammates depend on.
Definition: Python for DevOps means using Python to do the small jobs that keep a system running, instead of doing them by hand. I mean jobs like checking whether a deploy finished, restarting a stuck service, or reading a config file and comparing it with what is actually live. Python exists in this role because it is quick to write, easy to read at 2 AM, and it has a library for almost every API I need to talk to.
Types:
kubectl, and read what it prints.Uses: A deploy finishes and I want to know if the new version is really serving traffic. I run one Python script. It asks Kubernetes which pods are running, it checks that each pod says ready, and it prints one line: ready, or not ready. I test it on my laptop and I run it in the pipeline, because on my laptop it is fast to test and impossible to audit, and in the pipeline it is reviewed, versioned, and logged.
Dimensions: With 12 pods, one list call and one health check finish in seconds, and the wait is most of the run. As pods grow, the list call gets slower and the wait gets longer. What breaks is a script that exits zero on failure: the pipeline sees success, the dashboard stays green, and the deploy never happened. I notice it when the change is not live and everything says it is. What it costs is minutes to write and the discipline of checking the exit code in the pipeline. The trade is speed against audit. A script on my laptop is fast. A script in the pipeline is provable.
Definition: Calling kubectl from Python with subprocess is the fastest way to start. The command already exists, I know its output, and I can paste the same command into a terminal to check it. The price is parsing text. Text output changes between versions, and a script that parses text breaks quietly when the format moves. An API client costs more up front. I have to learn the client library and handle its objects. The price is paid once, and after that the script reads fields, not text.
Types:
Uses: I start with subprocess for a one-off job and move to the Kubernetes Python client the moment a teammate has to run the script. I read timeouts, retry counts, and the expected image tag from a YAML file with PyYAML, so a teammate who needs a different timeout edits the file, not the code. I log every event as one JSON line with a run id, so when ten runs overlap I can still follow one run through the pile. On call, when a script reports 100 pods ready, I ask whether it followed the pagination token, because pod 101 lives on the next page, and a script that never asks for it misses the 12 that are crashing.
Dimensions: At ten times the pods, I ask for only the fields I need instead of the whole pod object, and I watch one deployment at a time instead of listing everything in the namespace. What breaks is text parsing when the format moves, a token printed in a debug log line that lives longer than the token should, or an API version change a year later that turns a warning into an error. I notice the first as a script that worked yesterday and fails today with no code change, and the second only when I audit what the logs hold. What it costs is the upfront learning of one client library and the log volume of structured lines. The trade is a price paid once against a break that arrives quietly. I pin the client version and I read the deprecation warnings, because a warning today is an error after the next upgrade.
Definition: A retry policy is a promise about how the script behaves when the world says no. Idempotency is the promise that running the same action twice has the same result as running it once. Async automation is how the script waits on many API calls at once instead of one at a time. And a CLI is what the script becomes the second time someone else needs to run it: a tool with a front door, not a file on my laptop.
Types:
Uses: The rollout check retries the read, not the write, because retrying a read is safe and retrying a write can double-apply a change. The compare comes before the patch, and that order is the whole point: if the live image already matches, the script does nothing and says so. I package the script with click or typer the second time a teammate runs it, with a dry-run flag, so they can see what it would do before it does it. I write pytest tests for the compare logic, because a failed read treated as a difference is how a deployment restarts three times in one hour. On call, when a script retries five times in five seconds with no backoff, I know it was written before anyone read the retry policy.
Dimensions: At a hundred times the pods, five retries with a long backoff can outlast the timeout of the thing that called the script, so the retry budget is in time, not count. What breaks is a write retried without idempotency, an async script with no timeouts, or a CLI that only works on one machine because the dependencies were never pinned or packaged. I notice the first as a change applied twice, the second as a run that never returns, and the third as a ModuleNotFoundError in the pipeline. What it costs is test maintenance, packaging time, and the discipline of a dry-run flag on every destructive command. The trade is robustness against the speed of a one-off script. The one-off is faster to write. The tested CLI is faster to trust.
Definition: At a hundred times the pods, the script stops being a script and becomes a small service. It runs on a schedule, it keeps its own state between runs, and it writes its results to a dashboard instead of printing them. A dozen teammates depend on it, and nobody remembers what it does without reading the tests first.
Types:
Uses: A failed read now stops the run. The script exits with an error that says it could not read state, and it never patches on an unknown. The compare reads the image digest from the live pods, not a cached tag, so a stale cache can no longer invent a difference. The script checks the patch response and the exit code, and a failed patch exits nonzero, so the scheduler records a failure instead of a success. In a design review, the question I ask is who owns the automation when it breaks, because by now the automation is a service, and services need owners.
Dimensions: Every run is logged with a run id, and the dashboard shows the fleet, not one terminal. What breaks is the API version changing under a script nobody has touched in a year, a secret in the log, or a silent success that hides a failed deploy. I notice the first from a deprecation warning I read in time, and the last only when a user reports the change never landed. What it costs is on-call for the automation itself, review process on every change, and test suites that take real time to keep green. The trade is automation I trust against automation I babysit. The schedule stayed at fifteen minutes. The script got slower to fail, and that was the fix.
The lesson above is one script doing one job. Production is the rest of it. Here is what I actually spend my time on once the first script works.
Subprocess versus an API client. Calling kubectl from Python with subprocess is the fastest way to start. The command already exists, I know its output, and I can paste the same command into a terminal to check it. The price is parsing text. Text output changes between versions, and a script that parses text breaks quietly when the format moves. An API client costs more up front. I have to learn the client library and handle its objects. The price is paid once, and after that the script reads fields, not text. I start with subprocess for a one-off job and move to the API client the moment a teammate has to run the script. The second trade is where the script runs. On my laptop it is fast to test and impossible to audit. In the pipeline it is reviewed, versioned, and logged, and it is slower to change. I test on the laptop and I run in the pipeline. The third trade is the language itself. Python is fast to write and slow to run. For a script that waits on APIs, that speed does not matter, because the waiting is the work. For a script that crunches a million log lines, I would reach for something else or move the crunching into a library written in a faster language.
The quietest failure is a script that exits zero on failure. The pipeline sees success, the dashboard stays green, and the deploy never happened. I check the exit code in the pipeline and I fail the pipeline on purpose when I test the script, just to prove the failure path works. The second failure is a secret in the log. A token printed in a debug line ends up in the pipeline logs, and the logs live longer than the token should. I never log tokens, and I redact any field that looks like a secret before the log line is written. The third failure is a script that works until the API paginates. The first 100 pods come back fine. Pod 101 is on the next page, and the script never asks for it, so it reports 100 pods ready and misses the 12 that are crashing. I always follow the pagination token until the API says there is no next page. The fourth failure is a script that runs fine for a year and then breaks when the API version changes. I pin the client version and I read the deprecation warnings, because a warning today is an error after the next upgrade.
At ten times the pods, the list call gets slower and the wait gets longer, and the script starts to feel it. I ask for only the fields I need instead of the whole pod object, and I watch one deployment at a time instead of listing everything in the namespace. The log line grows a run id, so I can follow one run through the logs when ten runs overlap. And the script gets a dry-run flag, because at ten times the scale nobody wants to test a change by running it for real.
At a hundred times the pods, the script stops being a script and becomes a small service. It runs on a schedule, it keeps its own state between runs, and it writes its results to a dashboard instead of printing them. The retry policy gets a budget in time, not just in count, because five retries with a long backoff can outlast the timeout of the thing that called it. And the script gets tests, because by then a dozen teammates depend on it and nobody remembers what it does without reading the tests first.
import json
import logging
import time
# Structured logging: one JSON line per event, with a run id.
# I can search these lines. I cannot search a paragraph.
logging.basicConfig(level=logging.INFO, format="%(message)s")
log = logging.getLogger("rollout")
def run_id():
return f"run-{int(time.time())}"
def check_rollout(client, service, want_image, max_retries=2):
rid = run_id()
for attempt in range(max_retries + 1):
try:
live = client.get_deployment(service)
except Exception as e:
# Backoff grows: 1s, then 2s, then 4s. Jitter left out
# here for clarity; in production I add a random bit
# so ten scripts do not retry at the same second.
log.info(json.dumps({"run": rid, "event": "api_failed",
"attempt": attempt, "error": str(e)}))
time.sleep(2 ** attempt)
continue
# Idempotency: compare first, patch only on a real difference.
if live["image"] == want_image:
log.info(json.dumps({"run": rid, "event": "noop",
"service": service}))
return True
client.patch_image(service, want_image)
return wait_ready(client, service)
log.info(json.dumps({"run": rid, "event": "gave_up",
"service": service}))
return False
The compare comes before the patch, and that order is the whole point. The loop retries the read, not the write, because retrying a read is safe and retrying a write can double-apply a change. The backoff doubles each time, 1 second, then 2, then 4, so a short blip clears fast and a real outage does not get hammered. The log lines are JSON with a run id, so I can follow one run through a pile of logs. And the function returns False when it gives up, so the caller can exit nonzero and the pipeline knows the run failed.
This one stayed with me because the script did exactly what I wrote. The problem was what I wrote.
A rollout helper script ran from a scheduled job every fifteen minutes. Its job was simple: check whether the deployment was on the wanted image, and patch it if it was not. One night the image registry had a short outage. The script read the deployment, the read failed, and the script treated a failed read as "image does not match" instead of "I do not know". So it patched. The patch failed too, because the registry was down, and the script retried the patch five times with no backoff. Each retry hit the API server in the same second. Then the registry came back, the next scheduled run patched the deployment, the new pods started, and the readiness check passed. Fifteen minutes later the next run patched it again, because the compare read a cached tag that had not updated yet. The deployment restarted three times in one hour. Users saw brief errors each time a restart rolled through. The dashboards stayed green, because each restart was short and the health checks recovered between them.
Four things. First, a failed read now stops the run. The script exits with an error that says "could not read state", and it never patches on an unknown. Second, the retry got a backoff with jitter, and a retry budget in time, not just in count. Third, the script checks the patch response and the exit code, and a failed patch exits nonzero, so the scheduler records a failure instead of a success. Fourth, the compare reads the image digest from the live pods, not the cached tag, so a stale cache can no longer invent a difference. The schedule stayed at fifteen minutes. The script got slower to fail, and that was the fix.
This is the kind of question I would ask in an interview, and the kind I ask myself before I hand a script to a teammate. Take a minute with it before you read my approach.
Design a small command-line tool called deploy-check that your team will run after every deploy. It checks one service across three clusters, reports whether each cluster is on the wanted image and fully ready, and posts a one-line summary to the team chat. It must be safe to run twice, safe to run while a rollout is still in progress, and usable by a teammate who has never read its code. Sketch the commands, the checks, and the failure behavior.
I would ask five things before I sized anything. Who runs this tool, a human after a deploy or a pipeline step, because the output format is different for each? What does ready mean for this service: pods ready, health endpoint answering, or traffic actually flowing? How fresh must the answer be, and what happens when one cluster is unreachable: do we fail the whole run or report that cluster as unknown? What is the wanted image, and who decides it: a config file, a command-line flag, or the last deploy record? And what is the failure we design for: a cluster down, a deploy half finished, or the chat webhook down?
Start with the budget from this lesson. One cluster costs about 13.3 seconds on a clean run. Three clusters in sequence cost about 40 seconds. That is fine for a human, and slow for a pipeline step that runs on every deploy. I run the three cluster checks in parallel, so the total is the slowest cluster, about 15 seconds, plus 2 seconds for the chat post. Little's Law for the sizing: if the team deploys 20 times a day and each run makes 5 API calls per cluster, that is 300 calls a day, which is nothing for the API server, so I do not need caching. But if the tool runs every minute as a monitor, that is 21,600 calls a day, and then I add a cache and a slower poll. The chat post is one call with a timeout of 5 seconds, because a slow chat webhook must never make the check look slow. The tool prints a short summary to the screen for the human and writes one JSON log line per run for the machine, and it exits zero only when all three clusters are ready.
Parallel checks buy speed and cost clarity. When three clusters run at once, a failure in one is harder to read in the output, so I print one line per cluster with its own result. A single chat post buys a clean summary and costs detail, so the post links to the full log instead of carrying it. Checking the health endpoint buys a true answer and costs a network call to each cluster, so I check pods first and only call the endpoint when the pods say ready. Exiting nonzero on any unknown buys honesty and costs noise, so I separate unknown from failed: unknown means the cluster could not be reached, failed means it was reached and is not ready, and the exit code is nonzero for both but the message says which one. And the tool is a CLI, not a service, because the team needs it after deploys, not every minute. If they start running it every minute, I rebuild it as a service with its own state, and that is a different design.
Saved in this browser, so the Lessons index can show your progress across the course.