System Design · Lesson 73

Python

Python is a language for writing instructions a computer can follow. You write the instructions in a plain text file. A program called the interpreter reads the file and runs it, one line at a time. That is the whole idea, and it is why I reach for Python first when something needs doing once a day, every day, without a human watching. Here is one concrete example. Every morning I want a list of which services are healthy and which are not. I write a small Python script that reads a list of 50 service names, asks each one if it is healthy, and writes the answers into a report file. I run it with one command. It takes less than a second. When a service stops answering, the report says so in plain words, and I do not have to click through 50 dashboards to find out. This lesson follows that script from the moment I type the command to the moment the report file closes. You will see where the time goes, what breaks in real life, and how to write a script you can trust on a server.

Difficulty: Advanced · Know this first: Lesson 72
Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one run of the script, not a measurement from a live system, but the shape is honest: in most slow scripts I have chased, the waiting costs more than the code. The network hop is 120 ms of the 380 ms normal run. The loop is 80 ms. The code that does the real thinking is the cheap part. Step through the lesson or switch the scenario and watch the bar change. The system-Python run is nearly three times slower without the script doing anything different. The cached concurrent run is less than half, because the setup was already paid and the waiting overlapped.

Change the script and watch the run bend

The budget above is one fixed run of fifty services. This box is yours. Change the service count, run the checks together, or go cold, and watch the total bend. Then ask the question I ask before every deploy: how much of this run is setup, and how much is the work itself?

How many services the script checks in one run. Each service adds work to the loop and one network check. The loop costs 1.6 ms per service. The network costs 2.4 ms per service when the checks run one after another, and 0.8 ms per service when they run at the same time.

One at a time means the script finishes one check before it starts the next, so the waiting adds up. At the same time means the script starts all the checks and waits for the answers together, so the waiting overlaps. The network is no faster. The script just stops sitting idle.

Warm means the bytecode and the imports were already loaded on a recent run. Cold means the interpreter has to read every file and convert it again. Warm costs 15 ms to read the code and 60 ms to load the imports. Cold costs 45 ms to read the code and 240 ms to load the imports.

Loop cost80 ms50 services × 1.6 ms
Network cost120 ms50 services × 2.4 ms, one at a time
Setup cost125 msstart 40 + read 15 + imports 60 + packages 10
Total for one run380 ms105 fixed + read + imports + loop + network

Try this: (1) Set services to 500 with the checks one at a time. The total jumps from 380 ms to 2,180 ms and nothing else changed. (2) Now switch the checks to at the same time. The total falls to 1,380 ms. Same script, same 500 services. Only the waiting changed.

Check yourself

Three questions. No score is saved anywhere. This is just you against the material.

Basic to master

Same topic, four times. First the plain idea, then the habits that make a script trustworthy, then the wrappers, the waiting, and the packaging, then the level where one script is not enough and the work splits across workers.

Basic — Instructions in plain text

Definition: Python is a language for writing instructions a computer can follow. You write the instructions in a plain text file. A program called the interpreter reads the file and runs it, one line at a time. That is the whole idea, and it is why I reach for Python first when something needs doing once a day, every day, without a human watching.

Types:

  • Variables — a name that holds a value, like a service name or a count.
  • Lists and dictionaries — a list holds things in order, a dictionary holds them by name.
  • Loops and conditionals — do this for each item, and decide what to do when something is wrong.
  • Functions — a named block of instructions I can call more than once.

Uses: Every morning I want a list of which services are healthy and which are not. I write a small Python script that reads a list of 50 service names, asks each one if it is healthy, and writes the answers into a report file. I run it with one command. It takes less than a second. When a service stops answering, the report says so in plain words, and I do not have to click through 50 dashboards to find out.

Dimensions: With 50 services, the script runs in under a second and the whole job is one file. As the list grows, the loop is the first thing that costs real time, because each check waits for an answer in turn. What breaks is a script that stops on the first error, so one dead service means no report at all, or a file left open, so the report looks fine and the last lines are missing. I notice the first from a missing report and the second only when someone reads the report in a meeting. What it costs is interpreter startup and the time to learn the language. The trade is readability against speed. Python is easy to read at 2 AM, and it is slower than a compiled language at crunching numbers.

Intermediate — Writing code you can trust

Definition: A script I can trust closes what it opens, says what type each value should be, and runs in the same environment everywhere. The with line makes a promise that something gets cleaned up, and Python keeps it even if the code inside crashes. Typing means writing down what type each value should be. A checker called mypy enforces the types before the script runs, in the pipeline. A virtual environment means the pipeline has the same libraries as my laptop.

Types:

  • Comprehensions — a loop written in one line that builds a new list from an old one.
  • Generators — the same shape with round brackets, handing out one item at a time, only when asked.
  • Context managers — the with line that closes the file, returns the connection, and releases the lock.
  • Type hints and mypy — the types written in the signature and checked before the script ships.
  • Virtual environments — one isolated set of libraries per project, so the pipeline and the laptop match.

Uses: On 50 services the difference between a list and a generator does not matter. On ten million log lines it is the difference between a script that runs and a script that runs out of memory. My rule is simple. If the list fits in memory and I need it twice, I build it. If I only need it once, I generate it. I never open a file without with. I add types to shared code and to anything that runs on a schedule, because a scheduled script fails at 3 AM when nobody is watching, and types catch the wrong input before the script ships, not after it pages someone.

Dimensions: Five hundred checks one at a time take ten times as long as fifty, so I run them together and I set a timeout on every single one. What breaks is the system Python with no virtual environment, so the pipeline starts clean and the library the script imports is not there, or an unclosed file that silently drops the last lines. I notice the first as a ModuleNotFoundError in the pipeline that never happens on my laptop, and the second as a report that is short by a page. What it costs is virtual-environment setup and type-checking time in the pipeline. The trade is memory against convenience. A list is convenient and costs memory. A generator is cheap and can only be read once.

Advanced — Decorators, waiting, and packaging

Definition: A decorator is a wrapper around a function. It runs extra code before and after the function, without changing the function itself. Async is how Python waits for many things at once. The keyword async marks a function that can pause while it waits, and await is the pause. Packaging is the front door: a pyproject.toml file that names the tool, lists its dependencies, and says which function runs when the user types the command.

Types:

  • Timer decorators — a wrapper that logs how long a function took, put on every function that talks to the network.
  • Async and await — waiting for many network answers at once instead of one at a time.
  • Packaged CLIs — a pyproject.toml front door, installed with pip and run by name.
  • Typed shared code — mypy-checked signatures on anything more than one person runs.

Uses: I put the timer decorator on the boundaries: the network calls, the file writes, the database reads. When a script gets slow, the logs already tell me which function is responsible. I use async for network checks and I keep it out of everything else, because async code is harder to read and harder to debug at 2 AM. I package a script the second time someone else needs to run it. The first time it is mine. The second time it is a tool, and tools need an installer, so nobody has to find the script file or remember which Python to use.

Dimensions: Five hundred checks run together finish in a fraction of the sequential time, as long as the waiting is the work. What breaks is async used where the script is busy doing math, which helps nothing, or a decorator on a tiny helper, which adds noise, or an unpackaged script that only works on my machine because the pipeline never had its libraries. I notice the first as a script that is still slow after I made it async, and the last as the same ModuleNotFoundError from the tier below. What it costs is packaging overhead and async complexity. The trade is observability against simplicity. A decorator on every network call is cheap insurance. A decorator on everything is noise.

Master — Python as a scheduled system

Definition: At a hundred times the scale, one script is not enough. I split the work across workers and write the results into a shared store instead of one file. The budget stops being a target and becomes a contract, and every hop owns its share of it. The script runs on a schedule, and the people who depend on it are not the people who wrote it.

Types:

  • Worker pools — the checks split across workers that run at the same time.
  • Shared result stores — results written where every worker and every reader can reach them.
  • Scheduled runs — the script runs on a clock, with a timeout on every check.
  • Per-hop budgets — each step owns its share of the total time, and the budget is enforced.
  • Typed interfaces — the shapes between workers written down, so a wrong input fails before it runs.

Uses: The morning health report for 50 services becomes a fleet check for 5,000, split across workers, with the results in a shared store instead of one file on one machine. I use the per-hop budget the way I use a latency budget: when the total slips, the budget says which hop owns the slip. In a design review, the question I ask is what happens when one worker dies mid-run, because a scheduled system that cannot lose a worker is a system that pages someone at 3 AM for a single dead check.

Dimensions: Workers multiply throughput, and the shared store becomes the new bottleneck. What breaks is one slow check without a timeout, which stalls the whole batch, or a store that fills up, or a scheduled run that nobody watches until it has been failing quietly for a week. I notice the first from the budget, which names the hop, and the last from an alert on the run itself, not on the services it checks. What it costs is worker infrastructure, store upkeep, and the maintenance of typed interfaces between workers. The trade is throughput against the simplicity that made Python the pick. The language is still easy to read. The system around it is not, and that is the real cost of scale.

Go deeper

The lesson above is one script through nine hops. Production is the rest of it. Here is what I actually spend my time on once the imports are counted and the budget is written down.

Comprehensions and generators

A comprehension is a loop written in one line. It builds a new list from an old one. I use it when the job is simple, like turning a list of names into a list of addresses. A generator looks almost the same, with round brackets instead of square ones, but it does not build the list at all. It hands out one item at a time, only when asked. On 50 services the difference does not matter. On ten million log lines it is the difference between a script that runs and a script that runs out of memory. My rule is simple. If the list fits in memory and I need it twice, I build it. If I only need it once, I generate it.

Decorators

A decorator is a wrapper around a function. It runs extra code before and after the function, without changing the function itself. I use one decorator more than any other: a timer that logs how long a function took. I put it on every function that talks to the network. When a script gets slow, the logs already tell me which function is responsible. I do not decorate everything. A decorator on a tiny helper adds noise. I decorate the boundaries: the network calls, the file writes, the database reads.

Context managers

A context manager is a promise that something gets cleaned up. A file gets closed. A connection gets returned to the pool. A lock gets released. The with line makes the promise, and Python keeps it even if the code inside crashes. I never open a file without with. I never open a network connection in a long-running script without one either. The bugs I have seen from unclosed files are quiet. The file looks fine, the last lines are missing, and nobody notices until the report is read in a meeting.

Typing

Typing means writing down what type each value should be. A function that takes a list of strings says so in its signature. Python does not enforce the types when the script runs. A checker called mypy enforces them before the script runs, in the pipeline. I add types to shared code and to anything that runs on a schedule. A scheduled script fails at 3 AM when nobody is watching. Types catch the wrong input before the script ships, not after it pages someone.

Async basics

Async is how Python waits for many things at once. The keyword async marks a function that can pause while it waits. The keyword await is the pause. While one check waits for an answer, the script starts the next check instead of sitting still. Async only helps when the script is waiting. If the script is busy doing math, async does nothing, and threads or separate processes are the answer instead. I use async for network checks and I keep it out of everything else, because async code is harder to read and harder to debug at 2 AM.

Packaging a CLI

A command line tool is a script with a proper front door. The front door is a pyproject.toml file that names the tool, lists its dependencies, and says which function runs when the user types the command. Once it is packaged, anyone on the team installs it with pip and runs it by name. No one has to find the script file or remember which Python to use. I package a script the second time someone else needs to run it. The first time it is mine. The second time it is a tool, and tools need an installer.

At 10x / 100x

At 10x the services, the loop is the first thing that breaks. Five hundred checks one at a time take ten times as long, so I run them together and I set a timeout on every single one. At 100x, one script is not enough, so I split the work across workers and write the results into a shared store instead of one file. The budget stops being a target and becomes a contract, and every hop owns its share of it.

One small file, five ideas

This is a small command line tool with a decorator, a context manager, and types:

import time
from functools import wraps

def timed(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        start = time.monotonic()
        result = func(*args, **kwargs)
        print(f"{func.__name__} took {time.monotonic() - start:.3f}s")
        return result
    return wrapper

@timed
def check_service(url: str) -> str:
    # ask the service if it is healthy, return one word
    return "healthy"

def main() -> int:
    with open("services.txt") as f:
        names = [line.strip() for line in f if line.strip()]
    for name in names:
        print(name, check_service(name))
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Five ideas in one small file. The decorator timed wraps the function and prints how long it took, without changing the function itself. The type hints say the function takes a string and returns a string, so a checker can catch a wrong input before the script runs. The with line opens the file and closes it even if something goes wrong inside. The list comprehension builds the names in one line. And the last two lines are the front door: the script only runs main when you run the file directly, not when another script imports it. The exit code from main is what the scheduler reads. Zero means quiet.

One time it broke

This one stayed with me because nothing failed. The script ran fine, it exited zero, and the scheduler stayed quiet. The damage was done anyway.

What broke

A nightly cleanup script deleted the wrong files. It was supposed to remove log files older than 30 days from one folder. It removed log files from every folder it could reach, including the folder where the billing reports were kept. The script ran fine. It exited zero. The scheduler stayed quiet. Nobody noticed until the next morning, when a report job failed because its input files were gone.

Timeline

  • 02:00The cleanup script runs on its nightly schedule. It finishes in four seconds and exits zero.
  • 02:05The scheduler marks the run as successful. No alert fires, because nothing failed on paper.
  • 07:30I open the morning report job and it fails. Its input files are missing.
  • 07:45I check the folder. The billing reports are gone. The log folder is empty too, which is what the script was supposed to do.
  • 08:00I read the script. The path variable was set from an environment variable, and on the new server that variable was empty. An empty path meant the script started at the top of the disk instead of in the log folder.
  • 08:20I check the backups. The reports are recovered from the nightly backup. Nothing is lost for good, but the morning report is three hours late.
  • 09:00I confirm the recovered files match the backup list, and the report job runs clean.

What changed after

Three things. First, the script now refuses to run if the path variable is empty. An empty path is a hard error, not a default. Second, the script runs in dry-run mode first and prints what it would delete, and the dry-run output goes to the log. A delete that nobody can see before it happens is a delete I do not trust. Third, the script runs inside a virtual environment with pinned package versions, and the scheduler checks the exit code and the dry-run output together. A script that deletes files and exits zero is not a success. It is a script I have not checked yet.

If you remember nothing else

Design exercise

This is the kind of question I would ask in an interview, and the kind I ask myself before I sign off on a real script. Take a minute with it before you read my approach.

The prompt

Design a command line tool called podcheck that checks 200 services every 5 minutes and writes a report. It must finish in under 60 seconds, run the same way on a laptop and on a server, and page the on-call when a service is down. You have Python, pip, a virtual environment, a scheduler, and a shared file store. Sketch the tool and the budget.

Clarifying questions to ask first

I would ask five things before I sized anything. How fresh must the report be: is a check from five minutes ago good enough, or must it be live? What happens when a service is slow: do we wait, time out, or mark it slow and move on? Is 200 services the steady count or the peak, and how fast does it grow? Where does the tool run, and who installs it when a new engineer joins the team? And what is the failure we design for: one service down, the whole network slow, or the report file missing?

Rough sizing, worked

Start with the budget from this lesson. Fixed setup is 105 ms plus the read and import cost, about 180 ms warm. The loop at 200 services is 200 times 1.6 ms, which is 320 ms. The network one at a time is 200 times 2.4 ms, which is 480 ms. The total is about 980 ms, inside the 60 second limit with room to spare. But one slow service with no timeout can hang the whole run, so I set a 5 second timeout on every check. If all 200 checks timed out one at a time, the run would take 1,000 seconds and miss the window completely. So I run the checks at the same time. At 0.8 ms per service the network cost falls to 160 ms, and even with ten slow services timing out together the run finishes in about 6 seconds. Little's Law for the sizing: 200 checks every 300 seconds is less than one check a second on average, so one worker is enough, and I alert at 80 percent of the 60 second budget, which is 48 seconds.

Trade-offs to name

Running the checks together buys speed and costs complexity, because async code is harder to read and harder to debug at 2 AM, so I keep the async part small and the rest of the script plain. A short timeout buys a fast run and costs false alarms on a slow day, so I mark a service slow before I mark it down, and I page only on down. Pinning package versions buys a stable run and costs the work of updating the pins, so I update them on a schedule instead of on a bad day. A shared file store buys one report everyone can read and costs a single place that can fail, so the tool also writes a local copy. And the target is the worst run, not the average run. An average of 1 second with a worst run of 90 seconds fails this design.

Saved in this browser, so the Lessons index can show your progress across the course.

Next in this series
Lesson 74 — Python for DevOps

You can write a Python script now. Next, you will put it to work the way platform teams do: running it on a schedule, reading logs, talking to APIs, and turning a one-off script into a tool the whole team can rely on.

Coming next