A shell script is a text file full of commands. The computer reads it from the top and runs each command in order, the same way you would type them one by one. It exists because typing the same twenty commands every deploy is slow, boring, and easy to get wrong. You write the commands once, check them once, and then the machine repeats them exactly. Here is one I actually run. Every night at 2 AM a small script checks the disk space on three servers and pages me if any disk is over 90 percent full. I wrote it once. It has run a thousand times since. This lesson follows one script from its first line to its last, and shows you the lines that keep it honest: the safety net at the top, the loop that checks each server, the function that does the deploy, the pipe that reads the log, and the exit code that tells CI the truth. Most scripts fail quietly. The good ones fail loudly, clean up after themselves, and say so.
Every hop charges rent in milliseconds. The bar below is a worked example for one script run, not a measurement from a live system, but the shape is honest: the loop and the deploy function do almost all the work, because they run once per server. The safety net and the trap together cost 15 ms. Step through the lesson or switch the scenario and watch the bar change. The run with no safety net is 15 ms faster and wrong in every way that matters. The thirty-server run is nine times slower without a single failure, because the work scales with the server count.
The budget above is one fixed run. This box is yours. Change the fleet, grow the log, switch the safety net off, run the failure drill, and watch the bill move. The numbers are a teaching example, not live measurements. The shape is the lesson: the work that scales with the server count is the work that owns the run.
Each server costs a 100 ms health check in the loop and an 80 ms deploy in the function. The loop and the deploy function are the only hops that grow with the server count.
The pipe reads the whole log through grep. Bigger log, longer pipe. Pipe time is log size times 0.2 ms, so a 100 MB log costs 20 ms and a 2,000 MB log costs 400 ms.
On means the script stops at the first failure, reports it with a non-zero exit code, and the trap cleans up the temp files. Off means the script keeps going after a failure, exits 0 anyway, and leaves its temp files behind.
When on, server 2 fails its health check. This is a drill, not a real failure. Watch what the safety net does with it. With the net on, the script stops at server 2. With the net off, it deploys to the sick server and reports success.
Try this: (1) Turn the failure drill on with the safety net on. The total drops from 586 ms to 326 ms and the exit code is 1. The script stopped early and told the truth. (2) Now turn the safety net off with the drill still on. The total goes back to 571 ms, the exit code is 0, and the temp files are left behind. Same failure, opposite story. That is what the safety net is for.
Three questions. No score is saved anywhere — this is just you against the material.
Same topic, four times. First the plain idea, then the safety net and the tools that make a script honest, then debugging and knowing when bash is the wrong tool, then the level where the script is a platform job that runs itself.
Definition: A shell script is a text file full of commands. The computer reads it from the top and runs each command in order, the same way you would type them one by one. It exists because typing the same twenty commands every deploy is slow, boring, and easy to get wrong. You write the commands once, check them once, and then the machine repeats them exactly.
Types:
Uses: Every night at 2 AM a small script checks the disk space on three servers and pages me if any disk is over 90 percent full. I wrote it once. It has run a thousand times since. I use a script whenever the same twenty commands have to run the same way every time, with no step skipped because someone was tired.
Dimensions: With three servers, a loop that runs one server at a time is fast enough, and the whole run takes seconds. As servers grow, the sequential loop is the first thing that costs real time. What breaks is a script that fails quietly: it prints nothing, exits zero, and the deploy never happened. I notice it when the pipeline stays green and the change is not live. What it costs is the minutes to write it and the habit of checking it once, carefully, before it runs a thousand times. The trade is automation against a new thing to maintain. A script I never look at again is a script I cannot trust.
Definition: Most scripts fail quietly. The good ones fail loudly, clean up after themselves, and say so. The safety net is one line at the top: set -euo pipefail. It stops the script on the first error, treats an unset variable as an error, and fails the pipeline when any step in a pipe fails. On top of that sit the tools that make a script honest: traps for cleanup, arrays for lists, getopts for flags, and process substitution for comparing two things without temp files.
Types:
trap cleanup EXIT runs the cleanup function however the script exits, and it deletes only what the script itself created, by exact path, never by a wildcard.servers=(web-1 web-2 web-3), so the loop reads "${servers[@]}" instead of a space-separated string that breaks on names with spaces.-e staging -v, so the script stops guessing which positional argument is which and prints a usage line when a flag is wrong.diff <(sort old.txt) <(sort new.txt) hands the output of two commands to diff as if they were files, so I can compare the live config against the one in Git without writing temp files.xargs -P 4 runs four servers at a time instead of one.Uses: I hold the server list in an array, so adding a server means adding one word, not rewriting the loop. I parse flags with getopts, so deploy.sh -e production can never silently mean staging because someone swapped two arguments. I diff the live config against Git with process substitution before a deploy, and I check the important command separately, because the exit code inside <(...) does not reach the script the way I expect. On call, when a script misbehaves, I run it with bash -x and watch every command print right before it runs, with the variables already filled in.
Dimensions: Thirty servers in sequence cost about 30 seconds of checks. Four at a time with xargs -P 4 cuts that by roughly four, and the wall-clock time falls by the same factor. What breaks is a wildcard in a cleanup trap, which is how you delete the wrong folder at 2 AM, or two runs sharing a hard-coded temp path like /tmp/deploy, or parallel logs that interleave so I cannot tell which server failed. I notice the first from a missing folder, the second from a run that steps on another run, and the third when I give each server its own log file and merge them at the end. What it costs is clean logs and one exit code per server, collected and failed on if any of them is non-zero. The trade is speed against debuggability. Parallel is faster, and sequential is easier to read when it breaks.
Definition: When a script misbehaves, I run it with bash -x. That one flag has solved more script bugs for me than any dashboard. The second tool is shellcheck. It is a linter for shell scripts. It reads the script and flags the classic mistakes: unquoted variables, useless use of cat, a comparison that will never be true. I run shellcheck in CI on every script in the repo, and a warning fails the build. And there is a line where bash stops being the right tool at all.
Types:
Uses: A script that passes shellcheck can still be wrong, but it is wrong in interesting ways instead of the same five ways every time. I use the date-named backup folder so a rerun never overwrites the backup I might need. I use the guarded delete so a variable that comes back empty cannot turn a delete into a disaster. And I use the 200-line rule: a script over about 200 lines with nested loops and string surgery is a sign I have outgrown bash, and I rewrite it in Python, which is Lesson 73. Parsing JSON by hand with grep is a sign. Doing real math with dates is a sign. If I am fighting the shell instead of using it, I have outgrown it.
Dimensions: At three hundred servers, the parallel loop is the norm, and the wall-clock time falls by the parallel factor while the log problem grows with it. What breaks is a failure in one server that no longer stops the others, so the run looks finished while one server never got its backup, or a non-idempotent rerun that overwrites yesterday's folder. I notice the first by collecting one exit code per server and failing the run if any is non-zero, and the second when a rerun destroys the only good backup. What it costs is CI minutes on lint and the time to rewrite in Python when the logic gets big. The trade is bash glue against a real language. Bash is for gluing commands together. Python has real data structures, real error handling, and libraries for JSON and HTTP.
Definition: At three thousand servers, I stop running the script from my laptop or from one CI job entirely. The script becomes a job the platform runs on a schedule, with a lock file so two copies never run at once, with idempotency so a rerun is safe, and with the deploy split into waves so a bad deploy hits ten servers before it hits three thousand. The verify step is the whole point. A backup nobody verified is a hope, not a backup.
Types:
Uses: The nightly disk check from this lesson becomes a platform job that pages on-call for the failed server and still exits non-zero so CI goes red. The backup run writes each run to its own date-named folder, verifies every copy with a checksum, and never lets a second run overwrite the first. In a design review, the question I ask is who owns the backup store and what happens when it fills up, because a script that writes forever and nobody watches is a disk-full page waiting for a date.
Dimensions: The copy is the big cost. If each config folder is 500 MB and the copy runs at 50 MB per second, one server costs 10 seconds, and 40 servers in sequence cost 400 seconds. If the folders were 5 GB, the sequential run would be 90 minutes and would blow a 30-minute window, so I run four copies at a time and bring it back to about 23 minutes. What breaks is two copies running at once without a lock, a rerun that overwrites a good backup, or a backup that was never verified. I notice the first from a lock file that says the job is already running, and the last only when I try to restore and find the copy is short. What it costs is platform time, storage headroom for copies in flight, and the discipline of verifying every run. The trade is automation that runs itself against automation I can still reason about at 2 AM. The script got slower to fail, and that was the fix.
The lesson above is one script through nine hops. Production is the rest of it. Here is what I actually spend my time on once the safety net is in place and the loop is counted.
The trap in step 9 is the simplest version. The real version names a cleanup function and registers it once: trap cleanup EXIT. The cleanup function deletes the temp folder the script made with mktemp -d, removes the lock file, and prints one line saying what it removed. I make the temp folder with mktemp because it picks a name nobody else is using, and I never hard-code a temp path like /tmp/deploy, because two runs at once would share it and step on each other. One rule I never break: the trap deletes only what the script itself created, by exact path, never by a wildcard. A wildcard in a cleanup trap is how you delete the wrong folder at 2 AM.
Process substitution looks strange the first time you see it: diff <(sort old.txt) <(sort new.txt). The <(...) part runs a command and hands its output to another command as if it were a file. It exists because some commands only accept files, not pipes. The classic use is comparing the output of two commands side by side without writing temp files. I use it for diffing configs before a deploy: the live config against the one in Git. The warning is the same as with pipes. The exit code of the command inside <(...) does not reach the script the way you expect, so I check the important command separately and use process substitution only for reading and comparing, never for the step that decides pass or fail.
When a script misbehaves, I run it with bash -x, which prints every command right before it runs it, with the variables already filled in. That one flag has solved more script bugs for me than any dashboard. The second tool is shellcheck. It is a linter for shell scripts. It reads the script and flags the classic mistakes: unquoted variables, useless use of cat, a comparison that will never be true. I run shellcheck in CI on every script in the repo, and a warning fails the build. A script that passes shellcheck can still be wrong, but it is wrong in interesting ways instead of the same five ways every time.
Bash is for gluing commands together. It is the right tool when the script is mostly calling other programs: check the disk, copy the files, restart the app, read the log. It is the wrong tool when the logic itself gets big. Parsing JSON by hand with grep is a sign. Doing real math with dates is a sign. A script over about 200 lines with nested loops and string surgery is a sign. At that point I rewrite it in Python, which is Lesson 73. Python has real data structures, real error handling, and libraries for JSON and HTTP. The rule I use: if I am fighting the shell instead of using it, I have outgrown it.
At 10x the fleet, thirty servers become three hundred, and the sequential loop from this lesson costs 30 seconds of checks and 24 seconds of deploys. That is when I run the loop in parallel with xargs -P, four or eight servers at a time, and the wall-clock time falls by roughly that factor. The trade is that the logs interleave and a failure in one server no longer stops the others, so I collect one exit code per server and fail the run if any of them is non-zero. At 100x, three thousand servers, I stop running the script from my laptop or from one CI job entirely. The script becomes a job the platform runs on a schedule, with a lock file so two copies never run at once, with idempotency so a rerun is safe, and with the deploy split into waves so a bad deploy hits ten servers before it hits three thousand.
This is the skeleton I start every ops script from:
#!/usr/bin/env bash
set -euo pipefail
WORKDIR="$(mktemp -d)"
trap 'rm -rf "$WORKDIR"' EXIT
ENV="${1:?usage: deploy.sh <staging|production>}"
BACKUP_DIR="/var/backups/${ENV}"
echo "Deploying to ${ENV}, work in ${WORKDIR}"
Six ideas in ten lines. The shebang picks bash. set -euo pipefail is the safety net. mktemp makes a private work folder. The trap deletes that folder when the script exits, however it exits. The argument check stops the script with a usage message if no environment was given. And the backup path is built from the argument, so staging and production can never share a folder by accident. I start every ops script from this skeleton. It has saved me more times than any clever one-liner.
This one stayed with me because the pipeline said everything was fine. The job was green. The dashboard was green. The deploy had not happened, and a cleanup step had deleted the wrong folder.
A deploy script ran every night from CI. One night the config file it read had been moved during a cleanup of the repo, so the variable holding the target folder came back empty. The script had no set -u, so the empty variable was not an error. It kept going. The deploy step copied files into the wrong place and reported nothing. The cleanup step at the end deleted a folder it should never have touched. CI saw exit code 0 and went green. We found out the next morning, when the app was still running last week's code and a folder of old release notes was gone.
Three things. First, every script in the repo now starts with set -euo pipefail, and shellcheck runs in CI and fails the build on a warning. Second, no script deletes anything from a variable without a guard: the variable must be non-empty, and the path must be inside the folder it is supposed to be in. Third, the nightly deploy job now checks the deployed version after it runs and pages on-call if the version did not change. A green job is not proof. The version number on the server is the proof.
This is the kind of question I would ask in an interview, and the kind I ask myself before I hand a script the keys to production. Take a minute with it before you read my approach.
Design a nightly backup script for 40 servers. Every night at 2 AM it must copy each server's config folder to a backup store, verify the copy, and page on-call if any server fails. The run must finish inside a 30 minute window, it must be safe to rerun, and it must never delete a backup it should not delete. Sketch the script, the safety lines, and the time budget.
I would ask five things before I sized anything. How big is each config folder, and how long does one copy take? What does verify mean here: a file count, a checksum, or a test restore? What happens when one server fails: does the run stop, skip it and continue, or retry it? What does safe to rerun mean for this backup: can a second run overwrite the first, or does each run get its own folder? And who owns the backup store: what happens when it fills up?
Start with the numbers from this lesson. One server costs 100 ms for the health check and 80 ms for the work step, so 40 servers in sequence cost 4 seconds of checks and about 3 seconds of work. That is the small part. The copy is the big part. If each config folder is 500 MB and the copy runs at 50 MB per second, one server costs 10 seconds, and 40 servers in sequence cost 400 seconds, about 7 minutes. Verification with a checksum adds about 2 seconds per server, 80 seconds total. The whole sequential run is about 9 minutes, inside the 30 minute window with room to spare. If the folders were 5 GB instead of 500 MB, the sequential run would be 90 minutes and would blow the window, so I would run four copies at a time with xargs -P 4 and bring it back to about 23 minutes. Little's Law for the sizing: at 4 copies in flight and 10 seconds per copy, the script holds 4 copies inside the system at once, so the backup store needs 4 times 500 MB of headroom plus the finished backups, and I alert at 80 percent of the store.
Running in parallel buys time and costs clean logs, because four servers write at once, so I give each server its own log file and merge them at the end. Skipping a failed server buys a finished run and costs a gap in the backups, so I page on-call for the failed server and still exit non-zero so CI goes red. Making the run idempotent buys safe reruns and costs a naming rule: each run writes to a folder named by date, and never overwrites yesterday's folder. Guarding the delete buys safety and costs a check on every path: the script refuses to delete anything outside the backup store. And the verify step is the whole point. A backup nobody verified is a hope, not a backup.
Saved in this browser, so the Lessons index can show your progress across the course.
Bash glues commands together. Python is what you reach for when the logic outgrows the glue: real data structures, real error handling, and libraries for JSON, HTTP, and APIs. The next lesson rewrites the same kind of ops job in Python, and shows you where the switch pays off and where bash is still the right call.