Lesson 98 taught you ps, chmod, and df. This lab is where you use them on a live box. You find the busiest process, you fix a file's permissions and prove it, you back a folder up with tar and restore it, and you write a host-check script that passes or fails with an exit code. By the end you have a deploy gate you can run before any deploy.
Difficulty: Intermediate · Know this first: Lesson 98 — Advanced Linux; Lab 151
Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: on a cache hit the trip is almost free, on a miss the kitchen eats the budget. Step through the lesson or switch the scenario and watch the bar change.
Extra diagram · from scratch to master
Start with one folder. Add one box at a time until the host check passes and the gate opens.
The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the process hunt, the disk check, and the total change in the budget bar, in the hop timings, and in the total. The numbers are a teaching example, not live measurements. The shape is the lesson: a fuller disk means a longer check, and a failed gate costs a fix.
How full the disk is. Under 90% the check is quick. Over 95% the check fails, and the fix takes time.
How many processes ps lists before you find the busy one. More processes means more lines to read.
How many files over 50 MB the find step turns up. Each one has to be identified before anything is deleted.
Try this: (1) Set disk to 5%, processes to 5, big files to 0. The process hunt is about 2 min, the disk check about 1 min, the gate about 1 min, and the total is about 9 min. (2) Set all three to the worst end: 98% disk, 50 processes, 10 big files. The process hunt is about 6 min, the disk check about 5 min, the gate about 14 min, and the total is about 30 min. Same host. Only the inputs changed.
Three questions. No score is saved anywhere — this is just you against the material.
Here is the host-check script from this lab, ready to edit. Change it, then run the check. The check looks for the four things a working gate needs: a disk check, a readability check, a python3 check, and an exit with the failure count.
Try breaking it on purpose: change -ge 95 to -ge 5, or delete the exit line. Run the check and watch it fail. Then press Reset and try a change that keeps it passing, like adding a writability check.
Expected output when it passes: The script checks the disk, readability, and python3, and exits with the failure count. On a healthy host it prints PASS lines and exits 0.
I take the same host four times here. First I look at it. Then I change it. Then I back it up. Last I gate on it.
Flags in the next block, letter by letter. I read every flag out loud before I run a command I copied. A flag I cannot explain is a flag I do not run yet.
ps -eo pid,comm,%cpu --sort=-%cpu | head -4 lists the busiest processes. The -e means every process. The -o takes the column list: pid, command name, cpu percent. The --sort=-%cpu sorts by CPU use, and the minus in front means highest first. Without the minus, the quietest process comes first, which is the wrong end of the list when I am hunting the busy one. The head -4 keeps the header and the top three processes.df -h /tmp shows the space on one filesystem. The -h means human readable, MB and GB instead of 1K blocks. The /tmp at the end limits the answer to that filesystem. Without it, df lists every filesystem on the box.du -sh . shows how much space the current directory holds. The -s means summary, one total line. The -h means human readable. Without -s, du lists every subdirectory too, and I have to find the total in the noise.ls -l data/a.txt shows the permission bits as a string. The -l is the long listing. I read -rw-r--r-- left to right: file type, owner bits, group bits, others bits.df -i shows inode use instead of byte use. The -i means inodes. A disk can have free bytes and zero free inodes, and then nothing can create a file. I check both numbers, because they are two different ways to be full.Check it: run ps -eo pid,comm,%cpu --sort=-%cpu | head -4 and read the top line. Then run the same ps without the minus in --sort and see the quietest process on top instead.
Definition: ps lists the processes. df shows how full each disk is. du shows how big a folder is. ls -l shows the permission bits. These four commands are the first thing I run on any box I have never seen.
Types:
Uses: I use this for any new host, before I change anything. Try it now: run ps and df on your own box and read the top line of each.
Dimensions: At low volume everything is instant. What breaks first is reading the wrong column: %CPU is the busy column, and Use% is the full column.
Flags in the next block, letter by letter. I read every flag out loud before I run a command I copied. A flag I cannot explain is a flag I do not run yet.
chmod 640 data/a.txt sets the permission bits by number. 6 is read plus write for the owner, 4 is read only for the group, 0 is nothing for everyone else. The result reads -rw-r-----. I read the three digits as owner, group, others, in that order.chmod 644 data/a.txt gives read back to others. The last digit changes from 0 to 4, and the string becomes -rw-r--r--. The difference between 640 and 644 is one digit and one group of readers.chmod u+x file adds execute for the owner only. The u means user, the owner. The +x adds the execute bit. Plain chmod +x without the u would add execute for owner, group, and others, which is wider than I want for a plain file.chown nobody:nogroup file changes the owner and the group in one move. The part before the colon is the new owner, the part after it is the new group. Without the colon and group, chown nobody file changes only the owner and leaves the group alone.ls -l data/a.txt proves the change happened. The -l is the long listing. I read the bits before and after every chmod, because the command prints nothing and the listing is the only proof.Check it: run chmod 640 data/a.txt && ls -l data/a.txt, then chmod 644 data/a.txt && ls -l data/a.txt. The last three letters change from --- to r-- and nothing else moves.
Definition: chmod changes the permission bits. 640 means owner read+write, group read, others nothing. 644 adds read for others. chown changes the owner. A file with the wrong bits is either unreadable to the people who need it, or readable to the people who should not.
Types:
Uses: I use chmod when a file is too open or too closed, and I read the bits before and after. Try it now: run chmod 640 on the lab file, read the bits, and put them back.
Dimensions: As the team grows, permissions get harder, because more people need different access. What breaks is 777: it fixes the error today and opens the file to everyone forever.
Flags in the next block, letter by letter. I read every flag out loud before I run a command I copied. A flag I cannot explain is a flag I do not run yet.
tar czf demo.tar.gz data packs a directory into one compressed file. Read the letters one at a time. The c means create a new archive. The z means compress it with gzip. The f means the next word is the file name. Without the f, tar would write the archive to the screen instead of a file.tar tzf demo.tar.gz | sort lists what is inside without unpacking it. The t means list. The z and f mean the same as above. I always list before I unpack, so I know what is about to land.tar xzf demo.tar.gz -C restored --no-same-owner unpacks it into a chosen folder. The x means extract. The -C restored means change into the restored directory first, so the files land there and not in the current directory. The --no-same-owner means do not restore the original owner, keep the files owned by me. Without it, extracting an archive made by root would leave files owned by root in my folder.diff -r data restored/data compares two directories file by file. The -r means recursive, walk into subdirectories. Without -r, diff can only compare two files. Silence from diff means the two trees match, and that silence is the proof the restore worked.find . -size +10k -type f finds files bigger than 10 KB. The -size +10k means larger than 10 kilobytes. The + means more than. A - in front would mean smaller than, and no sign would mean exactly that size. The -type f means files only.Check it: run the round trip, then run diff -r data restored/data. Diff should print nothing. Then change one file in the restore, run diff again, and watch it catch the change.
Definition: tar czf makes a compressed archive. tar tzf lists it. tar xzf extracts it. diff -r compares two folders. A backup is only real when the restore has been tested, and diff is the cheapest restore test there is.
Types:
Uses: I use tar for any folder I cannot afford to lose, and I restore it somewhere else to prove it. Try it now: run the round trip from step 4 and check diff is silent.
Dimensions: At ten times the size, tar still works, and the time grows with the bytes. What breaks is ownership: extracting as the wrong user gives files the wrong owner, and --no-same-owner is the fix in a sandbox.
Flags in the next block, letter by letter. I read every flag out loud before I run a command I copied. A flag I cannot explain is a flag I do not run yet.
chmod +x host-check.sh && bash -n host-check.sh makes the script executable and checks its syntax. The +x adds the execute bit. The -n on bash means no execution, parse only. Silence from bash means the syntax is clean. I run both before I run a script I just wrote../host-check.sh runs the script. The ./ means run the file in this directory, because the current directory is not on the command search path.echo $? prints the exit code of the script. The $? holds it. 0 means every check passed and the deploy may go. 1 means at least one check failed and the deploy stops. I read the exit code, not just the PASS and FAIL lines, because a gate the next script calls can only see the code.df /tmp | awk 'NR==2 {print $5}' | tr -d '%' reads the disk use percent as a bare number. The NR==2 picks line 2, skipping the header. The $5 is the fifth field, the use percent. The tr -d '%' deletes the percent sign. The -d on tr means delete the named characters. Without it, the number would carry a % and the numeric compare after it would fail.[ -r /tmp ] tests readable. [ -w /tmp ] tests writable. The brackets are the test command with a different spelling. Other tests I use often: -x executable, -f regular file, -d directory, -e exists, -s not empty.command -v python3 >/dev/null checks whether python3 exists without printing anything. The -v prints the path when it exists and nothing when it does not. The >/dev/null throws the path away, because the script only needs the exit code.Check it: run ./host-check.sh; echo $?. The exit code should be 0. Then run df -h /tmp and compare the percent with what the script printed.
Definition: At this level the host check is a deploy gate. It runs before the deploy, it prints PASS or FAIL per check, and it exits 0 or 1. A failing gate stops the deploy, and the rerun after the fix is the proof the fix worked.
Types:
Uses: I use this before every deploy to a host I do not own. Try it now: run ./host-check.sh, read the exit code, and decide whether you would deploy.
Dimensions: At a hundred hosts, the script is the same shape, and it runs on every host. What breaks is a gate nobody runs: a check that exists but is never executed protects nothing.
Flags in the next block, letter by letter. I read every flag out loud before I run a command I copied. A flag I cannot explain is a flag I do not run yet.
fallocate -l 100M /tmp/bloat.bin reserves 100 MB of disk at once. The -l takes the length. The file appears full size at once, without writing 100 MB of data. I use it to make a big file fast when I need to test a full disk. Deleting the file frees the space again.df -h /tmp shows the space on one filesystem. The -h means human readable. Run it before and after the fallocate and watch the used number grow by about 100M.du -sh /tmp/bloat.bin shows how much space one file holds. The -s means summary, one total line. The -h means human readable.find /tmp -size +50M -type f finds files bigger than 50 MB. The -size +50M means larger than 50 megabytes. The + means more than. The -type f means files only.rm /tmp/bloat.bin deletes the file. No flags. The disk space comes back at once.mkdir -p ~/lab-host/data && cd ~/lab-host && echo hello > data/a.txt makes the lab folder and the first file. The -p means make the parents too. The && means run the next command only if this one worked. The > writes the word hello into the file.Check it: run fallocate -l 100M /tmp/bloat.bin && df -h /tmp, then rm /tmp/bloat.bin && df -h /tmp. The used number should rise by 100M and fall back again.
The host-check script, the whole file. Do this: save it as host-check.sh, run chmod +x host-check.sh && bash -n host-check.sh, then ./host-check.sh. Expected output: four PASS lines and exit 0. On this lab box the disk line reads disk /tmp: 5%.
#!/bin/bash
set -u
fail=0
disk=$(df /tmp | awk 'NR==2 {print $5}' | tr -d '%')
echo "disk /tmp: $disk%"
if [ "$disk" -ge 95 ]; then echo "FAIL: disk $disk%"; fail=1; else echo "PASS: disk under 95%"; fi
if [ -r /tmp ]; then echo "PASS: /tmp readable"; else echo "FAIL: /tmp not readable"; fail=1; fi
if command -v python3 >/dev/null; then echo "PASS: python3 present"; else echo "FAIL: python3 missing"; fail=1; fi
if [ -w /tmp ]; then echo "PASS: /tmp writable"; else echo "FAIL: /tmp not writable"; fail=1; fi
exit "$fail"
Make a disk fill up on purpose, safely, in /tmp. Do this: run fallocate -l 100M /tmp/bloat.bin && df -h /tmp. Expected output: the Use% moves up at once. Then run du -sh /tmp/bloat.bin and find /tmp -size +50M -type f, and expect both to name bloat.bin. Delete it with rm /tmp/bloat.bin and watch df move back. A sparse file made with truncate does not move df, because it holds no blocks. fallocate really allocates, so it really fills.
What the tar flags mean: c creates, z compresses with gzip, f names the file, t lists, x extracts. Do this: run tar czf demo.tar.gz data, then tar tzf demo.tar.gz | sort. Expected output: data/ then data/a.txt. The sort makes the listing deterministic.
At 10x the hosts, the shape holds and the script runs on every host. At 100x, the design changes: the checks move into a monitoring agent, and the gate reads the agent instead of running the commands. The exit code never changes, though. 0 still opens the gate, and 1 still shuts it.
A deploy went out on a Friday to a host whose disk was 99% full. The deploy wrote its files, ran out of space halfway, and left the app half updated. The old code was gone, the new code was incomplete, and the rollback failed too, because the rollback also needed disk space. The host check that would have caught it existed. It had not been run in three weeks, because the deploy script called it only when someone remembered to add the flag.
Two things. First, the host check runs inside the deploy script, every time, with no flag to forget. Second, the check fails the deploy at 90%, not 95%, so there is room to roll back. Try it now: run ./host-check.sh and lower the limit in your copy to 90, then watch it still pass on a healthy host.
Build a host-check script for your own box with five checks: disk under 90%, /tmp readable, /tmp writable, python3 present, and no file over 50 MB in /tmp you cannot name. It passes when it prints five PASS lines and exits 0.
Success looks like this: five PASS lines, exit 0, and a rerun after the filler is deleted that passes again. If the fifth check fails on a healthy host, the find path or size is wrong.
If df shows a different filesystem than you expected, check the mount point: /tmp may be its own filesystem. If find prints nothing, the size flag may be too big. If the script exits 1 with all PASS lines, the exit line is wrong. If fallocate is missing, the disk-full replay cannot run here, and the lesson says so.