Platform Engineering · Lesson 152

Linux Lab: Processes, Permissions, and Disk

Lesson 98 taught you ps, chmod, and df. This lab is where you use them on a live box. You find the busiest process, you fix a file's permissions and prove it, you back a folder up with tar and restore it, and you write a host-check script that passes or fails with an exit code. By the end you have a deploy gate you can run before any deploy.

Difficulty: Intermediate · Know this first: Lesson 98 — Advanced Linux; Lab 151

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: on a cache hit the trip is almost free, on a miss the kitchen eats the budget. Step through the lesson or switch the scenario and watch the bar change.

Extra diagram · from scratch to master

The host lab, mapped: process, perms, tar, disk, gate

Start with one folder. Add one box at a time until the host check passes and the gate opens.

7 boxes on the trip4 host checks1 full disk
Lab folder
0 min
ps
1 min
chmod
1 min
tar
2 min
df/du
2 min
Host check
3 min
Full disk
trip ends
Request / flowResponse · the way backFailure / dead end

Try it: change the inputs and watch the trip bend

The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the process hunt, the disk check, and the total change in the budget bar, in the hop timings, and in the total. The numbers are a teaching example, not live measurements. The shape is the lesson: a fuller disk means a longer check, and a failed gate costs a fix.

5

How full the disk is. Under 90% the check is quick. Over 95% the check fails, and the fix takes time.

5

How many processes ps lists before you find the busy one. More processes means more lines to read.

0

How many files over 50 MB the find step turns up. Each one has to be identified before anything is deleted.

Try this: (1) Set disk to 5%, processes to 5, big files to 0. The process hunt is about 2 min, the disk check about 1 min, the gate about 1 min, and the total is about 9 min. (2) Set all three to the worst end: 98% disk, 50 processes, 10 big files. The process hunt is about 6 min, the disk check about 5 min, the gate about 14 min, and the total is about 30 min. Same host. Only the inputs changed.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Try it: edit the pipeline and check it

Here is the host-check script from this lab, ready to edit. Change it, then run the check. The check looks for the four things a working gate needs: a disk check, a readability check, a python3 check, and an exit with the failure count.

Try breaking it on purpose: change -ge 95 to -ge 5, or delete the exit line. Run the check and watch it fail. Then press Reset and try a change that keeps it passing, like adding a writability check.

Press "Run the check" to see whether this pipeline does what the lesson asks.

Expected output when it passes: The script checks the disk, readability, and python3, and exits with the failure count. On a healthy host it prints PASS lines and exits 0.

Basic to master

I take the same host four times here. First I look at it. Then I change it. Then I back it up. Last I gate on it.

Basic — Look at the box

Flags in the next block, letter by letter. I read every flag out loud before I run a command I copied. A flag I cannot explain is a flag I do not run yet.

  • ps -eo pid,comm,%cpu --sort=-%cpu | head -4 lists the busiest processes. The -e means every process. The -o takes the column list: pid, command name, cpu percent. The --sort=-%cpu sorts by CPU use, and the minus in front means highest first. Without the minus, the quietest process comes first, which is the wrong end of the list when I am hunting the busy one. The head -4 keeps the header and the top three processes.
  • df -h /tmp shows the space on one filesystem. The -h means human readable, MB and GB instead of 1K blocks. The /tmp at the end limits the answer to that filesystem. Without it, df lists every filesystem on the box.
  • du -sh . shows how much space the current directory holds. The -s means summary, one total line. The -h means human readable. Without -s, du lists every subdirectory too, and I have to find the total in the noise.
  • ls -l data/a.txt shows the permission bits as a string. The -l is the long listing. I read -rw-r--r-- left to right: file type, owner bits, group bits, others bits.
  • df -i shows inode use instead of byte use. The -i means inodes. A disk can have free bytes and zero free inodes, and then nothing can create a file. I check both numbers, because they are two different ways to be full.

Check it: run ps -eo pid,comm,%cpu --sort=-%cpu | head -4 and read the top line. Then run the same ps without the minus in --sort and see the quietest process on top instead.

Definition: ps lists the processes. df shows how full each disk is. du shows how big a folder is. ls -l shows the permission bits. These four commands are the first thing I run on any box I have never seen.

Types:

  • ps -eo, which lists processes with PID, name, and CPU.
  • df -h, which shows disk size, used, and percent.
  • du -sh, which shows how big one folder is.
  • ls -l, which shows the permission bits.

Uses: I use this for any new host, before I change anything. Try it now: run ps and df on your own box and read the top line of each.

Dimensions: At low volume everything is instant. What breaks first is reading the wrong column: %CPU is the busy column, and Use% is the full column.

Intermediate — Change the bits

Flags in the next block, letter by letter. I read every flag out loud before I run a command I copied. A flag I cannot explain is a flag I do not run yet.

  • chmod 640 data/a.txt sets the permission bits by number. 6 is read plus write for the owner, 4 is read only for the group, 0 is nothing for everyone else. The result reads -rw-r-----. I read the three digits as owner, group, others, in that order.
  • chmod 644 data/a.txt gives read back to others. The last digit changes from 0 to 4, and the string becomes -rw-r--r--. The difference between 640 and 644 is one digit and one group of readers.
  • chmod u+x file adds execute for the owner only. The u means user, the owner. The +x adds the execute bit. Plain chmod +x without the u would add execute for owner, group, and others, which is wider than I want for a plain file.
  • chown nobody:nogroup file changes the owner and the group in one move. The part before the colon is the new owner, the part after it is the new group. Without the colon and group, chown nobody file changes only the owner and leaves the group alone.
  • ls -l data/a.txt proves the change happened. The -l is the long listing. I read the bits before and after every chmod, because the command prints nothing and the listing is the only proof.

Check it: run chmod 640 data/a.txt && ls -l data/a.txt, then chmod 644 data/a.txt && ls -l data/a.txt. The last three letters change from --- to r-- and nothing else moves.

Definition: chmod changes the permission bits. 640 means owner read+write, group read, others nothing. 644 adds read for others. chown changes the owner. A file with the wrong bits is either unreadable to the people who need it, or readable to the people who should not.

Types:

  • chmod 640, which takes read away from others.
  • chmod 644, which gives read back to others.
  • ls -l, which proves the change happened.
  • chown, which changes who owns the file.

Uses: I use chmod when a file is too open or too closed, and I read the bits before and after. Try it now: run chmod 640 on the lab file, read the bits, and put them back.

Dimensions: As the team grows, permissions get harder, because more people need different access. What breaks is 777: it fixes the error today and opens the file to everyone forever.

Advanced — Back up and restore

Flags in the next block, letter by letter. I read every flag out loud before I run a command I copied. A flag I cannot explain is a flag I do not run yet.

  • tar czf demo.tar.gz data packs a directory into one compressed file. Read the letters one at a time. The c means create a new archive. The z means compress it with gzip. The f means the next word is the file name. Without the f, tar would write the archive to the screen instead of a file.
  • tar tzf demo.tar.gz | sort lists what is inside without unpacking it. The t means list. The z and f mean the same as above. I always list before I unpack, so I know what is about to land.
  • tar xzf demo.tar.gz -C restored --no-same-owner unpacks it into a chosen folder. The x means extract. The -C restored means change into the restored directory first, so the files land there and not in the current directory. The --no-same-owner means do not restore the original owner, keep the files owned by me. Without it, extracting an archive made by root would leave files owned by root in my folder.
  • diff -r data restored/data compares two directories file by file. The -r means recursive, walk into subdirectories. Without -r, diff can only compare two files. Silence from diff means the two trees match, and that silence is the proof the restore worked.
  • find . -size +10k -type f finds files bigger than 10 KB. The -size +10k means larger than 10 kilobytes. The + means more than. A - in front would mean smaller than, and no sign would mean exactly that size. The -type f means files only.

Check it: run the round trip, then run diff -r data restored/data. Diff should print nothing. Then change one file in the restore, run diff again, and watch it catch the change.

Definition: tar czf makes a compressed archive. tar tzf lists it. tar xzf extracts it. diff -r compares two folders. A backup is only real when the restore has been tested, and diff is the cheapest restore test there is.

Types:

  • tar czf, which compresses a folder into one file.
  • tar tzf, which lists the archive.
  • tar xzf -C, which extracts it into a chosen folder.
  • diff -r, which proves the restore matches.

Uses: I use tar for any folder I cannot afford to lose, and I restore it somewhere else to prove it. Try it now: run the round trip from step 4 and check diff is silent.

Dimensions: At ten times the size, tar still works, and the time grows with the bytes. What breaks is ownership: extracting as the wrong user gives files the wrong owner, and --no-same-owner is the fix in a sandbox.

Master — Gate the deploy on the host

Flags in the next block, letter by letter. I read every flag out loud before I run a command I copied. A flag I cannot explain is a flag I do not run yet.

  • chmod +x host-check.sh && bash -n host-check.sh makes the script executable and checks its syntax. The +x adds the execute bit. The -n on bash means no execution, parse only. Silence from bash means the syntax is clean. I run both before I run a script I just wrote.
  • ./host-check.sh runs the script. The ./ means run the file in this directory, because the current directory is not on the command search path.
  • echo $? prints the exit code of the script. The $? holds it. 0 means every check passed and the deploy may go. 1 means at least one check failed and the deploy stops. I read the exit code, not just the PASS and FAIL lines, because a gate the next script calls can only see the code.
  • df /tmp | awk 'NR==2 {print $5}' | tr -d '%' reads the disk use percent as a bare number. The NR==2 picks line 2, skipping the header. The $5 is the fifth field, the use percent. The tr -d '%' deletes the percent sign. The -d on tr means delete the named characters. Without it, the number would carry a % and the numeric compare after it would fail.
  • [ -r /tmp ] tests readable. [ -w /tmp ] tests writable. The brackets are the test command with a different spelling. Other tests I use often: -x executable, -f regular file, -d directory, -e exists, -s not empty.
  • command -v python3 >/dev/null checks whether python3 exists without printing anything. The -v prints the path when it exists and nothing when it does not. The >/dev/null throws the path away, because the script only needs the exit code.

Check it: run ./host-check.sh; echo $?. The exit code should be 0. Then run df -h /tmp and compare the percent with what the script printed.

Definition: At this level the host check is a deploy gate. It runs before the deploy, it prints PASS or FAIL per check, and it exits 0 or 1. A failing gate stops the deploy, and the rerun after the fix is the proof the fix worked.

Types:

  • A host-check script, which runs the checks in one place.
  • An exit code, which is the gate: 0 opens, 1 shuts.
  • A rerun, which proves the fix worked.
  • A named failure, which says which check failed and why.

Uses: I use this before every deploy to a host I do not own. Try it now: run ./host-check.sh, read the exit code, and decide whether you would deploy.

Dimensions: At a hundred hosts, the script is the same shape, and it runs on every host. What breaks is a gate nobody runs: a check that exists but is never executed protects nothing.

Go deeper

Flags in the next block, letter by letter. I read every flag out loud before I run a command I copied. A flag I cannot explain is a flag I do not run yet.

  • fallocate -l 100M /tmp/bloat.bin reserves 100 MB of disk at once. The -l takes the length. The file appears full size at once, without writing 100 MB of data. I use it to make a big file fast when I need to test a full disk. Deleting the file frees the space again.
  • df -h /tmp shows the space on one filesystem. The -h means human readable. Run it before and after the fallocate and watch the used number grow by about 100M.
  • du -sh /tmp/bloat.bin shows how much space one file holds. The -s means summary, one total line. The -h means human readable.
  • find /tmp -size +50M -type f finds files bigger than 50 MB. The -size +50M means larger than 50 megabytes. The + means more than. The -type f means files only.
  • rm /tmp/bloat.bin deletes the file. No flags. The disk space comes back at once.
  • mkdir -p ~/lab-host/data && cd ~/lab-host && echo hello > data/a.txt makes the lab folder and the first file. The -p means make the parents too. The && means run the next command only if this one worked. The > writes the word hello into the file.

Check it: run fallocate -l 100M /tmp/bloat.bin && df -h /tmp, then rm /tmp/bloat.bin && df -h /tmp. The used number should rise by 100M and fall back again.

The host-check script, the whole file. Do this: save it as host-check.sh, run chmod +x host-check.sh && bash -n host-check.sh, then ./host-check.sh. Expected output: four PASS lines and exit 0. On this lab box the disk line reads disk /tmp: 5%.

#!/bin/bash
set -u
fail=0
disk=$(df /tmp | awk 'NR==2 {print $5}' | tr -d '%')
echo "disk /tmp: $disk%"
if [ "$disk" -ge 95 ]; then echo "FAIL: disk $disk%"; fail=1; else echo "PASS: disk under 95%"; fi
if [ -r /tmp ]; then echo "PASS: /tmp readable"; else echo "FAIL: /tmp not readable"; fail=1; fi
if command -v python3 >/dev/null; then echo "PASS: python3 present"; else echo "FAIL: python3 missing"; fail=1; fi
if [ -w /tmp ]; then echo "PASS: /tmp writable"; else echo "FAIL: /tmp not writable"; fail=1; fi
exit "$fail"

Make a disk fill up on purpose, safely, in /tmp. Do this: run fallocate -l 100M /tmp/bloat.bin && df -h /tmp. Expected output: the Use% moves up at once. Then run du -sh /tmp/bloat.bin and find /tmp -size +50M -type f, and expect both to name bloat.bin. Delete it with rm /tmp/bloat.bin and watch df move back. A sparse file made with truncate does not move df, because it holds no blocks. fallocate really allocates, so it really fills.

What the tar flags mean: c creates, z compresses with gzip, f names the file, t lists, x extracts. Do this: run tar czf demo.tar.gz data, then tar tzf demo.tar.gz | sort. Expected output: data/ then data/a.txt. The sort makes the listing deterministic.

At 10x the hosts, the shape holds and the script runs on every host. At 100x, the design changes: the checks move into a monitoring agent, and the gate reads the agent instead of running the commands. The exit code never changes, though. 0 still opens the gate, and 1 still shuts it.

One time it broke

What broke

A deploy went out on a Friday to a host whose disk was 99% full. The deploy wrote its files, ran out of space halfway, and left the app half updated. The old code was gone, the new code was incomplete, and the rollback failed too, because the rollback also needed disk space. The host check that would have caught it existed. It had not been run in three weeks, because the deploy script called it only when someone remembered to add the flag.

Timeline

  • 16:00,the deploy started on a host nobody had checked that week.
  • 16:02,the copy ran out of disk space halfway through. The app was half old, half new.
  • 16:05,the app stopped answering. The log said no space left on device.
  • 16:10,the rollback failed for the same reason: it also needed space to write.
  • 16:30,an operator deleted an old log archive by hand, freed 4 GB, and reran the deploy from the start.

What changed after

Two things. First, the host check runs inside the deploy script, every time, with no flag to forget. Second, the check fails the deploy at 90%, not 95%, so there is room to roll back. Try it now: run ./host-check.sh and lower the limit in your copy to 90, then watch it still pass on a healthy host.

If you remember nothing else

Build it yourself

Build a host-check script for your own box with five checks: disk under 90%, /tmp readable, /tmp writable, python3 present, and no file over 50 MB in /tmp you cannot name. It passes when it prints five PASS lines and exits 0.

Build it, step by step

  • Write host-check.sh from this lesson and add the fifth check with find -size +50M.
  • Run bash -n on it, make it executable, and run it. Confirm four or five PASS lines and exit 0.
  • Make a 100 MB filler with fallocate in /tmp and rerun the check. Confirm it names the filler.
  • Delete the filler, rerun the check, and confirm it passes again.
  • Lower the disk limit to 90 in your copy and confirm it still passes on a healthy host.

What success looks like

Success looks like this: five PASS lines, exit 0, and a rerun after the filler is deleted that passes again. If the fifth check fails on a healthy host, the find path or size is wrong.

If it fails, check this

If df shows a different filesystem than you expected, check the mount point: /tmp may be its own filesystem. If find prints nothing, the size flag may be too big. If the script exits 1 with all PASS lines, the exit line is wrong. If fallocate is missing, the disk-full replay cannot run here, and the lesson says so.

Next in this series
Lesson 71 — Git