Here is the whole lesson in one sentence. Lesson 97 taught you how a Linux box is put together. This lesson teaches you what to do when that box misbehaves at 2 AM, and how to tell whether the problem is the network, the CPU, the memory, the disk, or a limit you set yourself and forgot about. I learned this the way I learn most things in this job. A box went slow after a deploy, the dashboard blamed the network, the network team blamed the app, and the app blamed the box. The box was right and all three of us were wrong in different ways. The real cause was a process that had quietly eaten every file descriptor it was allowed, and nothing in the dashboard said so in plain words. This lesson walks one slow box through the whole debug, hop by hop, so you know exactly which command to run next and what its output is actually telling you.
Every hop charges rent in minutes. The bar below is a worked example for one slow request's trip through the box, not a measurement from a live system, but the shape is honest: the commands run in seconds, and the reading, deciding, and proving take the time. The minutes are the time I spend walking the request's path and checking each hop, not the request's travel time. The request crosses the box in milliseconds. The thinking is the bill. Step through the lesson or switch the scenario and watch the bar change. The black hole scenario costs the most, and most of that cost is proving a negative: every local check says fine while the packets vanish. The runaway process costs the least, because a single loud suspect points at itself early. Same box, same slow request, same first hop. Only the cause changed, and the bill changed with it.
The budget bar above shows three fixed stories. This simulator lets you write your own. It is a slow box with four dials, and each dial is one of the numbers from the debug above. Move a slider and watch the diagnosis time, the latency, and the verdict change. The numbers are a teaching example, not live measurements. The shape is the lesson: the biggest dial is the bottleneck, and fixing anything else is wasted work. The box is 4 CPU cores, 16 GB memory, one app, one container. Baseline healthy numbers: load 1.0, swap 0 percent, disk wait 2 percent, open file descriptors 2,000.
The CPU share of the diagnosis time and the latency. CPU time is 0.5 minutes per unit of load, and CPU adds 8 ms of latency per unit of load. A load of 8 on a 4-core box means the CPUs are twice over-subscribed.
The memory share of the diagnosis time and the latency. Memory time is 0.1 minutes per percent of swap used, and swap adds 1.5 ms of latency per percent. Swap is disk pretending to be memory, so every percent of it is slow.
The disk share of the diagnosis time and the latency. Disk time is 0.2 minutes per percent of I/O wait, and disk wait adds 2 ms of latency per percent. High wait means the disk is the queue.
The file-handle share of the diagnosis time and the latency. File time is 0.2 minutes per thousand open descriptors, and open descriptors add 1 ms of latency per thousand. A process near its limit spends its time failing to open new handles.
Try this: (1) Set swap to 0 and disk wait to 0 and load to 2 and file descriptors to 5. Latency falls to about 61 ms and the verdict reads healthy. That is what a boring box looks like. (2) Set load to 32 and leave the rest at default. Latency rises to about 448 ms, the verdict reads saturated, and CPU becomes the bottleneck. Same box, one dial. (3) Set disk wait to 90 and leave the rest at default. Latency rises to about 386 ms and disk becomes the bottleneck, even though CPU and memory did not move. The biggest bar is the fix.
Three questions. No score is saved anywhere — this is just you against the material.
The same box, four times, from first principles to the level where you can run it in production and debug it at 2 AM. Each level builds on the one before it. Skip a level and the next one will feel like magic, and magic breaks at 2 AM.
Definition: At this level, advanced Linux means reading the current state of one box with four commands. Uptime tells you the load. Free tells you the memory and swap. Df tells you the disk. Top tells you which processes are busiest. That is the whole level. You are not tuning anything. You are looking.
Types: The four reads at this level are load, memory, disk, and processes. Load is how many processes are waiting. Memory is what is used and what is swapped. Disk is space used and how busy the disk is. Processes are who owns the CPU and the memory.
Uses: I use this level every time an alert fires, before I touch anything. It is the first five minutes of any debug. It is also what I teach a new engineer first, because a wrong guess at this level costs an hour, and a right read costs five minutes.
Dimensions: Scale: one box, one reader, no tooling needed. Failure modes: the numbers are a snapshot, so a spike that already passed looks like nothing happened, and a slow leak looks normal until it is not. Cost: five minutes and four commands. Trade-offs: you get the truth about right now and nothing about why. That is enough to start, and never enough to finish.
Definition: At this level, you connect the symptom to a cause. You read the logs for the out-of-memory killer and the failed writes. You run ss to see the real connections, ip route to see where packets go, and a DNS lookup to split a network problem from an app problem. You check the file descriptor limit on the actual process, not the number you remember from the config.
Types: The tools at this level are log reading, connection listing with ss, routing with ip route, DNS testing, and limit checking with lsof and proc. Each one answers one question: is it the disk, the network, DNS, or a limit?
Uses: I use this level when the basic reads say the box is busy but not why. It is the middle of the debug in this lesson, hops 4 through 6. It is also the level where most on-call work actually happens, because most incidents are one of these four causes.
Dimensions: Scale: one box and the network next to it, still by hand. Failure modes: DNS is the classic trap, because a slow lookup looks like a slow service, and a routing mistake looks like a dead service. Cost: ten to twenty minutes and a habit of checking the second source before believing the first. Trade-offs: you get a cause, but only for the problems these tools can see. A kernel-level problem stays invisible at this level.
Definition: At this level, you ask the kernel what a process is actually doing. Strace shows the system calls, one by one. Perf shows where the CPU time goes. The proc filesystem shows the cgroup limits and counters for a container. Namespaces explain why a process sees a different network or a different process list than you do from the host.
Types: The tools at this level are strace for system calls, perf for CPU sampling, proc and sysfs for cgroup limits and counters, and namespace inspection with lsns and nsenter. Each one reads kernel state, not app state.
Uses: I use this level when the intermediate tools point at a suspect but cannot prove it. A process spinning on a failed open call, a container throttled by a cgroup limit I set myself, a process in a network namespace with its own routing table. Those are advanced-level answers, and they only come from kernel-level tools.
Dimensions: Scale: one box, one process, and the kernel between them. Failure modes: strace slows the process it watches, so a long attach makes the problem worse, and perf output is easy to misread if you do not know what a flame graph is actually counting. Cost: the learning curve is real, and a wrong read here leads to a confident wrong fix. Trade-offs: you get proof instead of suspicion, but you pay for it in care. These tools are sharp, and sharp tools cut the user as well as the problem.
Definition: At this level, you stop debugging one box and start running a fleet that debugs itself partway. You tune kernel parameters with sysctl and keep them in version control. You harden SSH, set up fail2ban, and write the runbook before the incident, not during it. You use eBPF to watch the kernel safely in production, and you choose latency or throughput on purpose, knowing what each choice costs.
Types: The work at this level is kernel tuning with sysctl, hardening with SSH config and fail2ban, observability with eBPF tools, and runbooks plus alerts that point at the right first command. The tools change. The discipline does not.
Uses: I use this level when the same incident happens twice. The second time, the fix is already written down, the alert already names the first command, and the tuning is already in version control. That is the difference between an engineer who debugs well and a team that does not have to.
Dimensions: Scale: a fleet, not a box, so a bad sysctl value ships everywhere at once if you are careless, and a good one fixes everywhere at once if you are not. Failure modes: a tuning change that helps latency hurts throughput, a fail2ban rule that locks out a real user, an eBPF program that watches everything and drowns the team in data. Cost: review, rollout, and rollback plans for every change, because a fleet multiplies both the fix and the mistake. Trade-offs: you trade speed for safety on every change, and you accept that trade, because at fleet scale a fast wrong change is an outage, and a slow right change is Tuesday.
The debug above is one box, worked by hand. Production is the rest of it. Here is what sits underneath the nine hops once you can run them without thinking.
eBPF is a way to run small programs inside the kernel, safely, without changing the kernel. You write a small program that watches something, like every time a process opens a file or a packet leaves an interface, and the kernel runs it and hands you the counts. I use it when strace is too slow and the logs say nothing. The classic first tool is a one-liner that counts file opens by process, and it answers in seconds a question that used to take an hour of guessing. The warning is simple. eBPF reads the kernel, so it needs privilege, and a badly written program can cost more than the problem it is measuring. I run the packaged tools first, and I write my own program only when the packaged tool does not answer the question.
Latency and throughput are a trade, and you pick one on purpose. A box tuned for latency answers fast and does less total work, because it wakes up often, interrupts often, and never lets a queue build. A box tuned for throughput does more total work and answers slower, because it batches, buffers, and lets queues build on purpose. I tune for latency on the path a user waits on, and for throughput on the path that runs in the background. The knobs are small and the effect is large: interrupt coalescing, the CPU frequency governor, the disk scheduler, the TCP buffers. I change one knob, measure, and write the number down. Two knobs at once means I never learn which one helped.
Most kernel parameters never need touching. Four of them come up in real incidents. The first is the file descriptor limit, because a busy service hits it quietly and the error looks like a network failure. The second is the TCP backlog and the ephemeral port range, because a box that opens many short connections runs out of ports before it runs out of anything else. The third is swappiness, because a box that swaps eagerly is a box that chose the disk over the memory, and on a database host that is the wrong choice. The fourth is the out-of-memory killer's behavior, because the kernel picks a victim when memory runs out, and I would rather it pick the process I can lose than the one I cannot. All four live in sysctl, all four belong in version control, and none of them get changed on a live box without a rollback plan.
SSH is the front door, and I treat it like one. Password logins off, key logins only. Root login off. The port stays where it is, because moving the port is not security, it is a hope that nobody looks. Fail2ban watches the logs and blocks an address after a few failed logins, which stops the bots that try every password they know. I check the fail2ban log after any login problem, because a real user locked out by fail2ban looks exactly like a user with the wrong key, and the fix is unblocking an address, not resetting a key. Hardening is not a project. It is a config file, a review, and a test login from outside before I call it done.
At 10x the fleet, a box stops being a box and becomes a number in a spreadsheet. I stop debugging by logging in and start debugging by reading the same four numbers from every box at once: load, swap, I/O wait, open descriptors. The simulator in this lesson is the shape of that dashboard. One dial per resource, one bottleneck per box, and the fix is always the biggest bar. At 100x, the tuning itself becomes the risk. A sysctl change that helps one service hurts another, so tuning moves out of the box and into the image build, reviewed like code, rolled out like a deploy, and rolled back the same way. The runbook stops being a document and becomes an alert that names the first command. The lesson does not change at that scale. The scale changes who reads it, and how fast a mistake travels.
# /etc/sysctl.d/99-latency.conf — latency-sensitive host, reviewed like code # file handles: a busy service hits the default limit quietly fs.file-max = 2097152 # swap: prefer real memory, swap only when truly out vm.swappiness = 10 # TCP: short connections need ports and a backlog that holds net.ipv4.ip_local_port_range = 1024 65535 net.core.somaxconn = 4096 # OOM: let the kernel kill, but log it so we see the victim vm.oom_kill_allocating_task = 0
This is a sysctl file, the way I keep kernel tuning: one file, one host role, in version control, with a comment on every line saying why the value is what it is. File-max is high because a busy service opens many handles and the default is too low. Swappiness is 10 because this host has real memory and should use it before touching disk. The port range is wide and the backlog is deep because short connections eat ports fast. The out-of-memory setting stays at the default, because I want the kernel to pick the victim and log it, not kill the process that happened to be allocating at the wrong second. Five settings, each doing one job, and the whole file is reviewable in one screen. That is how I want tuning kept: in the open, in version control, not typed into a live box at 2 AM and forgotten.
This one stayed with me because the breakage was not a bug. It was a door, built on purpose, inside a tool almost every Linux box already trusted.
In March 2024, a developer named Andres Freund noticed his SSH logins were taking half a second longer than they should, and he chased a feeling instead of ignoring it. What he found was a backdoor hidden inside xz utils, versions 5.6.0 and 5.6.1, a compression library that ships on nearly every Linux system. The backdoor was aimed at OpenSSH through the way some distributions link it with systemd. On an affected system, a remote attacker with the right key could have run commands as the SSH service. The code was careful, patient, and spread across test files where nobody looks. It was caught because one person measured a half-second delay and refused to let it go.
Three things, and they are all about trust, not tools. First, release tarballs are now checked against the source repository, because the backdoor lived in the tarball and not in the code anyone reviewed. A build from a tarball nobody diffed is a build from a stranger. Second, distributions and projects tightened who can cut a release and how a new maintainer earns that access, because two years of patient trust-building beat every automated check in place. Third, and this is the one I took home, I now treat a small unexplained slowdown as evidence, not noise. A half-second SSH delay was the only symptom, and it was enough. The lesson of this whole course, in one incident: measure, write the number down, and chase the number that does not fit, even when it is small.
This is the kind of question I would ask in an interview, and the kind I ask myself before I sign off on a real fleet. Take a minute with it before you read my approach.
Design the Linux tuning and debug runbook for a fleet of 500 hosts running a latency-sensitive API. The API has a p99 latency budget of 200 ms. Hosts run the app in containers, 4 cores and 16 GB each. You have one week, one on-call rotation, and a rule: no tuning change ships without a measurement before and after. Sketch the tuning, the runbook, and what you would measure.
Start with the numbers from this lesson. One host at the simulator defaults is 256 ms p95, which is already over a 200 ms budget, and I say so honestly. The fix is the bottleneck, which is swap at 90 ms of the latency, so the first tuning is swappiness 10 and a memory limit the container cannot cross quietly. For the fleet, Little's Law sets the concurrency: 500 hosts each handling 100 requests a second at 0.2 seconds in the system is 10,000 requests inside at once across the fleet, 20 per host, so I plan the file descriptor limit per container for 20 live connections plus headroom, not for a guess. The rollout is 5 hosts first, then 50, then the rest, with the same four numbers read from every host before and after: load, swap, I/O wait, open descriptors. If the p99 does not move on the first 5, the rollout stops.
Latency tuning buys fast answers and costs total work, because the box wakes up often and never batches. Throughput tuning buys total work and costs latency, so I tune the user-facing path for latency and the background path for throughput, and I never tune both the same way. A cgroup memory limit buys protection for the neighbors and costs a killed container when the limit is wrong, so the limit is set from measured peak usage plus headroom, not from a round number. Fail2ban buys quiet logs and costs a locked-out real user when the rule is too tight, so the unblock path is written into the runbook. And the rollout is staged because a tuning change that helps 5 hosts and hurts 500 is a fleet outage, and the only thing standing between the two is the size of the first batch.
Saved in this browser, so the Lessons index can show your progress across the course.