One machine, a hundred programs, and only one CPU to go around. Something has to decide who runs next, who gets the memory, and who talks to the disk. That something is the kernel, and it is not one thing. It is five parts that work together, and this lesson walks one read through all five of them. I learned the kernel the slow way, by watching a box go slow and blaming the wrong part first. The scheduler was the part I blamed last and the part that was guilty most often. Learn the five parts now, and the next slow box will take you minutes to read instead of hours.
Difficulty: Advanced · Know this first: Lesson F07, Lesson 98
By the end of this lesson you will have read the kernel's own numbers on your own machine, at small scale, and you will have the numbers to prove each hop works. First, read /proc/pressure/ and say what each of the three files means. Second, count context switches per second from /proc/stat and say whether the box is quiet or busy. Third, read the real scheduler knobs with sysctl and name the one knob that does not exist on a new kernel. Fourth, change a process's nice value with renice and watch the priority move. Fifth, count the system calls a trivial program makes with strace. You will also know where the kernel ends and the next lesson begins: allocation and the heap belong to Lesson 196.
What you need: a Linux machine with a shell. Every command in the lab ran on the build machine and the outputs shown are the outputs that ran. The perf step is reader-side, and the lab says so: perf is not installed in the build VM.
The kernel has one job: share one machine fairly. A hundred programs want the CPU, the memory, and the disk at the same time, and there is only one of each. So the kernel splits the job into five parts, and each part owns one piece of the sharing. Learn the five parts and you can read any slow box, because a slow box is always one of the five parts saying no.
| Part | What it does | The number I watch |
|---|---|---|
| Scheduler | Picks which process runs next, and for how long | Run queue length. A long queue means the CPUs are the bottleneck. |
| Memory manager | Maps each program's addresses to real pages in RAM | Pressure in /proc/pressure/memory. Stall time, not free space. |
| Drivers | Translate the kernel's general orders into one device's signals | dmesg. The driver writes its complaints there, and they are specific. |
| File system (VFS) | Turns a file name into a place on the disk | The lookup. A wrong path fails fast and honest. A slow path walks a deep tree or a dead mount. |
| IPC | Moves bytes and wakeups between programs | Who is waiting, and on what. A program that never wakes is waiting on the wrong thing. |
One more split runs underneath all five parts: user mode against kernel mode. A program runs in user mode, and user mode means it asks. It can use the CPU and its own memory, and nothing else. The kernel runs in kernel mode, and it can touch anything. The crossing between them is the system call, the door F07 taught first. Every read, every write, every open goes through that door. There is no other way in. This is the first lesson in the course that teaches the two modes by name, so take a second with it: the wall between them is what keeps one bad program from breaking the machine.
Lesson F07 taught the layers and the sharing idea without naming the parts. It taught that the kernel gives each process a turn, that switching costs a little time, that a driver is a translator, and that a system call is the door. This lesson names the five parts and shows each one working on a single read. Nothing here re-teaches F07. When a hop leans on something F07 already covered, the hop says so and moves on.
Write down, in one line each, what the scheduler decides, what the memory manager maps, what a driver translates, what the VFS turns a name into, and what IPC moves. Then write down which part you would check first if a box felt slow and the CPU looked idle. The answer is the run queue, and the scheduler owns it. If you wrote the disk first, you were checking the slowest part instead of the part that decides who waits.
The diagram above follows one read through the kernel. These four paths are the same idea drawn four ways. One read, done the steady way. One read, killed by memory pressure. One read, counted instead of timed. One read, answered from the cache. Run each path. One run takes 7.2 seconds across the path, so you can watch each hop light up in turn.
The return rule: the green dot only starts moving after the orange dot reaches the data hop. A return with no request is a lie. The diagram never draws one. The failure replay at the end of Run all four shows the other side of the rule. Break the scheduler, and the orange dot stops there and never comes back.
One program reads one file on a quiet box. The call crosses the door, the scheduler picks the reader, the map translates the address, the VFS resolves the name, the driver queues the read, the disk answers, and the wakeup carries the bytes home.
Not run yet.
Same read, one difference: memory is full and the reclaim fails. The memory manager cannot make room, so the kernel picks the biggest process and ends it. That is the OOM killer, and F07 named it first. The dot stops at Memory and never comes back. The read never gets its answer, because the reader is gone.
Not run yet.
The same read, counted instead of timed. Three system calls to get going, four processes waiting in the run queue, 1,204 pages mapped for the reader, 17 files open, 2 signals sent while the trip ran. The switches per second is the number I measured on the build machine just now. The other counts are a small teaching example.
Not run yet.
The same read, answered from the page cache instead of the disk. The call crosses the door, the scheduler picks the reader, the map translates the address, the VFS resolves the name, and the page is already in RAM. The driver and the disk are skipped. The fast path is not a different kernel. It is the same kernel with nothing left to wait for.
Not run yet.
Every hop charges rent in microseconds. The bar below is a worked example for one read through the kernel, not a measurement from a live system, but the shape is honest. The disk hop is the big one. It costs 180 µs at the simulator defaults, and the driver hop costs 60 µs. The other hops are plumbing. Switch the scenario and watch the bar change. In the runaway story the scheduler hop grows to 120 µs, because one process holds the CPU. In the memory pressure story the memory hop grows to 400 µs, because the manager is reclaiming pages to make room. Move a simulator slider and watch the scheduler hop and the memory hop move with it.
The budget bar above shows three fixed stories. This simulator lets you write your own. It answers the trade-off at the heart of this lesson: more runnable processes means a longer wait for a turn, a shorter slice means more switching, a loud process with a low nice value jumps the line, and memory pressure makes the map expensive. Move a slider and watch the scheduler hop, the memory hop, and the budget total change. The numbers are a teaching example, built from the formulas below, not live measurements. The shape is the lesson: the wait in the line costs more than the pick itself.
How many processes wait for a CPU. Four is a quiet box. Sixty-four is a line out the door.
How long one process runs before the scheduler picks again. A short slice shares fairly and switches often. A long slice switches rarely and makes everyone else wait.
The priority of the one loud process. A negative nice value means a higher priority, and the loud process jumps the line. Zero is neutral. A positive nice value means it waits its turn like everyone else.
How full memory is. Pressure makes the map expensive, because the manager reclaims pages to make room, and it makes the disk busy, because the disk writes those pages back.
Try this: (1) At the defaults, 4 processes, 6 ms slice, nice 0, no pressure, the scheduler hop is 8 µs and the memory hop is 12 µs. The budget total is 307 µs, and the box switches 2,267 times a second. (2) Drop to 1 process, 20 ms slice, nice 0, no pressure. The scheduler hop falls to 2 µs and the IPC hop to 9 µs. The budget total is 295 µs, and the switches fall to 450 a second. One process has nobody to share with. (3) Push to 32 processes, 1 ms slice, nice -10, pressure 80 percent. The scheduler hop grows to 124 µs, because the loud process jumps the line and the line is long. The memory hop grows to 332 µs. The budget total is 959 µs, and the switches reach 44,800 a second. The box is switching more than it is working. Raise the slice, quiet the loud process, or free some memory, and watch the total fall.
Formulas, in plain terms. Scheduler hop = 2 µs plus 2 µs per extra runnable process, plus 6 µs per nice point below zero. Memory hop = 12 µs plus 4 µs per pressure percent. Disk hop = 180 µs plus 2 µs per pressure percent. IPC hop = 9 µs plus 2 µs per extra runnable process. System call 2 µs, file system 25 µs, driver 60 µs, return 5 µs, fixed. Context switches per second = runnable processes times (1000 divided by the slice in ms, plus 400). The budget total is the nine hops added up. The timings are a teaching example, not live measurements. The shape will not change when the numbers do.
Do this now, on your own machine. You will read the pressure, count the context switches, read the real scheduler knobs, move a process down the line, and count the system calls a trivial program makes. Every output below is the output that ran on the build machine, and it is marked that way. Your numbers will differ. The shapes will not.
Run this on your own machine. It reads the three pressure files in order: cpu, io, memory.
cat /proc/pressure/cpu /proc/pressure/io /proc/pressure/memory
Expected output, exactly as it ran on this machine:
some avg10=0.37 avg60=6.13 avg300=7.76 total=543627782 full avg10=0.00 avg60=0.00 avg300=0.00 total=0 some avg10=0.74 avg60=6.16 avg300=5.92 total=272892126 full avg10=0.68 avg60=4.52 avg300=4.19 total=175815402 some avg10=0.00 avg60=0.15 avg300=0.22 total=21870338 full avg10=0.00 avg60=0.11 avg300=0.16 total=18401941
Read it as three files, two lines each. The first file is cpu, the second is io, the third is memory. The io full line is the one to watch here: avg10 0.68 means some task stalled on io for part of the last 10 seconds. The memory some avg10 is 0.00, so nothing stalled on memory. A full number above zero on memory is the loudest alarm in the file. It means the whole box stopped doing useful work to wait on memory.
Read the counter, wait 5 seconds, read it again. The difference is the switches in 5 seconds.
grep '^ctxt' /proc/stat; sleep 5; grep '^ctxt' /proc/stat
Expected output, exactly as it ran on this machine:
ctxt 71438070 ctxt 71458562
That is 20,492 switches in 5 seconds, or 4,098 per second. The box was quiet. Earlier the same day, while the box was busy, the same two reads gave about 14,000 per second. The number is the box, not the lesson. Run it twice, once quiet and once while something heavy runs, and you will see the number move with the load.
Read five knobs with one command. These are the real values on the build machine, not a tuning guide.
sysctl kernel.sched_rr_timeslice_ms kernel.sched_cfs_bandwidth_slice_us kernel.sched_autogroup_enabled vm.swappiness kernel.pid_max
Expected output, exactly as it ran on this machine:
kernel.sched_rr_timeslice_ms = 100 kernel.sched_cfs_bandwidth_slice_us = 5000 kernel.sched_autogroup_enabled = 1 vm.swappiness = 1 kernel.pid_max = 4194304
Now the honest catch. Try the two knobs every tuning post mentions, and watch them fail:
sysctl kernel.sched_min_granularity_ns kernel.sched_latency_ns
Expected output, exactly as it ran on this machine:
sysctl: cannot stat /proc/sys/kernel/sched_min_granularity_ns: No such file or directory sysctl: cannot stat /proc/sys/kernel/sched_latency_ns: No such file or directory
Those knobs do not exist on this kernel (7.0.0-44-generic). Knobs change between kernel versions. A tuning post written for one kernel version is a rumor on another. Read your own sysctl -a output before you change anything.
Start a sleep in the background, read its nice value, renice it, and read it again.
sleep 300 & ps -o pid,ni,comm -p $! renice -n 5 -p $! ps -o pid,ni,comm -p $!
Expected output, exactly as it ran on this machine:
PID NI COMMAND
34562 0 sleep
34562 (process ID) old priority 0, new priority 5
PID NI COMMAND
34562 5 sleep
The NI column moved from 0 to 5. The process is now nicer, which means it waits longer for a turn. One more read from the same process tells you how it switches:
grep -E 'voluntary_ctxt_switches|nonvoluntary_ctxt_switches' /proc/$P/status
Expected output, exactly as it ran on this machine:
voluntary_ctxt_switches: 2 nonvoluntary_ctxt_switches: 0
Voluntary 2 means the sleeping process gave the CPU up by itself twice. Nonvoluntary 0 means the scheduler never had to push it off. A sleeping process is polite. A hog is not, and the nonvoluntary number is where a hog shows up. Kill the sleep when you are done. Only root can renice to a negative value. A normal user can only make a process nicer, never meaner.
Count the system calls the program that does nothing makes.
strace -c -e trace=read,write,openat true
Expected output, exactly as it ran on this machine:
% time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 0.00 0.000000 0 1 read 0.00 0.000000 0 2 openat ------ ----------- ----------- --------- --------- ---------------- 100.00 0.000000 0 3 total
Even true, the program that does nothing, knocks on the door 3 times: one read, two opens. And one real read line, from strace -e trace=read,openat cat /etc/hostname, shows the shape of a single call: the program asked for 832 bytes of a library file and got 832 bytes back. The return value is the answer. A program that makes a million calls pays the door price a million times, and strace -c is how you find out it does.
There is no perf in this VM, so this step is yours to run where perf lives. The command is perf stat -e context-switches,page-faults sleep 1. It prints the switches and the page faults for one quiet second. No output is shown here, because none ran here. Run it, and compare the switches number with the one you counted in step 1. The two numbers should be in the same neighborhood. If they are far apart, one of the two measurements is measuring something else, and that is worth finding out before you trust either.
Change the nice value in step 3 from 5 to 10 and run it again. The NI column moves to 10. Then try renice -n -5 on the same process as a normal user and read the error. The kernel refuses, and the refusal is the rule: you can give up priority, you cannot take it.
Kill the sleep process. Nothing else to stop. The pressure files and the counters are read-only views. They reset themselves.
Three questions. No score is saved anywhere. This is just you against the material.
Knowing the kernel is not one skill. It is four levels of the same job, and every team I have worked with sits on one of them. The levels are not about how many knobs you can name. They are about whether you check the right part first when the box goes slow. Read them in order. Each level fixes the failure that hurt the most at the level below it.
At the basic level, you name the five parts out loud and follow one read through them: the program asks, the call crosses the door, the scheduler picks, the map translates, the VFS resolves the name, the driver translates, the disk answers, the wakeup lands, the answer comes home. You know user mode from kernel mode, and you know the system call is the only door between them. The failure that ends this level is blaming the disk for a wait that lives in the run queue.
Do this: run step 0 of the lab and write down what each of the three pressure files says. Then point at the diagram and name the hop where the file name becomes a place on the disk.
At the intermediate level, you read the run queue and say what a long one means. You move a nice value and watch the priority move with it. You count context switches per second and say whether the box is quiet or busy. You read pressure and say what stalled, instead of guessing from free memory. A hog is a priority problem before it is a capacity problem, and you stop buying CPU for a queue that a renice would drain.
Do this: run steps 1 and 3 of the lab. Write down the switches per second and the nice value before and after renice. Then say, in one sentence, what a long run queue means.
At the advanced level, you read the real knobs with sysctl and you know which ones do not exist on your kernel, because you checked. You can describe priority inversion: a low-priority process holds a lock, a high-priority process waits on it, and a medium one keeps running so the lock never frees. You read dmesg when a driver misbehaves, and you know a VFS error is fast and honest while a dead mount hangs. The failure that ends this level is tuning a knob from a blog post without reading your own kernel first.
Do this: run step 2 of the lab and write down the five knob values. Then write down which knob you would change first for a latency-sensitive service, and why, before you change anything.
At the master level, the kernel is a source of signals, not a black box. You alert on pressure and on the run queue, not on CPU percent alone, because CPU percent hides a queue. You watch the scheduler with eBPF and you can prove which process got picked and why. You tune one knob on one box, measure, and only then roll it out. The kernel is the floor the service stands on, and you treat it like a floor: measured, watched, and never tuned on a rumor.
Do this: write one alert on memory pressure some avg10 over 20 for five minutes, and one alert on the run queue staying long for five minutes. Then write one paragraph on what you would watch with eBPF (Lesson 126) to prove the alert fires for the right reason.
A program in user mode can use the CPU and its own memory and nothing else. The kernel runs in kernel mode and can touch anything. The switch between them is the trap, and it costs a couple of microseconds each way. That cost is why a program that makes a system call per byte is slow and a program that reads in big chunks is fast. The door is cheap. Knocking on it a million times is not.
The mode switch, the argument check, the work, the copy back. The lab counted 3 calls for true. A web server makes thousands per request. Count them before you optimize anything else, because the count is the first number that tells you whether the program is chatty. The syscall hop in this lesson is 2 µs. A million calls is two seconds of door, and the door is the cheapest part of the trip.
The Linux scheduler is called CFS, the Completely Fair Scheduler, and fair means each process gets time in proportion to its weight. Nice sets the weight. A process at nice -10 does not get all the CPU. It gets a bigger share than a process at nice 0, and the difference is the weight table, not a fixed number. Preemption is the scheduler taking the CPU back when a slice ends. A context switch is the save and the load: registers, stack pointer, page tables. It costs a little time each time, and F07 said so first.
The lab caught it live. sched_latency_ns and sched_min_granularity_ns do not exist on kernel 7.0.0-44-generic. The knobs that do exist here are sched_rr_timeslice_ms (100), sched_cfs_bandwidth_slice_us (5000), and sched_autogroup_enabled (1). A tuning post written for one kernel version is a rumor on another. Read your own sysctl -a output. Tune one knob on one box, measure, then roll it out.
Read first with sysctl <name>, change with sysctl -w <name>=<value>, and only write it to a file under /etc/sysctl.d/ when the measurement says it helped. vm.swappiness = 1 on the build machine means the kernel almost never swaps. That is a choice for a box with plenty of RAM. A box that swaps under load wants a different number, and the pressure files are how you tell. The kernel-parameters section of Lesson 98 is the tuning reference for this course. This lesson points there instead of repeating it.
Three files, two lines each. some means at least one task stalled. full means every non-idle task stalled, so nothing productive happened at all. The three averages are 10, 60, and 300 seconds. The total is microseconds of stall since boot. A full number above zero on memory is the loudest alarm in the file. It means the whole box stopped doing useful work to wait on memory. F07 taught the pressure files first. This lesson is where you learn to read them as a stall clock.
eBPF lets you watch the scheduler, the syscalls, and the pressure without changing the kernel. One paragraph is all this lesson needs, because Lesson 126 teaches eBPF properly: hooks, maps, the verifier, and the tools people already wrote. If you want to see which process the scheduler picks and why, that is an eBPF program. Go read Lesson 126, then come back and point it at the run queue.
A low-priority process holds a lock. A high-priority process waits on the lock. A medium-priority process keeps running, so the low one never releases the lock, and the high one waits forever. The priority order is upside down, and the scheduler cannot see it, because the scheduler sees priorities and the lock is invisible to it. The fix is in the lock, not the scheduler: priority inheritance, or a shorter critical section. The incident below is this shape.
One front door, many file systems. The VFS turns a path into an inode, and the inode belongs to ext4, tmpfs, or a network mount underneath. The program never knows which. That is the point. When a mount dies, the VFS still answers, and the answer is an error, fast, instead of a hang. The files themselves were taught in Lesson F09. This hop is the kernel half: the name becoming a place.
A driver that fails returns an error, and the kernel passes it up. A driver that hangs is worse, and dmesg is where it confesses. Load, probe, fail: the module loads, the probe asks the hardware if it is there, and if the answer is no, the driver unloads and the device stays dark. Check dmesg before you reinstall anything. The translator framing is F07's, and it still fits: the driver turns the kernel's general read into the exact signals one piece of hardware understands.
At 10x the processes, the run queue is the bottleneck, and the scheduler hop is the number that grows. The fix is not a faster scheduler. It is fewer runnable processes, or more CPUs, or a hog with its nice value fixed. At 100x, the pressure files become the dashboard, because memory is the scarce thing and the reclaim is the tax. The five parts do not change. The part that says no changes, and the numbers in this lesson tell you which one it is.
A batch job that rebuilt a search index every night started running at the same time as the evening web traffic. A deploy had set the batch job to nice -10, to make it finish faster. The web requests slowed down. The p99 latency doubled, then tripled. The CPUs were not full. The disks were not busy. The box looked fine on every dashboard except the one nobody had: the run queue.
Two things. First, the fix. Batch jobs run at nice 10 or higher on shared boxes, and the deploy template enforces it. A deploy that sets a negative nice value fails the check. Second, the rule. The run queue is on the dashboard now, next to CPU, and the alert fires when the queue stays long for five minutes. CPU at 60 percent with a queue of forty is a busy box pretending to be idle. The queue is the number that tells the truth.
Design the kernel-side view for a latency-sensitive web service that shares one box with a nightly batch job. The web service must answer in under 50 ms at p99. The batch job must finish by morning. The box has 4 CPUs and 16 GB of RAM. Name the priorities, the run queue alert, the pressure alert, and what you watch with eBPF. Say what breaks first at 10x traffic.
Start with the priorities. The web service runs at nice 0, the batch job at nice 10, and the deploy template refuses any negative nice value on a shared box. The batch job gets the leftover CPU, and leftover is enough, because the night is long and the web traffic is low after midnight.
Then the numbers. At 4 CPUs the run queue should stay under 8. The alert fires when it holds above 8 for five minutes, because a queue twice the CPU count means someone is waiting. Memory pressure some avg10 over 20 for five minutes pages the on-call engineer, because that is tasks stalling, not memory filling up. eBPF watches the scheduler: which process gets picked, how long each one waits, and whether the batch job ever jumps the line.
Then what breaks first at 10x. The run queue. Ten times the web traffic means ten times the runnable processes, and 4 CPUs cannot hide that. The fix is not a nicer batch job. It is a second box, or the batch job moving to its own machine, before the traffic arrives.
Trade-offs to name. Priority against throughput: a nice batch job finishes later, and later must still be before morning. Sharing against isolation: one box is cheaper than two, and the queue is the price. Alerting on pressure against alerting on usage: pressure pages when work stalls, usage pages when a number crosses a line, and only one of those is a user feeling slow.
Saved in this browser, so the Lessons index can show your progress across the course.