Platform Engineering · Lesson 195

How the Kernel Works

One machine, a hundred programs, and only one CPU to go around. Something has to decide who runs next, who gets the memory, and who talks to the disk. That something is the kernel, and it is not one thing. It is five parts that work together, and this lesson walks one read through all five of them. I learned the kernel the slow way, by watching a box go slow and blaming the wrong part first. The scheduler was the part I blamed last and the part that was guilty most often. Learn the five parts now, and the next slow box will take you minutes to read instead of hours.

Difficulty: Advanced · Know this first: Lesson F07, Lesson 98

What you will build

By the end of this lesson you will have read the kernel's own numbers on your own machine, at small scale, and you will have the numbers to prove each hop works. First, read /proc/pressure/ and say what each of the three files means. Second, count context switches per second from /proc/stat and say whether the box is quiet or busy. Third, read the real scheduler knobs with sysctl and name the one knob that does not exist on a new kernel. Fourth, change a process's nice value with renice and watch the priority move. Fifth, count the system calls a trivial program makes with strace. You will also know where the kernel ends and the next lesson begins: allocation and the heap belong to Lesson 196.

What you need: a Linux machine with a shell. Every command in the lab ran on the build machine and the outputs shown are the outputs that ran. The perf step is reader-side, and the lab says so: perf is not installed in the build VM.

The story: one read, five parts

The kernel has one job: share one machine fairly. A hundred programs want the CPU, the memory, and the disk at the same time, and there is only one of each. So the kernel splits the job into five parts, and each part owns one piece of the sharing. Learn the five parts and you can read any slow box, because a slow box is always one of the five parts saying no.

PartWhat it doesThe number I watch
SchedulerPicks which process runs next, and for how longRun queue length. A long queue means the CPUs are the bottleneck.
Memory managerMaps each program's addresses to real pages in RAMPressure in /proc/pressure/memory. Stall time, not free space.
DriversTranslate the kernel's general orders into one device's signalsdmesg. The driver writes its complaints there, and they are specific.
File system (VFS)Turns a file name into a place on the diskThe lookup. A wrong path fails fast and honest. A slow path walks a deep tree or a dead mount.
IPCMoves bytes and wakeups between programsWho is waiting, and on what. A program that never wakes is waiting on the wrong thing.

One more split runs underneath all five parts: user mode against kernel mode. A program runs in user mode, and user mode means it asks. It can use the CPU and its own memory, and nothing else. The kernel runs in kernel mode, and it can touch anything. The crossing between them is the system call, the door F07 taught first. Every read, every write, every open goes through that door. There is no other way in. This is the first lesson in the course that teaches the two modes by name, so take a second with it: the wall between them is what keeps one bad program from breaking the machine.

Lesson F07 taught the layers and the sharing idea without naming the parts. It taught that the kernel gives each process a turn, that switching costs a little time, that a driver is a translator, and that a system call is the door. This lesson names the five parts and shows each one working on a single read. Nothing here re-teaches F07. When a hop leans on something F07 already covered, the hop says so and moves on.

Do this now, for each of the five parts

Write down, in one line each, what the scheduler decides, what the memory manager maps, what a driver translates, what the VFS turns a name into, and what IPC moves. Then write down which part you would check first if a box felt slow and the CPU looked idle. The answer is the run queue, and the scheduler owns it. If you wrote the disk first, you were checking the slowest part instead of the part that decides who waits.

Run it as:

When it breaks, I check

One thing to remember

Four paths, one family

The diagram above follows one read through the kernel. These four paths are the same idea drawn four ways. One read, done the steady way. One read, killed by memory pressure. One read, counted instead of timed. One read, answered from the cache. Run each path. One run takes 7.2 seconds across the path, so you can watch each hop light up in turn.

The return rule: the green dot only starts moving after the orange dot reaches the data hop. A return with no request is a lie. The diagram never draws one. The failure replay at the end of Run all four shows the other side of the rule. Break the scheduler, and the orange dot stops there and never comes back.

Path 1: the steady read

Disk: reached

One program reads one file on a quiet box. The call crosses the door, the scheduler picks the reader, the map translates the address, the VFS resolves the name, the driver queues the read, the disk answers, and the wakeup carries the bytes home.

Program0 µs
Syscall2 µs
Scheduler8 µs
Memory12 µs
VFS25 µs
Driver60 µs
Disk180 µs
Answer5 µs

Not run yet.

Path 2: the OOM kill

Answer: never comes

Same read, one difference: memory is full and the reclaim fails. The memory manager cannot make room, so the kernel picks the biggest process and ends it. That is the OOM killer, and F07 named it first. The dot stops at Memory and never comes back. The read never gets its answer, because the reader is gone.

Program0 µs
Syscall2 µs
Scheduler8 µs
Memoryreclaim fails
OOM killerpicks the biggest

Not run yet.

Path 3: the count trail

Time: skipped

The same read, counted instead of timed. Three system calls to get going, four processes waiting in the run queue, 1,204 pages mapped for the reader, 17 files open, 2 signals sent while the trip ran. The switches per second is the number I measured on the build machine just now. The other counts are a small teaching example.

Syscalls3 calls
Run queue4 waiting
Pages mapped1,204 pages
Files open17 open
Signals sent2 sent
Switches/sec4,098/sec

Not run yet.

Path 4: the cache hit

Driver and disk: skipped

The same read, answered from the page cache instead of the disk. The call crosses the door, the scheduler picks the reader, the map translates the address, the VFS resolves the name, and the page is already in RAM. The driver and the disk are skipped. The fast path is not a different kernel. It is the same kernel with nothing left to wait for.

Program0 µs
Syscall2 µs
Scheduler8 µs
Memory12 µs
VFS25 µs
Page cachehit
Answer5 µs

Not run yet.

Where the time goes

Every hop charges rent in microseconds. The bar below is a worked example for one read through the kernel, not a measurement from a live system, but the shape is honest. The disk hop is the big one. It costs 180 µs at the simulator defaults, and the driver hop costs 60 µs. The other hops are plumbing. Switch the scenario and watch the bar change. In the runaway story the scheduler hop grows to 120 µs, because one process holds the CPU. In the memory pressure story the memory hop grows to 400 µs, because the manager is reclaiming pages to make room. Move a simulator slider and watch the scheduler hop and the memory hop move with it.

Run queue or time slice: move the sliders and watch the wait

The budget bar above shows three fixed stories. This simulator lets you write your own. It answers the trade-off at the heart of this lesson: more runnable processes means a longer wait for a turn, a shorter slice means more switching, a loud process with a low nice value jumps the line, and memory pressure makes the map expensive. Move a slider and watch the scheduler hop, the memory hop, and the budget total change. The numbers are a teaching example, built from the formulas below, not live measurements. The shape is the lesson: the wait in the line costs more than the pick itself.

4 processes

How many processes wait for a CPU. Four is a quiet box. Sixty-four is a line out the door.

6 ms

How long one process runs before the scheduler picks again. A short slice shares fairly and switches often. A long slice switches rarely and makes everyone else wait.

0

The priority of the one loud process. A negative nice value means a higher priority, and the loud process jumps the line. Zero is neutral. A positive nice value means it waits its turn like everyone else.

0%

How full memory is. Pressure makes the map expensive, because the manager reclaims pages to make room, and it makes the disk busy, because the disk writes those pages back.

Try this: (1) At the defaults, 4 processes, 6 ms slice, nice 0, no pressure, the scheduler hop is 8 µs and the memory hop is 12 µs. The budget total is 307 µs, and the box switches 2,267 times a second. (2) Drop to 1 process, 20 ms slice, nice 0, no pressure. The scheduler hop falls to 2 µs and the IPC hop to 9 µs. The budget total is 295 µs, and the switches fall to 450 a second. One process has nobody to share with. (3) Push to 32 processes, 1 ms slice, nice -10, pressure 80 percent. The scheduler hop grows to 124 µs, because the loud process jumps the line and the line is long. The memory hop grows to 332 µs. The budget total is 959 µs, and the switches reach 44,800 a second. The box is switching more than it is working. Raise the slice, quiet the loud process, or free some memory, and watch the total fall.

Formulas, in plain terms. Scheduler hop = 2 µs plus 2 µs per extra runnable process, plus 6 µs per nice point below zero. Memory hop = 12 µs plus 4 µs per pressure percent. Disk hop = 180 µs plus 2 µs per pressure percent. IPC hop = 9 µs plus 2 µs per extra runnable process. System call 2 µs, file system 25 µs, driver 60 µs, return 5 µs, fixed. Context switches per second = runnable processes times (1000 divided by the slice in ms, plus 400). The budget total is the nine hops added up. The timings are a teaching example, not live measurements. The shape will not change when the numbers do.

Lab: read the kernel yourself

Do this now, on your own machine. You will read the pressure, count the context switches, read the real scheduler knobs, move a process down the line, and count the system calls a trivial program makes. Every output below is the output that ran on the build machine, and it is marked that way. Your numbers will differ. The shapes will not.

Step 0: read the pressure

Run this on your own machine. It reads the three pressure files in order: cpu, io, memory.

cat /proc/pressure/cpu /proc/pressure/io /proc/pressure/memory

Expected output, exactly as it ran on this machine:

some avg10=0.37 avg60=6.13 avg300=7.76 total=543627782
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
some avg10=0.74 avg60=6.16 avg300=5.92 total=272892126
full avg10=0.68 avg60=4.52 avg300=4.19 total=175815402
some avg10=0.00 avg60=0.15 avg300=0.22 total=21870338
full avg10=0.00 avg60=0.11 avg300=0.16 total=18401941

Read it as three files, two lines each. The first file is cpu, the second is io, the third is memory. The io full line is the one to watch here: avg10 0.68 means some task stalled on io for part of the last 10 seconds. The memory some avg10 is 0.00, so nothing stalled on memory. A full number above zero on memory is the loudest alarm in the file. It means the whole box stopped doing useful work to wait on memory.

Step 1: count the context switches

Read the counter, wait 5 seconds, read it again. The difference is the switches in 5 seconds.

grep '^ctxt' /proc/stat; sleep 5; grep '^ctxt' /proc/stat

Expected output, exactly as it ran on this machine:

ctxt 71438070
ctxt 71458562

That is 20,492 switches in 5 seconds, or 4,098 per second. The box was quiet. Earlier the same day, while the box was busy, the same two reads gave about 14,000 per second. The number is the box, not the lesson. Run it twice, once quiet and once while something heavy runs, and you will see the number move with the load.

Step 2: read the real scheduler knobs

Read five knobs with one command. These are the real values on the build machine, not a tuning guide.

sysctl kernel.sched_rr_timeslice_ms kernel.sched_cfs_bandwidth_slice_us kernel.sched_autogroup_enabled vm.swappiness kernel.pid_max

Expected output, exactly as it ran on this machine:

kernel.sched_rr_timeslice_ms = 100
kernel.sched_cfs_bandwidth_slice_us = 5000
kernel.sched_autogroup_enabled = 1
vm.swappiness = 1
kernel.pid_max = 4194304

Now the honest catch. Try the two knobs every tuning post mentions, and watch them fail:

sysctl kernel.sched_min_granularity_ns kernel.sched_latency_ns

Expected output, exactly as it ran on this machine:

sysctl: cannot stat /proc/sys/kernel/sched_min_granularity_ns: No such file or directory
sysctl: cannot stat /proc/sys/kernel/sched_latency_ns: No such file or directory

Those knobs do not exist on this kernel (7.0.0-44-generic). Knobs change between kernel versions. A tuning post written for one kernel version is a rumor on another. Read your own sysctl -a output before you change anything.

Step 3: move a process down the line

Start a sleep in the background, read its nice value, renice it, and read it again.

sleep 300 &
ps -o pid,ni,comm -p $!
renice -n 5 -p $!
ps -o pid,ni,comm -p $!

Expected output, exactly as it ran on this machine:

    PID  NI COMMAND
  34562   0 sleep
34562 (process ID) old priority 0, new priority 5
    PID  NI COMMAND
  34562   5 sleep

The NI column moved from 0 to 5. The process is now nicer, which means it waits longer for a turn. One more read from the same process tells you how it switches:

grep -E 'voluntary_ctxt_switches|nonvoluntary_ctxt_switches' /proc/$P/status

Expected output, exactly as it ran on this machine:

voluntary_ctxt_switches:	2
nonvoluntary_ctxt_switches:	0

Voluntary 2 means the sleeping process gave the CPU up by itself twice. Nonvoluntary 0 means the scheduler never had to push it off. A sleeping process is polite. A hog is not, and the nonvoluntary number is where a hog shows up. Kill the sleep when you are done. Only root can renice to a negative value. A normal user can only make a process nicer, never meaner.

Step 4: count the calls through the door

Count the system calls the program that does nothing makes.

strace -c -e trace=read,write,openat true

Expected output, exactly as it ran on this machine:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  0.00    0.000000           0         1           read
  0.00    0.000000           0         2           openat
------ ----------- ----------- --------- --------- ----------------
100.00    0.000000           0         3           total

Even true, the program that does nothing, knocks on the door 3 times: one read, two opens. And one real read line, from strace -e trace=read,openat cat /etc/hostname, shows the shape of a single call: the program asked for 832 bytes of a library file and got 832 bytes back. The return value is the answer. A program that makes a million calls pays the door price a million times, and strace -c is how you find out it does.

Step 5: watch the switches with perf, reader-side

There is no perf in this VM, so this step is yours to run where perf lives. The command is perf stat -e context-switches,page-faults sleep 1. It prints the switches and the page faults for one quiet second. No output is shown here, because none ran here. Run it, and compare the switches number with the one you counted in step 1. The two numbers should be in the same neighborhood. If they are far apart, one of the two measurements is measuring something else, and that is worth finding out before you trust either.

Check yourself

Change the nice value in step 3 from 5 to 10 and run it again. The NI column moves to 10. Then try renice -n -5 on the same process as a normal user and read the error. The kernel refuses, and the refusal is the rule: you can give up priority, you cannot take it.

Clean up

Kill the sleep process. Nothing else to stop. The pressure files and the counters are read-only views. They reset themselves.

Check yourself

Three questions. No score is saved anywhere. This is just you against the material.

Basic to master

Knowing the kernel is not one skill. It is four levels of the same job, and every team I have worked with sits on one of them. The levels are not about how many knobs you can name. They are about whether you check the right part first when the box goes slow. Read them in order. Each level fixes the failure that hurt the most at the level below it.

Basic · Name the five parts, follow one read

At the basic level, you name the five parts out loud and follow one read through them: the program asks, the call crosses the door, the scheduler picks, the map translates, the VFS resolves the name, the driver translates, the disk answers, the wakeup lands, the answer comes home. You know user mode from kernel mode, and you know the system call is the only door between them. The failure that ends this level is blaming the disk for a wait that lives in the run queue.

Do this: run step 0 of the lab and write down what each of the three pressure files says. Then point at the diagram and name the hop where the file name becomes a place on the disk.

Intermediate · Run queue, nice, context switches, pressure

At the intermediate level, you read the run queue and say what a long one means. You move a nice value and watch the priority move with it. You count context switches per second and say whether the box is quiet or busy. You read pressure and say what stalled, instead of guessing from free memory. A hog is a priority problem before it is a capacity problem, and you stop buying CPU for a queue that a renice would drain.

Do this: run steps 1 and 3 of the lab. Write down the switches per second and the nice value before and after renice. Then say, in one sentence, what a long run queue means.

Advanced · Sysctl tuning, priority inversion, VFS and driver failure

At the advanced level, you read the real knobs with sysctl and you know which ones do not exist on your kernel, because you checked. You can describe priority inversion: a low-priority process holds a lock, a high-priority process waits on it, and a medium one keeps running so the lock never frees. You read dmesg when a driver misbehaves, and you know a VFS error is fast and honest while a dead mount hangs. The failure that ends this level is tuning a knob from a blog post without reading your own kernel first.

Do this: run step 2 of the lab and write down the five knob values. Then write down which knob you would change first for a latency-sensitive service, and why, before you change anything.

Master · Pressure-driven signals, eBPF watch, tuning for the workload

At the master level, the kernel is a source of signals, not a black box. You alert on pressure and on the run queue, not on CPU percent alone, because CPU percent hides a queue. You watch the scheduler with eBPF and you can prove which process got picked and why. You tune one knob on one box, measure, and only then roll it out. The kernel is the floor the service stands on, and you treat it like a floor: measured, watched, and never tuned on a rumor.

Do this: write one alert on memory pressure some avg10 over 20 for five minutes, and one alert on the run queue staying long for five minutes. Then write one paragraph on what you would watch with eBPF (Lesson 126) to prove the alert fires for the right reason.

Go deeper

User mode against kernel mode: two rings, one rule

A program in user mode can use the CPU and its own memory and nothing else. The kernel runs in kernel mode and can touch anything. The switch between them is the trap, and it costs a couple of microseconds each way. That cost is why a program that makes a system call per byte is slow and a program that reads in big chunks is fast. The door is cheap. Knocking on it a million times is not.

What a syscall costs

The mode switch, the argument check, the work, the copy back. The lab counted 3 calls for true. A web server makes thousands per request. Count them before you optimize anything else, because the count is the first number that tells you whether the program is chatty. The syscall hop in this lesson is 2 µs. A million calls is two seconds of door, and the door is the cheapest part of the trip.

The scheduler, fairly

The Linux scheduler is called CFS, the Completely Fair Scheduler, and fair means each process gets time in proportion to its weight. Nice sets the weight. A process at nice -10 does not get all the CPU. It gets a bigger share than a process at nice 0, and the difference is the weight table, not a fixed number. Preemption is the scheduler taking the CPU back when a slice ends. A context switch is the save and the load: registers, stack pointer, page tables. It costs a little time each time, and F07 said so first.

Knobs move between kernels

The lab caught it live. sched_latency_ns and sched_min_granularity_ns do not exist on kernel 7.0.0-44-generic. The knobs that do exist here are sched_rr_timeslice_ms (100), sched_cfs_bandwidth_slice_us (5000), and sched_autogroup_enabled (1). A tuning post written for one kernel version is a rumor on another. Read your own sysctl -a output. Tune one knob on one box, measure, then roll it out.

Sysctl tuning, the careful way

Read first with sysctl <name>, change with sysctl -w <name>=<value>, and only write it to a file under /etc/sysctl.d/ when the measurement says it helped. vm.swappiness = 1 on the build machine means the kernel almost never swaps. That is a choice for a box with plenty of RAM. A box that swaps under load wants a different number, and the pressure files are how you tell. The kernel-parameters section of Lesson 98 is the tuning reference for this course. This lesson points there instead of repeating it.

Reading /proc/pressure

Three files, two lines each. some means at least one task stalled. full means every non-idle task stalled, so nothing productive happened at all. The three averages are 10, 60, and 300 seconds. The total is microseconds of stall since boot. A full number above zero on memory is the loudest alarm in the file. It means the whole box stopped doing useful work to wait on memory. F07 taught the pressure files first. This lesson is where you learn to read them as a stall clock.

Watching the kernel with eBPF

eBPF lets you watch the scheduler, the syscalls, and the pressure without changing the kernel. One paragraph is all this lesson needs, because Lesson 126 teaches eBPF properly: hooks, maps, the verifier, and the tools people already wrote. If you want to see which process the scheduler picks and why, that is an eBPF program. Go read Lesson 126, then come back and point it at the run queue.

Priority inversion, the shape of the bug

A low-priority process holds a lock. A high-priority process waits on the lock. A medium-priority process keeps running, so the low one never releases the lock, and the high one waits forever. The priority order is upside down, and the scheduler cannot see it, because the scheduler sees priorities and the lock is invisible to it. The fix is in the lock, not the scheduler: priority inheritance, or a shorter critical section. The incident below is this shape.

The VFS in one paragraph

One front door, many file systems. The VFS turns a path into an inode, and the inode belongs to ext4, tmpfs, or a network mount underneath. The program never knows which. That is the point. When a mount dies, the VFS still answers, and the answer is an error, fast, instead of a hang. The files themselves were taught in Lesson F09. This hop is the kernel half: the name becoming a place.

Where the driver ends

A driver that fails returns an error, and the kernel passes it up. A driver that hangs is worse, and dmesg is where it confesses. Load, probe, fail: the module loads, the probe asks the hardware if it is there, and if the answer is no, the driver unloads and the device stays dark. Check dmesg before you reinstall anything. The translator framing is F07's, and it still fits: the driver turns the kernel's general read into the exact signals one piece of hardware understands.

What changes at 10x and 100x

At 10x the processes, the run queue is the bottleneck, and the scheduler hop is the number that grows. The fix is not a faster scheduler. It is fewer runnable processes, or more CPUs, or a hog with its nice value fixed. At 100x, the pressure files become the dashboard, because memory is the scarce thing and the reclaim is the tax. The five parts do not change. The part that says no changes, and the numbers in this lesson tell you which one it is.

One time it broke

What broke

A batch job that rebuilt a search index every night started running at the same time as the evening web traffic. A deploy had set the batch job to nice -10, to make it finish faster. The web requests slowed down. The p99 latency doubled, then tripled. The CPUs were not full. The disks were not busy. The box looked fine on every dashboard except the one nobody had: the run queue.

Timeline

  • 18:00The nightly batch job starts, at nice -10, on the same box as the web service.
  • 18:20Web p99 latency doubles. The on-call engineer checks CPU and sees 60 percent. The engineer blames the network.
  • 19:00p99 triples. The engineer adds a second web worker. Nothing changes.
  • 19:40The engineer reads the run queue for the first time. Forty processes wait, and the batch job holds the CPU.
  • 20:10The engineer renices the batch job to 5. The queue drains in two minutes. Latency is back to normal before 20:15.
  • 21:00The batch job finishes, an hour later than usual, and nobody misses it.

What changed after

Two things. First, the fix. Batch jobs run at nice 10 or higher on shared boxes, and the deploy template enforces it. A deploy that sets a negative nice value fails the check. Second, the rule. The run queue is on the dashboard now, next to CPU, and the alert fires when the queue stays long for five minutes. CPU at 60 percent with a queue of forty is a busy box pretending to be idle. The queue is the number that tells the truth.

If you remember nothing else

Design exercise

Design the kernel-side view for a latency-sensitive web service that shares one box with a nightly batch job. The web service must answer in under 50 ms at p99. The batch job must finish by morning. The box has 4 CPUs and 16 GB of RAM. Name the priorities, the run queue alert, the pressure alert, and what you watch with eBPF. Say what breaks first at 10x traffic.

Clarifying questions to ask

  • What is the p99 budget, and who wrote it down?
  • How long is the batch job, and what happens if it runs late?
  • How much memory does the web service hold, and does it grow?
  • What does the batch job read, and does it fight the web service for the disk?
  • Who can change a nice value, and what stops a bad one?

Worked approach

Start with the priorities. The web service runs at nice 0, the batch job at nice 10, and the deploy template refuses any negative nice value on a shared box. The batch job gets the leftover CPU, and leftover is enough, because the night is long and the web traffic is low after midnight.

Then the numbers. At 4 CPUs the run queue should stay under 8. The alert fires when it holds above 8 for five minutes, because a queue twice the CPU count means someone is waiting. Memory pressure some avg10 over 20 for five minutes pages the on-call engineer, because that is tasks stalling, not memory filling up. eBPF watches the scheduler: which process gets picked, how long each one waits, and whether the batch job ever jumps the line.

Then what breaks first at 10x. The run queue. Ten times the web traffic means ten times the runnable processes, and 4 CPUs cannot hide that. The fix is not a nicer batch job. It is a second box, or the batch job moving to its own machine, before the traffic arrives.

Trade-offs to name. Priority against throughput: a nice batch job finishes later, and later must still be before morning. Sharing against isolation: one box is cheaper than two, and the queue is the price. Alerting on pressure against alerting on usage: pressure pages when work stalls, usage pages when a number crosses a line, and only one of those is a user feeling slow.

Saved in this browser, so the Lessons index can show your progress across the course.

Next in this series: Lesson 196, CPU, Memory & Threads
CPU, Memory & Threads

The kernel shares the machine. The next lesson is about what it shares: the thread that runs, the CPU time it burns, and the memory it holds. Threads against processes, CPU utilization by name, and the heap and the stack for the first time.

Go to Lesson 196 →

Or browse the full course from the Lessons index.