Platform Engineering · Lesson F07

The Kernel: The Program That Runs the Machine

The machine runs many programs at once, and they all share one CPU, one memory, one disk. Something has to be in charge of sharing. That something is the kernel: the program that runs the machine, and the program every other program talks to first.

Difficulty: Beginner · Know this first: F06 — Building Software Together: Agile Methodologies

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

The bar below is a worked example for this lesson, not a measurement. It shows shares of 100 for the kernel’s attention on a normal day. The shape is the point: the programs themselves and the sharing between them take most of it. Switch the scenario and watch the sharing share grow.

Extra diagram · from scratch to master

The layers: hardware, kernel, applications

Start with the hardware alone. Add the kernel, then the drivers, then the applications, one layer at a time.

5 layers1 boss1 rule
Hardware
CPU, memory, disk
Kernel
the boss
Drivers
one per device
Applications
browser, editor
You
the user
Request / flowResponse · the way backFailure / dead end

Extra diagram · from scratch to master

Sharing one CPU between many programs

Follow three programs as the kernel gives each one a turn, thousands of times a second.

6 stops on the walk3 programs1 boss
Browser
waiting for you
Editor
waiting for keys
Music
playing
Kernel
hands out turns
CPU
one worker
All running
feels at once
Request / flowResponse · the way backFailure / dead end

Try it: change the inputs and watch it bend

Share a machine between programs yourself. Pick how many programs are running and how much memory the machine has, and see how much memory each program gets. The arithmetic is exact. The lesson is the sharing: more programs means less each.

4

How many processes are running. Each one gets its own piece of memory, and the kernel divides it between them.

8

Total memory in the machine. 1 GB is 1024 MB. The kernel keeps a little for itself, about 1 GB, and shares the rest.

Try this: (1) 8 GB of memory, 4 programs. The kernel keeps 1024 MB, shares 7168 MB, and each program gets 1792 MB. (2) 16 GB, 8 programs. Each program gets 1920 MB. (3) 4 GB, 20 programs. Each program gets only 154 MB, and that is tight.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

I take the same machine four times here. First I meet the kernel. Then I learn how it shares. Then I see what happens when sharing fails. Last I run the kernel for the world.

Basic — The program that runs the machine

Definition: The kernel is the program in charge of the machine. It starts first, it decides which program gets the CPU, it divides the memory, and it is the only program that talks to the hardware. System software is the kernel and its helpers. Application software is everything else.

Types:

  • The kernel: the boss, always running.
  • Drivers: small helpers that know one piece of hardware.
  • Applications: the programs you use, sitting on top.

Uses: I use this when a machine misbehaves. I ask which layer the problem is in: the application, the kernel, or the hardware underneath.

Dimensions: One machine, a few programs. What breaks first is usually an application, and I notice it when one program stops and the rest keep going. It costs nothing to learn the layers. It costs real time when I blame the kernel for an application problem. The trade is simple: name the layer first, then fix the problem in that layer.

Intermediate — How the kernel shares

Definition: At this level I see the sharing. The kernel gives each process a turn on the CPU, a piece of memory, and a turn at the disk. A process is a running program plus its memory and its place in the queue. Switching between processes is called a context switch, and it costs a little time each time.

Types:

  • CPU turns: short slices, handed out in turn. The part of the kernel that hands out the slices is called the scheduler.
  • Memory pieces: each process gets its own, and cannot see the others.
  • System calls: the door a program uses to ask the kernel for something.

The kernel is not one big job. It is five small jobs that work together. I name them here so I can point at each one later.

Five parts of the kernel:

  • The scheduler: it hands out the CPU slices, one process at a time.
  • The memory manager: it gives each process its own piece of memory and keeps the pieces apart.
  • Drivers: they talk to one piece of hardware each, as this lesson already showed.
  • The file system: it keeps track of files on the disk, so a program can ask for a file by name.
  • IPC, short for inter-process communication: it lets processes talk to each other, so they can share work instead of working alone.

Lesson 195, How the Kernel Works, takes these five parts one by one and shows how each one does its job.

Uses: I use this to explain why a machine with many programs feels slow. The work is not the problem. The sharing is.

Dimensions: Many programs at once, one machine. What breaks is the sharing when there is too much to share, and I notice it when the machine is busy but no single program is doing much. It costs attention to watch the sharing. The trade is that sharing lets many programs run at once, and it costs a little speed to switch between them.

Advanced — When sharing fails

Definition: At this level I see what happens when sharing fails. One program takes all the CPU. One program takes all the memory. The kernel has to protect the others, and it has tools for that: limits on how much a process may take, and the power to stop a process that will not share.

Types:

  • CPU hogs: a process that takes long turns and will not give the CPU back.
  • Memory hogs: a process that keeps asking for more until the machine has none left.
  • Limits: rules the kernel enforces, so one process cannot take everything.

Uses: I use this when a machine is slow and one process is the cause. I find the hog, I limit it or stop it, and the machine comes back.

Dimensions: One bad process, many good ones. What breaks is fairness, and I notice it when the machine feels frozen but the kernel is healthy. It costs a limit to protect the many from the one. The trade is that limits cost a little freedom for each program, and they buy a machine that stays usable when one program misbehaves.

Know your kernel. The kernel has a version, and that version decides what your machine can do. I check it before I install software that needs a new feature. An old kernel with new software is a mismatch that shows up as a strange error far from the cause.

$ uname -r
7.0.0-44-generic
$ grep -c processor /proc/cpuinfo
4

Every flag in the commands above, letter by letter.

  • uname -r: -r prints the kernel release only, like 7.0.0-44-generic. That one string is what I paste into a search when a driver problem looks kernel related.
  • grep -c: -c counts matching lines and prints only the number. Here it counts lines with the word processor, which is how I count cores from /proc/cpuinfo.
  • One more flag worth knowing: uname -a prints everything at once. Kernel name, host name, release, version, and machine type, on one line. I use it when I want the full identity of a machine in one paste.

Check it: run uname -r, then uname -a. The first prints one string. The second prints the whole line. Then run grep -c processor /proc/cpuinfo. It prints 4.

Try it: run both commands. The first is your kernel version. The second counts your CPU cores. Write both down. When a guide says it needs a kernel newer than yours, believe it and upgrade first.

Memory pressure has a number. The kernel tracks how much memory programs are waiting for. The pressure file shows it as an average. A high some number means programs are stalling, waiting for memory. I read it when a machine feels slow and free looks fine.

$ head -3 /proc/meminfo
MemTotal:        8124896 kB
MemFree:          126400 kB
MemAvailable:     459852 kB
$ cat /proc/pressure/memory
some avg10=0.46 avg60=0.51 avg300=0.21 total=3043028

Every flag in the commands above, letter by letter.

  • head -3: -3 means print the first 3 lines and stop. Without a number, head prints the first 10 lines. I use it to peek at a file instead of printing the whole thing.
  • cat takes no flags here. It prints the whole file. For a small file like /proc/pressure/memory, that is one line. For a big file, use head first.
  • Two more flags worth knowing: head -n 3 is the same thing written out, with -n taking the number. And cat -n numbers every line it prints, so I can point at line 3 instead of describing it.

Check it: run head -3 /proc/meminfo and head -n 3 /proc/meminfo. The output is identical. Then run cat -n board.txt in a folder with a small file and watch the numbers appear.

Try it: run both commands. MemAvailable is what a new program can use. The pressure avg10 is the stall rate over the last ten seconds. If avg10 is above 20, programs are waiting hard for memory, and that is your slowdown.

Master — Running the kernel for the world

Definition: At this level the kernel runs on machines that serve thousands of people, and I cannot see the machine at all. I watch the kernel through numbers: how busy it is, how much memory is free, how long programs wait for a turn. I tune the sharing so the machine stays fair under load.

Types:

  • Kernel numbers: CPU time in the kernel itself, context switches per second, memory pressure.
  • Tuning: changing how the kernel shares, for the work this machine does.
  • Isolation: containers that give each program its own walls, built on the kernel’s sharing.

Uses: I use this when a server slows down under load. I look at the kernel’s numbers before I blame the application, because the kernel is where the sharing happens.

Dimensions: Thousands of processes, thousands of users, one kernel. What breaks is the sharing under load, and I notice it in the kernel’s numbers before the users notice it in the slowness. It costs real skill to tune a kernel. The trade is that tuning costs hours, and it buys a machine that stays fair when everyone arrives at once.

The machine that kills your program. When memory runs out, the kernel picks a program and kills it to save the rest. That is the out-of-memory killer. It does not ask. It picks the biggest program and ends it. I learn the limits before I run a big job, so the kill does not surprise me.

$ ulimit -n
4096
$ cat /proc/pressure/memory
some avg10=0.46 avg60=0.51 avg300=0.21 total=3043028

Every flag in the commands above, letter by letter.

  • ulimit -n: -n prints the maximum number of open files this shell allows, 4096 here. A server that opens a socket per connection hits this limit. When it does, new connections fail with too many open files.
  • One more flag worth knowing: ulimit -a prints every limit, not just open files. Core file size, stack size, max processes, all of them. I run it when a program fails in a way that smells like a limit.

Check it: run ulimit -n, then ulimit -a. The first prints one number. The second prints the whole table of limits.

Try it: run ulimit -n. That is how many files one program may open at once. A program that opens more than that fails with too many open files. The fix is to raise the limit or to close files you no longer need. Both are better than finding out in production.

A kernel too old is a silent no. New software often needs a new kernel feature, and the install will fail with an error that says nothing about the kernel. I check the version first, then I install. Five seconds of checking saves an hour of reading error logs.

$ uname -r
7.0.0-44-generic

Try it: run uname -r and compare it with the minimum version in the install guide of the next tool you set up. If yours is older, upgrade the kernel before you install the tool. The guide assumes a kernel you may not have.

Go deeper

The kernel is the only program that runs in a special mode. It can touch the hardware. Applications run in a restricted mode. They cannot. That wall is enforced by the CPU itself, not by trust. An application that tries to touch the hardware is stopped by the hardware, not by the kernel asking nicely.

A process is not a program. A program is a file on the disk. A process is that program, running, with memory and a place in the queue. One program can become many processes. Your browser is one program and many processes, one per tab.

Context switches are the hidden cost of multitasking. Each switch saves one process and loads the next. It is fast, but it is not free. A machine that switches constantly spends its time switching instead of working. That is why too many processes can make a machine slow even when no single process is busy.

At 10x the programs, the kernel’s job does not change shape. It shares between more processes, switches more often, and divides the same memory into smaller pieces. The numbers grow. The idea stays.

At 100x, the kernel is the whole platform. Containers, virtual machines, and clouds are all built on the kernel’s sharing. The kernel never went away. It became the foundation everything else stands on.

Try it: read the pressure. Run cat /proc/pressure/memory and look at the some avg10 number. Under 1 means memory is fine. Over 20 means programs are stalling. Then run head -3 /proc/meminfo and compare MemAvailable with MemTotal. The two files tell the same story from two sides.

$ head -3 /proc/meminfo
MemTotal:        8124896 kB
MemFree:          126400 kB
MemAvailable:     459852 kB
$ cat /proc/pressure/memory
some avg10=0.46 avg60=0.51 avg300=0.21 total=3043028

One time it broke

What broke

A machine felt frozen. The mouse moved, but nothing opened. One program had taken all the memory, and the kernel was moving memory to the disk to make room, which made everything slower, which made the program ask for more. The machine was not broken. It was drowning in one program’s appetite.

Timeline

  • 15:00,the machine felt frozen. Clicks did nothing for seconds at a time.
  • 15:05,I checked which process was using the most memory. One program had nearly all of it.
  • 15:06,I stopped that program. The machine came back in seconds.
  • 15:10,I checked why the program wanted so much memory. It was opening every file in a folder at once, instead of one at a time.
  • 15:15,the program was fixed to open one file at a time, and a limit was set so it could not take all the memory again.

What changed after

Two things. First, when a machine feels frozen, I check which process is using the most memory before I blame the machine. Second, a limit on how much one process may take. The kernel can enforce it, and it protects every other program from one hog.

If you remember nothing else

Design exercise

A machine has 8 GB of memory and runs a browser, an editor, and a program that processes photos. The photo program keeps taking more memory until the machine slows down. Design the sharing: how much memory does each one get, what limit do you set, and what happens when the photo program hits the limit?

Clarifying questions to ask

  • How much memory does the kernel keep for itself?
  • How much does each program get, and why those numbers?
  • What limit do you set on the photo program, and what happens when it hits the limit?
  • What do you watch to know the sharing is working?
  • What changes if the machine has 32 GB instead of 8 GB?

Rough sizing

Start with the numbers. 8 GB is 8192 MB. The kernel keeps about 1024 MB. That leaves 7168 MB for three programs. Give the browser 2048 MB, the editor 1024 MB, and the photo program 4096 MB, with a hard limit at 4096 MB. When the photo program hits the limit, the kernel stops giving it more, and it must free memory or wait. The other two keep their pieces.

Trade-offs to name

A hard limit protects the other programs and can make the photo program fail if it truly needs more. A soft limit lets it borrow and risks the machine slowing down. The trade is that a limit costs the hog its freedom, and it buys a machine that stays usable for everyone else.

Next in this series
Lesson F08 — Application Software: The Programs You Actually Use