One request. One report. Forty-nine milliseconds on a healthy box, and I can tell you where every one of those milliseconds goes. A thread picks the request up. A core runs it. The heap gives it room. The pages map in. A lock guards the cache. A second core helps. The answer comes back. Three scarce things decide whether that trip is fast or slow: threads, CPU time, and memory. This lesson is the map of all three, and the sizing math that keeps them from fighting each other.
Difficulty: Advanced · Know this first: Lesson 98, Lesson F07, Lesson 195
By the end of this lesson you will have read a live box with top, free, and ps and named the load, the CPU split, and the available memory from your own output. You will have run a C program that shows the heap and the stack as two different addresses, and a Python program that shows two threads losing increments without a lock and losing none with one. You will have the sizing math for a box: cores from Little's law, memory from requests in flight, and limits from the measured peak plus 30 percent. The code parts run on your own machine and their outputs are shown exactly as they ran. The GPU parts of the story live in Lesson 193. This lesson is the CPU and the memory underneath them.
What you need: a Linux box or VM with gcc, Python 3, top, free, ps, and vmstat. The commands in this lesson were run on a 4-vCPU VM (AMD EPYC, x86_64) and the outputs shown are the outputs from that run. mpstat was not installed on that VM, so the per-core view in this lesson uses top press-1, which you run yourself. Your numbers will differ. The shape will not.
A thread is the unit that runs. CPU time is what it consumes. Memory is what it holds. One request needs all three, and the box has a fixed amount of each. When one runs out, the request waits or dies, and the waiting looks different for each one. A full CPU means a queue. Full memory means swap, then the OOM killer. Too many threads means a lock queue. Learn to tell the three apart and you stop fixing the wrong one.
| Process | Thread | |
|---|---|---|
| Shares with its siblings | Nothing. Its own memory, its own file descriptors. | Memory, file descriptors, open files. |
| Owns alone | Everything it was given. | Its stack and its registers. |
| Dies | Alone. The other processes keep running. | The process can die with it, if the thread takes the process down. |
| Costs to start | Dear. A new address space, a new set of tables. | Cheap. The memory is already there. |
| The danger | Isolation. One process cannot see another. | Sharing. Two threads can write the same memory at the same time. |
Most servers do not run one process per request. They run one process with many threads, so one slow request does not block the rest, and the price is shared state that threads can trip over. Lesson 119 shows the other way: Redis runs its commands on one thread, in order, and gives up parallelism to keep the sharing simple. Both are honest designs. The thread model is the general one. This lesson is the general model.
| RSS | VSZ | |
|---|---|---|
| Means | Resident set size. The memory really in RAM. | Virtual size. The memory the process thinks it has. |
| What it tells you | What the box is actually holding for this process. | The size of the illusion, including memory mapped but never touched. |
| Which one to watch | This one. A leak shows up in RSS first. | The gap between the two is the illusion. A big gap is normal. |
The process in the lab, PID 1, shows the gap in miniature. VmSize is 21,372 kB. VmRSS is 5,732 kB. The process thinks it has 21 MB. The box holds 6 MB for it. The rest is mapped and never touched. That gap is normal, and it is why you size from RSS, not from VSZ.
| Load average | %CPU | |
|---|---|---|
| Means | How many processes are waiting for a core, averaged over 1, 5, and 15 minutes. | The share of CPU time spent doing work instead of sitting idle. |
| Counts | Runners plus waiters, including processes waiting on disk. | Busy time only. Waiting is not counted. |
| When it lies | High load with idle CPUs means the wait is disk, not CPU. | Low %CPU with a slow app means the app is waiting, not working. |
| Read them together | Load high and %CPU high: the cores are the bottleneck. | Load high and %CPU low: the disk is the bottleneck. |
One request walks through all of it now. A report request on a 4-core box with 8 GB of memory. Eight worker threads wait in the pool. The request arrives, a thread picks it up, a core runs it, the heap gives it room, the pages map in, a lock guards the cache, a second core helps, and the answer comes back. Forty-nine milliseconds, end to end.
Run top -b -n 1 on your own box and read the %Cpu(s) line. Write down us, sy, wa, and id. The full decoding of that line lives in Lesson 98, which teaches it field by field. Then run ps -eo pid,user,%cpu,%mem,comm --sort=-%cpu and find the hottest process. If one process sits near 100 percent and the load average is near 1, that process is using one core fully. If the load average is 12 on a 4-core box and no process is hot, the queue is the story, not any one process.
The diagram above follows one report request through the box on 4 cores with 8 GB of memory. These four paths are the same idea drawn four ways. One request, answered. One request, killed for memory. One Little's law trail, hop by hop. One vmstat reading, mostly idle. Run each path. One run takes 7.2 seconds across the path, so you can watch each hop light up in turn.
The return rule: the green dot only starts moving after the orange dot reaches the data hop. A return with no request is a lie. The diagram never draws one. The failure replay at the end of Run all four shows the other side of the rule. Run it and watch the orange dot stop at the OOM killer and never come back.
One report walks the box from the story. Thread pool, core, heap, pages, lock, second core. The answer is born at the second core, and the green dot starts back from there.
Not run yet.
Same request, a leak. malloc stalls, the pages come back from swap, and the kernel kills the worker. The answer never comes back. The user gets an error instead of a report.
Not run yet.
The same request, measured instead of timed. Little's law says requests in flight equal arrival rate times trip time. One hundred times 49 ms is 5 requests in flight. Five times 16 MB, plus 512 MB of base, is 592 MB held. The trail adds up to 49 ms, and the memory math is the check that keeps the box alive.
Not run yet.
The same box, read with vmstat. The first sample from the real run in the lab: 2 runnable on a 4-core box, user 13 percent, system 7, idle 78, iowait 3. Two runnable on four cores means two cores are free at that instant. The queue is empty, and vmstat shows it in one line.
Not run yet.
Every hop charges rent in milliseconds. The bar below is a worked example for one report request through the box, not a measurement from a live system, but the shape is honest. The two core hops are the big ones at 30 ms, and the total is 49 ms. Switch the scenario and watch the bar change. In the saturated story the core hop grows to 240 ms, because the thread waits in the run queue. In the out-of-memory story the pages hop grows to 930 ms, because the pages come back from swap, and the answer hop disappears, because the worker is dead. Move a simulator slider and watch the pool wait, the core time, and the total move with it.
The budget bar above shows three fixed stories. This simulator lets you write your own. It answers the sizing question at the heart of this lesson: which scarce thing breaks first for your threads, your traffic, your memory per request, and your cores. You set how many worker threads wait in the pool, how many requests arrive per second, how much memory each request holds, and how many cores the box has. Move a slider and watch the pool wait, the core time, the lock wait, the memory held, the OOM killer, and the budget total change. The numbers are a teaching example, built from the formulas below, not live measurements. The shape is the lesson: the box breaks at the scarcest thing, and the scarcest thing moves when you move a slider.
How many threads wait in the pool. More threads run more requests at once. More threads also want the same lock, so the lock wait grows.
How many requests arrive each second. More arrivals mean more work for the same cores, so the run queue grows.
How many megabytes each request holds while it runs. Bigger requests fill the box faster, and the OOM killer is the end of that road.
How many cores the box has. More cores serve more requests at once. Fewer cores saturate sooner.
Try this: (1) At the defaults, 8 threads, 100 requests a second, 16 MB per request, 4 cores, utilization is 0.75, the pool wait is 2 ms, the core time is 18 ms, the lock wait is 3 ms, 5 requests are in flight, memory held is 592 MB, and the budget total is 49 ms. (2) Raise the threads to 32 and leave the rest at default. Utilization stays 0.75. The lock wait grows to 51 ms, the trip grows to 97 ms, 10 requests are in flight, memory held is 672 MB, and the budget total is 97 ms. The extra threads bought nothing. They only bought a longer lock queue. (3) Push the traffic to 200 requests a second and leave the rest at default. Utilization is 1.5. The pool wait grows to 296 ms, the core time grows to 128 ms, the trip grows to 453 ms, 91 requests are in flight, memory held is 1,968 MB, and the budget total is 453 ms. The memory holds. The cores do not. The status line says saturated. (4) Push to 32 threads, 1,000 requests a second, 256 MB per request, 4 cores. Utilization is 7.5. The pool wait caps at 400 ms, the core time caps at 400 ms, the lock wait is 51 ms, 877 requests are in flight, memory held is 225,024 MB, and the OOM killer fires. The kill hop costs 240 ms, the answer hop costs 0 ms, and the budget total is 1,113 ms. The request dies. The box did not run out of CPU first. It ran out of memory.
Formulas, in plain terms. CPU demand per request is 30 ms of core time, 18 ms on the first core and 12 ms on the second. Offered load is requests per second times 0.030 seconds. Utilization is offered load divided by cores. Pool wait is 2 ms while utilization is 0.8 or below. Above that it is 2 plus 600 times utilization minus 0.8, squared, capped at 400 ms. Core time is 18 ms while utilization is 1 or below. Above that it is 18 plus 220 times utilization minus 1, capped at 400 ms. Lock wait is 3 ms plus 2 ms per thread above 8, capped at 60 ms. Trip time without the kill is pool wait plus core time plus 4 plus 6 plus lock wait plus 12 plus 4. Requests in flight is arrival rate times trip time, by Little's law. Memory held is 512 MB of base plus in-flight requests times MB per request. The box holds 8,192 MB. If memory held is over 8,192 MB, the OOM killer fires, the kill hop costs 240 ms, and the answer hop costs 0 ms because the worker is dead. Otherwise the kill hop costs 0 ms and the answer hop costs 4 ms. Budget total is pool wait plus core time plus 4 plus 6 plus lock wait plus 12 plus the kill hop plus the answer hop.
Three steps, all run on this machine. The outputs shown are the outputs as they ran, on a 4-vCPU VM (AMD EPYC, x86_64, 7.7 GiB of memory). Nothing below is fabricated. Your numbers will differ. The shape will not.
Four reads, one picture. top shows the load and the CPU split. free shows the memory that matters, which is available, not free. ps shows the hottest process. vmstat shows the run queue. Run them in that order and write the four answers down before you open any other tool.
top -b -n 1 | head -5
Output, exactly as it ran:
top - 20:20:43 up 1:36, 0 user, load average: 0.49, 1.26, 1.74 Tasks: 9 total, 1 running, 8 sleeping, 0 stopped, 0 zombie %Cpu(s): 2.2 us, 2.2 sy, 0.0 ni, 93.3 id, 2.2 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 7934.5 total, 162.2 free, 7327.9 used, 1171.2 buff/cache MiB Swap: 1024.0 total, 577.7 free, 446.3 used. 606.6 avail Mem
The load average is 0.49, 1.26, 1.74. Falling, so the spike already passed. The %Cpu(s) line reads 2.2 us, 2.2 sy, 0.0 ni, 93.3 id, 2.2 wa, 0.0 hi, 0.0 si, 0.0 st. Mostly idle, a little user time, a little system time, a little disk wait, no steal. That is a healthy host and a quiet box.
free -h
Output, exactly as it ran:
total used free shared buff/cache available Mem: 7.7Gi 7.2Gi 162Mi 437Mi 1.1Gi 606Mi Swap: 1.0Gi 446Mi 577Mi
The line that matters is available, 606 Mi, not free, 162 Mi. Swap shows 446 Mi used. Swap in use is worth a look, not a panic. It says some pages were pushed out. It does not say the box is dying.
ps -eo pid,user,%cpu,%mem,comm --sort=-%cpu | head -6
Output, exactly as it ran:
PID USER %CPU %MEM COMMAND
67 root 11.9 5.2 hatch
752 root 1.5 0.3 hatch-execd
1 root 0.0 0.0 systemd
33114 ollama 0.0 0.2 ollama
17 root 0.0 0.0 systemd-journal
The hottest process sits at 11.9 percent of one core. Nobody is hot. That agrees with the load average and with the idle CPUs. One hot process would be a suspect. No hot process and a high load would be a queue. Here there is neither.
grep -E "VmRSS|VmSize|Threads" /proc/1/status
Output, exactly as it ran:
VmSize: 21372 kB VmRSS: 5732 kB Threads: 1
VmSize is 21,372 kB. VmRSS is 5,732 kB. The process thinks it has 21 MB. The box holds 6 MB for it. The gap is the illusion, and it is normal. Threads is 1. One process, one thread.
vmstat 1 2
Output, exactly as it ran:
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu------- r b swpd free buff cache si so bi bo in cs us sy id wa st gu 2 0 456976 165896 20 1202068 0 0 5390 2044 7843 30 13 7 78 3 0 0 0 0 456952 172864 20 1203760 0 0 1144 96 9571 12781 2 2 95 1 0 0
Read the first sample line, not the header. r is 2. Two runnable on a 4-core box, so two cores are free at that instant. us is 13, sy is 7, id is 78, wa is 3, st is 0. The queue is empty. The second sample says the same thing a second later: r is 0, idle is 95. A run queue that sits above the core count is the saturated box. This one is not.
One tool is missing on this VM, and the lesson says so plainly. mpstat was not installed here, so no mpstat output is shown. The per-core view in this lesson uses top press-1 instead. Run it yourself: start top, press 1, and the %Cpu(s) line splits into one line per core. One core at 100 percent and the rest idle is a single-threaded app. All cores high together is a genuinely busy box.
One int lives on the stack. Five ints live on the heap, asked for with malloc and given back with free. Print both addresses and the distance between them is the lesson.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main(void) {
int stack_var = 42;
int *heap_arr = malloc(5 * sizeof(int));
if (!heap_arr) return 1;
for (int i = 0; i < 5; i++) heap_arr[i] = i * 10;
printf("stack_var value: %d\n", stack_var);
printf("heap_arr values: %d %d %d %d %d\n",
heap_arr[0], heap_arr[1], heap_arr[2], heap_arr[3], heap_arr[4]);
printf("stack address: %p\n", (void *)&stack_var);
printf("heap address: %p\n", (void *)heap_arr);
free(heap_arr);
heap_arr = NULL;
printf("freed, pointer set to NULL\n");
return 0;
}
Output, exactly as it ran (gcc 13.3.0):
stack_var value: 42 heap_arr values: 0 10 20 30 40 stack address: 0x7fffe92c0188 heap address: 0x5a3a96a7a2a0 freed, pointer set to NULL
The stack address starts with 0x7fff. The heap address is far below it. Two different neighborhoods in the same process. Run it twice and the addresses change, because the kernel randomizes them. The values are the same. The addresses are not. That is address space layout randomization, and it is why you never hard-code an address from an output.
Two threads each add 1 to a shared counter 20,000 times. The right answer is 40,000. Without a lock the answer comes out wrong. With a lock it comes out exact. The sleep(0) in the no-lock version yields between the read and the write, so the two threads reliably step on each other.
"""race.py: two threads, one counter, with and without a lock.
Run with: python3 race.py
No third-party packages. The sleep(0) in the no-lock version yields the
interpreter between the read and the write, so the two threads reliably
step on each other. Your lost counts will differ run to run, which is
the point: without a lock the answer is wrong by a different amount
each time. With a lock it is exact, every time.
"""
import threading
import time
N = 20_000
counter = 0
def bump_no_lock():
global counter
for _ in range(N):
tmp = counter
time.sleep(0) # yield: the other thread runs between read and write
counter = tmp + 1
def bump_with_lock(lock):
global counter
for _ in range(N):
with lock:
counter += 1
t1 = threading.Thread(target=bump_no_lock)
t2 = threading.Thread(target=bump_no_lock)
t1.start(); t2.start(); t1.join(); t2.join()
print("no lock: counter =", counter, "expected", 2 * N,
"lost", 2 * N - counter)
counter = 0
lock = threading.Lock()
t1 = threading.Thread(target=bump_with_lock, args=(lock,))
t2 = threading.Thread(target=bump_with_lock, args=(lock,))
t1.start(); t2.start(); t1.join(); t2.join()
print("with lock: counter =", counter, "expected", 2 * N,
"lost", 2 * N - counter)
Output, exactly as it ran:
no lock: counter = 20000 expected 40000 lost 20000 with lock: counter = 40000 expected 40000 lost 0
Without a lock the counter lost 20,000 increments. Two threads read the same value, both added 1, and both wrote back the same answer. One increment vanished each time. With a lock the counter lost none. Your lost count will differ run to run. That is the point: without a lock the answer is wrong by a different amount each time.
Then the exit-code proof, run for real. A Python process that sends SIGKILL to itself exits with code 137. That is the number the OOM killer leaves behind.
python3 -c "import os, signal; os.kill(os.getpid(), signal.SIGKILL)"; echo "exit code: $?"
Output, exactly as it ran:
exit code: 137
Three questions. No score is saved anywhere. This is just you against the material.
Threads, CPU time, and memory are not three skills. They are four levels of the same job, and every team I have worked with sits on one of them. The levels are not about how many field names you can recite. They are about whether the numbers change what gets built. Read them in order. Each level fixes the failure that hurt the most at the level below it.
At the basic level you read the box and you name what you see. You run top, free, ps, and uptime. You can say which process is hot, how much memory is available, and what the load average is. You can say what a thread is, what a process is, and which one shares memory. The failure that ends this level is the first slow box where the numbers disagree, and you cannot say why.
Do this: run step 1 of the lab on your own box and write down the load average, the us and sy and wa fields, and the available memory. Then run step 2 and read the two addresses with your own eyes.
At the intermediate level you size from the traffic you actually see. Requests in flight is arrival rate times trip time. Memory held is in-flight requests times MB per request, plus the base. You read load average and %CPU together, and you know which one is lying: load counts disk wait, %CPU counts busy time only. You can say RSS from VSZ, and you know which one a leak shows up in.
Do this: take the simulator defaults and double the traffic. Watch the pool wait and the core time grow. Then halve the cores and watch the same thing happen. Same utilization, same queue, different cause.
At the advanced level you read the box per core, and you read the chip. Press 1 in top and one core at 100 percent with the rest idle means a single-threaded app, and more cores will not help it. Steal time means the hypervisor kept the CPU for another guest, and the fix is a bigger instance or a different host, not more threads. Swap is the failure mode, and the OOM killer is the end of it. You know x86 from ARM, and you know what changes when the chip changes: the instruction set, the core count, the memory bandwidth, and which accelerator the chip carries. You know the GIL in one paragraph: CPython runs one thread of Python bytecode at a time, so threads help Python wait and processes help Python compute.
Do this: run step 3 of the lab and read the lost count. Then check the flags line in /proc/cpuinfo on your own box and find ht for hyper-threading.
At the master level the box is sized from numbers, not from a feeling. You run the service under real traffic, you record the peak CPU and the peak memory, and you set the limits from those peaks plus 30 percent, for CPU and memory together. Lesson 146 teaches the rule. This lesson applies it to both scarce things at once. The cgroup is the wall: the kernel enforces the limit, and Lesson 195 explains the kernel side of that wall. A container that hits its memory limit dies with exit 137. A container that hits its CPU limit throttles and gets slow instead of dying. Both are the same decision, priced together. The team still reads the dashboard every week. The dashboard reads the measurements, not a guess.
Do this: pick one service you run, record its peak CPU and peak memory over a week, and write down the limit each peak plus 30 percent would give. Then check whether the current limit matches either number.
The decoding lives in Lesson 98, which teaches it field by field. The short version is here so you can read the line without flipping back. us is time running your code. sy is time running the kernel on your behalf. ni is niced work. id is plain idle. wa is idle waiting on disk. hi and si are interrupt time. st is steal, time the hypervisor took from this machine. The fields add up to 100 percent across the whole line. High us means your app is the work. High sy means the kernel is doing the work for your app, and syscalls and context switches are where you look next. High wa with idle CPUs means the disk, not the CPU. Read the fields before you blame the code.
Load average counts processes waiting for a core and processes waiting on disk. That is why a load of 12 on a 4-core box can sit next to idle CPUs: the wait is disk, and %CPU says so by staying low. %CPU counts busy time only. That is why a slow app can sit next to low %CPU: the app is waiting, not working, and %CPU has no column for waiting. The two numbers disagree on purpose. The disagreement is the diagnosis. Load high and %CPU high: the cores are the bottleneck. Load high and %CPU low: the disk is the bottleneck. %CPU high on one process and load near 1: one process, one core, and the fix is splitting the work, not adding cores.
The %Cpu(s) line is the average across all cores, and an average hides a busy core. Lesson 98 teaches the press-1 view. The diagnosis that follows is this lesson's: one core at 100 percent and the rest near idle is a single-threaded app. Adding cores does not help it. All cores high together is a genuinely busy box. Two cores at 50 percent each is one thread bouncing between cores, and the context switches are the tax. The per-core view is the check you do before you ask for a bigger box.
st is the time the CPU was ready to work for this machine and the hypervisor gave it to another guest instead. It only exists on virtual machines. A steal of 0 means the host is not oversubscribed, or this guest is getting what it asked for. Steal that stays above a few percent means the host is crowded, and no tuning inside the guest fixes it. The fix is a bigger instance, a different host, or fewer neighbors. The sample top output in the lab shows st at 0.0. That is a healthy host, not a guarantee.
RSS is the memory really in RAM. VSZ is the memory the process thinks it has, including pages mapped but never touched. A big gap is normal. A leak shows up in RSS, climbing steadily and never coming down. A spike shows up in RSS too, then comes back down. Page faults are the mechanism. A minor fault maps a page that is already in RAM. A major fault reads a page from disk. A major fault that reads from swap is the slowest kind, and a box doing many of them is a box that has run out of memory and is pretending it has not. free tells you the swap use. /proc/meminfo tells you MemAvailable, which is the number that matters, not MemFree. /proc/pressure/memory tells you how long processes stalled waiting on memory. Lesson 98 teaches the free reading. Lesson F07 teaches the pressure reading.
Swap is disk pretending to be memory. Lesson 98 teaches it. The failure mode is this: swap lets the box keep running past its memory, slowly, until the slowness is worse than the crash. A box with swap in use and major faults climbing is a box that needs more memory or less work, not more swap. Swap buys time. It does not buy memory.
A core is hardware. A thread is work. One core runs one thread at a time. Hyper-threading puts two threads on one core, sharing the core's execution units. The second thread is not a second core. It helps when the first thread is waiting on memory, because the core has something to do in the meantime. It does not help when both threads want the same execution unit. The flags line in /proc/cpuinfo shows ht when hyper-threading is on. The box in the lab has 4 vCPUs on an AMD EPYC, and the ht flag is set. Four vCPUs is the count you size from, not the count of physical cores underneath.
The two chip families run different instruction sets. x86_64 is the family in most servers and laptops. ARM is the family in most phones and in the newer cloud instances. Code compiled for one does not run on the other without a rebuild. What changes when the chip changes: the instruction set, the core count and layout, the memory bandwidth, and which accelerator sits next to the CPU. What does not change: the thread model, the paging model, and the sizing math in this lesson. The container image has to match the chip. An x86 image on an ARM node does not start, and the error will not say why in plain words. Check the architecture before you blame the image.
The main processor runs the program. The accelerator runs the one job it was built for. Lesson 167 teaches the split at device scale: a model that falls back to the main processor because the accelerator refused it runs five times slower. The server side of that story, GPUs, CUDA, and model formats, belongs to Lesson 193. This lesson does not teach what a GPU is. The rule carries over: a request that lands on the wrong processor pays for it in time, and the fix is routing, not a bigger box.
CPython runs one thread of Python bytecode at a time. That lock is the GIL. Threads still help Python when the work is waiting: network calls, disk reads, sleeping. The GIL is released while a thread waits, so the other threads run. Threads do not help Python when the work is computing: two threads doing math take turns on the one lock, and the total is no faster than one thread. For CPU-bound Python, processes are the answer, because each process has its own GIL. Lesson 73 says the same thing in one sentence: threads or separate processes are the answer instead. This paragraph is why.
A container is a process with a wall around it. The kernel builds the wall, and Lesson 195 explains how. The wall has two numbers: how much CPU the process may use, and how much memory it may hold. CPU is throttled when the process hits its limit: it runs slower, it does not die. Memory is fatal when the process hits its limit: the OOM killer ends it with exit 137. Lesson 48 teaches requests and limits as reservations. Lesson 146 teaches the measured peak plus 30 percent rule. This lesson is where the two meet the box: the limit is set from the peak, the kernel enforces it, and the exit code tells you which limit you got wrong.
At 10x the traffic, the pool queue is the first thing that breaks, and the fix is more cores or less work per request, not more threads. More threads move the queue to the lock. At 10x the memory per request, the OOM killer is the first thing that breaks, and the fix is a smaller batch or a bigger box, not a bigger swap. At 100x, the fleet is the product. The sizing runs in CI from the measured peaks, the limits are code, and the alert on available memory fires before the killer has to choose. The per-request math does not change. The number of boxes it runs on does.
The %Cpu(s) decoding, the per-core view, and saturation named: Lesson 98 Advanced Linux. The kernel side, the scheduler and the five components: Lesson 195 How the Kernel Works. The illusion this lesson gives a mechanism: Lesson F07 The Kernel: The Program That Runs the Machine. Thread pools sized from Little's law: Lesson 23 Circuit Breakers and Bulkheads. The single thread, for contrast: Lesson 119 Redis and In-Memory Data. The double check inside a lock: Lesson 07 Caching Strategies. The OOM killer and the measured peak plus 30 percent rule: Lesson 146 Docker in Production. Requests, limits, and QoS: Lesson 48 Kubernetes Core. The accelerator split at device scale: Lesson 167 On-Device AI. Server GPUs, CUDA, and model formats: Lesson 193 GPU and Model Formats for Inference. The async sentence this lesson answers for CPU-bound work: Lesson 73 Python.
A report service ran fine for months. Then the reports got bigger, and the service started dying at 3 AM with exit code 137. The supervisor restarted it. It died again. The on-call engineer raised the memory limit twice. It died a third time. The box had 8 GB. The service held every row it ever read and never gave the memory back. The leak was slow, about 40 MB an hour, so it took a day to show up and minutes to kill the box once swap filled. The OOM killer picked the report worker because it was the biggest memory user on the box. The database was smaller, so it survived. The reports were the only thing that died, every night, at the same hour.
Three things. First, the limit is set from the measured peak plus 30 percent, for CPU and memory together, and the number is written down where the team can see it. Second, an alert fires on MemAvailable, not on the kill. The kill is the failure. Available memory is the warning. Third, the report job streams its rows instead of loading them all. The fix that lasted was the streaming, not the limit. The limit buys time. The streaming removes the leak.
Size the box. A report API serves 200 requests a second. Each request needs 30 ms of CPU and holds 16 MB for 49 ms. The SLO is a 200 ms response time at p95. How many cores, how much memory, and how big a thread pool? Name the trade-off you are making.
Start with the cores. Offered load is 200 times 0.030 seconds, which is 6 cores worth of work. Six cores at 100 percent is a saturated box, so that is not the answer. At 0.75 utilization the queue stays short, so 6 divided by 0.75 is 8 cores. Eight cores is the size.
Then the memory. Trip time is 49 ms, so requests in flight are 200 times 0.049, which is about 10. Ten requests times 16 MB is 160 MB, plus 512 MB of base, which is 672 MB held. That is the steady state. The peak is what matters. Measure the peak over a week, add 30 percent, and that is the limit. If the measured peak is 2 GB, the limit is 2.6 GB.
Then the pool. Little's law says the pool needs at least the requests in flight, which is 10. A pool of 10 threads at 8 cores leaves room for bursts. A pool of 64 threads buys nothing but a longer lock queue. Size the pool from the traffic and the latency, not from a round number.
Trade-offs to name. Cores against latency: fewer cores are cheaper and slower under burst. Memory against batch size: a bigger batch holds more memory per request, and the limit has to move with it. Pool size against lock contention: more threads run more requests at once and queue longer on the shared lock. Say which SLO is the hard one before you size anything, because the sizing changes completely when it moves.
Saved in this browser, so the Lessons index can show your progress across the course.