Lesson 98 taught you to ask the kernel what a process is doing, with strace and perf. Lesson 46 taught you what a packet does on its way through the stack. This lesson teaches you to run your own small programs inside the kernel, safely, to watch both. eBPF is a way to load a tiny program into the kernel, attach it to a hook, and have it run every time something happens: a packet arrives, a file opens, a system call runs. The program cannot crash the kernel, because a checker called the verifier reads it first and refuses it if it might. I learned eBPF the way I learn most tools in this job. I chased a slow service for an hour with strace, and then a one-line eBPF tool answered the question in ten seconds: which process was opening which files, and how often. This lesson walks one packet through eBPF, step by step, from a hook firing to a policy being enforced: what eBPF is, hooks, maps, the verifier, and the tools built on it, Cilium and Tetragon, and where it stops being the right tool.
Every hop charges rent in microseconds. The bar below is a worked example for one packet arriving at a server, not a measurement from a live system, but the shape is honest: the kernel part of the trip is a few microseconds, and the userspace part is most of the budget. Step through the lesson or switch the scenario and watch the bar change. The reject scenario is the lesson in one picture. The kernel part costs nothing, because there is no program, and the userspace part costs everything, because polling is the fallback. A rejected program is not a slow program. It is no program, and the bill is paid in userspace.
I start from scratch: one packet, one hook, one program that watches one thing — the ready-made tool from the Basic tier. Then I add one piece at a time until I reach the full picture: the full kernel/userspace map with the boundary drawn, the verifier as a gate before any program runs, the map as shared memory bridging the two sides, and the policy engine enforcing rules in the kernel at the moment things happen.
Network card (the packet arrives) I start here. This is the simplest piece, and nothing else exists yet.
From scratch to master. Starts from scratch: the simplest decision trip — one packet, three possible answers (pass, drop, redirect), one microsecond each. Builds to the Master-level picture: the reject trip (verifier refuses, polling fallback, 400 microseconds) and the overflow incident (a buffer too small for the rate, 90,000 events a second dropped, a dashboard lying by omission) — the Master-level arithmetic of sizing the buffer from the rate.
Pass: the packet walks up the stack to the socket and the app reads it. I start here. This is the simplest piece, and nothing else exists yet.
The budget bar above shows three fixed stories. This simulator lets you write your own. It models a fleet of servers running eBPF programs, with events arriving, programs attached, and a ring buffer carrying the events to userspace. Move a slider and watch the kernel CPU, the dropped events, and the verdict change. The numbers are a teaching example, not live measurements. The shape is the lesson: the kernel cost grows with the events and the programs, and the drops grow when the buffer is smaller than the burst.
Every event runs every attached program, so the kernel work is the events times the programs.
Each program costs about 2 microseconds per event, so the cost per event is 2 microseconds times the number of programs.
The events per interval are the events per second times the interval in seconds. A longer interval means more events waiting at once.
The buffer holds that many events. If more events arrive in one interval than the buffer holds, the rest are dropped.
Try this: (1) Leave the defaults: 100,000 events a second, 3 programs, 10 ms interval, 10,000 event buffer. The kernel CPU is 60 percent of one core and nothing is dropped. (2) Push the events to 1,000,000 a second and the programs to 20. The kernel CPU jumps to 4,000 percent of one core, which is 40 cores worth of watching. Same programs, same events. Only the knobs changed. (3) Set everything back, then set the interval to 100 ms and the buffer to 1,000 events. The events per interval jump to 10,000, the buffer holds 1,000, and 90,000 events a second are dropped. That is what a small buffer and a long interval cost, and the dropped events are the ones you needed most.
Three questions. No score is saved anywhere — this is just you against the material.
eBPF looks like one big kernel trick. It is not. It is four layers stacked on each other, and each layer only makes sense after the one below it. I learned it in this order, and I teach it in this order: watch one thing with a tool someone else wrote, then understand the hook and the map, then write and load a program of your own, then build the networking and security on top of it. Skip a layer and the next one reads like magic. Build them in order and nothing in this lesson is magic.
Definition: eBPF at the basic level is a set of ready-made tools that watch the kernel for you. You run one, it prints what it sees, and you stop. The classic first tool counts file opens by process. Another counts network connections by destination. Another shows which process is using the disk. You do not write a program. You do not load anything. You run a tool, read the output, and answer a question that used to take an hour of guessing.
Types:
Uses: every time a box misbehaves and the logs say nothing. A process opening files thousands of times a second. A connection that never returns. The first win is an answer in seconds to a question that strace would take an hour to answer, because strace watches one process and eBPF watches the whole box.
Dimensions: Scale: holds for one box, one question, one run. Failure modes: a tool that watches everything and prints everything, which drowns the answer in data; and a tool run without the privilege it needs, which prints nothing and says nothing about why. Cost: a few seconds and the privilege to run it, which is root or the right capability. Trade-offs: you get answers and no understanding of how they were made. That is enough to debug, and never enough to build. The tools are the front door, not the house.
Definition: the machine underneath the tools. A hook is a place in the kernel where a program can be attached: a packet arriving, a system call running, a file opening. A program is a small piece of code, compiled to eBPF bytecode, that runs when the hook fires. A map is the shared memory between the program and the userspace tool: the program writes to it, the tool reads from it, and the tool writes the lists the program checks. The program runs in the kernel. The tool runs outside it. The map is the bridge.
Types:
Uses: when the packaged tool does not answer the question. A counter the tool does not have. A filter the tool cannot express. A map that holds the list I need to check. The point of this layer is that the tools are programs, and once you can read a program, you can read any tool, and once you can write one, you can answer any question the kernel can see.
Dimensions: Scale: holds to one box and a handful of programs, loaded by hand. Failure modes: a program attached to the wrong hook, which never fires; a map that fills up and starts dropping the events it was meant to catch; and a program that reads a map the userspace tool never populated, which passes everything because its list is empty. Cost: the cost is learning. The program is small, but the kernel is large, and a small program in a large kernel needs to know exactly where it is allowed to look. Trade-offs: you get the ability to watch anything the kernel can see, and you pay for it in care. A program that is wrong is not a bug in an app. It is a risk to the box, and the verifier is the only thing standing between the two.
Definition: writing a program, loading it, and living with the verifier. You write the program in C, compile it to eBPF bytecode, and load it with a loader. The verifier reads every instruction before the program runs: no unbounded loops, no memory reads outside what the program may touch, no calls it may not make. If the verifier refuses the program, it never runs, and the error names the instruction it did not like. The loader attaches the program to a hook, pins it so it survives, and the userspace tool reads the maps and the ring buffer. The program is small, the verifier is strict, and the two together are what make eBPF safe.
Types:
Uses: when the question needs a program of my own. A counter the packaged tools do not have. A filter that only my service needs. A hook in a place no tool watches. The point of this layer is that the program is mine, and I own its correctness, its verifier errors, and its cost on every event it runs on.
Dimensions: Scale: holds to a fleet, because a program is loaded the same way on every box, and CO-RE means one compiled program runs on many kernel versions. Failure modes: a verifier error that reads like a riddle, and the fix is a bounds check the program forgot; a program that is correct and slow, because it runs on every packet and every microsecond counts; and a loader that unloads the program on restart, because nothing pinned it. Cost: the cost is expertise. The verifier's errors are precise and hard to read, and a program that passes the verifier can still be wrong about the question it was written to answer. Trade-offs: writing your own program buys the exact answer you need and costs a build pipeline, a test harness, and an owner. The packaged tools buy speed and cost generality. I run the packaged tools first, and I write my own program only when the packaged tool does not answer the question.
The verifier is the gatekeeper, and it rejects programs that are too big, too complex, or unbounded. A loop without a fixed bound fails to load. A program that reads memory it has not checked fails to load. The fix is to keep programs small and to check every pointer before using it. I develop by loading the program on a test machine first, and I read the verifier error in full, because the error names the exact instruction that failed. A program that loads on one kernel version can fail on an older one, so I test on the oldest kernel in the fleet, not the newest one on my laptop.
Definition: what is built on eBPF, and where it stops. Cilium is a network fabric that uses eBPF for networking and policy: it replaces the old packet filter, it enforces which service may call which service, and it does it all in the kernel, with programs attached to the hooks. Tetragon is a security tool that uses eBPF to watch behaviour: which process opened which file, which process made which connection, and it enforces policy on that behaviour, in the kernel, at the moment it happens. Performance is the discipline of keeping the programs cheap, because a program runs on every event. The limits are the edges: eBPF cannot do everything, and knowing where it stops is the skill.
Types:
Uses: when the whole fleet needs the same watching and the same policy. Cilium for the network: one policy, enforced in the kernel, on every box. Tetragon for the behaviour: one rule about which files a process may open, enforced at the moment it opens one. And I use the limits every time someone proposes eBPF for a job it cannot do: a program cannot sleep, cannot wait, and cannot do work that takes time. It can decide, count, and report, in microseconds, and that is all it can do.
Dimensions: Scale: holds at thousands of boxes, because Cilium and Tetragon are deployed like any other fleet software, and the programs are the same on every box. Failure modes: a Cilium policy that blocks a service the team forgot about, which looks like a network outage; a Tetragon rule that stops a process the team needs, which looks like an app bug; and a program that costs 40 microseconds per event on a million events a second, which is 40 cores spent on watching. Cost: the cost is operational. Cilium and Tetragon need owners, policies need reviews, and the programs need measuring, because a program that is never measured is a program that can eat a core without anyone noticing. Trade-offs: Cilium buys the network in the kernel and costs a networking layer the team has to learn. Tetragon buys behaviour enforcement and costs a policy the team has to write and keep. eBPF buys the kernel's eyes and costs the discipline to keep the programs small, the maps bounded, and the verifier happy. There is no free watching. There is only a choice about what to watch, and how much of the kernel to spend on it.
Overhead in production is a budget, and I measure it before I ship the program. I start with sampling: observe one request in a hundred, not every request, and the overhead drops by the same factor. I use per-CPU maps so the cores do not fight over one lock, and I keep the map sizes bounded, because a map that grows without a limit grows until the machine runs out of memory. The check is a load test with the program on and off, comparing latency and CPU. If the program costs more than a few percent, I sample harder or I narrow what it watches. An observability tool that slows the system it watches is a tool that gets turned off during the incident it was built for.
Writing an eBPF program is writing a small C program with strict rules. No loops that might never end. No reading memory you have not proved is yours. No calling functions the kernel has not allowed. The verifier reads the whole program before it runs, and it is strict on purpose, because a program in the kernel that is wrong is a box that crashes. I write the program, compile it with clang, and load it with bpftool, and I read the verifier's error when it refuses it, because the error names the instruction it did not like. The most common error is a missing bounds check, and the fix is one line: check the length before you read the byte. I keep the program small, because a small program is easy to verify, easy to read, and cheap to run on every event.
The verifier is the reason eBPF is safe, and it is also the reason eBPF is hard to learn. It simulates the program, instruction by instruction, and it tracks what the program knows about every value: is this number a pointer, is it bounded, can it be zero. A program that might read past the end of a packet is refused, even if it never would in practice, because the verifier cannot prove it will not. I have spent an hour on a verifier error that turned out to be one missing check, and I have learned to read the error from the end, where it names the instruction, not from the start, where it lists the whole history. The verifier is not an enemy. It is the proof that the program cannot crash the kernel, and the proof is worth the hour.
Cilium replaces the old packet filter with eBPF programs, and it does three jobs at once: networking, load balancing, and policy. The policy is the one I care about: a rule that says this service may call that service, enforced in the kernel, on every packet, with no userspace in the path. Tetragon does the same for behaviour: a rule that says this process may open this file, and if it opens another one, the event is raised, or the process is stopped. I use Cilium when the network policy has to be enforced everywhere, the same way, without a firewall rule on every box. I use Tetragon when the question is what a process did, not what a packet did. Both are programs in the kernel, and both are only as good as the policy they were given.
Performance in eBPF is measured in microseconds per event, and the budget is the event rate times the cost. A program that costs 2 microseconds on a million events a second costs 2 seconds of CPU a second, which is two cores. A program that costs 40 microseconds on the same rate costs 40 cores, and the box does not have them. I keep programs small, I use per-CPU maps so the cores do not fight over one lock, and I measure the cost before I ship. The limits are just as important. A program cannot sleep, cannot wait, and cannot do work that takes time. It cannot read arbitrary memory. It cannot call a function the kernel has not allowed. And where the kernel has no hook, eBPF cannot watch. Knowing where it stops is the skill, because a tool used past its edge is a tool that fails quietly.
At 10x the boxes, the programs are the same, and the deployment is the problem. A program has to be loaded on every box, pinned so it survives a restart, and updated the same way everywhere. I load programs with the same pipeline that deploys the rest of the fleet, and I watch the load failures the way I watch any deploy. At 100x, the events are the problem. A million events a second per box is a billion events a second across the fleet, and nobody reads a billion events. I aggregate in the kernel: count in a map, report the count, and send the raw events only for the things that matter. The lesson does not change at that scale. The scale changes where the work happens, and the answer is always the same: do as much as possible in the kernel, and send as little as possible out of it.
# Count file opens by process, for ten seconds, then exit
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @opens[comm] = count(); } interval:s:10 { exit(); }'
One bpftrace line, and the flags decide whether it lists, runs, or runs forever.
sudo bpftrace -l 'tracepoint:syscalls:*openat*'
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @opens[comm] = count(); }'
sudo runs it as root. bpftrace needs root, because it loads programs into the kernel. Without sudo it fails on permissions.
-e 'program' runs the program from the command line. Without -e, bpftrace reads a program from a file. tracepoint:syscalls:sys_enter_openat is the probe. It fires on every openat syscall. tracepoint is the provider, syscalls is the group, sys_enter_openat is the event.
@opens[comm] = count() is a map. The key is comm, the process name. count() adds one each time the probe fires. When the program ends, bpftrace prints the map.
interval:s:10 { exit(); } is a timer probe. s means seconds, 10 is the count. After ten seconds it runs exit(), and bpftrace prints the map and stops. Without it, the program runs until Ctrl-C.
-l 'pattern' lists probes that match, without running anything. I run this first when I do not know the exact probe name. The * is a wildcard. -lv adds the arguments each probe carries. bpftool prog list is the check after loading a program with bpftool. It lists the programs already loaded in the kernel.
// xdp_drop.c — drop packets from a blocked source address
#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/ip.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_endian.h>
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(max_entries, 1024);
__type(key, __u32);
__type(value, __u32);
} blocklist SEC(".maps");
SEC("xdp")
int drop_blocked(struct xdp_md *ctx)
{
void *data = (void *)(long)ctx->data;
void *data_end = (void *)(long)ctx->data_end;
struct ethhdr *eth = data;
if ((void *)(eth + 1) > data_end)
return XDP_PASS;
if (eth->h_proto != bpf_htons(ETH_P_IP))
return XDP_PASS;
struct iphdr *ip = (void *)(eth + 1);
if ((void *)(ip + 1) > data_end)
return XDP_PASS;
__u32 *blocked = bpf_map_lookup_elem(&blocklist, &ip->saddr);
if (blocked)
return XDP_DROP;
return XDP_PASS;
}
char LICENSE[] SEC("license") = "GPL";
Two tools, two jobs. The bpftrace one-liner watches the openat system call and counts opens by process name, for ten seconds, then exits. It is the first eBPF tool I reach for, because it answers a question in ten seconds that strace would take an hour to answer. The C program is the real thing: an XDP program that drops packets from a blocked address. It reads the Ethernet header, checks the bounds before it reads a byte, because the verifier refuses the program without that check, then reads the IP header, checks the bounds again, looks the source address up in a map called blocklist, and drops the packet if it is there. The map is filled by the userspace tool, so the block list can change without reloading the program. Read them together and you can see the whole lesson: the one-liner watches, the program enforces, and the map is the bridge between the kernel and the outside.
Map sizing is the quiet failure mode. A map with ten thousand entries holds ten thousand keys, and the eleven thousandth write fails silently or overwrites, depending on the map type. I size maps from the real cardinality: the number of distinct processes, the number of distinct sockets, the number of distinct paths. Then I watch the map fill level, because a map that is 90 percent full is a map about to drop data. The fix is a bigger map or a shorter lifetime for the entries, and the choice depends on whether the data is a cache or a counter. A counter can be aggregated and reset. A cache cannot.
buckets = [(10, 1200), (25, 3400), (50, 2100), (100, 700), (250, 120)] # (upper ms, count)
total = sum(c for _, c in buckets)
target = total * 0.95
seen = 0
for upper, count in buckets:
seen += count
if seen >= target:
print(f"total {total} requests, p95 falls in the {upper} ms bucket")
break
Expected output: 7,520 requests, and the p95 falls in the 100 ms bucket. The bucket math is the check: I add the counts up from the fastest bucket until I pass 95 percent of the total, and the bucket where I stop is the p95 estimate. The estimate is only as good as the buckets, which is why I pick them from the latency I care about.
This one stayed with me because the watching was perfect and the watching was the problem. The tool saw everything, and the team could not see anything, because everything was too much.
A team rolled out an eBPF-based watcher across the fleet, to count every file open, every connection, and every exec, on every box. The watcher worked. It worked too well. On a busy box it generated a million events a second, and the ring buffer held 10,000 of them, and the userspace tool read the buffer every 100 milliseconds, which meant 100,000 events arrived between reads, and 90,000 of them were dropped. The dashboard showed a clean, steady count, because the dropped events were the ones that never arrived, and nobody could see what was missing. The team trusted the dashboard, and the dashboard was lying by omission. The incident was not a bug in the program. It was a buffer too small for the rate, and a team that had never done the arithmetic.
Three things. First, the buffer and the interval are sized from the event rate before the program ships, with the arithmetic from the simulator in this lesson, not after the rollout, when the drops are already happening. Second, the programs aggregate in the kernel: they count in a map and report the count, and they send the raw events only for the things that matter. A million events a second is a million events nobody reads, and a count is a number the dashboard can show. Third, the dropped-event counter is on the dashboard, next to the counts, so a drop is visible the day it starts. A watcher that drops events silently is worse than no watcher, because it gives the team confidence in a number that is wrong.
Design the eBPF observability and enforcement for a fleet of 500 hosts running a latency-sensitive API. The API has a p99 latency budget of 200 ms. You need to see every connection, every file open by the API process, and every packet dropped by policy. You have one week, one on-call rotation, and a rule: no program ships without the verifier accepting it and the cost measured. Sketch the programs, the maps, the buffer, the policies, and what you would measure.
Start with the numbers from this lesson. One host at 100,000 events a second with 3 programs costs 60 percent of one core, which is already too much for a latency-sensitive API, so I do not send every event out. I count connections in a map and report the count once a second, I count file opens by process in a map and report the count, and I send the raw events only for the drops, which are rare. The ring buffer is sized from the rate: the raw events are about 1,000 a second, the interval is 10 ms, so 10 events arrive per interval, and a buffer of 10,000 holds them a thousand times over. The policies are two: a Cilium network policy that says which services may call the API, and a Tetragon rule that says the API process may open only its own data files. Little's Law for the sizing: at 100,000 events a second, each spending 6 microseconds in the kernel, the kernel spends 0.6 seconds of CPU a second on watching, which is 60 percent of one core, so I cut the programs to the two that matter and the cost falls to 40 percent, and the counting in the map means the userspace tool reads once a second instead of once per event.
Counting in the kernel buys a small bill and costs the raw events, so the drops are still sent raw, because a drop is the thing the team needs to see. A bigger buffer buys room for bursts and costs memory on every host, so the buffer is sized from the measured rate, not from a round number. A shorter interval buys fresher data and costs more userspace wakeups, so the interval is 10 ms, not 1 ms, because the data does not need to be fresher than the dashboard. Cilium buys network policy in the kernel and costs a networking layer the team has to learn. Tetragon buys behaviour enforcement and costs a policy the team has to write and keep. And the verifier is the gate: a program the verifier refuses does not ship, and the host does not fall back to polling, because a fallback that drops events silently is worse than a deploy that stops.
Saved in this browser, so the Lessons index can show your progress across the course.