Platform Engineering · Lesson 201

Kubernetes Components: Control Plane, Node, and the Plugin Interfaces

A cluster looks like one machine from the outside. It is not. It is eleven named parts that talk to each other through one door, and each part can be swapped, killed, or upgraded on its own. I learned the parts the way I learn most things in this job. Something broke, and the name on the log line was a binary I had never looked at. This lesson names every part, says what each one does, and shows what still works when one of them dies.

Difficulty: Advanced · Know this first: Lesson 48, Lesson 105, Lesson 106, Lesson 126

What you will build

By the end of this lesson you will be able to name the five control-plane binaries and say what each one does. You will have read a node the way the kubelet sees it, told the kube-proxy modes apart, and said which parts of a cluster are mandatory and which ones you can swap. You will have traced one request through the parts, from kubectl to a Pod answering back, and killed one part in the simulator and said what still works after it.

What you need: a cluster you can read with kubectl. The lab steps run on your own cluster, and every expected output is described as what to look for, not quoted as a transcript. No cluster runs on this machine. This VM hands out no cpu controller in its cgroup, so no k3s server can start on it, and that was settled in earlier lessons, so I did not re-attempt it here. No kubectl output from this machine appears in this lesson, because there is no cluster on this machine to produce it. Your numbers will differ. The shape will not.

The story: one cluster, named parts

Lesson 48 taught you desired state. A Deployment says how many Pods should run, and the controllers make it true. In that lesson the API server was just "the API server". Now it gets its real name: kube-apiserver, a binary, a process you can see in a process list. Every part in this lesson gets the same treatment. The cluster is not one thing. It is a set of parts, and each part has one job. When a part dies, only its job stops. That is the whole lesson, and it is the reason a cluster can lose its control plane and keep serving traffic.

The control-plane map

PartBinaryIts one jobWhere it runs
API serverkube-apiserverThe only door. It checks who you are, checks the rules, and writes to etcd. Nothing else writes to etcd.Control-plane node, usually a static Pod.
The storeetcdThe single source of truth. Every object in the cluster lives here, and every part reads from here through the API server.Control-plane nodes, three or five of them.
Schedulerkube-schedulerPicks a node for each Pod that has no node yet. It never starts a container. It only writes a node name.Control-plane node.
Controller managerkube-controller-managerRuns the controller loops: the node controller, the endpoints controller, and more. One binary, many loops.Control-plane node, one active copy, the rest on standby.
Cloud controllercloud-controller-managerThe cloud half. It makes load balancers, sets up routes, and tells the cluster when a cloud machine is gone.Control-plane node, or run by the cloud provider.

Five binaries, five jobs, one door. Everything a user or a controller wants to change goes through kube-apiserver. Everything the cluster knows lives in etcd. The scheduler and the controllers only read and write through the API server. They never touch etcd directly, and they never talk to each other directly. That shape is what makes the parts replaceable. Swap the scheduler for another one and the rest of the cluster never notices, because the contract is the API server, not the scheduler.

etcd: the store, and why three

etcd keeps every object, and it keeps them on more than one machine so one dead disk does not lose the cluster. Three members is the usual number. A write needs a majority, which is two out of three. One member can die and writes still land. Two members die and writes stop. Reads stop too, because the API server cannot read from a store with no quorum, so kubectl stops answering. The running Pods keep running, because the kubelet already has its orders and needs nothing new from the store. Five members survives two dead members, and it costs two more machines and slower writes, because every write waits for three members to agree instead of two. I run three for a small cluster and five only when the control plane must survive two failures at once.

A store you cannot restore is not a backup. The backup is one command, etcdctl snapshot save, and the restore puts that snapshot into a fresh data dir. The restore is the part teams skip, and it is the part that matters. I test the restore on a scratch machine before I trust the backup. A snapshot nobody has ever restored is a hope, not a backup.

The scheduler, in plain terms

The scheduler watches etcd for Pods with no node named. For each one it runs two passes. The first pass cuts the nodes that do not fit: not enough CPU, not enough memory, a taint the Pod cannot tolerate, an affinity rule the node breaks. The second pass scores the nodes that are left, and the highest score wins. Then the scheduler writes the winning node into the Pod, and that write is the assignment. Lesson 197 teaches the full pipeline, the queue, the two cycles, and the binding. This lesson only needs the shape: cut, score, write. A Pod that fits nowhere stays Pending, and the scheduler writes an Event saying why, so you never have to guess.

The controller manager: one binary, many loops

Lesson 48 taught you the controller shape: read the desired state, compare it with the world, act until they match, repeat forever. kube-controller-manager is the binary that runs most of those loops. The node controller watches node health and marks a node NotReady when its kubelet stops reporting. The endpoints controller builds the list of healthy Pod IPs behind each Service, and kube-proxy reads that list when it writes its rules. Each loop is small and each one owns one kind of object. Only one copy of the binary is active at a time. The rest stand by, and leader election picks the active one. When the active copy dies, a standby takes over in seconds, and the loops keep running.

What the cloud owns

In a cloud cluster, three jobs belong to the cloud, and cloud-controller-manager is the translator that does them. When a Service asks for a load balancer, the cloud makes a real one. When nodes need routes to each other's Pod networks, the cloud sets the routes up. When a cloud machine is deleted, the cloud tells the cluster the node is gone, so the node controller does not wait for a heartbeat that will never come. On bare metal there is no cloud to do these jobs, so there is no cloud controller manager, or a small one that does only part of the work. The split is the point. Kubernetes does not know your cloud. The cloud controller manager is the one part that does, and swapping clouds means swapping that part, not the cluster.

The node, in one paragraph

Each node has three agents. The kubelet is the node agent: it watches the API server for Pods with its node name, runs the probes, and calls the runtime to start and stop containers. kube-proxy is the wiring: it watches Services and writes the rules that turn a Service IP into a pod IP, and it moves no packets itself. The container runtime is the hands: containerd is the reference runtime, and it does what the kubelet asks through the CRI, the contract between them. The rest of this lesson opens each of these up, and the plugin interfaces after them.

Do this now, before the diagram

On your own cluster, run kubectl get pods -n kube-system and write down which of the five control-plane binaries you can see. The ones whose names end with a node name are static Pods, and the Master tier of this lesson says why. Then run kubectl get nodes and write down how many nodes the cluster has. That count is the size of the node side of the map, and every part below it runs on those nodes.

Run it as:

When it breaks, I check

One thing to remember

Four paths, one family

The diagram above follows one Pod from apply to answer. These four paths are the same idea drawn four ways. One healthy request. One etcd outage. One kube-proxy outage. One numbers trail, hop by hop. Run each path. One run takes 7.2 seconds across the path, so you can watch each hop light up in turn.

The return rule: the green dot only starts moving after the orange dot reaches the data hop. A return with no finished work is a lie. The diagram never draws one. The failure replay at the end of Run all four shows the other side of the rule. Run it and watch the orange dot stop at etcd silent and never come back.

Path 1: the healthy request

Failure: skipped

One apply, one Pod, one answer. The API server writes to etcd in 2 s, the scheduler picks a node in 3 s, the kubelet starts the Pod in 17 s, and the Pod answers in 27 s end to end. The green dot starts back from the Pod answers hop, because that is where the finished work lives.

kubectl applies~0 s
API server1 s
etcd stores2 s
Scheduler picks3 s
kubelet starts17 s
Pod answers1 s

Not run yet.

Path 2: etcd is down

Pod born: skipped

The API server waits 5 s for a write that never lands, and kubectl fails. The request stops at etcd silent. No Pod is born, so the scheduler and the kubelet sit idle, and no green dot ever starts back. The running Pods keep running, because they already have their orders.

kubectl applies~0 s
API waits5 s
etcd silentno quorum
Scheduler idlewaits
kubelet idlewaits
No Pod bornnever

Not run yet.

Path 3: kube-proxy is down

Answer: skipped

The Pod runs and DNS answers, but no rule is written, so the call never arrives and the client times out. The request stops at kube-proxy silent. The Pod is healthy. The Service is silent. No green dot starts back, because the work was done and nobody could reach it.

Pod runs23 s
Client calls1 s
kube-proxy silentno rules
No rule writtennever
DNS answers1 s
curl times outno answer

Not run yet.

Path 4: the numbers trail

None skipped

Five binaries on the control plane, three agents on the node, two plugin shapes, one source of truth, and the static Pods that boot the whole thing. Eleven named parts, and the verdict is the count. The green dot starts back from the verdict, because the count is the answer this path carries.

Control plane5
Node agents3
Plugin shapes2
Source of truth1
Static Podsboot
Verdict11 parts

Not run yet.

Where the time goes

Every hop charges rent in seconds. The bar below is a worked example for one Pod, from apply to answer, not a measurement from a live system, but the shape is honest. The runtime starting the container is the big hop at 12 s, and the total is 27 s. Switch the scenario and watch the bar change. When etcd is down, every hop after the store disappears, because nothing was ever written. When kube-proxy is down, the answer hop disappears, because the call never arrives. Move a simulator control and watch the hops move with it.

Kill one part, and watch what still works

The budget bar above shows three fixed stories. This simulator lets you write your own. It answers the question at the heart of this lesson: when one part dies, what still works. Kill etcd, the scheduler, or kube-proxy, and move the image pull time from a warm cache to a cold pull. Watch whether kubectl apply still works, whether new Pods get scheduled, whether running Pods keep running, whether Service traffic reaches the Pod, and whether DNS still answers. The numbers are a teaching example, built from the formulas below, not live measurements. The shape is the lesson: the store failing stops all change, the scheduler failing stops new Pods only, and kube-proxy failing stops Service traffic only.

healthy

The store is down. No write lands, so no Pod is born, and the whole chain after the API server goes quiet. Running Pods keep running.

healthy

No Pod gets a node. Applies still work and etcd still stores the Pod, but it stays Pending, and the kubelet never hears about it.

healthy

No rule turns a Service IP into a pod IP. The Pod runs and DNS answers, but the call never arrives.

12 s

How long the runtime takes to start the container, from a warm cache at 5 s to a cold pull at 60 s. The pull moves the runtime hop, and the runtime hop moves the total.

Try this: (1) At the defaults, nothing killed and a 12 s pull, kubectl apply works, the Pod is scheduled in 3 s, the runtime starts it in 12 s, the rules are written, DNS answers, and the Pod answers in 27 s. (2) Kill etcd. Apply fails after about 5 s, no Pod is born, and the path total is 5 s. Running Pods keep running. Untick it. (3) Kill the scheduler. Apply works and etcd stores the Pod, but the Pod stays Pending, the scheduler hop shows 30 s of waiting, DNS still answers, and the path total is 34 s. Untick it. (4) Kill kube-proxy. The Pod runs in 23 s and DNS answers, but the answer hop costs 0 s and the path total is 24 s. The Pod is healthy and the Service is silent. (5) Back at the defaults, move the pull to 60 s. The runtime hop costs 60 s and the path total is 75 s. The pull is the whole story at that setting.

Formulas, in plain terms. The API hop costs 5 s when etcd is down and 1 s otherwise. The etcd hop costs 2 s when the store is up and 0 s when it is down. The scheduler hop costs 30 s when the scheduler is down, which is the watch window I use before I stop and read the Events, and 3 s when it is up. The kubelet hop costs 5 s when a Pod reaches a node and 0 s when no Pod is ever scheduled. The runtime hop is the pull time you set, 5 s warm to 60 s cold. The kube-proxy hop costs 2 s when the rules are written and 0 s when they are not. The DNS hop costs 1 s when CoreDNS can answer and 0 s when the store is down. The answer hop costs 1 s when the call reaches the Pod and 0 s when it does not. The path total is the eight hops added up, and hop 0 costs 0 s because it starts the clock.

Lab: read the control plane, the wiring, and the boot

Three steps, all run on your own cluster. There is no kubectl output from this machine in this lesson, because there is no cluster on this machine. The k3s install cannot start here, as the top of this lesson says. What follows is what to run and what to look for, not a transcript. Your output will differ. The shape will not.

Step 1: read the control plane

Run these on your own cluster. They only read, so they change nothing.

kubectl get pods -n kube-system -o wide
kubectl get nodes
kubectl cluster-info

What to look for: the kube-system Pods named kube-apiserver, etcd, kube-scheduler, and kube-controller-manager, each with a node name at the end of the Pod name. That suffix is the mark of a static Pod, and the Master tier says why it is there. The cloud controller manager shows up too, if your cluster runs one. The nodes list tells you how many machines the node side of the map runs on. kubectl cluster-info tells you where the API server listens. If any control-plane Pod is not Running, write down which one, and read its log next. That Pod is the suspect.

Step 2: the wiring and the names

Find kube-proxy and CoreDNS, then test a lookup from inside a throwaway Pod:

kubectl get pods -n kube-system -l k8s-app=kube-proxy
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl get configmap kube-proxy -n kube-system -o yaml
kubectl run dns-test --image=busybox:1.36 --restart=Never -i --rm -- nslookup kubernetes.default

What to look for: one kube-proxy Pod per node, because it runs as a DaemonSet, and two or more CoreDNS Pods, because it runs as a Deployment. The ConfigMap holds the kube-proxy mode on most clusters, usually iptables. On some clusters the mode is a flag on the DaemonSet instead, so if the ConfigMap has no mode line, read the DaemonSet. The nslookup must answer with the ClusterIP of the kubernetes Service. That answer is CoreDNS doing its job. The --rm flag deletes the test Pod when the command ends, so nothing is left behind. If the lookup fails, read the CoreDNS Pods first. A name that does not resolve is a DNS problem, and a name that resolves to a Service that never answers is a kube-proxy problem. The two failures look the same from the client, and they are not the same outage.

Step 3: static Pods and the kubeadm phases

This step reads the boot, and it changes nothing. On a control-plane node, list the static Pod manifests. The kubelet reads this directory and starts one Pod per file.

ls /etc/kubernetes/manifests
kubeadm init phase --help

What to look for: one manifest per control-plane binary, usually kube-apiserver.yaml, kube-controller-manager.yaml, kube-scheduler.yaml, and etcd.yaml. Each file is a Pod manifest, and each one makes a Pod whose name ends with the node name. The kubeadm command only lists the phases, it runs none of them: preflight, certs, kubeconfig, kubelet-start, control-plane, etcd, upload-config, mark-control-plane, bootstrap-token, and addon. That list is the outline of how a control plane boots itself, and the Master tier walks it. The shape of one manifest is below. It was checked against the Kubernetes 1.34 schemas with kubeconform, strict mode, on Nov 21, 2026. It parses and validates. It has not been run against a live API server from this machine, and it is shown as the shape only. Do not drop it into the manifests directory on a real cluster unless you mean to start that Pod, because the kubelet will start it the moment the file lands.

apiVersion: v1
kind: Pod
metadata:
  name: hello-static
  namespace: kube-system
spec:
  containers:
    - name: hello
      image: busybox:1.36
      command: ["sh", "-c", "echo I am a static Pod; sleep 3600"]

Read the real manifests instead of writing one. Open kube-apiserver.yaml on the node and find the etcd server flag, which tells you where the API server finds its store. Find the certificate flags too. They are the proof that the API server and etcd talk over TLS, and the Master tier says what breaks when those certificates expire.

Check yourself

Three questions. No score is saved anywhere. This is just you against the material.

Basic to master

A cluster, its node, its plugins, and its boot are not four skills. They are four levels of the same job, and every team I have worked with sits on one of them. The levels are not about how many binary names you can recite. They are about whether the names change what you check when something breaks. Read them in order. Each level fixes the failure that hurt the most at the level below it.

Basic · The control-plane map, and the one door

At the basic level you can draw the control plane from memory: kube-apiserver, etcd, kube-scheduler, kube-controller-manager, and cloud-controller-manager. You can say what each one does, and you can say the one rule that holds them together. Only the API server talks to etcd. Everything else asks the API server. You know etcd is the single source of truth, three members, quorum two, and you know a write needs a majority. The failure that ends this level is the first kubectl that hangs, and you cannot say whether the door is down or the store is down.

Do this: run step 1 of the lab on your own cluster and write down the five binaries you can see, which ones are static Pods, and how many nodes the cluster has.

Intermediate · The node, and the three agents

At the intermediate level you can say what happens on a node, hop by hop. The kubelet watches the API server for Pods with its node name, runs the probes, and calls the runtime through the CRI. kube-proxy watches Services and writes the rules, and it moves no packets. containerd starts the container. You can say which agent is the suspect for a Pod that stays Pending, a Service that answers nothing, and a container that never starts. The failure that ends this level is the first Service that goes silent while the Pod stays healthy, and you blame the app before you check the rules.

Do this: run step 2 of the lab, then read the kube-proxy mode from the ConfigMap or the DaemonSet and write down which mode your cluster runs, and why that mode fits its size.

Advanced · The plugin interfaces, as one idea

At the advanced level you see CNI and CSI as the same shape doing two jobs. Kubernetes calls a program when it needs a network, and it calls a program when it needs storage. You can pick a CNI plugin from the table: Flannel for a small cluster where simple wins, Calico where the network team owns the routes and wants policy, Cilium where the cluster is big enough that the lookup and the policy are the bottleneck. You know CoreDNS is a Deployment that answers names from Service objects, and you know what breaks when it is down: new lookups fail first, and the error says the name does not resolve. The failure that ends this level is the first plugin choice made on speed alone, and the cluster is fast and nobody can debug it at 2 AM.

Do this: run the nslookup from step 2, then delete the CoreDNS Pods on a scratch cluster if you have one, and watch what a fresh lookup does. Write down the error, word for word. That error is the CoreDNS failure mode, and you will meet it again.

Master · Mandatory vs replaceable, the boot, and the suspect

At the master level you know which parts must exist and which ones you can swap. etcd is mandatory. The API server is mandatory. kube-proxy is replaceable, and CoreDNS is replaceable. You know how a control plane boots itself: the kubelet starts first, reads the manifests, and starts the control plane as static Pods. You know the kubeadm phases in outline, and you know a managed control plane is the same parts run by the cloud provider instead of you. And when something breaks, you name the suspect from the symptom before you open a log: a hang at apply is the door or the store, a Pod that stays Pending is the scheduler or the kubelet, a Service that never answers is the rules. The master move is not a bigger cluster. It is the right suspect in the first minute.

Do this: pick one outage you have seen, write down the first symptom, and name the component that owned it. Then write down the log you would open first for that component, from the debugging table in Go deeper. If you cannot name both, that is the gap this level closes.

Go deeper

etcd quorum, in numbers

Three members need two to agree. Five members need three. The rule is a majority, and the number that matters is how many members can die at once: one for three, two for five. An even number buys nothing. Four members still need three to agree, so it survives one failure, the same as three, and costs an extra machine. I run odd numbers only. The backup is etcdctl snapshot save, and the restore is etcdctl snapshot restore into a fresh data dir, then a restart of the etcd member with that dir. The restore is the test. I run it on a scratch machine once a month, because a snapshot nobody has ever restored is a hope, not a backup. The other etcd discipline is disk space. etcd stops taking writes when its disk fills, and it stops before the disk is fully full, so the cluster stops changing while the Pods keep running. The incident at the end of this lesson is that failure, and the fix is a disk of its own for etcd plus an alert at 70 percent.

The scheduler, in plain terms

Lesson 197 teaches the full pipeline, and this is the short version. The scheduler keeps a queue of Pods with no node. For each Pod it runs the scheduling cycle: the predicates cut the nodes that do not fit, and the priorities score the nodes that are left. The highest score wins. Then the binding cycle writes the node name into the Pod. Preemption is the sharp edge: a high priority Pod can evict a low priority Pod to take its node, and the evicted Pod goes back to the queue. I set priorities only when a workload must win a node, because preemption is a Pod being killed by a scheduling decision, and the team that owns the evicted Pod will ask why. The scheduler never starts a container, never pulls an image, and never writes a rule. It writes one field, and the kubelet does the rest.

The loops inside kube-controller-manager

One binary, many loops, and each loop owns one kind of object. The node controller watches the nodes and marks one NotReady when its kubelet stops reporting, and it evicts the Pods from a node that stays dead. The endpoints controller builds the EndpointSlice behind each Service from the Pods that pass their readiness probe. The namespace controller deletes everything in a namespace when the namespace is deleted. The service account controller makes the token for each new ServiceAccount. Only one copy of the binary is active. Leader election picks it, and the standbys wait. When the active copy dies, a standby takes over in seconds, and the loops resume where they left off, because the state lives in etcd, not in the binary. That is why the controller manager can die and the cluster barely notices. The loops are stateless. The store is not.

What the cloud owns, in full

The cloud controller manager does three jobs, and each one is a translation from a Kubernetes object to a cloud API call. A Service of type LoadBalancer becomes a real load balancer in the cloud, with a real address, and the Service status carries that address back. The node controller in the cloud manager checks the cloud for each node, and when a cloud machine is gone it tells the cluster, so the node is removed instead of waiting for a heartbeat. The route controller sets up the routes between nodes so Pod networks can reach each other. In a managed cluster the cloud provider runs the whole control plane, including this part, and you see only the nodes and the API endpoint. That is a fair trade for a small team, and it costs you the ability to read the control-plane logs when the API server misbehaves. I take the managed control plane when the team is small and the logs matter less than the uptime, and I run my own when the team needs to see the parts.

The kubelet, the node agent

The kubelet is the only part that touches the node. It watches the API server for Pods with its node name, and it also reads static Pod manifests from /etc/kubernetes/manifests. It runs the liveness and readiness probes, and a failed liveness probe restarts the container. It calls the runtime through the CRI to create, start, and stop containers, and it reports the node status and the Pod status back to the API server. It authenticates to the API server with a certificate, and it rotates that certificate before it expires, because an expired kubelet certificate is a node that stops reporting and looks dead. The kubelet also runs the eviction order Lesson 48 teaches: BestEffort Pods go first, Guaranteed Pods go last. When a Pod stays Pending after it has a node, the kubelet log on that node is the first log I open, and the Pod Events are the second.

kube-proxy modes, and how each one fails

Lesson 105 teaches the modes and the rule walk, and this table adds the failure mode for each one. The modes are the same four options: userspace, iptables, IPVS, and the eBPF replacement.

ModeHow it worksWhen it winsHow it fails
userspacekube-proxy itself proxies every connection, in userspace.Almost never. I only meet it in old clusters.Slow under load, and the proxy process is a single point of failure for the traffic it carries.
iptablesOne rule per endpoint. The kernel walks the rules and rewrites the destination.Most clusters. It is the default, and it stays fast until the rule list gets long.The rule list grows with every Service and endpoint, and the walk gets slower as it grows. A Service with a thousand endpoints is a thousand rules to walk.
IPVSA table instead of a rule list: one virtual server per Service, one real server per endpoint.Clusters with hundreds of Services, where the iptables walk is the bottleneck.The table is another thing to read when traffic goes wrong, and the team has to learn IPVS to debug it.
eBPF (Cilium)Not a kube-proxy mode. It replaces kube-proxy. A program in the kernel does the lookup and the balancing.Big clusters where the lookup and the policy are the bottleneck, and the team can debug eBPF.A program that fails to load leaves the node with no dataplane at all, and the debugging moves from rules you can list to programs you have to dump.

I stay on iptables while the cluster is small. I move to IPVS when the rule list gets long. I move to eBPF when the team can debug it. The failure mode is the same shape in all four: the Pod lives, the Service goes silent, and the app log says nothing, because the app never saw the call.

The runtime and the CRI

The CRI is the contract between the kubelet and the runtime. The kubelet speaks CRI, and the runtime answers: pull this image, create this container, start it, stop it, report its status. containerd is the reference runtime, and it is the one most clusters run. CRI-O is the other runtime you will meet, and it does the same job behind the same contract. The history matters in one paragraph. Docker used to be built into Kubernetes through a shim called dockershim, and the kubelet spoke to Docker directly. That shim was removed in Kubernetes 1.24. Docker itself still works as a runtime through cri-dockerd, which speaks CRI on one side and Docker on the other, but the simple path now is containerd, straight through the CRI, with no shim in the middle. If you inherit a cluster that still talks about dockershim, it is running a Kubernetes older than 1.24, and the upgrade path is the runtime first, then the cluster.

CNI vs CSI, one idea twice

CNI and CSI are the same shape doing two jobs. Kubernetes needs something it does not want to build in, so it calls a program that does it. The shape is the plugin: a small program, a contract, and a call at the right moment. The jobs are different, and the table is the contrast.

CNICSI
JobThe network. It wires the Pod: the address, the routes, the policy.The storage. It provisions the volume, attaches it, and mounts it.
Called whenEvery Pod starts, and every Pod dies.A claim is made, a volume is attached, a snapshot is taken.
Taught inLesson 105 teaches the network side in full.Lesson 106 teaches the storage side in full.
When it breaksThe Pod starts and cannot reach anything. The network is silent.The Pod stays in ContainerCreating and the volume never mounts. The storage is silent.

The plugin shape is the point. Kubernetes does not build networks and it does not build storage. It defines the contract, and the plugin does the work. That is why a cluster can swap Flannel for Cilium without swapping Kubernetes, and swap one storage driver for another without touching a Pod spec.

CoreDNS: what answers, and what breaks

CoreDNS runs as a Deployment in the kube-system namespace, usually two replicas or more. A Pod that calls a Service by its short name asks the DNS server in its resolv.conf, and that server is CoreDNS. CoreDNS reads the Service objects from the API server and answers with the ClusterIP. The Pod never learns the pod IPs. It learns one virtual IP, and the kube-proxy rules do the rest. When CoreDNS is down, new lookups fail, and the error is a DNS error: the name does not resolve. The old answers linger in the Pod's cache for a short time, then they go too. A headless Service is the exception Lesson 105 teaches: no ClusterIP, no kube-proxy rules, and DNS answers the short name with the pod IPs themselves. The alternatives to CoreDNS are few, because the cluster DNS is a small job and CoreDNS does it well. The real alternative is running more replicas of it, so one dead replica is not an outage. I run at least two, and I check the CoreDNS Pods first whenever a lookup fails.

Flannel vs Calico vs Cilium, as a choice

Lesson 105 teaches the three plugins in one paragraph each, and this table is the choice, not the re-teaching. The internals of Cilium live in Lesson 126, which teaches how eBPF works. This lesson only asks which plugin to pick.

FlannelCalicoCilium
ShapeOverlay. Every cross-node packet travels inside a tunnel.Routed. Pod subnets are routed directly, no tunnel, when the real network allows it.eBPF. Programs in the kernel do the lookup, the policy, and the balancing.
PolicyNone. Flannel builds the network and stops there.Yes. NetworkPolicy on top of the routing.Yes. Policy in the kernel, with Hubble showing the flows.
Pick it whenThe cluster is small and simple wins.The network team owns the routes and wants policy without a new dataplane.The cluster is big enough that the lookup and the policy are the bottleneck, and the team is ready to debug eBPF.
CostsA tunnel on every cross-node call.A real network that has to allow the routes.A kernel version and a team that understands eBPF.

Weave is the name you will see in old blog posts. Its upstream is dead, so I do not teach it as a choice. If you inherit it, the migration target is Calico or Cilium.

Mandatory vs replaceable, the matrix

This is the table I keep in my head when a part breaks. Mandatory means the cluster cannot run without it. Replaceable means another part can do the job, or the cluster runs without it.

PartMandatoryReplacement
kube-apiserverYes. It is the only door.None. You run more copies of it, not a different one.
etcdYes. It is the single source of truth.None in a standard cluster. You run three or five members, not a different store.
kube-schedulerYes, something must schedule.Another scheduler. The contract is the API server, so a custom scheduler can take the job.
kube-controller-managerYes, something must run the loops.Split loops, or custom controllers, but the standard loops stay.
cloud-controller-managerNo. Bare metal runs without it.The cloud provider's own controller, or nothing, on bare metal.
kubeletYes. Every node needs one.None. One kubelet per node, always.
kube-proxyNo.Cilium eBPF replaces it, and the Service still works.
Container runtimeYes, something must run containers.containerd or CRI-O, behind the same CRI contract.
CNI pluginYes, something must wire the network.Flannel, Calico, or Cilium.
CSI driverNo, unless the Pods need volumes.Any CSI driver for your storage, or host paths on a scratch cluster.
CoreDNSYes, something must answer names.Another DNS server that reads Service objects, or more CoreDNS replicas.

The matrix is the debugging shortcut. When a part is replaceable, the question is which replacement is running, because the failure mode follows the replacement. When a part is mandatory, the question is how many copies run, because the failure mode is the loss of the last copy.

Static Pods and kubeadm: how a control plane boots itself

The control plane has a chicken and egg problem. The API server, the scheduler, and the controller manager are Pods, and Pods need a control plane to run them. The answer is the static Pod. The kubelet starts before the control plane does, and it reads Pod manifests from /etc/kubernetes/manifests on the node. One file, one Pod, started by the kubelet alone, with no API server involved. The API server itself starts this way, and once it is up, the kubelet tells it about the static Pods, which show up as mirror Pods with the node name at the end. You cannot delete a static Pod with kubectl, because the API server does not own it. You delete the file, and the kubelet stops the Pod. That is the whole trick, and it is how kubeadm boots a cluster.

kubeadm builds the control plane in phases, and the outline is the list, not a walkthrough. Preflight checks the machine. Certs makes the certificates. Kubeconfig writes the admin config. Kubelet-start starts the kubelet. Control-plane writes the static Pod manifests for the API server, the controller manager, and the scheduler. Etcd writes the etcd manifest. Upload-config stores the cluster config in the cluster. Mark-control-plane labels the node. Bootstrap-token makes the token new nodes join with. Addon installs CoreDNS and kube-proxy. That is the order, and each phase is a step you can run on its own when a phase fails. I do not teach the install here, because this course has no cluster install lesson, and this lesson should not become one. The phases are the map of the boot. The install is a separate job.

Debugging at the component level

When something breaks, the symptom names the suspect, and the suspect names the log. I open the log for the suspect first, and I do not open the app log until the chain says the app is the suspect.

SymptomSuspectWhat I open first
kubectl hangs or refuses to connectkube-apiserverThe API server log on the control-plane node, then the etcd health.
Applies fail, nothing new is storedetcdThe etcd log for disk errors, then the member list and the disk space.
Pod stays Pending, no node namedkube-schedulerThe Pod Events, which carry the scheduler's reason, then the scheduler log.
Pod has a node and never startskubelet, or the runtimeThe kubelet log on that node, then the runtime log for image pulls.
Pod runs, Service never answerskube-proxyThe rules on the node, then the kube-proxy log.
Name does not resolveCoreDNSThe CoreDNS Pods in kube-system, then the CoreDNS log.
Pod cannot reach another PodCNI pluginThe routes on both nodes, then the plugin log.
Pod stays in ContainerCreatingCSI driver, or the runtimeThe Pod Events for the mount error, then the driver log.
Node stops reportingkubelet, or its certificateThe node status, then the kubelet log for certificate errors.

The table is the habit. Symptom, suspect, log, in that order. The habit is what turns a two hour outage into a ten minute one, because the first log you open is the right one.

What changes at 10x and 100x

At 10x the Pods, the etcd write rate is the first thing that breaks, and the fix is faster disks for etcd and fewer writes from controllers that watch too much. At 10x the Services, the kube-proxy rule list is the first thing that breaks, and the fix is IPVS or eBPF, not more rules. At 100x, the control plane is the product. The API servers run behind a load balancer, etcd runs on five members with its own disks, the scheduler and the controllers run as many copies as leader election allows, and the alert on etcd disk space fires before the disk fills. The parts do not change. The number of copies does.

Further reading

The desired-state model underneath all of it: Lesson 48 Kubernetes Core. The full scheduling pipeline: Lesson 197 How a Pod Gets a Home and Starts Running. The network side, and the kube-proxy modes: Lesson 105 Kubernetes Networking & CNI. The storage side, and the CSI contract: Lesson 106 Kubernetes Storage & CSI. How Cilium works inside the kernel: Lesson 126 eBPF. The locks on what a Pod may do: Lesson 202 Kubernetes Pod Security in Depth.

One time it broke

What broke

A cluster stopped taking changes on a Tuesday afternoon. New deploys hung at apply. New Pods stayed Pending. The running Pods kept serving traffic, and nobody's app was down, so the first hour was spent looking at the wrong thing. The deploys were fine. The apps were fine. The cluster had stopped changing, and nothing in the apps could explain that. The suspect was the store, and the store was etcd, and etcd had run out of disk.

Timeline

  • Tue 13:00The etcd data dir shares a disk with the container logs. The logs grow past the free space, and the disk fills.
  • Tue 14:10etcd stops taking writes. It stops before the disk is fully full, because a store that cannot write its log cannot promise anything.
  • Tue 14:15The first deploy hangs at apply. kubectl waits on a write that never lands.
  • Tue 14:30New Pods stay Pending. The scheduler has nothing new to schedule, because nothing new was ever stored.
  • Tue 15:00I read the API server log and see etcd timeouts. I read the etcd log and see no space left on device. The suspect was right.
  • Tue 15:20I free disk space, compact etcd, and defrag it. Writes land again, the queued applies finish, and the Pending Pods get their nodes.
  • Tue 16:00The cluster is changing again. The running Pods never stopped, because they never needed the store during the outage.

What changed after

Three things. First, etcd got a disk of its own, so container logs can never fill it again. Second, a disk space alert fires at 70 percent on that disk, and it pages, because a store that stops writing is an outage even when every app is up. Third, the etcd snapshot restore is tested once a month on a scratch machine, because the outage proved the restore matters more than the snapshot. The fix that lasted was the disk, not the bigger cluster. The cluster was never the problem. The shared disk was.

If you remember nothing else

Design exercise

Design the control plane for a team of 20 engineers running 40 services on 6 nodes. The team deploys 30 times a day. The cluster must survive one dead control-plane machine without losing a write, and the team must be able to tell, in the first minute of an outage, which part broke. How many control-plane nodes, how many etcd members, which kube-proxy mode, which CNI plugin, how many CoreDNS replicas, and where does etcd store its data? Name the trade-off you are making.

Clarifying questions to ask

  • Is the control plane managed by the cloud, or does the team run it? A managed control plane changes every answer about copies and disks.
  • How many Services will the 40 services become, and how many endpoints does each one have? The Service count sizes the kube-proxy mode.
  • Do the services need NetworkPolicy, or is the network open inside the cluster? Policy is the reason to pick Calico or Cilium over Flannel.
  • What is the recovery target for etcd: how much data may be lost, and how fast must the restore run? The answer sizes the snapshot schedule and the restore test.
  • Who gets paged when etcd disk space hits 70 percent, and who owns the control-plane logs? An alert with no owner is a log nobody reads.

Worked approach

Start with the store. One dead control-plane machine without losing a write means etcd needs three members, quorum two. Three control-plane nodes, one etcd member each. The etcd data lives on a disk of its own on each node, with an alert at 70 percent that pages. Snapshots run every hour, and the restore is tested once a month on a scratch machine, because the incident in this lesson proved the restore matters more than the snapshot.

Then the door and the loops. Three API servers behind a load balancer, so one dead machine leaves two doors open. The controller manager and the scheduler run on all three nodes with leader election, so one active copy works and the standbys wait. The cloud controller manager runs only if the cluster is in a cloud. On bare metal it does not exist, and the load balancer for a Service is a problem the team solves another way.

Then the node side. Forty services on six nodes is a small Service table, so kube-proxy stays on iptables, the default, and the team spends nothing learning IPVS. The CNI choice is Calico, because the team will want NetworkPolicy before the year is out, and Calico gives policy without a new dataplane. Flannel is simpler and it has no policy, so the team would be back here in six months asking for it. CoreDNS runs three replicas, so one dead replica is not an outage, and the lookup failure mode is covered by the debugging table.

Trade-offs to name. Three etcd members against five: three survives one dead member, five survives two, and five costs two more machines and slower writes. iptables against eBPF: iptables is the default the team already knows, and eBPF is faster and costs a team that can debug it. Calico against Flannel: Calico costs a network that allows the routes, and Flannel costs a tunnel on every cross-node call and no policy. Say which promise is the hard one before you size anything, because the sizing changes completely when it moves.

Saved in this browser, so the Lessons index can show your progress across the course.

Next in this series: Lesson 202
Kubernetes Pod Security in Depth

This lesson named the parts that run the cluster. The next lesson locks down what a Pod may do once it runs: the securityContext fields, Pod Security Admission, the scanner and linter bench, and the runtime watch that catches what every gate missed.

Continue to Lesson 202 →

Or browse the full course from the Lessons index.