Platform Engineering · Lesson 90

How a Pod Gets a Home and Starts Running

I apply one YAML file. The API server writes a Pod with no node named on it. Then the scheduler finds it a home, the kubelet on that node builds it, and the probes say it is ready. Nine hops, about two seconds on a cold pull, and every hop can say no. This lesson walks the Pod's first minute in order, and it teaches you to read the Events when any hop says no.

Difficulty: Advanced · Know this first: Lesson 82, Lesson 05, Lesson 86

What you will build

By the end of this lesson you will have read a Pod's birth in order: the empty nodeName, the scheduler queue, the filter, the score, the bind, the kubelet sync loop, the image pull, the pause container, the CNI address, the init containers, and the probes. You will have run the failure reads on your own cluster: a Pod that asks for more CPU than any node has, a Pod with an image tag that does not exist, and a Pod whose readiness probe checks the wrong path. The cluster steps run on your machine. This VM could not run a kubelet, and the lab says so plainly instead of faking the output. The simulators carry the scheduling math here.

What you need: a cluster you can break: k3s or kind on your own machine, plus kubectl. The lab runs there, not here. This VM installed k3s 1.37.2 and started its API server, but the server could not start a kubelet because the VM has no CPU cgroup, so no node ever joined and every Pod stayed Pending. The lab shows that real output, then hands the rest to you.

The story: one Pod, nine hops, one minute

A Pod is not a process. It is a promise the cluster makes to run some containers together, on one node, sharing one network and some volumes. The promise starts as a YAML file on my machine. It ends as a Ready Pod with an IP address and a place in the Endpoints list. Between those two points, four pieces of machinery touch the Pod in order: the API server writes it, the scheduler places it, the kubelet builds it, and the probes judge it. Each one can say no. Each no looks different. Learn to tell them apart and you stop fixing the wrong one.

Born with no home

The API server is the only door into the cluster. Everything goes through it. When I apply a Pod, the API server checks the YAML, writes the Pod to etcd, and the Pod is born with nodeName empty. That empty field is the whole story of this lesson. It means the Pod exists and nobody has claimed it yet. The scheduler watches etcd for exactly that shape: a Pod with no node. The kubelet watches for the opposite shape: a Pod with its own node named. One field, two watchers, and the handoff between them is the Pod's first minute.

Filter, then score

FilterScore
MeansThrows nodes out.Ranks the survivors.
AsksDoes this node fit the Pod?Which node fits best?
ResultA shorter list, or an empty one.One winner.
When it failsFailedScheduling, and the Pod stays Pending.It never fails alone. No survivors means no score.
The knobsRequests, nodeSelector, taints, affinity, VolumeBinding.Free resources, spread, affinity weight.

The filter is the honest part. It does not guess. A node either has the free CPU and memory or it does not. The label either matches or it does not. The taint is either tolerated or it is not. The score is the preference part. It likes the node with the most room and the node that spreads Pods out. A Pod that fails the filter never reaches the score, and that is why the first question after a Pending Pod is always a filter question: which check threw my nodes out?

The kubelet loop

The kubelet runs a sync loop. It looks at the Pod as it should be, looks at the containers as they are, and closes the gap. It does this every few seconds for the life of the Pod. The loop is the reason a container that dies gets restarted without anyone asking, and the reason a Pod that changes gets rebuilt to match. The kubelet does not decide where the Pod goes. That decision was made at the bind. The kubelet only builds what the bind assigned to its node. When the loop cannot close the gap, it writes an Event that says which hop is stuck: the pull, the mount, or the probe.

Three probes, three jobs

LivenessReadinessStartup
AsksIs the app still alive?Can the app take traffic?Has the app finished starting?
When it failsThe container restarts.The Pod leaves Endpoints.Liveness and readiness wait.
When it passesNothing changes.Traffic flows.The other probes start.
The mistakeA slow start looks dead.A starting Pod takes traffic too early.No startup probe, and a slow app is killed while warming up.

Running is not ready. A container can run and serve nothing, and a Pod can be alive and take no traffic. The probes are how the kubelet tells those apart. The startup probe buys a slow app time to warm up. The readiness probe is the gate to traffic. The liveness probe is the restart button. Lesson 82 teaches what the probes mean. This lesson teaches who runs them, when they run, and what their failures look like in the Events.

imagePullPolicy, in one table

AlwaysIfNotPresentNever
MeansPull on every start.Pull only when the cache misses.Never pull, cache only.
The default forThe latest tag.Any other tag.Nothing. You set it yourself.
The trapA slow registry on every restart.A stale cache when the tag moves.A node without the image never starts.

The policy is the kubelet's instruction for the pull, and the default depends on the tag. A Pod with the latest tag pulls every time it starts. A Pod with a pinned tag trusts the cache. A tag that moves under the cache is how a Pod runs old code with a new name on it. Pin the tag, or pin the digest, and the cache stops lying to you.

Do this now, before the diagram

On your own cluster, run kubectl get pods -A and find one Pod that is Running. Then run kubectl describe pod <name> and read the Events at the bottom. The Events are the Pod's diary: Scheduled, Pulled, Created, Started, and the probe results. Every hop in this lesson leaves a line there. Then run kubectl get events --sort-by=.lastTimestamp and read the last ten lines of the whole cluster. The newest Events are where a failing Pod talks first.

Run it as:

When it breaks, I check

One thing to remember

Four paths, one family

The diagram above follows one Pod from kubectl apply to Ready on three nodes, pulling a 120 MB image cold. These four paths are the same idea drawn four ways. One Pod, running. One Pod, homeless. One Pod, unpullable. One filter, counted node by node. Run each path. One run takes 7.2 seconds across the path, so you can watch each hop light up in turn.

The return rule: the green dot only starts moving after the orange dot reaches the data hop. A return with no request is a lie. The diagram never draws one. The failure replay at the end of Run all four shows the other side of the rule. Run it and watch the orange dot stop at the filter and never come back.

Path 1: the healthy Pod

FailedScheduling: skipped

One Pod walks the chain from the story. kubectl, scheduler, kubelet, pull, pause and CNI, init and app, probes. The Pod is Ready at the probes hop, and the green dot starts back from there.

kubectl0 ms
Scheduler52 ms
Kubelet90 ms
Pull1,200 ms
Pause + CNI80 ms
Init + app260 ms
Probes Ready400 ms

Not run yet.

Path 2: no node fits

Bind: skipped

Same Pod, a bigger ask. The queue is fine. The filter throws all three nodes out for Insufficient cpu. The Pod never gets a bind, and the answer never comes back. The user sees Pending, not an error page.

kubectl0 ms
Scheduler queue15 ms
Filter25 ms
Score + bindskipped
Kubeletskipped
Pullskipped
Probesskipped

Not run yet.

Path 3: the image will not pull

Pod IP: skipped

Same Pod, a tag that does not exist. The scheduler binds it in 52 ms and the kubelet picks it up. Then the pull fails and the backoff starts. The pause container never starts and the Pod never gets an IP.

kubectl0 ms
Scheduler52 ms
Kubelet90 ms
Pullfails
Pause + CNIskipped
Init + appskipped
Probesskipped

Not run yet.

Path 4: the filter, counted

FailedScheduling: skipped

The same Pod, counted instead of timed. Five nodes enter the filter. NodeResourcesFit keeps three. Taints keep two. Affinity keeps one. VolumeBinding keeps one. One survivor wins, and the scheduling cycle took 52 ms. The count is the check that keeps a Pod off a node that cannot hold it.

Nodes5
Resources fit3
Taints pass2
Affinity pass1
VolumeBinding pass1
Winnernode-3
Verdict52 ms

Not run yet.

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one Pod's first minute, not a measurement from a live cluster, but the shape is honest. The pull hop is the big one at 1,200 ms, and the total is 2,082 ms. Switch the scenario and watch the bar change. In the no-node-fits story the trip stops at the filter, so the total is 40 ms of trying and no Pod. In the bad-image story the pull hop grows to 5,000 ms of backoff, because the kubelet waits longer after every failed pull, and the hops after it disappear. Move a simulator slider and watch the filter, the pull, and the total move with it.

Nodes, requests, taints, or image size: move the sliders and watch the Pod find a home or fail

The budget bar above shows three fixed stories. This simulator lets you write your own. It answers the scheduling question at the heart of this lesson: which nodes survive the filter for your Pod, who wins the score, and how long the pull takes. You set how many nodes the cluster has, how much CPU the Pod requests, how many nodes carry a taint the Pod does not tolerate, and how big the image is. Move a slider and watch the survivors, the winner, the pull time, the time to Ready, and the budget total change. The numbers are a teaching example, built from the formulas below, not live measurements. The shape is the lesson: the filter only removes, and a Pod with no survivors stays Pending.

3 nodes

How many nodes the scheduler can choose from. More nodes mean more chances one of them fits.

500 millicores

How many millicores the Pod requests. The scheduler counts requests, not limits. Every node here has 1,000 millicores free.

0 nodes

How many nodes carry a taint this Pod does not tolerate. The filter throws those nodes out before scoring.

120 MB

How many megabytes the image is. A cold pull moves about 10 MB a second, so a big image is a slow hop.

Try this: (1) At the defaults, 3 nodes, 500 millicores, no taints, a 120 MB image, 3 nodes pass the filter, node-1 wins, the pull takes 1,200 ms, the time to Ready is 2,082 ms, and the budget total is 2,082 ms. (2) Raise the request to 1,500 millicores and leave the rest at default. No node has 1,500 free, so 0 nodes pass the filter and the Pod stays Pending. The budget total is 40 ms: the queue and the filter, and nothing after them. The status line says FailedScheduling. (3) Taint 2 of the 3 nodes and leave the rest at default. One node passes the filter, node-3 wins, the pull takes 1,200 ms, the time to Ready is 2,082 ms, and the budget total is 2,082 ms. The taints removed nodes, but one survivor is enough. (4) Push the image to 800 MB and leave the rest at default. Three nodes pass the filter, node-1 wins, the pull takes 8,000 ms, the time to Ready is 8,882 ms, and the budget total is 8,882 ms. The status line says the pull is slow. The scheduling was fine. The image was the slow hop.

Formulas, in plain terms. Every node has 4,000 millicores of CPU and 3,000 already used, so 1,000 free. A node passes the filter when the request is 1,000 millicores or less and the node carries no taint this Pod does not tolerate. Survivors are the nodes that pass. The winner is the first untainted survivor. The pull takes image size times 10 ms, a cold pull at 10 MB a second. Time to Ready is 15 plus 25 plus 12 plus 90 plus the pull plus 80 plus 260 plus 400, when a survivor exists. When no survivor exists, the bind, the kubelet, the pull, the pause, the init, and the probes all cost 0 ms, and the budget total is 40 ms. The status is FailedScheduling when no node survives, slow pull when the pull passes 3,000 ms, and balanced otherwise.

Lab: make a Pod fail three ways, then read the Events

Three steps, all on your machine. This VM could not run them, and the lesson says so instead of faking the output. k3s 1.37.2 installed here on Nov 21, 2026. The full server failed at startup with "failed to find cpu cgroup (v2)". The server with the agent disabled started and answered kubectl, but no node ever joined, so every Pod stayed Pending. Then the VM service restarted and wiped the install before the first test Pod was applied. The one real output from this VM is below. The rest runs on your cluster.

What this VM actually showed:

kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE
kube-system   coredns-577d995dff-55fhf                  0/1     Pending   0          28s
kube-system   helm-install-gateway-api-crd-j78rp        0/1     Pending   0          28s
kube-system   local-path-provisioner-56c6584955-cgt8f   0/1     Pending   0          28s

kubectl get nodes printed "No resources found". No node, no kubelet, no pull, no probes. That Pending list is what a cluster with no nodes looks like, and it is the same shape your unschedulable Pod will show in step 2.

Step 1: get a cluster

Install k3s or kind on your own machine, then check the cluster answers before you apply anything.

kubectl get nodes
kubectl cluster-info

Read the node list first. A cluster with no Ready node cannot schedule anything, and every Pod you apply will sit Pending for the same reason the Pods above did. Do not debug a Pod until a node is Ready.

Step 2: a Pod no node can hold

This Pod asks for 100 whole CPUs. No node has that much free. Apply it, then read what the scheduler says.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: hungry
  labels:
    app: hungry
spec:
  containers:
  - name: web
    image: web:2.4.1
    resources:
      requests:
        cpu: "100"
        memory: "100Gi"
EOF
kubectl get pod hungry
kubectl describe pod hungry
kubectl get events --sort-by=.lastTimestamp

Look for a Warning event named FailedScheduling. Its message counts the nodes and names the reason each one failed. The Pod stays Pending and the event count climbs each time the scheduler retries. That climbing count is the scheduler working, not stuck. Delete the Pod when you are done: kubectl delete pod hungry.

Step 3: a Pod with an image that does not exist

This Pod asks for a tag the registry never had. Apply it, then watch the backoff grow.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: badtag
spec:
  containers:
  - name: web
    image: shop-web:does-not-exist
EOF
kubectl get pod badtag
kubectl describe pod badtag

Look for Failed to pull image in the Events, then the Pod status ImagePullBackOff. The kubelet retries with a longer wait each time, so run kubectl get pod badtag twice, a minute apart, and watch the status stay the same while the Events gain another Failed entry. The fix is the tag, not the cluster. Fix the tag to a real one, re-apply, and the Pod starts. Delete the Pod when you are done: kubectl delete pod badtag.

Step 4: a Pod that runs but never turns Ready

This Pod runs a web server and a Service in front of it. The readiness probe checks a path the server does not serve, so the Pod never turns Ready and the Service never gets an address.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: web
  labels:
    app: web
spec:
  containers:
  - name: web
    image: web:2.4.1
    ports:
    - containerPort: 80
    readinessProbe:
      httpGet:
        path: /healthz
        port: 80
      initialDelaySeconds: 2
      periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
  - port: 80
EOF
kubectl get pod web
kubectl describe pod web
kubectl get endpoints web

Look for Unhealthy events naming the readiness probe. The Pod says Running, but the Ready column says 0/1. kubectl get endpoints web shows the Service with no addresses, because a Pod that fails readiness never joins Endpoints. Now fix the probe path to /, re-apply, and watch the Pod turn 1/1 Ready and the Endpoints gain an address. That one-line fix is the whole lesson about probes: the probe and the app have to agree on the path. Delete the Pod and the Service when you are done: kubectl delete pod web service web.

Check yourself

Three questions. No score is saved anywhere. This is just you against the material. Every answer is a hop in the Pod's first minute.

Basic to master

Scheduling, pulling, and probing are not three skills. They are four levels of the same job, and every team I have worked with sits on one of them. The levels are not about how many field names you can recite. They are about whether the Events change what gets fixed. Read them in order. Each level fixes the failure that hurt the most at the level below it.

Basic · Read the Events and name the hop

At the basic level you read kubectl describe pod and you name the hop that said no. You can say what Pending means, what ImagePullBackOff means, and what Unhealthy means. You can say which container is stuck and which Event names it. The failure that ends this level is the first Pod that stays Pending with no obvious reason, and you cannot say which filter threw it out.

Types:

  • Pending: not yet scheduled, or waiting for volumes.
  • ContainerCreating, ImagePullBackOff, ErrImagePull: the image and sandbox phase.
  • CrashLoopBackOff: the container starts and keeps exiting.
  • Events from kubectl describe pod: the scheduler, kubelet, and probes explaining themselves.

Uses: The first five minutes of any "my Pod is not running" question: the Events name the component that said no.

Scale and failure: On a busy cluster, Events expire after about an hour by default, so the reason for an old failure may be gone. Capture Events to your logging system if you need them later.

Trade-off: kubectl describe is quick and shows one Pod at a time; cluster-wide views of Pending Pods and their reasons need monitoring built for it.

Do this: run step 2 of the lab on your own cluster and write down the FailedScheduling reason, word for word.

Try it: Basic

My pod sits in Pending with no obvious reason. What do I read first?

Now do it: On your cluster, run step 2 of the lab and write down the FailedScheduling reason, word for word.

Intermediate · Requests, filters, and the score

At the intermediate level you predict the filter before you apply the Pod. You can say how much CPU and memory each node has free, and you can say which nodes your Pod's requests, nodeSelector, taints, and affinity leave standing. You know the scheduler counts requests, not limits, and you know the bind is the write of nodeName. You can say why a Pod with no requests lands anywhere and why a Pod with big requests lands nowhere.

Types:

  • Resource requests: what the scheduler reserves on a node.
  • Filters: node selectors, taints and tolerations, node and Pod affinity, volume topology.
  • Scoring: spreading, packing, and image locality among the nodes that pass.
  • Binding: writing the chosen node name onto the Pod.

Uses: Placing workloads on the right nodes, such as GPU work on GPU nodes and system Pods off application nodes, and predicting whether a Pod will fit.

Scale and failure: The scheduler sees requests, not real usage, so a cluster can be "full" while nodes sit at 20 percent CPU, or overcommitted when requests are too low. Topology spread constraints keep replicas from landing in one zone.

Trade-off: Strict affinity rules place Pods precisely and leave them Pending when no node matches; soft preferences always schedule and sometimes in the wrong place.

Do this: take the simulator defaults and raise the request to 1,500 millicores. Watch the survivors fall to zero. Then lower it to 500 and taint two nodes. Watch one survivor win.

Try it: Intermediate

Two pods ask for the same CPU. One asks with requests and one with limits only. Which one does the scheduler count?

Now do it: Run the simulator drill above. Raise the request to 1,500 millicores and watch the survivors fall to zero, then lower it to 500 and taint two nodes.

Advanced · The kubelet loop, the pull, and the probes

At the advanced level you read the startup side. You can say what the CRI does, what imagePullPolicy each tag needs, and why the pause container starts first. You can say when the CNI plugin runs and why the Pod has no IP before it does. You can set all three probes, and you can say which threshold is wrong when a healthy Pod restarts or a ready Pod never takes traffic. You know init containers run in order and a native sidecar is an init container that stays.

Types:

  • Container runtime interface (CRI): containerd or CRI-O pulling images and starting containers.
  • imagePullPolicy: Always, IfNotPresent, Never.
  • Probes: startup, liveness, readiness, with their thresholds.
  • Init containers and native sidecars.

Uses: Diagnosing Pods that schedule and then fail to start, never become ready, or restart in a loop.

Scale and failure: Large images pulled by hundreds of Pods at once can saturate node disks and registry bandwidth during a scale-up. A liveness probe that is too strict restarts healthy Pods under load, which adds more load: a restart storm.

Trade-off: Aggressive probes detect real failures quickly and cause false restarts under load; lenient probes are stable and slow to notice a hung process.

Do this: run step 4 of the lab and fix the probe path. Then set imagePullPolicy to Always on a Pod with a latest tag and watch the pull run every time.

Try it: Advanced

My pod uses the latest tag and I want the pull to run every time. Which imagePullPolicy do I set?

Now do it: On your cluster, run step 4 of the lab and fix the probe path. Then set imagePullPolicy to Always on a pod with a latest tag and watch the pull run every time.

Master · Hold the line on Pending and on slow starts

At the master level the cluster is watched, not just debugged. An alert fires when a Pod sits Pending for more than five minutes, and the runbook starts with the FailedScheduling reason, not with a restart. Requests are set from measured use, so the filter has room to work and the nodes are not packed to the edge. Images are pinned by digest, so a pull never fails on a tag that moved. Probes are set from measured startup time, so a slow app is given room and a dead app is restarted fast. The team still reads the Events every week. The Events read the cluster, not a guess.

Types:

  • Alerts on Pods Pending longer than a threshold, with the scheduler's reason attached.
  • Requests set from measured usage and reviewed regularly.
  • PriorityClasses and preemption for critical workloads.
  • Cluster Autoscaler or Karpenter reacting to unschedulable Pods.

Uses: Multi-team clusters, where scheduling failures are a capacity problem, a configuration problem, or a priority decision, and each needs a different owner.

Scale and failure: Autoscalers add nodes in minutes, not seconds, so critical workloads need priority over best-effort ones to get capacity first. Preemption evicts lower-priority Pods, which must be designed to tolerate it.

Trade-off: Overprovisioned headroom schedules everything instantly and costs money every hour; tight clusters are cheap and make every spike a wait for new nodes.

Do this: pick one namespace you run, list the Pods that sat Pending in the last week, and write down the FailedScheduling reason for each one. Then check whether any of them asked for more than the nodes had free.

Try it: Master

A pod has sat Pending for more than five minutes. Where does the runbook start?

Now do it: On your cluster, list the pods that sat Pending in the last week and write down the FailedScheduling reason for each one. Then check whether any of them asked for more than the nodes had free.

Go deeper

The scheduling framework, named

The scheduler is not one big function. It is a framework with named extension points, and each point runs plugins in order: PreFilter, Filter, PostFilter, PreScore, Score, Reserve, Permit, PreBind, Bind, PostBind. NodeResourcesFit, NodeAffinity, PodTopologySpread, TaintToleration, InterPodAffinity, and VolumeBinding are Filter plugins. The framework is the reason a custom scheduler can add one plugin without rewriting the rest. You do not need to write one. You need to know the names, because the FailedScheduling message names the plugin that threw your nodes out.

Taints and tolerations, in one paragraph

A taint marks a node as picky. A toleration is the Pod saying it accepts the mark. The effects are NoSchedule, which keeps new Pods off the node, PreferNoSchedule, which asks the scheduler to avoid it, and NoExecute, which evicts Pods that are already there. A toleration names a key, an operator, a value, and an effect. A NoExecute toleration can carry tolerationSeconds, which says how long the Pod may stay after the taint appears. Taints are how a team keeps one workload off the nodes another workload owns.

Topology spread

topologySpreadConstraints say how evenly the Pods of one workload must spread across zones or nodes. maxSkew is the biggest allowed gap between the fullest place and the emptiest one. topologyKey names what counts as a place: a zone, a node, a rack. whenUnsatisfiable picks the behavior: DoNotSchedule makes the spread a rule, ScheduleAnyway makes it a preference. Spread is a score-level preference unless DoNotSchedule makes it a rule, and a rule that cannot be met is a Pod that stays Pending.

The WaitForFirstConsumer handshake

A StorageClass with volumeBindingMode WaitForFirstConsumer delays provisioning until the scheduler picks a node. The VolumeBinding filter checks that the volume can attach to each candidate node, so the scheduler will not pick a node the volume cannot reach. Then provisioning runs against that node. The handshake is the reason a Pod with a volume lands where its storage lives. The CSI calls themselves live in Lesson 89, Kubernetes Storage and CSI, which names the call chain end to end. This lesson teaches the handshake. That lesson teaches the calls.

imagePullPolicy, in one paragraph

Always pulls on every start. IfNotPresent trusts the cache. Never refuses to pull at all. The default is Always for the latest tag and IfNotPresent for any other tag. Always is honest and slow on a slow registry. IfNotPresent is fast and stale when a tag moves. Never is for nodes with preloaded images and no registry access. Pick the policy from the tag: a moving tag needs Always, a pinned tag can trust the cache, and a preloaded node needs Never.

The pause container, in one paragraph

The pause container holds the Pod's network namespace and its IPC namespace open. It is tiny and it never does app work. Every other container in the Pod joins that namespace, and that is why they share one IP and one localhost. When the pause container dies, the Pod's network dies with it, and the kubelet rebuilds the whole Pod. You never configure it. You only need to know it is first, and that the IP belongs to it, not to your app.

CNI ADD and DEL

The kubelet calls the CNI plugin twice in a Pod's life: ADD when the Pod starts, DEL when the Pod is deleted. ADD wires the interface into the pause container's namespace and returns the IP. DEL removes the interface and frees the address. The plugin binary and its config live on the node. Lesson 86, Kubernetes Networking and CNI, teaches how to pick the plugin. This hop is when the plugin runs, and the Pod has no IP before it does.

Native sidecars

A native sidecar is an init container with restartPolicy Always. It starts before the app containers, like any init container, and it keeps running beside the app after it starts. Proxies and log shippers are the usual sidecars. The sidecar terminates after the app containers stop, so it never outlives the Pod it serves. A plain init container exits and stays done. A sidecar starts early and stays late. That is the whole difference, and it is one field.

Probe thresholds

Five numbers set a probe: initialDelaySeconds waits before the first check, periodSeconds sets how often it runs, timeoutSeconds sets how long one check may take, successThreshold sets how many passes count as healthy, failureThreshold sets how many failures count as broken. failureThreshold times periodSeconds is the time a probe gets before it is called failed. The startup probe gates the other two: while it runs, liveness and readiness wait. A startup probe with a generous failureThreshold is how a slow app gets time to warm up without being killed for it.

Reading Events like a log

Events are the cluster's log for one object. kubectl get events --sort-by=.lastTimestamp shows the newest first. --field-selector involvedObject.name=web shows one Pod's Events only. kubectl describe pod shows the same Events at the bottom. The reasons to know: FailedScheduling, FailedMount, Failed, ImagePullBackOff as a state, Unhealthy, Killing, Started, Pulled. Warning means something failed. Normal means something happened. A Warning with a climbing count is a loop. Read the count before you read the message.

What changes at 10x and 100x

At 10x the Pods, the scheduler queue is the first thing that breaks, and the filter cost per node is the number that grows. A big image pulled by many Pods at once is a pull storm on one node, and the registry rate limit is the wall behind it. At 100x, the scheduler throughput is the product. The image cache is warmed before the Pods arrive, topology spread becomes a rule instead of a preference, and the alerts on Pending Pods fire before the users notice. The per-Pod math does not change. The number of Pods it runs on does.

Further reading

The one-liner chain this lesson opens up, and Pending and probe behavior: Lesson 82 Kubernetes Core. The CNI plugin choice: Lesson 86 Kubernetes Networking and CNI. The CSI call chain and WaitForFirstConsumer: Lesson 89 Kubernetes Storage and CSI. CPU and memory requests as the scheduler's inputs: Lesson 05 CPU, Memory and Threads. Admission and who may create Pods: Lesson 158 Kubernetes Security. The hands-on apply, scale, and describe: Lesson 83 Kubernetes Lab: First Pod. The controller that makes the Pods this lesson places: Lesson 91 Kubernetes Deployments.

One time it broke

What broke

A checkout Pod stayed Pending for a whole morning after a deploy. The deploy added a nodeSelector for a label the team had written down in a runbook but never applied to any node. The scheduler tried, the filter threw all twelve nodes out, and the Events said so in plain words. Nobody read the Events for three hours. The team restarted the deploy twice, rolled the image back once, and added a node that changed nothing, because the problem was never the image or the nodes. It was one label that did not exist anywhere in the cluster.

Timeline

  • 09:02The deploy applies. The new Pod stays Pending.
  • 09:20The deploy is restarted. The new Pod stays Pending.
  • 09:45The image is rolled back. The new Pod stays Pending.
  • 10:30A node is added. The new Pod stays Pending.
  • 11:05I read kubectl describe pod and see FailedScheduling: the nodeSelector matches no node.
  • 11:20The label is applied to three nodes. The Pod schedules in under a second and turns Ready.

What changed after

Three things. First, the label check runs in CI before any deploy with a nodeSelector: the label must exist on at least one node, or the deploy stops. Second, an alert fires when a Pod sits Pending for more than five minutes, and the runbook starts with the FailedScheduling reason. Third, the team reads the Events before restarting anything. A restart fixes a stuck process. It never fixes a filter.

If you remember nothing else

Design exercise

Place the checkout Pod. A cluster has three nodes. Node-1 has 1,200 millicores free and no taints. Node-2 has 800 free and a taint the Pod does not tolerate. Node-3 has 2,000 free and the zone label the Pod's affinity wants. The Pod requests 900 millicores, tolerates no taints, and wants the zone label. The image is 400 MB and cold. The readiness probe needs 400 ms to pass. Which node wins, how long to Ready, and what breaks first if traffic doubles?

Clarifying questions to ask

  • What are the requests, and are they measured or guessed? The filter counts requests.
  • Which nodes carry taints, and does the Pod tolerate them? A taint the Pod does not tolerate removes the node before scoring.
  • Is the image cached on any node? A cold pull is the slow hop.
  • What does the readiness probe check, and how long does the app take to warm up? The probe decides when traffic flows.
  • What happens to the free CPU when traffic doubles? The filter is the first thing that breaks.

Worked approach

Start with the filter. Node-1 has 1,200 free and the Pod asks 900, so it passes. Node-2 has 800 free, which is less than 900, and a taint the Pod does not tolerate, so it fails twice. Node-3 has 2,000 free and the zone label, so it passes. Two survivors.

Then the score. Node-3 has more free CPU and matches the affinity, so node-3 wins. The bind writes node-3 into nodeName.

Then the time. Scheduling is 52 ms. The kubelet takes 90 ms. The cold pull is 400 MB at 10 MB a second, which is 4,000 ms. Pause and CNI take 80 ms. Init takes 260 ms. The probe takes 400 ms. Ready in 4,882 ms. The pull is the slow hop.

Trade-offs to name. Requests against placement: a bigger request is safer for the Pod and harder to place. Affinity against spread: wanting one zone fills that zone first. Image size against start time: a big cold image is a slow first minute, and a cached image is a fast one. Say which SLO is the hard one before you place anything, because the placement changes completely when it moves.

Saved in this browser, so the Lessons index can show your progress across the course.

Next in this series · Lesson 91
Deployments & ReplicaSets: The Rolling Update, Step by Step

This lesson followed one Pod from nothing to Ready. The next lesson follows the controller that makes Pods: the Deployment that owns the ReplicaSet, the ReplicaSet that counts the gap, and the rolling update that replaces Pods one wave at a time.

Go to Lesson 91 →

Or browse the full course from the Lessons index.