Platform Engineering · Lesson 48

Kubernetes Core

You write a YAML file. You run one command. Then a dozen pieces of machinery argue about where your code should run, and a minute later it is serving traffic. Kubernetes is that machinery. This lesson walks a deploy from the command line to a live Pod, hop by hop, the way I trace it when a deploy stalls.

Difficulty: Advanced · Know this first: Lesson 47, Lesson 26

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: in most slow deploys I have chased, one or two hops eat almost the whole budget. Step through the lesson or switch the scenario and watch the bar change.

Try it: size the Deployment and watch the node fill up

The budget bar above shows three fixed stories. This simulator lets you size a real Deployment. Pick how many replicas you want and how much CPU each Pod requests. The node has 4 vCPU to give out. Watch which Pods fit, which wait in Pending, what QoS class you get, and what the node bill looks like. The numbers are a teaching example, not a live quote. The shape is the lesson: requests are reservations, and reservations run out.

3

How many copies of the Pod the Deployment asks for. From 1 to 10.

250

The CPU each Pod reserves. The scheduler adds up every request on a node and refuses to go past 4000 millicores. Total requested equals replicas times request. A request of 0 means the Pod asks for nothing.

500

The ceiling each Pod may burst to. A limit of 0 means no ceiling. If the limit is below the request (and the limit is not 0), the API server rejects the Pod: the limit must be greater than or equal to the request. Memory is fixed in this example at 512 Mi requested and 512 Mi limited, so the CPU numbers decide the QoS class: request and limit both 0 is BestEffort, request equal to limit is Guaranteed, anything else is Burstable.

Try this: (1) Leave the defaults: 3 replicas at 250 millicores. Total requested is 750 millicores, all 3 Pods fit on one node, the bill is $146 a month, QoS is Burstable, and the deploy total is about 210 ms. (2) Set replicas to 10 and request to 500. Total requested is 5000 millicores, only 8 Pods fit, 2 wait in Pending, the bill is $292 a month for 2 nodes, the scheduler hop is 135 ms, and the deploy total is about 330 ms. (3) Set request to 500 and limit to 500 for Guaranteed, then set both to 0 and watch it drop to BestEffort.

The cluster, from scratch: one command, a dozen pieces of machinery

Extra diagram. I read the picture first: the boxes are the system, the moving dot is a deploy, and each step adds one piece until the full picture is there. No text needed to follow it.

Desired state vs actual state, closed by loopsScheduler counts requests, not limitsDeployment makes a ReplicaSet, ReplicaSet makes Pods
123456789

I type itControl planeOn the nodeMoving dots: request · response

The deploy that stalls: Pending, and the probe that kills healthy Pods

Extra diagram. I read the picture first: the boxes are the system, the moving dot is a deploy that stalls, and each step adds one piece until the full picture is there. No text needed to follow it.

Pending means no room, not an errorReadiness decides traffic, liveness decides restartsEviction order: BestEffort first, Guaranteed last
12345678

Happy pathStalledThe loopMoving dots: request · response

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

The lesson above walks a deploy from the command line to a live Pod. This section covers the same machinery four times, from the first YAML file to the level where I debug a cluster at 2 AM.

Basic — Desired state, stated once

Definition: You write a YAML file. You run one command. Then a dozen pieces of machinery argue about where your code should run, and a minute later it is serving traffic. Kubernetes is that machinery. The file says what good looks like. The cluster does the steps.

Types:

  • The Pod — one or more containers, running together, sharing a network.
  • The Deployment — the object that says how many Pods, and which version.
  • The Service — the stable address in front of Pods that come and go.
  • The Namespace — the boundary that keeps one team's things apart from another's.

Uses: I use this level every time I deploy. I write the file, I apply it, and I check the Pods. If a Pod is not running, I read its events before I read its logs. The events say what the cluster tried to do.

Dimensions:

  • Load: At one Deployment, the cluster is quiet. At a hundred, the API server answers thousands of requests a minute, and a bad client that lists everything on a loop can slow it for everyone.
  • Breaks: It breaks when the YAML says one thing and the cluster does another, usually because I applied the file to the wrong namespace. I notice it when the deploy succeeds and the service does not change.
  • Cost: The cost is the control plane and the nodes, running all the time. A cluster with three nodes and no workloads still costs money.
  • Trade-off: I give up running the process myself to get a machine that watches it. The cluster restarts what dies. It also decides where things run, and I have to learn to ask it, not tell it.

Intermediate — Controllers and reconciliation

Definition: Nothing in Kubernetes runs once and stops. Every controller is a loop: read the desired state from etcd, read the actual state from the cluster, and close the gap. The Deployment controller makes a ReplicaSet. The ReplicaSet controller makes Pods. The scheduler places them. If I delete a Pod by hand, the loop notices within seconds and makes a new one.

Types:

  • The Deployment controller — desired replicas, kept true.
  • The ReplicaSet controller — the Pods, kept at the right count.
  • The scheduler — the node, chosen by resources and rules.
  • etcd — the source of truth, where the desired state lives.

Uses: I use this level when a deploy stalls. I trace the loop: did the Deployment make a ReplicaSet, did the ReplicaSet make Pods, did the scheduler place them, did the Pod start. The step that did not happen is the problem.

Dimensions:

  • Load: At small scale the loop is instant. At large scale the controllers queue up, and a Deployment that changes a thousand Pods takes minutes to settle. I watch the queue, not just the Pods.
  • Breaks: It breaks when the desired state is wrong. The loop is faithful. It will make the wrong thing, exactly as written, forever. I notice it when the cluster keeps recreating the thing I keep deleting by hand.
  • Cost: The cost is the control plane: etcd, the API server, the controllers, all running and all needing care. A small cluster pays it. A large one pays it and needs someone to watch it.
  • Trade-off: I give up imperative control to get a system that heals itself. I do not have to remember the steps to fix the system. I only have to state what good looks like, and the loop does the steps.

Advanced — Probes, scheduling, and failure modes

Definition: Readiness and liveness look alike and they do opposite jobs. Readiness decides traffic. Liveness decides restarts. A readiness probe should check whether the Pod can serve this minute, including its database connection. A liveness probe should check only whether the process itself is alive, nothing downstream.

Types:

  • The readiness probe — can this Pod serve traffic right now.
  • The liveness probe — is this process alive, checked with nothing downstream.
  • The startup probe — time to warm up, so the liveness probe does not fire while the app is still starting.
  • Resource requests and limits — the scheduler's numbers, and the numbers that kill a Pod that uses too much.

Uses: I use this level when Pods die for no good reason. The first thing I check is the probes. If the liveness probe depends on the database, a slow database kills healthy Pods, and the killing makes the database slower. I have seen that loop take down a whole Deployment in minutes.

Dimensions:

  • Load: At one Pod, a bad probe is one restart. At a hundred Pods, a bad probe is a restart storm, and the storm takes down the database it was checking.
  • Breaks: It breaks when the probe endpoint checks the database and the database is slow. The kubelet decides the Pod is dead and kills it. The replacement Pod does the same thing. I notice it in the restart count, climbing in a loop.
  • Cost: The cost is the probes themselves: a little CPU, a little latency, and the time to get them right. A probe that is too strict costs more than a probe that is too loose.
  • Trade-off: I give up the simplicity of one health endpoint to get three different questions. Answering them with one endpoint is how good Pods die.

Master — Running a cluster you can debug at 2 AM

Definition: At this level the cluster is a system I operate, not a tool I use. I know what normal looks like: the control plane latency, the etcd size, the scheduler queue, the node pressure. When a deploy stalls, I do not guess. I read the events, the conditions, and the controller logs, in that order.

Types:

  • The control plane SLO — API server latency and etcd health, watched like a service.
  • Node pressure — disk, memory, and PIDs, and the evictions they cause.
  • The upgrade — one control plane version at a time, nodes drained, Pods moved, never all at once.
  • The runbook — the deploy stall, the pending Pod, the crash loop, each with its first three checks.

Uses: I use this level on the night the cluster itself is the incident. A node runs out of disk, the Pods evict, the scheduler has nowhere to put them, and the app team pages me. The cluster is the service then, and I treat it like one.

Dimensions:

  • Load: At this scale the control plane is the bottleneck. A thousand Pods starting at once is a thundering herd on the API server. I roll deploys, I stagger them, and I watch the queue.
  • Breaks: It breaks when etcd fills up, or when a node pressure eviction takes out the Pods that were keeping the service up. I notice it when the cluster stops answering and the app dashboards go quiet at the same time.
  • Cost: The cost is an operator's time, every week, and the control plane bill. A cluster that nobody watches is a cluster that fails on a Friday night.
  • Trade-off: I give up the freedom to change things by hand to get a cluster that survives the change. Every fix goes through the API, with a record. A hand edit is invisible, and invisible is unfixable.

Go deeper

Controllers and reconciliation are the first thing to get straight. Nothing in Kubernetes runs once and stops. Every controller is a loop: read the desired state from etcd, read the actual state from the cluster, and close the gap. The Deployment controller makes a ReplicaSet. The ReplicaSet controller makes Pods. The scheduler places them. If I delete a Pod by hand, the loop notices within seconds and makes a new one. That is why the declarative style wins at 2 AM. I do not have to remember the steps to fix the system. I only have to state what good looks like, and the loop does the steps.

Readiness and liveness are the second thing. They look alike and they do opposite jobs. Readiness decides traffic. Liveness decides restarts. A readiness probe should check whether the Pod can serve this minute, including its database connection. A liveness probe should check only whether the process itself is alive, nothing downstream. If the liveness probe depends on the database, a slow database kills healthy Pods, and the killing makes the database slower. I add a startup probe for apps that warm up slowly, so the liveness probe does not fire while the app is still starting. The three probes are three different questions. Answering them with one endpoint is how good Pods die.

Requests, limits, and QoS classes are the third thing. Every container can set a CPU and memory request and a limit. The scheduler uses requests only. The kernel enforces limits: CPU over the limit gets throttled, memory over the limit gets the container killed with an OOMKill. The combination decides the QoS class. Guaranteed means every container sets request equal to limit for CPU and memory. Burstable means at least one container sets a request below its limit, or sets a request with no limit. BestEffort means nothing is set at all. When a node runs out of memory, the kubelet evicts BestEffort Pods first, then Burstable Pods that use more than they requested, and Guaranteed Pods last. The class is not a label. It is an eviction order, and it decides who dies first under pressure.

Pod disruption budgets are the fourth thing. Voluntary disruptions are the ones I cause: a node drain for an upgrade, a cluster autoscaler removing a node. A PodDisruptionBudget says how many Pods of a Deployment must stay up through those operations. I set minAvailable to 1 for a 3 replica service, or minAvailable to 2 when I cannot afford to lose more than one. Without a budget, a drain can take every replica at once, and the Service has an empty Endpoints list while the upgrade runs. The budget does not stop involuntary failures. A node that dies takes its Pods with it, budget or not. It only stops my own maintenance from doing the same damage.

At 10x the scale, the shape holds and the numbers grow. I add replicas and a HorizontalPodAutoscaler on CPU or on a custom metric like queue depth. Image pull time becomes the slow hop, so I keep images small and pre-warm them on the nodes. I split teams into namespaces with a ResourceQuota each, so one team cannot eat the whole cluster. At 100x, one cluster is not enough. I run several, and I push the same manifests to each with GitOps, which Lesson 50 covers. The control plane itself becomes the thing I protect: etcd backups, API server capacity, and admission control that rejects bad manifests before they land.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
  namespace: checkout
spec:
  replicas: 3
  selector:
    matchLabels: { app: checkout-api }
  template:
    metadata:
      labels: { app: checkout-api }
    spec:
      containers:
      - name: api
        image: registry.example.com/checkout-api:1.42.0
        ports:
        - containerPort: 8080
        env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef: { name: checkout-db, key: password }
        resources:
          requests: { cpu: "250m", memory: "512Mi" }
          limits:   { cpu: "500m", memory: "1Gi" }
        startupProbe:
          httpGet: { path: /healthz, port: 8080 }
          failureThreshold: 30
          periodSeconds: 2
        readinessProbe:
          httpGet: { path: /ready, port: 8080 }
          periodSeconds: 5
        livenessProbe:
          httpGet: { path: /healthz, port: 8080 }
          periodSeconds: 10
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-api
  namespace: checkout
spec:
  minAvailable: 2
  selector:
    matchLabels: { app: checkout-api }

What the snippet means: three replicas in the checkout namespace, each Pod requesting 250 millicores and 512 Mi and capped at 500 millicores and 1 Gi, so the QoS class is Burstable. The database password comes from a Secret, never from the YAML. The startup probe gives the app 60 seconds to warm up before liveness starts judging it. Readiness checks the /ready endpoint that knows about the database, liveness checks the cheap /healthz endpoint that only knows about the process. The disruption budget keeps at least 2 of the 3 Pods up through any drain or upgrade.

One time it broke

What broke

A new version of a checkout service shipped with a liveness probe that called the app health endpoint. That endpoint checked the database before answering. One slow morning the database took three seconds to answer. The probe timed out at one second. The kubelet decided the Pod was dead and killed it. The replacement Pod did the same thing. Within minutes every Pod in the Deployment was restarting in a loop, and users saw errors even though the app code was fine.

Timeline

  • 09:12,the first alert fired: Pod restart count climbing on the checkout service.
  • 09:15,I opened the Pod list. Restarts were rising on every replica at once. The same symptom everywhere meant a shared cause, not one bad Pod.
  • 09:20,the logs showed the app starting cleanly each time, then getting killed about thirty seconds later. The app was not crashing. Something was killing it.
  • 09:26,I read the probe settings. The liveness probe called the health endpoint with a one second timeout. The health endpoint waited on the database.
  • 09:31,I checked the database. It was slow but alive. A routine backup job was holding a lock.
  • 09:38,I changed the liveness probe to a cheap endpoint that only checks the process, and moved the database check to readiness only. The restarts stopped.
  • 09:50,traffic was normal. The backup job finished and the database recovered on its own.

What changed after

Four things. First, liveness probes check the process only, never a downstream dependency. Second, readiness probes check the dependencies, because readiness only costs traffic, not the Pod. Third, every app gets a startup probe when it needs more than a few seconds to warm up, so the liveness probe never fires during startup. Fourth, I alert on Pod restart rate, not just Pod count. A Deployment can show three Pods Running while every one of them is dying and restarting every minute. The restart count is the signal that catches that.

If you remember nothing else

Design exercise

Design the Kubernetes deployment for a checkout API that must serve 2,000 requests per second with three nines of availability. You have one cluster, three nodes with 4 vCPU each, and a database outside the cluster. The API is stateless. Each Pod handles 200 requests per second at 250 millicores. Decide the namespace, the replica count, the requests and limits, the probes, the rollout, and the disruption budget.

Clarifying questions to ask

  • Is 2,000 the steady load or the peak, and how fast does it spike?
  • What happens if a whole node dies: how many replicas can we afford to lose?
  • How slow may a deploy be: can we run old and new versions side by side?
  • What does the database allow: how many connections can the Pods open in total?
  • Who else shares the cluster, and what stops their workload from eating our CPU?

Rough sizing

2,000 requests per second at 200 per Pod means 10 Pods at full load. I run 12 so one node can die without dropping below the target. Each Pod requests 250 millicores, so 12 Pods reserve 3000 millicores, which fits on three 4 vCPU nodes with room for the system Pods. I set the limit at 500 millicores so a Pod can burst without starving its neighbours. I put the API in its own namespace with a ResourceQuota, a readiness probe on a cheap endpoint, a liveness probe on the process only, a startup probe with a 60 second window, a rolling update with maxUnavailable 1, and a PodDisruptionBudget with minAvailable 10.

Trade-offs to name

More replicas buy availability and cost money and database connections. A tight limit protects neighbours and throttles the Pod under load. A slow rollout is safe and leaves two versions serving at once. A PodDisruptionBudget protects availability and can block a node upgrade until someone looks. And the database is the real ceiling here. Twelve Pods mean twelve connection pools, so I size the pool per Pod and count the total before I scale the replicas.

Next in this series
Lesson 49 — Infrastructure as Code