You write a YAML file. You run one command. Then a dozen pieces of machinery argue about where your code should run, and a minute later it is serving traffic. Kubernetes is that machinery. This lesson walks a deploy from the command line to a live Pod, hop by hop, the way I trace it when a deploy stalls.
Difficulty: Advanced · Know this first: Lesson 47, Lesson 26
Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: in most slow deploys I have chased, one or two hops eat almost the whole budget. Step through the lesson or switch the scenario and watch the bar change.
The budget bar above shows three fixed stories. This simulator lets you size a real Deployment. Pick how many replicas you want and how much CPU each Pod requests. The node has 4 vCPU to give out. Watch which Pods fit, which wait in Pending, what QoS class you get, and what the node bill looks like. The numbers are a teaching example, not a live quote. The shape is the lesson: requests are reservations, and reservations run out.
How many copies of the Pod the Deployment asks for. From 1 to 10.
The CPU each Pod reserves. The scheduler adds up every request on a node and refuses to go past 4000 millicores. Total requested equals replicas times request. A request of 0 means the Pod asks for nothing.
The ceiling each Pod may burst to. A limit of 0 means no ceiling. If the limit is below the request (and the limit is not 0), the API server rejects the Pod: the limit must be greater than or equal to the request. Memory is fixed in this example at 512 Mi requested and 512 Mi limited, so the CPU numbers decide the QoS class: request and limit both 0 is BestEffort, request equal to limit is Guaranteed, anything else is Burstable.
Try this: (1) Leave the defaults: 3 replicas at 250 millicores. Total requested is 750 millicores, all 3 Pods fit on one node, the bill is $146 a month, QoS is Burstable, and the deploy total is about 210 ms. (2) Set replicas to 10 and request to 500. Total requested is 5000 millicores, only 8 Pods fit, 2 wait in Pending, the bill is $292 a month for 2 nodes, the scheduler hop is 135 ms, and the deploy total is about 330 ms. (3) Set request to 500 and limit to 500 for Guaranteed, then set both to 0 and watch it drop to BestEffort.
Extra diagram. I read the picture first: the boxes are the system, the moving dot is a deploy, and each step adds one piece until the full picture is there. No text needed to follow it.
Extra diagram. I read the picture first: the boxes are the system, the moving dot is a deploy that stalls, and each step adds one piece until the full picture is there. No text needed to follow it.
Three questions. No score is saved anywhere — this is just you against the material.
The lesson above walks a deploy from the command line to a live Pod. This section covers the same machinery four times, from the first YAML file to the level where I debug a cluster at 2 AM.
Definition: You write a YAML file. You run one command. Then a dozen pieces of machinery argue about where your code should run, and a minute later it is serving traffic. Kubernetes is that machinery. The file says what good looks like. The cluster does the steps.
Types:
Uses: I use this level every time I deploy. I write the file, I apply it, and I check the Pods. If a Pod is not running, I read its events before I read its logs. The events say what the cluster tried to do.
Dimensions:
Definition: Nothing in Kubernetes runs once and stops. Every controller is a loop: read the desired state from etcd, read the actual state from the cluster, and close the gap. The Deployment controller makes a ReplicaSet. The ReplicaSet controller makes Pods. The scheduler places them. If I delete a Pod by hand, the loop notices within seconds and makes a new one.
Types:
Uses: I use this level when a deploy stalls. I trace the loop: did the Deployment make a ReplicaSet, did the ReplicaSet make Pods, did the scheduler place them, did the Pod start. The step that did not happen is the problem.
Dimensions:
Definition: Readiness and liveness look alike and they do opposite jobs. Readiness decides traffic. Liveness decides restarts. A readiness probe should check whether the Pod can serve this minute, including its database connection. A liveness probe should check only whether the process itself is alive, nothing downstream.
Types:
Uses: I use this level when Pods die for no good reason. The first thing I check is the probes. If the liveness probe depends on the database, a slow database kills healthy Pods, and the killing makes the database slower. I have seen that loop take down a whole Deployment in minutes.
Dimensions:
Definition: At this level the cluster is a system I operate, not a tool I use. I know what normal looks like: the control plane latency, the etcd size, the scheduler queue, the node pressure. When a deploy stalls, I do not guess. I read the events, the conditions, and the controller logs, in that order.
Types:
Uses: I use this level on the night the cluster itself is the incident. A node runs out of disk, the Pods evict, the scheduler has nowhere to put them, and the app team pages me. The cluster is the service then, and I treat it like one.
Dimensions:
Controllers and reconciliation are the first thing to get straight. Nothing in Kubernetes runs once and stops. Every controller is a loop: read the desired state from etcd, read the actual state from the cluster, and close the gap. The Deployment controller makes a ReplicaSet. The ReplicaSet controller makes Pods. The scheduler places them. If I delete a Pod by hand, the loop notices within seconds and makes a new one. That is why the declarative style wins at 2 AM. I do not have to remember the steps to fix the system. I only have to state what good looks like, and the loop does the steps.
Readiness and liveness are the second thing. They look alike and they do opposite jobs. Readiness decides traffic. Liveness decides restarts. A readiness probe should check whether the Pod can serve this minute, including its database connection. A liveness probe should check only whether the process itself is alive, nothing downstream. If the liveness probe depends on the database, a slow database kills healthy Pods, and the killing makes the database slower. I add a startup probe for apps that warm up slowly, so the liveness probe does not fire while the app is still starting. The three probes are three different questions. Answering them with one endpoint is how good Pods die.
Requests, limits, and QoS classes are the third thing. Every container can set a CPU and memory request and a limit. The scheduler uses requests only. The kernel enforces limits: CPU over the limit gets throttled, memory over the limit gets the container killed with an OOMKill. The combination decides the QoS class. Guaranteed means every container sets request equal to limit for CPU and memory. Burstable means at least one container sets a request below its limit, or sets a request with no limit. BestEffort means nothing is set at all. When a node runs out of memory, the kubelet evicts BestEffort Pods first, then Burstable Pods that use more than they requested, and Guaranteed Pods last. The class is not a label. It is an eviction order, and it decides who dies first under pressure.
Pod disruption budgets are the fourth thing. Voluntary disruptions are the ones I cause: a node drain for an upgrade, a cluster autoscaler removing a node. A PodDisruptionBudget says how many Pods of a Deployment must stay up through those operations. I set minAvailable to 1 for a 3 replica service, or minAvailable to 2 when I cannot afford to lose more than one. Without a budget, a drain can take every replica at once, and the Service has an empty Endpoints list while the upgrade runs. The budget does not stop involuntary failures. A node that dies takes its Pods with it, budget or not. It only stops my own maintenance from doing the same damage.
At 10x the scale, the shape holds and the numbers grow. I add replicas and a HorizontalPodAutoscaler on CPU or on a custom metric like queue depth. Image pull time becomes the slow hop, so I keep images small and pre-warm them on the nodes. I split teams into namespaces with a ResourceQuota each, so one team cannot eat the whole cluster. At 100x, one cluster is not enough. I run several, and I push the same manifests to each with GitOps, which Lesson 50 covers. The control plane itself becomes the thing I protect: etcd backups, API server capacity, and admission control that rejects bad manifests before they land.
apiVersion: apps/v1
kind: Deployment
metadata:
name: checkout-api
namespace: checkout
spec:
replicas: 3
selector:
matchLabels: { app: checkout-api }
template:
metadata:
labels: { app: checkout-api }
spec:
containers:
- name: api
image: registry.example.com/checkout-api:1.42.0
ports:
- containerPort: 8080
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef: { name: checkout-db, key: password }
resources:
requests: { cpu: "250m", memory: "512Mi" }
limits: { cpu: "500m", memory: "1Gi" }
startupProbe:
httpGet: { path: /healthz, port: 8080 }
failureThreshold: 30
periodSeconds: 2
readinessProbe:
httpGet: { path: /ready, port: 8080 }
periodSeconds: 5
livenessProbe:
httpGet: { path: /healthz, port: 8080 }
periodSeconds: 10
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: checkout-api
namespace: checkout
spec:
minAvailable: 2
selector:
matchLabels: { app: checkout-api }
What the snippet means: three replicas in the checkout namespace, each Pod requesting 250 millicores and 512 Mi and capped at 500 millicores and 1 Gi, so the QoS class is Burstable. The database password comes from a Secret, never from the YAML. The startup probe gives the app 60 seconds to warm up before liveness starts judging it. Readiness checks the /ready endpoint that knows about the database, liveness checks the cheap /healthz endpoint that only knows about the process. The disruption budget keeps at least 2 of the 3 Pods up through any drain or upgrade.
A new version of a checkout service shipped with a liveness probe that called the app health endpoint. That endpoint checked the database before answering. One slow morning the database took three seconds to answer. The probe timed out at one second. The kubelet decided the Pod was dead and killed it. The replacement Pod did the same thing. Within minutes every Pod in the Deployment was restarting in a loop, and users saw errors even though the app code was fine.
Four things. First, liveness probes check the process only, never a downstream dependency. Second, readiness probes check the dependencies, because readiness only costs traffic, not the Pod. Third, every app gets a startup probe when it needs more than a few seconds to warm up, so the liveness probe never fires during startup. Fourth, I alert on Pod restart rate, not just Pod count. A Deployment can show three Pods Running while every one of them is dying and restarting every minute. The restart count is the signal that catches that.
Design the Kubernetes deployment for a checkout API that must serve 2,000 requests per second with three nines of availability. You have one cluster, three nodes with 4 vCPU each, and a database outside the cluster. The API is stateless. Each Pod handles 200 requests per second at 250 millicores. Decide the namespace, the replica count, the requests and limits, the probes, the rollout, and the disruption budget.
2,000 requests per second at 200 per Pod means 10 Pods at full load. I run 12 so one node can die without dropping below the target. Each Pod requests 250 millicores, so 12 Pods reserve 3000 millicores, which fits on three 4 vCPU nodes with room for the system Pods. I set the limit at 500 millicores so a Pod can burst without starving its neighbours. I put the API in its own namespace with a ResourceQuota, a readiness probe on a cheap endpoint, a liveness probe on the process only, a startup probe with a 60 second window, a rolling update with maxUnavailable 1, and a PodDisruptionBudget with minAvailable 10.
More replicas buy availability and cost money and database connections. A tight limit protects neighbours and throttles the Pod under load. A slow rollout is safe and leaves two versions serving at once. A PodDisruptionBudget protects availability and can block a node upgrade until someone looks. And the database is the real ceiling here. Twelve Pods mean twelve connection pools, so I size the pool per Pod and count the total before I scale the replicas.