Lesson 48 taught you what a pod is. This lesson teaches you what a pod is allowed to do, and who is allowed to ask it to do it. A cluster is a shared machine, and every workload on it is a tenant. The security of a cluster is not one wall. It is a series of gates, and a deploy has to pass every one of them. The API server asks who you are. RBAC asks what you may do. Admission control asks whether the pod you want is even allowed to exist. The image scanner asks whether the image is safe. The Pod Security Standards ask whether the pod runs as root. One kubectl apply deploys the checkout app, and this lesson follows it through every gate, from a first RoleBinding to the deep end: OPA and Kyverno policies, runtime security, and the CIS benchmarks. I learned cluster security the way I learn most things in this job. I watched a deploy fail with a forbidden error, fixed the permission, and then found the next gate waiting behind it.
I start with one gate, authentication, and one clean deploy that passes it. Then I walk the whole gate chain: RBAC, mutating admission, validating admission. The deploy is denied at the last gate, and nothing is stored.
When it breaks, I check:
Every hop charges rent in milliseconds. The bar below is a worked example for one deploy through the security gates, not a measurement from a live system, but the shape is honest: in most deploys I have chased, the gates are cheap and the pod start is the bill. Step through the lesson or switch the scenario and watch the bar change. The bad image and the privileged pod both fail, and both fail fast, without the pod ever starting. Same deploy, same manifest shape. Only the image and the privileges changed, and the gates caught both.
The budget bar above shows three fixed stories. This simulator lets you write your own. It models one deploy through the security gates, from the RBAC check to the pod starting. Move a slider and watch the admission time, the scan time, and the total change in the budget bar, in the hop timings, and in the status line. The numbers are a teaching example, not live measurements. The shape is the lesson: the gates are cheap, the scan grows with the image, and the pod start is the bill.
Each webhook adds 5 ms. Admission time is 3 ms of base plus the webhooks times 5 ms, so 2 webhooks cost 13 ms of admission time.
Each layer adds 1.5 ms to the scan. Scan time is the layers times 1.5 ms, so 12 layers cost 18 ms of scan time.
Each namespace adds 0.5 ms to the RBAC check. RBAC time is the namespaces times 0.5 ms, so 8 namespaces cost 4 ms of RBAC time.
Each secret adds 1 ms. Secret time is the secrets times 1 ms, so 3 secrets cost 3 ms of secret time.
Try this: (1) Leave the defaults: 2 webhooks, 12 layers, 8 namespaces, 3 secrets. The total is about 76 ms. (2) Push the webhooks to 5, the layers to 50, the namespaces to 50, and the secrets to 20. The total is about 186 ms. Same deploy, same gates. Only the knobs changed. (3) Set everything back, then set the layers to 1 and watch the scan fall to 1.5 ms. That is what a small image buys, and it is why the base image matters.
Three questions. No score is saved anywhere — this is just you against the material.
Kubernetes security looks like one wall. It is not. It is four layers stacked on each other, and each layer only makes sense after the one below it. I learned it in this order, and I teach it in this order: know who is asking, then limit what they may do, then limit what the pod may do, then watch the pod while it runs. Skip a layer and the next one reads like magic. Build them in order and nothing in this lesson is magic.
Definition: Kubernetes security at the basic level is authentication and RBAC. Authentication answers who you are: a certificate, a token, or an OIDC login. RBAC answers what you may do: a Role says what actions are allowed, and a RoleBinding ties the Role to a user or a service account in a namespace.
Types:
Uses: when a cluster is new and the question is just who may do what. One Role per team, one binding per namespace, and least privilege everywhere.
Scale: holds up to a handful of teams and namespaces without special care.
Failure modes: a binding that names the wrong subject allows the wrong person, or nobody at all; a human with cluster-admin is a human who can delete the cluster, and that is a permission nobody should hold every day.
Cost: the review; each Role needs an owner, and each binding needs a review, because a permission nobody reviews is a permission nobody understands.
Trade-offs: RBAC is deny by default, which is safe and annoying. Every new action needs a binding, and a team that is blocked by RBAC will ask for cluster-admin, which is the permission that defeats the whole system.
Definition: the intermediate layer is admission control and Pod Security Standards. Admission control is the gate the API server runs after RBAC: mutating webhooks change the request, validating webhooks say yes or no. Pod Security Standards are the built-in rules about what a pod may do, in three levels: privileged, baseline, and restricted. A namespace carries a label that says which level it enforces, and the API server rejects any pod that breaks it. The image scan lives here too: the scanner reads every layer and checks it against the CVE database.
Types:
Uses: when a deploy has to be safe, not just allowed. A pod that runs as a non-root user, with a read-only filesystem, with an image that passed the scan.
Scale: holds to a few dozen webhooks and policies, but the webhooks grow with every team that adds one, and each one is a service the API server has to call.
Failure modes: a mutating webhook that adds a sidecar to a namespace that should not have one; a validating webhook that is down blocks every deploy, because the API server waits for it; a namespace labeled restricted rejects a pod that runs as root, and the error names the exact field.
Cost: the webhook latency and the scan time; each webhook adds milliseconds, and each image layer adds scan time.
Trade-offs: admission control buys a gate and costs a dependency. A webhook that is down is a cluster that cannot deploy, so every webhook gets its own health check and its own failure policy.
Definition: the advanced layer is policy as code and the supply chain. OPA is a policy engine that runs Rego policies. Kyverno is a policy engine that runs policies written as YAML. Both live as validating webhooks, and both can say things the built-in standards cannot: this image must come from our registry, this pod must have a resource limit, this namespace must have a label. The supply chain is the path from the code to the image: the image is built in the pipeline, scanned, signed, and only a signed image may deploy.
Types:
Uses: when the cluster has to prove something, not just check it. An image that must come from the team's own registry. A pod that must have a resource limit.
Scale: the policy count becomes the bottleneck; every policy is a rule the API server has to evaluate, and a hundred policies is a hundred evaluations per deploy.
Failure modes: a policy that is too broad blocks every deploy, and the team learns to hate the policy instead of the risk; a policy exception with no expiry date is a permanent hole; a signed image that was never verified is a signature nobody checked.
Cost: operational; policies need owners, exceptions need expiry dates, and the supply chain needs a pipeline that signs every image.
Trade-offs: OPA buys expressive policies and costs a language to learn. Kyverno buys readable policies and costs the same review discipline. Signing buys provenance and costs a key management story.
The policy with no exception process. A policy that blocks everything with no way to ask for an exception gets worked around, and a worked-around policy is a policy that does nothing. I write the exception process next to the policy: who asks, who approves, and when it expires, because an exception that never expires is a hole.
Do this: apply a policy that blocks images from outside the company registry, then try to deploy an image from outside. Expected: the deploy is rejected, and the error names the policy. Add an exception with an expiry date, deploy again, and expect it to pass. The exception is a decision, and a decision needs a name and a date.
kubectl apply -f - <<'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-company-registry
spec:
validationFailureAction: Enforce
rules:
- name: check-registry
match:
anyKinds:
- Pod
validate:
message: "Images must come from the company registry."
pattern:
spec:
containers:
- image: "registry.internal/*"
EOF
What the snippet means: the policy rejects any pod whose image does not come from the company registry. A policy without an exception process gets bypassed, and a bypassed policy is worse than no policy, because it looks like protection.
Read the flags: kubectl apply -f - with a heredoc
The dash after -f means read the manifest from standard input. The heredoc feeds the manifest straight from the terminal, so I can apply one policy without writing a file first.
kubectl apply: This sends the manifests to the API server. Create makes a new object, apply creates or updates it. I use apply for changes I want to keep.-f -: The -f flag names the file to apply. The dash means standard input, not a file. Without the dash, -f would look for a file named dash and fail.<<'EOF': This is a heredoc. It sends every line until EOF as input. The quotes around EOF stop the shell from expanding variables inside the manifest, so the policy stays exactly as written.--dry-run=client (added): The lesson did not show this one yet. It checks the manifest without sending it. Client means the check happens on my machine. I use it before any apply, so a typo fails on my laptop and not on the cluster.Try it: run kubectl apply -f - with the policy heredoc, then run kubectl get clusterpolicy. Expect the policy to exist. Change the registry name in the heredoc and apply it again. Expect the policy to update, not duplicate.
The image that was signed, and the one that was not. Image signing is the proof of provenance: the image was built in the pipeline, scanned, and signed. An unsigned image is an image nobody vouches for, and a cluster that accepts it is a cluster that trusts the network. I check the signature at deploy time, because a scan that runs after the deploy is a report, not a gate.
Do this: try to deploy an unsigned image into a namespace that requires signatures, and read the rejection. Expected: the deploy is rejected, and the error names the signature. Sign the image, deploy again, and expect it to pass. The signature is the proof, and the rejection is the gate.
cosign verify --key cosign.pub registry.internal/shop-api:v1.2.0
# expect: the signature verifies, and the output names the signer
What the snippet means: the verify command checks the signature against the public key. A signature that verifies is an image the pipeline vouches for. A signature that fails is an image nobody vouches for.
Definition: the master layer is runtime security and the benchmarks. Runtime security watches the pod while it runs: a tool like Falco watches the pod's syscalls and alerts when the pod does something it should not, like open a shell or write to a system file. The CIS benchmarks are the checklist for the cluster itself: the API server flags, the kubelet flags, the etcd permissions, the network settings.
Types:
Uses: when the cluster has to be defended, not just gated. A pod that opens a shell at 2 AM is a pod that has been compromised, and the runtime is the only thing that sees it.
Scale: holds at any size, because the cost is per node and per pod, and it grows with the cluster.
Failure modes: a runtime tool that alerts on everything is a tool nobody reads, and the real alert drowns in the noise; a benchmark that is run once and never again is a score that goes stale the day after it is taken; an audit log that is never read is a record nobody uses.
Cost: expertise and attention; runtime alerts need a human to triage them, benchmarks need a schedule, and audit logs need storage.
Trade-offs: runtime security buys detection and costs noise, because every alert has to be triaged. Benchmarks buy a score and cost the work of fixing what the score finds. Audit logging buys the record and costs the storage to keep it.
Runtime security: the pod that opened a shell. A pod that opens a shell at runtime is a pod doing something it should not do, and runtime security is the tool that notices. I watch the syscalls, and I alert on the ones that never happen in normal operation, because an alert on a normal syscall is noise, and noise gets ignored.
Do this: run a test pod that opens a shell, and watch the runtime alerts. Expected: an alert that names the pod, the syscall, and the time. Run the normal workload, and expect silence. The alert is the signal, and the silence on the normal workload is the proof the rule is tuned.
kubectl exec -it my-pod -- sh
# expect a runtime alert: a shell in a pod that should never have one
# the normal workload runs without a shell, and without an alert
What the snippet means: the shell is the test, and the alert is the detection. A runtime rule that alerts on normal behaviour is a rule that gets turned off, and a rule that is turned off protects nothing.
Read the flags: kubectl exec -it with sh
A shell inside a pod is a debugging tool, and also a red flag. A pod that should never have a shell is a pod the policy should have blocked, and the exec proves it.
kubectl exec -it my-pod: Exec runs a command in the pod my-pod. The -i flag keeps standard input open. The -t flag allocates a terminal. Together they are the -it pair.--: This separates kubectl flags from the command to run inside the pod. Everything after it belongs to the pod.sh: This is the shell inside the pod. It runs with no command, so it waits for me to type. A workload pod should never need a shell, and a shell that opens is a finding, not a tool.-n (added): The lesson did not show this one yet. It picks the namespace, like kubectl exec -n shop -it my-pod -- sh. Without it, kubectl uses the current namespace, which may not hold the pod.Try it: run kubectl exec -it my-pod -- sh and expect a shell prompt. Expect a runtime alert too, because a shell in a pod that should never have one is the alert the policy is there to raise.
The audit log, and the request that changed prod. The audit log is the record of every request the API server handled, and it is the answer to who changed prod, when, and from where. I read it after a change I did not expect, because the log names the user, the resource, and the time, and that is the whole story.
Do this: change a deployment, then search the audit log for the change with the command in the snippet. Expected: an entry that names the user, the resource, and the time. A change with no entry is a change that bypassed the API, and that is a bigger problem than the change.
grep -i "shop-api" /var/log/kubernetes/audit.log | tail -5
# expect entries with the user, the verb, and the resource that changed
What the snippet means: the audit log is the record. A change that appears in the log is a change with an owner. A change that does not appear is a change that went around the API, and that is the thing to fix first.
Read the flags: grep -i on the audit log and tail
The audit log records who asked for what. Grep finds the service name. Tail keeps the last five lines, which are the most recent requests, and each line names the user, the verb, and the resource.
grep -i "shop-api": Grep searches for the text shop-api. The -i flag ignores case. Without it, a capital letter would hide the match./var/log/kubernetes/audit.log: This is the file to search. It is the audit log, and every API request is a line in it.| tail -5: The pipe hands the matches to tail. The -5 flag keeps the last five lines. Without it, grep would print every match in the log, which is pages long.grep -c (added): The lesson did not show this one yet. The -c flag counts matching lines instead of printing them. I use it when the question is how many times a user asked, not what they asked.Try it: run grep -i "shop-api" /var/log/kubernetes/audit.log | tail -5. Expect entries with the user, the verb, and the resource that changed. Run it with grep -c next and expect a count of the requests.
OPA and Kyverno answer the same question two ways. OPA runs Rego, a policy language. Rego is expressive and hard to read, and a Rego policy is a program. Kyverno runs policies written as YAML, which is easier to read and easier to get wrong in the same way, because YAML is familiar and the policy still has to be precise. I pick Kyverno when the team writes policies and the team reads YAML all day. I pick OPA when the policy needs logic that YAML cannot express, like a rule that compares two fields across two objects. The wrong move is picking the engine and skipping the review. A policy is code, and code that is never reviewed is code that breaks in production.
The gates decide whether a pod may start. The runtime decides whether it may keep running. A runtime tool watches the pod's syscalls: what files it opens, what processes it starts, what network connections it makes. A pod that opens a shell at 2 AM is a pod that has been compromised, and the runtime is the only thing that sees it, because the app logs will not show a shell. I tune the runtime rules the same way I tune alerts: start with the defaults, watch the noise for a week, and silence the rules that fire on normal behavior. A runtime tool that alerts on everything is a tool nobody reads, and the real alert drowns in the noise.
The CIS benchmarks are a checklist for the cluster itself. They cover the API server flags, the kubelet flags, the etcd permissions, and the network settings. Each check has a score, and the cluster gets a total. I run the benchmark on every new cluster, and I fix the failures before the cluster takes traffic, because a misconfigured API server is a door that is open from day one. The benchmark is not a security system. It is a measurement. A cluster that scores well and has no runtime security is a cluster with a good lock and no alarm.
A secret belongs in a Secret object, mounted into the pod as a file. It never belongs in the image, and it never belongs in the manifest. The next step is an external secrets operator: the secret lives in a vault, and the operator syncs it into the cluster. The vault is the source of truth, and the cluster holds a copy. The secret is rotated in the vault, and the operator brings the new value in. A secret mounted as a file is rotated when the Secret changes, and the pod sees the new value without a restart. A secret in an environment variable is not, and the pod has to restart. I mount secrets as files, and I rotate them on a schedule, because a secret that is never rotated is a secret that is stolen once and used forever.
At 10x the namespaces, the problem is the Roles. Every team wants a Role, every namespace wants a binding, and the permissions become a document nobody has read in full. I move the Roles into Git, review them like code, and audit them on a schedule, because a permission nobody reviews is a permission nobody understands. At 100x, the design changes. The policy moves from per-namespace rules to team-owned defaults with platform guardrails. The supply chain gets a signature on every image, and the cluster only runs signed images. And the audit log gets a pipeline, because a thousand requests a second is a log nobody can read by hand, and the log is the only record of who did what.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: deployer
namespace: checkout
rules:
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["create", "update", "patch", "get", "list"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: deployer-binding
namespace: checkout
subjects:
- kind: User
name: mohan
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: deployer
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: Namespace
metadata:
name: checkout
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-team-registry
spec:
validationFailureAction: Enforce
background: true
rules:
- name: require-team-registry
match:
any:
- resources:
kinds: ["Pod"]
validate:
message: "Images must come from the team registry."
pattern:
spec:
containers:
- image: "registry.example.com/*"
Four objects, one security story. The Role says what the deployer may do: create and update Deployments in the checkout namespace, and read pods, nothing more. The RoleBinding ties that Role to one user, in one namespace. The namespace label enforces restricted Pod Security, so any pod that runs as root is rejected by the API server itself, with no webhook needed. The Kyverno policy says every pod's image must come from the team's own registry, and the action is Enforce, so a pod with an image from anywhere else is denied. Read them together and you can see the whole story: who may deploy, what the pod may do, and where the image must come from. Change one of them without reading the other three and you have changed the security in a way you did not intend.
The CIS benchmark is the checklist for the cluster itself: the API server flags, the kubelet flags, the etcd permissions, the network settings. I run it against my own cluster, and I fix the failures one at a time, because a checklist that nobody runs is a checklist that protects nothing.
Do this: run the benchmark tool against a test cluster, and read the failures. Expected: a list, with the worst ones first. Fix the first failure, run again, and expect the list to shrink by one. A benchmark that runs on a schedule is a cluster that gets safer over time.
kube-bench run --targets master,node 2>&1 | grep -E "FAIL|WARN" | head -20
What the snippet means: the benchmark checks the cluster against the checklist and reports the failures. A failure is a setting to fix, and the list is the work, in order.
This one stayed with me because the deploy failed, and that was the good news. The bad news was that the same image had been running in production for a month, and the scan that caught it had never run on the running pods.
A team deployed a checkout service with an image built on an old base. The base had a critical CVE that was published after the image was built, and the deploy-time scan had never run on the image, because the scan gate was added after the image was already deployed. The pod ran fine for a month. Then a new deploy of the same image failed the scan, and the team learned the CVE existed. The running pod was still on the old image, and the CVE was still in it. The on-call team checked the running pods, found the same image on all of them, and realized the scan gate only protects new deploys. A running pod is never rescanned unless someone asks for it.
Three things. First, every running image is rescanned weekly, because a deploy-time scan is a snapshot in time, and a CVE published after the deploy is a vulnerability the scan never saw. Second, the base image is rebuilt on a schedule, not just when someone remembers, because a base that is never rebuilt is a base that accumulates CVEs. Third, the runtime alerts are tuned and triaged, because the runtime is the guard while the patch is built. The scan gate did nothing wrong in this incident. The lesson is that a gate that only runs at deploy time protects the next deploy, not the pods that are already running.
Design the security for a cluster with 50 services across 10 namespaces. Each team deploys its own services. Images must come from the team's own registry, pods must run as non-root, and secrets must never be in an image. A pod that opens a shell at 2 AM must be caught. The latency budget is not the question here. The question is the gates: sketch the RBAC, the admission policies, the image supply chain, and the runtime monitoring, and say what you would measure.
Start with the budget from this lesson. One deploy takes about 67 ms of gate time: authn 4, RBAC 3, mutating 5, validating 6, Pod Security 4, scan 12, secret 3, pod start 30. Fifty services deploying once a day is 50 deploys a day, and the gate time is trivial next to the pod starts. Little's Law for the sizing: at 5 deploys an hour in the busy window, each spending 0.067 seconds in the gates, the number inside the system at once is 5 times 0.067, which is less than 1. The gates are never the bottleneck. The bottleneck is the scan, and the scan is the image size. I size the webhook fleet for the deploy rate, not for the gate time: at 2 webhooks per deploy and 5 deploys an hour, the webhooks see 10 calls an hour, and a webhook that is down blocks every deploy, so each webhook gets its own health check and its own failure policy. The number I watch is the policy count: 50 services times 3 policies each is 150 policies, and each one is a rule the API server has to evaluate on every deploy, so I audit the policies on a schedule and delete the ones nobody needs.
RBAC buys least privilege and costs a binding for every new action. Admission control buys a gate and costs a dependency, and a webhook that is down is a cluster that cannot deploy. OPA buys expressive policies and costs a language to learn. Kyverno buys readable policies and costs the same review discipline. Signing buys provenance and costs a key management story. Runtime security buys detection and costs noise, because every alert has to be triaged. And the target is the gates, not the speed. A deploy that is fast and unguarded fails this design, and the gates are where the security lives.
Saved in this browser, so the Lessons index can show your progress across the course.