Platform Engineering · Lesson 159

Kubernetes Pod Security in Depth

Lesson 158 taught you the gates a deploy passes. This lesson teaches you the locks on the pod itself: what the pod is allowed to do once it starts. The pod asks for a user, a filesystem, a set of capabilities, and a profile. Three depths enforce the answer. The manifest says what the pod wants. The admission gates say whether it may start. The runtime watches what it does after it starts. A pod that passes the first two depths and misbehaves in the third is the pod this lesson is about. I learned pod security the way I learn most things in this job. I watched a deploy get denied, fixed the field, and then found the next gate waiting behind it.

Difficulty: Advanced · Know this first: Lesson 158, Lesson 157, Lesson 95, Lesson 155

What you will build

By the end of this lesson you will have run a hardened pod on your own cluster and read what its fields actually do. You will have watched a privileged pod get denied by a restricted namespace, read the error naming the exact field, and migrated the namespace from audit to enforce with your own hands. You will have run a linter over both manifests and read one finding per bad field. You will know when a gate should block a deploy, when it should only warn, and when only the runtime can catch the problem.

What you need: a cluster you can run pods on, with kubectl. The lab steps run on your own cluster, and every expected output is described as what to look for, not quoted as a transcript. I did not run the lab on a cluster here. This VM hands out no cpu controller in its cgroup, so no k3s server can start on it. The same wall stopped the Phase 9 lab work, and I did not re-attempt the install. No kubectl output from this machine appears in this lesson, because there is no cluster on this machine to produce it. Your numbers will differ. The shape will not.

The story: three depths, one pod

A pod is a set of asks. It asks to run as a user. It asks to write to its filesystem. It asks to keep a capability or two. It asks to escalate its privileges if a setuid binary offers the chance. The securityContext field is where those asks live, and it lives at two levels: the pod level, which sets the defaults for every container in the pod, and the container level, which overrides the pod level for one container. A container that says nothing inherits the pod. A container that speaks wins over the pod for itself only.

The fields, one table

Pod levelContainer levelWhat it does
runAsUseryesyesThe UID the processes run as. 0 is root. 1000 is the usual non-root user.
runAsGroupyesyesThe GID the processes run as.
runAsNonRootyesyesRefuses to start if the image user is UID 0. The pod never runs, it fails at start.
readOnlyRootFilesystemnoyesThe root filesystem is mounted read-only. Writes to / fail.
allowPrivilegeEscalationnoyesBlocks gaining more privilege than the parent, including through setuid binaries.
capabilities drop/addnoyesDrops Linux capabilities from the default set, or adds back only named ones.
seccompProfileyesyesThe syscall filter. RuntimeDefault is the runtime's own filter.
seLinuxOptionsyesyesThe SELinux label. Named here only: this is where the profile plugs in.

Lesson 158 taught the three Pod Security Standards levels. Baseline blocks the known privilege escalations: privileged containers, host namespaces, hostPath volumes, adding NET_RAW or capabilities beyond the default set. Restricted adds the hardening: non-root user, drop ALL capabilities (only NET_BIND_SERVICE may come back), seccomp RuntimeDefault, no privilege escalation, read-only root filesystem. This lesson teaches the fields behind those levels, so the levels stop being names and start being settings you can write.

The one capability a fresher meets first

NET_RAW. It is the capability behind raw sockets, and ping is the tool that needs it. A pod that drops ALL capabilities and then tries to ping gets a socket error, not a network error. That is the first time most people meet a capability: not in a security review, in a ping that fails. Restricted forbids adding NET_RAW back. Baseline forbids adding it too. The fix is not the capability. The fix is a different check: use a TCP connect or an HTTP probe instead of ping, because the network test you want does not need a raw socket.

Do this now, before the diagram

On your own cluster, run kubectl get namespaces --show-labels and read the pod-security labels. Write down which namespaces carry pod-security.kubernetes.io/enforce and which level each one enforces. A namespace with no label enforces nothing, and a pod in it may run as root. That list is the enforcement map, measured before you learn it.

Run it as:

When it breaks, I check

One thing to remember

Four paths, one family

The diagram above follows one pod from the manifest to the runtime. These four paths are the same idea drawn four ways. One clean deploy. One privileged pod, stopped at PSA. One shell at 2 AM. One numbers trail, hop by hop. Run each path. One run takes 7.2 seconds across the path, so you can watch each hop light up in turn.

The return rule: the green dot only starts moving after the orange dot reaches the data hop. A return with no finished work is a lie. The diagram never draws one. The failure replay at the end of Run all four shows the other side of the rule. Run it and watch the orange dot stop at PSA and never come back.

Path 1: the clean deploy

Privileged pod: skipped

One manifest, one pod. The linters find nothing, PSA passes it, Gatekeeper passes it, and the pod runs in 30 ms. The runtime watches and stays quiet. The green dot starts back from the Pod runs hop, because that is where the finished work lives.

Manifest written0 ms
Linter passes6 ms
PSA passes4 ms
Gatekeeper passes8 ms
Pod runs30 ms
Runtime quiet2 ms

Not run yet.

Path 2: the privileged pod, stopped at PSA

Gatekeeper: skipped

Same deploy, a pod that wants root. The linters flag every field, PSA denies the pod in a restricted namespace, and the pod never starts. No work is done, so no green dot ever starts back.

Manifest written0 ms
Linter warns6 ms
PSA denies15 ms
Never startsdenied

Not run yet.

Path 3: the shell at 2 AM

Nothing skipped

The manifest is clean, so every gate passes it. The pod runs, then opens a shell at 2 AM. Falco fires on the execve, and KubeArmor blocks the exec. The green dot starts back from the Falco alerts hop, because the alert is the finished work here: the detection the gates could not do.

Manifest written0 ms
Linter passes6 ms
PSA passes4 ms
Gatekeeper passes8 ms
Pod runs30 ms
Falco alerts40 ms
KubeArmor blocks5 ms

Not run yet.

Path 4: the numbers trail

Privileged pod: skipped

The same deploy, measured instead of timed. Seven securityContext fields set, four linters run, two gates passed, zero runtime alerts. The trail adds up to 65 ms, and the fields are the check that keeps the pod honest.

Fields set7
Linters run4
Gates passed2
Runtime alerts0
Verdict65 ms

Not run yet.

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one deploy through the pod security gates, not a measurement from a live system, but the shape is honest. The pod start is the big hop at 30 ms, and the total is 65 ms. Switch the scenario and watch the bar change. In the privileged story the PSA hop grows to 15 ms and everything after it disappears, because the pod never starts. In the runtime story the Falco hop grows to 40 ms, because the detection and the alert cost time the quiet deploy never pays. Move a simulator slider and watch the gates, the verdict, and the total move with it.

The fields and the PSA mode: move the sliders and watch the pod bend

The budget bar above shows three fixed stories. This simulator lets you write your own. It answers the sizing question at the heart of this lesson: which field gets the pod denied, which mode lets it through with a warning, and what the runtime catches when the gates let it through. You set the user, the capabilities, the escalation flag, the filesystem, and the PSA mode. Move a slider and watch the risk score, the gate verdicts, and the budget total change. The numbers are a teaching example, built from the formulas below, not live measurements. The shape is the lesson: the fields decide the verdict, the mode decides whether the verdict blocks, and the runtime catches what the gates let through.

uid 1000 (non-root)

The UID the processes run as. 0 is root. 1000 is the usual non-root user. Restricted demands non-root.

0 added

How many Linux capabilities the pod adds back after dropping ALL. Restricted allows none back except NET_BIND_SERVICE. Each one adds 10 to the risk score.

false (blocked)

Whether the container may gain more privilege than its parent, including through a setuid binary. Restricted demands false.

true (read-only)

Whether the root filesystem is mounted read-only. Restricted demands true. False means the pod can write anywhere in its own filesystem.

enforce

What the namespace label says to do with a violating pod. Enforce rejects it. Audit lets it start and logs the violation. Warn lets it start and prints a warning.

Try this: (1) At the defaults, uid 1000, 0 capabilities, escalation blocked, read-only filesystem, PSA enforce, the risk is 0, every gate passes, and the budget total is 65 ms. (2) Add 1 capability, NET_RAW. The risk is 10, the pod violates restricted, and PSA enforce denies it in 21 ms. The status line says denied at PSA. (3) Leave the capability added and set PSA to audit. The pod starts, Gatekeeper passes it because it does not run as root, and the runtime catches it: Falco fires, KubeArmor blocks, and the budget total is 108 ms. (4) Back at the defaults, set runAsUser to root and PSA to audit. The risk is 40, PSA audits and lets it through, and Gatekeeper denies it in 30 ms. The status line says denied at Gatekeeper. (5) Set runAsUser back to uid 1000, add 2 capabilities, allow escalation, make the filesystem writable, and set PSA to warn. The risk is 60, PSA warns and lets it through, Gatekeeper passes it, and the runtime catches it in 108 ms.

Formulas, in plain terms. Risk is root 40 plus capabilities times 10 plus escalation allowed 25 plus writable filesystem 15. The most a pod can score is 130. A pod violates restricted when it runs as root, allows escalation, adds any capability, or has a writable filesystem. PSA enforce denies a violating pod in 21 ms: 6 ms of linting plus 15 ms of PSA. PSA audit or warn lets a violating pod through, and Gatekeeper denies it when it runs as root, in 30 ms: 6 ms of linting, 4 ms of PSA, and 20 ms of Gatekeeper. A pod that reaches the runtime with escalation allowed or any capability added triggers the runtime catch in 108 ms: the clean 65 ms plus 40 ms of Falco detection plus 5 ms of KubeArmor blocking, minus the 2 ms of quiet watching. A clean pod costs 65 ms. Root never reaches the runtime in this simulator, because a gate always catches it first.

Lab: run a hardened pod, get a privileged pod denied, lint both

Three steps, all run on your own cluster. There is no kubectl output from this machine in this lesson, because there is no cluster on this machine. The k3s install was not re-attempted here: this VM hands out no cpu controller in its cgroup, the same wall Phase 9 hit, so no k3s server can start here. What follows is what to run and what to look for, not a transcript. Your output will differ. The shape will not.

Step 1: the hardened pod

Save this as namespace-psa.yaml. It creates the shop namespace with restricted enforced, audited, and warned. Then save the pod as pod-hardened.yaml. It is the clean deploy from the diagram: pod-level runAsUser 1000, runAsGroup 3000, runAsNonRoot true, seccompProfile RuntimeDefault; container-level runAsUser 1000, runAsNonRoot true, readOnlyRootFilesystem true, allowPrivilegeEscalation false, capabilities drop ALL. Both files were checked against the Kubernetes 1.34 schemas with kubeconform, strict mode, on Nov 21, 2026. They parse and validate. They have not been run against a live API server from this machine.

apiVersion: v1
kind: Namespace
metadata:
  name: shop
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted
apiVersion: v1
kind: Pod
metadata:
  name: shop-api
  namespace: shop
  labels:
    app: shop-api
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: api
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      securityContext:
        runAsUser: 1000
        runAsNonRoot: true
        readOnlyRootFilesystem: true
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
      resources:
        requests:
          cpu: 100m
          memory: 128Mi
        limits:
          cpu: 250m
          memory: 256Mi

Apply both and check the pod:

kubectl apply -f namespace-psa.yaml
kubectl apply -f pod-hardened.yaml
kubectl get pods -n shop
kubectl exec -n shop shop-api -- id
kubectl exec -n shop shop-api -- touch /etc/test

What to look for: the Pod runs, id shows uid 1000, and the touch fails with a read-only filesystem error. That failure is the field working, not the pod broken. Read the pod with kubectl get pod -n shop shop-api -o yaml and find the securityContext you wrote, now part of the running pod.

Step 2: the privileged pod, denied, then warned

Save this as pod-privileged.yaml. It is the pod that wants root: privileged true, runAsUser 0, allowPrivilegeEscalation true, capabilities NET_RAW and SYS_ADMIN. Same kubeconform check, same date, same caveat: validated, not run here.

apiVersion: v1
kind: Pod
metadata:
  name: shop-api-privileged
  namespace: shop
  labels:
    app: shop-api
spec:
  containers:
    - name: api
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      securityContext:
        privileged: true
        runAsUser: 0
        allowPrivilegeEscalation: true
        capabilities:
          add: ["NET_RAW", "SYS_ADMIN"]

Apply it, read the error, then move the namespace to audit and apply it again:

kubectl apply -f pod-privileged.yaml
kubectl label namespace shop pod-security.kubernetes.io/enforce=audit --overwrite
kubectl apply -f pod-privileged.yaml
kubectl get pods -n shop

What to look for: the first apply is rejected, and the error names the field that broke restricted. After the label change to audit, the same apply succeeds with a warning, and the pod runs. That is the migration in miniature: audit first, read the violations, then enforce. Put the label back and clean up:

kubectl label namespace shop pod-security.kubernetes.io/enforce=restricted --overwrite
kubectl delete pod -n shop shop-api-privileged
kubectl delete pod -n shop shop-api
kubectl delete namespace shop

What to look for: the namespace ends restricted again, the privileged pod is gone, and the shop namespace is deleted. A tidy cluster is a choice you make after every experiment, not a state the cluster keeps for you.

Step 3: lint both manifests

Install kube-linter on your machine. One binary, no cluster needed. Then run it over both files from step 1 and step 2:

kube-linter lint pod-hardened.yaml pod-privileged.yaml

What to look for: the hardened pod passes clean, and the privileged pod fails with one finding per bad field. Read the run-as-non-root finding first: it names the container and the check. The linter is the advice. The gate in step 2 is the law. Fix the privileged pod by hand until the linter passes it, then run the linter again and expect silence. Do not silence the check in the config. The finding is the lesson.

Check yourself

Three questions. No score is saved anywhere. This is just you against the material.

Basic to master

A pod, its gates, and its runtime are not three skills. They are four levels of the same job, and every team I have worked with sits on one of them. The levels are not about how many field names you can recite. They are about whether the settings change what gets built. Read them in order. Each level fixes the failure that hurt the most at the level below it.

Basic · The securityContext field, one field at a time

At the basic level you write the field by hand, one field at a time, and you read what each one does. The pod level sets the defaults for every container. The container level overrides them for one container. The full runnable pod that puts all of these together is in the lab. These are the pieces, each one on its own.

runAsUser, runAsGroup, runAsNonRoot, at pod level. The user and the group the processes run as, and the refusal to run as root at all. Put these at pod level and every container inherits them.

spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    runAsNonRoot: true

What the snippet means: the processes run as uid 1000 and gid 3000, and if the image user is root the pod fails to start. The refusal happens at start, not at deploy. A pod that cannot start as non-root is a pod that never runs, and that is the point.

readOnlyRootFilesystem, at container level. The root filesystem is mounted read-only. A container that tries to write to /etc gets an error, not a write.

containers:
  - name: api
    securityContext:
      readOnlyRootFilesystem: true

What the snippet means: the container can read its filesystem and cannot write to it. A log file, a cache directory, and a pid file all need a writable path, and each one gets an emptyDir volume of its own. The read-only root is the rule. The emptyDir is the exception, named and scoped.

allowPrivilegeEscalation, at container level. The container cannot gain more privilege than its parent, including through a setuid binary.

containers:
  - name: api
    securityContext:
      allowPrivilegeEscalation: false

What the snippet means: a setuid helper inside the container cannot lift the process to root. The field exists for exactly this case, and it is container-level only. There is no pod-level version of it.

capabilities, at container level. Drop the whole default set, then add back only what the container uses. NET_BIND_SERVICE is the one that comes back most often, for binding below port 1024.

containers:
  - name: api
    securityContext:
      capabilities:
        drop: ["ALL"]
        add: ["NET_BIND_SERVICE"]

What the snippet means: the container keeps one capability and loses the rest. A container that drops ALL and adds nothing back is the strong form, and most web servers need nothing at all. The usual fix for a port below 1024 is to listen above 1024 and let the Service forward the port, not to add the capability back.

seccompProfile and seLinuxOptions, named only. The profiles plug in here. The seccomp profile is the syscall filter, and RuntimeDefault is the runtime's own filter. The SELinux options are the label. Both are taught in the Master tier. For now, know where they live: the pod-level securityContext.

spec:
  securityContext:
    seccompProfile:
      type: RuntimeDefault

What the snippet means: the pod runs under the runtime's syscall filter. The Master tier teaches what the filter blocks and what a profile of your own looks like.

Do this: run step 1 of the lab on your own cluster and write down what id prints and what the failed touch prints. Then change one field at a time in the pod, apply it, and watch what breaks. Change runAsUser to 0 and watch the pod fail to start. Change readOnlyRootFilesystem to false and watch the touch succeed. One field, one change, one observation.

Try it: Basic

I set allowPrivilegeEscalation at pod level and the pod still accepts it. Why?

Now do it: I parsed the lab pod locally with Python and it checks out: runAsUser 1000, drop ALL, readOnlyRootFilesystem true. On your cluster, run step 1 of the lab and write down what id prints and what the failed touch prints.

Intermediate · Pod Security Admission, the mechanism

At the intermediate level you enforce the fields with labels, not with a webhook. Lesson 158 taught the three levels: privileged, baseline, restricted. This lesson teaches the mechanism that enforces them. Pod Security Admission is a built-in admission plugin. It runs inside the API server, and it reads one thing: the labels on the namespace.

apiVersion: v1
kind: Namespace
metadata:
  name: shop
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

What the snippet means: one namespace, three labels, one level each. The enforce label rejects a pod that breaks restricted. The audit label writes the violation to the audit log and lets the pod start. The warn label prints a warning to the user who applied it and lets the pod start. The three modes can carry three different levels: enforce baseline while you audit and warn on restricted, so the teams see what restricted would block before it blocks them.

The migration. A namespace moves from audit to enforce in three steps, and the order matters. First, label it audit: restricted and warn: restricted, and read the audit log for a week. Second, fix the violations the log shows, one pod at a time. Third, flip the label to enforce: restricted, and watch the deploys. A namespace that flips to enforce with violations still in it blocks deploys on the day it flips, and the teams learn to hate the gate instead of the risk.

PSA against a webhook. PSA differs from a webhook in what it costs to run. A webhook is a service the API server calls, and a service can be down, slow, or misconfigured. PSA is built into the API server. There is no webhook to run, no webhook to fail, and no webhook to be down on a deploy day. The price is expressiveness: PSA can only enforce the three levels, and anything finer needs a policy engine. That is the next tier.

Do this: run step 2 of the lab, both halves: once denied at enforce, once warned at audit. Write down the exact field the error names, and the exact warning the audit apply prints.

Try it: Intermediate

I want to move a running namespace to enforce restricted. What is the safe first step?

Now do it: On your cluster, run step 2 of the lab, both halves: once denied at enforce, once warned at audit. Write down the exact field the error names and the exact warning the audit apply prints. I parsed the three PSA labels locally with Python and they check out.

Advanced · The scanner and linter bench, plus Gatekeeper

At the advanced level you run the bench as a pipeline. Each tool checks the same manifest and reports a different thing, and each one runs at a different stage. kube-bench is the one Lesson 158 already taught: it runs the CIS benchmark against a live cluster, kube-bench run --targets master,node, and reports the FAIL and WARN lines. One paragraph is all it gets here, because Lesson 158 owns it. The other four are new.

What it checksWhich stageHow it reports
kube-benchThe cluster against the CIS benchmark listAgainst a live cluster, on a scheduleA FAIL and WARN list, worst first
kube-linterThe manifest against built-in checksCI, on the YAML, before applyNamed checks with a remediation line each
kubesecThe manifest as a security risk scoreCI and review, on the YAMLA score, plus critical, advise, and passed lists
kubeauditThe cluster or a manifest for the same concernsCluster audit, or pre-deployOne finding per concern, named by the auditor
PolarisThe workload against best practiceCI or the in-cluster dashboardPass, warning, or danger per check

kube-linter, one worked finding. Run it over the privileged pod from the lab. It flags the added capability by name: the container has NET_RAW added, check drop-net-raw-capability, and the remediation says to drop it. The finding names the container, the check, and the fix. That is what a linter finding looks like when it is good: specific enough to act on without reading the manual.

kube-linter lint pod-privileged.yaml
# look for: container "api" has NET_RAW capability added
# (check: drop-net-raw-capability)

What the snippet means: the linter reads the file and reports the check that fired. The check name is the rule. The remediation line is the fix. A linter that only says "bad" is a linter nobody reads.

kubesec, one worked finding. Run it over the same pod. It scores the manifest and shows the points lost. A privileged container is a critical deduction, and a container with no read-only filesystem is another. The score is the summary, and the critical list is the work. kubesec exits with code 2 when any scanned object scores 0 or lower, which is how CI gates on it.

kubesec scan pod-privileged.yaml
# look for: the score, and the critical list behind it
# a privileged container is a critical deduction

What the snippet means: the score is one number, and the lists behind it are the reasons. A score that drops below zero fails the CI gate. The fix is the same as the linter's: drop the privilege, drop the capabilities, fix the fields.

kubeaudit, one worked finding. Run it against the cluster. It audits the same concerns the gates check: run as non-root, read-only root filesystem, drop the scary capabilities, do not run privileged. A pod with no runAsNonRoot set is flagged by the run-as-non-root auditor, and the finding says the pod may run as root. The audit is the check after the fact, for the pods that are already running.

kubeaudit all
# look for: the run-as-non-root finding on any pod
# that has no runAsNonRoot set

What the snippet means: the audit reads the live cluster, not a file. A finding here is a pod that is already running with the weakness, and the fix is a rollout, not an edit.

Polaris, one worked finding. Run it over the workload, in CI or on the dashboard. It checks the pod against best practice and shows each check as pass, warning, or danger. A pod with no runAsNonRoot and no resource limits shows as danger on both checks, and the dashboard is the view a team reads in a review. Polaris is the broadest of the four: it checks security and reliability together, and the security findings sit next to the missing limits.

Gatekeeper, taught properly. Lesson 158 taught OPA and Kyverno, and named Gatekeeper once. Here is Gatekeeper in full. A ConstraintTemplate holds the Rego rule. A constraint instantiates it for a set of pods. Gatekeeper runs as a validating webhook, which means it is a service the API server calls, and a service that can be down. That is the price of its power: it can say anything Rego can express.

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: requirenonroot
spec:
  crd:
    spec:
      names:
        kind: RequireNonRoot
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package requirenonroot
        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          not container.securityContext.runAsNonRoot
          msg := sprintf("Container %v must run as non-root.", [container.name])
        }

What the snippet means: the template creates a new kind, RequireNonRoot, and the Rego says when a pod violates it: any container whose securityContext does not set runAsNonRoot. The template alone enforces nothing. The constraint is what turns it on.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: RequireNonRoot
metadata:
  name: pods-must-run-non-root
spec:
  enforcementAction: deny
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    excludedNamespaces:
      - kube-system

What the snippet means: every Pod outside kube-system must run as non-root, and the action is deny. Set enforcementAction to dryrun first, read the violations the audit controller reports, and only then flip it to deny. A constraint that flips straight to deny blocks deploys on the day it lands, and the teams learn to hate the policy instead of the risk.

Gatekeeper or Kyverno. Lesson 158 taught Kyverno in full: policies written as YAML, readable, with mutation and generation built in. Gatekeeper is Rego, expressive, and part of the OPA ecosystem. Pick Kyverno when the team writes policies and the team reads YAML all day, and when the rule fits a pattern. Pick Gatekeeper when the policy needs logic YAML cannot express, like a rule that compares two fields across two objects, or when the team already runs OPA elsewhere. Pick one engine, not both, because two engines means two places to look when a deploy is denied.

GatekeeperKyverno
Policy languageRego, a policy languageYAML patterns, readable as config
MutationYes, as a separate webhookYes, built in, same policy shape
AuditAudit controller reports violations on live objectsBackground mode reports violations on live objects
Pick it whenThe rule needs Rego logic, or the team already runs OPAThe team reads YAML all day and the rule fits a pattern

Do this: run step 3 of the lab, then write one Gatekeeper constraint of your own and apply it in dryrun first. Read the violations it reports for a week, then flip it to deny.

Try it: Advanced

I gate CI on kubesec. Which exit code fails the build?

Now do it: On your cluster, run kube-linter over the privileged pod from the lab and write down the check it flags by name. Then run kubesec over the same pod and write down the score and the critical list.

Master · Runtime security, and the OpenShift contrast

At the master level the pod is watched while it runs. The gates judged the manifest. The runtime judges the behaviour. Lesson 158 gave Falco one sentence. Here is the rule, the alert, and the syscall it fires on.

Falco, a rule and an alert. Falco watches the system calls, and a rule says which ones are suspicious. This rule fires when a shell is spawned inside a container. The syscall is execve, the process is a shell binary, and the pod should never have a shell. The macros in the condition ship with Falco's default rules: spawned_process means an execve exit event, container means the event came from inside a container, and shell_procs means the process name is a shell. Put the rule in the local rules file, never in the shipped one, because the shipped file is overwritten on upgrade.

- rule: Terminal shell in container
  desc: A shell was spawned inside a container
  condition: >
    spawned_process and container and shell_procs
    and proc.tty != 0
  output: >
    Shell spawned in container (user=%user.name container=%container.name
    shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)
  priority: WARNING
  tags: [container, shell, runtime]

What the snippet means: the condition is a filter over the syscall stream, and the output is the alert, with the fields filled in from the matching event. The alert names the user, the container, the shell, the parent process, and the command line. That is the whole story in one line, and it is the line you read at 2 AM. The priority is WARNING, and the tuning rule is the same as for any alert: start with the defaults, watch the noise for a week, and silence the rules that fire on normal behaviour.

KubeArmor, enforcement against Falco's detection. Falco detects. It watches the syscalls, matches the rule, and raises an alert. It never blocks anything. KubeArmor enforces. It blocks the call in the kernel, before it returns, and the process gets a permission error instead of a shell. The policy below blocks /bin/sh and /bin/bash in the shop pods, and it selects the pods by label, the way every Kubernetes object does.

apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: block-shell-in-shop
  namespace: shop
spec:
  selector:
    matchLabels:
      app: shop-api
  process:
    matchPaths:
      - path: /bin/sh
      - path: /bin/bash
  action: Block

What the snippet means: any process in a shop-api pod that tries to exec /bin/sh or /bin/bash is blocked in the kernel. Detection tells you it happened. Enforcement stops it happening. The two tools are a pair, and either one alone is half the job.

seccomp and AppArmor, applied to a pod. The seccomp profile is the syscall filter, and it loads with the pod: RuntimeDefault is the runtime's own filter, Localhost is a profile file on the node, Unconfined is no filter at all. Restricted demands RuntimeDefault or Localhost, and it demands the field be set. AppArmor is the file confinement, and it is set by annotation on the pod: container.apparmor.security.beta.kubernetes.io/api: runtime/default puts the default profile on the api container, and baseline forbids overriding it to anything else. Seccomp filters the calls. AppArmor confines the files. A pod with both is a pod that can neither make the call nor reach the file.

The SCC contrast. OpenShift's SecurityContextConstraints refuse the pod at admission, per service account, and they carry the allowed users, groups, capabilities, and volumes in the constraint itself. PSS labels a namespace and lets the mode decide. SCC is stricter and more specific. PSS is broader and simpler. A workload that moves from OpenShift to plain Kubernetes loses the SCC and must get the same guarantees from a restricted namespace plus a Gatekeeper constraint. Lesson 95 teaches SCC in full, with the blocked-rollout failure mode.

Do this: pick one pod you run, write its Falco rule for the one syscall it must never make, and read the alert the rule would fire. Then write the KubeArmor policy that blocks the same call, and say which one you would keep if you could only keep one.

Try it: Master

A shell spawned inside one of my pods. Which tool would have stopped it, and which one would only have told me?

Now do it: On your cluster, pick one pod you run, write its Falco rule for the one syscall it must never make, and read the alert the rule would fire. Then write the KubeArmor policy that blocks the same call.

Go deeper

seccompProfile, the three types

Unconfined is no filter. RuntimeDefault is the runtime's own filter, and it blocks the calls a container almost never needs, like mounting filesystems and loading kernel modules. Localhost is a profile file on the node, under the kubelet's seccomp directory, for a filter you wrote yourself. Restricted demands RuntimeDefault or Localhost, and it demands the field be set at all. A pod with no seccompProfile is a pod with no filter, and restricted refuses it.

AppArmor, the annotation

On a host with AppArmor, the profile is set by annotation, container.apparmor.security.beta.kubernetes.io/<container-name>, and the value runtime/default is the default profile. Baseline forbids overriding it to anything else. AppArmor confines what the process may read and write, which is a different job from seccomp: seccomp filters the calls, AppArmor confines the files.

Capabilities, the full picture

A container starts with a default set of capabilities, and that set is far more than a web server needs. Dropping ALL and adding back only what is used is the strong form. NET_RAW is the one a fresher meets first, because ping needs it. NET_BIND_SERVICE is the one restricted allows back, because binding to a port below 1024 needs it, and the usual fix is to listen above 1024 and let the Service forward the port. SYS_ADMIN is the one that ends the argument: it is close to root, and a pod that asks for it is a pod asking to be root.

PSA version labels

A namespace label can pin the version the level is checked against, pod-security.kubernetes.io/enforce-version: latest, or a version like v1.34. Latest moves when the cluster upgrades, and a level that moves can start rejecting pods that passed yesterday. Pin the version when the namespace must not change under you, and read the release notes before you move it.

ValidatingAdmissionPolicy, named only

Kubernetes ships a built-in CEL admission mechanism, ValidatingAdmissionPolicy, which writes simple policies without a webhook and without Rego. This lesson does not teach it. It is named here so you know it exists when a policy is too small for Gatekeeper and too big for a namespace label.

The SCC contrast, in full

OpenShift's SecurityContextConstraints refuse the pod at admission, per service account, and they carry the allowed users, groups, capabilities, and volumes in the constraint itself. PSS labels a namespace and lets the mode decide. SCC is stricter and more specific. PSS is broader and simpler. A workload that moves from OpenShift to plain Kubernetes loses the SCC and must get the same guarantees from a restricted namespace plus a Gatekeeper constraint. Lesson 95 teaches SCC in full.

ServiceAccount, pointed at only

The pod runs as a service account, and the account is the identity the pod carries to the API server. That identity belongs to Phase 11. This lesson points at it and stops. The fields in this lesson limit what the pod may do. The account limits what the pod may ask for.

What changes at 10x and 100x

At 10x the namespaces, the labels are the first thing that breaks, and the fix is the labels in Git, reviewed like code, not set by hand. At 10x the policies, the constraint count is the first thing that breaks, and the fix is one template per rule with many constraints, not one template per team. At 100x, the runtime is the product. The Falco rules are code, the KubeArmor policies are code, and the alert on a shell fires before the shell finishes. The fields do not change. The number of pods they run on does.

Further reading

The gate chain this lesson plugs into: Lesson 158 Kubernetes Security. The image scan that runs before these gates: Lesson 157 Container Supply Chain Security. The OpenShift contrast: Lesson 95 OpenShift. How an attacker thinks about these gates: Lesson 155 Threat Modeling & Security Practice. Runtime and detection, the wider view: secops.html. The scheduling rules a hardened pod still obeys: Lesson 90 How a Pod Gets a Home and Starts Running.

One time it broke

What broke

A pod passed every gate and then opened a shell at 2 AM. The manifest was clean: uid 1000, read-only filesystem, no escalation, no capabilities added. The image scan was clean. PSA passed it, Gatekeeper passed it, and the pod ran. Then the app took a request that carried a command, and the app shelled out to run it. The shell spawned, the shell wrote to /etc, and the pod that looked hardened on paper was running commands from the network. The gates had done their job. The gates read manifests. The behaviour was never in the manifest.

Timeline

  • Day 1The pod deploys clean. Linters find nothing, PSA passes it, Gatekeeper passes it.
  • Day 14A request arrives carrying a command string. The app passes it to a shell instead of rejecting it.
  • Day 14, 02:00The shell spawns inside the pod. Falco fires on the execve, and the alert names the pod, the shell, and the command line.
  • Day 14, 02:01KubeArmor blocks the write to /etc. The shell runs, but it cannot change the filesystem.
  • Day 14, 02:05The on-call engineer reads the Falco alert, sees the command line, and deletes the pod.
  • Day 14, 03:00The image is rebuilt without the shell binary, and the new pod has no shell to spawn.
  • Day 15The namespace moves from baseline to restricted, and a Gatekeeper constraint is added that drops ALL capabilities and demands a read-only filesystem.
  • Day 16A Falco rule is added that pages on any shell in a production pod, not just logs it.

What changed after

Three things. First, the image is built without a shell, because a pod with no shell binary cannot spawn one, and the runtime is the second line, not the first. Second, the namespace moved to restricted, because baseline allowed the pod that looked clean and still misbehaved. Third, the Falco alert pages now, because a runtime alert that only logs is a report nobody reads at 2 AM. The gates did nothing wrong in this incident. The lesson is that a gate that reads the manifest protects the deploy, not the behaviour, and the behaviour is what the runtime is for.

If you remember nothing else

Design exercise

Design the pod security for a cluster with 50 services across 10 namespaces. Each team deploys its own services. Pods must run as non-root, with a read-only filesystem, no privilege escalation, and no capabilities beyond NET_BIND_SERVICE. A pod that opens a shell at 2 AM must be caught and stopped. Two namespaces run on OpenShift. The question is the depths: sketch the fields, the PSA rollout, the linter bench, the policy engine, and the runtime, and say what you would measure.

Clarifying questions to ask

  • Which namespaces enforce restricted today, and which ones still run as root because the app needs it? A namespace that cannot go restricted needs a named exception with a date.
  • What happens when the Gatekeeper webhook is down: does the cluster fail closed and block deploys, or fail open and let them through?
  • Who owns the ConstraintTemplates: one platform team, or does each team write its own, and who reviews a Rego change?
  • What happens when a CVE is published for a running image: is there a rescan, and who owns the patch? (Lesson 157 owns the scan. This exercise owns the pod.)
  • How are the Falco alerts routed: do they page, or do they log, and who triages a shell alert at 2 AM?
  • What do the two OpenShift namespaces lose when their SCCs are translated to PSS plus Gatekeeper, and who signs off on the translation?

Worked approach

Start with the fields. Every pod gets the same five: runAsUser 1000, runAsNonRoot true, readOnlyRootFilesystem true, allowPrivilegeEscalation false, capabilities drop ALL. NET_BIND_SERVICE comes back only for the pods that bind below 1024, and the usual fix is to listen above 1024 and let the Service forward the port. The fields go in the pod template in Git, not in a ticket.

Then the rollout. Every namespace starts at audit: restricted for a week, warn: restricted alongside it so the teams see the warnings. Read the audit log, fix the violations, and only then flip enforce: restricted. A namespace that flips to enforce with violations still in it blocks deploys on the day it flips, and the teams learn to hate the gate instead of the risk.

Then the bench. kube-linter runs in CI on every manifest, kubesec scores the review, kubeaudit runs against the cluster weekly, Polaris shows the dashboard. One finding per bad field, and the finding names the field. The bench is the advice. PSA and Gatekeeper are the law.

Then the engine. Gatekeeper carries the rules PSA cannot express: the registry rule, the label rule, the run-as-non-root rule for the namespaces that cannot go restricted yet. Kyverno is the alternative when the team writes YAML all day and the rule fits a pattern. Pick Gatekeeper when the rule needs Rego logic, like comparing two fields. Pick Kyverno when the rule is a pattern and the team reads YAML. One engine, not both, because two engines means two places to look when a deploy is denied.

Then the runtime. Falco watches the syscalls and pages on a shell. KubeArmor blocks the exec in the shop namespaces. The seccomp profile is RuntimeDefault everywhere. The two OpenShift namespaces keep their SCCs, and the translation to PSS plus Gatekeeper is written down and signed off, not assumed.

Trade-offs to name. Restricted buys a simple guarantee and costs the apps that need root, and each one needs a named exception with a date. Gatekeeper buys expressive rules and costs a webhook that must stay up, and a webhook that is down is a cluster that cannot deploy. Falco buys detection and costs noise, because every alert has to be triaged. KubeArmor buys enforcement and costs a policy per workload, because a block that is too broad breaks the app. And the target is the depths, not the speed. A pod that is fast and unguarded fails this design, and the depths are where the security lives.

Saved in this browser, so the Lessons index can show your progress across the course.

Next in this series · Lesson 160
Kubernetes Secrets & ConfigMaps: The Cluster's Own Config System

This lesson locked down what a Pod may do once it runs. The next lesson teaches the system behind ConfigMaps and Secrets: the five kinds of Secret, the choice between an env var and a file, and the encryption that protects etcd. A Secret is not a password in a file. It is an object with a type, a namespace, and a set of readers.

Go to Lesson 160 →

Or browse the full course from the Lessons index.