Platform Engineering · Lesson 162

Kubernetes Authentication & Authorization: Who the API Server Trusts, and What It Allows

Lesson 158 taught you the gates a deploy passes. This lesson teaches you the front door itself. Every request to the API server answers three questions, in order. Who are you. What may you do. What got recorded. The first question is authentication. The second is authorization. The third is the audit log. A request that fails the first question never reaches the second. A request that passes both still leaves a record. I learned this the way I learn most things in this job. I watched a deploy fail with a forbidden error, fixed the permission, and then found the next gate waiting behind it.

Difficulty: Advanced · Know this first: Lesson 52, Lesson 158, Lesson 156, Lesson 94, Lesson 159

What you will build

By the end of this lesson you will have read a kubeconfig the way the API server reads it, and you will know what breaks when the context is wrong. You will have made a client certificate with openssl, asked the cluster to sign it, and used it to talk to the API server. You will have created a service account, bound it to a Role that allows one job and nothing else, and watched a request from that account get denied the moment it asks for more. You will have written an audit policy that records the full body of every Secret read and only the metadata of everything else. You will know which gate denied a request from the error alone.

What you need: a cluster you can run pods on, with kubectl. The lab steps run on your own cluster, and every expected output is described as what to look for, not quoted as a transcript. I did not run the lab on a cluster here. This VM hands out no cpu controller in its cgroup, so no k3s server can start on it. The same wall stopped the Phase 9 lab work, and I did not re-attempt the install. No kubectl output from this machine appears in this lesson, because there is no cluster on this machine to produce it. Your numbers will differ. The shape will not.

The story: three questions, in order

Every request to the API server passes three questions, and the order never changes. First, who are you. The API server checks the credential the request carries: a client certificate, a bearer token, or an OIDC token. If no credential checks out, the request stops here with a 401. There is no identity, so there is nothing to authorize. Second, what may you do. The API server checks the identity against the bindings: a Role, a ClusterRole, a Node rule, a webhook policy. If no binding allows this verb on this resource in this namespace, the request stops here with a 403. Third, what got recorded. The audit log writes the request down, at the level the audit policy sets for it. A request that passes all three still leaves a record. That is the point of the third question. The gates decide. The log remembers.

The front door, one table

Question it answersWhat it checksWhat a deny looks like
AuthenticationWho are youThe credential: client cert, bearer token, OIDC token401 Unauthorized. No identity was established.
AuthorizationWhat may you doThe bindings: Role, ClusterRole, Node, webhook403 Forbidden. The identity is known and the answer is no.
AdmissionMay this object existThe object itself: mutation, then validation, then Pod SecurityThe request is rejected, and the error names the rule or the field.
AuditWhat got recordedThe request, at the level the policy setsNothing is denied here. The record is written either way.

Lesson 158 taught the gate chain in one sentence each: the API server asks who you are, RBAC asks what you may do, admission asks whether the pod may exist. This lesson teaches the machinery behind each sentence. Lesson 94 taught where the API server runs and what the kubelet does. This lesson teaches what happens inside the API server when a request arrives. Lesson 159 taught Pod Security Admission as the last gate. This lesson teaches the three gates that run before it.

401 is not 403, and the difference tells you where to look

A 401 means the API server does not know who you are. The credential is missing, expired, or signed by a CA the API server does not trust. The fix lives on the client side: the kubeconfig, the cert, the token. A 403 means the API server knows exactly who you are and the answer is no. The credential is fine. The binding is missing, or the binding allows a different verb, a different resource, or a different namespace. The fix lives on the cluster side: the Role, the RoleBinding, the subject name. I read the status code first, because it tells me which side of the door to fix. A 401 with a good credential usually means the context is wrong, and the request went to a different cluster than the one I meant.

Do this now, before the diagram

On your own machine, run kubectl config current-context and kubectl config view --minify. Write down the cluster address, the user name, and the namespace the context carries. Then run kubectl auth whoami and write down the identity the API server sees. If the two do not match, you have found the first lesson of this file before reading it: the context picks the identity, and the identity is what the gates judge.

Run it as:

When it breaks, I check

One thing to remember

Four paths, one family

The diagram above follows one request from the context to the record. These four paths are the same idea drawn four ways. One clean deploy. One expired cert, stopped at authentication. One missing binding, stopped at RBAC. One numbers trail, hop by hop. Run each path. One run takes 7.2 seconds across the path, so you can watch each hop light up in turn.

The return rule: the green dot only starts moving after the orange dot reaches the data hop. A return with no finished work is a lie. The diagram never draws one. The failure replay at the end of Run all four shows the other side of the rule. Run it and watch the orange dot stop at Authn and never come back.

Path 1: the clean deploy

Nothing skipped

One request, one deployment. The context is right, the cert is live, the binding allows the create, admission passes the pod, etcd stores it, and the answer comes back in 33 ms. The green dot starts back from the etcd hop, because that is where the finished work lives.

Context picked0 ms
TLS passes4 ms
Authn passes3 ms
RBAC passes5 ms
Admission passes10 ms
etcd stores it8 ms
Audit logged2 ms
Response returns1 ms

Not run yet.

Path 2: the expired cert, stopped at Authn

RBAC: skipped

Same deploy, a certificate that expired yesterday. The context is right and TLS passes, because TLS only checks the server. Authentication reads the dead certificate and denies the request in 9 ms. No identity is established, so no work is done, and no green dot ever starts back.

Context picked0 ms
TLS passes4 ms
Authn denies5 ms
Never storeddenied

Not run yet.

Path 3: the missing binding, stopped at RBAC

Admission: skipped

Same deploy, no RoleBinding for mohan in shop. The context is right, the cert is live, and authentication passes. RBAC reads the bindings, finds nothing that allows create on deployments in shop, and denies the request in 13 ms. The identity is known. The permission is missing. No work is done, so no green dot ever starts back.

Context picked0 ms
TLS passes4 ms
Authn passes3 ms
RBAC denies6 ms
Never storeddenied

Not run yet.

Path 4: the numbers trail

Nothing skipped

The same deploy, measured instead of timed. Four authn modes, four authz modes, three audit levels, two bindings checked. The trail adds up to 33 ms, and the bindings are the check that keeps the request honest.

Authn modes4
Authz modes4
Audit levels3
Bindings checked2
Verdict33 ms

Not run yet.

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one deploy through the API server gates, not a measurement from a live system, but the shape is honest. The etcd write is the big hop at 8 ms, and the total is 33 ms. Switch the scenario and watch the bar change. In the expired-cert story the Authn hop grows to 5 ms and everything after it disappears, because no identity is ever established. In the missing-binding story the RBAC hop grows to 6 ms and everything after it disappears, because the identity is known and the answer is no. Move a simulator slider and watch the gates, the verdict, and the total move with it.

The credential, the binding, the policy, and the admission mode: move the sliders and watch the request bend

The budget bar above shows three fixed stories. This simulator lets you write your own. It answers the sizing question at the heart of this lesson: which gate denies the request, which credential gets it through, and what the audit log is left holding when the gates let it through. The request here is a deployer applying a Pod that runs as root to the shop namespace. You set the credential, the binding, the webhook policy, and the admission mode. Move a slider and watch the identity, the gate verdicts, and the budget total change. The numbers are a teaching example, built from the formulas below, not live measurements. The shape is the lesson: the credential decides the identity, the binding decides the permission, the policy decides the exception, and admission decides whether the object may exist.

valid client cert

What the request carries to prove who it is. A valid client cert identifies mohan. An expired cert identifies nobody. A stolen service account token identifies the deployer service account, from outside the cluster.

present

Whether a binding allows this identity to create deployments in the shop namespace. Present means the create is allowed. Missing means RBAC denies it with a 403.

allow

What the authorization webhook says about this request. Allow means it has no objection. Deny means the policy refuses it, and the webhook call costs 12 ms on top of the RBAC check.

baseline (pass)

What Pod Security enforces in the shop namespace. Baseline allows a pod that runs as root. Restricted blocks it, because the pod in this request runs as uid 0.

Try this: (1) At the defaults, valid cert, binding present, policy allow, admission baseline, the request passes every gate and the budget total is 33 ms. (2) Set the credential to expired cert. The request is denied at authentication in 9 ms, and the identity is nobody. (3) Set the credential back to valid cert and set the binding to missing. The request is denied at RBAC in 13 ms, and the identity is mohan with no permission. (4) Set the credential to stolen SA token, leave the binding missing. The request is still denied at RBAC in 13 ms. The token was real, and the binding was not there, so the real token bought nothing. (5) Set the binding back to present and the credential to stolen SA token. The request passes every gate in 33 ms, and the only record that it came from outside the cluster is the audit log. That is the incident in this lesson. Then set the policy to deny and watch it stop in 24 ms, and set the policy back to allow and admission to restricted and watch it stop in 27 ms.

Formulas, in plain terms. The gates run in order: authentication, then RBAC, then the webhook policy, then admission. An expired cert denies at authentication in 9 ms: 4 ms of TLS plus 5 ms of Authn. A missing binding denies at RBAC in 13 ms: 4 ms of TLS, 3 ms of Authn, and 6 ms of RBAC. A webhook deny costs 24 ms: 4 ms of TLS, 3 ms of Authn, and 17 ms of RBAC plus the webhook call. An admission block costs 27 ms: 4 ms of TLS, 3 ms of Authn, 5 ms of RBAC, 6 ms of mutation, and 9 ms of validation. A request that passes every gate costs 33 ms: 4 ms of TLS, 3 ms of Authn, 5 ms of RBAC, 6 ms of mutation, 4 ms of validation, 8 ms of etcd, 2 ms of audit, and 1 ms of response. A stolen token passes authentication, because the token is real. It is the binding that decides whether the real token gets anything.

Lab: read a kubeconfig, sign a certificate, bind a service account

Three steps, all run on your own cluster. There is no kubectl output from this machine in this lesson, because there is no cluster on this machine. The k3s install was not re-attempted here: this VM hands out no cpu controller in its cgroup, the same wall Phase 9 hit, so no k3s server can start here. What follows is what to run and what to look for, not a transcript. Your output will differ. The shape will not.

Step 1: the worked kubeconfig

Save this as kubeconfig.yaml. It is the file from the Basic tier: one cluster, one user, one context, and the context is current. The credential files are references, not embedded data. Both forms are taught in the Basic tier. This file was hand-written and checked as YAML in Python on Nov 21, 2026. It parses. It has not been run against a live API server from this machine.

apiVersion: v1
kind: Config
clusters:
  - name: shop-prod
    cluster:
      certificate-authority: /home/mohan/.kube/shop-ca.crt
      server: https://api.shop.example:6443
users:
  - name: mohan
    user:
      client-certificate: /home/mohan/.kube/mohan.crt
      client-key: /home/mohan/.kube/mohan.key
contexts:
  - name: shop-prod
    context:
      cluster: shop-prod
      user: mohan
      namespace: shop
current-context: shop-prod

Read it back and check the identity the API server sees:

kubectl config view --minify
kubectl config current-context
kubectl config use-context shop-prod
kubectl auth whoami

What to look for: the minified view shows the shop-prod cluster, the mohan user, and the shop namespace, and nothing else. current-context prints shop-prod. auth whoami prints the identity the API server sees, and it should match the user the context names. If it does not match, the context is wrong, and the gates will judge the wrong identity. Then try the wrong context on purpose: kubectl config use-context shop-prod is the right one, so pick a different context if your file has one, run kubectl auth whoami again, and watch the identity change. Switch it back when you are done.

Step 2: the client certificate, signed by the cluster

Make a private key and a certificate signing request on your own machine. The openssl commands below were run on Nov 21, 2026, against a throwaway CA, and they work: the signed certificate came back with subject CN mohan and O shop-team, issued by the shop CA. The key material was deleted after the check. Run the same commands on your machine, with your own CA, and keep the key private.

openssl genrsa -out mohan.key 2048
openssl req -new -key mohan.key -out mohan.csr -subj "/CN=mohan/O=shop-team"

What to look for: the key file exists and is readable only by you, and the CSR file exists. The subject carries the identity the API server will see: CN mohan becomes the username, O shop-team becomes the group. Then send the CSR to the cluster as a CertificateSigningRequest. This file carries the real base64 of a CSR made the same way, and it was checked against the Kubernetes 1.34 schemas with kubeconform, strict mode, on Nov 21, 2026. It validates. It has not been applied to a live API server from this machine.

apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: mohan
spec:
  request: MIICaTCCAVECAQAwJDEOMAwGA1UEAwwFbW9oYW4xEjAQBgNVBAoMCXNob3AtdGVhbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANdITu6cLvBa7RYbAeIGpp8PLeY04eE0pY002acKjMhbjE+70CgEv/PHSb0B3JSzdkeeVLTCD65TUlLDXDpaKigCf9Ep7Obw35kmrC0PpoL8t9D0u+s5EQqYlE02J+m9BHqJqKNSacpIJOvAaCbD4Z5depjLG/3rMeQfWCtt9dWvD8WTTE9U/cQGZvQzIrOQOM5dVqomCojzmR54t1sSP8aRKL0xkQqRNz8VnDB7iiAhk570TlLIdLZdjfkiA9JR4utlamlzI47TktkrSMOUd9clVjAFb4Ejmt4gmQ28vZDHPBuUJ34T48bWbYefhema2PPizrK2SvBJ4GklE8FTThUCAwEAAaAAMA0GCSqGSIb3DQEBCwUAA4IBAQAjMS5rf+WG7IfxXxkWFdBY7lpxCIxa7lWg0+wJ8iJy06oyFronnAARvZvHnoY4s8Edl6Un4xthYeHY56UZ1EC2WcVwkdQFYn1VOdtczSq2A01j43+IrySkEdTi0hvR39UDmpq0iRHA5TWvRaIf+rotiEMvk6zeklyJQA3Uycsr3PgLzmCt/QOV4oY3kmFtWSoQeGNhZrYteNKCQ55mZhEgza1D8swYirQbNj5ViQNqeCxhetEFkS/b3GMirApwLh+5wPtpCCnbvXtJSqNj3Jg6Is9VxC+eJW2kq397ZVaTMSX7Yi/MZ/MbLk+1SYQxUFUqIX7QEAk/L5hv0TCKWubR
  signerName: kubernetes.io/kube-apiserver-client
  usages:
    - client auth

Apply it, approve it, and fetch the signed certificate:

kubectl apply -f csr-mohan.yaml
kubectl certificate approve mohan
kubectl get csr mohan -o jsonpath='{.status.certificate}' | base64 -d > mohan.crt

What to look for: the CSR moves to Approved and then to Issued, and mohan.crt is a certificate, not an error message. Point your kubeconfig at it with kubectl config set-credentials mohan --client-certificate=mohan.crt --client-key=mohan.key and run kubectl auth whoami again. The identity should be mohan in the shop-team group. Then delete the CSR object when you are done: kubectl delete csr mohan. A signed certificate lives until it expires, and there is no revocation list to kill it early. That is why the expiry matters, and why a short-lived certificate is the safer one.

Step 3: the service account, the Role, the binding, and the bound token

Save these four files. The ServiceAccount is the identity the pod carries. The Role is the deployer Role from Lesson 158: create and update Deployments, read pods, nothing more. The RoleBinding ties the Role to the service account in the shop namespace. The Pod mounts a projected token with audience api and an expiry of one hour. All four files were checked against the Kubernetes 1.34 schemas with kubeconform, strict mode, on Nov 21, 2026. They parse and validate. They have not been run against a live API server from this machine.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: deployer
  namespace: shop
automountServiceAccountToken: true
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: deployer
  namespace: shop
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "create", "update", "patch"]
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: deployer-binding
  namespace: shop
subjects:
  - kind: ServiceAccount
    name: deployer
    namespace: shop
roleRef:
  kind: Role
  name: deployer
  apiGroup: rbac.authorization.k8s.io
apiVersion: v1
kind: Pod
metadata:
  name: shop-api
  namespace: shop
spec:
  serviceAccountName: deployer
  containers:
    - name: api
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      volumeMounts:
        - name: token
          mountPath: /var/run/secrets/tokens
          readOnly: true
  volumes:
    - name: token
      projected:
        sources:
          - serviceAccountToken:
              audience: api
              expirationSeconds: 3600
              path: token

Apply all four and ask the API server what the service account may do:

kubectl apply -f sa-deployer.yaml -f role-deployer.yaml -f rolebinding-deployer.yaml -f pod-projected-token.yaml
kubectl auth can-i create deployments --namespace shop --as system:serviceaccount:shop:deployer
kubectl auth can-i get secrets --namespace shop --as system:serviceaccount:shop:deployer
kubectl auth can-i create deployments --namespace default --as system:serviceaccount:shop:deployer

What to look for: the first check says yes, the second says no, and the third says no. The service account may create deployments in shop, and it may do nothing with Secrets, and it may do nothing in the default namespace. That is least privilege in three answers. Then read the projected token from inside the pod:

kubectl exec -n shop shop-api -- cat /var/run/secrets/tokens/token
kubectl exec -n shop shop-api -- ls /var/run/secrets/kubernetes.io/serviceaccount/

What to look for: the projected token exists at the path the volume mounts, and the old static token path is empty, because this pod does not mount the legacy token. The projected token is short-lived and bound to this pod. Delete the pod and the token dies with it. Clean up when you are done:

kubectl delete pod -n shop shop-api
kubectl delete rolebinding -n shop deployer-binding
kubectl delete role -n shop deployer
kubectl delete serviceaccount -n shop deployer

What to look for: the pod is gone, the binding is gone, the Role is gone, and the account is gone. A tidy cluster is a choice you make after every experiment, not a state the cluster keeps for you.

Check yourself

Three questions. No score is saved anywhere. This is just you against the material.

Basic to master

The front door is not one skill. It is four levels of the same job, and every team I have worked with sits on one of them. The levels are not about how many flag names you can recite. They are about whether the request that arrives is the request you meant to allow. Read them in order. Each level fixes the failure that hurt the most at the level below it.

Basic · kubeconfig first, then the authn modes as a set

At the basic level you learn the file before you learn the modes. The kubeconfig is the first thing kubectl reads, and it is the first thing that breaks. It has three lists and one pointer. The clusters list names the API servers. The users list names the credentials. The contexts list ties one cluster to one user in one namespace. The current-context pointer picks which context is live. The full worked file is in the lab. These are the pieces, each one on its own.

Clusters, users, contexts. A cluster entry is an address and a CA. A user entry is a credential. A context entry is the join between them, plus a namespace. The current-context is the one kubectl uses when you do not name another.

clusters:
  - name: shop-prod
    cluster:
      certificate-authority: /home/mohan/.kube/shop-ca.crt
      server: https://api.shop.example:6443
users:
  - name: mohan
    user:
      client-certificate: /home/mohan/.kube/mohan.crt
      client-key: /home/mohan/.kube/mohan.key
contexts:
  - name: shop-prod
    context:
      cluster: shop-prod
      user: mohan
      namespace: shop
current-context: shop-prod

What the snippet means: the file says the shop-prod cluster lives at that address, the mohan user proves who they are with that certificate and key, and the shop-prod context joins the two in the shop namespace. The last line makes it live. Change that line and every kubectl command goes somewhere else.

File references or embedded data. A credential can live in its own file, as above, or inside the kubeconfig as base64. The embedded form uses certificate-authority-data, client-certificate-data, and client-key-data instead of the file paths. The data is the same bytes, base64 encoded. A kubeconfig with embedded data is one file you can copy to another machine, and it carries the private key with it, which is why I keep the file references on a shared machine and the embedded form only on my own laptop. A private key in a copied file is a credential that travels, and a credential that travels is a credential that leaks.

What breaks when the context is wrong. Three things, and each one looks different. A context that points at the wrong cluster sends the request to the wrong API server, and the error comes from a cluster you did not mean. A context that names the wrong user sends the wrong credential, and the request fails with a 401 or passes as the wrong identity. A context that names the wrong namespace applies the object where you did not mean it, and the deploy succeeds in the wrong place, which is worse than a failure. The fix for all three is the same command: kubectl config use-context shop-prod, and then kubectl auth whoami to check the identity the API server sees. The KUBECONFIG variable can list several files, and kubectl merges them, so a context can also come from a file you forgot you had. When the context looks right and the identity is wrong, I check which files are merged.

Client certificates. A client certificate is the oldest credential in Kubernetes, and it is still the one I reach for on my own machine. The API server trusts certificates signed by its client CA, set with --client-ca-file on the API server. The common name becomes the username. The organization becomes the groups. A certificate for CN mohan and O shop-team is the user mohan in the group shop-team. The CSR flow, in outline, is four steps: make a key and a signing request with openssl, send the request to the cluster as a CertificateSigningRequest, approve it, and fetch the signed certificate. The lab runs all four. kubectl can also use a certificate directly, with --client-certificate and --client-key, without putting it in a kubeconfig at all. Two things to know about certificates. First, they expire, and an expired certificate is a 401. Second, there is no revocation list. A certificate that is stolen works until it expires, and the only fix is a short expiry and a quick rotation. That is why a certificate with a one-year life is a risk, and a certificate with a one-day life is a chore. The balance is the job.

Bearer tokens and service accounts. A bearer token is a string the request carries in the Authorization header. The API server can read static tokens from a file, set with --token-auth-file, where each line is a token, a username, a user id, and groups. Static tokens never expire until the file changes and the API server restarts, which is why they are the weakest credential in this lesson. Service account tokens are the bearer tokens for workloads. A ServiceAccount is the identity a pod carries to the API server, and this is its first teaching in the course. Every namespace has one service account by default, named default, and every pod that names no service account runs as default. The token is mounted into the pod at /var/run/secrets/kubernetes.io/serviceaccount/token, unless automount is off. The field lives in two places: the ServiceAccount itself, and the pod spec. Either one can turn it off. A pod that never talks to the API server should run with automount off, because a token the pod never uses is a token an attacker can still steal.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: deployer
  namespace: shop
automountServiceAccountToken: true

What the snippet means: the shop namespace gains an identity named deployer, and its token is mounted into every pod that runs as it. Set the field to false and the pod gets no token at all. The lab binds this account to the deployer Role, and the binding is what turns the identity into a permission.

OIDC for the API server. OIDC is the mode that lets the cluster trust your company's identity provider instead of its own certificates. The API server takes four flags and friends: --oidc-issuer-url names the provider, --oidc-client-id names the client, --oidc-username-claim picks which claim becomes the username, and --oidc-groups-claim picks which claim becomes the groups. The user logs in once, the provider returns an ID token, and kubectl sends that token with every request. The API server checks the issuer, the audience, and the expiry, and then trusts the identity. The protocol itself lives in Lesson 52, in the federation section that teaches OAuth, OIDC, SAML, and MFA. This lesson teaches only the API-server half: the flags, the claims, and the trust. JWT is mentioned here once, and only as the token format OIDC returns. The format is taught in Lesson 52. Here it is a detail, not a lesson.

--oidc-issuer-url=https://login.example.com
--oidc-client-id=kubernetes
--oidc-username-claim=email
--oidc-groups-claim=groups

What the snippet means: the API server trusts tokens from that issuer, for that client, and reads the username from the email claim and the groups from the groups claim. A token from any other issuer is rejected. A token past its expiry is rejected. The claims are the identity, and the identity is what RBAC judges.

Webhook authentication, named in full. The last authn mode is a webhook. The API server sends a TokenReview to a service outside the cluster, and the service answers with the identity or with nothing. The flags are --authentication-token-webhook-config-file for the kubeconfig that says where the service lives, and --authentication-token-webhook-cache-ttl for how long a good answer is cached. A webhook lets the cluster trust an identity system Kubernetes knows nothing about, and it costs a service that must stay up. If the service is down, authentication fails, and the request stops with a 401. That is the price of the flexibility, and it is why a webhook is never the only authenticator.

Do this: run step 1 and step 2 of the lab on your own cluster. Write down the context, the identity from auth whoami, and the expiry date of the certificate you made. Then set a calendar reminder for the day before it expires, because the 401 that arrives on expiry day is the one you will not expect.

Try it: Basic

I change one line in my kubeconfig and every kubectl command goes somewhere else. Which line did I change?

Now do it: On your own machine, run kubectl config use-context shop-prod, then run kubectl auth whoami and check the identity the API server sees.

Intermediate · Bound tokens and workload identity

At the intermediate level you stop using tokens that live forever and start using tokens that live for one job. The static service account token from the Basic tier lives in a Secret until someone deletes the Secret, and it works from anywhere until then. That is the token in the incident at the end of this lesson. The bound token is the fix.

SA token projection and bound tokens. A projected token is issued by the TokenRequest API for one pod, for one audience, with a short expiry. The kubelet asks for it, mounts it into the pod as a projected volume, and refreshes it before it expires. The pod dies, the token dies with it. The audience is the check that matters: the token is only valid for the service named in the audience, so a token stolen for one service does not work on another. The expiry is the second check: the default is one hour, and the kubelet refreshes it at eighty percent of its life, so the pod never sees a dead token and an attacker never gets a live one for long.

volumes:
  - name: token
    projected:
      sources:
        - serviceAccountToken:
            audience: api
            expirationSeconds: 3600
            path: token

What the snippet means: the pod gets a token at the path the volume mounts, issued for the audience api, expiring in one hour. The full pod is in the lab. A token with no audience is a token any service will accept, and a token with no expiry is a static token by another name. Set both, every time.

Static tokenBound token
Where it livesA Secret, until someone deletes itA projected volume, until the pod dies
ExpiryNone, until the Secret is deletedOne hour by default, refreshed by the kubelet
AudienceNone, any service accepts itNamed, only that service accepts it
StolenWorks from anywhere, until revokedWorks only for its audience, and only until it expires

Workload identity. A pod that needs a cloud service faces the same question as a human: how does it prove who it is without a static key. Workload identity answers it with federation. The cloud trusts the Kubernetes token, and the Kubernetes service account is bound to a cloud identity. No static key exists anywhere in the chain. Three clouds, one pattern. On AWS, IRSA binds a Kubernetes service account to an IAM role through the cluster OIDC provider, and the pod carries the annotation with the role ARN. On GCP, Workload Identity binds a Kubernetes service account to a Google service account, and the pod runs as the Google account. On Azure, Workload Identity binds a Kubernetes service account to a managed identity through a federated credential, and the pod carries the client id label. The names differ. The shape does not: the pod presents its bound token, the cloud checks the issuer, and the cloud hands back short-lived credentials for its own services. A static cloud key in a Secret is the old way, and the old way is the incident waiting to happen. Lesson 156 teaches the other half of this trust: Vault trusts the same service account token, and the Vault Kubernetes auth method is the same pattern pointed at a vault instead of a cloud.

Do this: run step 3 of the lab, then delete the pod and read the token path again from a new pod. The token is different, because the token belongs to the pod, not to the account. Then change the audience in the pod to a different service and watch the token change with it.

Try it: Intermediate

My pod gets a projected token with an audience of api and an expiry of one hour. When does the kubelet refresh it?

Now do it: On your own cluster, run step 3 of the lab, delete the pod, and read the token path again from a new pod to see the token change.

Advanced · The authz modes, in full

At the advanced level you learn that RBAC is one authorizer among four, and you learn when each one decides. The API server runs its authorizers in order. If any authorizer denies, the request is denied. If one allows and none denies, the request passes. If all of them say no opinion, the request is denied. That rule is the whole of authorization, and every mode below is one voice in it.

RBAC depth: aggregation. Lesson 158 taught the Role, the ClusterRole, the RoleBinding, and the ClusterRoleBinding, and it worked a deployer Role and binding in full. This lesson adds the piece Lesson 158 never named: aggregation. A ClusterRole can collect other ClusterRoles by label, through an aggregationRule. The collected roles carry a label, and the collecting role names the label in a selector. Add a label to a new role and the collection grows, without touching the collecting role at all. The built-in admin, edit, and view roles work this way: label a role aggregate-to-admin and every admin gets it.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: deployer-read
  labels:
    rbac.shop.example/aggregate-to-deployer: "true"
rules:
  - apiGroups: [""]
    resources: ["pods", "services"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: deployer
aggregationRule:
  clusterRoleSelectors:
    - matchLabels:
        rbac.shop.example/aggregate-to-deployer: "true"
rules: []

What the snippet means: the deployer-read ClusterRole carries the label, and the deployer ClusterRole collects every role with that label. Add another labeled role and the deployer gains its rules at once. The collecting role carries an empty rules list, because its rules come from the collection, not from itself. This file was checked against the Kubernetes 1.34 schemas with kubeconform, strict mode, on Nov 21, 2026.

Role or ClusterRole, the decision rules. A Role names permissions in one namespace. A ClusterRole names permissions across the whole cluster, or names a reusable set that a RoleBinding can use in one namespace. The decision is three questions. First, does the permission live in one namespace. If yes, a Role. Second, does the permission touch cluster-wide objects like nodes or namespaces. If yes, a ClusterRole, because a Role cannot name them. Third, is the same set needed in many namespaces. If yes, one ClusterRole and one RoleBinding per namespace, not one Role copied ten times. A ClusterRoleBinding is the dangerous one: it grants the permission everywhere, and a ClusterRoleBinding to a Role that allows Secrets is a permission to read every Secret in the cluster. I never bind a ClusterRole cluster-wide unless the job truly is cluster-wide, and a deployer never is.

RoleClusterRole
ScopeOne namespaceWhole cluster, or reusable across namespaces
Bound byRoleBinding, in that namespaceClusterRoleBinding, or RoleBinding in one namespace
Use it whenThe job lives in one namespaceThe object is cluster-wide, or the set repeats
Never whenThe object is a node or a namespaceThe job is a deployer in one namespace

Node authorization. The kubelet is a client of the API server too, and it has its own authorizer. Node authorization allows a kubelet to act only on the objects that belong to its own node. The kubelet identifies as system:node:<node-name>, in the group system:nodes. It may read the pods scheduled to its node, and the Secrets and ConfigMaps those pods mount. It may update the status of its own Node object. It may create events. It may not read a Secret for a pod on another node, and it may not list every pod in the cluster. The NodeRestriction admission plugin enforces the same boundary at admission: a kubelet may only modify the Node object that carries its own name. Why it matters is simple. A kubelet runs on every node, and a compromised node is a compromised kubelet. Node authorization is what stops one bad node from reading the Secrets of every pod in the cluster.

Webhook authorization. The webhook authorizer sends a SubjectAccessReview to a service outside the cluster, and the service answers allowed, denied, or no opinion. The flags are --authorization-webhook-config-file for the kubeconfig that says where the service lives, and --authorization-webhook-cache-authorized-ttl and --authorization-webhook-cache-unauthorized-ttl for how long each answer is cached. Two failure modes matter. First, the service is down. The API server treats an error as a deny, so a dead webhook is a cluster that cannot authorize anything, and every deploy stops. That is fail closed, and it is the safe failure, but it is still a failure, so the webhook needs the same uptime as the API server. Second, the cache is stale. An allowed answer cached for five minutes is a permission that lives five minutes after the policy changed. A denied answer cached for thirty seconds is a fix that takes thirty seconds to land. Set the TTLs short, and never cache a deny longer than you can wait for a fix.

ABAC, as legacy. ABAC is the authorizer that came before RBAC. The policy is a file of JSON lines, each one naming a user, a group, a resource, and a verb, and the API server reads it at startup with --authorization-mode=ABAC and --authorization-policy-file. It was replaced because the policy file needs an API server restart to change, it has no grouping, and a file with a thousand lines is a policy nobody can read. One paragraph is all it gets here, because nobody should write a new ABAC policy. If you find one running, the migration is to RBAC, one Role at a time, and the audit log tells you which lines are actually used.

Do this: write the aggregated ClusterRole from this tier, apply it on your own cluster, and add a second labeled role. Read the deployer ClusterRole back and watch its rules grow without an edit to the role itself. Then delete the label and watch the rules shrink.

Try it: Advanced

I label a new ClusterRole aggregate-to-admin. What happens to every admin?

Now do it: On your own cluster, label a test role aggregate-to-admin, then check that an admin gains its rules without any edit to the admin role.

Master · Audit as policy, and least privilege worked

At the master level the log stops being a file and becomes a policy. Lesson 158 taught the audit log as a concept: the record of every request the API server handled, with a pipeline behind it. This lesson teaches the policy that decides what the record holds.

Audit levels. Four levels, and each one costs more than the last. None records nothing. Metadata records who made the request, when, what they asked for, and whether it was allowed. Request records the headers too. RequestResponse records the full request and response bodies. The rule I use is simple. Secrets get RequestResponse, because the body shows what was asked for and what came back. Everything else gets Metadata, because who, when, and what is enough, and the bodies of a thousand Pod reads are a storage bill, not a record. A policy that logs everything at RequestResponse is a policy that drowns the one record you need in ten thousand you do not.

What it recordsWhat it costsUse it for
NoneNothingNothingNoisy system reads you have already decided to ignore
MetadataWho, when, what, allowed or deniedSmallMost requests, the default
RequestMetadata plus headersMediumRequests where the headers carry the story
RequestResponseRequest plus full bodiesLargeSecret reads, and nothing else

The audit policy file. The policy is a file the API server reads at startup, set with --audit-policy-file, and the log goes where --audit-log-path says. The rules run in order, and the first match wins, so the specific rules go first and the catch-all goes last. This policy records the full body of every Secret read, the metadata of every Pod read, and nothing from the default service account in kube-system, because that account is the noisiest reader in the cluster.

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["secrets"]
  - level: Metadata
    resources:
      - group: ""
        resources: ["pods"]
  - level: None
    users: ["system:serviceaccount:kube-system:default"]

What the snippet means: the first rule matches Secret reads and records them at RequestResponse. The second rule matches Pod reads and records them at Metadata. The third rule matches the noisiest system account and records nothing. A request that matches no rule is recorded at the default level, which is Metadata unless the policy says otherwise. This file parses as YAML and was checked in Python on Nov 21, 2026. kubeconform carries no schema for the audit Policy kind, so it is not schema-validated, and the lesson says so plainly.

Least privilege, worked three times. Lesson 53 teaches least privilege as a discipline: the smallest permission that works, one service, one secret, read only, one day. These are the Kubernetes versions of that discipline, and each one is a Role you can apply. First, the CI deployer, from the lab: create and update Deployments, read pods, in the shop namespace, and nothing else. The service account that runs the pipeline gets that binding, and the pipeline cannot read Secrets, cannot touch other namespaces, and cannot delete anything. Second, the debug-only role: read pods and pod logs, and create an exec session, in one namespace, for one incident, and then the binding is deleted. The exec verb is create on pods/exec, and it is the most powerful verb in this list, because an exec session is a shell inside a running pod. I grant it for the incident and revoke it when the incident closes.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: debugger
  namespace: shop
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create"]

What the snippet means: the debugger may read pods and logs, and may open an exec session, in the shop namespace only. No Secrets, no delete, no other namespace. The binding for this Role is temporary by design, and the deletion of the binding is part of the incident, not an afterthought.

Third, the namespace reader: read pods, services, ConfigMaps, and Deployments, in one namespace, for the person who watches the namespace but never changes it. No Secrets, no create, no update, no delete. The reader can see the shape of the system and cannot change any of it.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: namespace-reader
  namespace: shop
rules:
  - apiGroups: [""]
    resources: ["pods", "services", "configmaps"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch"]

What the snippet means: the reader sees the workloads and the config, and sees no Secrets at all. A reader who can read Secrets is a reader who can take the database password home, and that is a permission nobody needs to watch a namespace. All three roles were checked against the Kubernetes 1.34 schemas with kubeconform, strict mode, on Nov 21, 2026.

Do this: pick one service account on your own cluster, read its bindings, and write down every verb it holds that the job does not need. Then cut one, and watch the job keep working. The permission you cut and nobody missed is the permission that should never have been there.

Try it: Master

I write the audit policy for my cluster. Which level do I give Secret reads, and which level do I give everything else?

Now do it: On your own cluster, read the audit policy file and find the rule that matches Secret reads first, before the catch-all.

Go deeper

kubeconfig merging

The KUBECONFIG variable can list several files, separated by colons, and kubectl merges them. The first file wins on a conflict, and the current-context comes from the first file that names one. A context that appears in two files is the first file's version. The merge is why a context can look right in one file and behave differently at runtime: another file earlier in the list already defined it. kubectl config view --flatten shows the merged file, and it is the view I trust when the context misbehaves.

Impersonation, --as and --as-group

A cluster admin can act as another user with kubectl --as mohan --as-group shop-team, and the request is judged as mohan, not as the admin. The admin needs the impersonate permission to do it, and the audit log records both identities: who asked, and who they asked as. I use it to test a binding before I hand it to a person, because a binding that works for me may fail for them, and the failure is cheaper to find in a test than in a deploy.

Exec credential plugins

Some clusters hand out credentials through a plugin, not a file. The kubeconfig names an exec command, kubectl runs it, and the command prints a credential. OIDC login flows work this way, and so do cloud CLIs that mint a token on demand. The token the plugin prints is short-lived, and kubectl runs the plugin again when it expires. A plugin that prints a static token is a static token with extra steps, and the expiry is what makes it worth having.

OIDC groups, and the mapping that breaks

The groups claim is where OIDC bindings break most often. The API server reads the claim named by --oidc-groups-claim, and the RoleBinding names the group it expects. If the provider sends the groups under a different claim name, the user authenticates fine and every binding misses, because the identity carries no groups at all. The fix is the flag, not the binding. I check the groups in the ID token first, with the token decoded, before I touch any Role.

TokenReview, the webhook authn call

The authentication webhook receives a TokenReview: the token, and nothing else. It answers with the username, the groups, and whether the token is authenticated. It never sees the request, only the token. That is the difference from the authorization webhook, which sees the whole request. A TokenReview service that logs every token it sees is a service that logs credentials, and the logs of that service are a credential store. Keep them short, keep them few, and keep them out of the shared log bucket.

NodeRestriction, the admission half of Node authorization

Node authorization decides what a kubelet may ask for. NodeRestriction decides what a kubelet may change. It is an admission plugin, and it refuses any write to a Node or Pod object that does not belong to the kubelet's own node. The two together are the boundary: the kubelet can read its own pods and update its own node, and nothing else. Turn NodeRestriction off and Node authorization alone is a gate with no lock on the far side.

Audit backends: log file or webhook

The audit policy decides what is recorded. The backend decides where the record goes. The log backend writes to a file on the API server, set with --audit-log-path, and the file needs rotation, set with --audit-log-maxage and --audit-log-maxsize, or it grows until the disk fills. The webhook backend sends each event to a service, and the service stores it. The webhook backend can batch, and it can block the request until the event is written, which is the safe setting and the slow one. A log file that fills the disk stops the API server. A webhook that is down blocks the request. Pick the failure you can live with, and monitor the one you picked.

ValidatingAdmissionPolicy, named only

Kubernetes ships a built-in CEL admission mechanism, ValidatingAdmissionPolicy, which writes simple policies without a webhook and without Rego. This lesson does not teach it. It is named here so you know it exists when a policy is too small for a webhook and too big for a namespace label.

What changes at 10x and 100x

At 10x the namespaces, the bindings are the first thing that breaks, and the fix is the bindings in Git, reviewed like code, not set by hand. At 10x the service accounts, the static tokens are the first thing that breaks, and the fix is bound tokens with an audience, because a hundred static tokens is a hundred credentials to rotate. At 100x, the audit log is the product. The policy is code, the retention is a budget, and the alert on a service account used from outside the cluster fires before the exfiltration finishes. The gates do not change. The number of identities they judge does.

Further reading

The gate chain this lesson plugs into: Lesson 158 Kubernetes Security. The general auth protocols this lesson links to: Lesson 52 Proving Who You Are, Then What You May Do. The Vault half of service account trust: Lesson 156 Vault & Secrets Management. The components that run the front door: Lesson 94 Kubernetes Components. The last gate in the chain: Lesson 159 Kubernetes Pod Security in Depth. The discipline behind the examples: Lesson 53 Least Privilege and Secrets. The incident shape, told from the detection side: secops.html.

One time it broke

What broke

A service account token was copied out of a pod and used from outside the cluster. The token was a static one, minted for a debug pod that had long since been deleted, and it lived in a Secret nobody had cleaned up. The account it belonged to had a binding that allowed reading Secrets in the shop namespace, because the debug pod had needed it once, and the binding had never been removed. The token worked from a laptop outside the cluster, because a static token carries no audience and no expiry, and the API server cannot tell where a token is used from. The requests passed authentication, because the token was real. They passed authorization, because the binding was real. They were recorded in the audit log, and nobody read the log until the Secrets were already gone.

Timeline

  • Day 0A debug pod is created with a static service account token, to read a Secret during an incident. The pod is deleted when the incident closes. The Secret holding the token is not deleted.
  • Day 9The token is copied out of the cluster in a support bundle, pasted into a shared chat thread by mistake, and read by someone outside the team.
  • Day 9, 21:40The first request arrives from outside the cluster: a Secret read in the shop namespace, authenticated as the debug service account.
  • Day 9, 21:52Eleven more Secret reads follow, one every minute, all from the same outside address, all as the same service account.
  • Day 10, 09:00An engineer reading the audit log for an unrelated deploy notices the service account name next to an address that is not a cluster node, and raises it.
  • Day 10, 09:20The RoleBinding is deleted, and the token stops working at once. The Secret holding the token is deleted next.
  • Day 10, 10:00Every Secret the account could read is rotated, because a read is a copy, and a copy cannot be unmade.
  • Day 11An alert is added: any request authenticated as a service account from outside the cluster pages the on-call engineer.
  • Day 12Static tokens are removed from every namespace. Pods that need the API server get projected bound tokens with an audience and a one-hour expiry.

What changed after

Three things. First, no static service account tokens remain anywhere. Every pod that talks to the API server uses a projected bound token, with an audience and a short expiry, and the token dies with the pod. Second, debug bindings are temporary. The debug Role exists, the binding is created for the incident and deleted when it closes, and the deletion is part of the incident record, not a cleanup ticket. Third, the audit log pages now. A service account used from outside the cluster is an alert, not a log line, because a record nobody reads is a record that only helps after the damage. The gates did nothing wrong in this incident. The token was real and the binding was real, and both gates did exactly what they were told. The lesson is that a credential that lives forever and a binding that nobody removes are the same failure, and the fix is an expiry on one and a deletion on the other.

If you remember nothing else

Design exercise

Design the authentication and authorization for a cluster with 50 services across 10 namespaces. Each team deploys its own services. A CI pipeline deploys them. Engineers debug them. Pods need to read cloud storage, and the cloud must never see a static key. Every read of a Secret must leave a record with the full request body. A service account used from outside the cluster must page the on-call engineer. The question is the front door: sketch the identities, the bindings, the tokens, the workload identity, and the audit policy, and say what you would measure.

Clarifying questions to ask

  • Who logs in as a human, and who runs as a service account? A human who shares a service account is a human whose actions cannot be traced to a person.
  • Which identity provider do the humans use, and what do the groups look like in the ID token? A group that arrives under the wrong claim name is a binding that never matches.
  • What may the CI pipeline do, and in which namespaces? A pipeline that can deploy anywhere is a pipeline that can deploy a bad pod everywhere.
  • How long may a debug session last, and who deletes the binding when it ends? A debug binding with no expiry is a permanent permission with a temporary excuse.
  • Which cloud services do the pods call, and what does each one need? A pod that can read every bucket is a pod that can read the wrong bucket.
  • What happens when the authorization webhook is down: does the cluster fail closed and block deploys, or fail open and let them through?
  • How long is the audit log kept, and who reads it? A log that nobody reads is a record that only helps after the damage.

Worked approach

Start with the humans. OIDC for every human, through the company identity provider, with the username from the email claim and the groups from the groups claim. No client certificates for humans, because a certificate that lives for a year is a credential that lives for a year after the person leaves. No static tokens for humans, for the same reason. The kubeconfig each engineer carries names their context, and the context is checked with auth whoami on the first day.

Then the service accounts. One account per service, in the namespace the service runs in. The CI pipeline gets one account per namespace, bound to the deployer Role from this lesson, and nothing else. The pipeline cannot read Secrets, cannot touch other namespaces, and cannot delete anything. A debug account gets the debugger Role, bound for the incident and deleted when it closes, and the deletion is part of the incident record.

Then the tokens. Every pod that talks to the API server gets a projected bound token, with an audience and a one-hour expiry. No static tokens anywhere, and a scan of the Secrets in every namespace proves it. The automount is off for every pod that never calls the API server, because a token the pod never uses is a token an attacker can still steal.

Then the cloud. Workload identity for every cloud call, one federation per cloud, no static keys. The pod presents its bound token, the cloud checks the issuer, and the cloud hands back short-lived credentials for the one bucket the pod needs. A pod that needs a different bucket gets a different binding, not a broader one.

Then the audit policy. RequestResponse for Secret reads, Metadata for everything else, None for the noisiest system readers. The policy is a file in Git, reviewed like code. An alert fires when a service account is used from outside the cluster, and the alert pages, because a record nobody reads is a record that only helps after the damage.

Trade-offs to name. OIDC buys one login and costs a provider that must stay up, and a provider that is down is a cluster nobody can log into. Bound tokens buy a short life and cost a refresh, and a refresh that fails is a pod that loses its identity mid-run. Workload identity buys no static keys and costs a federation per cloud, and a federation that breaks is a pod that cannot reach its bucket. The audit policy buys the record and costs the storage, and a policy that logs everything at RequestResponse is a bill, not a record. And the target is the front door, not the speed. A request that is fast and unauthenticated fails this design, and the door is where the security lives.

Saved in this browser, so the Lessons index can show your progress across the course.

Next in this series · Lesson 163
Service Design from Scratch: Requirements, NFRs, and the Design Brief

Every service I have designed started as one sentence from someone who did not know it was a design problem. "Customers keep calling to ask where their parcel is. Can we show them?" That sentence is not a design. It is an ask.

Go to Lesson 163 →

Or browse the full course from the Lessons index.