Lesson 84 taught you how to create a Secret and a ConfigMap and hand them to a pod. This lesson teaches you the system behind them: the five kinds of Secret, the choice between an env var and a file, the encryption that protects etcd, and the six ways to configure a workload. A Secret is not a password in a file. It is an object with a type, a namespace, and a set of readers. A ConfigMap is not a text file. It is the cluster's own config system, and the pod reads it the way the cluster says it may. I learned this the slow way. I put a password in an env var, saw it in a log, and spent the next week rotating it.
Difficulty: Advanced · Know this first: Lesson 84, Lesson 158, Lesson 156
By the end of this lesson you will have created all five kinds of Secret on your own cluster, mounted one as files and read one as an env var, and watched a ConfigMap edit reach the mounted file while the env var stays frozen. You will have locked an object immutable and read the rejection when you tried to edit it, generated config with Kustomize and watched the name change on every edit, and written a Role that lets one person read one Secret and nothing else. You will know when a value belongs in an env var and when it belongs in a file, and you will know which of the six ways to configure a workload fits the change you are about to make.
What you need: a cluster you can run pods on, with kubectl. The lab steps run on your own cluster, and every expected output is described as what to look for, not quoted as a transcript. I did not run the lab on a cluster here. This VM hands out no cpu controller in its cgroup, so no k3s server can start on it. The same wall stopped the Phase 9 lab work, and I did not re-attempt the install. No kubectl output from this machine appears in this lesson, because there is no cluster on this machine to produce it. Your numbers will differ. The shape will not.
Every workload needs two kinds of input. The first kind is config: the log level, the feature flag, the cache size. It is not sensitive, and it changes often. The second kind is secrets: the password, the TLS key, the registry credential. It is sensitive, and it changes rarely. Kubernetes gives each kind its own object. The ConfigMap holds the config. The Secret holds the secrets. The two objects look alike on the surface. They are created the same way, read the same way, and mounted the same way. The difference is not the shape. The difference is the promise: a Secret is stored encrypted in etcd, read only by the accounts you name, and never printed in plain text by a command that lists it. A ConfigMap promises none of that, because it does not need to.
| Type | What it holds | What breaks if you get it wrong |
|---|---|---|
| Opaque | Any key and value pair. The default when you name no type. | Nothing breaks. It is the plain box, and every other type is a box with a shape the cluster checks. |
| kubernetes.io/tls | A certificate and a key, under the keys tls.crt and tls.key. | An expired cert. The pod still runs, the handshake fails, and the clients reject it. |
| kubernetes.io/dockerconfigjson | A registry credential, built from a docker login, under the key .dockerconfigjson. | A wrong password, or a credential in the wrong namespace. The kubelet cannot pull the image, and the pod waits. |
| kubernetes.io/service-account-token | A token for one service account, created by the controller, not by you. | A token that outlives the account it belongs to. Delete the account and the token dies with it. |
| kubernetes.io/basic-auth | A username and a password, under the keys username and password. | A client that reads the wrong key name. The shape is fixed, and a typo in the key is a failed login. |
Opaque is the type you reach for first, and it is the type Lesson 84 showed you as expected output: one row, type Opaque, never taught as a type. The other four types are the same object with a shape the cluster enforces. A TLS Secret with no tls.crt key is rejected at creation, not at use. A basic-auth Secret with no username key is rejected the same way. The type is a check that runs early, when the mistake is cheap. An Opaque Secret has no shape to check, so a typo in the key sails through creation and fails at 2 AM, when the pod reads a key that is not there.
A Secret lives in one namespace, and a pod in another namespace cannot read it. That is the first boundary, and it is the one people forget. A ConfigMap has the same boundary. The second boundary is etcd, the cluster's own database. The Secret is stored there, and the API server is the only way in. The third boundary is the readers: a Role that grants secrets get, and the accounts the RoleBinding names. A Secret with no Role behind it is readable by nobody except the cluster admin, and that is the default you want. The lab writes a Role that lets one person read one Secret, and that Role is the whole lesson in one file.
On your own cluster, run kubectl get secrets --all-namespaces and kubectl get configmaps --all-namespaces. Write down which secrets exist, which namespace each one lives in, and which type each one carries. A secret in the wrong namespace is a pod that cannot pull its image, and a secret with no type is an Opaque box you will have to open to understand. That list is the config map, measured before you learn it.
The diagram above follows one secret and one ConfigMap from creation to a running pod. These four paths are the same idea drawn four ways. One clean config. One expired cert, stopped at the handshake. One stale env var, fixed by a rollout. One numbers trail, hop by hop. Run each path. One run takes 7.2 seconds across the path, so you can watch each hop light up in turn.
The return rule: the green dot only starts moving after the orange dot reaches the data hop. A return with no finished work is a lie. The diagram never draws one. The failure replay at the end of Run all four shows the other side of the rule. Run it and watch the orange dot stop at the expired cert and never come back.
One secret, one ConfigMap, one pod. The secret is created, the ConfigMap is created, etcd stores both, the pod asks for them, the kubelet mounts them, and the pod runs in 30 ms. A ConfigMap edit arrives, and the mounted file picks it up. The green dot starts back from the Pod runs hop, because that is where the finished work lives.
Not run yet.
Same deploy, one difference: the TLS Secret holds a cert that expired yesterday. The pod still starts, because the kubelet does not check the date on the cert. The first client that connects gets a handshake failure, and the pod serves nothing. No work is done after the failure, so no green dot ever starts back.
Not run yet.
Same deploy, one difference: the pod reads the ConfigMap as env vars, not as files. A ConfigMap edit arrives, and the pod keeps the old value, because env vars are frozen at Pod start. The checksum annotation changes, the Deployment rolls the pods, and the new pods start with the new value. The green dot starts back from the Rollout restarts hop, because the rollout is the finished work here.
Not run yet.
The same config, measured instead of timed. Five secret types, six ways to configure, two consumption paths, one decision table. The trail adds up to 65 ms, and the types are the check that keeps the secret honest.
Not run yet.
Every hop charges rent in milliseconds. The bar below is a worked example for one secret and one ConfigMap reaching a pod, not a measurement from a live system, but the shape is honest. The pod start is the big hop at 30 ms, and the total is 65 ms. Switch the scenario and watch the bar change. In the expired story the Value changes hop grows to 15 ms, because the handshake fails and the pod serves nothing after it. In the stale story the Pods catch up hop grows to 40 ms, because the rollout restarts every pod. Move a simulator slider and watch the pickup, the rollout, and the total move with it.
The budget bar above shows three fixed stories. This simulator lets you write your own. It answers the sizing question at the heart of this lesson: which pods pick up an edit, which ones need a rollout, and which ones fail. You set what changed, how the pod consumes it, whether the object is immutable, what reload mechanism exists, and whether etcd encrypts at rest. Move a slider and watch the risk score, the pickup verdict, the rollout verdict, and the budget total change. The numbers are a teaching example, built from the formulas below, not live measurements. The shape is the lesson: a mounted file updates, an env var freezes, an immutable object refuses, and a rollout is the only way an env var ever changes.
What changed. No change means the pods already run the current values. A ConfigMap edit changes a flag. A Secret rotation changes a password.
How the pod reads the value. Env vars are simple, frozen at Pod start, and visible in describe. A volume mount is files, updatable, and the right home for certs and multi-line values.
Whether the object refuses edits after creation. True locks it: any edit is rejected at apply. False lets anyone with write access change it in place.
What restarts the pods when an env var changes. None means the pods stay stale until a human restarts them. A reloader sidecar watches and restarts. A checksum annotation changes the Pod template, so the Deployment rolls.
Whether the API server encrypts Secrets before writing them to etcd. On means the KMS provider encrypts them. Off means they sit in plain text in the cluster database.
Try this: (1) At the defaults, no change, volume mount, immutable true, checksum annotation, etcd encryption on, the risk is 0, nothing needs a rollout, and the budget total is 65 ms. (2) Set immutable to false and edit the ConfigMap. The mounted file picks it up in 68 ms, the risk is 20, and the status line says picked up without a restart. (3) Keep the edit and switch the pod to env vars. The checksum annotation changes the Pod template, the Deployment rolls the pods, and the budget total is 108 ms. The risk is 45. (4) Keep the env vars and set the reload mechanism to none. The pods keep the old value forever, the budget total is 68 ms, and the status line says stale until someone restarts them. The risk is 60. (5) Rotate the Secret with immutable back on. The edit is rejected at apply in 21 ms, the risk is 40, and the status line says immutable. The old password keeps working, and that is the point of the lock. (6) Back at the defaults, turn etcd encryption off. The risk is 40, the budget total stays 65 ms, and the status line says the secrets sit in plain text in etcd.
Formulas, in plain terms. Risk is etcd encryption off 40 plus env vars 25 plus mutable 20 plus no reload mechanism 15. The most a setup can score is 100. A change to an immutable object is rejected at apply in 21 ms: 6 ms to create, 4 ms to store, 8 ms in etcd, and 3 ms for the rejected edit. A change picked up by a volume mount costs 68 ms: the clean 65 ms plus 3 ms for the edit. A change that needs a rollout costs 108 ms: the clean 65 ms plus 3 ms for the edit plus 40 ms for the rollout. A change with no reload mechanism costs 68 ms and the pods stay stale, because nothing restarts them. No change costs 65 ms. An env var never updates on its own, and a volume mount never needs a rollout.
Three steps, all run on your own cluster. There is no kubectl output from this machine in this lesson, because there is no cluster on this machine. The k3s install was not re-attempted here: this VM hands out no cpu controller in its cgroup, the same wall Phase 9 hit, so no k3s server can start here. What follows is what to run and what to look for, not a transcript. Your output will differ. The shape will not.
Save these five files. The Opaque Secret uses stringData, so you write the password in plain text and the API server encodes it for you. The TLS Secret carries a real self-signed certificate and key, generated with openssl on Nov 21, 2026 for the name shop.internal, valid one year. The dockerconfigjson Secret carries a registry credential for registry.internal.example. The basic-auth Secret carries a username and a password. The service-account-token Secret names the shop-api account in an annotation, and the controller fills in the token. All eight core files in this lab were checked against the Kubernetes 1.34 schemas with kubeconform, strict mode, on Nov 21, 2026. They parse and validate. They have not been run against a live API server from this machine.
apiVersion: v1 kind: Secret metadata: name: shop-api-password namespace: shop type: Opaque stringData: password: s3cret-password
apiVersion: v1 kind: Secret metadata: name: shop-tls namespace: shop type: kubernetes.io/tls data: tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURMekNDQWhlZ0F3SUJBZ0lVRm13UnNJblNPOEQ4dFZ4elRZbldzQlcxZ09Vd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0p6RVdNQlFHQTFVRUF3d05jMmh2Y0M1cGJuUmxjbTVoYkRFTk1Bc0dBMVVFQ2d3RWMyaHZjREFlRncweQpOakV4TWpFeU1UQTROVE5hRncweU56RXhNakV5TVRBNE5UTmFNQ2N4RmpBVUJnTlZCQU1NRFhOb2IzQXVhVzUwClpYSnVZV3d4RFRBTEJnTlZCQW9NQkhOb2IzQXdnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUsKQW9JQkFRQ3FNQU5PeFliZ2dlcENwQUs3RXIvODVianZKeFVBTFcyQzZVa1Q5Wmdaa3JzajhhVzNSYnZMTnNhUApMT1daanoyUHVsa3hPcElSN3VKaGhiSUE5VUNtM0F3Qi84aWh6UjdRZ3Zrdkx6QVZtVFJZN01wak1DcWVoekRxCk0xMTh0dmJ3Vjl0ZFNXaFg3azY3dm1RcDdSbmUydVVlZ2hWa1h0YU83Y3pQQ2lmZzN3ZkxHazhJUjNBMWpwTFMKWGhkWXFyMFZWRVNwbHQxQ0NsUTVhVDJyZ2dpTWMxR1R6SnlxeTMvbWIvOStIeWhoYkxwb0dvZFcrZzRzcHNxSwpzZXkreXFOd082c0tnVzczaG9ZRFo1c0JLL3hGdy9IZkFuZ3RPRTBtdDZ4VVFJUUlvTmo0RUdYcmw5K1luS2VhCnM4TG1xM1ZSYzE1aVBEamFxNWxSQ2hqeXVpSDNBZ01CQUFHalV6QlJNQjBHQTFVZERnUVdCQlJFQWl2UjAyVG0KYVIzbzhhWTFtbDRRQlpKNk1qQWZCZ05WSFNNRUdEQVdnQlJFQWl2UjAyVG1hUjNvOGFZMW1sNFFCWko2TWpBUApCZ05WSFJNQkFmOEVCVEFEQVFIL01BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQmR0RWRSeGtQK2R0YWc3SzNXCnhaRlZXclNLMDVST0k3UksrWFlyK2tDU2pQZTFjYzFjRGdhQlliWDRjYzZtMlNGNWZYeUlWNVNndGdSNHlmbmUKdXhZNUh0S3hQR0RzSU54YXdIWFVRZFZPOVMreCs3Mmh3RkM5LzRkU0YwRXI5cXIrajJWVGh6Q1czTVU3a0RtUQpnMEJPUERVektBNFZScEo2UUFXMXJaSnZmWmtLbWEzUEN1SUNJSlQxMTdoVTZEN3NtVnM0S1FvUmxjaUhkdUp1CmFNV21xTE5CcWl1TTBTNGs4eDJRQjhVVzN2RkRYanU2T3FMQkJRVkd3aUFWaGQ2UldlUExYMUdQU3hkeVk0eU4KQXMwTFlIZGk4bXc3VUhjZ212TzU4L0NDRnJaY01SRUhhaXNBUlJuWXpKQk83SThNcjNZZ1YyMTBmWlRqb2hHegpwRHN1Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRQ3FNQU5PeFliZ2dlcEMKcEFLN0VyLzg1Ymp2SnhVQUxXMkM2VWtUOVpnWmtyc2o4YVczUmJ2TE5zYVBMT1daanoyUHVsa3hPcElSN3VKaApoYklBOVVDbTNBd0IvOGloelI3UWd2a3ZMekFWbVRSWTdNcGpNQ3FlaHpEcU0xMTh0dmJ3Vjl0ZFNXaFg3azY3CnZtUXA3Um5lMnVVZWdoVmtYdGFPN2N6UENpZmczd2ZMR2s4SVIzQTFqcExTWGhkWXFyMFZWRVNwbHQxQ0NsUTUKYVQycmdnaU1jMUdUekp5cXkzL21iLzkrSHloaGJMcG9Hb2RXK2c0c3BzcUtzZXkreXFOd082c0tnVzczaG9ZRApaNXNCSy94RncvSGZBbmd0T0UwbXQ2eFVRSVFJb05qNEVHWHJsOStZbktlYXM4TG1xM1ZSYzE1aVBEamFxNWxSCkNoanl1aUgzQWdNQkFBRUNnZ0VBQi85OWdFMzd0TzZmTngxQVBIVzQ1Kzk2L2F2MlR3M0JNMnFURjJqM3J0N1kKRnNlUTFGOHZ6aFZwMlUzNXh1ZWZNL05YTjB6N2t4aWJhZm03QWxQTmNSZkJTUDhOWnRFNWV0eG5LV0luaTlqZwpJeURiK2xnUTdtWXNPVCs1SnhidDNkZU5jb3IwUThHaEdjQ2plMjNKb0lnSklYMk4rRDNNZDdCa1JycFRod1ZRCm1sM2hJeVBvY21qbldmUmdZRmF5di9nd0V0eC94cTN0ZXNDSEpYUzVXNjR5dGd2bVhaUThSK2VVdWpFMTMzZ0kKMHVzemlieE1UWERCdWIycHBoTCszK1N1a3NsWXZONCtPeFMwdkZVWWVpRVQ4bExXcmplMzRwYlkzV1IrU201WApLV3RoVkhSYXcwckptWkJHWVJJaHc3UW1qVWk4NElSMFRLcU5uV2g4elFLQmdRRGNuNHBqVkJjeHdlRmtjb2hOCmUwWTQvb3lBT25BMWdZQ3ZQZ01MMnhEK21mLzlvbk1vSmR5VkhPeTNINENzZ2E0VUpIUkdJQ0RObCt6MldSYjkKbXdXWk05cjFBcTkrdjA3bHNwVFJiUFVBaWlraEI1aUdZZlhYS1NmdVVGNHJhU2wyYk5QM3ByM2VwN3BlRmQreQpVK25JdWkzUFNzS1VySzlpV2dwZUZBb3d6UUtCZ1FERmVoM3F3SlpURFFCbG9XV2NLVWtwbXkwTXlQZDVVOGlVCjlmWFNDZmgxOWtOclBrSy9UZUJnaG5LS2hYYm9jU09ieWd0bForY0JXV3ByTndSb3dpdVd6NVVhT0pUemJQSW0KQTl4UjNHWlJTUE1jZ1cxdlprN3dJRFZyYnp6ckorSVNkSUx1TGNWUmx2SDRVWGNtQ3Vqb2Q4bmkrRTRtbm94TQpiMktTQ3orTjB3S0JnUURYOE1neEpKTjNQajl3WTdJSnRJa3RLK2sxY1AxM1FZUkZBanRWRlZ4eTl5VGFpbkNRCmNVNlFITG1ORzZhRU1vSENjVlBsZDNqa0Fyb0svWFJGVWNDUERYUllzQUk1MnVKTWJSakFTeGNiS2gvTGxuV3gKd3B3dnRyYkF5YVVDZEhJQzJjNDI3ZHVWM1RISytUbjBmN0dwcUtGQjdmaTZNTE9SWG5RYlZSNE1XUUtCZ0d0aQppcDQ3ZUFOVmtKK3pjb1BHMTZndVlBTVJ6YkZHWkU3TXFZNDhLY1pMMGtHaEs3N0g1SG5ZNHJSOCt6cFBPK1BkCnJQREozWTZZUnd4aWNKMWU2VUVyUkpOcnJXUHFISXVia2RWYUdyR2REOGF5Z3dZUEJtSUJ3Y3JZUkxKRGorWWwKUVlXdmtodGFEOTdPdmhJTDRMZDVYR2wyU1ZrcTRuUTJKYzN4Ry9CTkFvR0FZU2pvUWZhNDRwN0dXRzNVbWJ2dQo2RHlyL0tQU0w5ZVhwM0JBZFdzb2p5K3N3U0o2dmZaVzNYcU5tTDlvWGJzMVNLWEE1aFJjUi9XdlVTNEdkV1l0CndYeUpETWtpTkNPZ0pxbE80SDJzVDF4VzVDcExrd2IzL3BtWlV6TEtmbC9IS3VjMC9iMSsxQWpVKzVWcHUvdEQKTE1qYXUwUWd3bGltdkhPS2ZDRlZrMFE9Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
apiVersion: v1 kind: Secret metadata: name: registry-creds namespace: shop type: kubernetes.io/dockerconfigjson data: .dockerconfigjson: eyJhdXRocyI6eyJyZWdpc3RyeS5pbnRlcm5hbC5leGFtcGxlIjp7InVzZXJuYW1lIjoic2hvcC1jaSIsInBhc3N3b3JkIjoiczNjcmV0LXBhc3N3b3JkIiwiYXV0aCI6ImMyaHZjQzFqYVRwek0yTnlaWFF0Y0dGemMzZHZjbVE9In19fQ==
apiVersion: v1 kind: Secret metadata: name: shop-basic-auth namespace: shop type: kubernetes.io/basic-auth data: username: c2hvcC1jaQ== password: czNjcmV0LXBhc3N3b3Jk
apiVersion: v1
kind: Secret
metadata:
name: shop-api-token
namespace: shop
annotations:
kubernetes.io/service-account.name: shop-api
type: kubernetes.io/service-account-token
Apply them and read them back:
kubectl apply -f secret-opaque.yaml kubectl apply -f secret-tls.yaml kubectl apply -f secret-dockerconfig.yaml kubectl apply -f secret-basic-auth.yaml kubectl apply -f secret-sa-token.yaml kubectl get secrets -n shop
What to look for: five rows, one per type, each in the shop namespace. Read the Opaque Secret back and decode the password yourself, the way Lesson 84 taught: the value is base64, and base64 is not encryption. Run kubectl get secret -n shop shop-api-password -o jsonpath='{.data.password}' | base64 -d and read the password back. Anyone who can read the Secret can decode it. The protection is the access, not the encoding. Then read the service-account-token Secret the same way and find the token the controller wrote in. You did not write that token. The controller did, because the annotation asked it to.
The registry credential has a third creation way worth knowing. The file above is the YAML way. The kubectl way builds the same Secret from a docker login, and it is the way you use when the credential already lives in a docker config file:
kubectl create secret docker-registry registry-creds \ --docker-server=registry.internal.example \ --docker-username=shop-ci \ --docker-password=s3cret-password \ --namespace=shop
What to look for: the command prints that the Secret was created, and kubectl get secret -n shop registry-creds -o jsonpath='{.type}' reads kubernetes.io/dockerconfigjson. The two ways make the same object. The YAML way fits Git. The kubectl way fits a login you already ran. The decision is the workflow, and the commands are in Lesson 84.
Save the ConfigMap, the pod, and the Role. The pod reads LOG_LEVEL as an env var from the ConfigMap, reads DB_PASSWORD as an env var from the Secret, mounts the TLS Secret as files under /etc/tls, and mounts the ConfigMap as files under /etc/config. It pulls its image with the registry-creds Secret. The Role lets one user, mohan, read one Secret, shop-api-password, and nothing else.
apiVersion: v1
kind: ConfigMap
metadata:
name: shop-config
namespace: shop
data:
LOG_LEVEL: INFO
FEATURE_FLAG: "on"
app.properties: |
log.level=INFO
feature.flag=on
cache.size=256
apiVersion: v1
kind: Pod
metadata:
name: shop-api
namespace: shop
labels:
app: shop-api
spec:
imagePullSecrets:
- name: registry-creds
containers:
- name: api
image: busybox:1.36
command: ["sh", "-c", "sleep 3600"]
env:
- name: LOG_LEVEL
valueFrom:
configMapKeyRef:
name: shop-config
key: LOG_LEVEL
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: shop-api-password
key: password
volumeMounts:
- name: tls
mountPath: /etc/tls
readOnly: true
- name: app-config
mountPath: /etc/config
readOnly: true
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 250m
memory: 256Mi
volumes:
- name: tls
secret:
secretName: shop-tls
- name: app-config
configMap:
name: shop-config
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: secret-reader
namespace: shop
rules:
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["shop-api-password"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: read-shop-secret
namespace: shop
subjects:
- kind: User
name: mohan
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: secret-reader
apiGroup: rbac.authorization.k8s.io
Apply them, then read the pod from the inside:
kubectl apply -f configmap.yaml kubectl apply -f pod-config.yaml kubectl apply -f secrets-role.yaml kubectl exec -n shop shop-api -- printenv LOG_LEVEL kubectl exec -n shop shop-api -- printenv DB_PASSWORD kubectl exec -n shop shop-api -- cat /etc/config/app.properties kubectl exec -n shop shop-api -- ls /etc/tls
What to look for: LOG_LEVEL prints INFO, DB_PASSWORD prints the password, app.properties prints the three lines from the ConfigMap, and /etc/tls lists tls.crt and tls.key. The env var and the file hold the same value, and the pod does not care which way it arrived. Now edit configmap.yaml, change LOG_LEVEL to DEBUG, apply it, and read both again:
kubectl apply -f configmap.yaml kubectl exec -n shop shop-api -- printenv LOG_LEVEL kubectl exec -n shop shop-api -- cat /etc/config/app.properties
What to look for: the file shows DEBUG, and the env var still shows INFO. The mounted file updated, because the kubelet syncs it. The env var did not, because env vars are frozen at Pod start. This is the choice the Intermediate tier teaches, and you have just measured it. The propagation mechanics behind the file update are in the P9 fold-in in Lesson 84. Now check the Role:
kubectl auth can-i get secret/shop-api-password --as=mohan -n shop kubectl auth can-i get secret/shop-tls --as=mohan -n shop kubectl auth can-i list secrets --as=mohan -n shop
What to look for: the first check says yes, and the other two say no. One Secret, one reader, one verb. A Role that grants secrets get with no resourceNames lets that reader read every Secret in the namespace, and that is the failure mode the Master tier names.
Save the immutable ConfigMap and the kustomization file. The immutable ConfigMap is the same shop-config with one field added, and the kustomization file generates both objects from literals, so their names carry a hash suffix.
apiVersion: v1
kind: ConfigMap
metadata:
name: shop-config-immutable
namespace: shop
immutable: true
data:
LOG_LEVEL: INFO
FEATURE_FLAG: "on"
app.properties: |
log.level=INFO
feature.flag=on
cache.size=256
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: shop
commonLabels:
app: shop-api
configMapGenerator:
- name: shop-config
literals:
- LOG_LEVEL=INFO
- FEATURE_FLAG=on
secretGenerator:
- name: shop-api-password
literals:
- password=s3cret-password
Apply the immutable ConfigMap, then try to edit it:
kubectl apply -f configmap-immutable.yaml
kubectl patch configmap shop-config-immutable -n shop --type=merge -p '{"data":{"LOG_LEVEL":"DEBUG"}}'
What to look for: the apply succeeds, and the patch is rejected with an error that names the immutable field. The object refuses the edit. The only way forward is to delete it and create it again, and that is what immutability buys: no silent in-place change, ever. Now generate the config with Kustomize and watch the names:
kustomize build . | kubectl apply -f - kubectl get configmaps -n shop kustomize build . | kubectl apply -f - kubectl get configmaps -n shop
What to look for: the first build creates a ConfigMap and a Secret whose names end in a hash, like shop-config-7d9f4b6c8d. The second build, after you change a literal in the kustomization file, creates new objects with a different hash, and the old ones stay until you delete them. The name changes on every edit, and that is the point: the Deployment that references the generated name rolls when the name changes, and a pod never runs with a half-updated config. Rotate the password the same way: change the literal, rebuild, apply, and the new Secret gets a new name. The rotation patterns behind that rollout, the reloader sidecar and the checksum annotation, are in the Master tier. Clean up when you are done:
kubectl delete pod -n shop shop-api kubectl delete secret -n shop shop-api-password shop-tls registry-creds shop-basic-auth shop-api-token kubectl delete configmap -n shop shop-config shop-config-immutable kubectl delete role -n shop secret-reader kubectl delete rolebinding -n shop read-shop-secret
What to look for: the shop namespace ends empty, and nothing you created in this lab keeps running. A tidy cluster is a choice you make after every experiment, not a state the cluster keeps for you.
One thing this lab does not do: apply the EncryptionConfiguration. That file lives on the API server, not in kubectl reach. You do not apply it. Your cluster admin sets it in the API server flags, and on a managed cluster you ask the provider what it encrypts. The file is in the Advanced tier so you can read it. To check your own cluster, ask the admin for the API server flags, or read the provider docs for the encryption setting, and write down which resources it covers. If the answer is "nothing", your Secrets sit in plain text in etcd, and the simulator above shows you what that costs.
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- kms:
name: shop-kms
endpoint: unix:///var/run/kms-plugin/kms.sock
cachesize: 100
- identity: {}
Three questions. No score is saved anywhere. This is just you against the material.
A secret, its readers, and its rotations are not three skills. They are four levels of the same job, and every team I have worked with sits on one of them. The levels are not about how many type names you can recite. They are about whether the settings change what gets built. Read them in order. Each level fixes the failure that hurt the most at the level below it.
At the basic level you write each type by hand, one at a time, and you read what the type checks. The full runnable files are in the lab. These are the pieces, each one on its own, with the shape that matters.
Opaque, the default. Any key and value pair, no shape to check. Write it with stringData and the API server encodes it for you. Write it with data and you encode it yourself, with base64. The two spellings make the same object.
type: Opaque stringData: password: s3cret-password
What the snippet means: the password is stored under the key password, and nothing checks the shape. A typo in the key sails through creation and fails when the pod reads a key that is not there. That is the price of the plain box.
TLS, the cert and the key. The keys are fixed: tls.crt and tls.key. The values are base64, and the file in the lab carries a real self-signed cert for shop.internal, generated with openssl on Nov 21, 2026, valid one year.
type: kubernetes.io/tls data: tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0t tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0t
What the snippet means: the cert and the key travel together, under names the cluster knows. A TLS Secret with no tls.crt key is rejected at creation. The check runs early, when the mistake is cheap. What breaks when the cert expires is in the Go deeper section: the pod still runs, and the handshake fails.
dockerconfigjson, the private-registry credential. One key, .dockerconfigjson, holding the whole docker config file as base64. It is built from a docker login, and the lab shows both ways: the YAML file, and the kubectl create secret docker-registry command that builds the same object from the flags.
type: kubernetes.io/dockerconfigjson data: .dockerconfigjson: eyJhdXRocyI6eyJyZWdpc3RyeS5pbnRlcm5hbC5leGFtcGxlIjp7fX19
What the snippet means: the credential is the login, encoded once, stored as one value. The kubelet reads it when it pulls the image. A credential in the wrong namespace is a pod that cannot pull, and the error says the image cannot be found, not that the credential is missing.
service-account-token, the token the controller writes. You do not write the token. You write the annotation that names the account, and the controller fills in the token. Delete the account and the token dies with it.
type: kubernetes.io/service-account-token
metadata:
annotations:
kubernetes.io/service-account.name: shop-api
What the snippet means: the Secret is a request, not a value. The controller sees the annotation, mints a token for that account, and writes it into the Secret. A token you mint by hand never expires on its own. A token the controller mints is bound to the account, and it dies when the account dies.
basic-auth, the username and the password. Two keys, fixed names: username and password. A client that reads the wrong key name gets a failed login, not a clear error.
type: kubernetes.io/basic-auth data: username: c2hvcC1jaQ== password: czNjcmV0LXBhc3N3b3Jk
What the snippet means: the shape is fixed, and the client reads it by key name. The values are base64, and the decode is one command. Anyone who can read the Secret can decode them, which is why the next tier is about who can read.
Creation, three ways, one decision. The commands live in Lesson 84, and this lesson does not repeat them. The decision is the teaching. Use kubectl create secret when the value already exists somewhere, like a password in your head or a credential in a docker login, and you want it in the cluster now. Use a YAML file when the Secret is part of a deploy, reviewed in Git like the rest of the manifest, with the warning that a Secret in a file is a secret in git history. Use --from-literal for a short value you type, and --from-file for a long value that already lives in a file, like a cert or a config blob. The workflow picks the way. The object is the same either way.
Do this: run step 1 of the lab on your own cluster and write down the type of each Secret you created. Then read the TLS Secret with kubectl get secret -n shop shop-tls -o yaml and find the two keys. Then decode the Opaque password with base64 and write down the plain text. That decode is the whole base64 lesson in one command.
Try it: Basic
I write a TLS Secret and leave out the tls.crt key. What happens at creation?
Now do it: I parsed a TLS Secret locally with Python and it checks out: the keys are exactly tls.crt and tls.key. On your cluster, run step 1 of the lab and write down the decode of one value.
At the intermediate level you stop asking how to consume a Secret and start asking which way fits the value. Lesson 84 taught both mechanisms: valueFrom with secretKeyRef reads one key into one env var, and a volume mount turns the keys into files. This lesson teaches the choice, and the lab measures it: edit the ConfigMap, and the file updates while the env var stays frozen.
| Env vars | Volume mounts | |
|---|---|---|
| Shape | One key, one variable, in the process environment | One key, one file, under a mount path |
| When it updates | Never. Frozen at Pod start. A new value needs a new pod. | When the kubelet syncs it. The file changes under the running pod. |
| Who can see it | Anyone who can run describe on the pod, and anyone who reads the logs if the app dumps its env | Anyone who can exec into the pod and read the file |
| Best for | Short flags and one-line values that rarely change | Certs, keys, and multi-line values like a properties file |
| Pick it when | The value is a flag, and a rollout on change is fine | The value is a file in the app's mind already, or it must update without a restart |
Env vars, the snippet. One key into one variable. The pod in the lab reads LOG_LEVEL from the ConfigMap and DB_PASSWORD from the Secret the same way, and the app cannot tell which object the value came from.
env:
- name: LOG_LEVEL
valueFrom:
configMapKeyRef:
name: shop-config
key: LOG_LEVEL
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: shop-api-password
key: password
What the snippet means: the value arrives in the process environment at start, and it stays there. A ConfigMap edit after start changes nothing. The pod runs with the value it was born with, and the only way to change it is to start a new pod.
Volume mounts, the snippet. The keys become files. The TLS Secret becomes tls.crt and tls.key under /etc/tls, and the ConfigMap becomes app.properties under /etc/config.
volumeMounts:
- name: tls
mountPath: /etc/tls
readOnly: true
- name: app-config
mountPath: /etc/config
readOnly: true
volumes:
- name: tls
secret:
secretName: shop-tls
- name: app-config
configMap:
name: shop-config
What the snippet means: the values arrive as files, and the kubelet keeps them in sync. Edit the ConfigMap, and the file under the running pod changes. The app that re-reads the file sees the new value. The app that read it once at start does not, and that is a property of the app, not of the mount. The propagation mechanics, the kubelet watch and the update delay, are in the P9 fold-in in Lesson 84. This lesson owns the choice. That fold-in owns the timing.
Base64, one paragraph. Base64 is not encryption. The protection is the access, not the encoding. Lesson 84 teaches it twice, in the Basic tier and the Intermediate tier, and the lab in this lesson makes you run the decode yourself. Anyone who can read the Secret can decode it in one command. The fix is never a better encoding. The fix is fewer readers, and the Advanced tier is where the readers get small.
Do this: run step 2 of the lab, the edit half, and write down the two readings: the file that changed and the env var that did not. Then pick one value in a pod you run and move it from an env var to a volume mount, or the other way, and write down why the new home fits better.
Try it: Intermediate
I edited a ConfigMap. The file under my running pod changed but the env var did not. What does that tell me?
Now do it: On your cluster, run step 2 of the lab, the edit half, and write down the two readings: the file that changed and the env var that did not.
At the advanced level the config system meets the cluster itself. Four hard parts, each one a failure you have not met yet.
etcd encryption at rest, first teaching. A Secret is stored in etcd, the cluster's own database, and by default it is stored in plain text there. Anyone who can read the etcd data can read every Secret. The fix is EncryptionConfiguration, a file the API server reads at start. It lists the resources to encrypt and the providers that encrypt them, in order. This lesson teaches it first. Lesson 158 names etcd encryption once, as one item in a summary list, and EncryptionConfiguration appears nowhere else in the course. Do not cite Lesson 158 as coverage. There is none.
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- kms:
name: shop-kms
endpoint: unix:///var/run/kms-plugin/kms.sock
cachesize: 100
- identity: {}
What the snippet means: the secrets resource is encrypted with the KMS provider first, and the identity provider second. The KMS provider sends the data key to a key management service and stores the encrypted key next to the data. The identity provider encrypts nothing, and it sits last as the fallback for resources the list does not cover. What is encrypted: every Secret, and any other resource you list. What is not encrypted: the values in a running pod, the values in a log, and the values in a backup of etcd taken before encryption was turned on. Turning encryption on does not re-encrypt the Secrets already stored. You re-encrypt them yourself, by reading every Secret and writing it back, and the Go deeper section has the runbook. Rotating the encryption key is the same job: point the provider at a new key, then read and write every Secret again, so each one is stored under the new key.
Image pull secrets, per namespace, per ServiceAccount. A private registry says no to a pull with no credential, and the pod waits in ImagePullBackOff. The credential is a dockerconfigjson Secret, and it lives in one namespace. A pod uses it in one of two ways: the pod names it in imagePullSecrets, or the ServiceAccount names it and every pod that runs as that account inherits it. The per-ServiceAccount way is the one that scales: set it once on the account, and every pod in the namespace pulls with it. The failure mode is the same either way, and it is quiet: the pull fails, the pod never starts, and the event says the image cannot be pulled. Read the Secret first, not the image name.
spec:
imagePullSecrets:
- name: registry-creds
What the snippet means: the pod pulls with the registry-creds Secret from its own namespace. A Secret in another namespace does not work, and the error will not tell you that. The fix is the credential in the right namespace, on the right account.
Immutable Secrets and ConfigMaps. Set immutable: true and the object refuses every edit after creation. What immutability buys: no silent in-place change, ever. A value that cannot change in place cannot drift, and a drift you cannot see is the drift that breaks a deploy at 2 AM. When it breaks a rollout: a Deployment that updates the ConfigMap in place and expects the pods to pick it up. The edit is rejected, the rollout never starts, and the error names the immutable field. The fix is a new object with a new name, which is exactly what the generators do next. The lab makes you read the rejection yourself.
immutable: true data: LOG_LEVEL: INFO
What the snippet means: the object is locked at creation. The lock is the feature. A lock that breaks a rollout is a rollout that was going to change config in place, and in-place change is the thing the lock exists to stop.
configMapGenerator and secretGenerator. Kustomize generates the objects from literals or files, and each generated object gets a name with a hash suffix. The hash changes on every edit, so the name changes on every edit. That is the point. A Deployment that references the generated name rolls when the name changes, and a pod never runs with a half-updated config. The full file is in the lab.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: shop
commonLabels:
app: shop-api
configMapGenerator:
- name: shop-config
literals:
- LOG_LEVEL=INFO
- FEATURE_FLAG=on
secretGenerator:
- name: shop-api-password
literals:
- password=s3cret-password
What the snippet means: two generators, one ConfigMap and one Secret, built from literals. The names in the file are the base names. The names in the cluster carry the hash. Helm teaches the same idea from the other side, as a comparison with Kustomize, and the Master tier puts the two side by side.
Do this: run step 3 of the lab, both halves: once rejected at the immutable ConfigMap, once renamed by the generator. Write down the hash suffix on each generated object, change one literal, and write down the new suffix.
Try it: Advanced
I turned on etcd encryption today. Are my old Secrets now encrypted?
Now do it: On your cluster, read the EncryptionConfiguration the API server loads and list which resources it encrypts. Then check whether any Secret predates the day encryption was turned on.
At the master level the question is not how to configure a workload. It is which of the six ways fits the change you are about to make. Five of the six are taught elsewhere in the course. One is taught here first.
Pod command: and args:, first teaching in the k8s track. A Pod can take its config on the command line, in command: and args:. Nothing in the course teaches this in the Kubernetes track. The command overrides the image entrypoint, and the args override the image arguments. It is the simplest way to pass a value, and the worst way to pass a secret: the command line is visible in describe, in the audit log, and in the process list.
containers:
- name: api
command: ["sh", "-c"]
args: ["./api --log-level=INFO"]
What the snippet means: the container starts by running that command with those args. A flag on the command line is fine. A password on the command line is a password in the audit log. The rule is short: flags go on the command line, secrets never do.
The decision table. Six ways, one row each. Read down the change rate column first: how often does this value change. Then the sensitivity column: what happens if it leaks. Then the team size column: who else has to live with the choice.
| Way | Change rate | Secret sensitivity | Team size | Pick it when |
|---|---|---|---|---|
| Env vars | Low. A change needs a new pod. | Low. Visible in describe, so no real secrets. | Any. The simplest way there is. | The value is a flag, and a rollout on change is fine. |
| Volume mounts | High. The file updates under the running pod. | High. The right home for certs and keys. | Any. One mount per object, and the app reads files. | The value is a file already, or it must update without a restart. |
| Pod command: / args: | Low. A change needs a new pod. | None. Never a secret. The command line is logged. | One person, one pod, one debug session. | A flag for one run, or a debug override you will delete. |
| Kustomize | Medium. A change rebuilds the generated names. | Medium. Generators build Secrets, but the values live in the overlay. | Small. A few environments, a few overlays. | The variation is small, and the team reads YAML all day. |
| Helm values | Medium. A change is a new release. | Medium. Values files hold config, and secrets stay out of them. | Medium to large. Many releases, many overrides. | The variation is large, and the chart is shared. Helm teaches values in full. |
| GitOps | High. A change is a commit, and the sync loop applies it. | High, with a store behind it. Git holds the reference, never the value. | Any, once the loop exists. The bigger the team, the more it pays. | Git is the truth, and the cluster follows it. GitOps teaches the loop in full. |
Kustomize, generators plus overlays. The generators build the objects, and the overlays change them per environment: one base, one overlay per namespace, and the differences live in the overlay. Helm teaches the comparison in its Advanced tier: pick Kustomize when the variation is small, pick Helm when the variation is large. This lesson adds the generators to the Kustomize side of that comparison, and the decision table above is where the two meet.
Rotation patterns. A secret that never rotates is a secret that leaks once and stays leaked. Three patterns, and the propagation mechanics behind them are in the P9 fold-in in Lesson 84. First, the mounted-secret update: the value changes, the kubelet syncs the file, and the app that re-reads the file picks it up with no restart. Second, the reloader sidecar: a small watcher runs next to the app, sees the Secret change, and restarts the app, or sends it a signal. Third, the checksum annotation: the Pod template carries an annotation with the hash of the ConfigMap or Secret, the hash changes when the value changes, the template changes, and the Deployment rolls the pods. The annotation is the one you write by hand, and it looks like this:
template:
metadata:
annotations:
checksum/config: abc123
What the snippet means: the annotation holds the hash of the config, and a new hash means a new template, and a new template means a rollout. The value abc123 stands for the real hash, which you compute from the ConfigMap. The pattern is the same one the traffic paths use in path 3: the edit arrives, the annotation changes, the pods roll.
RBAC on Secrets, the worked Role. Lesson 158 taught RBAC and worked one Role: a deployer who may create deployments and read pods. It never granted secrets to anyone, and no resourceNames scoping appears anywhere in the course. Here is the Role that does both. The full file, with its RoleBinding, is in the lab.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: secret-reader
namespace: shop
rules:
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["shop-api-password"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: read-shop-secret
namespace: shop
subjects:
- kind: User
name: mohan
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: secret-reader
apiGroup: rbac.authorization.k8s.io
What the snippet means: one user, mohan, may get one Secret, shop-api-password, in the shop namespace, and nothing else. The resourceNames field is the scope: without it, the same Role would let mohan read every Secret in the namespace. The failure mode is a Role that grants secrets get in kube-system, or cluster-wide, and the rule is the one this lesson repeats: Secrets are the crown jewels, and the default is deny. A reader gets named, scoped, and read-only, or the reader gets nothing.
Admission as the last gate for secrets. One paragraph, and it links instead of teaching. A policy engine can police secrets at admission: a rule that denies a Secret committed to Git, a rule that requires immutable true on every production Secret, a rule that denies a Pod that mounts a Secret it has no Role to read. The engines are taught in Lesson 158, OPA and Kyverno, and in Lesson 159, Gatekeeper. This lesson does not re-teach them. The point to carry away is the position: admission is the last gate, and it catches the mistake the author missed, the reviewer missed, and the pipeline missed.
Do this: pick one service you run, write down which of the six ways each of its values uses today, and move one value to a better way. Then write the secrets-scoped Role for its namespace, and run the three can-i checks from the lab against it.
Try it: Master
I need to pass a password to a container for one debug run. Why do I not put it in command or args?
Now do it: Pick one value in a pod you run and walk it down the Master decision table: change rate first, then sensitivity, then team size. Write down which of the six ways fits and why.
The encoding demo is two commands, and you can run both on this machine, no cluster needed. Encode a value, then decode it back. The round trip is the lesson: nothing was encrypted, and anyone with the encoded value can run the second command.
printf 's3cret-password' | base64 printf 'czNjcmV0LXBhc3N3b3Jk' | base64 -d
What the snippet means: the first command prints czNjcmV0LXBhc3N3b3Jk, and the second prints the password back. That is all base64 does. It changes the shape, not the secrecy. Lesson 84 teaches it, and this lesson makes you run it.
Nothing in the cluster checks the date on a TLS Secret. The kubelet mounts the cert, the pod serves it, and the first client that connects gets a handshake failure. The pod keeps running, the Service keeps routing to it, and the readiness probe keeps passing, because the probe checks the port, not the cert. The fix is a renewal before the expiry, not a restart after it. Renew the cert, update the Secret, and the mounted file updates under the running pod, because the TLS Secret is consumed as a volume. The rotation is the mounted-secret update from the Master tier, and the cert that never gets renewed is the incident the traffic paths show in path 2.
Rotating the encryption key is a read-and-write job. Point the KMS provider at the new key, restart the API server so it reads the new config, then read every Secret and write it back. The read and the write re-encrypt the value under the new key. The command is one line, and it runs on your own cluster:
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
What the snippet means: every Secret is read as JSON and written back, and the write encrypts it under the current key. A Secret you miss stays under the old key. Run it after every key rotation, and run it once after turning encryption on, because turning it on does not re-encrypt what is already stored.
A volume mount with subPath mounts one file, not the whole directory, and that one file does not update when the ConfigMap changes. The kubelet syncs the directory, not the single file. This lesson does not teach subPath. It is named here so you know why a pod that mounts one file with subPath keeps the old value after an edit, while a pod that mounts the whole directory picks it up.
A Secret written with stringData takes plain text values, and the API server encodes them to base64 on write. A Secret written with data takes values you encoded yourself. Read the Secret back and both spellings look the same: data, base64, no trace of which spelling you used. The rule is short: write with stringData, read with data, and never paste a base64 value you encoded by hand into a file you review in Git, because a reviewer cannot read it.
This lesson is the cluster's own config system, and it ends where Lesson 156 begins. Vault is where secrets come from when the cluster is not the source of truth: dynamic secrets, leases, the agent injector that puts the secret into the pod. The next lesson in this series, Lesson 161, teaches the stores beyond the cluster: the operators that sync a secret from a store into the cluster, and the drivers that mount it straight from the store. This lesson owns the object. Lesson 156 owns the source. Lesson 161 owns the sync.
At 10x the namespaces, the per-namespace pull secret is the first thing that breaks, and the fix is the credential on the ServiceAccount, not on each pod. At 10x the secrets, the manual rotation is the first thing that breaks, and the fix is the generator with the hash suffix, so a new value is a new object and the rollout follows the name. At 100x, etcd encryption is the product: the KMS key is rotated on a schedule, the re-encryption runs after every rotation, and the audit log records every read of every Secret, because at that scale the question is not whether a secret leaked. It is who read it last. The types do not change. The number of readers does.
The creation and consumption this lesson links to: Lesson 84 Kubernetes Lab: Config, Secrets, and Storage. The RBAC definitions and the deployer Role: Lesson 158 Kubernetes Security. The admission engines behind the last gate: Lesson 158 Kubernetes Security for OPA and Kyverno, Lesson 159 Kubernetes Pod Security in Depth for Gatekeeper. Where secrets come from: Lesson 156 Vault & Secrets Management. Values and releases: Helm. Git as truth: GitOps. The components this lesson runs on: Lesson 94 Kubernetes Components. Encryption at rest, the wider view: Data in Transit and at Rest.
A database password leaked through a log. The pod read the password as an env var, and the app printed its whole environment to the log on startup, one line per variable, as a debug aid. The log went to the central logging system, where every engineer on the team could read it, and the retention was 90 days. The password was valid for all 90 of those days. Nothing about the deploy was wrong. The Secret was typed Opaque, the pod read it with secretKeyRef, and the Role allowed the deployer to read it. The leak was not in the Secret. It was in the env var, which is visible to anything that can read the process environment, and the app was the thing that read it.
Three things. First, the password moved from an env var to a volume mount, because a file is read by the app and nothing else, while an env var is read by anything that can dump the environment. Second, the startup dump was deleted, because a debug aid that prints secrets is a leak with a helpful name. Third, the log retention was cut for the service, and the rotation runbook was written down: rotate the value, update the Secret, roll the pods, revoke the old value, and purge the logs that hold it. The Secret did nothing wrong in this incident. The lesson is that a secret is only as safe as its most visible copy, and an env var is the most visible copy there is.
Design the config system for a cluster with 50 services across 10 namespaces. Each team deploys its own services. Each service has a log level, a feature flag, a database password, a TLS cert, and a private-registry credential. The database password rotates every 30 days. The TLS cert expires every year. A config edit must reach the pods that mount files without a restart, and the pods that read env vars must roll. No team may read another team's secrets. The question is the system: sketch the types, the consumption, the encryption, the rotation, the RBAC, and the last gate, and say what you would measure.
Start with the types. Every password is an Opaque Secret, every cert is a TLS Secret, every registry credential is a dockerconfigjson Secret, and every service account gets its token from the controller, not from a file. The types are the checks, and a type that rejects a bad shape at creation is a failure that never reaches 2 AM.
Then the consumption. Flags and one-line values go in env vars, because a rollout on change is fine for them. Certs, keys, and the properties file go in volume mounts, because they are files in the app's mind already and they must update without a restart. No secret goes on the command line, because the command line is logged. The decision table in the Master tier is the record of these choices, and the table is reviewed when a value moves homes.
Then the encryption. EncryptionConfiguration covers secrets first, with the KMS provider, and the identity provider last as the fallback. The key rotates on a schedule, and the read-and-write re-encryption runs after every rotation. A backup of etcd taken before encryption was turned on is treated as a leak, because it holds the old plain-text values.
Then the rotation. The database password rotates every 30 days through the generator, so the new Secret gets a new name and the Deployment rolls. The TLS cert is renewed a month before expiry, the Secret is updated, and the mounted file updates under the running pods. The registry credential rotates with the CI account, and the per-ServiceAccount pull secret means the rotation lands in one place.
Then the readers. One Role per namespace, one Secret per resourceNames entry, read-only verbs. No Role grants secrets get without a resourceNames scope, and no Role grants it in kube-system. The can-i checks from the lab run in CI against every Role, and a Role that fails them blocks the deploy.
Then the last gate. Admission carries two rules: no Secret in Git, and every production Secret immutable. The rules are written in the engine Lesson 158 teaches, and the gate is the last check, not the first. A rule that blocks a deploy on the day it lands is a rule that was never run in audit mode, and the rollout is audit first, then enforce.
Trade-offs to name. Env vars buy simplicity and cost a rollout on every change, and a secret in an env var is a secret in the logs. Volume mounts buy live updates and cost an app that re-reads its files, and an app that reads once at start gains nothing from the mount. Immutability buys no silent drift and costs a delete and recreate on every change, and a team that edits in place will fight the lock until the generator gives them a new name. KMS encryption buys real protection for etcd and costs a key service that must stay up, and a KMS that is down is a cluster that cannot start its API server. And the target is the system, not the secret. A password that is safe and unrotated fails this design, and the rotation is where the safety lives.
Saved in this browser, so the Lessons index can show your progress across the course.