Lesson 48 taught you ConfigMaps and Secrets, and the storage lesson taught you PVCs. This lab is where you use them: a ConfigMap that sets the app environment, a Secret that holds a password the logs never show, a PVC that keeps the data across a pod restart, and probes that decide when a pod may receive traffic. By the end you have a Deployment that survives a restart with its data, and you know the difference between config and a secret, because you put one in the open and one behind a lock.
Difficulty: Intermediate · Know this first: Lesson 48 — Kubernetes Core; Lab 159; minikube or k3s installed, kubectl installed
Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: on a cache hit the trip is almost free, on a miss the kitchen eats the budget. Step through the lesson or switch the scenario and watch the bar change.
Extra diagram · from scratch to master
Start with one ConfigMap. Add one box at a time until the data survives a restart and the secret stays sealed.
The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the wiring, the storage, and the total change in the budget bar, in the hop timings, and in the total. The numbers are a teaching example, not live measurements. The shape is the lesson: more config means more wiring, and a leaked secret costs a rotation.
How many ConfigMaps the app reads. Each one is wired by name, and each one is one more thing to check.
How many Secrets the app reads. Each one is sealed, and each one must stay out of the logs.
How big the PVC is, in Gi. A bigger volume takes longer to bind, and it holds more data.
Try this: (1) Set ConfigMaps to 1, Secrets to 1, volume to 1 Gi. The wire is about 3 min, the store about 2 min, the check about 1 min, and the total is about 11 min. (2) Set all three to the worst end: 5 ConfigMaps, 3 Secrets, 10 Gi. The wire is about 13 min, the store about 6 min, the check about 1 min, and the total is about 25 min. Same app. Only the inputs changed.
Three questions. No score is saved anywhere — this is just you against the material.
Here is the Deployment from this lab, ready to edit. Change it, then run the check. The check looks for the five things a working Deployment needs: a ConfigMap read, a Secret read, a PVC mount, a readiness probe, and a liveness probe.
Try breaking it on purpose: change configMapKeyRef to the wrong ConfigMap name, or delete the readinessProbe. Run the check and watch it fail. Then press Reset and try a change that keeps it passing, like adding a second env var.
Expected output when it passes: The Deployment reads the ConfigMap and the Secret by key, mounts the PVC, and has both probes. kubectl apply would create it, and the pods would become Ready.
kubectl apply -f, flag by flag. apply reads a file and moves the cluster toward it. The file is the desired state. The flag says which file.
-f deploy.yaml names the file to apply. Without it the command has nothing to read. The file is the desired state, and apply is what moves the cluster toward it.Flags you will meet next: --dry-run=client prints what would happen without doing it, and -n picks the namespace.
Try it: apply the same file twice. The second run prints unchanged, not created. That is the proof apply is declarative: it compares, and it only changes what differs.
I take the same pod four times here. First I configure it. Then I seal it. Then I store its data. Last I prove it.
Definition: A ConfigMap holds plain values: the environment name, the log level, the feature flags. valueFrom configMapKeyRef reads one key into one environment variable. Config is for values anyone may read, and a password is never config.
Types:
Uses: I use a ConfigMap for every value that is not a secret, so the config can change without rebuilding the image. Try it now: read APP_ENV inside the pod and check it says production.
Dimensions: At low volume everything is instant. What breaks first is a ConfigMap name that does not exist: the pod fails to start, and the error names the ConfigMap.
Try it: Basic
I need to store the environment name and the log level. Where do they go, and where does the password go?
Now do it: On your own cluster, read APP_ENV inside the pod and check it says production.
Definition: A Secret holds passwords, tokens, and keys. kubectl create secret keeps the value out of git. secretKeyRef reads one key into one environment variable. Base64 is not encryption: anyone who can read the Secret can decode it, and the protection is the access, not the encoding.
kubectl create secret generic, flag by flag. A secret holds values the app needs but the manifest should not show. --from-literal puts one value in, on the command line.
--from-literal=password=changeme puts one key and value straight into the secret. The part before the = is the key. The part after is the value. Without it the secret is created empty.Flags you will meet next: --from-file=key=path reads the value from a file, and --dry-run=client -o yaml prints the secret without creating it.
Try it: create the secret, then read it back with kubectl get secret hello-secret -o yaml. The value is base64, not plain text. Base64 is encoding, not encryption. Anyone who can read the secret can decode it.
Types:
Uses: I use a Secret for every password, token, and key, and I never print one. Try it now: create the Secret, then grep the logs for the password and check the result is empty.
Dimensions: As the secrets grow, the access grows, and the risk grows with it. What breaks is a Secret in a file: a secret in git history is a secret in every clone, forever.
Try it: Intermediate
I read my Secret back with kubectl get secret and the value is base64. Is it encrypted?
Now do it: On your own cluster, create the secret, then grep the logs for the password and check the result is empty.
Definition: A PVC is a claim for storage. Bound means the cluster has bound a volume to it. The data lives in the volume, not in the pod, so the pod can die and the data stays. The marker file is the proof: write it, delete the pod, read it back from the replacement.
Types:
Uses: I use a PVC whenever the data must outlive the pod, which is every database and every queue. Try it now: write the marker, delete the pod, and read it back.
Dimensions: At ten times the data, the volume grows, and the binding takes longer. What breaks is a PVC that is Pending: the cluster has no storage to give, and the pod waits forever.
Try it: Advanced
I write a marker file, delete the pod, and read the marker back from the replacement. What does that prove?
Now do it: On your own cluster, write the marker, delete the pod, and read it back from the replacement.
Definition: At this level the Deployment is a promise that survives change. kubectl set image rolls the image forward, one pod at a time. kubectl rollout undo rolls it back the same way. kubectl rollout status is the watch, and it waits until the rollout is done.
Types:
Uses: I use rollouts for every change, and I test the undo before I need it. Try it now: roll forward to 1.28, watch it succeed, and roll it back.
Dimensions: At a hundred pods, the rollout is the same shape, and the probes are the same. What breaks is a rollout with no readiness probe: the Service sends traffic to pods that are not ready, and the first requests fail.
Try it: Master
I roll the image forward with kubectl set image, then roll it back. What watches the rollout, and what breaks if I skip the readiness probe?
Now do it: On your own cluster, roll forward to 1.28, watch it succeed with kubectl rollout status, and roll it back.
The Deployment, the Service, the ConfigMap, and the PVC, one file. Do this: save it as deploy.yaml and run kubectl apply -f deploy.yaml. Expected output: four created lines, one per resource. This file was checked in this lab: it parses clean with yaml.safe_load, it holds four documents, the Deployment reads the ConfigMap by key, and mounts the PVC at /data.
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello
spec:
replicas: 3
selector:
matchLabels: {app: hello}
template:
metadata:
labels: {app: hello}
spec:
containers:
- name: hello
image: nginx:1.27
ports: [{containerPort: 80}]
env:
- name: APP_ENV
valueFrom:
configMapKeyRef: {name: hello-config, key: APP_ENV}
resources:
requests: {memory: "64Mi", cpu: "100m"}
limits: {memory: "128Mi", cpu: "250m"}
livenessProbe:
httpGet: {path: /, port: 80}
initialDelaySeconds: 5
periodSeconds: 10
readinessProbe:
httpGet: {path: /, port: 80}
initialDelaySeconds: 3
periodSeconds: 5
volumeMounts: [{name: data, mountPath: /data}]
volumes:
- name: data
persistentVolumeClaim: {claimName: hello-data}
---
apiVersion: v1
kind: Service
metadata:
name: hello
spec:
type: ClusterIP
selector: {app: hello}
ports: [{port: 80, targetPort: 80}]
---
apiVersion: v1
kind: ConfigMap
metadata:
name: hello-config
data:
APP_ENV: production
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: hello-data
spec:
accessModes: [ReadWriteOnce]
resources:
requests: {storage: 1Gi}
The Secret, created by command, never by file. Do this: run kubectl create secret generic hello-secret --from-literal=password=changeme. Expected output: secret/hello-secret created. Then run kubectl get secret hello-secret -o yaml and expect the password to be base64, not plain text. Change the password in your own run: changeme is the lab placeholder, and a real password is never changeme.
kubectl get, flag by flag. get lists objects. The label flag is how I ask for the three pods I care about instead of every pod in the cluster.
-l app=web filters by label. Without it I get every pod in the namespace, and the three I care about hide in the list.-o yaml prints the whole object as YAML instead of the short table. Without it I see a summary, and the field I need is not in it.Flags you will meet next: -n picks the namespace, -o wide adds the node and IP columns, and --all-namespaces lists every namespace at once.
Try it: run kubectl get pods, then kubectl get pods -l app=web. The first lists everything. The second lists only the web pods. Then add -o yaml to one pod and read the full object.
The marker test, the proof the PVC works. Do this: run kubectl exec deploy/hello -- sh -c 'echo keep-me > /data/marker.txt' — the volume is mounted at /data in the Deployment above — then delete the pod, wait for the replacement, and run kubectl exec deploy/hello -- cat /data/marker.txt. Expected output: keep-me. The pod died, the replacement came back, and the data is still there.
kubectl exec, flag by flag. exec runs a command inside a running container. The -- separator is the flag that makes it work at all.
deploy/hello names the workload. kubectl picks a pod from it. Without a target, exec has nowhere to run.-- marks the end of kubectl flags. Everything after it is the command to run inside the container. Without it kubectl tries to read the container command as its own flag, and the command fails before it starts.cat /data/marker.txt the command that runs inside the container. It reads a file and prints it. If the file is missing, the error is the answer: the data did not survive, or it was never written.Flags you will meet next: -it gives you a terminal inside the container, -c container picks one container in a multi-container pod, and -- sh -c '...' runs a compound command.
Try it: run the cat command, then run it again after writing the marker file. The first read fails or prints nothing. The second prints the marker. The file is the proof the volume survived, and exec is how I checked it.
Honest limit, stated plainly: this lab was verified by parsing the manifests and by reading the kubectl docs, not by running kubectl apply. There is no cluster in the machine this lab was written on, so no pod was actually started here. The commands, the outputs, and the checks are the ones the reader runs on their own machine, where minikube or k3s is running.
At 10x the config, the shape holds and the ConfigMaps multiply. At 100x, the design changes: the config moves into a dedicated store, and the Secrets move into a vault. The split never changes, though. Config in the open, secrets sealed, data in the volume.
This lab practices Lesson 48 — Kubernetes Core: ConfigMaps and Secrets, the objects this lab puts to work; and Lesson 106 — Kubernetes Storage & CSI: the PVC that keeps the data across a pod restart.
I edit a ConfigMap and I expect the pods to pick it up. That expectation is half right. How the change travels depends on how the pod reads the ConfigMap, and the two ways behave differently.
A ConfigMap mounted as a volume is a live file. The kubelet watches the ConfigMap through the API server and keeps a local cache of it. When I edit the ConfigMap, the kubelet sees the change, refreshes its cache, and writes the new value into the mounted file inside the pod. The pod keeps running. No restart happens. The update is not instant. I wait about a minute, then I read the file again. That wait is the kubelet cache and its sync loop doing their work, not a bug.
One exception matters. A file mounted with subPath does not update. It is a snapshot taken when the pod started, and it stays frozen until the pod is replaced. If I need a live file, I mount the whole ConfigMap, not one key through subPath.
An environment variable is the opposite. It is set once, when the container starts, and it never changes after that. A ConfigMap edit does not change the env of a running pod, and no amount of waiting fixes it. I learned this the slow way. I edited the ConfigMap, I waited, and printenv still showed the old value. The file had the new value. The env did not. Only a new pod gets the new env.
The same split holds for Secrets. A Secret mounted as a volume updates the same way a ConfigMap does, through the kubelet watch and cache, with the same wait. A Secret read into env is frozen at pod start, the same as config. And base64 is still not encryption. The value in etcd is only safe when etcd encryption is turned on, and that part lives in Lesson 107, Kubernetes Security.
That frozen env is also why a Deployment does not roll when I edit a ConfigMap. The Deployment watches its pod template. A ConfigMap is not in the template, so the template did not change, and the controller sees nothing to do. The pods keep running with the old env, and no rollout starts. The fix is to make new pods. I run kubectl rollout restart deployment/hello on my own machine, and the Deployment creates new pods that read the new ConfigMap at start. The output you should see on your own machine is deployment.apps/hello restarted. The other fix is a checksum annotation. I put a hash of the ConfigMap into an annotation on the pod template, so the template changes whenever the config changes, and the Deployment rolls on its own. Lesson 198, Deployments and ReplicaSets teaches the rollout itself, step by step.
Try it: edit APP_ENV in hello-config, wait a minute, then read the mounted file and run printenv in the same pod. The file shows the new value. The env shows the old one. Then run the rollout restart and read printenv again. Now it shows the new value, because the pod is new.
Immutable ConfigMaps and Secrets live in Lesson 203, Kubernetes Secrets and ConfigMaps. The generators that build them live there too. That lesson also teaches the five secret types, etcd encryption, and image pull secrets.
The stores outside the cluster live in Lesson 204, External Secret Stores. That lesson covers Vault, Conjur, and the operators that sync secrets into the cluster.
An app I owned printed its whole environment at startup, for debugging, and the log line included the database password. The logs went to a central place, where everyone on the team could read them. The password was in the logs for three months before anyone noticed, and by then it was in every log archive the team kept. The fix was not deleting the logs. The fix was rotating the password, cleaning the archives, and removing the print.
Two things. First, no app prints its environment, ever, and the pipeline greps the logs for Secret values before a deploy. Second, Secrets are rotated on a schedule, so a leak has a short life even when nobody notices it. Try it now: grep the lab logs for the word password and confirm the result is empty.
Build a Deployment for an app you choose with a ConfigMap, a Secret, a PVC, and both probes. It passes when the config reads inside the pod, the PVC is Bound, the data survives a pod delete, and the logs hold no Secret value.
Success looks like this: printenv shows both config keys, the PVC reads Bound, the marker survives a pod delete, both probes are listed, the log grep is empty, and both rollouts succeed. If the marker is gone, the volume was not mounted. If the grep finds the value, the app prints its environment.
If the pod fails to start, read the Events in describe: a missing ConfigMap or Secret is named there. If the PVC stays Pending, the cluster has no storage class to bind. If the marker is gone after the delete, the volumeMount path is wrong. If the rollout hangs, the readiness probe is failing, and the pod never becomes Ready.