A Secret in Git is a leak. A Secret in etcd is a copy with no owner. I keep the real secret in a store built for secrets, and I let something in the cluster fetch it. This lesson is the map of where secrets live when they do not live in etcd, and the three ways they get into a Pod.
Difficulty: Advanced · Know this first: Lesson 156, Lesson 127, Lesson 89, Lesson 94
By the end of this lesson you will be able to name the three sync paths and say which one fits your team. You will have written an ExternalSecret that syncs a cloud secret into the cluster, sealed a secret for Git with kubeseal, encrypted a file with SOPS and age, and mounted a secret as a file through the CSI driver. You will know when Vault wins, when Conjur wins, and when the cloud store you already pay for is enough.
What you need: a cluster you can read with kubectl, and a cloud account or a Vault for the worked syncs. The operator, Sealed Secrets, Conjur, and CSI steps run on your own cluster, and every expected output is described as what to look for, not quoted as a transcript. No cluster runs on this machine. This VM hands out no cpu controller in its cgroup, so no k3s server can start on it, and that was settled in earlier lessons, so I did not re-attempt it here. The SOPS step is the exception. SOPS and age are small static binaries, and I installed them on this machine and ran the encrypt and decrypt round trip here, so that step carries real output. No kubectl output from this machine appears in this lesson, because there is no cluster on this machine to produce it. Your numbers will differ. The shape will not.
Lesson 160 teaches the cluster's own config system. A Secret in etcd is the cluster holding a value. This lesson starts one step earlier. Where did that value come from, who may change it, and who knows when it changed. The answer is almost never etcd. etcd is where the copy sits. The store is where the secret lives.
A Secret in Git is a leak. Git remembers everything. You can delete the file, and the value is still in the history, in every clone, on every laptop that ever pulled the repo. A secret that was committed once is a secret you must rotate, because you cannot uncommit it. The only thing Git may hold is ciphertext, and this lesson shows two kinds of it.
A Secret in etcd alone is not enough, and the gap has three parts. First, rotation. A value in etcd changes only when a person changes it, and nobody is watching the calendar. A store built for secrets rotates on a schedule and tells you when it did. Second, audit. etcd can tell you the Secret changed. It cannot tell you which person read the value, from where, and why. A store logs every read. Third, one vault per team. etcd is one store for the whole cluster, and every team shares it. A team that owns its own store sets its own rules, its own readers, and its own rotation, and a leak in one team stays in one team. Lesson 53 teaches rotation as a practice. This lesson is the machinery that makes the practice automatic.
There are three ways a secret gets from the store into a Pod, and the choice is the lesson. The operator syncs it into a Kubernetes Secret. The CSI driver mounts it at Pod start. The agent injects it at Pod start. Same store, same Pod, three different landings.
| Path | How it works | The secret lands as | When it wins |
|---|---|---|---|
| Operator sync | An operator reads the store on a timer and writes a normal Kubernetes Secret. The Pod reads it like any Secret. | A Secret in etcd, refreshed on a timer. | Most teams. The Pod changes nothing, and the sync is one object to read. |
| CSI mount | The CSI driver reads the store when the Pod starts and mounts the secret as a file. No Secret object is written unless you ask for the sync. | A file in the Pod, read at Pod start. | The secret must never sit in etcd. The app reads a file. |
| Agent injection | A sidecar or an init container fetches the secret when the Pod starts. The Vault Agent Injector is the shape Lesson 156 teaches. | A file or an environment variable, written by the agent. | The secret changes during the Pod's life, and the Pod must pick it up without a restart. |
The operator path is the default for most teams, because the Pod changes nothing and the failure is one object to read. The CSI path wins when the secret must never sit in etcd, because the file is the only copy in the cluster. The agent path wins when the secret is short lived and the Pod must renew it while it runs. Pick the path by what the secret needs, not by which tool is newest. The decision table in the Master tier makes the call explicit.
On your own cluster, run kubectl get secrets -n shop and write down, for each Secret, where the value came from. If the answer is "someone typed it once", that Secret has no owner and no rotation plan. That is the gap this lesson closes. One line per Secret, one source per line. If you cannot name the source, the store is the answer, and the path is the choice below.
The diagram above follows one secret from the store to the Pod. These four paths are the same idea drawn four ways. One healthy operator sync. One rotation the operator missed. One CSI mount, where no Secret object is ever written. One numbers trail, hop by hop. Run each path. One run takes 7.2 seconds across the path, so you can watch each hop light up in turn.
The return rule: the green dot only starts moving after the orange dot reaches the data hop. A return with no finished work is a lie. The diagram never draws one. The failure replay at the end of Run all four shows the other side of the rule. Run it and watch the orange dot stop at the missed version and never come back.
The store holds the secret, the operator reads it in 2 s, writes the Secret in 1 s, the Pod mounts it and starts, the store rotates the value, the operator re-syncs after the 60 s poll wait, and the Pod reads the new value in 72 s end to end. The green dot starts back from the Pod reads new value hop, because that is where the finished work lives.
Not run yet.
The store rotates the password. The operator polls on its timer, but the ExternalSecret pins the old version, so the poll reads the old value and writes nothing new. The Pod keeps the old password, the app calls with it, and auth says no. The request stops at the missed version. No green dot ever starts back, because the new value never reached the Pod.
Not run yet.
The store holds the secret and the driver reads it when the Pod starts. No Secret object is ever written, so nothing lands in etcd. The Pod mounts the file and starts, the store rotates the value, the kubelet re-reads the file, and the Pod reads the new value in 12 s end to end. The green dot starts back from the Pod reads the file hop, because that is where the finished work lives.
Not run yet.
Three sync paths, three cloud stores, two operators that sync or seal, two file tools that hold ciphertext in Git, and four CSI providers that mount secrets as files. One decision at the end of it. The green dot starts back from the verdict, because the decision is the answer this path carries.
Not run yet.
Every hop charges rent in seconds. The bar below is a worked example for one secret, from the store to the Pod, not a measurement from a live system, but the shape is honest. The re-sync wait is the big hop at 60 s, and the total is 72 s. Switch the scenario and watch the bar change. When the operator misses the rotation, the re-sync hop and the Pod reads hop disappear, because the new value never arrived. On the CSI path, the operator write hop disappears, because no Secret object is ever written. Move a simulator control and watch the hops move with it.
The budget bar above shows three fixed stories. This simulator lets you write your own. It answers the question at the heart of this lesson: when a secret rotates in the store, how long does the Pod keep the old value. Rotate the secret in the store, break the operator sync, and move the refresh interval from 15 seconds to 10 minutes. Watch whether the Pod sees the new value, how long the pickup takes, and whether the app can still log in. The numbers are a teaching example, built from the formulas below, not live measurements. The shape is the lesson: the operator polls, so the pickup wait is up to one refresh interval, and a missed sync means the Pod keeps the old value forever.
The store holds a new version of the secret. The Pod still holds the old one until the operator syncs it.
The ExternalSecret pins the old version, so the poll reads the old value and writes nothing new. The operator reports Ready, because the sync worked. It synced the wrong version.
How often the operator polls the store, from 15 seconds to 10 minutes. The pickup wait is up to one interval. A short interval finds the new value fast and reads the store often. A long interval reads the store rarely and leaves the Pod stale for longer.
Try this: (1) At the defaults, a rotation happened and the sync is healthy at a 60 s interval, the Pod picks up the new value in about 60 s and the path total is 72 s. (2) Tick the missed sync. The re-sync hop costs 0 s, the Pod keeps the old value, the path total is 11 s, and auth fails. Untick it. (3) Move the interval to 600 s. The Pod waits up to 10 minutes for the new value and the path total is 612 s. The interval is the whole story at that setting. (4) Untick the rotation. Nothing changed in the store, so there is nothing to pick up, and the path total is 12 s.
Formulas, in plain terms. The store hop costs 1 s. The operator read costs 2 s. The operator write costs 1 s. The mount costs 2 s. The Pod start costs 3 s. The rotation hop costs 2 s when a rotation happened and 0 s when nothing changed. The re-sync hop costs 2 s when nothing rotated, 0 s when the operator missed it, and the refresh interval you set when a rotation happened and the sync is healthy. The Pod reads hop costs 1 s when the Pod holds a working value and 0 s when it holds a stale one. The path total is the eight hops added up, and hop 0 costs 0 s because it starts the clock. The diagram above uses a 60 s refresh interval, so its re-sync hop costs 60 s and its total is 72 s. The simulator shows what changes when the interval moves.
Three steps. Steps 1 and 2 run on your own cluster, and every expected output is described as what to look for, not quoted as a transcript. Step 3 ran on this machine, because SOPS and age need no cluster, and it carries the real output. There is no kubectl output from this machine in this lesson, because there is no cluster on this machine. The k3s install cannot start here, as the top of this lesson says.
Install the External Secrets Operator on your own cluster, then point it at a cloud store. The install is one Helm command.
helm repo add external-secrets https://charts.external-secrets.io helm install external-secrets external-secrets/external-secrets \ -n external-secrets --create-namespace
What to look for: the operator Pods Running in the external-secrets namespace. Then apply the SecretStore. This one points at AWS Secrets Manager, and the operator reads it with the service account, not with a static key. The YAML was parsed and checked on this machine with PyYAML. It has not been applied to a live cluster from this machine.
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: shop-store
namespace: shop
spec:
provider:
aws:
service: SecretsManager
region: us-east-1
auth:
jwt:
serviceAccountRef:
name: eso-reader
Then the ExternalSecret. It says which remote value to fetch, how often to poll, and what Kubernetes Secret to write. The refresh interval is one hour. That is the pickup wait when the value rotates.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: db-password
namespace: shop
spec:
refreshInterval: 1h
secretStoreRef:
name: shop-store
kind: SecretStore
target:
name: db-password
creationPolicy: Owner
data:
- secretKey: password
remoteRef:
key: shop/db-password
property: password
What to look for: the ExternalSecret reports Ready, and a Secret named db-password appears in the shop namespace with one key, password. Read the status of the ExternalSecret first when the Secret does not appear. The status names the failure: the store cannot be reached, the key does not exist, or the service account may not read it. The app log is the wrong place to start. The operator is the one doing the fetching, so the operator is the one that knows why it stopped.
Write a plain Secret, then seal it with kubeseal. The sealed file is safe to commit. The plain file is not, and it never goes into Git.
kubectl create secret generic db-password \ --from-literal=password='change-me-in-production' \ --dry-run=client -o yaml > db-secret.yaml kubeseal --format=yaml < db-secret.yaml > sealed-secret.yaml
What to look for: sealed-secret.yaml holds a SealedSecret, and the value under encryptedData is ciphertext, not the password. The shape is below. The ciphertext value shown here is a placeholder. kubeseal writes the real one, and the real one is long, so do not type a value here by hand. Commit the sealed file, delete the plain one, and apply the sealed file to the cluster. The controller decrypts it and writes the normal Secret.
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
name: db-password
namespace: shop
spec:
encryptedData:
password: <kubeseal output goes here>
template:
type: Opaque
What to look for after the apply: a Secret named db-password appears in the shop namespace, and its value is the password you sealed. The cluster key is the whole point. The controller made a key pair when it started. The public key sealed the value, and only the controller holds the private key that opens it. That is why Git can hold the ciphertext. A repo with the sealed file and no cluster is a locked box with no key.
This step needs no cluster, so it ran here. SOPS 3.10.2 and age 1.2.1 were installed on this machine from their release binaries, and the commands below are the ones that ran. First the key. age-keygen writes one private key file and prints the public key. Guard the private key file like a password. The public key is the one that goes into the config.
age-keygen -o sops-age-key.txt # Public key: age1... (your key prints here; use that value below)
Then the rule file. It says which files get encrypted, with which key, and which fields stay plain. Only the data fields are encrypted. The kind, the name, and the namespace stay readable, so Git can still tell you what the file is.
creation_rules:
- path_regex: .*\.enc\.yaml$
age: <your age public key>
encrypted_regex: '^(data|stringData)$'
Then the encrypt and the decrypt. The input file is a plain Secret with one password. The encrypted file keeps the shape and hides the value behind an ENC block. The decrypt brings the plain file back.
sops --encrypt --in-place sops-demo-secret.enc.yaml sops --decrypt sops-demo-secret.enc.yaml
What this run proved, on this machine, on Nov 21, 2026. The encrypted file parses as YAML. The kind stays plain, Secret, and the stringData value starts with ENC[, so the value is ciphertext and the shape is still a Secret. The decrypt round trip is identical to the input file, checked field by field with PyYAML, so nothing is lost in the round trip. The ciphertext itself is different on every run, because encryption picks a fresh key each time, so I do not quote it as a transcript. The shape is the proof, and the round trip check is the result. Keep the age private key out of Git. A repo with the encrypted file and the private key next to it is a locked box with the key taped to the lid.
Three questions. No score is saved anywhere. This is just you against the material.
Where a secret lives and how it gets into a Pod are not two skills. They are four levels of the same job, and every team I have worked with sits on one of them. The levels are not about how many store names you can recite. They are about whether the names change what you check when a Pod holds the wrong value. Read them in order. Each level fixes the failure that hurt the most at the level below it.
At the basic level you can say why a Secret in Git is a leak, why a Secret in etcd alone is not enough, and what the three sync paths are. Rotation, audit, and one vault per team are the three reasons the secret lives outside the cluster. The operator writes a Secret, the CSI driver mounts a file, and the agent injects at Pod start. You can name which landing each path gives you. The failure that ends this level is the first Secret whose source nobody can name, and the team has no answer for where the value came from.
Types:
Uses: Any cluster where secrets must rotate, be audited, or be shared with systems outside Kubernetes.
Scale and failure: Secrets copied into Git or into many namespaces by hand multiply with every environment, and nobody can say where each copy came from. One source of truth per secret is the point of the external store.
Trade-off: An external store gives rotation and audit and adds a dependency that must be available whenever Pods start.
Do this: run the exercise at the top of the story on your own cluster. List the Secrets in one namespace and write down the source of each value. One line per Secret, one source per line.
Try it: Basic
Why does the secret live outside the cluster?
Now do it: Run the exercise at the top of the story on your own cluster: list the Secrets in one namespace and write down the source of each value, one line per Secret.
At the intermediate level you can run the two operators. You can write a SecretStore and an ExternalSecret, set the refresh interval, and read the ExternalSecret status when the Secret does not appear. You can seal a Secret with kubeseal, commit the ciphertext, and say why Git may hold it: only the controller holds the private key. You know the operator polls, so the pickup wait is up to one refresh interval, and a restart never fixes a Secret the operator has not synced. The failure that ends this level is the first rotation the operator did not pick up, and you restart Pods before you read the ExternalSecret.
Types:
Uses: GitOps clusters, where everything is declared in Git but the secret values must not be.
Scale and failure: Operators poll the store, so a rotated value reaches the cluster within one refresh interval; and environment variables in a running Pod never change, so the Pod must restart to see it. Short refresh intervals across thousands of secrets can hit the store's API rate limits.
Trade-off: Sealed Secrets need no external store and make rotation a manual re-seal; ESO gives automatic sync and adds a store to run and pay for.
Do this: run steps 1 and 2 of the lab, then change the value in the cloud store and time how long the synced Secret takes to follow. Write down the interval and the wait. That wait is the refresh interval, and now you have measured it.
Try it: Intermediate
I changed the value in the cloud store and restarted the Pods, but the Secret is still old. What did I miss?
Now do it: Run steps 1 and 2 of the lab above on your own cluster, then change the value in the cloud store and time how long the synced Secret takes to follow.
At the advanced level you can say when each store wins. Conjur when policy is code and the broker must hold the secret instead of the app. Vault when the team needs dynamic secrets and leases, and Lesson 156 is the depth for that. The cloud store when the team already pays for the cloud and the secrets live there. SOPS when the file must live in Git and only the values may be hidden. You can encrypt a file with age, decrypt it, and prove the round trip. The failure that ends this level is the first encrypted file committed next to its private key, and the ciphertext is worth nothing.
Types:
Uses: Choosing a store per team and use case: dynamic database credentials, regulated environments, cloud-native teams, or config files that must stay in Git.
Scale and failure: Dynamic secrets shrink the window an attacker can use a stolen credential, and they mean thousands of short-lived credentials whose issue and revocation must keep up with Pod churn. A private key committed beside the SOPS file it decrypts makes the encryption worthless.
Trade-off: Dynamic secrets are the most secure and the most operationally demanding; static secrets with rotation are simpler and leave a longer window if leaked.
Do this: run step 3 of the lab, then put the encrypted file in a scratch repo and confirm the plain value appears nowhere in the Git history. Then delete the repo. That check is the habit.
Try it: Advanced
I committed an encrypted file next to its private key. What is the ciphertext worth?
Now do it: Run step 3 of the lab above, then put the encrypted file in a scratch repo and confirm the plain value appears nowhere in the Git history. Then delete the repo.
At the master level you can mount a secret as a file through the CSI driver, say when the sync-as-Secret option helps and when it puts the secret back in etcd, and say how rotation reaches the Pod through the kubelet. You can state the Vault recap in one paragraph: the injector, the CSI driver, and the operator, and when each wins. And you can make the decision for a team from its size and its rotation need, using the table at the end of Go deeper. The master move is not a bigger store. It is the right path for the secret in front of you, and the alert that fires when the Pod holds a value older than the store.
Types:
Uses: Platform teams setting the secret-delivery standard for every team on the cluster.
Scale and failure: The bootstrap problem, how a Pod proves who it is to the store, is solved at scale by workload identity rather than by another secret. Syncing CSI mounts back into Kubernetes Secrets brings the value back into etcd, which may defeat the reason for using the store.
Trade-off: One standard path is easier to support and audit and fits some teams badly; a menu of paths fits everyone and multiplies what the platform team must operate.
Do this: pick one Secret on your cluster and write down its path, its store, its rotation interval, and who gets paged when the Pod holds a value older than the store. If you cannot name all four, that is the gap this level closes.
Try it: Master
I pick sync-as-Secret for a secret mounted through the CSI driver. What is the cost?
Now do it: Pick one Secret on your cluster and write down its path, its store, its rotation interval, and who gets paged when the Pod holds a value older than the store.
The operator has three objects, and each one has one job. A SecretStore names the store and how to reach it, and it lives in one namespace. A ClusterSecretStore is the same thing for the whole cluster, and any namespace may use it. An ExternalSecret names what to fetch, how often to poll, and what Kubernetes Secret to write. The refresh interval is the pickup wait. Set it to one hour and a rotation takes up to one hour to reach the Pod. Set it to one minute and the operator reads the store sixty times an hour. The right value is the rotation need, not the smallest number. A store that rotates daily does not need a one minute poll.
The operator can also build one Secret from several remote values. That is templating. The ExternalSecret carries a template, and the template writes the Secret data from the fetched values, so one database host, one username, and one password from three remote keys become one Secret the Pod reads. The status is the first thing to read when the Secret does not appear. The ExternalSecret carries Ready or NotReady, and the reason names the failure: the store cannot be reached, the key does not exist, or the service account may not read it. The app log is the wrong place to start. The operator does the fetching, so the operator knows why it stopped.
Sealed Secrets solves one problem: the value must live in Git, and Git must never hold it in plain text. kubeseal encrypts the value with the cluster's public key. The controller holds the private key, and only the controller can open it. The sealed file goes into Git like any other manifest, and ArgoCD or Flux applies it like any other manifest. Lesson 127 states the trade-off in one line: sealed secrets buy Git as the single source of truth and cost a rotation plan. This is the rotation plan it costs. The cluster key is the thing to guard. If the key is lost, every SealedSecret in Git is unreadable, and the team re-seals them all. If the key is rotated, old sealed files keep working only while the controller still holds the old key, so the team re-seals what matters and retires the old key on a schedule. The key never leaves the cluster, and the public key is safe to share, because a public key can only lock, never open.
Conjur is the enterprise store for teams that want policy as code. The Conjur server holds the secrets. The policy is a file, and the file is the access control. A policy names the groups, the hosts, and the variables, and it says which group may read which variable. The shape is below. It is a short file, and it is the whole access model.
- !policy
id: shop
body:
- !group apps
- !host shop-api
- !variable db-password
- !permit
role: !group apps
privilege: [read, execute]
resource: !variable db-password
The Secrets Provider is the bridge into the cluster. It reads Conjur and writes Kubernetes Secrets, so the Pod reads a normal Secret and never learns Conjur exists. The authenticators are how a workload proves itself. The Kubernetes authenticator trusts the Pod's service account token, so a Pod proves it is who it says it is without a static credential. Other authenticators cover the rest: a JWT from an identity provider, an OIDC login, or a cloud IAM role. The secretless broker is the pattern on top of all of it. The app never sees the secret at all. A broker holds the connection to the database, and the app talks through the broker. The secret stays in the broker, and a stolen Pod has nothing to steal. That is the strongest shape in this lesson, and it costs the most to build, because the broker is a new part to run.
AppRole is Vault's way to let a machine log in, and it takes one paragraph because the shape is small. Vault issues a role ID (role_id), which names the role and is safe to share, and a secret ID (secret_id), which is the password for that role and is not. The machine sends both to Vault, Vault checks them, and Vault hands back a token. That is the whole flow. The discipline is the delivery: the role ID can live in a config file, and the secret ID must reach the machine through a trusted channel, because anyone holding both can log in as that role. Lesson 156 names AppRole in its list of auth methods. This paragraph is the flow behind the name.
Each cloud has its own store, and each one does the same three jobs: hold the value, version it, and rotate it on a schedule. AWS Secrets Manager holds the secret and rotates it with a Lambda function the team writes. GCP Secret Manager holds the secret and versions every value, so an old version stays readable until the team deletes it. Azure Key Vault holds secrets, keys, and certificates in one vault, and it is the store the rest of the cloud already trusts. The worked sync for each one is an ExternalSecret, and the only part that changes is the remote reference.
The AWS sync is the lab, step 1. The GCP sync is below. The store is named gcp-store, and it points at the project. The remote key is the secret name in GCP, and the operator writes a Secret named api-key.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: api-key
namespace: shop
spec:
refreshInterval: 30m
secretStoreRef:
name: gcp-store
kind: SecretStore
target:
name: api-key
data:
- secretKey: key
remoteRef:
key: shop-api-key
The Azure sync is the same shape. The store is named azure-store, and it points at the vault. The remote key is the secret name in the vault, and the operator writes a Secret named db-password.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: db-password
namespace: shop
spec:
refreshInterval: 1h
secretStoreRef:
name: azure-store
kind: SecretStore
target:
name: db-password
data:
- secretKey: password
remoteRef:
key: shop-db-password
Rotation and versioning are the same idea in all three stores. A rotation writes a new version, and the old version stays readable for a grace window, so the app keeps working while the new value spreads. The operator must read the version you mean, not a pinned old one. A pinned version is a rotation that never lands, and the incident at the end of this lesson is exactly that. The keys versus config split is the rule for what goes in the store. A value that is sensitive goes in the store. A value that changes per environment and is not sensitive is config, and it belongs in a ConfigMap. A database host is config. A database password is a secret. A feature flag is config. An API key is a secret. When in doubt, ask what breaks if the value leaks. If the answer is a login, it is a secret.
SOPS encrypts files, not stores. The file lives in Git, the values are ciphertext, and the shape stays readable. The key is age or PGP. Age is the simple one: one key pair, one public key in the config, one private key file guarded like a password. PGP is the older one, and it works the same way behind a different key. Decryption happens at apply time. The pipeline decrypts the file, applies the plain text, and the plain text never lands in Git. Flux has SOPS built in: the Flux sync reads the encrypted file, decrypts it with the age key stored in the cluster, and applies the result. The key lives in the cluster as a Secret, and that Secret is the thing to guard, because anyone holding it can open every encrypted file in the repo. The lab step 3 ran the round trip on this machine, and the check is the proof: the encrypted file parses, the shape stays a Secret, and the decrypt is identical to the input.
The driver is a CSI driver that mounts secrets instead of disks. Lesson 94 teaches what a CSI driver is, and Lesson 89 teaches the CSI call chain in Go deeper, so this section teaches only the secrets half. The object is a SecretProviderClass. It names the provider and the secret to fetch. The providers are Vault, AWS, GCP, and Azure, one per store. The Pod mounts the secret as a file, and the app reads the file. The mount happens at Pod start, which is the NodePublishVolume step of the call chain, so the secret is fresh when the Pod is born.
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: shop-vault-db
namespace: shop
spec:
provider: vault
parameters:
roleName: shop-db
vaultAddress: https://vault.example.internal:8200
objects: |
- objectName: db-password
secretPath: secret/data/shop/db
secretKey: password
The Pod that uses it mounts a CSI volume, and the volume points at the class. The Pod below was checked against the Kubernetes 1.34 schemas with kubeconform, strict mode, on Nov 21, 2026. It parses and validates. It has not been run against a live API server from this machine, and it is shown as the shape only.
apiVersion: v1
kind: Pod
metadata:
name: shop-api
namespace: shop
spec:
serviceAccountName: shop-api
containers:
- name: api
image: nginx:1.27
volumeMounts:
- name: secrets
mountPath: /mnt/secrets
readOnly: true
volumes:
- name: secrets
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: shop-vault-db
The driver can also write a Kubernetes Secret from the mounted value. That is the sync-as-Secret option. It helps when the app can only read environment variables and cannot read a file. It costs the thing the CSI path was chosen to avoid: the secret lands in etcd as a normal Secret. Turn it on only when the app gives you no other way in, and say so in the file. Rotation reaches the Pod through the kubelet. The kubelet re-reads the mounted file on a timer, so the file in the Pod changes when the store changes, and the Pod sees the new value without a restart, if the app re-reads the file. An app that reads the file once at start still needs a restart. The mount moves the secret. It does not move the app's reading habit.
Vault is one store, and Lesson 156 teaches it in full: the KV store, dynamic secrets, leases, and the unseal ceremony. This lesson only says how Vault reaches a Pod, and there are three ways. The Agent Injector puts a sidecar in the Pod, and the sidecar fetches and renews the secret while the Pod runs. The CSI driver mounts the secret as a file at Pod start, and the kubelet re-reads it. The External Secrets Operator polls Vault and writes a normal Secret. The injector wins when the secret is short lived and must renew without a restart. The CSI driver wins when the secret must never sit in etcd. The operator wins when the Pod must change nothing and the team wants one sync shape for every store. No re-teaching here. The depth is in Lesson 156.
Four paths, one choice each. Team size and rotation need decide it.
| Path | Team size | Rotation need | Secret lands as | Watch out for |
|---|---|---|---|---|
| GitOps-native Sealed Secrets, SOPS | Small. One repo, one cluster. | Slow. Rotation is a new commit. | Ciphertext in Git, a normal Secret in the cluster. | The cluster key, or the age key, is the whole secret. Lose it and every file is unreadable. |
| Operator External Secrets Operator | Any size. One operator serves every namespace. | Scheduled. The pickup wait is the refresh interval. | A normal Secret in etcd. | A pinned version is a rotation that never lands. A long interval is a long stale window. |
| CSI mount Secrets Store driver | Medium and up. One driver per store. | At Pod start, then the kubelet re-reads the file. | A file in the Pod. No Secret object unless you ask. | An app that reads the file once at start still needs a restart. The sync option puts the secret back in etcd. |
| Agent injection Vault Agent Injector | Medium and up. One injector per cluster. | Continuous. The agent renews while the Pod runs. | A file or an environment variable, written by the agent. | A sidecar in every Pod. The agent is a new part to run, and its failure is the Pod's failure. |
The table is the lesson in one place. A small team with GitOps picks the GitOps-native path, because the repo is already the source of truth. A team that rotates on a schedule picks the operator, because the timer does the work. A team that must keep the secret out of etcd picks the CSI mount. A team with short lived secrets picks the agent. Say which promise is the hard one before you pick, because the sizing changes completely when it moves.
The Vault depth underneath all of it: Lesson 156 Vault & Secrets Management. The cluster's own config system, and where a synced Secret lands: Lesson 160 Kubernetes Secrets & ConfigMaps. The GitOps trade-off that Sealed Secrets answers: Lesson 127 GitOps. The CSI call chain, and what a CSI driver is: Lesson 89 Kubernetes Storage & CSI and Lesson 94 Kubernetes Components. Rotation as a practice, and the discipline behind it: Lesson 53 Least Privilege and Secrets.
A database password rotated on schedule in the cloud store, and the app kept working for an hour. Then every login failed at once. The Pods were healthy. The database was healthy. The store held the new password. The cluster held the old one. The operator had polled on its timer, reported Ready, and synced the wrong version, because the ExternalSecret pinned the version the team had tested with months earlier. The rotation had happened. The pickup had not.
Three things. First, no version pins in ExternalSecrets unless a rollback needs one, and a pinned version gets an expiry date in the same file, so the pin cannot outlive its reason. Second, an alert on the age of the synced Secret: if the store version is newer than the Secret by more than one refresh interval, it pages, because a sync that reports Ready and moves nothing is a silent failure. Third, the rotation runbook now includes a check that the Pod holds the new value before the grace window ends, not after. The fix that lasted was the alert, not the unpinning. The unpinning fixed one Secret. The alert catches the next one.
Design the secret flow for a team of 8 engineers running 12 services on 2 clusters. The team rotates database passwords every 30 days, keeps API keys in the cloud store, and deploys with GitOps. Which path carries the database passwords, which path carries the API keys, where does the rotation alert live, and what breaks first when the operator stops syncing? Name the trade-off you are making.
Start with the operator. Twelve services on two clusters is an operator job. One External Secrets Operator per cluster, one ClusterSecretStore pointing at the cloud store, and one ExternalSecret per secret. The database passwords get a refresh interval of one hour, because the rotation is monthly and an hour of stale is cheap. The API keys get thirty minutes, because a leaked key must die fast. No version pins anywhere, because the incident in this lesson proved a pin is a rotation that never lands.
Then the GitOps half. The ExternalSecrets live in Git, because the sync definitions are config, not secrets. The values never enter Git. The one service whose database password must never sit in etcd gets the CSI mount instead, with a SecretProviderClass per cluster, and the app reads the file. The rest stay on the operator, because the Pod changes nothing and the failure is one object to read.
Then the alert. The rotation alert lives on the age of the synced Secret, not on the operator health. If the store version is newer than the Secret by more than one refresh interval, it pages. The operator can report Ready while syncing the wrong version, and only the age check catches that. The check runs before the grace window ends, so the team fixes the sync while the old value still works.
Trade-offs to name. The operator against the CSI mount: the operator is simpler and the secret lands in etcd, and the CSI mount keeps it out of etcd and costs a driver per store and an app that reads a file. A one hour interval against a one minute interval: the hour reads the store rarely and leaves the Pod stale for up to an hour, and the minute finds the new value fast and reads the store sixty times an hour. Say which promise is the hard one before you size anything, because the sizing changes completely when it moves.
Saved in this browser, so the Lessons index can show your progress across the course.