I once watched a deploy ask for a database password it should never have had. The code was not evil. It was copied from an older service, and the older service had a broad role, so the new one got the same role. Nobody checked what the code actually needed. It needed to read three tables. It could read the whole database, write to it, and delete from it. That is this lesson. I follow one deploy's request for a secret, from the code asking for it to the token dying at the end. Least privilege means the smallest permission that still lets the job run. A secret means a password, a key, or a token that must never live in code, in an env dump, or in a log. Short-lived credentials shrink the damage when one leaks. Every access gets logged. Rotation is boring and automatic, not a 2 AM event.
Difficulty: Advanced · Know this first: Lesson 29, Lesson 26
Extra diagram · Architecture map
One service asking for one secret. I add the service account and the policy check, then I put the broad role beside it for comparison.
Extra diagram · Second trip
A token dies and a fresh one arrives, boring and automatic. Then a secret leaks, and the audit log earns its keep.
Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: in a healthy trip the role check and the database are the real work, and the identity and token hops are cheap enough to run every time. Switch the scenario and watch the bar change. In the broad-role scenario the trip gets faster because the checks stop checking. In the leaked-secret scenario the trip gets much slower because revocation and rotation have to happen under pressure.
The budget above is one fixed trip. This box is yours. Move the sliders and watch the trade I live with every day: one secret shared by many services means one leak hits them all, a long token life means a stolen token works for longer, and more permissions per role means a leak reaches further. There is no setting that wins on all three. I pick the one I can defend at 2 AM.
How many services one leaked secret reaches, how long revocation takes, and how much work the identity check does.
How long a stolen secret stays usable, and how often rotation runs. A short lifetime means more rotations. A long lifetime means a stolen secret works for longer.
How much the role check has to evaluate, and how far one leaked token reaches.
The simulator works on the normal scenario. When it is active, the identity, role, and token hops in the diagram and budget above follow the sliders. Switch to another scenario to see its fixed timings again.
Three questions. No score is saved anywhere — this is just you against the material.
The same topic four times. First the smallest permission, then the short life, then the failures that never page, then secrets as a platform.
Definition: Least privilege means the smallest permission that still lets the job run. If a service can do something it never needs, the role is wrong. A secret is a password, a key, or a token that must never live in code, in an env dump, or in a log. Code asks for a secret by name, and the secret itself stays in the secret manager. Short-lived credentials shrink the damage when one leaks. Every access gets logged. Rotation is boring and automatic, not a 2 AM event.
Types:
Uses: I use it the way the lesson opens. A deploy asks for a database password it should never have had. The code was not evil. It was copied from an older service, and the older service had a broad role, so the new one got the same role. Nobody checked what the code actually needed. It needed to read three tables. It could read the whole database, write to it, and delete from it. That is the whole lesson in one deploy.
Dimensions: One secret shared by 20 services means one rotation, one policy, one thing to manage, and one leak that hits 20 services. Separate secrets mean 20 rotations and 20 policies. The cost is a secret manager and the discipline to ask for a secret by name. The trade is convenience against blast radius. I would rather rotate 20 small secrets than explain one big leak.
Definition: A long-lived password works for years, and it works for an attacker for years. A short-lived token is safer and busier. The service has to ask again every day, rotation has to run every day, and a failed rotation breaks the service at 2 AM. I pay that cost because the leak window shrinks from years to a day. The scoped policy is the same idea for permissions. One service, one secret, three tables, read only, one day. Everything not listed is denied. That is the default.
Types:
Uses: I use this for the checkout read service in this lesson. It reads three tables, read only, for one day, and the policy says so in plain words. I use the template when a new service is created, because the template starts narrow and a service that needs more asks for it, and the ask is reviewed. I use the explicit deny line, because a policy with only an allow list grows one permission at a time until it is broad again.
Dimensions: As services grow, rotation becomes a scheduled job with an alert, not a manual step. The failure I watch for is rotation that works in the scheduler and fails in the pickup. The job runs, the fresh secret exists, and the service keeps using the old one until it dies. Then the service fails at 2 AM, and the on-call engineer finds a rotation job that says success and a service that says access denied. The cost is the rotation job and the alert. The trade is a busy, short-lived credential against a quiet, long-lived one that an attacker can use for years.
Definition: The quietest secrets failure is a secret that works but is too broad. Nothing errors. Nothing pages. The service runs fine for a year with a role that can delete the database, and the only evidence is a policy file nobody reads. The second failure is a secret in a log. The service works fine. The log works fine. The secret is sitting in a file that gets copied to a shared drive, and nobody knows until an audit finds it. The third failure is the rotation that succeeds and the pickup that fails, from the tier above. In 2021, the Codecov breach showed the same shape in public: an attacker got credentials out through a tampered build script, and the credentials worked because they were broad enough to be useful.
Types:
Uses: I use this in the incident from this lesson. A service account was shared by 12 services, and its role could read and write the whole database. A config change pointed one service at the wrong table, and it overwrote rows it should never have touched. The write worked, because the role allowed it. The audit log named the shared account, so it could not say which of the 12 services made the write. I use the pipeline check after that, so a broad role cannot slip through in a copy-paste again.
Dimensions: At ten times the services, I stop managing secrets by hand. The audit log becomes a dashboard I look at every week, not a file I open after an incident. I watch two numbers: how many services share one secret, and how many permissions the broadest role has. Both numbers should go down over time, not up. The cost is the pipeline check and the weekly dashboard review. The trade is strictness against deploy friction. A deploy that stops for a review is slower. A deploy that ships a broad role is a breach waiting for a reason.
Definition: At a hundred times the services, the secret manager is a system of its own, with its own on-call rotation and its own backup. I stop letting services read secrets directly. The platform injects a short-lived token at startup, and the service never sees the password at all. Revocation becomes a single call that kills a token everywhere at once, because at this scale I cannot chase 100 services one by one. The audit log becomes a first-class system with its own retention and its own access control, because the log that says who had which secret is itself sensitive.
Types:
Uses: I use this when the secret count outgrows hand management and the revocation question becomes the whole design. I use it when I revoke the shared account and issue 12 separate service accounts, one per service, each with a narrow role, the way the incident in this lesson ended. And I use it when I test revocation the way I test backups. A revocation that has never been tested is a revocation I do not trust.
Dimensions: The secret manager is now a system with its own on-call rotation, its own backup, and its own failure modes. If it is down, services cannot start, so its availability is the availability of every service that depends on it. The cost is a dedicated system and a dedicated rotation. The failure mode is a platform outage that stops every service from getting its token at once. The trade is platform control against service autonomy. I take the platform, because at a hundred services, one revocation call beats a hundred manual ones, and the audit log is the only record that says who had what.
The lesson above is one deploy asking for one secret. Production is the rest of it. Here is what I actually spend my time on once secrets are running.
A long-lived password is simple and dangerous. I store it once, the service reads it, and it works for years. The price is the leak window. A password that works for years works for an attacker for years. A short-lived token is safer and busier. The service has to ask again every day, rotation has to run every day, and a failed rotation breaks the service at 2 AM. I pay that cost because the leak window shrinks from years to a day. A shared secret is simple and dangerous in the same way. One secret for 20 services means one rotation, one policy, one thing to manage. The price is the blast radius. One leak hits 20 services. Separate secrets mean 20 rotations and 20 policies. I pay that cost too, because I would rather rotate 20 small secrets than explain one big leak.
The quietest secrets failure is a secret that works but is too broad. Nothing errors. Nothing pages. The service runs fine for a year with a role that can delete the database, and the only evidence is a policy file nobody reads. The second failure is a secret in a log. The service works fine. The log works fine. The secret is sitting in a file that gets copied to a shared drive, and nobody knows until an audit finds it. The third failure is rotation that works in the scheduler and fails in the pickup. The job runs, the fresh secret exists, and the service keeps using the old one until it dies. Then the service fails at 2 AM, and the on-call engineer finds a rotation job that says success and a service that says access denied.
At ten times the services, I stop managing secrets by hand. Every service gets its own service account and its own narrow role, created by the same template, not copied from an older service. The template gives read access to that service's own tables and nothing else. Rotation becomes a scheduled job with an alert when it fails, not a manual step. The audit log becomes a dashboard I look at every week, not a file I open after an incident. I watch for two numbers: how many services share one secret, and how many permissions the broadest role has. Both numbers should go down over time, not up.
At a hundred times the services, the secret manager is a system of its own, with its own on-call rotation and its own backup. I stop letting services read secrets directly. The platform injects a short-lived token at startup, and the service never sees the password at all. Revocation becomes a single call that kills a token everywhere at once, because at this scale I cannot chase 100 services one by one. The audit log becomes a first-class system with its own retention and its own access control, because the log that says who had which secret is itself sensitive. And I test revocation the way I test backups. A revocation that has never been tested is a revocation I do not trust.
This is a scoped policy plus a rotation setting, in the shape I use in production. The point is the scope: one service, one secret, three tables, read only, one day. Everything else is denied by default.
# Scoped secret policy for the checkout read service.
# One service, one secret, three tables, read only, one day.
# Everything not listed here is denied. That is the default.
service_account: checkout-read
secret_name: checkout-db-read-token
allow:
- action: db.read
tables: [orders, payments, items]
scope: checkout-db
deny: all-other-tables
token:
lifetime: 24h
renewable: true
rotation:
schedule: daily 03:00
automatic: true
alert_on_failure: on-call
audit:
log_every_access: true
store: tamper-proof-audit-log
The line I care about most is deny: all-other-tables. The allow list says what the service can do. The deny line says the rest is not an accident. A policy with only an allow list and no explicit deny is a policy that grows one permission at a time until it is broad again.
This one stayed with me because nothing failed. Every service was up. The database was up. The problem was a secret that could do far more than anyone remembered, and a log that could not say who had used it.
A service account was shared by 12 services, and its role could read and write the whole database. The account was created years ago for one service, then copied to the next service because it was easier than making a new one, then copied again. Nobody trimmed it. One evening a config change in one of the 12 services pointed it at the wrong table, and it overwrote rows it should never have touched. The write worked, because the role allowed it. The audit log named the shared account, so it could not say which of the 12 services made the write. We knew a bad write had happened. We did not know which service did it, and we did not know how many rows it had touched until we diffed the backup.
Three things, and I still do all three. First, no service account is shared. Every service gets its own account, created from a template, and the template starts narrow. A service that needs more asks for it, and the ask is reviewed. Second, the audit log names the service, not just the account. The log entry carries the service name, the pod name, and the secret name, so the next incident has a straight line from a bad write to the service that made it. Third, I added a check to the deploy pipeline. If a role allows write access to a table the service has never written to in the last 30 days, the deploy stops and asks for a review. A broad role cannot slip through in a copy-paste again, because the pipeline now reads the role before the code ships.
This is the kind of question I would ask in an interview, and the kind I ask myself before I change a real secrets setup. Take a minute with it before you read my approach.
You run a checkout platform with 20 services. Orders, payments, inventory, pricing, shipping, and refunds each have their own service. Each service talks to the database and to two or three other services. Today every service shares one service account, and that account can read and write the whole database. A secret leaked last month and took a day to revoke. Design secrets handling for this platform. Who gets which secret, how long do credentials live, how does rotation work, and what happens the next time a secret leaks?
I would ask five things before I sized anything. Which services actually write, and which ones only read? A read-only service should never hold a write credential. How many secrets are there today, and how many services share each one? What does the audit log have to answer after a leak, and can it answer it today? How fast does revocation have to be, and is a day acceptable or is an hour the real number? And what happens when rotation fails? Does the service keep working on the old credential, or does it fail, and who gets paged?
20 services, one database, about 30 tables. I give each service its own service account and its own narrow role, created from one template. A read service gets read access to its own tables only. A write service gets write access to its own tables only. That is 20 accounts, 20 roles, and about 60 permissions in total, an average of 3 per role. Each service gets a short-lived token with a 1-day lifetime, issued by the secret manager at startup and renewed automatically. Rotation runs daily at 03:00, automatic, with an alert to on-call if it fails. The audit log records every access with the service name, the secret name, and the time. One leaked secret now reaches 1 service and about 3 permissions, not 20 services and the whole database. Revocation is one call to kill the token, and it takes about 7 minutes, not a day, because only one service shares that secret.
I chose 20 separate accounts because the blast radius matters more than the management cost, and I accepted 20 rotations a day because they are automatic. I chose a 1-day token lifetime because a leaked token then works for at most a day, and I accepted the daily renewal traffic because renewal is cheap and a failed renewal is visible in the morning, not hidden for a year. I chose to log every access because the log is how I scope the next leak, and I accepted the log volume because the audit store is cheap compared to an unscoped incident. And I chose read-only roles wherever a service only reads, because a read-only service that gets compromised can leak data but cannot overwrite it, and that difference is the whole design.