System Design · Lesson 29

Authentication and Authorization

Two questions sit at the door of every service I run. The first is "are you who you say you are". The second is "are you allowed to do this". They sound like one question. They are not. I have watched a system answer the first one perfectly and never ask the second, and a user who should only read a dashboard walked out with the power to delete it. This lesson follows one request from login to a protected action: the user logs in, gets a token, and that token is checked at the door of a service that has to decide both questions, on every request, not just once at login.

Difficulty: Intermediate · Know this first: Lesson 01, Lesson 18

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: in a healthy trip the password check and the protected action are the real work, and the verify and permission hops are cheap enough to run on every request. Switch the scenario and watch the bar change. In the expired-token scenario the refresh eats the budget. In the stolen-token scenario the trip ends early at the door, and the total is the smallest of the three, because the expensive hops never ran.

Move the token lifetime and watch the trade move

The budget above is one fixed trip. This box is yours. Move the sliders and watch the trade I live with every day: a short token life means more refreshes, a long token life means a stolen token works for longer, and more permission checks mean every request pays more. There is no setting that wins on all three. I pick the one I can defend at 2 AM.

15 minutes

How often a refresh happens, and how long a stolen token stays usable. A short TTL means more refreshes. A long TTL means a stolen token works for longer.

10000

The cost of the revocation lookup in the verify hop. More sessions means a bigger store to check when the token has to be looked up instead of verified locally.

2

The cost of the permission hop. Each extra check is one more question the service has to answer before the action runs.

Token issued: 16 ms Token verified: 6 ms Permission checked: 14 ms Refreshes per user per hour: 4 Stolen-token exposure: 15 minutes

Simulator total, normal trip: ~142 ms

4 ms of this total is the refresh cost spread across requests. The scenario above charges the full refresh only when it happens.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Go deeper

The lesson above is the happy path with one refresh and one revocation. Production is the rest of it. Here is what I actually spend my time on once login is running.

Trade-offs I weigh every time

Sessions and JWTs are the same promise with opposite costs. A session id means the server keeps the truth. Revoking it works at once, because the next request asks the server and the server says no. The price is a lookup on every request, and a store that has to stay up, stay fast, and stay in sync across every instance. A JWT means the client keeps the proof. Verifying it is local and fast, and no store has to answer. The price is revocation. A JWT that is still inside its lifetime is still valid to every verifier that trusts the signature, and the only way to kill it early is a revocation list, which is a store again. I pick sessions when revocation speed matters more than lookup cost, and JWTs when the service has to verify without calling home. I never pick one because it is fashionable. I pick it because I know which failure I am buying.

Failure modes that do not look like failures

The quietest auth failure is a permission check that only runs at login. The token carries the role, the role changes, and nothing re-checks it until the token expires. That is a user who was demoted on Monday still deleting things on Friday, and every log will say the request was authorized, because it was, by a stale token. The second failure is a signing key that rotates in one place and not in another. Half the verifiers trust the new key, half trust the old one, and the user gets a 401 every other request. The third is a token stored where a script can read it. A cross-site scripting hole becomes an account takeover, and the auth logs will show a perfectly valid login, because the token was valid. It was just stolen.

What changes at 10x

At ten times the users, the session store stops being a small table I ignore and becomes a capacity number I track. I watch its size, its lookup latency, and what happens when it is slow, because a slow session store makes every request slow, not just logins. I shorten token lifetimes a little, because ten times the users means ten times the stolen tokens, and a shorter life means a smaller window. I also stop putting permissions in the token. The token carries the user id and nothing else. The permission check reads the live role from the service that owns it, on every request. It costs more per request, and it is worth it, because roles change and tokens do not.

What changes at 100x

At a hundred times the users, I stop verifying tokens in each service by hand. The gateway does it once, at the edge, from Lesson 18, and passes the verified identity downstream in a header the services trust because only the gateway can set it. The services still check permissions themselves, because permission is a business decision and it belongs to the service that owns the data. Key rotation becomes a scheduled job with a rollout, not a manual step I do at 2 AM. Old keys stay valid for a short overlap so tokens signed minutes ago keep working, and the overlap window is written down, not remembered. And the audit log becomes a first-class system. Every sensitive action writes who did it, with which token, from where, and I can answer "who deleted this" without guessing.

The check I actually write

This is generic middleware pseudo-code, not tied to any one framework. The order is the lesson: verify first, then check permission, then run the action. Nothing runs before both pass.

// Auth middleware. Order matters: verify, then authorize, then act.
// Nothing below this runs until both checks pass.
async function protect(req, res, next) {
  const token = req.headers.authorization?.replace("Bearer ", "");
  if (!token) return res.status(401).json({ error: "no token" });

  const claims = await verifyToken(token); // signature + expiry + revocation
  if (!claims) return res.status(401).json({ error: "bad or expired token" });

  // Authorization is a live check, not a claim from the token.
  const role = await getLiveRole(claims.userId);
  if (!canDo(role, req.action)) {
    return res.status(403).json({ error: "not allowed" });
  }

  req.user = { id: claims.userId, role };
  auditLog(req.user, req.action); // sensitive actions always leave a record
  next();
}

The line I care about most is getLiveRole, not verifyToken. Verifying the token answers who you are. Reading the live role answers what you may do right now. A system that skips that line will pass every test on the day it ships and fail quietly the day someone's role changes.

One time it broke

This one stayed with me because nothing failed. Every log said the request was authorized. The problem was that authorized meant something from last week.

What broke

A user was moved off an admin team on a Monday morning. Their role changed in the directory. Their token did not. The token carried the role inside it, and the permission check trusted the token, so for the rest of the token's life that user was still an admin in every service that read it. On Friday they deleted a shared configuration object they no longer had any business touching. The delete worked. The audit log said an admin did it, because as far as the system knew, they were one. Nobody noticed until the team that owned the object found it gone on Monday and asked who had deleted it.

Timeline

What changed after

Three things, and I still do all three. First, tokens stopped carrying roles. A token carries the user id and nothing else. The permission check reads the live role from the source of truth on every request, and a role change takes effect on the next request, not the next login. Second, the login service stopped caching roles for a day. It reads the role live when it issues a token, and the cache, where it still exists, expires in minutes, not hours. Third, I added a test I run before every release that touches auth: log in as a user, change their role, and try the old action with the same token. If it still works, the release does not ship. A permission that only changes at login is a permission that does not change when it matters.

If you remember nothing else

Try it yourself

This is the kind of question I would ask in an interview, and the kind I ask myself before I change a real login flow. Take a minute with it before you read my approach.

The prompt. You run an internal ops dashboard for a company with 20,000 employees. It has three actions that matter: viewing a dashboard, changing a deploy setting, and deleting a service. Peak traffic is 2,000 requests per second. Employees change teams often, and a role change has to take effect within five minutes. Design login and authorization for this dashboard. Sessions or JWTs? Where do the checks happen? What happens when a token is stolen?

Clarifying questions to ask first

I would ask five things before I sized anything. How fast does a role change have to land, really, and is five minutes a hard rule or a wish? Which of the three actions is the dangerous one, and does it need a second step like re-entering a password? How many of the 2,000 requests per second are reads, and how many are writes? Where do employees log in from, and do I have to support logins from outside the office network? And what does the audit trail have to answer, if someone asks "who changed this deploy setting last Tuesday"?

Rough sizing, worked

2,000 requests per second, and the dashboard is mostly reads. I use short-lived JWTs for reads and a server-side session check for the two write actions, because the writes are rare and the reads are not. Token lifetime is 15 minutes, so a stolen token is usable for at most 15 minutes, and refreshes run about 4 times per user per hour. Roles are never in the token. The permission check reads the live role on every write, and on reads it reads a role cached for at most 5 minutes, so a role change lands inside the five-minute rule. The gateway, from Lesson 18, verifies the token once at the edge and passes the identity downstream. The service checks the permission itself, because the service owns the data. Deleting a service needs a password re-entry, because it is the one action I never want a stolen token alone to do.

Trade-offs to name

I chose JWTs for reads because verifying locally keeps the read path fast at 2,000 requests per second, and I accepted that a stolen read token works until it expires, because reads do not change anything. I chose a server-side check for writes because revocation speed matters more there than lookup cost, and writes are rare enough that the lookup does not dominate. I cache the role for reads for up to 5 minutes, and I accept that a demoted user can still read for up to 5 minutes, because reading a dashboard is not the same risk as changing a deploy. And I put the token check at the gateway and the permission check in the service, because identity is one question for the whole system and permission is a different question for each service that owns its own data.

Next in this series
Lesson 30 — Least Privilege and Secrets
Coming next