System Design · Lesson 05

Stateless Services

A stateless service forgets you the second your request is done. It keeps nothing in memory between requests. No session in RAM. No cart on disk. No “I saw this user before” tucked away in a local variable. That sounds like a limitation. In production it is the thing that lets me kill a pod in the middle of traffic and watch nobody notice. This lesson follows one logged-in request through a stateless service and shows where its state actually lives, because the state has to live somewhere. It just does not live in the pod.

Difficulty: Intermediate · Know this first: Lesson 01, Lesson 04

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson’s scenario. It is not a measurement from a live system. The shape is honest though. In most slow requests I have chased, one or two hops eat almost the whole budget. Step through the lesson or switch the scenario and watch the bar change. In the normal case the database is the biggest slice, which is why the cache exists. In the replica-dies case the retry is the biggest slice. In the state-in-memory case the broken session hop dwarfs everything else.

Try it: what breaks first?

I can tell you a stateless service survives a dead pod. It is better if you kill the replicas yourself and watch the numbers move. Drag the sliders. Watch which hop grows first. That hop is the one you would be paged about.

Replicas in the fleet10
Session-store latency (ms)6
Requests per second200

At 10 replicas, 6 ms session store, 200 requests per second: about 148 ms end to end.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Go deeper

Stateless sounds clean on a whiteboard. In production it is a set of trades, and every trade has a bill that arrives later. The first trade is the token. A token that carries the user id and the permissions means the replica can verify it without calling anyone. That is fast. It also means the token grows every time someone adds one more claim to it. I have seen tokens get fat enough to matter on every request. Keep the token small. Put the user id and the roles in it. Put nothing else in it. If you need the rest, fetch it from the shared store once and cache it short.

The second trade is the shared session store. Moving the session out of the pod fixes the random logout bug. It also makes the session store a dependency of every request. If that store goes down, every replica is blind at the same time. That is a fair trade, but only if you treat the store like production. It needs its own replicas. It needs its own health check. It needs an alert that fires before users notice, not after. A session store that nobody monitors is a single point of failure with a friendly name.

The third trade is revocation. A token that is valid for an hour stays valid for an hour, even after the user changes their password. That is the price of not calling a central service on every request. The fix is short token lifetimes and a refresh step, or a small denylist in the shared cache for revoked tokens. Short lifetimes mean more refresh traffic. A denylist means one more lookup. Pick one. Do not pretend the problem does not exist.

Failure modes I actually watch for: the signing key rotates and reaches only half the fleet, so half the requests fail and the graph looks random. The clocks on two replicas drift apart, so a token that is valid on one is expired on the other. The session store fills up because nobody set a TTL, and then it starts evicting live sessions. A deploy sets a pod disruption budget too tight, so the rollout stalls and old pods stay up for hours holding old code. None of these show up in a unit test. All of them show up on a dashboard at the wrong hour.

What changes at 10x scale: the session store needs connection limits. Every replica opens connections to it, and the count grows with the fleet. Set a pool size per replica and watch the wait time. The database needs the same treatment. Token verification stays local, which is the part that scales well. That is the whole point of verifying from the token instead of calling a service. The part that does not scale for free is the write path. Ten times the writes means ten times the rows, and the database is the hop that was already the biggest slice.

What changes at 100x scale: one session store in one region stops being enough. Sessions need to live near the user, or the fetch crosses the country on every request. Shard the session store by user id so one shard can grow without moving the rest. Put a hard budget on token size and enforce it in review, because at this scale a fat token is a bandwidth bill. And the idempotency key from Step 6 stops being a nice extra. At 100x, retries are constant. A retry without an idempotency key is a duplicate order waiting to happen.

This is the guardrail that keeps a deploy from killing the last healthy replica. I set it on every stateless service.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-api
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: checkout-api

minAvailable 2 means a voluntary drain or a rollout can never take the service below two healthy replicas. If the fleet is already at two, the drain waits. That is the behaviour I want. A stateless service is only disposable if there is always somewhere for the traffic to go.

One time it broke

What broke: users started getting logged out at random, right after we scaled out. Login worked fine. The next click logged them out. Then the click after that worked again. Support tickets came in waves. Some users never saw it. Some saw it every few minutes. Nothing had changed in the login code. The only change that day was traffic going up and the autoscaler adding replicas.

Timeline: 14:02, traffic spikes and the autoscaler adds four replicas to the fleet. 14:09, the first logout reports arrive. They look like user error, so support resets a few passwords. 14:20, I pull the login logs and see the pattern. Every failed request after a login landed on a new replica. Every successful one landed on the replica that handled the login. 14:35, I check the service code and find it. The session was stored in the pod’s memory. The login replica knew the user. The new replicas had never seen them. 15:10, we pin users to the replica that logged them in as a stopgap, just to stop the bleeding. 16:40, the session moves to the shared store and the pinning comes off. 17:05, the logout reports stop. No user data was lost. The carts that vanished were gone for good though. Those users had to rebuild them by hand.

What changed after: first, the session lives in the shared store, full stop, and a code review that adds session state to pod memory gets blocked. Second, I added a test that logs in on one replica and makes the next request on a different one. That test runs on every deploy now. If the session does not follow the user across replicas, the deploy fails. Third, I added an alert on the re-login rate. A spike in logins that were not preceded by a logout is the fingerprint of this bug, and I would rather get paged on the fingerprint than read about it in support tickets the next morning.

If you remember nothing else

Design exercise

Design a checkout service for a small online shop. It has to stay stateless. Users log in, build a cart, and pay. The shop expects 500 orders a minute at peak. Sketch how a request flows, where each piece of state lives, and what happens when a replica dies mid-checkout. Keep it to one page of notes.

Clarifying questions to ask

  • How long does a cart need to live if the user walks away?
  • Do we need to survive a full region outage, or just a dead pod?
  • Is the payment step allowed to retry, and who owns the idempotency key?
  • How big is a cart, and does it carry files like a prescription or a custom upload?
  • What does “logged in” mean here, a token the client holds or a session the server looks up?
Show a worked approach

Start with the token. The client holds a signed token with the user id and the roles, short lived, verified locally on every replica. No replica calls an auth service per request. The cart lives in the database, keyed by user id, not in pod memory. The session details that change fast, like the half-finished form, live in the shared cache with a 30 minute TTL. Files go to object storage and the database holds only the pointer. The payment step carries an idempotency key generated at the client, so a retry after a dead pod writes the order once, not twice. A replica dying mid-checkout costs one retry on another replica. The retry fetches the same cart from the database and the same session from the cache, so the user sees a small pause, not a lost order. Rough sizing: 500 orders a minute is about 8 a second, small, but the read traffic behind it is ten times that, so size the cache for the reads and the database for the writes. One database handles this. Two replicas of the session store handle the failure case.

Trade-offs to name

  • token size against lookup cost
  • short token lifetime against refresh traffic
  • cache TTL against stale carts
  • one database against the cost of running a second one before you need it
  • sticky sessions as a tempting shortcut that breaks the moment a replica dies.
Next in this series
Lesson 06 — Load Balancing
Coming next