System Design · Lesson 06

Load Balancing

One server cannot carry a real service. Sooner or later it fills up, or it dies at 2 AM, and every user finds out at the same time. The fix is not a bigger server. It is several servers and one box in front of them that decides, request by request, who does the work. That box is the load balancer. This lesson follows one request from the client, through the balancer, to a backend and back, and shows what happens when a backend dies, when a health check lies, and when the balancer itself is the thing that breaks.

Difficulty: Intermediate · Know this first: Lesson 04, Lesson 05

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: in a healthy trip the backend does the real work and the balancer's hops are nearly free. Switch the scenario and watch the bar change. In the lying health check scenario, one hop eats almost the whole budget, and it eats it by waiting on a backend that will never answer.

Move the fleet and watch the average move

The budget above is one fixed trip. This box is yours. Move the sliders and watch what happens to the average request. I learned this the slow way: the balancer is not magic. It just shares work, and how it shares work decides how slow the average user feels.

Healthy backend latency after load sharing: 40 ms.

Round robin: the slow backend gets 1 in 6 requests (about 17%). Every request counts the same, fast or slow.

Average backend work is now about 67 ms, so the whole trip is about 127 ms. The slow backend is not just slow for its own users. It pulls the average up for everyone.

The average is climbing. One slow backend is now a real part of what users feel.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Go deeper

The lesson above is the happy path with one failure. Production is the rest of it. Here is what I actually spend my time on once the balancer is running.

Trade-offs I weigh every time

Layer 4 is fast and blind. Layer 7 is slower and can see the request. I do not pick the smart one by default. I pick it only when the routing decision needs the path, a header, or a cookie. Sticky sessions are the same kind of deal. Pinning a user to one backend is simple and it breaks the whole idea of sharing work. I use it only when the session really cannot live anywhere else, and then I fix the session storage so I can drop the pinning later. Connection draining is the quiet one people skip. When I pull a backend out of the pool, I let its open requests finish instead of cutting them off. It costs a few seconds on a deploy. It saves a wave of errors that look like a bug and are really just my own deploy.

Failure modes that do not look like failures

A slow backend is worse than a dead one. A dead backend fails a check and leaves the pool. A slow backend passes the check and keeps getting fed. That is the pattern in this lesson, and it is the one I watch for in dashboards: one backend with rising latency and rising connections while the rest stay flat. The balancer saturating is the second one. Every request flows through it, so when it runs out of connections or CPU, every backend looks broken at once. The fix is not more backends. It is a bigger balancer, or a second one. The third failure mode is the check itself. A health check that only asks if the process is alive will say yes to an app that cannot serve a single real request. I write checks that do real work: hit the database, call the one downstream service users need. If any of those fail, the check fails.

What changes at 10x

At ten times the traffic, the balancer stops being a box I think about once a year and becomes a capacity number I track every week. I watch connections per backend, not just requests per second. I set a slow start on new backends, so a fresh replica does not get full traffic in its first minute while its caches are cold. I also stop hand-tuning weights. I let the algorithm do it, and I fix the slow backend instead of routing around it for months.

What changes at 100x

At a hundred times the traffic, one balancer pair is not enough. I run balancers in more than one place, and I spread traffic across them before it reaches any one box. The health checks get stricter, not looser, because one bad backend at that scale takes down a visible slice of users in seconds. And the failover test stops being a nice-to-have. I test it on a schedule, on a quiet day, with someone watching the error rate. A standby I have never failed over to is a hope, not a plan.

Health-check config I actually use

# Health check: prove the app can serve, not just that the process is alive. interval: 5s timeout: 2s healthy_after: 2 passes unhealthy_after: 2 failures endpoint: /healthz # /healthz must check the database and the one downstream call users need. # If any of them fails, /healthz returns 503. # A backend that fails twice in a row leaves the pool. # A backend that passes twice in a row comes back, slowly.

Two passes to come back matters. A backend that flaps in and out of the pool is worse than one that stays out. I would rather wait ten seconds for a clean recovery than watch users bounce onto a backend that is still half dead.

One time it broke

This one stayed with me because nothing looked broken. Every dashboard was green. Users were getting errors anyway.

What broke

One backend in a pool of six stopped serving real requests. Its database connection had died after a deploy, so every real request failed. But the health check still passed. The check hit the root path and only asked if the process answered. The process answered fine. So the balancer kept sending one in six requests to a backend that could not do any work. Users hit it at random. Some got a fast answer. Some got an error. The team kept saying it was flaky, because it was. It was flaky in exactly a one-in-six pattern, and nobody read the pattern.

Timeline

What changed after

Three things, and I still do all three. First, the health check changed. It now hits a dedicated endpoint that runs a real database query and calls the one downstream service the app needs. If any of those fail, the check returns 503. A process that is alive but cannot serve is not healthy. Second, I added an alert on error rate per backend, not just on the pool total. A pool total hides one bad backend behind five good ones. A per-backend alert catches it in minutes. Third, I added a rule for deploys: after a deploy to one backend, I watch that backend's real error rate for ten minutes before the deploy rolls to the next one. A deploy that breaks one backend quietly is a deploy I want to catch on the first backend, not the sixth.

If you remember nothing else

Try it yourself

This is the kind of question I would ask in an interview, and the kind I ask myself before I change a real balancer. Take a minute with it before you read my approach.

“You run a checkout service. It gets 2,000 requests per second at peak. Each request takes about 50 ms on a healthy backend. One backend can handle about 400 requests per second before its latency climbs. Design the balancing for this service. How many backends do you run, which algorithm do you pick, and what do you do when one backend gets slow?”

Clarifying questions to ask first

I would ask four things before I sized anything. Is the traffic steady or spiky, and how long does the peak last? Are the requests all the same, or do some cost ten times more than others? Does a user need to land on the same backend twice, or can any backend serve any request? And what happens if a request fails once: can the client retry it safely, or would a retry charge the card twice?

Rough sizing, worked

2,000 requests per second divided by 400 per backend is 5 backends at full load. I never run at full load. I run at about 60 percent, so one backend dying does not push the rest over the edge. 2,000 divided by 240 is about 8. So I run 8 backends, and I plan for 7 to carry the peak if one dies. The balancer sits in front of all 8 as an active and standby pair, so the balancer itself is not the single point of failure.

Trade-offs to name

I pick least connections, not round robin, because checkout requests are not all the same cost. A slow backend under round robin keeps getting fed while its queue grows. Least connections sends new work to the backends that are free. I keep the health check deep: it must prove the app can reach the database and the payment call, not just that the process answers. I add connection draining on deploys, so in-flight checkouts finish instead of failing mid-payment. And I set a timeout that is shorter than the user's patience. A request that waits 3 seconds on a dead backend is worse than a fast error the client can retry safely, if the retry is safe. If the retry is not safe, I do not retry at all. I fail fast and I log it.

Next in this series
Lesson 07 — Caching Strategies
Coming next