I have watched a service grow from one quiet box to a fleet, and I have watched that same fleet fall over because one box was doing all the work. This lesson walks that growth, step by step. You will see when a bigger box is the right call, when it stops working, and what breaks the moment you add a second box. Scaling is not about buying hardware. It is about knowing which limit you are hitting today.
Difficulty: Advanced · Know this first: Lesson 01, Lesson 02, Lesson 03
Every stage charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system. The shape is the honest part. In the steady case the total stays small. In the ceiling case one stage eats almost the whole budget. Step through the lesson or switch the scenario and watch the bar change.
Extra diagram · from scratch to master
Start with a single box and one stream of traffic. Grow it, hit its ceiling, then build the fleet and move the state out of the pods.
Extra diagram · from scratch to master
Start from the steady fleet. Traffic rises, the scaler watches, a box appears, and the load falls again.
The steps above show one fixed story. This panel lets you change the story. Move the sliders and watch the budget bar move with them. I use this same math on call. I do not guess how many boxes I need. I divide the traffic by the boxes, then I compare that number with what one box can hold.
The total traffic arriving at the service. More traffic means more work for each box, unless the replica count rises with it.
How many identical boxes share the traffic. More replicas means less work per box. One replica means one box carries everything, and one failure takes the service down.
How many requests per second one box can serve before it starts to queue and then fail. This is a teaching number for this lesson, not a benchmark for a real box. A bigger number here is a bigger or better tuned box.
Each box has room. This is the safe zone. The fleet can absorb a small spike without queuing.
Three questions. No score is saved anywhere — this is just you against the material.
I take the same growth story four times here. First I learn the two directions. Then I learn the math that picks between them. Then I learn what a fleet needs to act like one service. Last I learn where the ceilings hide.
Definition: Scaling up means a bigger box: more CPU, more memory, more disk, same one server. Scaling out means more boxes: identical replicas behind a load balancer, each carrying a share of the traffic. I have watched a service grow from one quiet box to a fleet, and the first question is always which limit I am hitting today.
Types:
Uses: I use this when a service is slow and someone says "just make it bigger." I ask what is actually full. If memory is at 95 percent and CPU is fine, a bigger box buys time. If traffic keeps growing, it only buys a little time.
Dimensions: At low traffic one box is cheap and simple, and scaling up is the fastest fix. What breaks is the ceiling: every instance family has a largest size, and traffic that keeps growing will find it. I notice it when the box is on the largest size and the memory graph still climbs. It costs a monthly bill that jumps in big steps. The trade is that one box is simple to run and it is a single point of failure: one box dying takes the whole service down.
Definition: I do not guess how many boxes I need. I divide the traffic by the boxes, then I compare that number with what one box can hold. Six hundred requests a second across three boxes is 200 per box. If one box holds 250, that is 80 percent, with 50 of headroom. I never run boxes at full capacity. I target 70 percent, so one box dying does not push the rest over the edge.
Types:
Uses: I use this math on call and in design reviews. Peak is 5,000 requests a second, one box holds 250, I target 175 per box, so I need about 29 boxes and I plan for 30. That line fits on a whiteboard and it is the line I would defend.
Dimensions: As traffic grows, the math stays the same and the box count grows with it, until the shared parts become the limit. What breaks is the hot replica: one box at 98 percent CPU while five sit near 30 percent means the work is not reaching the fleet. I notice it in traffic per replica, not in total traffic. It costs a load-per-box dashboard and an autoscaler with a slow scale down and a fast scale up. The trade is that a fleet costs more to operate than one huge box, and it buys survival when one box dies.
Definition: Adding boxes is the easy part. Making them behave as one service is the work. Sessions, cache, and files must live outside the replicas, or any replica can serve any request only on paper. Deploys must roll one replica at a time. Logs and metrics must add up across boxes. The load balancer must be right.
Types:
Uses: I use this when a fleet is misbehaving and the total traffic graph looks fine. I compare image tags and config across replicas. I watch version counts during a rollout, not just the replica count. I check traffic per replica when one box burns and the rest idle.
Dimensions: At ten times the traffic the shared database becomes the new ceiling: ten replicas hammer one database and it falls over. What breaks is the part I fixed last time but did not recheck, and I notice it when the app tier is healthy and the database connection count is pegged. It costs coordination work that never goes away: config checks, deploy watches, and a load-per-box dashboard. The trade is that I trade one big limit for a dozen small jobs, and the small jobs are the price of the fleet.
Definition: At this level I assume every part has a ceiling and I hunt for them before traffic does. The largest instance size. The database connection limit. The load balancer's own capacity. The one hot key that pulls all the traffic to one replica. I write the ceilings into the runbook with the exact command to add a replica, so the next person on call does not have to discover them at 2 AM.
Types:
Uses: I use this in a monthly capacity check: I look at load per box against the largest size and ask whether the fleet still fits. I also use it after an incident where the service ran on one box, grew quietly for months, and hit the ceiling on a night when nobody had looked at the memory graph in weeks. The fix that night was replicas, not a bigger box, because there was no bigger box left to buy.
Dimensions: At a hundred times the traffic the shape changes: more than one region, a sharding or read replica plan, and a way to shed load when full. What breaks is the ceiling nobody wrote down, and I notice it in a monthly check, not in a page, if I do the check. It costs one hour a month and the discipline to do it when nothing is on fire. The trade is that the check is boring while the service is healthy, and it is the thing that keeps a quiet growth curve from becoming a 2 AM restart loop.
The steps above give you the shape. Production adds the sharp edges. I have paid for every trade-off below at least once, usually at night. Read this section when the simple answer stops working.
At 10x the traffic in this lesson, one load balancer and one database start to matter as much as the app boxes. I stop asking how many app replicas I need and I start asking which shared part breaks first. I load test the database and the cache at 10x before I need them at 10x.
At 100x, the answers from this lesson still hold, but the shape changes. I need more than one region or more than one database copy. I need sharding or a read replica plan, and I need a way to shed load when the system is full. The rule does not change. I still find the limit in front of me and fix that one. The limits just arrive faster and they cost more when I miss them.
At both scales, the check that saves me is the same. I look at load per box, not total load. Total load hides a hot replica. Load per box shows it.
Use this Kubernetes HorizontalPodAutoscaler as the example. It scales the replica count, not the box size. It fits this lesson because it turns the simulator math into a rule the cluster runs on its own.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: checkout-api
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: checkout-api
minReplicas: 3
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 75
behavior:
scaleUp:
stabilizationWindowSeconds: 60
scaleDown:
stabilizationWindowSeconds: 300
What each part does, in plain sentences. Minimum 3 replicas means one box can die and the service still has two. Maximum 10 caps the bill and stops a runaway scaler. The CPU target of 70 percent and the memory target of 75 percent leave headroom, so the scaler acts before the box is full. Scale up waits 60 seconds, so a short spike does not cause a flurry of new boxes. Scale down waits 300 seconds, so the scaler does not remove boxes the moment load dips and then add them back a minute later.
What this config does not fix. It does not fix a hot key. It does not fix sessions stuck on one replica. It does not fix a database that cannot take ten replicas worth of connections. I check those before I trust the autoscaler.
This is the pattern I have seen more than once. The names and the system are changed. The failure is the shape of it.
A small customer facing service ran on one box. It had grown quietly for months. Nobody had looked at the memory graph in weeks. One night the traffic rose a little past the usual peak, the box ran out of memory, and the process was killed and restarted in a loop. Each restart pulled traffic back in, memory climbed again, and the process died again. The service was up between restarts and down during them, so the alerts were noisy and the cause was not obvious at first. I checked the instance family that night. We were already on the largest size it offered. There was no bigger box to buy.
You run a profile service. It serves user profiles to a web app. Peak traffic is 5,000 requests per second. Reads are almost all of the traffic. Writes are rare. The service must stay up if one box dies. You have one box today. Start with one box or a fleet? Show your sizing and name the trade-offs. You have ten minutes, a whiteboard, and no perfect answer.
Clarifying questions to ask first. What is the read to write ratio, really, and what is the latency target at p95? How big is one profile, and does traffic spike at one hour of the day or stay flat? Where does the profile data live today, and is any of it stored on the box? What is the budget, and how often do you deploy? I ask these before I size anything, because the answers change the box count.
Rough sizing. Assume one box can serve 250 requests per second at full capacity, the same teaching number as the simulator. I do not run boxes at full capacity. I target 70 percent, which is 175 requests per second per box. Peak is 5,000 requests per second. 5,000 divided by 175 is about 29 boxes. I want one spare, so I plan for 30 boxes at peak. I set the autoscaler to a minimum of 10 and a maximum of 30, so the fleet grows into the peak and shrinks at night. That math fits on one line, and it is the line I would defend in the interview.
Shape of the answer. A fleet of stateless replicas behind a load balancer. Profiles in the database, hot profiles in a shared cache, sessions and files outside the replicas. Rolling deploys, one replica at a time, with health checks that pull a bad replica out of rotation. A memory alert at 75 percent and a load per box dashboard, not just a total traffic graph.
Trade-offs to name. Thirty small boxes cost more to operate than one huge box, and they add coordination cost, cache invalidation, and deploy complexity. One huge box is simpler and cheaper to run, but one failure takes the whole service down and the largest size is a hard ceiling. A cache makes reads fast, but a cache miss at peak can stampede the database, so I need a plan for that. More replicas also mean more database connections, so I check the connection limit before I promise 30 boxes.
What a good answer sounds like. It names the bottleneck, shows the division, keeps headroom, and says what breaks first. It does not pick up or out forever. It picks the move for the limit in front of it, and it says how the answer changes when traffic doubles.
Saved in this browser only. The Lessons index will use it to show your progress across the course.