I have watched a sale day go sideways in the first ten minutes. The traffic was not even that surprising. We had just planned for the average day instead of the peak hour, and the average day never shows up on sale day. This lesson follows one Black Friday traffic spike from the forecast to the recovery, hop by hop. Forecast, headroom, load test, autoscale, queue, load shedding. Every hop is a decision I made before the spike or during it.
Difficulty: Advanced · Know this first: Lesson 02, Lesson 03
Extra diagram · Architecture map
One pod and one database carrying one request. I add the load balancer and the warm instances, then the bottleneck the forecast has to name.
Extra diagram · Second trip
One number, last peak, one forecast. I follow the load test from the forecast file to the bottleneck it finds before sale day.
Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: in most spikes I have worked, one or two hops eat almost the whole budget. Step through the lesson or switch the scenario and watch the bar change. The first hop is the tap itself, so it costs no time. The forecast, the headroom, and the load test all happened before the sale, and they live in the lesson, not on this track. What you see here is the spike itself, from the tap to the answer back.
Move the three controls and watch what the spike does to the system. The budget bar above and the hop timings follow the served latency you land on. Extra latency above the 160 ms healthy baseline is spread across hops 2 to 7 in proportion to their normal-peak values. The first hop stays at 0 ms, because the tap itself costs no time, and the last hop keeps its 25 ms.
Assumptions, stated plainly: one instance serves 200 requests a second at comfortable load. The queue absorbs up to 500 requests a second before shedding starts. New instances take about 4 minutes from launch to serving traffic.
How hard the spike hits, in requests per second.
Spare capacity held warm above the instance count.
App instances warm when the spike arrives.
Instances × 200 × (1 + headroom). At the defaults: 10 × 200 × 1.5 = 3,000 requests a second. Utilization is peak divided by this number.
At the defaults the system is loafing: a third loaded, no queue, nothing shed, and served latency sits at the healthy 160 ms, the same total as the normal peak scenario.
20,000. Utilization 0.5%. Latency 160 ms, queue 0, time to scale 0 min, shed 0/sec.360 ms. Queue 0, time to scale 4 min, shed 0/sec.2,000 ms. Queue = (10,000 − 200) × 2 = 19,600 requests. Shed = 10,000 − 200 − 500 = 9,300/sec.Three questions. No score is saved anywhere — this is just you against the material.
The same topic, four times. One sale-day spike, from the first forecast to the level where I can run the sale when the forecast is wrong.
Definition: Capacity planning is deciding, before the spike, how much the system can carry and what breaks first. A load test is proving it before the spike, at the shape of real production. The average day never takes the site down. The peak hour does. I plan for the peak hour, and I start the forecast from the last sale's peak hour, never the average day.
Types:
Uses: I use this level before any big sale day. The first questions are simple: what is the peak in requests a second, which endpoints take the load, and what do I shed first. In this lesson, one instance serves 200 requests a second at comfortable load, and the queue absorbs up to 500 a second before shedding starts.
Dimensions: As load grows, the first minutes of a spike run on whatever I already have, because autoscaling is slow and new instances take about four minutes from launch to serving traffic. What breaks is the bottleneck I did not name, usually the database connection pool, sized for the normal instance count and never redone after the pool grew. What it costs is idle headroom every hour. What I give up is the cheap bill; an outage on the best sales hour costs more than the headroom.
Definition: The math is plain. Effective capacity is instances times per-instance rate times one plus headroom. At 10 instances, 200 requests a second each, and 50 percent headroom, that is 3,000 requests a second, and a 1,000 request peak leaves the system a third loaded, with no queue and nothing shed. Utilization is peak divided by that number. When utilization passes 1, latency climbs and shedding starts.
Types:
Uses: I use this level to size a real sale. A store peaking at 500 requests a second, with a sale day at ten times that for two hours, means planning for 7,500 a second with 50 percent headroom, which is about 38 instances, rounded to 40. Then the database check: size the pool for 40 instances, move reads to replicas, and test in steps, 2,500, then 5,000, then 7,500, then 10,000, holding at each level until latency is flat.
Dimensions: As load grows, the database stops being one box I tune and becomes a set of decisions, and the pool limit must be recalculated from the instance count every time the count changes. What breaks is a test that ramps in equal steps and hides the cliff, or a shed switch that was never flipped in a test. What it costs is a test environment at prod shape. What I give up is over-provisioning the database writes, which is the expensive mistake; reads scale with replicas, writes do not.
Definition: At a hundred times the traffic, the sale itself changes. I pre-warm the cache for the sale pages, I serve product pages from the content delivery network, and checkout becomes the only path that must reach the database in real time. The load test changes too, and the last thing I plan is the rollback of the plan itself: when traffic falls, the pool scales down slowly, one step at a time, so a second wave does not find me cold.
Types:
Uses: I use this level when the sale has waves, not one peak. Recommendations, product reviews, full-size images, and background emails shed first, in that order. Checkout and payment are never shed. If the test bends at the database before the planned peak, the answer is not more app instances. It is fewer queries, a bigger pool, or more replicas.
Dimensions: As load grows, every new instance is another set of database connections, and the pool is the hard limit that matches the database, not the app tier. What breaks is the scale-down: a pool that shrinks during a lull between waves leaves the second wave with nothing warm. What it costs is the pre-warm work and the replica fleet. What I give up is fast scale-down, and I get back a system that survives the second wave.
Definition: At this level the forecast will be wrong, and the system is the runbook, the shed list, and the practiced rollback of the plan. The shed list is written, tested, and printed in the runbook, with checkout last. Headroom is pre-provisioned the night before. The scaler threshold is 60 percent CPU on sale days. Every big sale gets a load test at prod shape the week before.
Types:
Uses: I use this level in the incident in this lesson, a big sale day at a retailer. The forecast came from the average day, not the last sale's peak hour. Traffic hit 2.5 times the forecast in four minutes. The autoscaler threshold sat at 85 percent CPU. New instances joined and opened more connections, and the database started refusing them. Checkout timed out forty minutes into the sale while the app servers looked fine. Someone turned off recommendations by hand from a laptop, and it helped a little, but there was no written shed list, so each switch was a debate. The sale lost its best hour.
Dimensions: As load grows, the bottleneck moves, and the only defense is naming it before sale day. What breaks is the plan that was never tested: a shed switch that was never flipped is a hope, not a control. What it costs is a load test every big sale and the headroom bought the night before. What I give up is recommendations revenue first, and I never give up checkout.
At 10x the traffic, the numbers change shape. The database stops being one box I tune and becomes a set of decisions: reads go to replicas, writes stay on the primary, and the connection pool gets a hard limit that matches the database, not the app tier. A pooler like PgBouncer sits in front so a hundred app instances do not open a thousand connections. At 100x, the sale itself changes. I pre-warm the cache for the sale pages, I serve product pages from the CDN, and checkout becomes the only path that must reach the database in real time. The load test changes too. A test that ramps in equal steps hides the cliff. I ramp in uneven steps and hold at each level, because the bottleneck shows up at a hold, not during a ramp. And I test the shed switches in the test, not on sale day. A shed switch that was never flipped is a hope, not a control. The last thing I plan is the rollback of the plan itself: when traffic falls, the pool scales down slowly, one step at a time, so a second wave does not find me cold.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: checkout-api
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: checkout-api
minReplicas: 15
maxReplicas: 60
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 60
behavior:
scaleUp:
stabilizationWindowSeconds: 60
scaleDown:
stabilizationWindowSeconds: 600minReplicas 15 is the headroom, bought before the sale. The 60 percent CPU target fires the scaler early. The 600 second scale-down window stops the pool shrinking during a lull between waves.
A big sale day at a retailer. The forecast came from the average day, not the last sale's peak hour. The autoscaler threshold sat at 85 percent CPU. The database connection pool was sized for the normal instance count, and nobody had redone the math after the pool grew. Checkout started timing out forty minutes into the sale while the app servers looked fine.
The forecast now starts from the last sale's peak hour, never the average. Headroom is pre-provisioned the night before, and the pool limit is recalculated from the instance count every time the count changes. The scaler threshold is 60 percent CPU on sale days. The shed list is written, tested, and printed in the runbook, with checkout last. And every big sale gets a load test at prod shape the week before, not the month before.
The prompt: An online store normally peaks at 500 requests a second. A sale day is coming and marketing expects 10 times that, for about two hours. Plan the capacity: the forecast, the headroom, the load test plan, and what you shed first.
Forecast 5000 requests a second at peak, then plan for the top of the range, not the middle. Buy 50 percent headroom, so provision for 7500 a second. At 200 requests a second per instance, that is 7500 divided by 200, which is 37.5, so 38 instances, and I round to 40 for a round number and a little slack. The database is the check that matters: size the connection pool for 40 instances, move reads to replicas, and put a pooler in front so the connection count stays flat. Load test in steps: 2500, then 5000, then 7500, then 10000, holding at each level until latency is flat. The test passes only if the database stays under its latency budget at 10000. Shed list, in order: recommendations, product reviews, full size images, background emails. Checkout and payment are never shed. If the test bends at the database before 7500, the answer is not more app instances. It is fewer queries, a bigger pool, or more replicas.
Idle headroom costs money every hour, but an outage on the best sales hour costs more. Over-provisioning the database writes is the expensive mistake; reads scale with replicas, writes do not. Shedding recommendations costs some revenue, but losing checkout costs all of it.