Platform Engineering · Lesson 02

Speed and Volume

Latency is how long one request takes. Throughput is how many requests the system handles in a second. They sound like two separate numbers. They are not. When throughput pushes past what the system can hold, latency is the price, and the user pays it. This lesson follows one request through its time budget, hop by hop, the way I trace a slow page at 2 AM.

Difficulty: Beginner · Know this first: Lesson 01

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: in a normal request the time spreads out, and in a bad one a single hop eats almost the whole budget. Step through the lesson or switch the scenario and watch the bar change.

Extra diagram · from scratch to master

The system, mapped, and where the waiting lives

Start with one client and one server and one number. Add the boxes one at a time, and watch a queue appear at every hop.

3 kinds of latency1 queue at every hopp50 120 ms
Client
you, waiting
The wire
network time
App pod
queue in front
Cache
fast answers
Downstream
its own queue
Database
queue at the disk
Request / flowWaiting / queueResponse · the way back

Extra diagram · from scratch to master

The loop that makes everything slow

Start from the healthy system. Then the downstream slows, and each step adds one more turn of the loop until it closes.

downstream 900 msp99 1,100 msp50 stays 120 ms
Healthy trip
fast downstream
Downstream slows
900 ms now
Workers block
waiting, not working
Queue grows
line gets longer
Latency climbs
everyone waits
The loop closes
slower makes slower
p99 user
waits 1,100 ms
Timeout
the wait is cut short
Request / flowWaiting / queueFailure / dead endResponse · the way back

Try it: push the system until it bends

The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the queue, the downstream call, and the database change in the budget bar, in the hop timings, and in the total. The numbers are a teaching example, not live measurements. The shape is the lesson: push any one input too far and the total bends upward fast.

100

How fast requests arrive. The system in this example handles 200 requests per second before the database starts to strain. Utilization equals requests per second divided by 200. Database time is 60 ms at 200 requests per second or less; above that, add 0.8 ms for each request per second over 200, capped at 480 ms.

50

The downstream hop, millisecond for millisecond. At 600, every worker waits 600 ms on that one call.

20

The line made visible. Little's Law says the number of requests stuck inside the system equals how fast requests arrive times how long each one stays. Here, queue wait equals concurrent requests times 2 ms.

Try this: (1) Set requests per second to 400, downstream to 200, concurrent to 100. Queue wait is 200 ms, database is 220 ms, total is about 710 ms. (2) Set all three to the max: 1000 requests per second, 600 ms downstream, 200 concurrent. Queue wait is 400 ms, database is capped at 480 ms, total is about 1,570 ms. Same system, same code. Only the load changed.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

I take the same two numbers four times here. First I learn what each one means. Then I learn how they are tied together. Then I learn to find the loop that makes both worse. Last I set the contract that keeps the system honest.

Basic — Two numbers that are really one

Definition: Latency is how long one request takes. Throughput is how many requests the system handles in a second. They look separate. They are not. When the system fills up, the extra requests wait, and waiting is latency. A checkout page that answers in 40 ms at low traffic can answer in 900 ms at high traffic with the same code.

Types:

  • Network time, the time on the wire.
  • Service time, the time doing real work.
  • Queue wait, the time standing in line.

Uses: I use this on call when a page is slow. I ask first whether the work got slower or the line got longer. If the line got longer, I look for what is filling the system, not for a bug in the code.

Dimensions: At low traffic latency is flat and throughput climbs freely. What breaks is the limit nobody wrote down: past it, latency climbs fast while throughput stays flat. I notice it when p99 climbs and p50 stays flat, because only some users are waiting. It costs a habit of watching p99 instead of the average. The trade is that averages are easy to read and they hide the users who are hurting.

Intermediate — Little's Law and the percentiles that tell the truth

Definition: Little's Law says the requests stuck inside the system equal how fast they arrive times how long each one stays. Five hundred requests a second times 0.24 seconds is 120 requests in flight at once. I need room for at least 120, plus headroom for a spike, so I plan for 200 and alert at 80 percent.

Types:

  • p50, the lucky half.
  • p95, the unlucky twentieth.
  • p99, the user who is actually hurting.
  • Queue depth, the line made visible.

Uses: I use this when I size workers or connections before a launch. I also use it in a design review: I ask for the p99 target, not the average, and I ask what the queue depth alert is set to, because latency tells me users are already hurting and queue depth tells me they are about to.

Dimensions: As traffic grows ten times, the queue grows ten times unless I add capacity first, and the database feels it first because it is the hardest part to scale. What breaks is the connection pool or the worker pool, and I notice it in queue depth climbing before latency moves. It costs capacity planning up front and alerts on the queue, not just on latency. The trade is that headroom costs money while it sits idle, and it buys the minutes I need when traffic spikes.

Advanced — Finding the loop that makes everything slow

Definition: Most latency incidents are not one bug. They are a loop. A slow downstream call holds my workers. Held workers grow my queue. A grown queue makes p99 climb. A climbed p99 pages someone. And if my retries add load to the sick downstream service, the loop tightens. I find the loop by checking each hop in order and asking which one started it.

Types:

  • A slow dependency that never errors.
  • A queue that grows because workers are held.
  • Retries that add a second wave of traffic.
  • A fallback that serves a slightly old answer while the dependency recovers.

Uses: I use this at 2 AM when the page is slow and nothing has failed outright. Slow does not raise the same alarms as down, so I check the downstream p99, the queue depth, and the retry rate together. The hop where all three meet is the cause. The rest are symptoms.

Dimensions: As load grows, the loop feeds itself: each slow call holds a worker longer, which grows the queue, which slows the next call. What breaks is the worker pool, and I notice it when CPU is low, p50 is flat, and p99 and queue depth climb together. It costs a timeout on every dependency and a fallback I have actually tested. The trade is that a fallback costs freshness or accuracy, and it buys a fast page while the dependency is sick.

Master — The budget as a contract

Definition: At this level the numbers stop being targets and become a contract. Each dependency gets its own timeout. The queue gets a depth limit, and past it new requests get a fast error instead of a slow answer. Shedding load is not failure. It is the system choosing which requests to serve well instead of serving all of them badly.

Types:

  • Per-dependency timeouts.
  • A queue depth guard with a fast error.
  • A retry policy with backoff, only for calls that are safe to repeat.
  • A load-shedding rule written down before the incident.

Uses: I use this when I own a service other teams depend on. I write the contract down, I load test against it, and I review it after every incident. In a design review I ask what the system does when it is full, because every system gets full eventually.

Dimensions: At a hundred times the traffic I split reads and writes, I cache at the edge, and I shed load on purpose. What breaks is the contract nobody enforced: a dependency with no timeout, or a queue with no limit, and I notice it when one slow dependency takes the whole service down with it. It costs discipline: the contract has to be tested, and the test has to run before launch, not after the incident. The trade is that a fast error for a few users feels wrong, and it is better than a slow answer for everyone. One public anchor: in February 2017 a single mistyped command took down a large cloud storage service for hours, and services that depended on it slowed or failed across the internet. Their latency became its latency. That is why the contract matters.

Go deeper

Timeouts are the first trade-off. A long timeout is polite to a slow dependency and cruel to your own users, because every worker waits the full timeout before giving up. A short timeout protects your workers but can fail requests that would have finished a moment later. I set the timeout to the slowest answer I am willing to wait for, and I set it per dependency, not one number for the whole system.

A queue depth guard is the second trade-off. A deep queue absorbs a burst and pays for it in latency. A shallow queue drops work early and keeps the requests it accepts fast. I would rather fail fast with a clear error than accept a request I will answer in five seconds. Failing fast is a kindness at 2 AM. It keeps the system alive for the next request.

Retries are the third trade-off. One retry with a backoff can paper over a blip. Three immediate retries are a second wave of traffic aimed at a service that is already down. I retry only when the call is safe to repeat, and I back off. An idempotent read can be retried. A payment charge cannot.

The failure modes stack. A slow downstream call holds my workers. Held workers grow my queue. A grown queue makes my p99 climb. A climbed p99 pages someone. And if my retries add load to the sick downstream service, the loop tightens. That is the shape of most latency incidents I have worked: not one bug, but a loop.

At 10x the traffic, the numbers change but the shape stays. Ten times the requests means ten times the queue unless I add capacity first. The database is the first thing to feel it, because it is the hardest to scale. I add read replicas, I shorten the TTL on hot keys, and I check the connection pool before I add app servers. App servers are cheap. Database connections are not.

At 100x, the design changes. One region is not enough. I split reads and writes, I cache at the edge, and I shed load on purpose when the system is full. Shedding load is not failure. It is the system choosing which requests to serve well instead of serving all of them badly. At that scale the budget is not a target. It is a contract, and every hop owns its share of it.

timeouts:
  downstream_ms: 200
  database_ms: 150
  total_budget_ms: 500
queue:
  max_depth: 100
  shed_load_above: 80
  shed_with_status: 503
retries:
  max_attempts: 1
  backoff_ms: 100

What the snippet means: the downstream call gets 200 ms and no more. The database gets 150 ms. The whole request gets 500 ms. The queue holds 100 requests, and once 80 are waiting, new requests get a fast 503 instead of a slow answer. One retry at most, after a 100 ms pause.

One time it broke

What broke

A pricing lookup slowed down after a deploy on the downstream side. Nothing failed outright. The lookup just took 900 ms instead of 50. My service had no timeout on that call, so every worker waited the full 900 ms. The queue behind them grew. The checkout page slowed for everyone, and no error fired anywhere. Slow does not raise the same alarms as down, and that is what made it expensive.

Timeline

  • 02:04,the p99 alert fired. The page was slow for one user in a hundred, then one in twenty.
  • 02:11,I opened the dashboard. CPU was low, p50 was flat, p99 was climbing, and the queue depth was climbing with it.
  • 02:18,I checked the downstream service. Its p99 was 900 ms. My code was fine. My wait was the problem.
  • 02:26,we served cached prices for five minutes while the downstream team looked at their deploy. The page got fast again.
  • 02:41,p99 was back under 300 ms.
  • 03:05,the downstream team fixed the slow query and we turned the live lookup back on.

What changed after

three things. First, a 200 ms timeout on that pricing call, with a cached price as the fallback. Second, a circuit breaker that trips after five slow calls in a row, so one bad dependency cannot hold all my workers. Third, an alert on queue depth, not just latency. Latency tells me users are already hurting. Queue depth tells me they are about to.

If you remember nothing else

Design exercise

Design a product page that answers in under 300 ms at p99 while serving 500 requests per second. The page needs product details, price, and stock. Price changes often. Stock changes often. You have a cache, a database, and a pricing service.

Clarifying questions to ask

  • How fresh must the price be, to the second or to the minute?
  • How fresh must stock be, and what happens if we sell one item we do not have?
  • What should the page show when the pricing service is slow: a stale price, no price, or a slow page?
  • Is traffic steady at 500 per second or does it spike at sale time?
  • What share of requests are reads, and how many are writes?

Rough sizing

Start with the budget from this lesson. Network 30, queue 40, app 35, cache 5, downstream 50, database 60, trip back 20. That is 240 ms, which leaves 60 ms of headroom under the 300 ms target. Then size the workers. Little's Law says concurrent requests equal arrival rate times time in the system, so 500 requests per second times 0.24 seconds is 120 requests in flight at once. I need room for at least 120, plus headroom for a spike, so I plan for 200 workers or connections and I alert at 80 percent of that. Cache the product details with a short TTL, cache the price for seconds not minutes, and read stock from the database on the buy path, not the browse path.

Trade-offs to name

caching the price buys speed and costs freshness. Serving a stale price when the pricing service is slow buys availability and costs accuracy, so say it on the page. Reading stock live buys correctness and costs database load, so do it only where money changes hands. Shedding load above capacity buys a fast answer for most users and costs a failed request for a few, and that is better than a slow answer for everyone. And the target is p99, not the average. An average of 150 ms with a p99 of 900 ms fails this design.

Next in this series
Lesson 03 — Napkin Math: Back-of-the-Envelope Estimation