A customer watches an order page. The status flips from “Preparing” to “Out for delivery.” Nobody pressed refresh. That flip crossed nine hops to get to the screen, and every hop had a chance to drop it, delay it, or deliver it twice. This lesson follows one status change from the source event to a browser screen, and shows what polling, long-polling, server-sent events, and WebSockets each cost.
Difficulty: Advanced · Know this first: Lesson 01, Lesson 16
Every hop charges rent in milliseconds. The bar below is a worked example for this lesson’s scenario — not a measurement from a live system — but the shape is honest: in a push system the fan-out hop is the one that grows with your audience, and in a polling system the trip looks cheap because the real cost is the wait between polls, which never shows up in a millisecond bar. Step through the lesson or switch the scenario and watch the bar change.
Extra diagram · from scratch to master
Start with one client polling one server on a timer. Add the push gateway and the open connection, then the broker topic, then thousands of clients.
Extra diagram · from scratch to master
Start from the calm push system. Then a routine deploy restarts the push gateway at lunch peak, and every client reconnects at the same moment.
Move the sliders. The budget bar, the fan-out hop, and the status lines update live.
How many screens are watching right now.
How many status changes happen each second.
Only matters in poll mode. Push ignores it.
Fan-out deliveries / sec
200,000
Fan-out hop (push)
12 ms
Idle connection memory (push)
500 MB
Poll requests / sec
333
Average staleness (poll)
15 s
Trip total on the bar above
~72 ms
Three questions. No score is saved anywhere — this is just you against the material.
The lesson above follows one status change from the source event to a browser screen. This section covers the same trip four times, from the first poll to the level where ten thousand screens stay fresh through a gateway restart.
Definition: Polling means the client asks again later. Push means the server speaks the second something changes. A status page that polls every 30 seconds is up to 30 seconds stale, and on average 15 seconds stale. A push page is fresh the moment the event lands. The choice is a freshness question first and a technology question second.
Types:
Uses: I use this level when I pick the transport. If the server does the talking and the client only watches, I pick server-sent events. If the client talks back, like in chat, I pick WebSockets. If the freshness need is loose, polling at 30 seconds is honest and simple, and I say plainly that those clients see updates up to 30 seconds late.
Dimensions:
Definition: Fan-out is a multiplier. One event times ten thousand clients is ten thousand deliveries, and that is the number to size for. A push gateway holds the open sockets, and each pod only knows about its own. When a client connects to pod A and the event arrives on pod B, somebody has to route between them. That somebody is usually a shared presence store, and it becomes the new bottleneck.
Types:
Uses: I use this level when I size the system. I start with the fan-out math, not the transport. 200 status changes a second sounds small until you multiply it. Most orders have two or three watchers, so the real fan-out is about 200 times 3, roughly 600 deliveries a second, not 200 times 50,000. That changes the whole sizing.
Dimensions:
Definition: A connection that drops is normal. The design is what happens next. Every update carries a sequence number. The client applies in order, drops anything older than what is on screen, and asks for a replay from its last sequence when it reconnects. A reconnecting client waits with exponential backoff and jitter, so ten thousand clients do not all knock at once.
Types:
Uses: I use this level the day a deploy restarts the gateway at lunch peak. Every client notices the drop at the same moment. The backoff spreads the reconnects over a minute instead of a second. The gateway caps how fast it accepts new connections. The fan-out queue stops growing, and the screens catch up in a wave.
Dimensions:
Definition: At this level the client is part of the system, not the end of it. I shard the fan-out by region so an event never crosses the country to reach a watcher next door. At a hundred times the scale, I stop fanning out from one gateway entirely and push from the edge, close to the client. And I measure to the paint, not to the send, because the user sees the screen, not my server log.
Types:
Uses: I use this level when the alert that matters fires. CPU looked fine the whole time in this lesson's incident. The queue was the truth. Fan-out lag is the alert I page on, not gateway CPU, because a gateway can be idle and still four minutes behind.
Dimensions:
Push feels instant until it is not. The first thing that breaks at scale is the gateway’s connection list. A stateless service scales by adding pods. A push gateway holds ten thousand open sockets, and each pod only knows about its own. When a client connects to pod A and the event arrives on pod B, somebody has to route between them. That somebody is usually a shared presence store, and it becomes the new bottleneck. I have watched a team scale gateways beautifully and then fall over on the presence lookup they forgot to size.
Backpressure is the second thing. Fan-out is a queue, and a queue that grows without a limit will eat the gateway’s memory and then the gateway. Shed load early. Drop a slow consumer before it drowns the fast ones. A client that cannot keep up gets a resume point and a replay, not an infinite buffer.
Server-sent events deserve a real look before you reach for WebSockets. SSE is plain HTTP, it reconnects by itself, and it is one way, which is all a status page needs. WebSockets are two way, which you need for chat, not for watching an order. Two way costs you heartbeats, ping frames, and a harder proxy story. I pick SSE when the server does the talking and WebSockets when the client talks back.
At ten times the scale, shard the fan-out by region so an event never crosses the country to reach a watcher next door. At a hundred times, stop fanning out from one gateway entirely and push from the edge, close to the client. The shape stays the same. Only the distance changes.
And the client is part of the system, not the end of it. Every update carries a sequence number. The client applies in order, drops anything older than what is on screen, and asks for a replay from its last sequence when it reconnects. Without that, a flaky mobile connection will paint lies on the screen and nobody on the server will ever know.
let lastSeq = 0;
let attempts = 0;
function connect() {
const ws = new WebSocket("/updates?since=" + lastSeq);
ws.onmessage = (e) => {
const msg = JSON.parse(e.data);
if (msg.seq !== lastSeq + 1) { resync(); return; }
lastSeq = msg.seq;
applyUpdate(msg);
};
ws.onclose = () => {
attempts++;
setTimeout(connect, backoff());
};
}
function backoff() {
const wait = Math.min(30000, 1000 * 2 ** attempts);
return wait + Math.random() * 1000; // jitter spreads the storm
}The two lines that matter are the sequence check and the jitter. The sequence check stops out-of-order paints. The jitter stops ten thousand clients from reconnecting in the same second.
A routine deploy restarted the push gateway at lunch peak. Every connected client noticed the drop at the same moment and reconnected at once. The gateway accepted handshakes faster than it could register them, the fan-out queue filled with new events and replay requests at the same time, and order status updates started arriving four minutes late. Screens froze on old statuses. Customers called support to ask where their food was. The food was on its way. The screen just did not know.
Clients now reconnect with exponential backoff and jitter, so a gateway restart spreads reconnects over a minute instead of a second. The gateway caps how fast it accepts new connections and sheds slow consumers instead of buffering them forever. Every update carries a sequence number, and a reconnect replays from the last sequence instead of re-reading everything. And the alert that matters is fan-out lag, not gateway CPU. CPU looked fine the whole time. The queue was the truth.
Design live order tracking for a food delivery app. 50,000 customers watch their orders at peak. The service records about 200 status changes per second. Clients are mostly on flaky mobile networks. How do you get every status change to the right screen, quickly, without melting the backend?
Start with the fan-out math, not the transport. 200 updates a second sounds small until you multiply it. Most orders have two or three watchers: the customer, the courier, maybe the restaurant. So the real fan-out is about 200 × 3, roughly 600 deliveries a second, not 200 × 50,000. That changes the whole sizing. Use server-sent events, since the flow is one way and SSE reconnects by itself. Hold connections on a push gateway, shard it by region, and keep the connection list in a shared presence store so any gateway pod can reach any client. Give every update a sequence number per order. On reconnect, the client sends its last sequence and gets a replay of only what it missed. Keep a replay window of at least fifteen minutes in the broker, because a phone in a dead zone comes back late. Poll as a fallback for clients behind proxies that kill long-lived connections, at a 30-second interval, and say plainly that those clients will see updates up to 30 seconds late.
Push costs idle connection memory on the server and a harder gateway to scale; polling costs request volume that grows with clients instead of updates. SSE is simpler than WebSockets but cannot carry client-to-server messages, so anything interactive needs a second channel. A longer replay window costs broker storage and makes catch-up slower; a short one forces full re-reads. Perfect freshness for every client is not the goal. Freshness for the clients who are actually watching, without falling over when they all reconnect at once, is.
Saved in this browser only. It tracks your progress through the series on this device.