Platform Engineering · Lesson 17

Live Data, As It Happens

A customer watches an order page. The status flips from “Preparing” to “Out for delivery.” Nobody pressed refresh. That flip crossed nine hops to get to the screen, and every hop had a chance to drop it, delay it, or deliver it twice. This lesson follows one status change from the source event to a browser screen, and shows what polling, long-polling, server-sent events, and WebSockets each cost.

Difficulty: Advanced · Know this first: Lesson 01, Lesson 16

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson’s scenario — not a measurement from a live system — but the shape is honest: in a push system the fan-out hop is the one that grows with your audience, and in a polling system the trip looks cheap because the real cost is the wait between polls, which never shows up in a millisecond bar. Step through the lesson or switch the scenario and watch the bar change.

Extra diagram · from scratch to master

The push system, mapped: gateway pods and ten thousand lines

Start with one client polling one server on a timer. Add the push gateway and the open connection, then the broker topic, then thousands of clients.

10,000 connected clientspoll: 333 requests a second, mostly nothingpush: server speaks the second it changes
Ordering service
source of the change
Broker topic
updates keyed by order id
Push gateway
more than one pod
Presence store
which client is on which pod
10,000 clients
each holding one open line
Poll branch
30-second timer, mostly nothing
Push branch
server speaks at once
Request / flowBuild / fixWaiting / queueResponse · the way back

Extra diagram · from scratch to master

The reconnect storm: ten thousand clients come back at once

Start from the calm push system. Then a routine deploy restarts the push gateway at lunch peak, and every client reconnects at the same moment.

updates 4 minutes latescreens freeze on old statusescatch-up: I last saw event 4812
Normal push
one change, 10,000 screens, fast
Gateway restarts
routine deploy, lunch peak
Everyone reconnects
at the same moment
Handshakes pile up
faster than they can register
Queue fills
new events plus replay requests
Screens freeze
four minutes late
Recovery
reconnects spread out, queue drains
Catch-up
only what it missed, from 4812
Request / flowWaiting / queueFailure / dead endBuild / fixResponse · the way back

Try it: push vs poll

Move the sliders. The budget bar, the fan-out hop, and the status lines update live.

How many screens are watching right now.

How many status changes happen each second.

Only matters in poll mode. Push ignores it.

Transport:

Fan-out deliveries / sec

200,000

Fan-out hop (push)

12 ms

Idle connection memory (push)

500 MB

Poll requests / sec

333

Average staleness (poll)

15 s

Trip total on the bar above

~72 ms

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

The lesson above follows one status change from the source event to a browser screen. This section covers the same trip four times, from the first poll to the level where ten thousand screens stay fresh through a gateway restart.

Basic — Poll or push, and what the screen sees

Definition: Polling means the client asks again later. Push means the server speaks the second something changes. A status page that polls every 30 seconds is up to 30 seconds stale, and on average 15 seconds stale. A push page is fresh the moment the event lands. The choice is a freshness question first and a technology question second.

Types:

  • Polling — the client asks on a timer, and pays a request even when nothing changed.
  • Long-polling — the server holds the request open until there is something to say, then the client asks again.
  • Server-sent events — plain HTTP, one way, and the browser reconnects by itself when the connection drops.
  • WebSockets — two way, which you need when the client talks back, not when it only watches.

Uses: I use this level when I pick the transport. If the server does the talking and the client only watches, I pick server-sent events. If the client talks back, like in chat, I pick WebSockets. If the freshness need is loose, polling at 30 seconds is honest and simple, and I say plainly that those clients see updates up to 30 seconds late.

Dimensions:

  • Load: At a hundred clients, polling is cheap. At ten thousand clients polling every 30 seconds, the poll requests alone are 333 a second, and most of them learn that nothing changed.
  • Breaks: It breaks when the client assumes the screen is live and the transport is polling. I notice it when a customer calls to ask where their food is and the food is on its way. The screen just did not know.
  • Cost: The cost of push is idle connection memory on the server. The cost of polling is request volume that grows with clients instead of updates.
  • Trade-off: I give up instant freshness to get a transport any proxy understands. Polling is simple and stale. Push is fresh and needs a gateway that can hold the connections.

Intermediate — Fan-out and the gateway

Definition: Fan-out is a multiplier. One event times ten thousand clients is ten thousand deliveries, and that is the number to size for. A push gateway holds the open sockets, and each pod only knows about its own. When a client connects to pod A and the event arrives on pod B, somebody has to route between them. That somebody is usually a shared presence store, and it becomes the new bottleneck.

Types:

  • The push gateway — the box that holds the open connections.
  • The presence store — the shared list of which client sits on which pod.
  • The fan-out queue — the queue that turns one event into ten thousand deliveries.
  • The replay window — the recent events kept around so a reconnecting client can catch up.

Uses: I use this level when I size the system. I start with the fan-out math, not the transport. 200 status changes a second sounds small until you multiply it. Most orders have two or three watchers, so the real fan-out is about 200 times 3, roughly 600 deliveries a second, not 200 times 50,000. That changes the whole sizing.

Dimensions:

  • Load: At 20 updates a second and 10,000 clients, the simulator shows 200,000 deliveries a second. The fan-out hop is the one that grows with the audience, and the bar moves when either number moves.
  • Breaks: It breaks when the presence lookup is the part nobody sized. I notice it when the gateways scale fine and the updates still lag, because the lookup that routes between pods is the slow hop.
  • Cost: The cost is idle connection memory on the gateway, about 500 MB for 10,000 clients in this lesson's numbers, plus the presence store itself, running all the time.
  • Trade-off: I give up stateless scaling to get live delivery. A stateless service scales by adding pods. A push gateway needs shared state so any pod can reach any client, and that shared state is a new thing to watch.

Advanced — Reconnects, order, and backpressure

Definition: A connection that drops is normal. The design is what happens next. Every update carries a sequence number. The client applies in order, drops anything older than what is on screen, and asks for a replay from its last sequence when it reconnects. A reconnecting client waits with exponential backoff and jitter, so ten thousand clients do not all knock at once.

Types:

  • The sequence number — one number per order, so the client can tell old from new.
  • The replay — only what the client missed, from its last sequence, not a full re-read.
  • Backoff with jitter — the wait doubles each try, capped at 30 seconds, with a random spread so reconnects do not land in lockstep.
  • Load shedding — a slow consumer gets dropped before it drowns the fast ones.

Uses: I use this level the day a deploy restarts the gateway at lunch peak. Every client notices the drop at the same moment. The backoff spreads the reconnects over a minute instead of a second. The gateway caps how fast it accepts new connections. The fan-out queue stops growing, and the screens catch up in a wave.

Dimensions:

  • Load: At ten thousand clients, one gateway restart is a reconnect storm. The handshake rate spikes past anything the gateway has ever seen, and the queue fills with new events and replay requests at the same time.
  • Breaks: It breaks when a flaky mobile connection paints an update out of order and nobody on the server ever knows. I notice it when the screen shows a status the order already passed. Without the sequence check, the lie stays on the screen.
  • Cost: The cost is the replay window in the broker, at least fifteen minutes in this lesson's design, because a phone in a dead zone comes back late. A longer window costs storage and makes catch-up slower.
  • Trade-off: I give up buffering everyone forever to get a system that survives a restart. A slow consumer gets a resume point and a replay, not an infinite buffer. The buffer would eat the gateway's memory and then the gateway.

Master — Real-time at scale without melting

Definition: At this level the client is part of the system, not the end of it. I shard the fan-out by region so an event never crosses the country to reach a watcher next door. At a hundred times the scale, I stop fanning out from one gateway entirely and push from the edge, close to the client. And I measure to the paint, not to the send, because the user sees the screen, not my server log.

Types:

  • The regional shard — fan-out that stays inside the region where the watchers are.
  • Edge push — the delivery point moves close to the client, so the distance stops being the delay.
  • The fan-out lag alert — the number that tells the truth when gateway CPU looks fine.
  • The client contract — apply in order, drop anything older, replay from the last sequence.

Uses: I use this level when the alert that matters fires. CPU looked fine the whole time in this lesson's incident. The queue was the truth. Fan-out lag is the alert I page on, not gateway CPU, because a gateway can be idle and still four minutes behind.

Dimensions:

  • Load: At ten times the scale, I shard by region. At a hundred times, one gateway cannot fan out for everyone, and the edge takes over. The shape stays the same. Only the distance changes.
  • Breaks: It breaks when every watcher reconnects at once and the storm lands on one gateway. I notice it in fan-out lag climbing while the data itself is correct. The data was never wrong. The delivery was four minutes behind.
  • Cost: The cost is edge capacity and a presence store that now spans regions. A single gateway is cheaper to run and it is a single place where a restart becomes a storm.
  • Trade-off: I give up one central gateway to get delivery close to the watcher. The system gets more pieces, and each piece is smaller and closer. The payoff is that a restart in one region stays in one region.

Go deeper

Push feels instant until it is not. The first thing that breaks at scale is the gateway’s connection list. A stateless service scales by adding pods. A push gateway holds ten thousand open sockets, and each pod only knows about its own. When a client connects to pod A and the event arrives on pod B, somebody has to route between them. That somebody is usually a shared presence store, and it becomes the new bottleneck. I have watched a team scale gateways beautifully and then fall over on the presence lookup they forgot to size.

Backpressure is the second thing. Fan-out is a queue, and a queue that grows without a limit will eat the gateway’s memory and then the gateway. Shed load early. Drop a slow consumer before it drowns the fast ones. A client that cannot keep up gets a resume point and a replay, not an infinite buffer.

Server-sent events deserve a real look before you reach for WebSockets. SSE is plain HTTP, it reconnects by itself, and it is one way, which is all a status page needs. WebSockets are two way, which you need for chat, not for watching an order. Two way costs you heartbeats, ping frames, and a harder proxy story. I pick SSE when the server does the talking and WebSockets when the client talks back.

At ten times the scale, shard the fan-out by region so an event never crosses the country to reach a watcher next door. At a hundred times, stop fanning out from one gateway entirely and push from the edge, close to the client. The shape stays the same. Only the distance changes.

And the client is part of the system, not the end of it. Every update carries a sequence number. The client applies in order, drops anything older than what is on screen, and asks for a replay from its last sequence when it reconnects. Without that, a flaky mobile connection will paint lies on the screen and nobody on the server will ever know.

let lastSeq = 0;
let attempts = 0;

function connect() {
  const ws = new WebSocket("/updates?since=" + lastSeq);
  ws.onmessage = (e) => {
    const msg = JSON.parse(e.data);
    if (msg.seq !== lastSeq + 1) { resync(); return; }
    lastSeq = msg.seq;
    applyUpdate(msg);
  };
  ws.onclose = () => {
    attempts++;
    setTimeout(connect, backoff());
  };
}

function backoff() {
  const wait = Math.min(30000, 1000 * 2 ** attempts);
  return wait + Math.random() * 1000; // jitter spreads the storm
}

The two lines that matter are the sequence check and the jitter. The sequence check stops out-of-order paints. The jitter stops ten thousand clients from reconnecting in the same second.

One time it broke

What broke

A routine deploy restarted the push gateway at lunch peak. Every connected client noticed the drop at the same moment and reconnected at once. The gateway accepted handshakes faster than it could register them, the fan-out queue filled with new events and replay requests at the same time, and order status updates started arriving four minutes late. Screens froze on old statuses. Customers called support to ask where their food was. The food was on its way. The screen just did not know.

Timeline

  • 12:04Deploy restarts the push gateway. Connection count drops to zero.
  • 12:05Ten thousand clients reconnect in the same minute. Handshake rate spikes past anything the gateway has ever seen.
  • 12:06Fan-out lag alert fires. The queue is growing faster than it drains.
  • 12:09Support reports customers seeing stale order statuses. The data is correct. The delivery is four minutes behind.
  • 12:20Gateway sheds slow consumers and starts replaying from sequence numbers instead of full state. Lag starts to fall.
  • 12:35Lag back to zero. Screens catch up in a wave. Nobody lost data, but an hour of trust did.

What changed after

Clients now reconnect with exponential backoff and jitter, so a gateway restart spreads reconnects over a minute instead of a second. The gateway caps how fast it accepts new connections and sheds slow consumers instead of buffering them forever. Every update carries a sequence number, and a reconnect replays from the last sequence instead of re-reading everything. And the alert that matters is fan-out lag, not gateway CPU. CPU looked fine the whole time. The queue was the truth.

If you remember nothing else

Design exercise

Design live order tracking for a food delivery app. 50,000 customers watch their orders at peak. The service records about 200 status changes per second. Clients are mostly on flaky mobile networks. How do you get every status change to the right screen, quickly, without melting the backend?

Clarifying questions to ask

  • How many watchers does one order have, on average?
  • How fresh must the screen be: one second, five seconds?
  • What happens when a client is offline for ten minutes and comes back?
  • Is the update one way, or does the client send anything back?

Saved in this browser only. It tracks your progress through the series on this device.

Next in this series
Lesson 18 — API Gateways