Platform Engineering · Lesson 38

Building Chat That Feels Instant

One message. Two phones. It looks simple, and the message part is simple. The hard part is everything around it: the connection that has to stay open, the ordering that has to hold inside one conversation, the 200-member group that turns one send into 200 writes, and the recipient who is offline and still has to get the message. This lesson follows one message from send to read receipt, and it pulls together the queue from Lesson 15, the events from Lesson 16, and the real-time push from Lesson 17. I have debugged every hop in this chain at 2 AM. The connection is the hard part. The message is easy.

Difficulty: Advanced · Know this first: Lesson 15, Lesson 16, Lesson 17

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds, except the wait, which charges in patience. The bar below is a worked example for this lesson's scenario, not a measurement from a live system. The shape is honest: in a normal message the push and the store eat most of the time, in a group message the fan-out eats it, and in an offline message the wait eats everything. Step through the lesson or switch the scenario and watch the bar change.

Try it: size the system yourself

Three numbers decide whether a chat system keeps up: how many sockets are open, how many messages arrive per second, and how wide each message fans out. Move the sliders and watch what the multiplication does. The latency here is latency under load, not the single-message budget above: a single group message costs 660 ms in the budget bar, and the same group under load also pays queue delay on top.

One persistent socket per user. Users are connections.

Sends across the whole system at peak.

Every message is copied once per member.

Delivery latency

210 ms

Fan-out writes / sec

40,000

Connection count

500,000

How it works: fan-out writes per second = messages per second × average group size. Delivery latency = 210 ms + queue delay, where queue delay = max(0, (fan-out writes/sec − 40,000) / 1,000) ms. The 40,000 is the fan-out capacity of one worker pool in this teaching example. Below it, the queue keeps up and adds nothing. Above it, every extra 1,000 fan-out writes per second adds 1 ms of waiting. At the maximums (10M users, 100k messages/sec, group size 500) the latency reads 50,170 ms. The system is not slow at that point. It is down. That is the point of the check.

The chat system, from scratch: one socket per user

Extra diagram. I read the picture first: the boxes are the system, the moving dot is a message, and each step adds one piece until the full picture is there. No text needed to follow it.

One open socket per userRegistry sharded by user idStore first, then send
12345678910

Send pathLive deliveryNudge onlyMoving dots: request · response

The offline message and the reconnect storm

Extra diagram. I read the picture first: the boxes are the system, the moving dot is an offline message, then a reconnecting crowd, and each step adds one piece until the full picture is there. No text needed to follow it.

Mailbox holds it while they are awayReconnect jitter 0-30 sPresence never blocks messages
12345678

Mailbox pathReconnectStorm defenceMoving dots: request · response

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

The same topic, four times. One message, from the first socket to the level where I can deploy the connection servers on a weekday afternoon and trust them.

Basic — The connection is the hard part

Definition: A chat system looks simple. One message, two phones. The message part is simple. The hard part is everything around it: the connection that has to stay open, the ordering that has to hold inside one conversation, the group that turns one send into many writes, and the recipient who is offline and still has to get the message.

Types:

  • One-to-one messages, one socket per user, one copy per recipient.
  • Group messages, where one send fans out to every member.
  • Offline delivery, where the message waits in a mailbox until the recipient reconnects.
  • Read receipts, which are best-effort and must never block the message.

Uses: I use this level for the first version of a chat product. The rule I start with is that the connection is the system: one persistent socket per user, and users are connections.

Dimensions: As load grows, connections grow with users, one socket each. What breaks is the reconnect storm, when many clients reconnect at once. What it costs is socket servers and a connection registry. What I give up is the simple request-response shape; a chat server holds state for every user, all the time.

Intermediate — Store first, fan out second

Definition: The send handler has one order, and the order is the lesson: dedupe on the client message id, store first, assign the sequence number once, then fan out. Nothing delivers before the store returns. A crash must never eat a message. Ordering matters inside one conversation, not across conversations.

Types:

  • Idempotent sends, deduped on the client message id, so a retry is not a new message.
  • A per-conversation sequence counter, one counter per conversation, giving exact order inside it.
  • An offline mailbox, where a user who reconnects asks for everything since their last sequence number.
  • Push as a nudge only, never the delivery path, because push is best-effort and outside my control.

Uses: I use this level for the design exercise in this lesson, 1 million concurrent users with groups up to 500 members and a user offline for a week who must still get every message on reconnect. The sizing is plain: fan-out writes per second equals messages per second times average group size. At 5,000 messages a second and an average group of 8, that is 40,000 writes a second, which one worker pool handles.

Dimensions: As load grows, the group size is the multiplier that matters. At an average group of 200, the same 5,000 messages a second is 1,000,000 writes a second, which needs sharded workers and a separate queue for group fan-out. What breaks is a device that renders a message before it knows the sequence number: it shows messages out of order and then shuffles them, and that shuffle is what users call a bug. What it costs is a write on every message. What I give up is the fast path that delivers before storing; storing first is what makes offline delivery and crash safety possible.

Advanced — Groups flip the direction

Definition: Write-path fan-out, copying the message into every mailbox, works up to a few hundred members. Past that, one announcement means a hundred thousand writes, and the send never finishes. The fix is read-path fan-out: store the message once in the group, and let each reader pull it when they open the group. I pick the direction per group size, not per system.

Types:

  • Write-path fan-out for small groups, which makes reads cheap and sends expensive.
  • Read-path fan-out for big groups, which makes sends cheap and reads slower.
  • Ephemeral presence and typing indicators, sent down the socket, never stored, never retried.
  • End-to-end encryption, where only the phones hold the keys and the server can store the message but cannot read it, search it, or moderate it.

Uses: I use this level when a group passes a few hundred members, and when the product team asks for server-side search and end-to-end encryption in the same system. Those two do not fit without a compromise, usually search on the device, and that is a product decision, not an engineering detail. It has to be made before the store is designed, not after.

Dimensions: As load grows, the connection registry becomes the system. A million sockets is a routing problem. Ten million is a sharding problem: which registry shard holds which user, and what happens when a shard dies. I shard the registry by user id, I replicate it, and I accept that after a crash some entries are stale for a few seconds. What breaks is a registry that blocks delivery while it heals. What it costs is registry replication. What I give up is perfect presence; stale entries mean a few messages take the offline path and arrive a moment late, and that is a fine price.

Master — Running chat through a deploy

Definition: At this level the deploy is the incident waiting to happen, and I plan for it. Connection servers drain before they restart: they stop accepting new sockets and wait for existing ones to close naturally, so a restart never drops ten thousand clients at once. Clients reconnect with jitter, a delay picked at random between zero and thirty seconds, which spreads the spike into a slope. Presence updates get their own path, separate from message delivery, so a presence flood can never queue behind a message again.

Types:

  • Drain-before-restart, so no restart drops a whole server of clients at once.
  • Jittered reconnect, spreading the reconnect wave over thirty seconds.
  • A separate presence path, so presence can never block messages.
  • A deploy runbook that lists the reconnect rate as a metric to watch, next to error rate and latency.

Uses: I use this level in the incident in this lesson. A new version of the connection servers was deployed on a weekday afternoon, restarting the servers one by one, which was the plan. What was not in the plan: every client connected to a restarting server reconnected immediately, all at once, and each reconnect wrote to the connection registry and then asked for its backlog. The registry took a million writes in under a minute. Presence updates flooded in behind them. Real messages queued behind the reconnects. Users saw messages arrive minutes late, and some saw their contacts flicker online and offline. I had been watching the wrong dashboard.

Dimensions: As load grows, the reconnect wave grows with the user count, and the registry is the bottleneck, not the database. What breaks is the deploy that restarts servers without draining them. What it costs is slower deploys, because draining takes time. What I give up is fast restarts, and I get back a deploy that does not take messaging down with it.

Go deeper

At 10x scale, the pieces in this lesson stop being separate problems and start being the same problem. Presence, typing indicators, message history, and search all sit on top of the same connection registry and the same store. A few things I would want a reader to carry into a real design.

Typing indicators and presence are ephemeral. They are never stored. If a typing indicator is lost, nothing is lost. I send them down the socket, I never write them to the database, and I never retry them. The moment presence becomes durable state, it becomes a second chat system to maintain.

End-to-end encryption changes the store step. If only the phones hold the keys, the server can store the message but it cannot read it, search it, or moderate it. That is a product decision, not an engineering detail, and it has to be made before the store is designed, not after. Server-side search and end-to-end encryption do not fit in the same system without a compromise, usually search on the device.

Big groups flip the fan-out direction. Write-path fan-out, copying the message into every mailbox, works up to a few hundred members. Past that, one announcement means a hundred thousand writes, and the send never finishes. The fix is read-path fan-out: store the message once in the group, and let each reader pull it when they open the group. The tradeoff is that reading gets slower and offline delivery gets harder, because there is no per-user mailbox to wait in. I pick the direction per group size, not per system.

Ordering across devices is a second ordering problem. The same user on a phone and a laptop needs the same order in both places. The per-conversation sequence number from step 4 solves this too, as long as both devices sync from the same number. If a device renders a message before it knows the sequence number, it will show messages out of order and then shuffle them. That shuffle is what users call a bug.

At 100x scale, the connection registry becomes the system. A million sockets is a routing problem. Ten million is a sharding problem: which registry shard holds which user, and what happens when a shard dies. I shard the registry by user id, I replicate it, and I accept that after a crash some entries are stale for a few seconds. Stale entries mean a few messages take the offline path and arrive a moment late. That is a fine price. The alternative is a registry that blocks delivery while it heals, and that is worse.

The code below is the send handler I would sketch on a whiteboard. It is the whole lesson in fifteen lines: dedupe on the client id, store first, assign the sequence number once, then fan out. Nothing delivers before the store returns.

// Idempotent send: dedupe on clientMsgId, store first, fan out second.
async function handleSend(userId, conversationId, clientMsgId, text) {
  const existing = await store.findByClientId(clientMsgId);
  if (existing) return existing; // retry, not a new message

  const seq = await counter.next(conversationId); // one counter per conversation
  const msg = await store.insert({
    clientMsgId, conversationId, senderId: userId,
    seq, text, createdAt: Date.now(),
  });
  await fanout.publish({ conversationId, msg }); // queue from Lesson 15
  return msg;
}

One time it broke

What broke

We deployed a new version of the connection servers on a weekday afternoon. The deploy restarted the servers one by one, which was the plan. What was not in the plan: every client connected to a restarting server reconnected immediately, all at once, and each reconnect wrote to the connection registry and then asked for its backlog. The registry took a million writes in under a minute. Presence updates flooded in behind them. Real messages queued behind the reconnects. Users saw messages arrive minutes late, and some saw their contacts flicker online and offline.

Timeline

  • 14:02Deploy starts, first connection server restarts.
  • 14:03Reconnect spike begins. Registry write latency climbs past one second.
  • 14:06Fan-out queue depth starts growing. One-to-one messages slow down.
  • 14:11On-call pages. Dashboard shows the registry, not the database, as the bottleneck.
  • 14:19Deploy paused, remaining servers left running the old version.
  • 14:26Registry catches up as the reconnect wave finishes. Messages start flowing again.
  • 14:40Backlog drained. Latency back to normal. Deploy rolled back for the day.

What changed after

Three things. First, connection servers drain before they restart: they stop accepting new sockets and wait for existing ones to close naturally, so a restart never drops ten thousand clients at once. Second, clients reconnect with jitter. A reconnect delay picked at random between zero and thirty seconds spreads the spike into a slope. Third, presence updates got their own path, separate from message delivery, so a presence flood can never queue behind a message again. The deploy runbook now lists the reconnect rate as a metric to watch, next to error rate and latency. I had been watching the wrong dashboard.

If you remember nothing else

Design exercise

The prompt

Design a chat system for 1 million concurrent users, with group chat and offline delivery. One-to-one messages, groups up to 500 members, and a user who is offline for a week must still get every message on reconnect.

Clarifying questions to ask

How many messages per second at peak, and what is the average group size? How long do undelivered messages live: a week, a month, forever? Is end-to-end encryption required, and does the product need server-side search? What does read mean in a group: read by one member, or read by all?

Worked approach

Start with the shape from this lesson. Gateway for auth and rate limits. Store first, then fan out. One sequence counter per conversation. A connection registry sharded by user id, holding one socket per user, so 1 million users means 1 million connections spread across connection servers. Fan-out workers copy each message into each recipient's mailbox. Offline users get their backlog from the mailbox on reconnect, asking for everything since their last sequence number. Push is a nudge only, never the delivery path. Size it with the simulator's formula: fan-out writes per second equals messages per second times average group size. At 5,000 messages per second and an average group of 8, that is 40,000 writes per second, which one worker pool handles. At an average group of 200, that is 1,000,000 writes per second, which needs sharded workers and a separate queue for group fan-out. Groups above 500 members flip to read-path fan-out: store once, let readers pull. Keep receipts aggregated and best-effort.

Trade-offs to name

Write-path fan-out makes reads cheap and sends expensive. Read-path fan-out does the opposite. Storing first costs a write on every message, but it is what makes offline delivery and crash safety possible. A per-conversation counter gives exact ordering and becomes a hot spot in huge groups. Draining connections before a restart makes deploys slower and reconnect storms smaller. Push as the only delivery path would be simpler and wrong: it is best-effort and outside your control.

Saved in this browser, so the Lessons index can show your progress.

Next in this series
Lesson 39 — Design a Notification System