One message. Two phones. It looks simple, and the message part is simple. The hard part is everything around it: the connection that has to stay open, the ordering that has to hold inside one conversation, the 200-member group that turns one send into 200 writes, and the recipient who is offline and still has to get the message. This lesson follows one message from send to read receipt, and it pulls together the queue from Lesson 15, the events from Lesson 16, and the real-time push from Lesson 17. I have debugged every hop in this chain at 2 AM. The connection is the hard part. The message is easy.
Difficulty: Advanced · Know this first: Lesson 15, Lesson 16, Lesson 17
Every hop charges rent in milliseconds, except the wait, which charges in patience. The bar below is a worked example for this lesson's scenario, not a measurement from a live system. The shape is honest: in a normal message the push and the store eat most of the time, in a group message the fan-out eats it, and in an offline message the wait eats everything. Step through the lesson or switch the scenario and watch the bar change.
Three numbers decide whether a chat system keeps up: how many sockets are open, how many messages arrive per second, and how wide each message fans out. Move the sliders and watch what the multiplication does. The latency here is latency under load, not the single-message budget above: a single group message costs 660 ms in the budget bar, and the same group under load also pays queue delay on top.
One persistent socket per user. Users are connections.
Sends across the whole system at peak.
Every message is copied once per member.
Delivery latency
210 ms
Fan-out writes / sec
40,000
Connection count
500,000
How it works: fan-out writes per second = messages per second × average group size. Delivery latency = 210 ms + queue delay, where queue delay = max(0, (fan-out writes/sec − 40,000) / 1,000) ms. The 40,000 is the fan-out capacity of one worker pool in this teaching example. Below it, the queue keeps up and adds nothing. Above it, every extra 1,000 fan-out writes per second adds 1 ms of waiting. At the maximums (10M users, 100k messages/sec, group size 500) the latency reads 50,170 ms. The system is not slow at that point. It is down. That is the point of the check.
Extra diagram. I read the picture first: the boxes are the system, the moving dot is a message, and each step adds one piece until the full picture is there. No text needed to follow it.
Extra diagram. I read the picture first: the boxes are the system, the moving dot is an offline message, then a reconnecting crowd, and each step adds one piece until the full picture is there. No text needed to follow it.
Three questions. No score is saved anywhere — this is just you against the material.
The same topic, four times. One message, from the first socket to the level where I can deploy the connection servers on a weekday afternoon and trust them.
Definition: A chat system looks simple. One message, two phones. The message part is simple. The hard part is everything around it: the connection that has to stay open, the ordering that has to hold inside one conversation, the group that turns one send into many writes, and the recipient who is offline and still has to get the message.
Types:
Uses: I use this level for the first version of a chat product. The rule I start with is that the connection is the system: one persistent socket per user, and users are connections.
Dimensions: As load grows, connections grow with users, one socket each. What breaks is the reconnect storm, when many clients reconnect at once. What it costs is socket servers and a connection registry. What I give up is the simple request-response shape; a chat server holds state for every user, all the time.
Definition: The send handler has one order, and the order is the lesson: dedupe on the client message id, store first, assign the sequence number once, then fan out. Nothing delivers before the store returns. A crash must never eat a message. Ordering matters inside one conversation, not across conversations.
Types:
Uses: I use this level for the design exercise in this lesson, 1 million concurrent users with groups up to 500 members and a user offline for a week who must still get every message on reconnect. The sizing is plain: fan-out writes per second equals messages per second times average group size. At 5,000 messages a second and an average group of 8, that is 40,000 writes a second, which one worker pool handles.
Dimensions: As load grows, the group size is the multiplier that matters. At an average group of 200, the same 5,000 messages a second is 1,000,000 writes a second, which needs sharded workers and a separate queue for group fan-out. What breaks is a device that renders a message before it knows the sequence number: it shows messages out of order and then shuffles them, and that shuffle is what users call a bug. What it costs is a write on every message. What I give up is the fast path that delivers before storing; storing first is what makes offline delivery and crash safety possible.
Definition: Write-path fan-out, copying the message into every mailbox, works up to a few hundred members. Past that, one announcement means a hundred thousand writes, and the send never finishes. The fix is read-path fan-out: store the message once in the group, and let each reader pull it when they open the group. I pick the direction per group size, not per system.
Types:
Uses: I use this level when a group passes a few hundred members, and when the product team asks for server-side search and end-to-end encryption in the same system. Those two do not fit without a compromise, usually search on the device, and that is a product decision, not an engineering detail. It has to be made before the store is designed, not after.
Dimensions: As load grows, the connection registry becomes the system. A million sockets is a routing problem. Ten million is a sharding problem: which registry shard holds which user, and what happens when a shard dies. I shard the registry by user id, I replicate it, and I accept that after a crash some entries are stale for a few seconds. What breaks is a registry that blocks delivery while it heals. What it costs is registry replication. What I give up is perfect presence; stale entries mean a few messages take the offline path and arrive a moment late, and that is a fine price.
Definition: At this level the deploy is the incident waiting to happen, and I plan for it. Connection servers drain before they restart: they stop accepting new sockets and wait for existing ones to close naturally, so a restart never drops ten thousand clients at once. Clients reconnect with jitter, a delay picked at random between zero and thirty seconds, which spreads the spike into a slope. Presence updates get their own path, separate from message delivery, so a presence flood can never queue behind a message again.
Types:
Uses: I use this level in the incident in this lesson. A new version of the connection servers was deployed on a weekday afternoon, restarting the servers one by one, which was the plan. What was not in the plan: every client connected to a restarting server reconnected immediately, all at once, and each reconnect wrote to the connection registry and then asked for its backlog. The registry took a million writes in under a minute. Presence updates flooded in behind them. Real messages queued behind the reconnects. Users saw messages arrive minutes late, and some saw their contacts flicker online and offline. I had been watching the wrong dashboard.
Dimensions: As load grows, the reconnect wave grows with the user count, and the registry is the bottleneck, not the database. What breaks is the deploy that restarts servers without draining them. What it costs is slower deploys, because draining takes time. What I give up is fast restarts, and I get back a deploy that does not take messaging down with it.
At 10x scale, the pieces in this lesson stop being separate problems and start being the same problem. Presence, typing indicators, message history, and search all sit on top of the same connection registry and the same store. A few things I would want a reader to carry into a real design.
Typing indicators and presence are ephemeral. They are never stored. If a typing indicator is lost, nothing is lost. I send them down the socket, I never write them to the database, and I never retry them. The moment presence becomes durable state, it becomes a second chat system to maintain.
End-to-end encryption changes the store step. If only the phones hold the keys, the server can store the message but it cannot read it, search it, or moderate it. That is a product decision, not an engineering detail, and it has to be made before the store is designed, not after. Server-side search and end-to-end encryption do not fit in the same system without a compromise, usually search on the device.
Big groups flip the fan-out direction. Write-path fan-out, copying the message into every mailbox, works up to a few hundred members. Past that, one announcement means a hundred thousand writes, and the send never finishes. The fix is read-path fan-out: store the message once in the group, and let each reader pull it when they open the group. The tradeoff is that reading gets slower and offline delivery gets harder, because there is no per-user mailbox to wait in. I pick the direction per group size, not per system.
Ordering across devices is a second ordering problem. The same user on a phone and a laptop needs the same order in both places. The per-conversation sequence number from step 4 solves this too, as long as both devices sync from the same number. If a device renders a message before it knows the sequence number, it will show messages out of order and then shuffle them. That shuffle is what users call a bug.
At 100x scale, the connection registry becomes the system. A million sockets is a routing problem. Ten million is a sharding problem: which registry shard holds which user, and what happens when a shard dies. I shard the registry by user id, I replicate it, and I accept that after a crash some entries are stale for a few seconds. Stale entries mean a few messages take the offline path and arrive a moment late. That is a fine price. The alternative is a registry that blocks delivery while it heals, and that is worse.
The code below is the send handler I would sketch on a whiteboard. It is the whole lesson in fifteen lines: dedupe on the client id, store first, assign the sequence number once, then fan out. Nothing delivers before the store returns.
// Idempotent send: dedupe on clientMsgId, store first, fan out second.
async function handleSend(userId, conversationId, clientMsgId, text) {
const existing = await store.findByClientId(clientMsgId);
if (existing) return existing; // retry, not a new message
const seq = await counter.next(conversationId); // one counter per conversation
const msg = await store.insert({
clientMsgId, conversationId, senderId: userId,
seq, text, createdAt: Date.now(),
});
await fanout.publish({ conversationId, msg }); // queue from Lesson 15
return msg;
}
We deployed a new version of the connection servers on a weekday afternoon. The deploy restarted the servers one by one, which was the plan. What was not in the plan: every client connected to a restarting server reconnected immediately, all at once, and each reconnect wrote to the connection registry and then asked for its backlog. The registry took a million writes in under a minute. Presence updates flooded in behind them. Real messages queued behind the reconnects. Users saw messages arrive minutes late, and some saw their contacts flicker online and offline.
Three things. First, connection servers drain before they restart: they stop accepting new sockets and wait for existing ones to close naturally, so a restart never drops ten thousand clients at once. Second, clients reconnect with jitter. A reconnect delay picked at random between zero and thirty seconds spreads the spike into a slope. Third, presence updates got their own path, separate from message delivery, so a presence flood can never queue behind a message again. The deploy runbook now lists the reconnect rate as a metric to watch, next to error rate and latency. I had been watching the wrong dashboard.
Design a chat system for 1 million concurrent users, with group chat and offline delivery. One-to-one messages, groups up to 500 members, and a user who is offline for a week must still get every message on reconnect.
How many messages per second at peak, and what is the average group size? How long do undelivered messages live: a week, a month, forever? Is end-to-end encryption required, and does the product need server-side search? What does read mean in a group: read by one member, or read by all?
Start with the shape from this lesson. Gateway for auth and rate limits. Store first, then fan out. One sequence counter per conversation. A connection registry sharded by user id, holding one socket per user, so 1 million users means 1 million connections spread across connection servers. Fan-out workers copy each message into each recipient's mailbox. Offline users get their backlog from the mailbox on reconnect, asking for everything since their last sequence number. Push is a nudge only, never the delivery path. Size it with the simulator's formula: fan-out writes per second equals messages per second times average group size. At 5,000 messages per second and an average group of 8, that is 40,000 writes per second, which one worker pool handles. At an average group of 200, that is 1,000,000 writes per second, which needs sharded workers and a separate queue for group fan-out. Groups above 500 members flip to read-path fan-out: store once, let readers pull. Keep receipts aggregated and best-effort.
Write-path fan-out makes reads cheap and sends expensive. Read-path fan-out does the opposite. Storing first costs a write on every message, but it is what makes offline delivery and crash safety possible. A per-conversation counter gives exact ordering and becomes a hot spot in huge groups. Draining connections before a restart makes deploys slower and reconnect storms smaller. Push as the only delivery path would be simpler and wrong: it is best-effort and outside your control.
Saved in this browser, so the Lessons index can show your progress.