System Design · Lesson 15

Message Queues

A customer places an order. The order service does not call the inventory service, the payment service, and the shipping service one by one and hope they all answer. It drops one event into a queue and moves on. The queue holds the event until a consumer is ready for it. That gap between "sent" and "handled" is the whole lesson. It is what keeps a slow downstream service from taking the checkout page down with it, and it is also where duplicates, lost messages, and 2 AM backlogs are born.

Difficulty: Intermediate · Know this first: Lesson 01, Lesson 02

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: in most slow orders I have chased, one hop eats almost the whole budget, and in a queue that hop is usually the wait. Step through the lesson or switch the scenario and watch the bar change.

Try it: how fast can your consumers keep up?

Move the sliders. The budget bar above, the wait hop in the diagram, and the status lines update live. There is no right answer to find. There is a line to cross, and you will see it when you cross it.

One consumer: 25/sec Total capacity: 100/sec Utilization: 80% Wait in queue: 10 ms Total end to end: ~97 ms
  • One consumer handles 1000 / processing time orders per second. At 40 ms that is 25 orders per second.
  • Total capacity = consumers × orders per second per consumer. At the defaults: 4 × 25 = 100 orders per second.
  • Utilization = producer rate / total capacity. At the defaults: 80 / 100 = 0.8, or 80 percent.
  • Above 100 percent utilization the broker feels it too: publish, disk write, and ack all slow down to the backlog values.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Go deeper

A queue gives you at-least-once delivery. That is the whole contract. It does not give you exactly-once, and it does not give you ordering across the whole queue. If your design needs either of those, you build it yourself on top. I have watched teams assume exactly-once because the queue felt reliable. The queue was reliable. The duplicates still shipped.

The standard answer to duplicates is an idempotency key. The consumer stores the message id with the result of the work, in the same database write. Second delivery with the same id finds the stored result and skips the work. The key detail is "same write". If the result and the key land in two separate writes, a crash between them leaves the exact hole you were trying to close.

Ordering is narrower than people think. A queue orders messages inside one partition or one queue, not across all of them. If order matters for one customer, send all of that customer's messages to the same partition, using the customer id as the partition key. If order matters globally, you have one partition, one consumer, and a hard ceiling on throughput. That is a real trade-off, not a config mistake.

At 10x scale the broker stops being the interesting part. The interesting part is the consumers. Ten times the orders means ten times the database writes on the consumer side, and the database is where the queue's buffer stops helping. The queue absorbs a burst. It does not absorb a sustained 10x load. If the consumers cannot drain at the new rate, the queue grows until the disk fills, and then the broker stops accepting new orders. Plan the drain rate first, then the queue size.

At 100x scale you stop thinking about one queue and start thinking about many. Split by order type, by region, or by tenant. A poison message in the refunds queue should never block the orders queue. Separate queues are bulkheads. One bad message type gets its own dead-letter queue and its own alarm, and the rest of the system keeps moving.

The dead-letter queue needs an owner and a runbook. A DLQ nobody reads is a trash can with a fancy name. Set an alarm on DLQ depth. Write down who looks at it, how they replay a fixed message, and when they delete one for good. I check DLQ depth in the same morning pass as error rates. It takes ten seconds.

{
  "queue": "orders",
  "visibility_timeout_seconds": 90,
  "max_receive_count": 3,
  "dead_letter_queue": "orders-dlq",
  "message_retention_seconds": 345600,
  "notes": "visibility timeout must be longer than the slowest real processing time. max_receive_count 3 means a message fails three times, then moves to orders-dlq. retention is 4 days, long enough to replay a bad deploy."
}

This is the shape of the config I want on every queue: a visibility timeout longer than the work, a small retry count, and a dead-letter queue with an alarm on it.

One time it broke

An online store ran a sale. Orders came in fast, the queue filled, and the inventory consumer fell behind. Nothing crashed. No alarm fired at first. The queue depth dashboard was on a screen nobody was watching at midnight. By morning, customers had paid for items that showed as out of stock, and some customers had been charged twice for the same order.

Timeline

  • 11:48 PMThe sale starts. Producer rate jumps from 40 orders a second to 260.
  • 11:52 PMQueue depth passes 10,000 messages. The oldest message is 4 minutes old. No alarm exists on queue age, so nobody is paged.
  • 12:20 AMThe inventory consumer's database slows under the write load. Processing time per message climbs from 40 ms to 300 ms. Consumer capacity falls below producer rate.
  • 12:41 AMA deploy rolls out a new consumer version. Pods restart mid-processing. Messages that were processed but not yet acknowledged get redelivered. The consumer has no duplicate check. Some orders reserve stock twice. Some customers are charged twice.
  • 06:15 AMThe morning shift sees the queue depth at 180,000 messages and the duplicate charge complaints. The sale is paused.
  • 07:00 AMA second consumer pool is added and the database write path is throttled to a safe rate. The queue drains by 10:30 AM.

What changed after

  • Alarms on queue depth and on the age of the oldest message, not just depth. Age is the number that tells you a customer is actually waiting.
  • An idempotency check on the consumer: the message id is stored with the reservation in the same database write, so a redelivery is a no-op.
  • Deploys stop restarting consumers mid-processing. The rollout drains each pod first, waits for in-flight messages to finish, then kills the pod.
  • A runbook for the dead-letter queue, with a named owner and a replay procedure, so the next bad message does not sit unread for a week.

If you remember nothing else

Design exercise

Design the order queue for a flash sale. One store, one checkout page. Peak traffic is 10,000 orders a minute for 30 minutes. Orders must not be lost if a consumer crashes, and no customer may be charged twice. Sketch the producer, the broker, the consumers, and the dead-letter queue. Name your numbers.

Clarifying questions to ask

A worked approach

Trade-offs to name

Next in this series
Lesson 16 — Pub-Sub and Events
Coming next