A customer places an order. The order service does not call the inventory service, the payment service, and the shipping service one by one and hope they all answer. It drops one event into a queue and moves on. The queue holds the event until a consumer is ready for it. That gap between "sent" and "handled" is the whole lesson. It is what keeps a slow downstream service from taking the checkout page down with it, and it is also where duplicates, lost messages, and 2 AM backlogs are born.
Difficulty: Intermediate · Know this first: Lesson 01, Lesson 02
Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: in most slow orders I have chased, one hop eats almost the whole budget, and in a queue that hop is usually the wait. Step through the lesson or switch the scenario and watch the bar change.
Move the sliders. The budget bar above, the wait hop in the diagram, and the status lines update live. There is no right answer to find. There is a line to cross, and you will see it when you cross it.
Three questions. No score is saved anywhere — this is just you against the material.
A queue gives you at-least-once delivery. That is the whole contract. It does not give you exactly-once, and it does not give you ordering across the whole queue. If your design needs either of those, you build it yourself on top. I have watched teams assume exactly-once because the queue felt reliable. The queue was reliable. The duplicates still shipped.
The standard answer to duplicates is an idempotency key. The consumer stores the message id with the result of the work, in the same database write. Second delivery with the same id finds the stored result and skips the work. The key detail is "same write". If the result and the key land in two separate writes, a crash between them leaves the exact hole you were trying to close.
Ordering is narrower than people think. A queue orders messages inside one partition or one queue, not across all of them. If order matters for one customer, send all of that customer's messages to the same partition, using the customer id as the partition key. If order matters globally, you have one partition, one consumer, and a hard ceiling on throughput. That is a real trade-off, not a config mistake.
At 10x scale the broker stops being the interesting part. The interesting part is the consumers. Ten times the orders means ten times the database writes on the consumer side, and the database is where the queue's buffer stops helping. The queue absorbs a burst. It does not absorb a sustained 10x load. If the consumers cannot drain at the new rate, the queue grows until the disk fills, and then the broker stops accepting new orders. Plan the drain rate first, then the queue size.
At 100x scale you stop thinking about one queue and start thinking about many. Split by order type, by region, or by tenant. A poison message in the refunds queue should never block the orders queue. Separate queues are bulkheads. One bad message type gets its own dead-letter queue and its own alarm, and the rest of the system keeps moving.
The dead-letter queue needs an owner and a runbook. A DLQ nobody reads is a trash can with a fancy name. Set an alarm on DLQ depth. Write down who looks at it, how they replay a fixed message, and when they delete one for good. I check DLQ depth in the same morning pass as error rates. It takes ten seconds.
{
"queue": "orders",
"visibility_timeout_seconds": 90,
"max_receive_count": 3,
"dead_letter_queue": "orders-dlq",
"message_retention_seconds": 345600,
"notes": "visibility timeout must be longer than the slowest real processing time. max_receive_count 3 means a message fails three times, then moves to orders-dlq. retention is 4 days, long enough to replay a bad deploy."
}This is the shape of the config I want on every queue: a visibility timeout longer than the work, a small retry count, and a dead-letter queue with an alarm on it.
An online store ran a sale. Orders came in fast, the queue filled, and the inventory consumer fell behind. Nothing crashed. No alarm fired at first. The queue depth dashboard was on a screen nobody was watching at midnight. By morning, customers had paid for items that showed as out of stock, and some customers had been charged twice for the same order.
Design the order queue for a flash sale. One store, one checkout page. Peak traffic is 10,000 orders a minute for 30 minutes. Orders must not be lost if a consumer crashes, and no customer may be charged twice. Sketch the producer, the broker, the consumers, and the dead-letter queue. Name your numbers.