Platform Engineering · Lesson 28

Event-Driven Architecture

A customer places an order. Then three things have to happen. Inventory has to hold the item. Payments has to charge the card. Notifications has to send the confirmation. In the old way, the order service calls each of them in turn, waits for each answer, and hopes nothing fails in the middle. I have run that version. When the second call fails, the first one already happened, and nobody knows what state the order is in. This lesson follows a different way. The order service publishes one event, order.placed, and stops. Inventory, payments, and notifications each hear it and do their own job. Nobody calls anybody directly. That is event-driven architecture. The hard parts are not the publish. They are who owns the flow when steps fail, what the user sees while the system catches up, and how you undo work that already happened.

Difficulty: Advanced · Know this first: Lesson 15, Lesson 16

Run it as:

When it breaks, I check

One thing to remember

Extra diagram · Architecture map

One event, three listeners

One publisher, one consumer, one event. I add the second and third listeners, then the saga and the read model.

Event: order.placedConsumers: 3 groups, independentReplay: from the broker log
order.placedwatchesbuildsOrder servicepublishesBroker logevents stay hereInventory groupreserves stockPayments groupchargesNotifications grouptells the customerSaga coordinatorwatches all eventsRead modelbuilt from events
order.placed
order.placedeach group reads aloneread modelrequest · flowingresponse · the way back

    Extra diagram · Second trip

    When a step fails: the undo trip

    The payment fails after the stock was reserved. I follow the half-finished order through the saga's undo, to the replay that proves it.

    Order: half-finishedUndo: release the stockProof: replay the log
    order.placedpublishedInventory reservesstockPayment failsSaga sees itpayment failed afterstock reservedRelease-stockcommandsentStock releasedFailure message sentnot a confirmationReplay from the logverifies thesequence
    the order
    the orderpayment failedstock releasedreplayrequest · flowingresponse · the way back

      Where the time goes

      Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario - not a measurement from a live system - but the shape is honest: in a healthy order the payment call eats the budget and the broker hops are nearly free. Switch the scenario and watch the bar change. In the consumer-down scenario, waiting eats the budget. In the saga scenario, the undo eats a real share. There is no single total for an event-driven flow. There is one total per scenario, and one total per consumer group.

      Move the load and watch the lag grow

      The budget above is one fixed order. This box is yours. Move the sliders and watch what happens to one order when the event rate climbs, when I add consumers, and when I keep more history to replay. I learned this on a sale day: the broker does not fall over first. The slowest consumer does, and the lag is the bill.

      Events per second200

      How fast events arrive, so how much queue lag builds up when consumers cannot keep up.

      Consumer count3

      How many workers share the load in each consumer group, so how fast the queue drains. Each consumer handles about 120 events per second.

      Replay depth (days)7 days

      How far back the log is kept, so how long a replay takes and how many events are stored.

      Consistency lag

      What the customer sees lags by about 35 ms.

      Events stored

      The log holds about 120,960,000 events for 7 days of history.

      Check yourself

      Three questions. No score is saved anywhere - this is just you against the material.

      Basic to master

      The same topic four times. First the publish, then the delivery promise, then the undo, then the platform.

      Basic — Publish once, many react

      Definition: A customer places an order. Then three things have to happen. Inventory has to hold the item. Payments has to charge the card. Notifications has to send the confirmation. In the old way, the order service calls each of them in turn, waits for each answer, and hopes nothing fails in the middle. I have run that version. When the second call fails, the first one already happened, and nobody knows what state the order is in. In the event-driven way, the order service publishes one event, order.placed, and stops. Inventory, payments, and notifications each hear it and do their own job. Nobody calls anybody directly.

      Types:

      • Choreography: each consumer reacts on its own. No one component owns the flow.
      • Orchestration: one component, a saga, owns the flow and the undo.
      • Event sourcing: every state change is an event in the log, and the databases are copies built from it.

      Uses: I use it when three things have to happen after one event and none of them should block the others. I use choreography when each reaction is independent. Inventory hears the event and reserves. Notifications hears the event and sends. Neither cares what the other did. A new consumer just subscribes, and that is the whole reason to choose it.

      Dimensions: One publish fans out to every consumer, so adding a consumer costs nothing to the publisher. The cost is the broker and the consumer code. The trade is decoupling against visibility. When nothing calls anything directly, no one call graph shows the whole flow, and the flow lives in the log instead of in the code.

      Intermediate — The delivery promise

      Definition: The broker delivers at least once, never exactly once. A consumer can get the same event twice, and if it charges the card twice, the user pays twice. So every consumer checks the event id before doing the work, and commits the offset after the work, not before. Commit before the work and a crash loses the event. Never commit at all and every restart replays everything. Commit after the work and a crash costs one duplicate, which the idempotency check absorbs.

      Types:

      • Dedupe by event id: do the work only if this event id was never processed.
      • Offset commit after work: the commit is the receipt, written last.
      • Replay from the log: re-read the events to rebuild a copy, or to find every order that got a wrong message.

      Uses: I use this when a consumer restarts and replays, and when a duplicate delivery arrives during a slow hour. I use replay after an incident to find every order affected, the way the confirmation incident in this lesson did: pull the events for one order id, see the confirmation sent seven minutes before the payment failed, and find every order with the same pattern.

      Dimensions: At ten times the events, duplicate deliveries are ten times the duplicate charges and duplicate messages, so I make every consumer idempotent before the traffic grows, not after. The failure mode is a consumer that commits first and works second, losing events on every crash. The cost is the dedupe store and the check on every event. The trade is at-least-once with dedupe against exactly-once, which the network will not give you.

      Advanced — Flows with undo

      Definition: A saga is a list of steps and a list of undo steps. The undo steps have to be written before the saga ships, and they have to be tested like real code, because they run at 2 AM when the payment fails. The hardest undo is the one that cannot be undone cleanly. A charge can be refunded. A reservation can be released. A message that was already sent cannot be unsent. So I order the saga so the irreversible steps come last. Send the confirmation after the money and the stock are settled, never before.

      Types:

      • Choreography with undo scattered: each service guesses what the others did. It works until a failure leaves the order in a state nobody can explain.
      • Saga with compensation: one undo per step that already ran. Payment failed after inventory reserved, so release the stock, mark the order failed, publish order.failed.
      • Terminal-event triggers: a consumer reacts to the event that means the outcome is known, never to the first event in the flow.

      Uses: I use a saga the moment step three can fail after step two succeeded, because then the undo logic belongs in one place, not scattered across five services. I use a terminal event for notifications, after the confirmation incident in this lesson: the notifications consumer listened to order.placed and sent confirmations for orders that had not been paid for, because the payment provider was slow that hour. The fix was a new event, order.confirmed, published only after inventory and payment both succeeded.

      Dimensions: The read model lags the truth by the time events take to travel. On a good day that is tens of milliseconds. On a bad day it is minutes and the customer refreshes a page that lies. I design the screen for the lag: show processing while the events are in flight, and never show a success the events have not confirmed. The cost is the saga code and the tests for every undo step. The trade is simple choreography against owned undo. I pick choreography until a failure has to undo earlier work, and then I pick a saga.

      Master — Events at platform scale

      Definition: At a hundred times the events, one topic is not one problem anymore. I split topics by what changes together. The order topic stays small and fast. The analytics events go to their own topic, where a slow consumer cannot touch checkout. Event versions get a real process. New fields are optional, old fields are never renamed in place, and every version is tested against every consumer before it ships. Replay becomes a planned operation, and I rehearse it, because the first time I replay a hundred times the events should not be during an incident.

      Types:

      • Per-domain topics: orders, analytics, and audit events each get their own topic, so a slow consumer in one cannot hurt the others.
      • Event versioning: every event carries a version, and a consumer that cannot read version two of order.placed fails loudly at the door.
      • Schema evolution rules: optional new fields, no in-place renames, every version tested against every consumer.

      Uses: I use this when the event volume outgrows one topic and the consumers start hurting each other. I use the versioning process when a deploy changes the event shape, because a consumer that silently misreads a new field is worse than a consumer that fails loudly. I use the gap alert between order.placed and order.confirmed. If that gap grows, something in the middle is slow or failing, and I want to know before the customers do.

      Dimensions: The log grows by the event rate times retention, and retention is a business decision, not a default. I keep enough history to debug and rebuild, and I archive the rest. The cost is the versioning process and the rehearsed replay. The failure mode is an event shape change that ships without testing every consumer, so half the consumers read the new event wrong and nobody notices until the warehouse ships the old price. The trade is evolution speed against consumer safety, and I take safety, because a slow consumer is a bad day and a misread event is a wrong answer.

      Go deeper

      The lesson above is one order on a good day. Production is the rest of it. Here is what I actually spend my time on once the events are flowing.

      Choreography versus orchestration, and when I pick which

      I use choreography when each reaction is independent. Inventory hears the event and reserves. Notifications hears the event and sends. Neither cares what the other did. That is simple to add to, because a new consumer just subscribes. I use orchestration, a saga, when a failure has to undo earlier work. The moment step three can fail after step two succeeded, I want one component that knows the whole flow and owns the undo. The mistake I see is choreography for a flow with real undo needs. Then the undo logic is scattered across five services, each guessing what the others did, and a failure leaves the order in a state nobody can explain.

      Eventual consistency and what the user sees

      The read model lags the truth by the time it takes events to travel. On a good day that is tens of milliseconds and nobody notices. On a bad day it is minutes and the customer refreshes a page that lies. I design the screen for the lag, not against it. I show processing while the events are in flight. I never show a success the events have not confirmed. And I give the customer a way to see the real state, like an order status page that reads from the order service, not from the cached copy.

      Sagas and the halfway problem

      A saga is a list of steps and a list of undo steps. The undo steps have to be written before the saga ships, and they have to be tested like real code, because they run at 2 AM when the payment fails. The hardest undo is the one that cannot be undone cleanly. A charge can be refunded. A reservation can be released. A message that was already sent cannot be unsent. So I order the saga so the irreversible steps come last. Send the confirmation after the money and the stock are settled, never before.

      Event sourcing and replay

      If every state change is an event in the log, the log is the system of record and the databases are copies built from it. That gives me replay: fix the projection, re-read the log, rebuild the copy. It also gives me a bill. The log grows by the event rate times retention, and retention is a business decision, not a default. I keep enough history to debug and rebuild, and I archive the rest. I also version every event. A consumer that cannot read version two of order.placed should fail loudly at the door, not silently misread the order.

      What changes at 10x

      At ten times the events, I add consumers before I add brokers, because the broker is rarely the bottleneck. I watch lag per consumer group with an alert, not a dashboard I check when someone complains. And I make every consumer idempotent before the traffic grows, because duplicate deliveries at ten times the volume are ten times the duplicate charges and duplicate messages.

      What changes at 100x

      At a hundred times the events, one topic is not one problem any more. I split topics by what changes together. The order topic stays small and fast. The analytics events go to their own topic, where a slow consumer cannot touch checkout. Event versions get a real process: new fields are optional, old fields are never renamed in place, and every version is tested against every consumer before it ships. And replay becomes a planned operation. I rehearse it, because the first time I replay a hundred times the events should not be during an incident.

      Consumer and saga snippet I actually use

      # Consumer: do the work only if this event id was never processed.
      # Then commit the offset. Work first, commit second.
      def handle_order_placed(event):
          if already_processed(event.id):
              return  # duplicate delivery, safe to skip
          charge_card(event.order_id, event.total,
                      idempotency_key=event.order_id)
          mark_processed(event.id)
      
      # Saga compensation: one undo per step that already ran.
      # Payment failed after inventory reserved, so:
      #   1. release_stock(order_id)   # undo step 4
      #   2. mark_order_failed(order_id)
      #   3. publish order.failed      # notifications reacts to this
      # A step with no undo step listed here does not ship.

      Committing after the work is the whole trick. Commit before the work and a crash loses the event. Never commit at all and every restart replays everything. Commit after the work and a crash costs one duplicate, which the idempotency check absorbs. The saga list is the same idea at the flow level: every step names its undo before it ships.

      One time it broke

      This one stayed with me because nothing crashed, nothing errored, and customers still got the wrong message. The system did exactly what I told it to do. I had told it the wrong thing.

      What broke

      A confirmation message went out for orders that had not been paid for. The notifications consumer was listening to order.placed, the first event in the flow, and it sent the confirmation the moment it read that event. On a good day the payment landed a fraction of a second later and nobody noticed the order of events. Then the payment provider had a slow hour. Charges took minutes instead of milliseconds. Some failed outright. The confirmations had already gone out. Customers got a message saying their order was confirmed, then a second message saying the payment failed, then a support ticket asking which one was true. The bug was not in the consumer. It was in my choice of trigger. I had told notifications to react to the start of the flow instead of the end of it.

      Timeline

      What changed after

      Three things, and I still do all three. First, notifications react to a terminal event, never to the first event in a flow. A message goes out when the outcome is known, not when the flow starts. Second, I added an alert on the gap between order.placed and order.confirmed. If that gap grows, something in the middle is slow or failing, and I want to know before the customers do. Third, every new consumer gets a review question before it ships: which event triggers it, and what does the customer hear if the steps after that event fail? If the answer is a lie, the trigger is wrong.

      If you remember nothing else

      Try it yourself

      This is the kind of question I would ask in an interview, and the kind I ask myself before I wire a new flow with events. Take a minute with it before you read my approach.

      You run an online store. When a customer places an order, inventory has to reserve the item, payments has to charge the card, and notifications has to send a confirmation. Peak is 5,000 orders a minute. Design the event-driven flow. Which events do you publish, who owns the flow when a step fails, and what does the customer see while the system catches up?

      Clarifying questions to ask first

      I would ask four things before I sized anything.

      Rough sizing, worked

      5,000 orders a minute is about 83 events a second for order.placed. That is small for a broker, so the topic is not the bottleneck. The payment call is. One topic, orders, with 16 partitions, keyed by order id, so all the events for one order stay in order in one partition. Inventory, payments, and notifications each get their own consumer group, because each one has to see every event and each one moves at its own speed. A saga owns the flow: it watches for inventory reserved and payment succeeded, then publishes order.confirmed, and notifications reacts to that event only. If payment fails, the saga releases the stock and publishes order.failed. At 83 events a second across 16 partitions, each partition sees about 5 events a second, so even a consumer at 150 ms per event keeps up easily. The numbers only get interesting when a consumer gets slow or a step fails halfway, and that is the part the saga is for.

      Trade-offs to name

      I run inventory and payments in parallel, because neither depends on the other, and that saves the customer time. I pay for that with a saga, because parallel steps need an owner when one fails. I make the payment consumer idempotent first, because a duplicate charge is the one mistake I cannot apologize away. I let the read model lag on purpose, and I show processing while it catches up, because a fast screen that lies is worse than a slow screen that is honest. I set retention to seven days, so a crashed consumer can replay a full week, and I pay the disk cost for that because the alternative is lost events. And I alert on lag per group, with a tighter threshold for payments than for notifications, because not all lag is equal.

      Stored only in this browser, so the course index can show your progress.

      Next in this series
      Lesson 29 — Proving Who You Are, Then What You May Do