Platform Engineering · Lesson 26

One Big App or Many Small Ones

A monolith is one deployable thing. One codebase, one database, one deploy that takes everything down or ships everything at once. Microservices are the same system cut into pieces, each with its own codebase, its own database, its own deploy. Both can work. Both can fail. I have run both, and the teams that got hurt were not the ones who picked the wrong side. They were the ones who split too early, before they knew where the real boundaries were, and then spent a year debugging the network instead of shipping features. This lesson follows one change, a new checkout discount rule, through a monolith and through a set of services, so you can see what the split actually costs and what it actually buys.

Difficulty: Advanced · Know this first: Lesson 01, Lesson 05

Run it as:

When it breaks, I check

One thing to remember

Extra diagram · Architecture map

One box, or five

The same checkout, drawn twice. I start with one box and one database, then I split services out one by one.

Monolith: 1 box, 1 database, 1 deployServices: 5, each its own databasePayment: a network call either way
Gatewayone door inMonolith appeverything in oneboxOne databaseall tablesGatewayroutes by pathCheckout serviceorder databasePricing servicepricing databaseInventory serviceinventory databasePayment servicealways a networkcallEventwarehouse, email,analytics later
your request
same checkout in bothown databaseevent, laterrequest · flowingresponse · the way back

    Extra diagram · Second trip

    One service down, and the undo

    Pricing stops answering in the middle of a checkout. I follow the order from the silent call to the truth the customer is told.

    Timeout + retry: 450 ms burnedFallback: full price, no discountSaga: undoes a half-finished order
    Checkout callspricingPricingdoes not answerTimeout and retryburn 450 msFallbackfull price, nodiscountOrder savedpayment chargedPayment failssaga startsCompensating stepsunreserve stock,cancel orderCustomertold the truth
    the checkout
    the checkoutpricing silent / payment failedorder savedcompensating steprequest · flowingresponse · the way back

      Where the time goes

      Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario. It is not a measurement from a live system. The shape is honest though. In most slow requests I have chased, one or two hops eat almost the whole budget. Step through the lesson or switch the scenario and watch the bar change. In the monolith the payment and the save are the biggest slices, because there is no network in the way. In the services version the price hop, the stock hop, and the payment hop are the biggest slices, because each one is a network round trip now. In the one-service-down version the price hop dwarfs everything else.

      Try it: split it yourself

      I can tell you splitting costs latency and buys deploy independence. It is better if you split it yourself and watch the numbers move. Drag the sliders. Watch which hop grows first. That hop is the one you would be paged about.

      5

      Every service in the path adds one more route the gateway and the checkout have to know about. Routing cost grows with the number of services. It grows slowly, a millisecond per service, but the failure surface grows with it. Each service is one more thing that can be down.

      4

      The four network hops together (price, stock, payment, tell the rest) cost calls times 45 ms. Each call costs about 45 ms: roughly 25 ms of network round trip and 20 ms of work inside the service.

      3

      Deploys do not cost latency. They buy independence. Each deploy touches 1 of the services in the path, so the blast radius of one deploy is 1 divided by the service count. In the monolith, one deploy touches everything.

      Check yourself

      Three questions. No score is saved anywhere — this is just you against the material.

      Basic to master

      The same topic four times. First the two shapes, then the bills that arrive, then the boundaries that hold, then redrawing the boundary.

      Basic — One box or many boxes

      Definition: A monolith is one deployable thing. One codebase, one database, one deploy that takes everything down or ships everything at once. Microservices are the same system cut into pieces, each with its own codebase, its own database, its own deploy. Both can work. Both can fail. I have run both, and the teams that got hurt were not the ones who picked the wrong side. They were the ones who split too early, before they knew where the real boundaries were, and then spent a year debugging the network instead of shipping features.

      Types:

      • Monolith: one deploy, one database, one log stream, one on-call rotation.
      • Modular monolith: one deploy, but the code is split into modules with clear boundaries inside it.
      • Microservices: separate deploys, separate databases, separate teams, connected by the network.

      Uses: I use a monolith for a new system and a small team, where one deploy is a feature, not a problem. I split when a boundary is obvious and painful, when one part needs to deploy or scale on its own schedule and the rest of the system is in its way. I do not split because splitting is fashionable. I split because something specific is blocked.

      Dimensions: One monolith is one deploy, one dashboard, one log stream. Five services is five of each. The cost of the split is the network. A function call cannot fail independently of the caller. A network call can fail, time out, return slowly, or return a partial answer. The trade is deploy independence against simplicity. I give up the simple deploy to buy the independent one.

      Intermediate — The bills that arrive

      Definition: The whiteboard version is simple. Split the monolith, get independent deploys, go faster. The production version is a list of bills that arrive one at a time. The first bill is the network. Every call between services needs a timeout, a retry budget, and a fallback. The second bill is the transaction. The monolith gave me atomicity for free. The split takes it away, and the replacement is a saga, a sequence of steps where each step has a compensating step that undoes it. The third bill is versioning. The fourth bill is operations.

      Types:

      • Saga: steps with compensating steps. If the payment fails, something has to unreserve the stock and cancel the order.
      • Outbox pattern: the order and the event are written in the same local transaction, and a publisher sends the event later.
      • API versioning: the old version keeps working until every caller has moved, on purpose, with an end date.
      • Trace id: one id that travels with the request from the gateway through every service.

      Uses: I use this when I am splitting a real checkout flow. I set the timeout first and the retry count second, because a retry without a timeout is just a slower failure. I never retry a payment without an idempotency key. That rule is not negotiable. I use the outbox pattern when an event must not be lost if the publisher crashes, because the event is still in the outbox.

      Dimensions: A checkout that calls pricing, stock, and payment pays a network call for each, about 45 ms each, and each one can fail on its own. The failure I watch for is a slow service with no timeout, so the checkout waits and the thread pool fills up and the whole checkout goes down because one service is slow. The cost is timeouts, retries, fallbacks, and five dashboards instead of one. The trade is independence against distributed complexity. The team that splits without adding observability ends up debugging across five log streams at 2 AM with no trace id tying them together.

      Advanced — Boundaries that hold

      Definition: Data ownership decides the boundary, not the org chart. The team that owns the data owns the rule, and the service follows the data. A shared database that was supposed to be split but never was means the services are separate in name and coupled in fact. The price has one owner, and the checkout never computes a discount itself. If the checkout code contains a discount formula, the review blocks it.

      Types:

      • Data-owned boundaries: the service boundary follows the data boundary, not the team boundary.
      • Version-mix monitoring: a dashboard that shows which API version each replica is calling, so two versions running at once is a planned state with an end date.
      • Business checks: a check that compares the price charged against the price the rule would produce, on a sample of orders, every hour.

      Uses: I use this when a split goes wrong quietly, the way a pricing split did in this lesson. Half the orders were priced right and half were priced wrong. Nothing had crashed. Every dashboard was green. The bug was invisible to every monitor, because no monitor was watching the price itself. I use version-mix monitoring after that, and I use the business check to page the on-call when the price itself is wrong.

      Dimensions: At ten times the scale, the call graph gets deep enough that one slow service at the bottom slows every request at the top. I add timeouts that shrink as the call goes deeper, and a breaker on the pricing call so a dead pricing service fails fast. The cost is versioning discipline and a dashboard for the version mix. The trade is speed of the split against correctness of the boundary. A boundary drawn around the wrong data is a boundary I will redraw later, at full price.

      Master — Redrawing the boundary

      Definition: At a hundred times the scale, the network calls that cost 45 ms each now cost real money in the latency budget, and the team starts merging the hottest calls back together or caching the price for a few seconds. That is not a failure of the split. It is the split maturing. The boundary that made sense at one size stops making sense at another, and the honest move is to redraw it. Amazon learned this in public in the early 2000s, when it split its monolith into services and then spent years tuning which calls stayed separate and which came back together.

      Types:

      • Re-merging: two hot services become one again, because the call between them costs more than the split saves.
      • Caching hot paths: the price cached for a few seconds, so the hottest call stops crossing the network on every request.
      • Per-service cadence: some services deploy weekly, some deploy twice a day. The point was never to deploy everything independently.

      Uses: I use this when the system outgrows its first split and the honest answer is to change the boundary, not defend it. I use it when a deploy cadence argument starts, and I say the point was to deploy the fast-moving parts without waiting on the slow ones, not to deploy everything independently. I use it when the fallback decision comes up, because falling back to full price is a business decision the product owner makes, written down before the first outage, not during it.

      Dimensions: At a hundred times the scale, the operations bill is the dominant cost: five services means five deploys, five dashboards, and five things that can be down at 2 AM. The failure mode is a boundary defended long after the workload outgrew it, with latency paid on every request to protect a split nobody needs anymore. The cost is the rework of redrawing, and the trade is the purity of the split against what the workload actually needs. Start with a monolith. Split when a boundary is obvious and painful. Redraw when it stops being either.

      Go deeper

      The whiteboard version of this lesson is simple. Split the monolith, get independent deploys, go faster. The production version is a list of bills that arrive one at a time. The first bill is the network. A function call cannot fail independently of the caller. A network call can fail, time out, return slowly, or return a partial answer. Every call between services needs a timeout, a retry budget, and a fallback. I set the timeout first and the retry count second, because a retry without a timeout is just a slower failure. And I never retry a payment without an idempotency key. That rule is not negotiable.

      The second bill is the transaction. The monolith gave me atomicity for free. The split takes it away, and the replacement is a saga: a sequence of steps where each step has a compensating step that undoes it. Order saved, stock reserved, payment charged. If the payment fails, something has to unreserve the stock and cancel the order. That something is code I have to write, test, and monitor. The outbox pattern helps. The order service writes the order and the event in the same local transaction, in the same database. A separate publisher reads the outbox and sends the event. If the publisher crashes, the event is still in the outbox. Nothing is lost. It is not free, but it is honest.

      The third bill is versioning. In the monolith, the discount rule and the checkout change in the same deploy. They cannot disagree, because they ship together. Across services, the pricing service deploys on Tuesday and the checkout deploys on Thursday, and for two days the checkout is calling a version of the rule it was not written for. So the pricing API gets a version, and the old version keeps working until every caller has moved. That means running two versions at once, on purpose, and knowing which callers use which. I have seen teams skip this and pay for it with half the orders priced wrong.

      The fourth bill is operations. One monolith is one deploy, one dashboard, one log stream, one on-call rotation. Five services is five of each. The team that splits without adding observability ends up debugging across five log streams at 2 AM with no trace id tying them together. A trace id that travels with the request from the gateway through every service is not optional. It is the only way to answer the simplest question in an incident: what did this one request actually do?

      Failure modes I actually watch for: a slow service with no timeout, so the checkout waits and the thread pool fills up and the whole checkout goes down because one service is slow. A retry storm, where every caller retries at once and the recovering service gets crushed by the retries. A shared database that was supposed to be split but never was, so the services are separate in name and coupled in fact. A deploy that changes the event shape, so the consumers silently ignore the new field and the warehouse ships the old price.

      What changes at 10x scale: the call graph gets deep enough that one slow service at the bottom slows every request at the top. I add timeouts that shrink as the call goes deeper, and I add a circuit breaker on the pricing call so a dead pricing service fails fast instead of eating 450 ms on retries. The event volume grows, and the consumers need their own scaling. The database per service stops being a nice idea and becomes the thing that lets each team scale its own storage.

      What changes at 100x scale: the network calls that cost 45 ms each now cost real money in latency budget, and the team starts merging the hottest calls back together or caching the price for a few seconds. That is not a failure of the split. It is the split maturing. The boundary that made sense at one size stops making sense at another, and the honest move is to redraw it. I also stop pretending every service needs its own deploy cadence. Some services deploy weekly. Some deploy twice a day. The point was never to deploy everything independently. The point was to deploy the fast-moving parts without waiting on the slow ones.

      This is the pricing call as I write it now. A timeout, one retry, a fallback, and an idempotency key. No part of this is optional.

      # checkout calls pricing: timeout first, one retry, then fall back
      response = pricing_client.get_price(
          item_id=item_id,
          user_id=user_id,
          timeout_ms=100,          # fail fast; never wait on a dead service
          retries=1,               # one retry, not three; retries are load too
          idempotency_key=order_id # a retry must never charge or price twice
      )
      if response.timed_out:
          price = full_price(item_id)  # fallback: no discount, order still completes
          log.warn("pricing timeout, fell back to full price", order_id=order_id)

      The fallback is a business decision, not a code detail. Falling back to full price keeps the order moving and charges the customer more than the discount promised. I do not get to make that call alone. The product owner decides the fallback, and the decision is written down before the first outage, not during it.

      One time it broke

      What broke

      We split the discount rule out of the monolith into a pricing service, and for two days half the orders were priced right and half were priced wrong. Some customers got the ten percent off. Some paid full price. A few got the discount applied twice. Support tickets came in waves, and every ticket looked different. Nothing had crashed. Every dashboard was green. The pricing service was up. The checkout was up. The bug was invisible to every monitor we had, because no monitor was watching the price itself.

      Timeline

      • 10:00, the pricing service deploys the new discount rule, version two of the API.
      • 10:15, the checkout team deploys their change to call version two.
      • 10:20, the checkout deploy rolls back. A test failed in the pipeline and nobody noticed the rollback, so half the checkout replicas call version two and half still call version one.
      • 11:40, the first support ticket arrives. A customer was charged full price on a first order.
      • 13:05, I pull the orders and see the pattern. Every order priced wrong came from a replica running the old checkout. Every order priced twice came from a replica that called version two with the old discount still applied locally.
      • 14:30, I pin all checkout replicas to one version and the wrong prices stop.
      • 16:00, the duplicate discounts are refunded by hand, one by one.
      • 17:20, the deploy pipeline gets a check that blocks a rollback nobody sees.

      What changed after

      First, the price has one owner, and the checkout never computes a discount itself. If the checkout code contains a discount formula, the review blocks it. Second, every deploy now reports which API version each replica is calling, and a dashboard shows the mix. Two versions running at once is a planned state with an end date, not an accident we discover from support tickets. Third, I added a check that compares the price charged against the price the rule would produce, on a sample of orders, every hour. A mismatch pages the on-call. The monitors watch the system. That check watches the business.

      If you remember nothing else

      Design exercise

      Design the checkout for a small online shop. It does 200 orders a minute at peak. The shop wants a first-order discount, stock checks, payment, and a confirmation email. Decide: monolith or microservices? Draw the boundary lines, say where each piece of data lives, and say what happens when one piece is down. Keep it to one page of notes.

      Clarifying questions to ask

      • How many developers are on the team, and how often do they deploy today?
      • Does the discount rule change often, and who changes it?
      • Is the payment step allowed to retry, and who owns the idempotency key?
      • What happens if the stock check and the payment disagree, who wins?
      • Does the confirmation email need to arrive in seconds, or can it wait a minute?
      • What does the team do at 2 AM when one piece is down, and who gets paged?
      See a worked approach

      Start with a monolith. A team small enough to ask this question is small enough to ship one deployable thing. One codebase, one database, one transaction that saves the order, the discount, and the stock together. The discount rule is a function with one owner, and the price is computed in exactly one place. The payment is the one real boundary, because the money lives outside the system, so wrap it in a client with a timeout, one retry, and an idempotency key. The email goes out through an outbox row in the same transaction, so a crash never loses it. Split later, and split on data ownership: pricing first, because the discount rule changes often and its team wants to deploy without waiting on checkout. Then inventory, when the stock team needs to scale on its own. Never split the order and the payment into two databases on the same day. Rough sizing: 200 orders a minute is about 3 a second, small, but the reads behind it are ten times that, so size the cache for the reads and the database for the writes. One database handles this. One deploy handles this.

      Trade-offs to name

      • One deploy against independent deploys
      • One transaction against sagas and outboxes
      • Function calls against network calls
      • One dashboard against five
      • Splitting early and paying the network bill against splitting late and paying the coordination bill
      • A discount fallback that charges full price against one that blocks the order

      Stored in this browser only, so the Lessons index can show your progress.

      Next in this series
      Lesson 27 — The Big Map, Then the Small Details