Platform Engineering · Lesson 23

Circuit Breakers and Bulkheads

A downstream service stops answering. My service keeps calling it anyway, and every call waits the full timeout. The threads pile up. The queue behind them grows. Then my service falls over too, and the outage spreads one hop further. I have watched that exact chain happen on a dashboard at 2 AM. This lesson follows one call to a failing downstream through the three states of a circuit breaker, closed, open, and half-open, and shows the second guard that sits next to it, the bulkhead, which stops one slow dependency from eating every worker I have. The breaker makes failure fast. The bulkhead makes failure small. The fallback makes failure survivable.

Difficulty: Advanced · Know this first: Lesson 02, Lesson 19

Run it as:

When it breaks, I check

One thing to remember

Extra diagram · Architecture map

One pool per downstream

One caller, three downstreams, one pool each. I start with one pool and build up to the wall that keeps a slow service contained.

Pricing pool: 10 slotsPools: 1 per downstreamSlow pricing: cannot touch checkout
when openCaller serviceone callerPricing pool +breaker10 slotsCheckout pool +breakerits own slotsSearch pool +breakerits own slotsFallback cachestale price beatsnonePricing serviceslow today
the call to pricing
call to pricingpool full / breaker openother pools still openrequest · flowingresponse · the way back

    Extra diagram · Second trip

    The breaker, round in a circle

    The breaker is a cycle, not a straight line. I follow its state around the circle: closed, open, half-open, and back.

    Trips open: 5th failureFail fast: 2 msBack to open: 30 s
    5th failurereset timeoutprobe succeedsprobe failsClosedcalls flow,failurescountedOpencalls failfast in 2 msHalf-openone probegoes through
    the breaker state
    breaker stateclosedopenhalf-openrequest · flowingresponse · the way back

      Where the time goes

      Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: in the dying case one timeout eats almost the whole budget, and everything after it is cheap because the breaker stops the bleeding. Step through the lesson or switch the scenario and watch the bar change.

      Try it: trip the breaker yourself

      A burst of 100 calls hits the same pricing downstream. Move the sliders and watch what the breaker saves. A call that answers costs 45 ms. A call that times out costs 2,000 ms. A call the breaker blocks costs 2 ms to fail fast plus 15 ms for the fallback.

      The share of calls that time out instead of answering.

      How many failed calls the breaker tolerates before it opens.

      How long the downstream rests with the breaker open.

      How many probe calls get through at once when half-open.

      Calls reaching downstream

      6 of 100

      Total for 100 calls with breaker

      ~10,666 ms (about 107 ms per call)

      Same 100 calls with no breaker

      ~180,450 ms

      Time saved

      ~169,784 ms (about 94%)

      Downstream rest before probe

      30 seconds

      Probes per minute

      2

      Check yourself

      Three questions. No score is saved anywhere — this is just you against the material.

      Basic to master

      The same topic four times. First the fast failure, then the three knobs, then the crowd of callers, then the call graph that degrades on purpose.

      Basic — Stop calling what is already dead

      Definition: A circuit breaker watches calls to one downstream service. When enough calls fail, it trips open. After that, calls fail at once, in about 2 ms, instead of waiting the full 2,000 ms timeout. The breaker makes failure fast. A bulkhead is the wall next to it. Each downstream gets its own small pool of workers, say 10 slots for pricing. A slow pricing service can fill its own pool. It cannot touch the pool that serves checkout. The bulkhead makes failure small. A fallback is what answers while the breaker is open. A cached price with its age stamped on it beats an error page.

      Types:

      • Closed: calls go through. The breaker is counting failures.
      • Open: calls fail at once. The downstream gets time to recover.
      • Half-open: one probe call gets through to test whether the downstream is back.
      • Bulkhead pool: one small pool per downstream, with a hard limit, and no shared pool behind it.

      Uses: I use it on every outbound call that can hang. A pricing call inside checkout is the example in this lesson. The pricing service stops answering. My service keeps calling it. Every call waits the full timeout. The threads pile up. Then my service falls over too, and the outage spreads one hop further. The breaker stops that chain at the first hop.

      Dimensions: One dead downstream with a 2,000 ms timeout fills a 10-slot pool in under a minute. The cost of adding a breaker is a few lines of config. The trade is small. A call that fails fast is a call I can handle. A call that hangs for two seconds is a thread held hostage. The fallback is what makes open survivable for the user. A breaker without a fallback is just a faster error.

      Intermediate — Tuning the three knobs

      Definition: A breaker has three settings that matter, and all three are guesses until an incident proves them wrong. The threshold is how many failures before it trips. The reset timeout is how long the downstream rests. The probe count is how many calls get through at half-open. I start with a threshold of 5, a reset of 30 seconds, and 1 probe, then I tune from the state-change log, not from a feeling.

      Types:

      • Count-based window: 5 failures in the last 20 calls trips the breaker.
      • Time-based window: failures counted over the last minute instead of the last calls.
      • Error counting rules: count timeouts and 500s. Never count 400s. A bad request from my own caller is not the downstream being sick.

      Uses: I use this when I am tuning a breaker after its first incident, and when I am reviewing one in a design review. I ask what it counts, because the errors you count decide when it trips. A breaker that trips on 400s will open every time a client ships a bug. I also ask where the fallback gets its data, because a fallback that calls the same sick database is not a fallback.

      Dimensions: Too low a threshold and a short blip opens the breaker for no reason. Too high and the incident is half over before the breaker moves. Too short a reset and the probe hits a service that is still down. Too long and users sit on the fallback longer than they need to. As traffic grows tenfold, the pools get bigger and the thresholds get tighter. The cost is tuning time and one dashboard panel that shows breaker state per downstream in plain words: closed, open, or half-open. The trade is sensitivity against stability.

      Advanced — Breakers in a crowd

      Definition: The breaker lives in the caller, so every caller has its own breaker for the same downstream. Ten callers with a threshold of 5 means the downstream takes 50 failed calls before all the breakers open. That is why the bulkhead matters next to it. The pool limit is per caller too, and it is the thing that stops the pileup while the breakers are still counting. Netflix built Hystrix for exactly this problem: one slow dependency taking down every service that called it.

      Types:

      • Per-caller breaker: each caller trips on its own count. Simple, and slow to converge across many callers.
      • Gateway-level breaker: one breaker at the front door that sees all the traffic at once.
      • Fallback budget: the fallback gets its own limit, so a flood of fallback work cannot sink the caller either.

      Uses: I use this when the call graph has many callers for one downstream, and when I am the one explaining why a breaker that works in testing did not open fast enough in production. I use it on call, when I read the breaker state per downstream and see which ones are open, instead of guessing from a wall of red latency graphs.

      Dimensions: At ten times the traffic, the numbers change but the shape does not. The pools get bigger, the thresholds get tighter, and the fallback gets its own budget. The failure I watch for is the shared pool behind the separate pools. If the pools are labels on one big pool, the bulkhead is a lie, and one slow downstream still eats every worker. The cost is real separation: one pool per downstream, hard limits, and the health check endpoint on its own threads so a busy service can still answer that it is alive. The trade is local protection against a global view. A gateway breaker sees everything. A per-caller breaker sees only itself.

      Master — Failure as a designed state

      Definition: At this level the breaker, the bulkhead, and the fallback are not three tricks. They are one design: the system degrades on purpose, in an order I chose, instead of falling over in an order the incident chose. The pricing call falls back to a cached price. The checkout keeps selling. The payment call never retries without an idempotency key. The dashboard shows which breakers are open and why. And I test it by breaking things on purpose, because a breaker I have never seen open is a breaker I am guessing about.

      Types:

      • Cascading-failure drills: kill a downstream in staging and watch the breakers open, the fallbacks serve, and the callers stay up.
      • Fallback freshness budgets: a cached price older than 15 minutes is not served. A plain message is served instead.
      • Breaker state alerts: a breaker that stays open for an hour pages someone, because a fallback is a bandage, not a fix.

      Uses: I use this when I am designing a call graph that has to survive a downstream outage without paging anyone, the way the pricing outage in this lesson ended the second time: the breaker opened in seconds, the fallback served cached prices, and nobody paged me at all. I use it in design reviews when someone proposes a new dependency, and I ask for the breaker, the bulkhead, and the fallback in the same review, not as follow-up work.

      Dimensions: At a hundred times the traffic, I stop trusting any single breaker and add a second layer: the gateway sheds load before the breakers even see it. The cost is ongoing drill time and dashboard space. The failure mode is a fallback that goes stale silently, serving a price from yesterday while the monitors stay green, because no monitor watches the price itself. The trade is graceful degradation against full fidelity. I would rather serve a slightly stale price and keep selling than serve nothing at all.

      Go deeper

      A breaker has three settings that matter, and all three are guesses until an incident proves them wrong. The threshold is how many failures before it trips. Too low and a short blip opens the breaker for no reason. Too high and the incident is half over before the breaker moves. The reset timeout is how long the downstream rests. Too short and the probe hits a service that is still down. Too long and users sit on the fallback longer than they need to. The probe count is how many calls get through at half-open. One is the safe default. More than one is a bet that the downstream is really back. I start with a threshold of 5, a reset of 30 seconds, and 1 probe, then I tune from the state-change log, not from a feeling.

      There is a second trap. A breaker counts errors, and the errors you count decide when it trips. Count timeouts and 500s. Do not count 400s. A bad request from my own caller is not the downstream being sick, and a breaker that trips on 400s will open every time a client ships a bug.

      There is a third trap. The breaker lives in the caller, so every caller has its own breaker for the same downstream. Ten callers with a threshold of 5 means the downstream takes 50 failed calls before all the breakers open. That is why the bulkhead matters next to it. The pool limit is per caller too, and it is the thing that stops the pileup while the breakers are still counting.

      At 10x traffic, the numbers change but the shape does not. The pools get bigger, the thresholds get tighter, and the fallback gets its own budget, because a fallback that calls the same sick database is not a fallback. At 100x, I stop trusting any single breaker and add a second layer: the gateway sheds load before the breakers even see it. That is Lesson 24, and it is the same idea one hop further out.

      # circuit breaker + bulkhead for the pricing downstream
      # numbers are a starting point, tuned from the state-change log
      pricing-breaker:
        failure-threshold: 5          # failures in the rolling window, then open
        window-size: 20               # last 20 calls counted
        count-these: [timeout, http-500, http-503]
        never-count: [http-400, http-404]
        reset-timeout-seconds: 30     # open -> half-open
        half-open-probes: 1           # one call tests the water
      pricing-bulkhead:
        max-concurrent-calls: 10      # pricing pool only, checkout has its own
        max-wait-for-slot-ms: 50      # wait briefly, then fail fast
      fallback:
        serve: cached-price           # last known price, stamped with its age
        max-age-minutes: 15           # older than this, show a plain message

      One time it broke

      A pricing service went down on a weekday afternoon. It did not return errors. It just stopped answering. My service kept calling it with a 2,000 ms timeout and no breaker. Every call waited the full two seconds. The pool for pricing filled up in under a minute. Then the shared thread pool behind it started filling too, because the pools were not really separate. They were one big pool with labels on it. Checkout slowed down. Search slowed down. Then the health check endpoint slowed down, because it ran on the same threads. The load balancer saw the slow health checks and pulled the whole service out of rotation. One dead downstream had taken down a healthy caller, and the caller going dark made the dashboard look like everything was down at once.

      Timeline

      • 14:02Pricing stops answering. No errors, just silence. My service's pricing latency climbs to the timeout.
      • 14:04The pricing pool fills. Calls start queueing for a slot.
      • 14:07The shared pool fills. Checkout and search latency climb even though their own downstreams are fine.
      • 14:09Health checks time out. The load balancer pulls two instances out of rotation. Traffic concentrates on the rest, and they fill faster.
      • 14:12Someone pages me. The dashboard shows everything red. The real cause is one silent downstream.
      • 14:20We set the pricing timeout to 500 ms by hand and restart the service. Threads free up. Checkout recovers.
      • 14:40Pricing comes back. We spend the rest of the afternoon explaining why a pricing outage took down checkout.

      What changed after

      Three things, in order. First, a real breaker on every outbound call, threshold 5, reset 30 seconds, one probe, counting timeouts and 500s only. Second, real bulkheads: one small pool per downstream, with hard limits, and no shared pool behind them. The health check endpoint got its own threads too, so a busy service can still answer "I am alive." Third, a fallback for the pricing call: serve the cached price with its age stamped on it. We also added one dashboard panel we did not have that day: breaker state per downstream, in plain words, closed, open, or half-open. The next time pricing died, the breaker opened in seconds, the fallback served cached prices, and nobody paged me at all.

      If you remember nothing else

      Design exercise

      Design the pricing call in a checkout flow. Checkout calls a pricing service that is flaky: it answers in 45 ms when healthy, times out half the time when sick, and goes fully down for a few minutes about once a month. Checkout must keep selling while pricing is down. The page may show a slightly stale price, but it must never charge a stale price. How do you design the call?

      Clarifying questions to ask

      • How stale may the displayed price be, and who decides that number?
      • Is the charge price read from pricing at payment time, or is the displayed price reused at charge time?
      • How many checkouts per second at peak, and how many threads does checkout have?
      • What does pricing return besides the price, and does checkout need all of it?
      • Who owns the cached price, and how is it invalidated when the real price changes?
      Show a worked approach

      Put a breaker on the pricing call with a threshold of 5, a 30 second reset, and 1 probe. Give pricing its own bulkhead pool of 10 slots, separate from the payment pool, so a pricing pileup cannot touch payment. Set the pricing timeout short, 500 ms, because a price is not worth a long wait. Add a fallback that serves the cached price with its age shown, and cap its age at 15 minutes. At payment time, never use the cached price: the charge path calls pricing again through the breaker, and if the breaker is open, checkout stops and tells the user plainly instead of charging a guess.

      Size it roughly: at 100 checkouts a second, 10 pool slots at 45 ms each handle about 220 calls a second, so the pool has headroom when healthy and fails fast when sick.

      Say the trade-offs out loud: a stale displayed price can confuse a user who just saw a different price, so stamp the age on it; a short timeout can fail a call that would have succeeded at 600 ms, and that is a price worth paying; and the breaker protects checkout but does nothing for pricing itself, so pricing still needs its own owner and its own fix.

      Saved on this device only, so you can pick the course back up where you left it.

      Next in this series
      Lesson 24 — Saying No Before the System Falls Over