A downstream service stops answering. My service keeps calling it anyway, and every call waits the full timeout. The threads pile up. The queue behind them grows. Then my service falls over too, and the outage spreads one hop further. I have watched that exact chain happen on a dashboard at 2 AM. This lesson follows one call to a failing downstream through the three states of a circuit breaker, closed, open, and half-open, and shows the second guard that sits next to it, the bulkhead, which stops one slow dependency from eating every worker I have. The breaker makes failure fast. The bulkhead makes failure small. The fallback makes failure survivable.
Difficulty: Advanced · Know this first: Lesson 02, Lesson 19
Extra diagram · Architecture map
One caller, three downstreams, one pool each. I start with one pool and build up to the wall that keeps a slow service contained.
Extra diagram · Second trip
The breaker is a cycle, not a straight line. I follow its state around the circle: closed, open, half-open, and back.
Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: in the dying case one timeout eats almost the whole budget, and everything after it is cheap because the breaker stops the bleeding. Step through the lesson or switch the scenario and watch the bar change.
A burst of 100 calls hits the same pricing downstream. Move the sliders and watch what the breaker saves. A call that answers costs 45 ms. A call that times out costs 2,000 ms. A call the breaker blocks costs 2 ms to fail fast plus 15 ms for the fallback.
The share of calls that time out instead of answering.
How many failed calls the breaker tolerates before it opens.
How long the downstream rests with the breaker open.
How many probe calls get through at once when half-open.
Calls reaching downstream
6 of 100
Total for 100 calls with breaker
~10,666 ms (about 107 ms per call)
Same 100 calls with no breaker
~180,450 ms
Time saved
~169,784 ms (about 94%)
Downstream rest before probe
30 seconds
Probes per minute
2
Three questions. No score is saved anywhere — this is just you against the material.
The same topic four times. First the fast failure, then the three knobs, then the crowd of callers, then the call graph that degrades on purpose.
Definition: A circuit breaker watches calls to one downstream service. When enough calls fail, it trips open. After that, calls fail at once, in about 2 ms, instead of waiting the full 2,000 ms timeout. The breaker makes failure fast. A bulkhead is the wall next to it. Each downstream gets its own small pool of workers, say 10 slots for pricing. A slow pricing service can fill its own pool. It cannot touch the pool that serves checkout. The bulkhead makes failure small. A fallback is what answers while the breaker is open. A cached price with its age stamped on it beats an error page.
Types:
Uses: I use it on every outbound call that can hang. A pricing call inside checkout is the example in this lesson. The pricing service stops answering. My service keeps calling it. Every call waits the full timeout. The threads pile up. Then my service falls over too, and the outage spreads one hop further. The breaker stops that chain at the first hop.
Dimensions: One dead downstream with a 2,000 ms timeout fills a 10-slot pool in under a minute. The cost of adding a breaker is a few lines of config. The trade is small. A call that fails fast is a call I can handle. A call that hangs for two seconds is a thread held hostage. The fallback is what makes open survivable for the user. A breaker without a fallback is just a faster error.
Definition: A breaker has three settings that matter, and all three are guesses until an incident proves them wrong. The threshold is how many failures before it trips. The reset timeout is how long the downstream rests. The probe count is how many calls get through at half-open. I start with a threshold of 5, a reset of 30 seconds, and 1 probe, then I tune from the state-change log, not from a feeling.
Types:
Uses: I use this when I am tuning a breaker after its first incident, and when I am reviewing one in a design review. I ask what it counts, because the errors you count decide when it trips. A breaker that trips on 400s will open every time a client ships a bug. I also ask where the fallback gets its data, because a fallback that calls the same sick database is not a fallback.
Dimensions: Too low a threshold and a short blip opens the breaker for no reason. Too high and the incident is half over before the breaker moves. Too short a reset and the probe hits a service that is still down. Too long and users sit on the fallback longer than they need to. As traffic grows tenfold, the pools get bigger and the thresholds get tighter. The cost is tuning time and one dashboard panel that shows breaker state per downstream in plain words: closed, open, or half-open. The trade is sensitivity against stability.
Definition: The breaker lives in the caller, so every caller has its own breaker for the same downstream. Ten callers with a threshold of 5 means the downstream takes 50 failed calls before all the breakers open. That is why the bulkhead matters next to it. The pool limit is per caller too, and it is the thing that stops the pileup while the breakers are still counting. Netflix built Hystrix for exactly this problem: one slow dependency taking down every service that called it.
Types:
Uses: I use this when the call graph has many callers for one downstream, and when I am the one explaining why a breaker that works in testing did not open fast enough in production. I use it on call, when I read the breaker state per downstream and see which ones are open, instead of guessing from a wall of red latency graphs.
Dimensions: At ten times the traffic, the numbers change but the shape does not. The pools get bigger, the thresholds get tighter, and the fallback gets its own budget. The failure I watch for is the shared pool behind the separate pools. If the pools are labels on one big pool, the bulkhead is a lie, and one slow downstream still eats every worker. The cost is real separation: one pool per downstream, hard limits, and the health check endpoint on its own threads so a busy service can still answer that it is alive. The trade is local protection against a global view. A gateway breaker sees everything. A per-caller breaker sees only itself.
Definition: At this level the breaker, the bulkhead, and the fallback are not three tricks. They are one design: the system degrades on purpose, in an order I chose, instead of falling over in an order the incident chose. The pricing call falls back to a cached price. The checkout keeps selling. The payment call never retries without an idempotency key. The dashboard shows which breakers are open and why. And I test it by breaking things on purpose, because a breaker I have never seen open is a breaker I am guessing about.
Types:
Uses: I use this when I am designing a call graph that has to survive a downstream outage without paging anyone, the way the pricing outage in this lesson ended the second time: the breaker opened in seconds, the fallback served cached prices, and nobody paged me at all. I use it in design reviews when someone proposes a new dependency, and I ask for the breaker, the bulkhead, and the fallback in the same review, not as follow-up work.
Dimensions: At a hundred times the traffic, I stop trusting any single breaker and add a second layer: the gateway sheds load before the breakers even see it. The cost is ongoing drill time and dashboard space. The failure mode is a fallback that goes stale silently, serving a price from yesterday while the monitors stay green, because no monitor watches the price itself. The trade is graceful degradation against full fidelity. I would rather serve a slightly stale price and keep selling than serve nothing at all.
A breaker has three settings that matter, and all three are guesses until an incident proves them wrong. The threshold is how many failures before it trips. Too low and a short blip opens the breaker for no reason. Too high and the incident is half over before the breaker moves. The reset timeout is how long the downstream rests. Too short and the probe hits a service that is still down. Too long and users sit on the fallback longer than they need to. The probe count is how many calls get through at half-open. One is the safe default. More than one is a bet that the downstream is really back. I start with a threshold of 5, a reset of 30 seconds, and 1 probe, then I tune from the state-change log, not from a feeling.
There is a second trap. A breaker counts errors, and the errors you count decide when it trips. Count timeouts and 500s. Do not count 400s. A bad request from my own caller is not the downstream being sick, and a breaker that trips on 400s will open every time a client ships a bug.
There is a third trap. The breaker lives in the caller, so every caller has its own breaker for the same downstream. Ten callers with a threshold of 5 means the downstream takes 50 failed calls before all the breakers open. That is why the bulkhead matters next to it. The pool limit is per caller too, and it is the thing that stops the pileup while the breakers are still counting.
At 10x traffic, the numbers change but the shape does not. The pools get bigger, the thresholds get tighter, and the fallback gets its own budget, because a fallback that calls the same sick database is not a fallback. At 100x, I stop trusting any single breaker and add a second layer: the gateway sheds load before the breakers even see it. That is Lesson 24, and it is the same idea one hop further out.
# circuit breaker + bulkhead for the pricing downstream # numbers are a starting point, tuned from the state-change log pricing-breaker: failure-threshold: 5 # failures in the rolling window, then open window-size: 20 # last 20 calls counted count-these: [timeout, http-500, http-503] never-count: [http-400, http-404] reset-timeout-seconds: 30 # open -> half-open half-open-probes: 1 # one call tests the water pricing-bulkhead: max-concurrent-calls: 10 # pricing pool only, checkout has its own max-wait-for-slot-ms: 50 # wait briefly, then fail fast fallback: serve: cached-price # last known price, stamped with its age max-age-minutes: 15 # older than this, show a plain message
A pricing service went down on a weekday afternoon. It did not return errors. It just stopped answering. My service kept calling it with a 2,000 ms timeout and no breaker. Every call waited the full two seconds. The pool for pricing filled up in under a minute. Then the shared thread pool behind it started filling too, because the pools were not really separate. They were one big pool with labels on it. Checkout slowed down. Search slowed down. Then the health check endpoint slowed down, because it ran on the same threads. The load balancer saw the slow health checks and pulled the whole service out of rotation. One dead downstream had taken down a healthy caller, and the caller going dark made the dashboard look like everything was down at once.
Three things, in order. First, a real breaker on every outbound call, threshold 5, reset 30 seconds, one probe, counting timeouts and 500s only. Second, real bulkheads: one small pool per downstream, with hard limits, and no shared pool behind them. The health check endpoint got its own threads too, so a busy service can still answer "I am alive." Third, a fallback for the pricing call: serve the cached price with its age stamped on it. We also added one dashboard panel we did not have that day: breaker state per downstream, in plain words, closed, open, or half-open. The next time pricing died, the breaker opened in seconds, the fallback served cached prices, and nobody paged me at all.
Design the pricing call in a checkout flow. Checkout calls a pricing service that is flaky: it answers in 45 ms when healthy, times out half the time when sick, and goes fully down for a few minutes about once a month. Checkout must keep selling while pricing is down. The page may show a slightly stale price, but it must never charge a stale price. How do you design the call?
Put a breaker on the pricing call with a threshold of 5, a 30 second reset, and 1 probe. Give pricing its own bulkhead pool of 10 slots, separate from the payment pool, so a pricing pileup cannot touch payment. Set the pricing timeout short, 500 ms, because a price is not worth a long wait. Add a fallback that serves the cached price with its age shown, and cap its age at 15 minutes. At payment time, never use the cached price: the charge path calls pricing again through the breaker, and if the breaker is open, checkout stops and tells the user plainly instead of charging a guess.
Size it roughly: at 100 checkouts a second, 10 pool slots at 45 ms each handle about 220 calls a second, so the pool has headroom when healthy and fails fast when sick.
Say the trade-offs out loud: a stale displayed price can confuse a user who just saw a different price, so stamp the age on it; a short timeout can fail a call that would have succeeded at 600 ms, and that is a price worth paying; and the breaker protects checkout but does nothing for pricing itself, so pricing still needs its own owner and its own fix.
Saved on this device only, so you can pick the course back up where you left it.