Platform Engineering · Lesson 34

Feature Flags and Rollouts

I used to think shipping code and releasing a feature were the same thing. They are not. A flag lets me ship code on Tuesday and release it on Friday, to ten percent of users first, and switch it off in seconds if it misbehaves. This lesson follows one flag flip from definition to cleanup, through a gradual rollout, a bad flag killed fast, and a flag nobody remembered to remove.

Difficulty: Advanced · Know this first: Lesson 33, Lesson 24

Run it as:

When it breaks, I check

One thing to remember

Extra diagram · Architecture map

The switchboard

One flag, one user, on or off. I add the console and the push to the app servers, then the metrics and the audit log.

Evaluation: 3 ms, localDefault: safe when service is downBucketing: sticky by user id
pushed outlocalcohortsevery flipFlag consoleflip it hereFlag serviceholds the configConfig pushto app serversApp serversin-memory configcopyRequest evaluation3 ms, localCohort metricsflagged vs unflaggedAudit logevery flip writtendown
the flag config
flag configlocal evaluationmetrics + auditrequest · flowingresponse · the way back

    Extra diagram · Second trip

    Kill it in four seconds

    A flag flips to 10%, and the flagged cohort starts erroring. I follow it from the split to the kill, and the cleanup that keeps flags from becoming permanent.

    Kill: off in 4 secondsDetect: cohort split, not blended rateCleanup: delete the flag
    and the branch that skippedFlag flipped to 10%Flagged cohortsplitserror rate vsunflaggedError thresholdtripsKill switch flippedoff in 4 secondsNo build, no deployno rollback trainFlag deletedold branch removedSkipped-steps branch0 to 100% in oneflip, blended ratehides it
    the bad flag
    bad flagerror rate splitskilled in 4 srequest · flowingresponse · the way back

      Where the time goes

      The clock below is a worked example for this lesson's scenario, not a measurement from a live system. The shape is the lesson: most of the time goes to watching, not to flipping. A flip takes seconds. A soak takes minutes. That ratio is what keeps a rollout safe. Steps that cost 0 ms in a scenario are left out of the bar but still listed in the legend.

      Try it: the rollout simulator

      Same worked example: 100,000 total users. Move the controls and watch what changes. The rollout percentage decides how many users a bad flag can reach. The error-rate threshold decides how long it runs before anyone notices. The live flag count decides how long the kill takes and how much cleanup debt the team is carrying. In the "A bad flag, killed fast" scenario above, steps 6 and 7 follow these controls live.

      How much of the user base the flag can reach right now.

      A low threshold trips sooner and cries wolf more often. A high threshold trips later and lets more users hit the bug.

      More live flags means a slower hunt for the right switch and more stale branches to untangle later.

      Exposed users

      10,000

      of 100,000 total users

      Time to detect a bad flag

      3 min

      180,000 ms of watching

      Time to kill it

      4 sec

      4,000 ms, no deploy needed

      Flag debt

      Watch it

      15 flags over the healthy budget of 25 = 7.5 cleanup hours

      At the defaults, detect plus kill is 184,000 ms: 180,000 ms to detect plus 4,000 ms to kill. That is exactly steps 6 plus 7 of the bad flag scenario above. Push the rollout to 100% with a 5.0% threshold and 200 live flags and the same bad flag reaches all 100,000 users, runs 15 minutes before detection, and sits on 87.5 hours of flag debt.

      Check yourself

      Three questions. No score is saved anywhere — this is just you against the material.

      Basic to master

      The same topic, four times. One flag flip, from the first switch to the level where I can kill a bad flag at 2 AM and trust the switch.

      Basic — Ship code, release later

      Definition: A feature flag is a named switch the code reads at runtime. Shipping code and releasing a feature are not the same thing. A flag lets me ship code on Tuesday and release it on Friday, to ten percent of users first, and switch it off in seconds if it misbehaves. No deploy is needed to turn it off.

      Types:

      • Boolean flags, on or off for everyone.
      • Percentage rollouts, where a share of users gets the new path.
      • Kill switches, flags whose only job is to turn a feature off fast.
      • Overrides, a short list of internal users and test accounts who see the new path first.

      Uses: I use this level when a feature is ready to ship but not ready to be seen, or when I want a way back that is faster than a rollback. The flip takes seconds. The soak takes minutes. That ratio is what keeps a rollout safe.

      Dimensions: As load grows, one flag is still one flag, and the cost stays small. What breaks is the flag nobody remembered to remove: the code keeps carrying two paths, old and new, and the old path rots. What it costs is one more config to maintain and one more branch in the code. What I give up is a codebase with one path through it; until cleanup, there are two.

      Intermediate — Rollouts in steps

      Definition: A percentage rollout hashes the user id together with the flag name, so the same user always lands in the same bucket. I open the flag to internal users first, then to ten percent of real users for a soak, watching the error rate split by flag cohort, then step to 50 percent, then to 100 percent. If the flagged cohort errors, I flip the flag off. That takes seconds and touches only flagged users.

      Types:

      • Percentage rollout with sticky bucketing, so a user does not flip between old and new mid-session.
      • Cohort dashboards, metrics split by flag so the flagged group is visible on its own.
      • Error-rate thresholds, a low one that trips sooner and cries wolf more often, a high one that trips later and lets more users hit the bug.
      • A flag lifecycle: created, rolling out, at 100 percent, cleanup scheduled, deleted.

      Uses: I use this level for the new pricing engine in this lesson. The plan was 10 percent, then 50, then 100. The on-call engineer skipped the steps and flipped the flag straight from 0 to 100 percent, because the sale started in an hour and the code had passed tests. The new engine rounded tax wrong on orders with a discount code. No alert fired, because the dashboard showed the blended error rate. The first signal was a support ticket, then three more. The flag was flipped off 22 minutes after the flip, and the overcharges stopped.

      Dimensions: As load grows, the flag service becomes a critical dependency, because every request asks it a question even if the answer comes from a local cache. What breaks is a percentage typed wrong, a bucket that is not sticky, or a cohort dashboard that was never opened. What it costs is evaluation on every request and the soak time at each step. What I give up is speed: a slower rollout learns more safely but delays the revenue or savings the feature brings.

      Advanced — Flags at fleet scale

      Definition: At this level flag evaluation is local and fast. The code evaluates the flag from a cached config and never blocks a request on the flag service. If the flag service goes down, evaluation fails to the safe default and checkout keeps running on the old engine. Config changes push out in seconds, and the push is versioned, so a bad config can be rolled back like a bad deploy.

      Types:

      • Local evaluation with a cached config that has a time to live and an age metric.
      • Versioned config pushes, so a bad push is reversible.
      • Live-flag caps per team, so exceeding the cap is a conversation, not silent drift.
      • An audit log of every flip: who, when, old value, new value.

      Uses: I use this level when the flag service is down or a config push stalls. Servers keep serving the last known good config and alert on its age. When an incident review asks what changed, the flip log is the first place I look, because most flag incidents I have seen were not bad code. They were a flip nobody recorded, a percentage typed wrong, or a flag whose owner left the team.

      Dimensions: As load grows, the count matters more than any single flag, and I cap live flags per team at around 50. What breaks is a config push that stalls, or two flags interacting in a way nobody reviewed. What it costs is audit storage and the review time before a new flag ships. What I give up is instant global flips with no record; every flip is now reviewable, which is the point.

      Master — The flag as a liability you manage

      Definition: At this level every flag has an owner and an expiry date at creation, and the kill switch gets a runbook, not just a button. The runbook says who can flip it, how to reach the flag console when the normal network path is down, and what to check after the flip. Then I drill it: flip a harmless flag off and on during a quiet hour, once a month, and time it. A kill switch I have never flipped is a hope, not a control.

      Types:

      • Cleanup tickets filed on day one, with a date six weeks out and the flag deleted when the date arrives.
      • Monthly kill-switch drills on a harmless flag.
      • New-flag reviews against the live list, so two flags cannot interact unnoticed.
      • Flag debt measured in cleanup hours: in this lesson, 15 flags over a healthy budget of 25 means 7.5 hours of cleanup owed.

      Uses: I use this level after the sale-week incident in this lesson. The rules that came out of it are still the rules: no jump larger than 25 percentage points in one step, ever. A cohort dashboard split by flag is required before any flip, not after. Every flag gets an owner and an expiry date at creation. The sale still ran, on the old engine, and nobody outside the team knew anything had happened.

      Dimensions: As load grows, flag debt grows with the team, not with traffic, and the live count is the number I watch. What breaks is a flag in limbo at 100 percent with no cleanup date, or an override list nobody removes, which is a backdoor. What it costs is cleanup work and drill time every month. What I give up is shipping speed on flag creation, and I get back a live list small enough to reason about at 2 AM.

      Go deeper

      A flag is a small idea with a long tail. The rollout in this lesson is the easy part. The hard part is everything around it.

      Targeting is where flags get subtle. A percentage rollout hashes the user id together with the flag name, so the same user always lands in the same bucket. An override list sits on top for internal users and test accounts. Overrides are how the team tries the new path before any real user sees it. Keep the override list short and review it, because an override nobody removes is a backdoor.

      The kill switch needs a runbook, not just a button. Write down who can flip it, how to reach the flag console when the normal network path is down, and what to check after the flip. Then drill it. Flip a harmless flag off and on during a quiet hour, once a month, and time it. A kill switch you have never flipped is a hope, not a control.

      Give every flag a lifecycle: created, rolling out, at 100 percent, cleanup scheduled, deleted. A flag that sits at 100 percent with no cleanup date is a flag in limbo. Put the cleanup date in the flag description on day one, and put a ticket on the board the same day. I set the expiry for six weeks out and let the ticket remind me.

      At 10x scale the flag service becomes a critical dependency. Every request asks it a question, even if the answer comes from a local cache. That cache needs a time to live and an age metric, so a stale config shows up on a dashboard instead of hiding in production. Config changes should push out in seconds, and the push should be versioned, so a bad config can be rolled back like a bad deploy.

      At 100x scale the count matters more than any single flag. Cap live flags per team, say 50, and make exceeding the cap a conversation, not a silent drift. Keep an audit log of every flip: who, when, old value, new value. When an incident review asks what changed, that log is the first place I look. Most flag incidents I have seen were not bad code. They were a flip nobody recorded, a percentage typed wrong, or a flag whose owner left the team.

      Failure modes to name: the flag service goes down, so evaluation must fail to the safe default and keep serving. A config push stalls, so servers must keep serving the last known good config and alert on its age. A percentage change lands mid-request, so bucketing must stay sticky for the session. Two flags interact, so new flags get reviewed against the live list before they ship.

      The flag, as config

      flag: new-pricing-engine
      default: false
      rollout:
        percentage: 10
        bucket_key: user_id
      kill_switch: true
      owner: pricing-team
      expires: 2027-02-01

      Evaluation, in code

      // Evaluate locally. Never block a request on the flag service.
      const on = flags.evaluate("new-pricing-engine", user.id, { default: false });
      if (on) { return newPricing(user, cart); }
      return legacyPricing(user, cart);

      One time it broke

      During a sale week, a new pricing engine went live behind a flag. The plan was 10 percent, then 50, then 100. The on-call engineer skipped the steps and flipped the flag straight from 0 to 100 percent, because the sale started in an hour and the code had passed tests. The new engine rounded tax wrong on orders that carried a discount code. A slice of customers was overcharged by small amounts. No alert fired, because the dashboard showed the blended error rate, and the checkout error rate looked normal. The first signal was a support ticket, then three more.

      Timeline

      • 09:12Flag flipped from 0 to 100 percent in one jump, no soak, no cohort dashboard open.
      • 09:26First support ticket: a customer charged more than the quoted total on a discounted order.
      • 09:31Three more tickets with the same shape. The engineer opens the flag console and the pricing logs.
      • 09:34The cohort split is finally checked. Flagged orders show the tax rounding error. The flag is flipped off.
      • 09:40Overcharges stop. The error rate for the pricing path falls back.
      • 10:05The rounding bug is found: the new engine applied the discount after tax instead of before it.
      • 11:30Affected orders are listed from the logs. Refunds go out the same day.

      What changed after

      No jump larger than 25 percentage points in one step, ever. A cohort dashboard split by flag is required before any flip, not after. The kill switch gets a monthly drill on a harmless flag. Every flag gets an owner and an expiry date at creation, and the pricing flag got both. The sale still ran. It ran on the old engine, and nobody outside the team knew anything had happened.

      If you remember nothing else

      Design exercise

      The prompt: You run checkout for a store with 100,000 daily users. A new pricing engine is ready. Roll it out to 10 percent of users, then to 100 percent, with a kill switch and a rollback plan. Walk through it.

      Clarifying questions to ask

      • How many checkouts run per minute at peak, and what is the current error rate?
      • What exactly changes in the pricing math, and which order shapes are affected?
      • What metric proves the new engine is correct: error rate, completed checkouts, revenue per order?
      • Who is allowed to flip the kill switch at 2 AM?
      • What is the safe default if the flag service goes down?

      Worked approach

      Ship the new engine dark behind a flag that defaults to off. Open the flag to internal users first, then to 10 percent of real users, about 10,000 people, for a 15 minute soak. Watch the error rate split by flag cohort, latency p99, and completed checkouts against the unflagged group. If the lines stay together, step to 50 percent for 10 minutes, then to 100 percent with a 5 minute confirm. Set the error threshold at 1.0 percent, so a bad flag is detected in about 3 minutes and killed in about 4 seconds. File the cleanup ticket on day one with a date six weeks out, and delete the flag and the old branch when the date arrives.

      Rollback plan

      If the flagged cohort errors, flip the flag off. That takes seconds and touches only flagged users. If the flag service itself is down, evaluation fails to the safe default, which is off, and checkout keeps running on the old engine. Only roll back the deploy if the code itself is broken for the unflagged path too. A deploy rollback is the last resort, not the first move.

      Trade-offs to name

      A slower rollout learns more safely but delays the revenue or savings the engine brings. A targeted rollout to internal users first is cheaper but proves less than real traffic. Percentage rollout is simple, but it cannot target a specific region or customer type, so a regional bug can hide in the average. The kill switch is fast, but it only helps if someone is watching the cohort dashboard when the spike starts.

      Saved in this browser, so the Lessons index can show your progress across the course.

      Next in this series
      Lesson 35 — Capacity Planning and Load Testing