I used to think shipping code and releasing a feature were the same thing. They are not. A flag lets me ship code on Tuesday and release it on Friday, to ten percent of users first, and switch it off in seconds if it misbehaves. This lesson follows one flag flip from definition to cleanup, through a gradual rollout, a bad flag killed fast, and a flag nobody remembered to remove.
Difficulty: Advanced · Know this first: Lesson 33, Lesson 24
Extra diagram · Architecture map
One flag, one user, on or off. I add the console and the push to the app servers, then the metrics and the audit log.
Extra diagram · Second trip
A flag flips to 10%, and the flagged cohort starts erroring. I follow it from the split to the kill, and the cleanup that keeps flags from becoming permanent.
The clock below is a worked example for this lesson's scenario, not a measurement from a live system. The shape is the lesson: most of the time goes to watching, not to flipping. A flip takes seconds. A soak takes minutes. That ratio is what keeps a rollout safe. Steps that cost 0 ms in a scenario are left out of the bar but still listed in the legend.
Same worked example: 100,000 total users. Move the controls and watch what changes. The rollout percentage decides how many users a bad flag can reach. The error-rate threshold decides how long it runs before anyone notices. The live flag count decides how long the kill takes and how much cleanup debt the team is carrying. In the "A bad flag, killed fast" scenario above, steps 6 and 7 follow these controls live.
How much of the user base the flag can reach right now.
A low threshold trips sooner and cries wolf more often. A high threshold trips later and lets more users hit the bug.
More live flags means a slower hunt for the right switch and more stale branches to untangle later.
Exposed users
10,000
of 100,000 total users
Time to detect a bad flag
3 min
180,000 ms of watching
Time to kill it
4 sec
4,000 ms, no deploy needed
Flag debt
Watch it
15 flags over the healthy budget of 25 = 7.5 cleanup hours
At the defaults, detect plus kill is 184,000 ms: 180,000 ms to detect plus 4,000 ms to kill. That is exactly steps 6 plus 7 of the bad flag scenario above. Push the rollout to 100% with a 5.0% threshold and 200 live flags and the same bad flag reaches all 100,000 users, runs 15 minutes before detection, and sits on 87.5 hours of flag debt.
Three questions. No score is saved anywhere — this is just you against the material.
The same topic, four times. One flag flip, from the first switch to the level where I can kill a bad flag at 2 AM and trust the switch.
Definition: A feature flag is a named switch the code reads at runtime. Shipping code and releasing a feature are not the same thing. A flag lets me ship code on Tuesday and release it on Friday, to ten percent of users first, and switch it off in seconds if it misbehaves. No deploy is needed to turn it off.
Types:
Uses: I use this level when a feature is ready to ship but not ready to be seen, or when I want a way back that is faster than a rollback. The flip takes seconds. The soak takes minutes. That ratio is what keeps a rollout safe.
Dimensions: As load grows, one flag is still one flag, and the cost stays small. What breaks is the flag nobody remembered to remove: the code keeps carrying two paths, old and new, and the old path rots. What it costs is one more config to maintain and one more branch in the code. What I give up is a codebase with one path through it; until cleanup, there are two.
Definition: A percentage rollout hashes the user id together with the flag name, so the same user always lands in the same bucket. I open the flag to internal users first, then to ten percent of real users for a soak, watching the error rate split by flag cohort, then step to 50 percent, then to 100 percent. If the flagged cohort errors, I flip the flag off. That takes seconds and touches only flagged users.
Types:
Uses: I use this level for the new pricing engine in this lesson. The plan was 10 percent, then 50, then 100. The on-call engineer skipped the steps and flipped the flag straight from 0 to 100 percent, because the sale started in an hour and the code had passed tests. The new engine rounded tax wrong on orders with a discount code. No alert fired, because the dashboard showed the blended error rate. The first signal was a support ticket, then three more. The flag was flipped off 22 minutes after the flip, and the overcharges stopped.
Dimensions: As load grows, the flag service becomes a critical dependency, because every request asks it a question even if the answer comes from a local cache. What breaks is a percentage typed wrong, a bucket that is not sticky, or a cohort dashboard that was never opened. What it costs is evaluation on every request and the soak time at each step. What I give up is speed: a slower rollout learns more safely but delays the revenue or savings the feature brings.
Definition: At this level flag evaluation is local and fast. The code evaluates the flag from a cached config and never blocks a request on the flag service. If the flag service goes down, evaluation fails to the safe default and checkout keeps running on the old engine. Config changes push out in seconds, and the push is versioned, so a bad config can be rolled back like a bad deploy.
Types:
Uses: I use this level when the flag service is down or a config push stalls. Servers keep serving the last known good config and alert on its age. When an incident review asks what changed, the flip log is the first place I look, because most flag incidents I have seen were not bad code. They were a flip nobody recorded, a percentage typed wrong, or a flag whose owner left the team.
Dimensions: As load grows, the count matters more than any single flag, and I cap live flags per team at around 50. What breaks is a config push that stalls, or two flags interacting in a way nobody reviewed. What it costs is audit storage and the review time before a new flag ships. What I give up is instant global flips with no record; every flip is now reviewable, which is the point.
Definition: At this level every flag has an owner and an expiry date at creation, and the kill switch gets a runbook, not just a button. The runbook says who can flip it, how to reach the flag console when the normal network path is down, and what to check after the flip. Then I drill it: flip a harmless flag off and on during a quiet hour, once a month, and time it. A kill switch I have never flipped is a hope, not a control.
Types:
Uses: I use this level after the sale-week incident in this lesson. The rules that came out of it are still the rules: no jump larger than 25 percentage points in one step, ever. A cohort dashboard split by flag is required before any flip, not after. Every flag gets an owner and an expiry date at creation. The sale still ran, on the old engine, and nobody outside the team knew anything had happened.
Dimensions: As load grows, flag debt grows with the team, not with traffic, and the live count is the number I watch. What breaks is a flag in limbo at 100 percent with no cleanup date, or an override list nobody removes, which is a backdoor. What it costs is cleanup work and drill time every month. What I give up is shipping speed on flag creation, and I get back a live list small enough to reason about at 2 AM.
A flag is a small idea with a long tail. The rollout in this lesson is the easy part. The hard part is everything around it.
Targeting is where flags get subtle. A percentage rollout hashes the user id together with the flag name, so the same user always lands in the same bucket. An override list sits on top for internal users and test accounts. Overrides are how the team tries the new path before any real user sees it. Keep the override list short and review it, because an override nobody removes is a backdoor.
The kill switch needs a runbook, not just a button. Write down who can flip it, how to reach the flag console when the normal network path is down, and what to check after the flip. Then drill it. Flip a harmless flag off and on during a quiet hour, once a month, and time it. A kill switch you have never flipped is a hope, not a control.
Give every flag a lifecycle: created, rolling out, at 100 percent, cleanup scheduled, deleted. A flag that sits at 100 percent with no cleanup date is a flag in limbo. Put the cleanup date in the flag description on day one, and put a ticket on the board the same day. I set the expiry for six weeks out and let the ticket remind me.
At 10x scale the flag service becomes a critical dependency. Every request asks it a question, even if the answer comes from a local cache. That cache needs a time to live and an age metric, so a stale config shows up on a dashboard instead of hiding in production. Config changes should push out in seconds, and the push should be versioned, so a bad config can be rolled back like a bad deploy.
At 100x scale the count matters more than any single flag. Cap live flags per team, say 50, and make exceeding the cap a conversation, not a silent drift. Keep an audit log of every flip: who, when, old value, new value. When an incident review asks what changed, that log is the first place I look. Most flag incidents I have seen were not bad code. They were a flip nobody recorded, a percentage typed wrong, or a flag whose owner left the team.
Failure modes to name: the flag service goes down, so evaluation must fail to the safe default and keep serving. A config push stalls, so servers must keep serving the last known good config and alert on its age. A percentage change lands mid-request, so bucketing must stay sticky for the session. Two flags interact, so new flags get reviewed against the live list before they ship.
flag: new-pricing-engine default: false rollout: percentage: 10 bucket_key: user_id kill_switch: true owner: pricing-team expires: 2027-02-01
// Evaluate locally. Never block a request on the flag service.
const on = flags.evaluate("new-pricing-engine", user.id, { default: false });
if (on) { return newPricing(user, cart); }
return legacyPricing(user, cart);
During a sale week, a new pricing engine went live behind a flag. The plan was 10 percent, then 50, then 100. The on-call engineer skipped the steps and flipped the flag straight from 0 to 100 percent, because the sale started in an hour and the code had passed tests. The new engine rounded tax wrong on orders that carried a discount code. A slice of customers was overcharged by small amounts. No alert fired, because the dashboard showed the blended error rate, and the checkout error rate looked normal. The first signal was a support ticket, then three more.
No jump larger than 25 percentage points in one step, ever. A cohort dashboard split by flag is required before any flip, not after. The kill switch gets a monthly drill on a harmless flag. Every flag gets an owner and an expiry date at creation, and the pricing flag got both. The sale still ran. It ran on the old engine, and nobody outside the team knew anything had happened.
The prompt: You run checkout for a store with 100,000 daily users. A new pricing engine is ready. Roll it out to 10 percent of users, then to 100 percent, with a kill switch and a rollback plan. Walk through it.
Ship the new engine dark behind a flag that defaults to off. Open the flag to internal users first, then to 10 percent of real users, about 10,000 people, for a 15 minute soak. Watch the error rate split by flag cohort, latency p99, and completed checkouts against the unflagged group. If the lines stay together, step to 50 percent for 10 minutes, then to 100 percent with a 5 minute confirm. Set the error threshold at 1.0 percent, so a bad flag is detected in about 3 minutes and killed in about 4 seconds. File the cleanup ticket on day one with a date six weeks out, and delete the flag and the old branch when the date arrives.
If the flagged cohort errors, flip the flag off. That takes seconds and touches only flagged users. If the flag service itself is down, evaluation fails to the safe default, which is off, and checkout keeps running on the old engine. Only roll back the deploy if the code itself is broken for the unflagged path too. A deploy rollback is the last resort, not the first move.
A slower rollout learns more safely but delays the revenue or savings the engine brings. A targeted rollout to internal users first is cheaper but proves less than real traffic. Percentage rollout is simple, but it cannot target a specific region or customer type, so a regional bug can hide in the average. The kill switch is fast, but it only helps if someone is watching the cohort dashboard when the spike starts.
Saved in this browser, so the Lessons index can show your progress across the course.