Platform Engineering · Lesson 33

From Commit to Running Code

I push a commit. Then I wait. Build, tests, artifact, staging, canary, rollout. That wait is the pipeline, and the pipeline is the product. A slow pipeline does not just annoy engineers. It delays the fix that users are already waiting for. This lesson follows one commit from my laptop to production, hop by hop, and shows what happens when a test catches a bad commit and when a canary catches a bad deploy.

Difficulty: Advanced · Know this first: Lesson 01, Lesson 26

Run it as:

When it breaks, I check

One thing to remember

Extra diagram · Architecture map

One artifact, three homes

One commit pushed straight to production. I add the pipeline gates and staging, then the same image in canary and production.

Artifact: 1 image, tagged by commitCanary: 5%Same image: staging, canary, production
same imagesame imagesame imageDeveloper laptop /repoone commitPipelinebuild, testsArtifact registryone image, tagged bycommitStagingsame imageProduction canary5% sliceProduction100%
one commit
one commitsame image everywherewatching canaryrequest · flowingresponse · the way back

    Extra diagram · Second trip

    The way back

    A bad canary goes live to 5%. I follow it from the climbing error rate to the flip back, and the failing-test branch that ships nothing.

    Rollback: flip back, no rebuildHalt: automatic on error rateFailing tests: nothing ships
    and the other branchCanary goes live5%Error rate climbsRollout halts itselfTraffic flips backold version, norebuildDashboard watchedafter the flipFix preparedon a calm branchFailing-test branchpipeline stops,nothing ships
    the bad version
    bad versionerror rate climbsold version backrequest · flowingresponse · the way back

      Where the time goes

      Every hop charges rent in minutes, not milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: in a healthy deploy the test suite is the longest hop, and that is fine, because it is the hop that earns the right to ship. Switch the scenario and watch the bar change. In the failing test scenario, the bar stops early, and that is the pipeline doing its job. In the bad canary scenario, the rollout never happens, and the rollback takes the last slice of the bar.

      Push the pipeline and watch the blast radius

      The budget above is one fixed deploy. This box is yours. Move the sliders and watch what happens to the time to production, the number of users a bad canary touches, and how long the way back takes. I learned this the slow way: a slow pipeline does not just annoy engineers. It delays the fix that users are waiting for.

      How often the team ships. It does not change the time of one deploy. It changes how much pipeline time and risk the week holds.

      How much traffic the new version sees first. A bigger canary touches more users when it is bad, and takes longer to drain on the way back.

      The slow suite, Step 4 above. Move this and watch the diagram, the step panel, and the budget bar change with it.

      Time to production

      43 min

      Total trip, incl. watch

      45 min

      Blast radius, one bad canary

      5,000 users, of 100,000

      Expected users hit / week

      25,000 users

      Rollback time

      2.5 min

      Pipeline time / week

      215 min (3.6 hrs)

      Check yourself

      Three questions. No score is saved anywhere — this is just you against the material.

      Basic to master

      The same topic, four times. One commit, from the first pipeline to the level where I can roll back at 3 AM and trust the button.

      Basic — One commit to production

      Definition: A pipeline is the path one commit takes from my laptop to production: build, tests, artifact, staging, canary, rollout. Each step is a gate. A slow pipeline does not just annoy engineers. It delays the fix that users are already waiting for.

      Types:

      • Build, which turns the commit into one versioned artifact, one commit, one version, no exceptions.
      • Tests, fast ones first so a bad commit fails in minutes.
      • Staging, a deploy plus smoke tests that prove the service starts, answers, and reaches the database.
      • Canary, a small share of real traffic watching for trouble before the rollout.
      • Rollout, traffic stepping up in stages until the new version serves everyone.

      Uses: I use this level every time I ship. The shape to remember is that each gate is cheaper than the one after it, because a failure at the test hop costs minutes and a failure in production costs users.

      Dimensions: As load grows, the test suite is the longest hop, and that is fine, because it is the hop that earns the right to ship. What breaks is the green pipeline that ships a broken version, because the bug only shows up at full traffic or after an hour. What it costs is pipeline minutes on every deploy. What I give up is shipping straight to production; the wait is the price of the gates.

      Intermediate — Gates that earn trust

      Definition: At this level the gates have owners and rules. Fast tests block the merge. The slow suite blocks the deploy. The canary blocks the rollout. Rollback is one button: it points traffic at the previous image and stops. It builds nothing and tests nothing. If going back needs a build, I do not have a rollback. I have a second deploy, and it will take 40 minutes on the worst night of the quarter.

      Types:

      • Canary with a soak, five percent of traffic and ten minutes of watching the same dashboard every time.
      • Rolling deploys, where the load balancer drains one instance at a time.
      • Config deploys split from code deploys, so a bad timeout can never ride in with a code change again.
      • Database changes in two steps, expand then contract, so the old version never breaks the moment the new schema lands.

      Uses: I use this level in the incident in this lesson. A deploy went out with a code change and a config change in the same commit, straight to all traffic, because the release was late and someone decided the canary was the thing to cut. The code was fine. The config change raised a timeout from 2 seconds to 30, so the service waited on a slow dependency instead of failing fast. The deploy job was green. The dashboard was red. There was no rollback button, so going back meant checking out the old commit, building it again, and deploying it like a new release. Forty minutes of broken checkouts.

      Dimensions: As load grows, each service gets its own pipeline and its own test environment, so one team's deploy never blocks another's, and I cache the build and split the test suite across machines in parallel. What breaks is the pipeline that is green because nobody looks at it: a test disabled six months ago, a gate skipped for one urgent deploy and never put back. What it costs is test environments and artifact storage. What I give up is the freedom to skip a gate when a release is late; a late release with a canary is late, and a late release without one is an incident waiting for a time slot.

      Advanced — The pipeline as a system

      Definition: At ten times the deploys, the pipeline itself becomes a system I have to run. Builds queue behind each other. Test environments collide. Two deploys to the same service at the same time need a lock, or they overwrite each other. At a hundred times the deploys, humans stop watching each one, so the canary decision becomes automatic: error rate and latency compared against the old version, and the rollout halts itself when the numbers move.

      Types:

      • Automatic canary halt on error rate or latency moving against the old version.
      • Automatic rollback for the failures with a clear signal.
      • Deploys in dependency order, where a deploy to a service the others depend on waits for the canary to pass first.
      • Artifact retention rules, because a hundred deploys a day means a hundred images a day.

      Uses: I use this level for a team of five engineers and five services deploying daily. One shared pipeline would queue them behind each other, so each service gets its own pipeline, and one shared pipeline definition with a small config file per service, because five copies of the same pipeline drift apart within a month.

      Dimensions: As load grows, the artifact registry becomes a real system, and the disk bill arrives before the quarter ends. What breaks is a deploy to a dependency that lands before its canary passes. What it costs is the runner fleet and the retention rules. What I give up is a human watching every deploy; the automation has to be right, because nobody is there to catch it being wrong.

      Master — Deploying when it goes wrong

      Definition: At this level the deploy is judged by the dashboard after the rollout, not by the job status. I watch the same dashboard after every rollout, and the rollback is a button I have pressed before, tested every month, because a rollback I have never pressed is a rollback I do not have. Anyone on the team can press it, no approval needed, and the canary halts itself when the error rate moves.

      Types:

      • Post-rollout watch on the same dashboard, every time, including the 3 AM case.
      • Monthly rollback drills on a harmless deploy.
      • Gate audits, checking the gates the way I check the code.
      • A deploy rule written down: anyone can deploy, the pipeline is the adult in the room, and the button is free while the gates are strict.

      Uses: I use this level when the job is green and the dashboard is red, which is the gap this whole lesson is about. The public case I keep in mind is Knight Capital in August 2012: a deploy reactivated old code and the firm lost about 440 million dollars in under an hour, because there was no fast way back. The lesson I took from it is the same as the one in this lesson: the rollback is the deploy. A real public incident, not a story from any employer.

      Dimensions: As load grows, humans stop watching each deploy, so the drill and the audit are what keep the system honest. What breaks is the bug that only shows up at full traffic or after an hour, when the pipeline has already called it a success. What it costs is the discipline of drills, audits, and daytime deploys. What I give up is the Friday afternoon deploy; a deploy at 2 PM has five engineers awake to fix it, and a deploy at 2 AM has one tired person on call.

      Go deeper

      The lesson above is one commit on a good day. Production is the rest of it. Here is what I actually spend my time on once the pipeline is running.

      Trade-offs I weigh every time

      Speed and safety pull in opposite directions, and the pipeline is where I decide how much of each I get. A pipeline that runs every test ever written is safe and slow, and a slow pipeline gets skipped. A pipeline that runs only the fast tests is quick and blind. I split the difference. Fast tests block the merge. The slow suite blocks the deploy. The canary blocks the rollout. Each gate is cheaper than the one after it, because a failure at the test hop costs minutes and a failure in production costs users. The other trade-off is who can deploy. If only one person can press the button, deploys queue behind their calendar. If everyone can press it, someone will press it at 5 PM on a Friday. I let anyone deploy, and I make the pipeline the adult in the room: tests, canary, automatic rollback. The button is free. The gates are strict.

      Failure modes that do not look like failures

      The first is the green pipeline that ships a broken version. The tests passed, the canary was clean, and production is still broken, because the bug only shows up at full traffic or after an hour. The fix is not more tests. The fix is watching the same dashboard after the rollout, and a rollback that works when the dashboard turns red at 3 AM. The second is the pipeline that is green because nobody looks at it. A test was disabled six months ago, a gate was skipped for one urgent deploy and never put back, and now the pipeline passes everything because it checks nothing. I audit the gates the way I audit the code. The third is the deploy that works but takes so long that the team stops deploying. They batch a week of changes into one release, the release is huge, the rollback is scary, and the team deploys even less. That spiral is how a team ends up deploying once a month and calling it stability.

      What changes at 10x

      At ten times the deploys, the pipeline itself becomes a system I have to run. Builds queue behind each other. Test environments collide. Two deploys to the same service at the same time need a lock, or they overwrite each other. I give each service its own pipeline and its own test environment, so one team's deploy never blocks another's. I also start caching the build. Dependencies do not change on every commit, so I reuse the layers that did not change and rebuild only what did. And I split the test suite across machines in parallel, because a 12 minute suite that runs in 4 minutes across three runners is the same confidence for a third of the wait.

      What changes at 100x

      At a hundred times the deploys, humans stop watching each one. The canary decision becomes automatic: error rate and latency compared against the old version, and the rollout halts itself when the numbers move. Rollback becomes automatic too, for the failures with a clear signal. I also stop deploying services one at a time by hand. The pipeline deploys in dependency order, and a deploy to a service the others depend on waits for the canary to pass first. And the artifact registry becomes a real system, with retention rules, because a hundred deploys a day means a hundred images a day, and the disk bill arrives before the quarter ends.

      Pipeline config I actually use

      A generic pipeline definition, not tied to any one vendor:

      # Deploy pipeline. One commit in, one decision out: ship it or stop it.
      pipeline:
        build:
          runs_on: clean_runner
          steps:
            - checkout
            - build
            - tag_image: ${COMMIT_SHA}   # one commit, one version, no exceptions
        test:
          needs: build
          steps:
            - unit_tests_and_lint        # fast tests first, fail in 3 minutes
            - integration_suite          # slow suite, 12 minutes, split across 3 runners
        stage:
          needs: test
          steps:
            - deploy_image: staging
            - smoke_tests                # starts, answers, reaches the database
        canary:
          needs: stage
          steps:
            - deploy_image: production
              traffic: 5%                # small blast radius
            - watch: 10m                 # same dashboard, every time
              halt_if:
                error_rate: "> 1%"       # canary vs old version, side by side
        rollout:
          needs: canary
          steps:
            - traffic: 25% -> 50% -> 100%
              watch_each_step: true
        rollback:
          # One button. No build, no tests, no new deploy.
          # Points traffic at the previous image and stops.
          trigger: manual_or_auto
          target: previous_image

      The rollback job is the whole lesson in one block. It builds nothing. It tests nothing. It points traffic at the previous image and stops. If your rollback needs a build, you do not have a rollback. You have a second deploy, and it will take 40 minutes on the worst night of the quarter.

      One time it broke

      This one stayed with me because the deploy job was green the whole time. Green job, broken checkout. That is the gap this lesson is about.

      What broke

      A deploy went out with a code change and a config change in the same commit, straight to all traffic. The release was already late, so the team skipped the canary to catch up. The code was fine. The config change raised a timeout on a call to a slow dependency, so instead of failing fast, the service waited. Requests piled up, checkout latency climbed, and then checkout started failing. The deploy job was green. The dashboard was red. The gap between those two facts was the incident.

      16:40

      A deploy goes out to all traffic. Code and config in one commit. No canary, because the release was late and someone decided the canary was the thing to cut.

      16:55

      Error rate climbs from 0.5 percent to 6 percent. Checkout failures start. The deploy job still shows green, because the pipeline checks the build, not the users.

      17:05

      I read the diff. The code change is small and clean. The config change raised a timeout from 2 seconds to 30. The service now waits on a slow dependency instead of failing fast. The config was the bug.

      17:10

      I reach for the rollback button. There is no button. Rollback means checking out the old commit, building it again, and deploying it like a new release. That takes 40 minutes on a good day, and this is not a good day.

      17:50

      The old version is back. Error rate drops to normal. Forty minutes of broken checkouts, because going back cost as much as going forward.

      18:30

      I write down what has to change before the next deploy. Not a postmortem for a file drawer. A list with names and dates.

      What changed after

      Three things, and I still do all three. First, the canary became mandatory. No deploy goes to all traffic without a canary, and no one can skip it to catch up a late release. A late release with a canary is late. A late release without one is an incident waiting for a time slot. Second, config deploys split from code deploys. A config change ships on its own, through its own canary, so a bad timeout can never ride in with a code change again. Third, rollback became one button. It points traffic at the previous image. It builds nothing. We test it every month, because a rollback you have never pressed is a rollback you do not have.

      If you remember nothing else

      Try it yourself

      This is the kind of question I would ask in an interview, and the kind I ask myself before I change a real pipeline. Take a minute with it before you read my approach.

      You run a team of five engineers and five services. Each service deploys daily. Users must never see downtime, and a bad deploy must be reversible in under five minutes. Design the pipeline. How do the services deploy without taking each other down, what does the pipeline check before it ships, and what happens when the canary goes bad?

      Clarifying questions to ask first

      I would ask five things before I sized anything. How do the services talk to each other, and can an old version and a new version run side by side? Is the database change backwards compatible, or does the old version break the moment the new schema lands? What is the traffic pattern, and is there a quiet hour when a deploy is safer? Who watches the deploy, and who is allowed to press the rollback button at 2 AM? And what does a bad deploy look like for these services: errors, latency, or silent wrong answers?

      Rough sizing, worked

      Five services, one deploy each per day, is 5 deploys a day and 25 a week. One deploy takes about 43 minutes from push to all users on this lesson's pipeline, so the team spends about 25 x 43 = 1,075 minutes a week in pipeline time, roughly 18 hours, spread across five pipelines that run in parallel. That is why each service gets its own pipeline. One shared pipeline would queue them behind each other. Each deploy runs its own canary at 5 percent with a 10 minute soak, then rolls out in steps, 25 percent, then 50, then all. Old and new versions run side by side during the rollout, behind the load balancer, so no user ever hits a service that is down. Database changes go in two steps. First the expand: add the new column, ship the code that writes to both. Then the contract: remove the old column in a later deploy, after the old version is gone.

      Trade-offs to name

      I use rolling deploys, not blue-green, because five services each keeping a full spare copy doubles the bill and the team is small. Rolling costs nothing extra, and the load balancer drains one instance at a time. I keep one pipeline definition shared across the five services, with a small config file per service, because five copies of the same pipeline drift apart within a month. I set the rollback rule in advance: anyone on the team can press it, no approval needed, and the canary halts itself when the error rate moves. A rollback that needs a meeting is not a rollback. And I accept that the team deploys during the day, not at night, because a deploy at 2 PM has five engineers awake to fix it, and a deploy at 2 AM has one tired person on call.

      Stored in this browser only, so you can pick the course back up where you left it.

      Next in this series
      Lesson 34 — Feature Flags and Rollouts