Platform Engineering · Lesson 139

Jenkins Downstream Builds and Multibranch Pipelines

Lesson 138 gave you one library that twenty pipelines share. This lesson is about pipelines calling other pipelines. An upstream build triggers a downstream build, passes it parameters, and waits for it. A multibranch pipeline builds every branch and every pull request on its own. By the end you have built a chain of four jobs and a multibranch pipeline that tests every branch before it merges.

Difficulty: Master · Know this first: Lesson 138 — Jenkins Shared Libraries: Write One from Scratch

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: on a cache hit the trip is almost free, on a miss the kitchen eats the budget. Step through the lesson or switch the scenario and watch the bar change.

Extra diagram · from scratch to master

The chain, mapped: trigger, wait, branch, deploy

Start with one build. Add one box at a time until four jobs chain together and every branch builds itself.

7 boxes on the trip4 jobs in the chain1 failed job
Build
0 min
build job
2 min
Parameters
2 min
Downstream
2 min
Wait
6 min
Multibranch
3 min
Failed job
trip ends
Request / flowResponse · the way backFailure / dead end

Try it: change the inputs and watch the trip bend

The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the downstream calls, the waiting, and the deploy change in the budget bar, in the hop timings, and in the total. The numbers are a teaching example, not live measurements. The shape is the lesson: the chain is cheap, the wait is the cost, and a long chain waits the longest.

4

How many jobs chain together. Every job adds its own trigger, its own wait, and its own deploy step. The chain is written once. The waiting is paid once per job.

10

How many branches the multibranch pipeline builds. Every branch adds its own job, and the scan must find all of them.

3

How long each downstream job takes, in minutes. The upstream waits for all of them, so the wait is the chain length times this number.

Try this: (1) Set jobs to 6, branches to 20, wait to 5. The downstream is about 14 min, the waiting about 30 min, the deploy about 12 min, and the total is about 65 min. (2) Set all three to the worst end: 8 jobs, 50 branches, 15 min wait. The downstream is 21 min, the waiting is 120 min, the deploy is 16 min, and the total is about 166 min. Same chain. Only the inputs changed.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Try it: edit the pipeline and check it

Here is an upstream Jenkinsfile that triggers a downstream job, passes it a version, and waits for it. Change it, then run the check. The check looks for the three things this lesson teaches: a build job step, a parameters block, and wait true.

Try breaking it on purpose: change wait to false, or delete the parameters line. Run the check and watch it fail. Then press Reset and try a change that keeps it passing, like adding a second downstream trigger.

Press "Run the check" to see whether this pipeline does what the lesson asks.

Expected output when it passes: The Jenkinsfile triggers the downstream job, passes it a version, and waits for it. This pipeline would deploy only after the downstream job passes.

Basic to master

I take the same chain four times here. First I learn the trigger. Then I learn parameters and waiting. Then I learn multibranch. Last I run it for the world.

Basic — One job triggers another

Definition: A build step triggers a downstream job by name. Parameters carry values down the chain. Wait true makes the upstream build wait for the downstream, and pass only if the downstream passes. The upstream and downstream jobs link to each other, so a bad deploy can be traced back to the exact change.

Types:

  • A build step, which triggers a job by name.
  • Parameters, which carry values down the chain.
  • Wait true, which makes the upstream wait for the downstream.
  • The upstream link, which says which build caused this one.

Uses: I use this when the build and the deploy live in separate jobs and I want one build to mean the whole chain. I pass the version down so the downstream deploys the exact build that was tested. Try it now: write a build step with a version parameter in the Try it box, and run the check.

Dimensions: At low traffic everything is nearly free. What breaks first is the name mismatch: the job name must match exactly, including the folder path, or the trigger fails the upstream build. It costs one look at the job name. It costs a red build when you skip it.

Try it: Basic

I trigger a downstream job and the upstream build fails at once. What do I check first?

Now do it: On your Jenkins write a build step that triggers a downstream job by name with a version parameter, and check the name matches exactly, including the folder path.

Intermediate — Parameters, waiting, and propagation

Definition: Parameters are typed: string, boolean, choice. The downstream job declares them, and the upstream passes them. Propagate false lets the downstream fail without failing the upstream, for jobs that report but do not block. The build result can be read in the upstream, so the pipeline can decide what to do when the downstream fails.

Types:

  • Typed parameters, which the downstream job declares.
  • Propagate false, which lets a downstream failure not fail the upstream.
  • The build result, which the upstream reads after the wait.
  • A quiet period, which waits before the downstream starts.

Uses: I use propagate false for reporting jobs that must not block the deploy. I use the build result when the upstream must decide, not just pass or fail. I use typed parameters so a wrong value fails fast. Try it now: add propagate false to a reporting trigger in your head, and name what it would report.

Dimensions: As the chain grows, the wait grows, and the upstream build stays running while the whole chain runs. What breaks is the long chain: four jobs waiting in a row is four times the wait of one. It costs parallel jobs, which is the next tier.

Try it: Intermediate

A reporting job runs downstream and fails, but the deploy must still go ahead. What do I set?

Now do it: Add propagate false to a reporting trigger on your Jenkins and name what it reports. Then read the build result in the upstream and write down what the pipeline does when the downstream fails.

Advanced — Multibranch and pull requests

Definition: A multibranch pipeline scans the repo and creates one job per branch that has a Jenkinsfile. A pull request is a branch, so it gets its own build and its own status tick. Branch sources say where to look: one repo, or a whole org. Discovering pull requests from forks needs trust settings, or a fork can run code that reads your secrets.

Types:

  • A multibranch pipeline, which scans the repo for branches.
  • A pull request build, which reports its status back to the pull request.
  • Branch sources, which say which repos to scan.
  • Trust settings, which decide what a fork pull request may do.

Uses: I use multibranch so every branch builds itself without anyone creating a job by hand. I use PR builds so a branch that does not compile never gets reviewed. I use trust settings so a fork cannot read my secrets. Try it now: open a pull request in a repo you own and find its build status. If it has none, the branch has no Jenkinsfile.

Dimensions: At ten times the branches, the scan is the lesson. What breaks is the branch that deletes its Jenkinsfile, or the fork pull request that runs untrusted code. It costs trust settings and a scan schedule that matches how fast branches appear.

The downstream that ran with the wrong version. A downstream job triggered without a version parameter builds whatever is on its main branch, which may not be the version the upstream just tested. I pass the version as a parameter, every time, so the downstream deploys the thing the upstream built.

Do this: trigger a downstream job without a version parameter and read its log. Expected: it builds the latest main, not the upstream commit. Add the parameter from the snippet, trigger it again, and expect the log to show the upstream version. A downstream without a version is a deploy of whatever happens to be newest.

build job: 'deploy-service', parameters: [
  string(name: 'VERSION', value: env.BUILD_VERSION),
  string(name: 'COMMIT', value: env.GIT_COMMIT)
], wait: true

What the snippet means: the parameters carry the version and the commit from the upstream into the downstream. The downstream deploys the tested thing, not the newest thing.

The fork pull request that read your secrets. A pull request from a fork runs code from a stranger, and a pipeline that hands it secrets hands them to the stranger. The trust setting decides what a fork build may do, and the safe default is: build the code, touch nothing secret.

Do this: open the multibranch trust settings and read which strategies are allowed for fork pull requests. Expected: a setting that says whether fork builds may run, and whether they may use credentials. Set it so fork builds run without credentials, open a test fork pull request, and expect the build to run and the deploy step to fail on the missing secret. That failure is the boundary working.

pipeline {
  agent any
  stages {
    stage("Test") {
      steps { sh "make test" }
    }
  }
}

What the snippet means: the test stage needs no secrets, so a fork build can run it safely. The deploy stage is the one that needs credentials, and it never runs for a fork.

Try it: Advanced

A pull request comes in from a fork. What is the risk?

Now do it: Run the drill above: open the multibranch trust settings, set fork builds to run without credentials, open a test fork pull request, and expect the build to run and the deploy step to fail on the missing secret.

Master — Chains as a platform

Definition: At this level the chains run across teams, with owners, dashboards, and a failure drill. A chain of four jobs is a system, and the wait is the budget. Parallel stages cut the wait when the jobs do not depend on each other. A shared library holds the trigger code, so every chain calls the same step. And I rehearse the worst day: a downstream job is down, and the whole chain waits for it.

Types:

  • Parallel stages, which cut the wait when jobs do not depend on each other.
  • A shared library step, which holds the trigger code every chain uses.
  • A chain dashboard, which shows every link and its last result.
  • Rehearsed failures, where the team practises a downstream job being down.

Uses: I use this when more than one team ships through the same chains, or when a bad chain can take down a service other teams depend on. It is also how I review a setup I did not build: I ask where the chains are drawn, who owns each link, and what happens when a downstream job is gone. Try it now: draw the chain for one deploy you own, on paper, with every link. If you cannot draw it, nobody owns it.

Dimensions: At a hundred times the jobs, the chain is not the bottleneck, the blast radius is. What breaks is the single downstream job that every chain waits for, or the wait that one stuck job holds for an hour. It costs real operational work: parallel stages, dashboards, and drills.

The chain that waits forever. A chain of four jobs waits for the slowest link, and a downstream job that is down makes the whole chain wait. I put a timeout on the wait, and I write down what happens when the timeout fires: the chain fails loudly, and a human decides whether to retry or to roll back.

Do this: add the timeout wrapper from the snippet around a downstream trigger, set it short, and run the chain while the downstream is paused. Expected: the chain fails on the timeout with a message that names the downstream job. The timeout is the difference between a stuck chain and a failed chain, and a failed chain is better, because a failed chain pages someone.

timeout(time: 30, unit: "MINUTES") {
  build job: 'deploy-service', wait: true
}

What the snippet means: the timeout bounds the wait. A chain without a timeout waits forever, and a chain that waits forever tells nobody it is stuck.

Parallel stages, and the shared resource they fight over. Parallel stages cut the wait when the jobs do not depend on each other. They also share the agents, the database, and the test environment, and two jobs writing to the same test database will fail in ways that look random. I parallelise the stages that touch nothing shared, and I keep the rest in order.

Do this: run two parallel stages that both write to the same test database and watch the result. Expected: flaky failures, sometimes one, sometimes the other. Give each stage its own database name, run again, and expect both to pass. The parallel speedup is real, and the shared resource is the price.

parallel(
  unit: { sh "make test UNIT_DB=test_unit_a" },
  api:  { sh "make test UNIT_DB=test_unit_b" }
)

What the snippet means: the two stages run at the same time with different databases. Parallel stages share the machine, so they must not share the data.

Try it: Master

A chain of four jobs waits for the slowest link, and the downstream job is down. What do I put on the wait?

Now do it: Run the drill above: add the timeout wrapper around a downstream trigger, set it short, run the chain while the downstream is paused, and expect the chain to fail on the timeout with a message that names the downstream job.

Go deeper

Start with the upstream Jenkinsfile, then the multibranch setup. The upstream file below builds, then triggers the downstream job with a version parameter, and waits for it. Do this: save it as Jenkinsfile in a scratch repo, check the braces balance, and push it. Expected output: the upstream build stays running while the downstream runs, then turns green when the downstream passes.

pipeline {
  agent { label 'build' }
  stages {
    stage('Build') {
      steps { sh 'make build' }
    }
    stage('Test') {
      steps { sh 'make test' }
    }
    stage('Trigger downstream') {
      steps {
        build job: 'deploy-service',
          parameters: [string(name: 'VERSION', value: env.BUILD_NUMBER)],
          wait: true
      }
    }
  }
  post {
    failure {
      echo 'The chain failed. Check the downstream job first.'
    }
  }
}

What the snippet means: three stages, then a trigger that passes the build number down as the version and waits for the downstream job. The post block runs only on failure, and it says where to look first. The upstream build is the whole chain.

Propagate false is for jobs that report but do not block. Do this: add a second build step with propagate: false that triggers a reporting job, push, and watch the upstream build. Expected output: the upstream build turns green even when the reporting job fails, and the reporting job still shows its own red result. If the upstream turns red too, propagate is not set on that step.

Multibranch is a job type, not a Jenkinsfile change. Do this: create a multibranch pipeline, point it at a scratch repo, add a Jenkinsfile to two branches, and run the scan. Expected output: two jobs appear, one per branch. Then open a pull request and look for the build status on the pull request page. Expected output: a green tick or a red cross, with a link to the build.

Parallel stages cut the wait when jobs do not depend on each other. Do this: wrap two independent test stages in a parallel block, push, and watch the run. Expected output: both stages run at the same time, and the total is the slower of the two, not the sum. If they run one after another, the parallel block is in the wrong place.

At 10x the jobs, the shape holds and the numbers grow. At 100x, the design changes: the chains get a dashboard, the triggers live in the shared library, and the waits get a budget. The chain never stops being jobs calling jobs. It just gets more links. This is the last lesson in the pipelines track.

The chain dashboard, read in one look

A chain of four jobs is a system, and a system needs a view. I want one page that shows every link, its last result, and how long it waited. When a chain is slow, the dashboard says which link is slow, and I fix that link.

Do this: list the last five builds of each job in a chain with the command in the snippet, and write down the duration of each. Expected: five numbers per job, and the slowest job is visible. That job is the work. A chain dashboard is that list, kept current, and the slowest link is always the first thing I fix.

curl -s -u admin:TOKEN "http://localhost:8080/job/deploy-service/api/json?tree=builds[number,result,duration]" | python3 -m json.tool | head -30

What the snippet means: the API lists recent builds with their result and duration. Five builds is enough to see whether a job is slow once or slow always.

Read the flags: curl -s -u and ?tree=

The job API lists the last builds with their numbers, results, and durations. One call answers whether the downstream job is green and how long it takes.

  • -s: This is silent mode. Curl prints no progress output, so the JSON stays clean for the pipe to Python.
  • -u admin:TOKEN: The -u flag sends basic auth. Without it, the API answers 403 and the JSON is an error page.
  • ?tree=builds[number,result,duration]: The tree part asks for only three fields per build: number, result, and duration. Without it, the answer is the whole job object, which is pages long.
  • | python3 -m json.tool | head -30: The first pipe pretty-prints the JSON. The -m flag runs json.tool as a program. The second pipe keeps the first thirty lines, which is enough to see the last few builds.
  • -f (added): The lesson did not show this one yet. It makes curl fail on an HTTP error instead of printing the error page as if it were data. I add it in scripts, so a missing job fails the run instead of printing a confusing JSON error.

Try it: run the curl for the deploy-service job and read the last builds. Expect a number, a result, and a duration per build. Compare the duration of the last green build with the one before it.

One time it broke

What broke

A deploy I owned ran while the downstream job was still failing. The build step had wait false, so the upstream build turned green the moment the trigger fired. The deploy job ran on the next stage, because nothing told it to wait. The chain did exactly what the file said. The file said nothing about waiting.

Timeline

  • 11:04,the upstream build ran. It triggered the downstream deploy job with wait false, and turned green at once.
  • 11:05,the downstream job failed on a bad config. Nobody noticed, because the upstream was already green.
  • 11:06,the next stage in the upstream pipeline ran the deploy, because the upstream had passed.
  • 11:12,the app started serving the broken config. The team reported it as a deploy bug.
  • 11:20,I set wait true on the build step, rolled back the deploy, and re-ran the chain.

What changed after

Two things. First, every build step that gates a deploy uses wait true, and a build step with wait false is treated as a reporting trigger in review. Second, the deploy stage lists the downstream job as its gate, so it cannot run while the chain is still red. Try it now: open any build step you own that gates a deploy and read its wait value. If it is false, change it before the next run.

If you remember nothing else

Build it yourself

Build a chain of two jobs and a multibranch pipeline. The upstream Jenkinsfile must trigger a downstream job with a version parameter and wait true. The multibranch pipeline must build every branch that has a Jenkinsfile, and every pull request must show a build status. It passes when the upstream build turns green only after the downstream passes, and a pull request shows a green tick.

Build it, step by step

  • Create a downstream job called deploy-service that takes a string parameter VERSION and prints it in its log.
  • Write an upstream Jenkinsfile with build and test stages, then a trigger stage with a build step, a VERSION parameter, and wait true.
  • Push to main and confirm the upstream build stays running while the downstream runs, then turns green when it passes.
  • Break the downstream job on purpose in a scratch branch and confirm the upstream build turns red and the deploy never runs.
  • Create a multibranch pipeline for the same repo, open a pull request, and confirm the pull request shows a build status.

What success looks like

Success looks like this: the downstream log shows the version the upstream passed. The upstream build turns green only after the downstream passes. When the downstream fails on purpose, the upstream turns red and nothing deploys. The pull request shows a green tick with a link to the build.

If it fails, check this

If the downstream job never starts, the job name in the build step is wrong. If the upstream turns green at once, wait is false. If the version arrives empty, the parameter name does not match on both sides. If the pull request shows no status, the branch has no Jenkinsfile or the scan has not run yet.

Next in this series
Lesson 112 — GitHub Actions