Platform Engineering · Lesson 135

GitLab Shared Templates and Cross-Pipeline Triggers

Lesson 134 gave you one pipeline in one repo. This lesson is about five repos sharing one set of steps, and pipelines calling other pipelines. A template repo holds the shared jobs. Include pulls them in. A parent pipeline triggers child pipelines, and one project triggers a pipeline in another project. By the end you have built a shared template and a pipeline that calls another pipeline.

Difficulty: Advanced · Know this first: Lesson 134 — GitLab CI YAML: Your First Pipeline, Written by Hand

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: on a cache hit the trip is almost free, on a miss the kitchen eats the budget. Step through the lesson or switch the scenario and watch the bar change.

Extra diagram · from scratch to master

The shared pipeline, mapped: template, include, trigger

Start with one copied job. Add one box at a time until five repos share one template and pipelines call pipelines.

7 boxes on the trip5 repos1 failed child
Copied job
0 min
Template file
1 min
include
2 min
extends
2 min
Template repo
1 min
Child pipeline
6 min
Failed child
trip ends
Request / flowResponse · the way backFailure / dead end

Try it: change the inputs and watch the trip bend

The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the include, the child pipelines, and the downstream work change in the budget bar, in the hop timings, and in the total. The numbers are a teaching example, not live measurements. The shape is the lesson: sharing multiplies the value of one fix, and it multiplies the blast radius of one mistake.

5

How many repos include the same template. Every repo adds its own include and its own downstream work. The template is written once. The work is paid once per repo.

2

How many child pipelines one parent triggers. Each child costs about 4 min to run, and the parent waits for all of them when the strategy is depend.

4

How many jobs the shared template holds. More jobs means more to include, and more that can break every repo at once when the template changes.

Try this: (1) Set repos to 10, child pipelines to 4, template jobs to 6. The include is about 7 min, the children about 16 min, the downstream about 10 min, and the total is about 41 min. (2) Set all three to the worst end: 20 repos, 10 children, 10 jobs. The include is 13 min, the children are 40 min, the downstream is 20 min, and the total is about 81 min. Same template. Only the inputs changed.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Try it: edit the pipeline and check it

Here is a repo file that includes a shared template and triggers a child pipeline. Change it, then run the check. The check looks for the four things this lesson teaches: an include block, an extends line, a trigger block, and an anchor merged with <<:.

Try breaking it on purpose: delete the ref line, or change the trigger strategy to something else. Run the check and watch it fail. Then press Reset and try a change that keeps it passing, like adding a second child trigger.

Press "Run the check" to see whether this pipeline does what the lesson asks.

Expected output when it passes: The file includes a pinned template, extends a template job, triggers a child pipeline, and merges an anchor. This file would start a pipeline that shares code and calls another pipeline.

Basic to master

I take the same template four times here. First I learn to share one job. Then I learn versions and trust. Then I learn parent-child splits. Last I run it for the world.

Basic — Share one job

Definition: A template is a file of jobs that more than one repo uses. Include pulls it in. Extends copies a job and changes one part. The repo that includes the template decides the stages and the order. The template decides the steps.

Types:

  • An include block, which names the project, the file, and the version.
  • A template repo, which holds the shared files and nothing else.
  • Extends, which copies a job and overrides one field.
  • !reference, which reuses one block inside another job.

Uses: I use this when two repos have the same build job and I am tired of fixing the same bug twice. I move the job into the template repo, include it from both repos, and fix it once. Try it now: take the build job from the Try it box, move it into a template file in your head, and write the include that pulls it back.

Dimensions: At low traffic everything is nearly free. What breaks first is the unpinned include: a template change lands in your repo on the next push, and you did not ask for it. It costs one line to pin the ref. It costs a broken pipeline when you skip it.

Try it: Basic

Two repos share the same build job and I am tired of fixing the same bug twice. What do I do?

Now do it: Do the Try it above: take the build job, move it into a template file, and write the include that pulls it back.

Intermediate — Versions and trust

Definition: A template repo is a library, so it gets versions. Tags mark releases. A changelog says what changed. A major version bump warns every consumer that the change can break them. The template repo also needs tests: a pipeline that runs the template jobs against a scratch repo before the tag ships.

Types:

  • Tags, which mark a release like v1.2.0.
  • A changelog, which says what changed in each release.
  • Template tests, which run the shared jobs before the tag ships.
  • A CODEOWNERS file, which says who must approve a template change.

Uses: I use tags so no repo is forced onto a breaking change. I use template tests so a bad template never ships. I use CODEOWNERS so a template change gets a second pair of eyes. Try it now: look at any shared file you own and ask what version it is on. If the answer is newest, pin it.

Dimensions: As the consumers grow, the template grows, and the blast radius grows with it. What breaks is the template change that five repos did not expect. It costs a test pipeline and a changelog. The trade is that sharing is powerful and unforgiving in the same breath.

Try it: Intermediate

I ship a breaking change to the shared template. What do I do first?

Now do it: Do the Try it above: look at any shared file you own, ask what version it is on, and pin it if the answer is newest.

Advanced — Parent-child and multi-project pipelines

Definition: A parent pipeline triggers child pipelines, one per service or per team. A multi-project pipeline triggers a pipeline in another project. Strategy depend makes the parent wait. Without it, the parent finishes while the child is still running. The deploy job lists every child as a need, so it starts only when all of them pass. YAML anchors and hidden jobs are the third way to reuse: an anchor reuses a block inside one file, and a hidden job never runs but exists to be extended.

Types:

  • Parent-child triggers, which split one big pipeline into smaller ones.
  • Multi-project triggers, which let one project start a pipeline in another project.
  • Strategy depend, which makes the caller wait for the pipeline it started.
  • Needs lists, which say which jobs the deploy job waits for.

Uses: I use a parent-child split when one pipeline has grown past twenty jobs. I use a multi-project trigger when the app repo and the deploy repo live apart. I use needs on the deploy job so a failing child stops the deploy. Try it now: add a second child trigger to the Try it box and run the check.

Dimensions: At ten times the pipelines, the graph is the lesson. What breaks is the trigger that crosses a trust boundary nobody configured, or the child that is not listed in the deploy job's needs. It costs a diagram of who triggers whom, kept next to the template repo.

Strategy depend, and the parent that finished early. Without strategy depend, the parent finishes while the child is still running, and the parent reports success for a child that has not finished. I set depend when the parent must wait, and I check the child result before the parent reports.

Do this: trigger a child pipeline without strategy depend, and watch the parent. Expected: the parent finishes green while the child is still running. Add the strategy line from the snippet, trigger again, and expect the parent to wait. If the child fails, expect the parent to fail too. A parent that reports success for a failed child is lying.

trigger-child:
  stage: test
  trigger:
    include: child-pipeline.yml
    strategy: depend

What the snippet means: strategy depend makes the parent wait for the child, and the parent result follows the child result. Without it, the parent is a fire-and-forget, and the child result is invisible.

The template that every repo includes, and the version it pins. A template repo is a library, and a library change reaches every repo that includes it. I pin the include to a tag, and I bump the tag one repo at a time, because a bad template breaks every repo at once.

Do this: list the repos that include a template with the command in the snippet, and write down the ref each one pins. Expected: a list, and the repos on the old tag are the migration list. Bump one repo, run its pipeline, and expect it to pass before the next one moves.

grep -r "pipeline-templates" --include="*.yml" . | sort

What the snippet means: the grep is the version dashboard for the template repo. Every include names the version its repo loads, and the list shows which repos a bad template reaches.

Read the flags: grep -r --include and sort

A shared template is found by searching for the repos that use it. The search is the list of callers, and the list is the blast radius of any change to the template.

  • grep -r: The -r flag searches files under a directory. Without it, grep would stop at the directory and search nothing.
  • "pipeline-templates": This is the text to find. Every repo file that uses the template names it, and the name is the link between the template and its callers.
  • --include="*.yml": This limits the search to .yml files. Without it, grep would search every file, and the list would include docs that only mention the name.
  • . (the dot): This is the directory to search, the current one. It means start here and go down through every subfolder.
  • | sort: The pipe hands the matches to sort. Sort orders them, so the same repo groups together and I can count the callers.
  • grep -n (added): The lesson did not show this one yet. It prints the line number with each match. I use it when I need to open the file at the match and read the include line.

Try it: search the repo for the pipeline-templates name. Expect one line per caller. Change one job in the template and expect every caller on the list to be affected.

Try it: Advanced

My parent pipeline is green but the child is still running. What did I forget?

Now do it: In your GitLab project, trigger a child pipeline without strategy depend and watch the parent finish green, then add the strategy line and trigger again and watch the parent wait.

Master — Templates as a platform

Definition: At this level the template repo is a platform with owners, versions, tests, and a deprecation policy. Old versions get a sunset date. Breaking changes ship with a migration guide. A dashboard shows which repos pin which version, so a bad version can be found in one look. And I rehearse the worst day: the template repo is down, and no pipeline can include it.

Types:

  • A deprecation policy, which says when an old version stops working.
  • A version dashboard, which shows every consumer and its pinned version.
  • A migration guide, which ships with every major version.
  • A fallback plan, for the day the template repo cannot be reached.

Uses: I use this when more than one team ships through the same templates, or when a bad template can take down services other teams depend on. It is also how I review a setup I did not build: I ask where the templates live, who can change them, and what happens when they are gone. Try it now: pick one shared file you own and write its sunset date. If it has none, it has no owner.

Dimensions: At a hundred times the consumers, the template repo is not the bottleneck, the blast radius is. What breaks is the single template that everything depends on, or the version nobody can find. It costs real operational work: owners, tests, dashboards, and drills.

The template repo is down. Every pipeline includes files from the template repo. When that repo is down, no pipeline can include them, and no pipeline runs. I rehearse that day with a mirror, so the first outage is not the first test.

Do this: point a test pipeline at a template path that does not exist and run it. Expected: the pipeline fails before any job starts, with an error that names the missing file. That is the blast radius. The fallback is a mirror of the template repo, and the drill is changing the include to the mirror and watching the pipeline run.

include:
  - project: my-group/pipeline-templates-mirror
    ref: v1.2.0
    file: /templates/jobs.yml

What the snippet means: the include is the only thing that changes in the fallback. A mirror that nobody has included from is not a fallback, it is a hope.

The deprecation policy, with a sunset date. An old template version stops working one day, and the day it stops is the sunset date. I write the date in the template repo, and I warn every repo that still pins the old version, because a version that stops without warning is an outage with a calendar.

Do this: pick a template version, write its sunset date in the repo readme, and list the repos that still pin it. Expected: a list of repos and a date. Warn each repo owner, and expect the list to shrink before the date. A sunset date with no warning list is a date nobody meets.

# templates/jobs.yml v1.2.0 sunsets on 2027-03-01
# repos still on v1.2.0: list them, warn them, track the count weekly

What the snippet means: the comment is the deprecation notice. The date is the deadline, the list is the work, and the weekly count is the proof the migration is moving.

Try it: Master

The template repo is down and no pipeline can include it. What should I have rehearsed?

Now do it: In your GitLab project, point a test pipeline at a template path that does not exist, watch it fail before any job starts, then switch the include to the mirror and watch it run.

Go deeper

Start with the template file, then the repo file that uses it. The template below holds one shared job. Do this: save it as templates/jobs.yml in a repo called pipeline-templates, tag the repo v1.2.0, and run python3 -c "import yaml; yaml.safe_load(open('templates/jobs.yml'))". Expected output: the parse check prints nothing, and the file holds one job with no repo-specific values in it.

Read the flags: python3 -c and yaml.safe_load

The template file is YAML too, and it needs the same parse check as the pipeline file. A broken template breaks every repo that includes it.

  • python3 -c: The -c flag runs the code string that follows it. Without it, python3 would expect a file name.
  • yaml.safe_load(open('templates/jobs.yml')): Safe_load reads the file and turns it into Python data. Safe means it never runs code from the file. A broken file raises an error, and the check fails before any repo uses the template.
  • json.dumps (added): The lesson did not show this one yet. Wrapping the result in json.dumps prints the parsed file as JSON. I use it when safe_load passes but I want to see the jobs the template defines.

Try it: run the python3 -c check on templates/jobs.yml. Expect silence. Break one line of indentation and run it again. Expect an error that names the line.

.base-test:
  image: node:20
  script:
    - make test
  artifacts:
    reports:
      junit: report.xml

What the snippet means: a hidden job, named with a leading dot so it never runs on its own, that other jobs extend. It sets the image, the script, and the test report. Any repo that extends it gets all three, and overrides only what it must.

The repo file is short, because the template does the work. Do this: save the file below as .gitlab-ci.yml in a scratch repo, validate it the same way, and push it. Expected output: the pipeline shows the test job from the template, and the child pipeline appears as a downstream pipeline on the pipeline page.

include:
  - project: my-group/pipeline-templates
    file: templates/jobs.yml
    ref: v1.2.0

stages: [build, test, deploy]

test-job:
  extends: .base-test

child-a:
  stage: deploy
  trigger:
    include: child-service-a.yml
    strategy: depend

What the snippet means: the repo includes a pinned template, extends the base test job without changing it, and triggers a child pipeline that the parent waits for. Fourteen lines, and the repo owns almost none of the logic.

A multi-project trigger is one more block. Do this: add a job with trigger: {project: my-group/deploy-config, branch: main, strategy: depend} to a scratch repo, push, and open the deploy repo. Expected output: the deploy repo shows a pipeline started by the trigger. If it shows nothing, the project path is wrong or the deploy repo does not allow the trigger.

Anchors are the third way to reuse, and they work inside one file only. A block marked with &base-test is an anchor. A job that writes <<: *base-test merges that block into itself. Use anchors when two jobs in the same file share a block, like an image and a script. Use extends when the shared job lives in another file, because anchors cannot cross files: an anchor is a YAML feature, and GitLab only ever sees the merged result. The limit to remember is that anchors merge mappings, not jobs. An anchor fills in what is missing, and a key written in the job itself always wins over the merged key.

stages: [test, deploy]

.base-test: &base-test
  image: node:20
  script:
    - make test

test-job:
  <<: *base-test
  stage: test

test-job-lint:
  extends: .base-test
  script:
    - make lint

What GitLab sees after merging. Do this: save the file above as .gitlab-ci.yml and run python3 -c "import yaml, json; print(json.dumps(yaml.safe_load(open('.gitlab-ci.yml')), indent=2))". Expected output: the test job shows the image and the script merged in from the anchor, and the lint job still shows its extends line, because the YAML parser resolves anchors but not extends. GitLab resolves extends next, and the final result is the two runnable jobs below. The hidden job itself never runs.

test-job:
  image: node:20
  script: [make test]
  stage: test
test-job-lint:
  image: node:20
  script: [make lint]

Hidden jobs are jobs whose name starts with a dot, like .base-test. They never run. They exist only to be reused, and they combine with both of the other ways: a hidden job can carry an anchor, as .base-test does above, and a real job can extend it, as test-job-lint does. Do this: add a hidden job to a scratch repo file, push it, and open the pipeline page. Expected output: the pipeline shows only the real jobs. The hidden job is nowhere on the page, because it never runs. If it appears as a runnable job, the dot is missing from its name.

At 10x the repos, the shape holds and the numbers grow. At 100x, the design changes: the template repo gets owners, tests, a version dashboard, and a deprecation policy. The template never stops being shared code. It just gets more readers, and more ways to break them. The next lesson in this track does the same job on GitHub.

The anchor, the hidden job, and extends, in one file

Three reuse tools, one file, and each one has a job. The anchor reuses a block inside the file. The hidden job holds the shared job for extends. Extends copies the hidden job and changes one part. I use all three in the same file, and each one is visible in the merged result.

Do this: save the file from the snippet, run the YAML parse check, and push it. Expected: the parse check prints nothing, the pipeline shows two jobs, and both use the anchored script. Change the anchor once, and expect both jobs to change. That is the reuse, proved in one file.

.defaults: &defaults
  image: node:20
  before_script:
    - npm ci

.base-test:
  <<: *defaults
  script:
    - make test

test-unit:
  extends: .base-test

test-api:
  extends: .base-test
  script:
    - make test-api

What the snippet means: the anchor holds the defaults, the hidden job holds the test job, and the two real jobs extend it. GitLab sees only the merged result, so the anchor and the hidden job are invisible to the pipeline.

One time it broke

What broke

A template change I shipped broke five repos at once. The include had no pinned ref, so every repo pulled the newest template on its next push. I renamed a job in the template, and five pipelines failed with a job-not-found error within the hour. The template did exactly what it was told. Nobody had told it to stay still.

Timeline

  • 09:12,I renamed a job in the template repo and pushed to main, to clean up a name I did not like.
  • 09:20,the first repo pushed a change. Its pipeline failed: the job name it referenced no longer existed.
  • 09:35,four more repos failed the same way, each on its next push. Five teams asked what had changed.
  • 09:50,I reverted the rename, tagged the template v1.2.0, and pinned the ref in all five repos.
  • 10:15,all five pipelines were green again. The rename shipped a week later as v2.0.0, with a note in the changelog.

What changed after

Three things. First, every include pins a ref, and an unpinned include is treated as broken in review. Second, the template repo gets a tag and a changelog for every release. Third, a template change runs a test pipeline before it ships. Try it now: open any include you own and read its ref. If there is none, add one before the next push.

If you remember nothing else

Build it yourself

Build a shared template and a pipeline that uses it. Create a template repo with one shared test job, tag it v1.2.0, and include it from a scratch repo with a pinned ref. Add a child pipeline trigger with strategy depend. It passes when the scratch repo pipeline runs the template job, the child pipeline appears as downstream, and the deploy job turns green only after the child passes.

Build it, step by step

  • Create a repo called pipeline-templates with templates/jobs.yml holding one hidden base test job, and tag it v1.2.0.
  • In a scratch repo, write .gitlab-ci.yml with an include block naming the template project, file, and ref v1.2.0.
  • Add a test job that extends the base test job, and validate both files with the YAML parse check before pushing.
  • Add a child pipeline trigger with strategy depend, push, and confirm the child appears as a downstream pipeline.
  • Break the child pipeline on purpose in a scratch branch and confirm the parent stops and the deploy job never runs.
  • In the scratch repo file, add a hidden job .base-test with an anchor &base-test holding the image node:20 and the script make test, and a test job that merges it with <<: *base-test.
  • Add a second job that extends .base-test and overrides only the script with make lint, then validate the file with the YAML parse check and read the merged result.

What success looks like

Success looks like this: both files parse with no output from the check. The scratch repo pipeline shows the template test job running. The pipeline page shows the child as a downstream pipeline. When the child fails on purpose, the parent reports failed and the deploy job stays grey. The merged result must show the test job with image node:20, script make test, and stage test, and the lint job with image node:20 and script make lint. The hidden job .base-test must not appear as a runnable job on the pipeline page.

If it fails, check this

If the template jobs do not appear, the include project, file, or ref is wrong. If the parent finishes at once, the strategy is not depend. If the deploy runs while a child is still failing, the deploy job's needs list is missing that child. If a template change breaks repos that did not change, a ref is unpinned.

Next in this series
Lesson 68 — ArgoCD