Platform Engineering · Lesson 134

GitLab CI YAML: Your First Pipeline, Written by Hand

Lesson 67 showed you a GitLab pipeline running. This lesson is the one where you write it. One file, .gitlab-ci.yml, in the root of the repo. Three stages, three jobs, and a deploy that only runs when the tests pass. By the end you have a pipeline you typed yourself, and you know what every line does.

Difficulty: Intermediate · Know this first: Lesson 67 — GitLab Pipelines

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: on a cache hit the trip is almost free, on a miss the kitchen eats the budget. Step through the lesson or switch the scenario and watch the bar change.

Extra diagram · from scratch to master

The pipeline, mapped: push, file, stages, jobs

Start with one push. Add one box at a time until the whole pipeline is on the map, failed test included.

7 boxes on the trip3 stages1 failed test
Push
0 min
YAML file
1 min
Stages
2 min
Build job
8 min
Test job
6 min
Deploy
5 min
Failed test
trip ends
Request / flowResponse · the way backFailure / dead end

Try it: change the inputs and watch the trip bend

The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the build, the test, and the deploy change in the budget bar, in the hop timings, and in the total. The numbers are a teaching example, not live measurements. The shape is the lesson: more services means more of everything, and the test job is the gate.

3

How many services run this same file. Every service adds its own build, its own test run, and its own deploy. The file is written once. The work is paid once per service.

2

How many test jobs each service runs: unit tests, then integration tests, for example. Each test job costs about the test time below, in each service that runs it.

2

How long one test job takes, in minutes. A slow suite bends the whole pipeline, because the deploy waits for every test job to finish.

Try this: (1) Set services to 5, test jobs to 3, test time to 3. The build is about 15 min, the tests about 45 min, the deploy about 10 min, and the total is about 78 min. (2) Set all three to the worst end: 10 services, 8 test jobs, 10 min per job. The build is 30 min, the tests are 800 min, the deploy is 20 min, and the total is about 858 min. Same file. Only the inputs changed.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Try it: edit the pipeline and check it

Here is the pipeline from this lesson, ready to edit. Change it, then run the check. The check looks for the three things a working pipeline file needs: a stages list, a build job, and a script block. It also fails the file if it finds a tab, because YAML forbids tabs.

Try breaking it on purpose: delete the stages line, or replace two spaces with a tab. Run the check and watch it fail. Then press Reset and try a change that keeps it passing, like adding a second test job.

Press "Run the check" to see whether this pipeline does what the lesson asks.

Expected output when it passes: The file has stages, a build job, and a script block, and no tabs. This file would parse and the pipeline would start.

Basic to master

I take the same file four times here. First I learn the loop: push, file, stages, jobs. Then I learn rules and variables. Then I learn to split the file apart. Last I run it for the world.

Basic — One file, three stages

Definition: A pipeline is a file called .gitlab-ci.yml in the repo root. It lists stages in order, and each stage holds jobs. A job is a script that runs on a runner. The push starts the pipeline. The stages decide the order. The test decides whether the code may ship.

Types:

  • A stages list, which says the order: build, test, deploy.
  • A job, which names its stage and runs a script.
  • A runner, which is the machine that runs the job.
  • An environment, which names where the deploy lands.

Uses: I use this for one service and one repo. I write the file by hand, I push it, and I watch the pipeline run. Try it now: create the file from the Try it box, push it to a scratch repo, and watch the three columns turn green.

Dimensions: At low traffic everything is nearly free. What breaks first is the YAML: a tab, a wrong indent, a stage name that does not match. It costs one second to validate the file before pushing. It costs a confused hour when you skip that step.

Try it: Basic

My pipeline file has a tab and a stage name that does not match. What breaks first?

Now do it: In your repo, create the file from the Try it box, push it to a scratch repo, and watch the three columns turn green.

Intermediate — Rules, variables, and artifacts

Definition: Rules decide whether a job runs. Variables carry values into the jobs, like the app name or the target region. Artifacts carry files between jobs. Cache carries dependencies between runs. A job that needs another job's output lists it with needs, and starts as soon as that job finishes instead of waiting for the whole stage.

Types:

  • Rules, which run a job only on main, or only on a tag, or only when a file changed.
  • Variables, which are set in the file or in the project settings, and read as $NAME in scripts.
  • Artifacts, which carry built files from one job to the next.
  • Cache, which carries dependencies between runs so the second run is faster.

Uses: I use rules so the deploy runs only on main. I use variables so the same file deploys to staging and prod with different values. I use needs so the deploy starts the moment the test job passes. Try it now: add a rule to the deploy job in the Try it box so it runs only on main, and run the check.

Dimensions: As the team grows, the file grows, and the pipeline slows down. What breaks is the shared file: one team waiting on another team's job, or one bad variable breaking every job. It costs a split of the file, which is Lesson 135.

Try it: Intermediate

My deploy job runs on every branch. I want it to run only on main. What do I add?

Now do it: Add a rule to the deploy job in the Try it box so it runs only on main, and run the check.

Advanced — Splitting the file apart

Definition: One big config file gets hard to read, so the pipeline splits. Include pulls in another file. Extends copies a job and changes one part of it. The !reference tag reuses one block inside another job. A parent pipeline triggers a child pipeline, one per service or per team. A multi-project pipeline triggers a pipeline in another repo.

Types:

  • Include, which pulls a shared file into this pipeline.
  • Extends, which copies a job definition and overrides one field.
  • !reference, which reuses a single block, like a script, in several jobs.
  • Parent-child pipelines, which split one big pipeline into smaller ones that run on their own.

Uses: I use include when three repos share the same build steps. I use extends when the jobs are nearly the same and differ in one line. I use a parent-child split when one pipeline has grown past twenty jobs. Lesson 135 builds all of this, with the full code.

Dimensions: At ten times the jobs, the split is the lesson. What breaks is the shared file: a change in the template breaks five repos at once, and nobody knows which change did it. It costs version pins on every include and a test that runs the template before it ships.

The include that broke every repo. An include pulls a shared file into the pipeline, and a bad change to that file breaks every repo that includes it. I version the include by pinning the ref, and I test the change in one repo before the rest take it.

Do this: change an included file in a test branch, point one repo at that branch with the ref in the snippet, and run its pipeline. Expected: the test repo uses the new file and the other repos keep the old one. A green pipeline is the permission to merge, and the merge is what the rest of the repos get.

include:
  - project: my-group/pipeline-templates
    ref: test-branch
    file: /templates/jobs.yml

What the snippet means: the ref pins the version of the shared file. A test branch lets one repo try the change while the rest of the org stays on the release.

The extends that changed one line too many. Extends copies a job and changes one part of it, and the part it changes is the part the next person has to find. I extend small and override little, because a job that overrides half its parent is a job that should be its own job.

Do this: take a job that extends a hidden job, override one field, and run the pipeline. Expected: the job runs with the parent values and the one override. Override three fields, run again, and ask whether the job still belongs to the parent. If the answer is no, copy the job and own it.

.base-test:
  image: node:20
  script:
    - make test

test-api:
  extends: .base-test
  script:
    - make test-api

What the snippet means: the hidden job holds the shared parts, and the real job overrides only the script. One override is reuse. Three overrides is a copy wearing a disguise.

Try it: Advanced

I change a shared included file and five repos break at once. What would have contained the blast radius?

Now do it: Run the include drill above. Change an included file in a test branch, point one repo at that branch with the ref, and run its pipeline. A green pipeline is the permission to merge.

Master — The pipeline as production

Definition: At this level the pipeline stops being a tool and becomes a production system. I watch runner queue time the way I watch latency, with an alert on it. The config lives in a shared template repo. Policy checks run before merge. And I rehearse the worst day: the runner pool is down, the registry is down, the deploy is stuck halfway.

Types:

  • Queue-time alerts, which page a human when jobs wait too long for a runner.
  • Template repos, which hold the shared pipeline code with versions, like a library.
  • Policy checks, which reject a pipeline change that deploys to prod without a test.
  • Rehearsed failures, where the team practises the runner pool being down.

Uses: I use this when more than one team ships through the same pipelines, or when a bad pipeline can take down a service other teams depend on. It is also how I review a setup I did not build: I ask where the config lives, who can change it, and what happens when the runners are gone. Try it now: open any pipeline you own and find its queue time. If you cannot find it, that is the first gap to close.

Dimensions: At a hundred times the pushes, the pipeline is not the bottleneck, the blast radius is. What breaks is the single shared template that everything depends on, or the queue that one stuck job blocks for an hour. It costs real operational work: template versions, queue monitoring, and failure drills.

The pipeline as a production system. At this level the pipeline stops being a tool and becomes a production system. It has an owner, an alert on its queue time, and a runbook for the day the runner pool is down. I treat it like production because by then it is production.

Do this: write the runbook for a down runner pool in three lines: what waits, what goes by hand, and who decides. Expected: three lines, and the hotfix line says who approves it. A runbook that fits on a card is a runbook someone will read at 2 AM.

# runner pool down: 1) merges continue, deploys wait
# 2) hotfix deploys go by hand, with two reviewers
# 3) the on-call decides, and the decision is written in the incident channel

What the snippet means: the comment is the runbook. Three lines, and the third line names the person who decides. A runbook without a decider is a list of suggestions.

Policy checks before the merge. A policy check that runs after the merge is a report. A policy check that runs before the merge is a gate. I put the checks in the pipeline that runs on the merge request: no secrets in code, no unpinned dependencies, no latest tags.

Do this: add a policy job to a merge request pipeline that fails when a job uses the latest tag, and open a merge request with one. Expected: the pipeline fails and names the job. Fix the tag, push again, and expect the pipeline to pass. The gate is the check, and the check is in the pipeline, not in a document.

policy-check:
  stage: test
  script:
    - python3 -c "import yaml,sys; d=yaml.safe_load(open('.gitlab-ci.yml')); bad=[j for j,v in d.items() if isinstance(v,dict) and 'image' in v and str(v['image']).endswith(':latest')]; print(bad); sys.exit(1 if bad else 0)"

What the snippet means: the policy job reads the config and fails when any job pins the latest tag. A tag that moves is a deploy that changes without a review, and the check stops it before the merge.

Try it: Master

My policy check runs after the merge and reports a latest tag. What is wrong with that timing?

Now do it: Run the policy drill above. Add a policy job to a merge request pipeline that fails when a job uses the latest tag, open a merge request with one, and expect the pipeline to fail and name the job.

Go deeper

Start with the smallest file that works, and grow it one job at a time. The full file below is the one this lesson builds. Do this: save it as .gitlab-ci.yml in a scratch repo, run python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))", and push it. Expected output: the parse check prints nothing, and the pipeline page shows three stages with three jobs.

Read the flags: python3 -c and yaml.safe_load

A pipeline file that does not parse fails the whole run. This one-liner parses it before I push, so the failure happens on my machine, not in the pipeline.

  • python3 -c: The -c flag runs the code string that follows it. Without it, python3 would expect a file name, and the check would need a script file for one line of code.
  • import yaml: This loads the YAML library. The library knows how to read YAML. Without the import, the next call has no name to call.
  • yaml.safe_load(open('.gitlab-ci.yml')): Safe_load reads the file and turns it into Python data. Safe means it never runs code from the file. If the YAML is broken, safe_load raises an error and the check fails.
  • json.dumps (added): The lesson did not show this one yet. Wrapping the result in json.dumps prints the parsed file as JSON. I use it when safe_load passes but I want to see the structure it read, like the jobs and their stages.

Try it: validate the file with python3 -c and fix every error before pushing. Break one line of indentation and run it again. Expect an error that names the line.

stages: [build, test, deploy]

build-job:
  stage: build
  script:
    - make build
  artifacts:
    paths: [dist/]

test-job:
  stage: test
  script:
    - make test
  needs: [build-job]

deploy-job:
  stage: deploy
  script:
    - make deploy
  environment:
    name: production
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

What the snippet means: three stages in order, a build job that saves its output, a test job that starts as soon as the build job finishes, and a deploy job that runs only on main. The needs line is the difference between waiting for the whole stage and waiting for one job.

Rules are the cheapest gate in the pipeline. Do this: add a second rule to the deploy job that also requires a tag, push to main without a tag, and watch the deploy job skip. Expected output: the deploy job shows as skipped, and the pipeline still passes. A skipped job is a decision with a record, not a failure.

Cache is the second run being faster than the first. Do this: add cache: {paths: [node_modules/], key: $CI_COMMIT_REF_SLUG} to the build and test jobs, push twice, and compare the two test times in the job logs. Expected output: the second test run is shorter, because the dependencies came from the cache. If it is not shorter, the cache key or the path is wrong, and the log will show the cache as missed.

Variables keep secrets out of the file. Do this: put the deploy token in the project settings as a masked variable called DEPLOY_TOKEN, then use $DEPLOY_TOKEN in the deploy script. Expected output: the job log shows the token as asterisks, never in plain text. A token in the file is a token in git history, and git history is forever.

At 10x the jobs, the shape holds and the numbers grow. At 100x, the design changes: the file splits into includes, the shared parts live in a template repo, and the pipeline gets queue-time alerts. The file never stops being the checklist. It just gets more readers. Lesson 135 is where the split happens.

Anchors and hidden jobs, side by side

Anchors and hidden jobs are two ways to reuse a block, and they solve different problems. An anchor reuses a block inside one file. A hidden job reuses a job across files, through extends. I use an anchor when the reuse stays in one file, and a hidden job when the reuse crosses files.

Do this: save the file from the snippet, run the YAML parse check on it, and push it. Expected: the parse check prints nothing, the anchor job never appears in the pipeline, and both real jobs use the anchored script. Change the anchored block once, and expect both jobs to change. That is the reuse, proved.

.test-anchor: &test-anchor
  script:
    - make test

test-unit:
  image: node:20
  <<: *test-anchor

test-api:
  image: node:20
  <<: *test-anchor
  script:
    - make test-api

What the snippet means: the anchor names a block, and the merge key pulls it into both jobs. The second job overrides the script, which is the part that differs. GitLab sees only the merged result, so the anchor is invisible to the pipeline.

One time it broke

What broke

A deploy job I owned ran on a scratch branch. It had no rules block, so it ran on every push, including the branch I made to try out a risky change. The deploy overwrote the staging app with broken code, and the team testing staging lost an afternoon. The pipeline did exactly what the file said. The file said nothing about branches.

Timeline

  • 10:04,I pushed a scratch branch to try a risky change to the checkout page.
  • 10:05,the pipeline started. The build passed, the tests passed, and the deploy job ran, because nothing told it not to.
  • 10:09,the staging app started serving the broken checkout page. The team testing staging reported it as a staging bug.
  • 10:20,I opened the pipeline page and saw the deploy job had run on my scratch branch.
  • 10:25,I added a rules block to the deploy job so it runs only on main, and redeployed staging from main.

What changed after

Two things. First, every deploy job gets a rules block, and a deploy job without one is treated as broken in review. Second, I check the pipeline page after every first push of a new branch, before I trust it. Try it now: open any deploy job you own and read its rules. If it has none, add one before the next push.

If you remember nothing else

Build it yourself

Build a working pipeline for a small web service. It must have three stages, build, test, and deploy. The test job must run the test suite. The deploy job must run only on main, must name its environment production, and must wait for the test job. It passes when the pipeline succeeds on main and the deploy job turns green.

Build it, step by step

  • Create .gitlab-ci.yml in the repo root with stages: [build, test, deploy], one job per stage, and a script in each job.
  • Add artifacts to the build job so the test job uses the built files instead of rebuilding.
  • Add a rules block to the deploy job so it runs only when the branch is main.
  • Validate the file with python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))" and fix every error before pushing.
  • Push to a scratch branch first and confirm the deploy job skips. Then merge to main and confirm the deploy job turns green.

What success looks like

Success looks like this: the parse check prints nothing. The pipeline on the scratch branch passes with the deploy job skipped. The pipeline on main passes with all three jobs green, and the environment page shows the deploy. If the deploy job runs on the scratch branch, the rules block is wrong. If the test job rebuilds, the artifact is missing.

If it fails, check this

If the pipeline does not appear, the file is misnamed or in the wrong folder. If the pipeline fails at once with a YAML error, the indent is wrong or a tab slipped in. If the deploy job runs on the scratch branch, the rule is on the wrong job or the branch check is wrong. If the test job is slow, check the cache before you blame the suite.

Next in this series
Lesson 135 — GitLab Shared Templates and Cross-Pipeline Triggers