Platform Engineering · Lesson 132

GitLab Runners and Environments

Lesson 67 gave you GitLab pipelines. A pipeline is a checklist that runs itself. This lesson is about the machines that run the checklist, and the places the checklist deploys to. Runners pick up the jobs. Environments receive the deploys. When a job waits twenty minutes, the runner is usually the story.

Difficulty: Advanced · Know this first: Lesson 67 — GitLab Pipelines

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: on a cache hit the trip is almost free, on a miss the kitchen eats the budget. Step through the lesson or switch the scenario and watch the bar change.

Extra diagram · from scratch to master

The runner, mapped: queue, runner, environment

Start with one push. Add one box at a time until the whole trip from push to staging is on the map, offline runner included.

7 boxes on the trip1 queue1 offline runner
Push
0 min
Queue
2 min
Runner
1 min
Executor
1 min
Job
9 min
Environment
3 min
Offline
trip ends
Request / flowResponse · the way backFailure / dead end

Try it: change the inputs and watch the trip bend

The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the queue, the job, and the deploy change in the budget bar, in the hop timings, and in the total. The numbers are a teaching example, not live measurements. The shape is the lesson: the queue is usually the wait, and the job is usually the work.

9

How many jobs the pipeline runs. The job time grows with the number of jobs, but only by about three fifths of each job's time, because some jobs run in parallel.

2

How long a job waits for a free runner. Two minutes is a normal busy hour. Thirty minutes means the shared runners are full, or the job asked for something no runner has.

5

How long one job takes to run. Five minutes is a small build and test. Twenty minutes is a job that should probably be split in two.

Try this: (1) Set jobs to 20, queue wait to 10, job time to 10. The queue is 10 min, the jobs are 120 min, the deploy is 10 min, and the total is about 144 min. (2) Set all three to the worst end: 50 jobs, 30 min queue, 20 min jobs. The queue is 30 min, the jobs are 600 min, the deploy is 25 min, and the total is about 659 min. Same pipeline. Only the inputs changed.

Type your own numbers: Get the total under 150 minutes. Type 20 jobs, 10 minutes of queue wait, 10 minutes per job, and run it. Then set the queue wait to 30 and watch the total.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

I take the same pipeline four times here. First I learn what a runner is. Then I learn tags, executors, and caching. Then I learn environments, review apps, and deploys. Last I run the runners as production.

Basic — What a runner is

Definition: A runner is a small agent that asks GitLab for work. It picks up a job, runs it in an executor, and sends the result back. Shared runners belong to GitLab and serve everyone. Specific runners belong to you and serve your jobs. A job waits in a queue until a runner that matches it is free.

Types:

  • Shared runners, free, busy, and shared with everyone.
  • Specific runners, yours, on your machines, with your tags.
  • Group runners, shared by a group of projects.
  • Executors, docker, shell, or virtual machine, the room the job runs in.

Uses: I use shared runners for small jobs that need nothing special. I use my own runner the moment a job needs a private network, a big disk, or a warm cache.

Dimensions: At low traffic everything is nearly free. What breaks first is the queue: a busy hour, and every job waits. It costs almost nothing to watch the queue time. It costs an afternoon when a slow pipeline turns out to be a waiting pipeline.

Try it yourself

Save this file as .gitlab-ci.yml. It has three stages and four jobs. Parse it with Python and count the jobs.

build:
  artifacts:
    paths:
    - app.tar
  script:
  - echo "building"
  stage: build
  tags:
  - docker
deploy-prod:
  environment:
    name: prod
    url: https://prod.example.com
  script:
  - echo "deploying to prod"
  stage: deploy
  tags:
  - deploy
  when: manual
deploy-staging:
  environment:
    name: staging
    url: https://staging.example.com
  script:
  - echo "deploying to staging"
  stage: deploy
  tags:
  - deploy
stages:
- build
- test
- deploy
test:
  script:
  - echo "testing"
  stage: test
  tags:
  - docker
python3 -c "import yaml; d=yaml.safe_load(open('.gitlab-ci.yml')); print(len([k for k,v in d.items() if isinstance(v,dict)]), 'jobs')"

What you should see: 4 jobs

It passes when: the parse prints 4 jobs and no error. A YAML error here is a pipeline that never starts.

Intermediate — Tags, executors, and caching

Definition: Tags match jobs to runners: a job that asks for a GPU tag waits for a runner with a GPU. Executors decide the room: docker starts clean every time, shell runs on the machine itself. Caching saves dependencies between jobs, so a warm cache skips the download. Artifacts pass work between jobs, and they are a promise, not a speed trick.

Types:

  • Tags, which match a job to the runner that can run it.
  • Docker executors, clean every time, nothing survives unless saved.
  • Shell executors, fast, on the machine, and able to touch anything on it.
  • Caches and artifacts, the two ways jobs remember, with different promises.

Uses: I use tags when a job needs something specific: a GPU, a private network, a big disk. I use the docker executor when a job must start clean. I use the cache for dependencies and artifacts for outputs, and I never confuse the two.

Dimensions: As the pipeline grows, the cache grows, and the artifact grows with it. What breaks is the cache: a stale cache that serves yesterday's dependencies, or an artifact so big the next job spends longer downloading it than rebuilding it. It costs a cache policy and an artifact size check. The trade is that speed tricks need owners.

Try it yourself

Save runners.yml and check-pipeline.py next to the CI file. The checker reads the pipeline and the runner list and fails on any tag no runner carries.

runners:
  docker: 3
  deploy: 2
#!/usr/bin/env python3
"""Check a .gitlab-ci.yml the way the pipeline would: structure, tags, gates."""
import sys, yaml

def load(path):
    with open(path) as f:
        return yaml.safe_load(f)

def check(path, runners):
    ci = load(path)
    errors = []
    stages = ci.get("stages", [])
    for name, job in ci.items():
        if name.startswith(".") or not isinstance(job, dict):
            continue
        if "stage" in job and job["stage"] not in stages:
            errors.append(f"{name}: stage '{job['stage']}' is not in stages {stages}")
        for tag in job.get("tags", []):
            if tag not in runners:
                errors.append(f"{name}: tag '{tag}' is carried by no runner")
        if job.get("stage") == "deploy" and job.get("environment", {}).get("name") == "prod":
            if job.get("when") != "manual":
                errors.append(f"{name}: prod deploy is not manual")
    if errors:
        print("FAIL")
        for e in errors:
            print("  - " + e)
        return 1
    print("PASS: %d jobs, stages %s, every tag has a runner, prod deploy is manual" % (
        len([k for k, v in ci.items() if isinstance(v, dict) and not k.startswith(".")]), stages))
    return 0

if __name__ == "__main__":
    runners = load(sys.argv[2])["runners"]
    sys.exit(check(sys.argv[1], runners))
python3 check-pipeline.py .gitlab-ci.yml runners.yml

What you should see: PASS: 4 jobs, stages ['build', 'test', 'deploy'], every tag has a runner, prod deploy is manual

Read the flags: python3 check-pipeline.py with file arguments

A check script takes files as arguments. The two files here are the pipeline and the runner list, and the script checks that every job has a runner that can run it.

  • python3 check-pipeline.py: This runs the file check-pipeline.py. Python3 is the interpreter, and the file name is the program. No flag is needed, because a file name is the default input.
  • .gitlab-ci.yml runners.yml: These are the two arguments, in order. The first is the pipeline file. The second is the runner list. Swap them and the script reads the wrong file as the pipeline.
  • python3 queue.py 50 3 10: The same pattern on a different script. The three numbers are jobs, runners, and minutes per job. The script prints how long the queue takes with that many runners.
  • python3 -c (added): The lesson did not show this one yet. The -c flag runs one code string instead of a file. The lesson uses it elsewhere to parse YAML in one line. I use -c for checks and a file for anything longer.

Try it: run python3 check-pipeline.py .gitlab-ci.yml runners.yml. Expect PASS with the job count and the stages. Run python3 queue.py 50 3 10, then 50 1 10, and expect the time to triple with one runner.

It passes when: the checker prints PASS. Break it on purpose: change one tag to gpu and run it again. The checker prints FAIL and names the tag. Change it back. A tag is a promise. Keep it.

Advanced — Environments, review apps, and deploys

Definition: An environment is a named place with a URL, a history, and a stop button. A review app is an environment per merge request, created when the request opens and destroyed when it closes. Manual jobs gate the deploy to prod: a human clicks, and only then does it go. Protected environments say who may deploy to prod, and nobody else.

Types:

  • Environments, staging and prod, each with a history of deploys.
  • Review apps, one environment per merge request, alive while the request is open.
  • Manual jobs, which wait for a human to click before they run.
  • Protected environments, which say who may deploy where.

Uses: I use review apps so a reviewer can click the change instead of imagining it. I use manual jobs for prod, so no deploy happens while nobody is watching. I use protected environments so the deploy to prod is a decision, not an accident.

Dimensions: At ten times the merge requests, the review apps pile up. What breaks is the environment that was never destroyed, still running, still costing money. It costs a stop job that runs when the request closes. The trade is that a live preview for every change is worth the cleanup.

Try it yourself

Add a review app job to the CI file. It deploys one environment per merge request, with a URL the reviewer can click, and an on_stop job that tears it down.

review-app:
  stage: deploy
  tags: [deploy]
  script:
    - echo "deploying review app for $CI_MERGE_REQUEST_IID"
  environment:
    name: review/$CI_MERGE_REQUEST_IID
    url: https://review-$CI_MERGE_REQUEST_IID.example.com
    on_stop: stop-review-app

stop-review-app:
  stage: deploy
  tags: [deploy]
  script:
    - echo "tearing down the review app"
  environment:
    name: review/$CI_MERGE_REQUEST_IID
    action: stop
  when: manual
python3 -c "import yaml; d=yaml.safe_load(open('.gitlab-ci.yml')); r=d['review-app']['environment']; print(r['name'], '|', r['url'], '| on_stop:', r['on_stop'])"
python3 check-pipeline.py .gitlab-ci.yml runners.yml

What you should see:

review/$CI_MERGE_REQUEST_IID | https://review-$CI_MERGE_REQUEST_IID.example.com | on_stop: stop-review-app
PASS: 6 jobs, stages ['build', 'test', 'deploy'], every tag has a runner, prod deploy is manual

It passes when: the one-liner prints the review name, URL, and on_stop, and the checker prints PASS on 6 jobs. The review app exists only while the request is open.

The tag no runner carries. A job that asks for a tag waits for a runner that carries it, and no other runner will take the job, even if it is idle. A tag that no runner carries is a job that waits forever. I check the runner list before I add a tag to a job.

Do this: add a tag to a job that no runner carries, and run the pipeline. Expected: the job waits, and the pipeline page says it is pending with no runner assigned. Remove the tag, run again, and expect the job to start. A tag is a promise. Keep it.

test-job:
  tags:
    - docker
  script:
    - make test

What the snippet means: the tags list is the job asking for a specific runner. The runner list is the answer, and a tag with no runner is a job that never runs.

The manual job that waited for nobody. A manual job waits for a human to click, and a deploy that waits for a human at 2 AM waits until morning. I use manual jobs for prod, because a deploy nobody is watching is a deploy nobody can stop. But I also write down who watches, because a gate with no watcher is a wall.

Do this: add a manual deploy job to a pipeline, push to main, and watch. Expected: the pipeline stops at the deploy job and waits. Click the play button and expect the deploy to run. Do not click it, and expect the pipeline to stay waiting. The waiting job is the gate, and the click is the decision.

deploy-prod:
  stage: deploy
  script:
    - make deploy-prod
  when: manual
  environment:
    name: production

What the snippet means: when manual makes the job wait for a click. A prod deploy that runs without a click is a deploy nobody approved, and a manual job is the approval.

Master — Running runners as production

Definition: At this level the runners are production. They run on machines you own, with monitoring, autoscaling, and a runbook for when they go down. Autoscaling adds runners when the queue grows and removes them when it shrinks. The runner fleet is sized by the queue, not by gut feel. And the queue is the metric you watch, because it is the wait your developers feel.

Types:

  • Autoscaling runners, which grow with the queue and shrink when it empties.
  • Runner monitoring, which watches the queue, the failures, and the disk.
  • Runner fleets per team, so one team's jobs cannot starve another's.
  • Cost tracking, because a runner fleet is a bill, and the bill grows with the queue.

Uses: I use this when the pipeline is the way code ships, and a slow pipeline is a slow team. It is also how I review a setup I did not build: I ask how long jobs wait, who owns the runners, and what happens when the fleet goes down.

Dimensions: At a hundred times the jobs the runners are not the bottleneck, the queue is. What breaks is the fleet that cannot scale, or the queue that grows on a Monday morning and nobody notices until lunch. It costs real operational work: autoscaling, monitoring, and a runbook. The trade is that the runners are now a service, and a service needs an owner.

Try it yourself

Run the queue math before you size the runners. Fifty jobs, three runners, ten minutes each.

#!/usr/bin/env python3
"""Queue wait from jobs, runners, and job time. No runner, no math."""
import sys
jobs, runners, job_min = int(sys.argv[1]), int(sys.argv[2]), float(sys.argv[3])
batches = -(-jobs // runners)
print(f"{jobs} jobs, {runners} runners, {job_min} min each")
print(f"batches: {batches}, wall time about {batches * job_min:.0f} min")
print(f"with {runners + 1} runners: {-(-jobs // (runners + 1)) * job_min:.0f} min")
python3 queue.py 50 3 10
python3 queue.py 50 1 10

What you should see:

50 jobs, 3 runners, 10.0 min each
batches: 17, wall time about 170 min
with 4 runners: 130 min

With 1 runner the same command prints batches: 50, wall time about 500 min.

It passes when: the numbers match. One runner is a single point of failure with a name. Three runners cut the Monday peak from 500 minutes to 170.

The runner fleet, sized by the queue. A runner fleet is sized by the queue, not by gut feel. I count the waiting jobs at peak, count the running jobs, and size the fleet so the wait stays under the number the team agreed. A fleet that is too big is a bill. A fleet that is too small is a wait.

Do this: record the queue depth at three times of day, morning, midday, and evening, and write down the numbers. Expected: a peak and a quiet time, and the peak is the sizing number. Size the fleet for the peak, and expect the morning wait to fall. The three numbers are the sizing, and the gut feel is not.

# record the queue three times a day for a week
# morning: 12 waiting, midday: 3 waiting, evening: 1 waiting
# peak is 12, so the fleet grows until the morning wait is under 5 minutes

What the snippet means: the comment is the sizing note. A fleet sized from a week of queue numbers is a decision. A fleet sized from a feeling is a guess.

The runner that filled its disk. A runner that builds images fills its disk with old images, and then every job fails the same way. I watch the disk on the runners, and I clean it on a schedule, because a full disk is a fleet outage with a boring cause.

Do this: check the disk on a runner with the command in the snippet, then run a job that builds a large image, and check again. Expected: the free space drops. Add a scheduled cleanup job that removes unused images, and expect the free space to recover. A runner fleet without cleanup is a fleet with a countdown.

df -h /var/lib/docker
docker system prune -af --filter "until=72h"

What the snippet means: the first command reads the disk, and the second removes images older than three days. The cleanup job is the fix, and the disk check is the proof it works.

Read the flags: df -h and docker system prune -af --filter

Docker eats disk silently. Df shows how full the disk is. Prune deletes what Docker no longer needs. The filter keeps the last three days, so a recent image is never the one that gets deleted.

  • df -h /var/lib/docker: Df reports free disk space. The -h flag prints sizes in human units. The path is the Docker data directory, which is where images and layers live.
  • docker system prune: This deletes unused Docker data: stopped containers, unused networks, dangling images, and build cache. It never deletes a running container.
  • -af: Two flags in one set, broken down letter by letter. The a means all unused images, not just dangling ones. The f means force, so prune does not ask for confirmation. Without a, only dangling images go. Without f, prune waits for me to type yes.
  • --filter "until=72h": This keeps anything used in the last 72 hours. The filter is the safety net. Without it, prune would delete every unused image, including the one I deployed yesterday.
  • docker system df (added): The lesson did not show this one yet. It shows how much disk each kind of Docker data uses: images, containers, volumes, and build cache. I run it before prune, so I know which kind is eating the disk.

Try it: run df -h /var/lib/docker, then docker system df, then docker system prune -af --filter "until=72h". Expect more free space after, and the newest images still present.

Go deeper

Tags are a contract between the job and the runner. A job that asks for a tag waits for a runner that carries it, and no other runner will take the job, even if it is idle. That is the strength and the trap. A tag that no runner carries is a job that waits forever. Before you add a tag to a job, check that a runner carries it, and check that the runner is online. A tag is a promise. Keep it.

Executors decide what the job can touch. A docker executor starts a fresh container, runs the job, and throws the container away. Nothing survives, which is safe and slow when the job needs the same dependencies every time. A shell executor runs on the machine itself, which is fast and dangerous: the job can touch anything the machine can touch. I use docker when the job must start clean, and shell only when the job needs the host itself.

Caching and artifacts are two different promises. A cache is a speed trick: dependencies saved between jobs, which may vanish at any time. If the cache goes missing, the pipeline must still work, just slower. An artifact is a promise: the built image, the test report, saved for the next job to rely on. I treat the cache as a trick and the artifact as a contract, and I never build a pipeline that breaks when the cache disappears.

Environments are the deploy story. Staging first, prod second, each with its own history. A manual job between them means a human decides when prod gets the change. A protected environment means only certain people may click that job. The deploy to prod is a decision, and the pipeline should make the decision visible: who clicked, when, and which version went out.

Review apps are environments per merge request. The pipeline creates one when the request opens, with a URL the reviewer can click, and destroys it when the request closes. The reviewer stops imagining the change and starts clicking it. The cost is the cleanup: an app that is never destroyed keeps running and keeps billing. The stop job is part of the design, not an afterthought.

At 10x the jobs, the shape holds and the numbers grow. The queue is longer, the runners are busier, the artifacts are bigger. At 100x, the design changes: autoscaling runners, a queue metric on a dashboard, and a fleet per team so one team's jobs cannot starve another's. The runner never stops being a small agent that asks for work. It just gets more company.

deploy-staging:
  stage: deploy
  script:
    - deploy --env staging
  environment:
    name: staging
    url: https://staging.example.com
  tags:
    - docker

What the snippet means: a deploy job in the deploy stage that runs a deploy script, records the deployment in the staging environment with its URL, and asks for a runner with the docker tag. The environment is the record: what was deployed, where, and when. Those eight lines are the whole deploy, and the tag is the promise that the job runs on a machine that can run it.

Try it yourself

Save check-cache-artifacts.py and run it. It fails on any job that leans on the cache for something the next job needs.

#!/usr/bin/env python3
"""Cache is a speed trick, artifacts are a contract. Fail if a job
depends on the cache for something the next job needs."""
import sys, yaml
with open(".gitlab-ci.yml") as f:
    ci = yaml.safe_load(f)
fails = []
for name, job in ci.items():
    if not isinstance(job, dict) or name.startswith("."):
        continue
    if "cache" in job and "artifacts" not in job and job.get("stage") in ("build", "test"):
        fails.append(f"{name}: uses cache but declares no artifacts; if the cache vanishes the next job starves")
if fails:
    print("FAIL")
    for e in fails:
        print("  - " + e)
    sys.exit(1)
print("PASS: no job leans on the cache for something another job needs")
python3 check-cache-artifacts.py

What you should see: PASS: no job leans on the cache for something another job needs

Read the flags: python3 check-cache-artifacts.py

Cache is for dependencies. Artifacts are for outputs. A job that leans on the cache for something another job needs will fail when the cache is cold, and this script catches that before the pipeline does.

  • python3 check-cache-artifacts.py: This runs the file check-cache-artifacts.py with no arguments. The script reads the pipeline file itself. No flag is needed, because a file name is the default input.

Try it: run python3 check-cache-artifacts.py. Expect PASS when no job leans on the cache for something another job needs. Make one job read a cache file another job writes, and run it again. Expect a failure that names the job.

It passes when: the checker prints PASS. Break it on purpose: give the build job a cache and take away its artifacts, and the checker fails with the job name. The cache is a speed trick. The artifact is the contract.

The environment that stops itself

A review app that never stops is a leak, and the leak is fixed by a stop job that runs when the branch closes. I write the stop job next to the deploy job, and I test it by closing a merge request and watching the environment disappear.

Do this: add the stop job from the snippet, open a merge request, and watch the review app appear. Close the merge request without merging, and watch the environment. Expected: the stop job runs and the environment disappears. If it stays, the stop job is missing or broken, and the leak is still open.

stop-review-app:
  stage: deploy
  script:
    - make teardown-review-app
  when: manual
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    action: stop

What the snippet means: the action stop marks the job as the teardown for the environment. Closing the merge request triggers it, and the review app is gone the same day it is no longer needed.

One time it broke

What broke

A pipeline I owned waited forty minutes for a runner that was offline. The job asked for a tag that only one runner carried, and that runner had run out of disk the night before. The queue showed the job waiting. Nobody looked at the queue. The team spent the morning blaming the tests, which had not run at all.

Timeline

  • 09:05,a developer pushed a fix and the pipeline started. The build job picked up at once. The deploy job waited.
  • 09:12,the deploy job was still waiting. The team assumed the build was slow.
  • 09:25,someone opened the pipeline page. The build had finished at 09:08. The deploy job was still in the queue.
  • 09:31,I checked the runner list. The tagged runner was offline. Its disk was full, and it had stopped asking for work hours earlier.
  • 09:40,I cleared the disk on the runner and it came back online. The deploy job picked up within a minute.
  • 09:52,the deploy finished. The fix was live forty-seven minutes after the push, and forty of those minutes were waiting.

What changed after

Three things. First, the queue time is on a dashboard now, with an alert when a job waits more than ten minutes. Second, every tagged runner has a disk alert, because a full disk is a quiet way for a runner to die. Third, no job asks for a tag that only one runner carries, because one runner is a single point of failure with a name.

Replay it yourself

Replay the offline runner. Remove the deploy runner from runners.yml and run the checker. Every deploy job now waits for a runner that does not exist.

sed -i '/deploy: 2/d' runners.yml
python3 check-pipeline.py .gitlab-ci.yml runners.yml

What you should see:

FAIL
  - review-app: tag 'deploy' is carried by no runner
  - stop-review-app: tag 'deploy' is carried by no runner
  - deploy-staging: tag 'deploy' is carried by no runner
  - deploy-prod: tag 'deploy' is carried by no runner

It passes when: the checker names all four jobs. Put the runner back when you are done. That is the incident: the queue showed the job waiting, and nobody looked at the queue.

If you remember nothing else

Design exercise

Design runners and environments for a team of thirty developers pushing fifty times a day. Jobs need docker, a private network, and a warm cache. Deploys go to staging automatically and to prod only when a human clicks. A runner going down must never block the team for more than ten minutes.

Clarifying questions to ask

  • How many runners do you run, and how are they sized: shared, specific, or both?
  • Which jobs ask for which tags, and what happens when a tag has only one runner?
  • How does the queue behave at the Monday morning peak, and what alerts when a job waits too long?
  • How are the environments set up, and who may deploy to prod?
  • What happens when a runner goes down: who notices, and what runs while it is down?

Rough sizing

Start with the budget from this lesson. Push 0, queue 2, runner 1, executor 1, job 9, artifact 1, environment 3, rollback 1, about 18 min end to end on shared runners. Fifty pushes a day means fifty pipelines, and at nine minutes of job time each that is about seven and a half hours of job time a day, before the queue. With five runners the queue stays short. With two, the Monday peak waits. Little's Law says the jobs in the system equal the arrival rate times the time in the system, so size the fleet from the peak, not the average.

Trade-offs to name

Shared runners buy zero maintenance and cost the queue: you wait with everyone else. Your own runners buy speed and cost ownership: disk, updates, and a runbook. Autoscaling buys a fleet that fits the load and costs a bill that moves with it. Manual gates buy safety and cost a human who must be there to click. And the target is ten minutes. A job that waits longer than that is a job the team feels.

Build it: the checkable version

Build the runners and environments from the exercise: the CI file, the runner list, and the checker. Thirty developers, fifty pushes a day, staging automatic, prod manual, and no single runner holding a tag alone.

Steps:

  • Build the CI file from the Basic and Advanced sections: six jobs, three stages.
  • Save runners.yml with two deploy runners and check-pipeline.py next to it.
  • Run the checker and the queue math.
python3 check-pipeline.py .gitlab-ci.yml runners.yml
python3 queue.py 50 3 10

What you should see: PASS: 6 jobs, stages ['build', 'test', 'deploy'], every tag has a runner, prod deploy is manual and the queue math showing 170 minutes with 3 runners.

It passes when: the checker prints PASS, the prod deploy is manual, and the deploy tag has two runners behind it. A runner going down must never block the team for more than ten minutes: that is the queue alert from the incident above, and the checker is what stops the tag from pointing at one runner.

Next in this series
Lesson 133 — GitLab Beyond Pipelines