Lesson 132 gave you runners and environments. This lesson is about everything else GitLab does, and when it matters. Merge requests, reviews, security scans, the container registry, and releases. A pipeline builds the code. The rest of GitLab decides whether the code is worth shipping.
Difficulty: Advanced · Know this first: Lesson 132 — GitLab Runners and Environments
Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: on a cache hit the trip is almost free, on a miss the kitchen eats the budget. Step through the lesson or switch the scenario and watch the bar change.
Extra diagram · from scratch to master
Start with one branch. Add one box at a time until the whole trip from branch to prod release is on the map, blocked merge included.
The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the review, the scan, and the deploy change in the budget bar, in the hop timings, and in the total. The numbers are a teaching example, not live measurements. The shape is the lesson: the review is the longest step, and it should be.
How many approvals the merge request needs. Each reviewer adds about 8 minutes of reading. One reviewer is fast and misses things. Five reviewers are thorough and slow.
How long the security scan takes. Four minutes is a clean scan of a small change. Sixty minutes is a big change with findings to triage.
How many merge requests are open at the same time. Each open request adds about half a minute to the deploy queue, because the pipeline is shared and the deploy waits its turn.
Try this: (1) Set reviewers to 3, scan time to 20, open requests to 25. The review is about 24 min, the scan is 20 min, the deploy is about 18 min, and the total is about 64 min. (2) Set all three to the worst end: 5 reviewers, 60 min scan, 50 open requests. The review is 40 min, the scan is 60 min, the deploy is 30 min, and the total is about 132 min. Same code. Only the inputs changed.
Type your own numbers: Get the total under 70 minutes. Type 3 reviewers, 20 minutes of scan, 25 open requests, and run it. Then set the scan to 60 and watch the total.
Three questions. No score is saved anywhere — this is just you against the material.
I take the same merge request four times here. First I learn what it is. Then I learn reviews, approvals, and gates. Then I learn security scanning and the registry. Last I see GitLab as the platform.
Definition: A merge request is a proposal to merge one branch into another, with a description, a diff, a pipeline, and reviewers. The request is the unit of work. It is reviewed, tested, approved, and merged, in that order. The branch is where the code lives. The request is what happens to it.
Types:
Uses: I use a merge request for every change, even a one-line fix. The request is the review, the test, and the record, and a change without one has none of the three.
Dimensions: At low traffic everything is nearly free. What breaks first is the big request: five hundred changed lines that nobody reviews well. It costs almost nothing to keep requests small. It costs a bad merge when they grow.
Save this file as CODEOWNERS. It says who maintains each path. Read it with Python and print every path with its owner.
# Owners for the checkout service
/app/ @checkout-team
/db/ @data-team
*.md @docs-teampython3 -c "
owners = {}
for line in open('CODEOWNERS'):
line = line.strip()
if not line or line.startswith('#'):
continue
path, *who = line.split()
owners[path] = ', '.join(who)
for path, who in owners.items():
print(f'{path} -> {who}')
" What you should see:
/app/ -> @checkout-team
/db/ -> @data-team
*.md -> @docs-team It passes when: every path prints with an owner. A path with no owner is a change nobody reviewed properly.
Definition: Approval rules say how many reviewers must approve, and who they must be. A code owner must approve changes to the files they own. Pipeline gates say the merge is blocked until the pipeline passes: tests, scans, and checks. Draft requests say the change is not ready for review yet, so nobody wastes time on it.
Types:
Uses: I use approval rules so a change cannot merge on one person's say-so. I use code owners so the people who maintain a file see changes to it. I use draft requests so reviewers never read a change that is half finished.
Dimensions: As the team grows, the rules grow, and the merge slows down. What breaks is the rule that nobody remembers why it exists. It costs a review of the rules themselves, once in a while. The trade is that gates buy safety and cost speed, and the balance is a decision, not a default.
Run check-mr-flow.py. It checks the owners, the scan gate, and the release notes in one pass. Save the CI file and the release notes next to it first: the CI file is in the Advanced section below and the notes are in the Master section.
#!/usr/bin/env python3
"""Check the merge-request flow: CODEOWNERS, scan gate, release notes."""
import re, sys, yaml
errors = []
# 1. every path in CODEOWNERS has an owner
with open("CODEOWNERS") as f:
for line in f:
line = line.strip()
if not line or line.startswith("#"):
continue
parts = line.split()
if len(parts) < 2 or not any(p.startswith("@") for p in parts[1:]):
errors.append(f"CODEOWNERS: '{line}' names no owner")
# 2. the scan job blocks on critical
with open(".gitlab-ci.yml") as f:
ci = yaml.safe_load(f)
scan = ci.get("security-scan", {})
if scan.get("allow_failure") is not False:
errors.append("security-scan: allow_failure is not false, the scan cannot block")
if "critical" not in " ".join(scan.get("script", [])):
errors.append("security-scan: script never mentions critical")
# 3. a release needs notes
try:
with open("RELEASE_NOTES.md") as f:
notes = f.read().strip()
if len(notes) < 20:
errors.append("RELEASE_NOTES.md is too short to tell a user anything")
except FileNotFoundError:
errors.append("RELEASE_NOTES.md is missing")
if errors:
print("FAIL")
for e in errors:
print(" - " + e)
sys.exit(1)
print("PASS: every path has an owner, the scan blocks on critical, release notes exist")python3 check-mr-flow.py What you should see: PASS: every path has an owner, the scan blocks on critical, release notes exist
Read the flags: python3 check-mr-flow.py and python3 check-waiver.py
Two small scripts, two gates. One checks the merge request flow. The other checks that a waived library is not imported by production code. Both run with a file name and no flags.
python3 check-mr-flow.py: This runs the file check-mr-flow.py. It checks that every path has an owner, that the scan blocks on critical, and that release notes exist. No arguments, because the script reads the files itself.python3 check-waiver.py vulnlib: This runs the file check-waiver.py with one argument, the library name. The script checks that production code does not import the waived library. Change the argument and the check follows a different library.python3 -c (added): The lesson did not show this one yet. The -c flag runs one code string instead of a file. The lesson uses it elsewhere to parse YAML in one line. I use -c for checks and a file for anything longer.Try it: run python3 check-mr-flow.py. Expect PASS. Run python3 check-waiver.py vulnlib, then add the import to app.py and run it again. Expect FAIL, then remove the import and expect PASS.
It passes when: the checker prints PASS. Break it on purpose: delete the owner from one line of CODEOWNERS and run it again. The checker fails with the line. Put the owner back.
Definition: The pipeline scans the code and its dependencies for known vulnerabilities: static analysis, dependency scanning, container scanning, and secret detection. The registry holds the built images, tagged by commit or release, scanned on push. A critical finding blocks the merge, or the release, until it is fixed or waived by a named person.
Types:
Uses: I use scanning on every pipeline, because a vulnerability found in review is cheap and a vulnerability found in prod is not. I use secret detection because a key in the history is a key that must be revoked, not just removed.
Dimensions: At ten times the dependencies, the findings pile up. What breaks is the triage: a hundred low findings that nobody reads, hiding the one critical one. It costs a policy: critical blocks, high gets a deadline, low gets a ticket. The trade is that a scan that blocks everything gets disabled, and a scan that blocks nothing is decoration.
Add the security scan job to the CI file. It blocks the merge on a critical finding, and allow_failure is false, so the scan is a gate and not a suggestion.
security-scan:
stage: test
tags: [docker]
script:
- scan --fail-on critical
allow_failure: falsepython3 check-mr-flow.py What you should see: PASS: every path has an owner, the scan blocks on critical, release notes exist
It passes when: the checker prints PASS. Break it on purpose: set allow_failure to true and run it again. The checker fails with security-scan: allow_failure is not false, the scan cannot block. Set it back to false.
The critical finding, and the waiver with a name. A critical finding blocks the merge until it is fixed or waived by a named person. A waiver with no name is a hole. I require the waiver to name the person, the reason, and the date it expires, because a waiver that never expires is a vulnerability that never gets fixed.
Do this: open a merge request with a critical finding, and try to merge it. Expected: the merge is blocked, and the page names the finding. Add a waiver with your name, the reason, and an expiry date, and expect the merge to proceed. The waiver is a decision, and a decision needs a name and a date.
# in the merge request: request a vulnerability exception
# name, reason, expiry date. All three, or the waiver is not approved
# example: "waived by A. Rao, false positive in test data, expires 2026-12-31"
What the snippet means: the comment is the waiver, written down where the merge request can see it. A waiver without an expiry is a finding that was forgotten, not a finding that was fixed.
The secret that the scanner caught. Secret detection stops a password or a key from being committed, and it catches the one that slips through in review. But a secret that reached the history is a secret that must be revoked, not just removed. Removing the file does not un-leak the secret.
Do this: commit a test secret to a scratch repo, push it, and watch the pipeline. Expected: the secret detection job fails and names the file. Rotate the secret, then remove it from the history. The rotation is the fix. The removal is the cleanup.
git log -S "aws_secret_access_key" --oneline --all | head -5
# rotate the key first, then clean the history
What the snippet means: the log search finds every commit that touched the key. A key that was in the history is a key that was seen, and a seen key is a revoked key.
Read the flags: git log -S --oneline --all
The same pickaxe search as the GitOps lesson, aimed at a cloud key. A key in history is a key that has to be rotated, even after the file is fixed.
git log -S "aws_secret_access_key": The -S flag finds commits where the count of the string changed. A commit that added the key shows up. A commit that only moved it does not.--oneline: This prints one line per commit. Without it, log prints the full message and the diff, and five commits fill the screen.--all: This searches every branch. A key can hide on a branch nobody merges. Without --all, that branch is never searched.| head -5: The pipe keeps the first five lines. The newest matches are usually the ones I need.--since (added): The lesson did not show this one yet. It limits the search by time, like git log --since="30 days ago". I use it when the key is recent and the full history is years long.Try it: run git log -S "aws_secret_access_key" --oneline --all | head -5. Expect the commits that added or removed the key. Rotate the key first, then clean the history.
Definition: At this level GitLab is the platform, not just the repo. The pipeline builds, the registry stores, the review decides, the release promises, and the deploy hands off to GitOps. Value stream analytics show where the time goes: from idea to prod, step by step. And the honest part: GitLab is the right home when one tool for the whole flow beats five tools wired together, and the wrong home when the team already has a flow that works.
Types:
Uses: I use this when the team wants one place for the whole flow, from the first commit to the prod deploy. It is also how I review a setup I did not build: I ask where the time goes, what blocks merges, and whether the team trusts the gates or works around them.
Dimensions: At a hundred times the merge requests the platform is not the bottleneck, the process is. What breaks is the gate everyone works around, or the review queue nobody clears. It costs real operational work: queue reviews, gate reviews, and an honest look at the time from idea to prod. The trade is that one tool for everything is simple until it is not, and the moment it is not is the moment to say so.
Save RELEASE_NOTES.md and add the release job to the CI file. The release tags the version and points at the notes.
## v1.4.0
- Checkout retries a failed payment once, then shows the error.
- Fixed the balance page showing a cached number after a refund.release:
stage: deploy
tags: [docker]
script:
- echo "tagging v1.4.0"
release:
tag_name: v1.4.0
description: RELEASE_NOTES.mdpython3 -c "import yaml; d=yaml.safe_load(open('.gitlab-ci.yml')); print(d['release']['release'])"
python3 check-mr-flow.py What you should see:
{'tag_name': 'v1.4.0', 'description': 'RELEASE_NOTES.md'}
PASS: every path has an owner, the scan blocks on critical, release notes exist It passes when: the release job prints the tag and the notes file, and the checker prints PASS. A release with no notes is a release the users cannot judge.
Value stream analytics: where the time goes. Value stream analytics show where the time goes, from idea to prod, step by step. I read it once a month, and I fix the longest step first, because the longest step is the bottleneck and the bottleneck is the work.
Do this: open the value stream analytics page and write down the time for each stage. Expected: one stage is clearly the longest, usually review or testing. Fix that stage, wait a month, and read the numbers again. If the longest stage did not move, the fix did not work, and the numbers will say so.
# Operate > Value stream analytics (in the GitLab UI)
# record: issue to plan, plan to code, code to review, review to merge, merge to prod
# the longest stage is the work
What the snippet means: the comment is the checklist of stages to record. The numbers are the pipeline, and the longest number is the first thing I fix.
One tool for the flow, or five tools wired together. GitLab is the right home when one tool for the whole flow beats five tools wired together, and the wrong home when the team already has a flow that works. I make the decision on purpose, with the numbers, not by habit.
Do this: list the five tools your flow uses today, and write down what each one costs in money and in maintenance hours. Expected: a list with numbers, and one tool is clearly the most expensive to maintain. That tool is the case for moving, or the case for staying. A decision with numbers is a decision. A decision with a preference is a habit.
# flow inventory: repo, CI, registry, review, deploy
# cost each one: money per month + maintenance hours per month
# the most expensive to maintain is the case, either way
What the snippet means: the comment is the inventory to fill in. The numbers decide whether one tool or five tools is cheaper, and the answer is different for every team.
Approval rules are the merge policy written down. Two approvals, one from a code owner, no approval from the author. The rules say who must approve, and the merge is blocked until they do. A rule that nobody remembers why it exists is a rule that gets worked around. Review the rules once in a while, the way you review the code.
Code owners are the people who maintain a file. A change to their file needs their approval, and the request asks them automatically. The point is not control. It is that the person who will fix the bug gets a say before the bug ships. A file with no owner is a file nobody maintains, and a change to it is a change nobody reviewed properly.
Security scanning finds known problems, not new ones. A clean scan means no known vulnerability in the code or its dependencies. It does not mean the code is safe. The scan is a floor, not a ceiling. The review is the ceiling: humans reading the logic, asking whether the change does what it claims. The two together are the gate. Either one alone is half a gate.
Secret detection stops a password or a key from being committed. The pipeline scans every push, and a match blocks it. A secret that reaches the history is a secret that must be revoked, not just removed, because the history keeps it forever. The fix is to revoke the key, remove it from the history, and put it in a secret store. The prevention is a pre-commit hook that checks before the push.
Releases are a promise to the people who use the software. The notes say what changed, what is new, what to watch. The assets ride along: the image, the binaries, the changelog. A release with no notes is a release the users cannot judge, and they will judge it anyway, by whether it breaks them. Write the notes as if the person reading them is about to upgrade on a Friday. Because they might be.
At 10x the merge requests, the shape holds and the numbers grow. The review queue is longer, the scan findings pile up, the registry holds more images. At 100x, the design changes: the queue gets owners, the gates get reviewed, and the time from idea to prod gets measured. The merge request never stops being the unit of work. It just gets more company.
security-scan:
stage: test
script:
- scan --fail-on critical
allow_failure: false
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
What the snippet means: a scan job in the test stage that fails the pipeline on any critical finding, may not fail softly, and runs only on merge request pipelines. The gate is the point: a critical finding blocks the merge, and the block is not optional. Those six lines are the whole gate, and the allow_failure line is the one that makes it real.
Save this file as pre-commit and run it. It blocks any commit that carries a secret. Test it with a bad file first, then a clean tree.
#!/bin/sh
# Block a commit that carries a secret. Run: sh pre-commit
if grep -rInE '(AKIA[0-9A-Z]{16}|password[[:space:]]*=[[:space:]]*["'\''][^"\'']+["'\''])' --include='*.py' --include='*.yml' --include='*.yaml' . 2>/dev/null | grep -v pre-commit; then
echo "BLOCKED: a secret is in the change. Remove it and use the secret store."
exit 1
fi
echo "OK: no secret in the change."echo 'db_password = "hunter2"' > bad.py
sh pre-commit
rm bad.py
sh pre-commit What you should see: the first run prints the bad line and BLOCKED: a secret is in the change. Remove it and use the secret store. and exits 1. The second prints OK: no secret in the change. and exits 0.
Read the flags: echo and quoting
Echo prints words. The quotes decide what the shell does with them. Single quotes print everything literally. Double quotes let variables expand. In a hook, the difference decides whether a secret prints or a variable does.
echo "BLOCKED: a secret is in the change. Remove it and use the secret store.": Double quotes print the words as written. The message is the block. A hook that prints BLOCKED stops the change, and the words tell the developer what to do next.echo "OK: no secret in the change.": The same command with a pass message. The hook prints OK and the change goes through.echo 'db_password=<redacted>': Single quotes print the text exactly, with no expansion. The value is redacted, not the real secret. Single quotes are how I print a secret-shaped value without ever printing the secret.printf (added): The lesson did not show this one yet. It prints with a format string, like printf '%s\n' "$value". I use it when echo would mangle backslashes or the value starts with a dash.Try it: run the hook script on a change with a secret and on a clean change. Expect BLOCKED on the first and OK on the second. Check the output for the real secret value. Expect it nowhere.
It passes when: the bad file is blocked and the clean tree passes. A secret that reaches the history must be revoked, not just removed, because the history keeps it forever.
The registry holds the built images, tagged by commit or release, and scanned on push. An image that was never scanned is an image nobody checked, and a deploy that uses it is a deploy of an unchecked thing. I block the deploy when the scan is missing, not just when it fails.
Do this: push an image without a scan stage in the pipeline, and try to deploy it. Expected: the deploy runs, because nothing checked. Add the container scanning job from the snippet, push again, and expect the scan to run and the deploy to wait for it. A scan that runs after the deploy is a report, not a gate.
container-scanning:
stage: test
script:
- echo "scan the image before the deploy stage runs"
What the snippet means: the scanning job sits in the test stage, before the deploy. A finding blocks the deploy, and an unscanned image never reaches prod.
A merge request I owned shipped a dependency with a known critical vulnerability. The scan had found it. The finding was waived by the author, with a comment that said the dependency was only used in tests. It was not only used in tests. The vulnerable code ran in prod, and the vulnerability was disclosed a week later.
Three things. First, a waiver needs a second person, not just the author, because the author is the person most likely to be wrong about their own code. Second, a waiver must name the reason and the expiry, so it cannot live forever. Third, the scan now checks whether a waived dependency is imported by production code, and a waiver that claims test-only is checked, not trusted.
Replay the waiver. A dependency is waived as test-only, but the production code imports it. The checker must catch that, not trust the waiver.
flask
requests
vulnlib==1.2.3import flask
import requests#!/usr/bin/env python3
"""A waiver that says 'test-only' is checked, not trusted."""
import sys
waived = sys.argv[1] # dependency name
with open("requirements.txt") as f:
prod_deps = [l.strip() for l in f if l.strip()]
with open("app.py") as f:
code = f.read()
in_prod = any(d == waived or d.startswith(waived + "==") for d in prod_deps)
if in_prod and f"import {waived}" in code:
print(f"FAIL: {waived} is waived as test-only but app.py imports it")
sys.exit(1)
print(f"PASS: {waived} is not imported by production code")python3 check-waiver.py vulnlib
sed -i '/import vulnlib/d' app.py
python3 check-waiver.py vulnlib What you should see: the first run prints FAIL: vulnlib is waived as test-only but app.py imports it and exits 1. The second prints PASS: vulnlib is not imported by production code and exits 0.
Read the flags: sed -i with a delete
Sed deletes the line that imports the waived library. One command, and the waiver check passes again. The delete is the fix, and the check proves it.
sed -i: The -i flag edits the file in place. Without it, sed prints the changed text and the file stays as it was.'/import vulnlib/d': This is the command. The slashes hold the pattern to match. The d means delete the matching line. Only lines with the import are deleted. The rest of the file is untouched.app.py: This is the file being edited. Sed reads it, deletes the line, and writes it back.-i.bak (added): The lesson did not show this one yet. It keeps a backup copy with the .bak ending. I use it when the file matters, so a bad pattern can be undone by copying the backup back.Try it: run python3 check-waiver.py vulnlib, then sed -i '/import vulnlib/d' app.py, then run the check again. Expect FAIL first, then PASS after the delete.
It passes when: the first run fails and the second passes. A waiver needs a second person, a reason, and an expiry. And a waiver that claims test-only is checked, not trusted.
Design the merge request flow for a team of twenty developers shipping ten times a day. Every change needs review, a security scan, and a release note. A critical finding must block the merge. A release must be traceable from the tag back to the merge request that made it. The team must not wait more than an hour from request to merge.
Start with the budget from this lesson. Branch 0, request 1, review 8 per reviewer, scan 4 clean, registry 2, release 1, deploy 5 plus half a minute per open request, audit 1. For 2 reviewers, a clean scan, and 10 open requests that is about 16 for the review, 4 for the scan, 10 for the deploy, and 5 for the fixed hops, about 35 min in this teaching example. Ten merges a day means ten reviews, and at 16 minutes of review each that is about three hours of review time a day, split across the team. The review is the bottleneck, not the pipeline.
More reviewers buy better reviews and cost slower merges. A scan that blocks everything buys safety and cost a team that learns to waive everything. Tagging by commit buys traceability and cost a registry full of images. Release notes buy trust and cost the author's time. And the target is an hour. A merge request that waits longer than that is a change that goes stale.
Build the merge request flow from the exercise: CODEOWNERS, the request template, the CI file with the scan and the release, the secret hook, and the waiver check. Twenty developers, ten merges a day, and no merge in more than an hour.
Steps:
.gitlab/merge_request_templates/default.md.## What changed
## Why
## How I tested it
## Release notepython3 check-mr-flow.py
sh pre-commit
python3 check-waiver.py vulnlib What you should see: the flow checker prints PASS, the hook prints OK: no secret in the change., and the waiver check prints PASS.
It passes when: all three checks print their pass line. That means every path has an owner, the scan blocks on critical, the release has notes, no secret is in the change, and no waived dependency runs in production. The merge request is a gate with all three parts working.