GitLab CI/CD From Zero to Pipeline Architect: Part 3
In Post 1 and Post 2, you built a solid pipeline:
Reusable GitLab CI Templates & Shared CI Libraries (From Single Repo to Org-Wide Pipelines)
- Multiple stages (.pre, build, test, deploy, .post)
- Artifacts to pass data across stages
- Smart variables and branch-aware behavior
That’s great for one project.
Now imagine you have 20 projects:
- Frontend apps (React, Angular, Vue)
- Backend services (Node, Go, Java)
- Shared libraries and tools
If every repo has its own custom .gitlab-ci.yml, you’ll get:
- Copy-paste pipelines everywhere
- Slightly different behavior in each project
- A nightmare when you want to change a policy (“enable SAST everywhere”, “change Docker conventions”, etc.)
This is where reusable templates and shared CI libraries come in.
By the end of this post, you’ll know how to:
- Build hidden job templates and reuse them with extends
- Use YAML anchors to avoid repeating blocks
- Split pipelines using include (local, project, remote)
- Create a shared CI library repo used by many projects
- Version that library with tags and safely override behavior per app
1. The goal: one CI “brain”, many projects
Imagine this setup:
- One repo: ci-templates – your shared CI library
- Many app repos: app-frontend, app-backend, app-worker, …
Each app’s .gitlab-ci.yml is tiny:
include:
- project: "my-org/ci-templates"
ref: v1.0.0
file: "node/pipeline.yml"Almost all logic is central:
- How you build Node apps
- How you run tests
- How you build Docker images
- How you run lint, SAST, etc.
Each app only:
- sets some variables,
- maybe overrides a job or adds extra jobs.
That’s our target.
2. Hidden jobs: templates you don’t run directly
A hidden job is just a job whose name starts with a dot (.):
.node-build:
stage: build
image: node:20-alpine
cache:
paths: [ node_modules/ ]
script:
- npm ci
- npm run build
artifacts:
paths: [ dist/ ]
expire_in: 1 weekGitLab won’t run .node-build by itself.
Instead, you create real jobs that extend it.
2.1 Using extends to reuse hidden jobs
build_frontend:
extends: .node-build
build_admin_panel:
extends: .node-build
script:
- npm ci
- npm run build:adminWhat’s happening:
- build_frontend inherits everything from .node-build.
- build_admin_panel inherits everything but overrides script.
You can think of .node-build as a class, and build_frontend as a specific object based on it.
Real jobs extend and customize them.
3. YAML anchors: reuse blocks inside a file
Inside a single .yml file, you can reuse blocks (like rules: or variables:) using anchors (&) and aliases (*).
Example: shared rules for your main branch:
.main_branch_rules: &main_branch_rules
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
when: on_success
- when: neverNow you can attach this block to multiple jobs:
build_frontend:
extends: .node-build
<<: *main_branch_rules
test_frontend:
extends: .node-test
<<: *main_branch_rules<<: *main_branch_rules means: “merge in everything from this anchor”.
Key points:
- Anchors work within the same file.
- They’re great for shared rules, variables, or before_script blocks.
- They are a YAML feature, not GitLab-specific.
4. Splitting pipelines with include
As your CI grows, a single .gitlab-ci.yml becomes hard to read.
GitLab lets you split it into multiple files and glue them with include.
There are three main types you’ll likely use:
- include:local – files from the same repo
- include:project – files from a different repo
- include:remote – files from an external URL (less common, be careful)
Let’s see each quickly, then we’ll build the shared library.
4.1 include:local – organize within one repo
Imagine you have some structure:
.gitlab-ci.yml
ci/
defaults.yml
node/
pipeline.yml
jobs/
build.yml
test.yml
docker.ymlIn .gitlab-ci.yml you can write:
include:
- local: "ci/defaults.yml"
- local: "ci/node/pipeline.yml"Each included file can define:
- stages
- default (image, tags, retry)
- hidden jobs
- visible jobs
GitLab merges all of these configs into one big pipeline.
4.2 include:project – the shared CI library
This is where things get powerful.
You put all your shared CI code in a dedicated project, e.g.:
- my-org/ci-templates
Inside that repo, you might have:
ci-templates/
node/
pipeline.yml
jobs/
build.yml
test.yml
go/
pipeline.yml
defaults.ymlThen, in an app repo:
include:
- project: "my-org/ci-templates"
ref: v1.0.0
file:
- "defaults.yml"
- "node/pipeline.yml"- project: → full GitLab path of your CI library repo.
- ref: → branch, tag, or commit SHA. For stability, use tags like v1.0.0.
- file: → one or more YAML files from that project.
Once included, all jobs/templates defined in ci-templates are available in the app pipeline.
Shows one “ci-templates” box on the left, arrows to multiple “app” repos on the right, each with tiny .gitlab-ci.yml that includes the library.
4.3 include:remote – from a URL
Less commonly, you can include raw YAML from a URL:
include:
- remote: "https://example.com/path/to/template.yml"Caution:
- This must be publicly reachable.
- You have less control over changes (the file could change anytime).
For internal org use, include:project is almost always better.
5. How GitLab merges includes and overrides
Understanding the merge rules is key to doing advanced stuff.
Simplified rules:
- GitLab resolves all includes and reads those YAML files.
- Those configs are merged in order:
- first included file, then the next, etc.
- Finally, the root .gitlab-ci.yml of the project is merged last.
Conflicts:
- If two includes define the same job name, the later one wins.
- If the root .gitlab-ci.yml also defines that job name, it wins over all includes.
- For maps (like variables: blocks), GitLab does a deep merge:
1. new keys are added,
2. existing keys are overwritten by the later file.
What this means in practice:
- Your CI library repo provides defaults.
- An app repo can override any job or variable from the library by:
1. reusing the same job name,
2. redefining global variables:,
3. redefining default:.
Example:
# In ci-templates/node/pipeline.yml
variables:
NODE_VERSION: "20"
build_app:
extends: .node-buildIn the app repo:
include:
- project: "my-org/ci-templates"
ref: v1.0.0
file: "node/pipeline.yml"
variables:
NODE_VERSION: "22" # override default from templates
build_app:
extends: .node-build
script:
- echo "Custom step before standard build"
- npm ci
- npm run build- NODE_VERSION becomes 22 for this project.
- build_app in the app repo replaces the one from the template, but still extends: .node-build from the library.
6. Designing a mini “CI standard library”
Let’s build a simple example: a Node CI library.
6.1 Structure of the ci-templates repo
ci-templates/
.gitlab-ci.yml # optional; can be empty or internal checks
defaults.yml
node/
pipeline.yml
jobs/
build.yml
test.yml
docker.yml6.2 defaults.yml – stages and global defaults
# ci-templates/defaults.yml
stages:
- validate
- build
- test
- package
- deploy
default:
image: node:20-alpine
cache:
paths:
- node_modules/
retry:
max: 1
when:
- runner_system_failure
- unknown_failureThis sets a common stage layout and default behavior for all projects that include it.
6.3 Node build template: jobs/build.yml
# ci-templates/node/jobs/build.yml
.node-build:
stage: build
script:
- npm ci
- npm run build
artifacts:
paths:
- dist/
expire_in: 1 week6.4 Node test template: jobs/test.yml
# ci-templates/node/jobs/test.yml
.node-test:
stage: test
script:
- npm ci
- npm test -- --watch=false6.5 Docker build template: jobs/docker.yml
# ci-templates/node/jobs/docker.yml
.node-docker-build:
stage: package
variables:
IMAGE_TAG: "$CI_COMMIT_SHORT_SHA"
script:
- docker build -t "$CI_REGISTRY_IMAGE:$IMAGE_TAG" .
- docker push "$CI_REGISTRY_IMAGE:$IMAGE_TAG"6.6 node/pipeline.yml – wiring it all together
# ci-templates/node/pipeline.yml
include:
- local: "defaults.yml"
- local: "node/jobs/build.yml"
- local: "node/jobs/test.yml"
- local: "node/jobs/docker.yml"
# Optional shared rules for main branch
.main_branch_rules: &main_branch_rules
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
when: on_success
- when: never
# Real jobs that apps can override if needed
build_app:
extends: .node-build
test_app:
extends: .node-test
docker_package:
extends: .node-docker-build
<<: *main_branch_rulesNow every Node app in your company can include node/pipeline.yml and get:
- A standard build job
- A standard test job
- A standard Docker packaging job
…with all the defaults and retries set.
Shows a “.node-build” template on the left and a “build_frontend — extends .node-build” on the right, with an arrow.
7. Using the CI library from an app repo
In the app repo (app-frontend), your .gitlab-ci.yml becomes tiny:
include:
- project: "my-org/ci-templates"
ref: v1.0.0
file: "node/pipeline.yml"
# Overrides or project-specific jobs
variables:
APP_NAME: "frontend-app"
build_app:
extends: .node-build
script:
- echo "Building $APP_NAME"
- npm ci
- npm run buildYou:
- Reuse shared templates from ci-templates.
- Override only what you need.
If later you decide to add a security job or change how Node is built:
- You update ci-templates.
- All apps that include v1.0.0 can upgrade by changing ref: v1.0.0 → v1.1.0.
8. Versioning your CI library with tags
Treat your CI library like real code:
- Use branches for development (main, feature/*).
- Use tags for releases: v1.0.0, v1.1.0, etc.
In app repos:
include:
- project: "my-org/ci-templates"
ref: v1.1.0
file: "node/pipeline.yml"Strategy:
- Start with v1.0.0.
- When you add features or fix bugs in the templates, tag v1.1.0.
- Each app chooses when to bump from v1.0.0 to v1.1.0.
This avoids surprises from using ref: main, where templates can change under you without warning.
9. Safe overrides in app repos
Two key override techniques:
9.1 Override a job (same name, new content)
If the CI library defines:
build_app:
extends: .node-buildYou can override it in an app repo:
build_app:
extends: .node-build
script:
- echo "Custom build for this app"
- npm ci
- npm run custom-buildBecause the root .gitlab-ci.yml is merged last, your build_app wins.
9.2 Add extra jobs that still extend shared templates
You can also add new jobs:
test_smoke:
extends: .node-test
stage: test
script:
- npm ci
- npm run test:smokeYou haven’t changed the shared template. You just reuse it for your own extra checks.
10. Advanced patterns (for when you’re the CI architect)
Here are some extra tricks to make your CI library feel pro.
10.1 Generic “build-base” template with variables
Instead of a Node-specific template, you can build a generic one:
.build-base:
stage: build
image: $BUILD_IMAGE
script:
- eval "$BUILD_COMMAND"
artifacts:
paths:
- $BUILD_OUTPUT_DIR
expire_in: 1 weekIn a Node app:
variables:
BUILD_IMAGE: "node:20-alpine"
BUILD_COMMAND: "npm ci && npm run build"
BUILD_OUTPUT_DIR: "dist"
build_app:
extends: .build-baseIn a Go app (same template, different variables):
variables:
BUILD_IMAGE: "golang:1.22-alpine"
BUILD_COMMAND: "go test ./... && go build -o app ./cmd/app"
BUILD_OUTPUT_DIR: "app"
build_app:
extends: .build-baseOne template, many languages.
10.2 “Policy” templates
You can create templates that encode org-wide policies, like:
- Every repo must run a linter.
- Every repo must run SAST.
- Every repo must upload test reports.
Example:
.ci-policy-lint:
stage: validate
script:
- ./scripts/run-lint.sh
.ci-policy-sast:
stage: security
script:
- ./scripts/run-sast.shThen in language-specific pipelines, you just:
include:
- local: "policies.yml"
lint:
extends: .ci-policy-lint
sast:
extends: .ci-policy-sastIf your security team updates how SAST works, they change it once in the CI library.
10.3 Avoiding breaking changes
Some best practices:
- Don’t remove or rename shared jobs in a minor tag.
- If you must, bump a major version (v1.x → v2.0.0).
- Keep a changelog in the CI library repo so app teams know what changed.
- Add new behavior as optional at first (allow_failure: true), then make it strict later.
10.4 Testing your CI library
Your ci-templates repo can have its own .gitlab-ci.yml that:
- Lints the YAML (e.g., with yamllint).
- Runs “example pipelines” using test projects.
- Maybe even uses the GitLab CI Lint API to validate templates.
This way, you don’t ship broken templates to all app repos.
11. Recap
In this post, you leveled up from “my project has a good pipeline” to:
You learned:
- Hidden jobs (.job) and extends for reusable templates.
- YAML anchors to share rules and other blocks.
- include with local, project, and remote to split and share configs.
- How GitLab merges configs and how overrides work.
- How to build a mini “CI standard library” repo with:
- shared defaults,
- language-specific pipelines,
- shared policy jobs.
- How to version your CI library with tags and safely override behavior in app repos.
- Advanced patterns: generic templates, policy templates, testing the library itself.
With this, you’re not just using GitLab CI — you’re architecting it.
Conclusion
By centralizing your CI logic, you’ve evolved from managing individual files to architecting a scalable system that empowers every team in your organization. You now possess the toolkit — hidden jobs, extends, and include—to enforce standards while maintaining the flexibility each project requires. This shift not only eliminates code duplication but allows you to roll out improvements and security policies across dozens of repos instantly. With the structural foundation set, you are ready to add sophisticated decision-making to your automation. In the next post, we will explore advanced rules and workflows to give your pipelines a brain of their own.
In the next post, we can go deep on:
Where you’ll master rules:, workflow:rules, changes:, manual gates, and build pipelines that behave differently based on branches, file paths, and environments.
Comments
Questions, corrections, war stories — all welcome. Sign in with GitHub to join the discussion.