Platform Engineering · Lesson 137

GitHub Actions at Scale: Cross-Workflow Calls, Environments, and OIDC

Lesson 136 gave you one reusable workflow and one caller. This lesson is the org-wide version. One workflow triggers another with workflow_run. One repo tells another repo to run with repository_dispatch. Environments add approvals. OIDC replaces stored cloud keys with short-lived tokens. By the end you have built a chain of workflows that deploys across repos, with no stored keys anywhere.

Difficulty: Master · Know this first: Lesson 136 — GitHub Reusable Workflows and Composite Actions

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: on a cache hit the trip is almost free, on a miss the kitchen eats the budget. Step through the lesson or switch the scenario and watch the bar change.

Extra diagram · from scratch to master

The chain, mapped: trigger, approval, token, deploy

Start with one push. Add one box at a time until ten repos share one chain, with a human gate and a token that lives for minutes.

7 boxes on the trip10 repos1 refused token
Push
0 min
workflow_run
3 min
Dispatch
2 min
Environment
4 min
OIDC token
2 min
Runners
6 min
Refused
trip ends
Request / flowResponse · the way backFailure / dead end

Try it: change the inputs and watch the trip bend

The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the fan-out, the runner wait, and the deploy change in the budget bar, in the hop timings, and in the total. The numbers are a teaching example, not live measurements. The shape is the lesson: the chain is cheap, the queue is the wait, and the token is the gate.

10

How many repos react to the same event. Every repo adds its own chain and its own deploy. The chain is written once. The runs are paid once per repo.

100

How many workflow runs the org starts each day. More runs means a longer queue when the runner pool is small.

4

How many self-hosted runners share the load. More runners means a shorter queue. One runner for five hundred runs a day is a queue that never clears.

Try this: (1) Set repos to 25, runs to 250, runners to 8. The fan-out is about 25 min, the runner wait about 3 min, the deploy about 13 min, and the total is about 51 min. (2) Set all three to the worst end: 50 repos, 500 runs, 1 runner. The fan-out is 50 min, the runner wait is 50 min, the deploy is 25 min, and the total is about 135 min. Same chain. Only the inputs changed.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Try it: edit the pipeline and check it

Here is a deploy workflow that chains from a build workflow, waits for an environment approval, and mints an OIDC token. Change it, then run the check. The check looks for the three things this lesson teaches: a workflow_run trigger, an environment gate, and the id-token permission.

Try breaking it on purpose: delete the environment line, or change id-token to read. Run the check and watch it fail. Then press Reset and try a change that keeps it passing, like adding a conclusion check on the build workflow.

Press "Run the check" to see whether this pipeline does what the lesson asks.

Expected output when it passes: The workflow chains from the build workflow, waits at the production environment, and mints an OIDC token with no stored key. This workflow would deploy only after the build passes and the reviewer approves.

Basic to master

I take the same chain four times here. First I learn to chain two workflows. Then I learn gates and tokens. Then I learn runners and cost. Last I run it for the world.

Basic — Chain two workflows

Definition: A workflow with on workflow_run starts when the named workflow finishes. A workflow with on repository_dispatch starts when another repo sends it an event. The chain is two workflows and one trigger, and the second one checks whether the first passed before it deploys.

Types:

  • A workflow_run trigger, which chains workflows in the same repo.
  • A repository_dispatch trigger, which lets one repo start a workflow in another repo.
  • A conclusion check, which stops the chain when the first workflow failed.
  • A payload, which carries values from the sender to the listener.

Uses: I use workflow_run when the build and the deploy live in the same repo. I use repository_dispatch when the app repo must tell the deploy repo to run. I use a conclusion check so a failed build never deploys. Try it now: write a workflow_run trigger for a build workflow in the Try it box, and run the check.

Dimensions: At low traffic everything is nearly free. What breaks first is the name mismatch: the trigger names the workflow by its name field, not its file name, and a mismatch means the chain never fires. It costs one look at the name field. It costs a silent afternoon when you skip it.

Intermediate — Environments, approvals, and OIDC

Definition: An environment is a named target with its own rules: required reviewers, a wait timer, and a branch restriction. A job that names an environment waits for approval. OIDC mints a short-lived token for the cloud, with no stored key. The permissions block says what the token may do: id-token write to mint it, contents read to check out the code.

Types:

  • An environment, which names the target and its rules.
  • Required reviewers, who must approve before the deploy runs.
  • An OIDC token, minted per run, alive for minutes.
  • A permissions block, which limits what the workflow may do.

Uses: I use environments so prod deploys wait for a human. I use OIDC so no cloud key is stored in the repo. I use the permissions block so a workflow that only builds cannot mint a deploy token. Try it now: add id-token write to the permissions in the Try it box, and run the check.

Dimensions: As the team grows, the reviewers become the bottleneck. What breaks is the approval queue: ten repos wait for the same two reviewers, and the deploy waits behind them. It costs more reviewers, or a smaller blast radius per approval.

Advanced — Runners, minutes, and policies

Definition: Self-hosted runners are machines you own, with labels the jobs match on. Minutes are billed per run on hosted runners, and the trigger is the budget. Policies limit which actions a repo may use, and starter workflows ship a caller file to every new repo. A reusable workflow in a central repo is the shared deploy, and the org watches which version each repo pins.

Types:

  • Self-hosted runners, with labels that jobs match on.
  • Minute budgets, with alerts per repo and per team.
  • Action policies, which allow or block actions by name.
  • Starter workflows, which ship a caller to every new repo.

Uses: I use self-hosted runners for private networks and special hardware. I use minute budgets so one noisy workflow cannot eat the org's budget. I use policies so a repo cannot call an action nobody reviewed. Try it now: open your org usage page and find the most expensive workflow. That is the first place to look.

Dimensions: At ten times the runs, the queue is the lesson. What breaks is the runner pool: one runner for five hundred runs a day is a queue that never clears. It costs more runners, or fewer triggers. The trade is that runners are cheap and waiting is not.

The runner that ran out of disk. A self-hosted runner that runs a hundred jobs fills its disk with old workspaces and old images, and then every job fails the same way. I watch the disk on the runner fleet the way I watch any other resource, and I clean it on a schedule, not when it breaks.

Do this: check the disk on a runner with the command in the snippet, then run a job that builds a large image, and check again. Expected: the free space drops by the size of the image. Add a scheduled cleanup job that removes old workspaces and unused images, and expect the free space to recover.

df -h /home/runner
# after the cleanup job runs
df -h /home/runner

What the snippet means: the two readings are the before and the after. A runner fleet without a cleanup job is a fleet with a countdown to full disks.

Read the flags: df -h

A runner that runs out of disk fails builds in strange ways. Df shows the disk before the failure, and -h makes the numbers readable.

  • df -h: Df reports free disk space. The -h flag prints sizes in human units like G and M instead of blocks. Without it, the numbers are block counts and I have to do the maths in my head.
  • /home/runner: This is the directory to check. It is the runner home, where caches and workspaces live. A disk can be full in one place and fine everywhere else, so I check the place that matters.
  • df -i (added): The lesson did not show this one yet. The -i flag shows free inodes instead of free blocks. A disk can run out of inodes while it still has space, and then no new file can be created at all.
  • du -sh (added): The lesson did not show this one yet. Du shows how much space a directory uses. The -s flag totals it into one line, and -h makes it readable. I use it to find which folder ate the disk.

Try it: run df -h /home/runner before the cleanup job and after it. Expect more free space after. Run df -i too, and expect the inode count to rise as well.

The matrix that tested everything, slowly. A test matrix with five versions and three operating systems is fifteen jobs, and fifteen jobs on every push is a queue that never drains. I run the full matrix on the main branch and on a schedule, and I run one version on pull requests, because a pull request needs a fast answer, not a complete one.

Do this: change a workflow so pull requests run one matrix entry and the main branch runs the full matrix, then open a pull request. Expected: the pull request runs one job and finishes fast, and the merge to main runs all fifteen. The full matrix still runs. It just runs where the wait is cheap.

strategy:
  matrix:
    node: ${{ github.ref == 'refs/heads/main' && fromJSON('[18, 20, 22]') || fromJSON('[20]') }}

What the snippet means: the matrix reads the branch and picks its size. Pull requests get the fast check. Main gets the full check. The tradeoff is written in the workflow, not in a chat message.

Master — Actions as a platform

Definition: At this level Actions runs across an org with owners, policies, budgets, and a version dashboard. Every shared workflow has a version and a sunset date. Every deploy has a chain, an approval, and a token, and the audit log shows all three. And I rehearse the worst day: the runner pool is down, the shared repo is down, and a deploy is stuck halfway.

Types:

  • A version dashboard, which shows every caller and its pinned version.
  • A minute budget per team, with alerts before it runs out.
  • An audit log, which shows the chain, the approval, and the token for every deploy.
  • Rehearsed failures, where the team practises the runner pool being down.

Uses: I use this when more than one team ships through the same workflows, or when a bad workflow can take down a service other teams depend on. It is also how I review a setup I did not build: I ask where the shared workflows live, who can change them, and what happens when the runners are gone. Try it now: pick one workflow you own and find its cost per run. If you cannot find it, that is the first gap to close.

Dimensions: At a hundred times the runs, the chain is not the bottleneck, the blast radius is. What breaks is the single shared workflow that everything depends on, or the approval queue that one stuck deploy blocks for an hour. It costs real operational work: owners, budgets, dashboards, and drills.

The audit log is the deploy record. When production changes, someone will ask who changed it, when, and with what approval. The audit log is the answer, and it is only an answer if it shows the chain, the approval, and the token for every deploy. I check the audit log after a deploy the way I check the deploy itself.

Do this: run a deploy, then open the org audit log and find the entry for it. Expected: the entry names the workflow, the person who approved it, and the environment it touched. If the approval is missing, the environment protection rule is wrong, and that is the fix.

gh api /orgs/my-org/audit-log --jq ".[] | select(.action | startswith(\"workflows\")) | [.action, .actor, .created_at]"

What the snippet means: the audit log is the record of who did what. A deploy without an approval in the log is a deploy the environment rules did not protect.

Read the flags: gh api --jq on the audit log

The audit log is the record of who did what. The --jq filter picks the fields I care about, so one command answers one question instead of dumping the whole log.

  • gh api /orgs/my-org/audit-log: This asks the API for the org audit log. Each entry has an action, an actor, and a time. Without a filter, the answer is pages of JSON.
  • --jq ".[] | select(.action | startswith(\"workflows\")) | [.action, .actor, .created_at]": The --jq flag runs a filter on the answer. The .[] part walks every entry. Select keeps only actions starting with workflows. The last part prints three fields per entry: the action, the actor, and the time. Without the filter, I would read the whole log by eye.
  • gh api /orgs/my-org/actions/permissions --jq ".allowed_actions": The same pattern on a different endpoint. It asks what actions the org allows, and the filter prints only the allowed_actions field.
  • gh api /orgs/my-org/actions/runners --jq ".runners | group_by(.status) | map({status: .[0].status, count: length})": The same pattern again. Group_by collects runners by status, and the map counts each group. The answer is one line per status with a count, which is the shape a human can read.
  • --paginate (added): The lesson did not show this one yet. It fetches every page of a list endpoint. The audit log is long, and the first page hides the rest, so a real audit always paginates.

Try it: run the audit-log gh api call with --jq. Expect one line per workflow action, with the actor and the time. Run it once without --jq and expect the full JSON.

The policy that blocks the unreviewed action. An action is code you did not write, running on your runner, with your secrets. The org policy is the allow list, and a repo that calls an action outside the list fails before it runs. I review the allow list the way I review code, because it is a security boundary.

Do this: add an action to the org allow list, then call a different action from a repo and run the workflow. Expected: the workflow fails before any step runs, with an error that names the policy. Add the action to the allow list, run again, and expect it to pass. The failure is the policy working.

gh api /orgs/my-org/actions/permissions --jq ".allowed_actions"

What the snippet means: the API reports the org action policy. A repo that calls an unlisted action fails fast, and the fix is a reviewed change to the list, not a quiet exception.

Go deeper

Start with the chain, then the gate, then the token. The workflow below runs after the build workflow completes, waits at the production environment, and mints an OIDC token. Do this: save it as .github/workflows/deploy.yml in a scratch repo, and run python3 -c "import yaml; yaml.safe_load(open('.github/workflows/deploy.yml'))". Expected output: the parse check prints nothing, and the file triggers on workflow_run only.

Read the flags: python3 -c and yaml.safe_load

The same one-line parse check as the reusable workflows lesson. A workflow that does not parse fails the whole run, so I check it before I push.

  • python3 -c: The -c flag runs the code string that follows it. Without it, python3 would expect a file name.
  • yaml.safe_load(open('.github/workflows/deploy.yml')): Safe_load reads the file and turns it into Python data. Safe means it never runs code from the file. A broken file raises an error, and the check fails fast.

Try it: run the python3 -c check on the deploy workflow. Expect silence. Break one line of indentation and run it again. Expect an error that names the line.

name: deploy
on:
  workflow_run:
    workflows: [build]
    types: [completed]

jobs:
  deploy:
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    runs-on: ubuntu-latest
    environment: production
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: actions/checkout@v4
      - run: make deploy
        shell: bash

What the snippet means: the workflow starts when the build workflow completes, runs the deploy job only if the build succeeded, waits at the production environment for approval, and mints an OIDC token with no stored key. Four gates in one file.

Dispatch is the cross-repo version. Do this: add a workflow that listens for on: {repository_dispatch: {types: [deploy]}} in a second repo, then send it an event from the first repo. Expected output: the second repo shows a run started by the event, with the payload you sent. If nothing appears, the event type in the sender does not match the type in the listener.

Self-hosted runners are one line and one machine. Do this: change a job's runs-on to [self-hosted, linux], push, and watch the run. Expected output: the job runs on your machine, and the log shows its name. If the job waits forever, the runner is offline, or its labels do not match the runs-on list.

Minutes are a budget line. Do this: open the org usage page, read the minutes used this month, and divide by the number of runs. Expected output: a cost per run, in minutes. If one workflow costs ten times the others, it is the first place to look for a slow step or a needless trigger.

At 10x the repos, the shape holds and the numbers grow. At 100x, the design changes: the chain gets a version dashboard, the runners get autoscaling, and the deploys get a queue with a human watching it. The chain never stops being workflows calling workflows. It just gets more callers. The next lesson in this track does the same job on Jenkins.

Runner scale sets: the pool that grows itself

A fixed pool of self-hosted runners is either idle or queued, and both cost money. A runner scale set grows the pool when the queue grows and shrinks it when the queue empties, and the queue is the metric that drives it.

Do this: watch the queue depth during a busy hour, then check the scale set size at the same time. Expected: the size follows the queue, up when jobs wait and down when they do not. Set a maximum size so a bad workflow cannot grow the pool without limit, and expect the queue to grow instead. A queue that grows is a visible problem. A bill that grows is a surprise.

gh api /orgs/my-org/actions/runners --jq ".runners | group_by(.status) | map({status: .[0].status, count: length})"

What the snippet means: the API groups the runners by status. Idle runners are money doing nothing. Queued jobs are developers waiting. The scale set trades one against the other.

One time it broke

What broke

A deploy I owned ran after a failed build. The deploy workflow triggered on workflow_run, which fires on completion, not on success. It had no conclusion check, so it deployed the broken build to production. The chain did exactly what the file said. The file said nothing about the build passing.

Timeline

  • 14:02,the build workflow failed on a test. The run went red.
  • 14:03,the deploy workflow started, because workflow_run fires on completion. It had no conclusion check.
  • 14:05,the deploy job ran. It deployed the broken build, because the build job's output was still there from the last green run.
  • 14:12,the app started serving the broken checkout page. The team reported it as a deploy bug.
  • 14:20,I added a conclusion check to the deploy job, rolled back the deploy, and re-ran the build.

What changed after

Two things. First, every workflow_run chain gets a conclusion check, and a chain without one is treated as broken in review. Second, the deploy job reads the build output only when the build passed, so a stale artifact cannot deploy. Try it now: open any workflow_run chain you own and read its if line. If there is none, add one before the next push.

If you remember nothing else

Build it yourself

Build a chained deploy across two workflows with a human gate and an OIDC token. The build workflow must run on push. The deploy workflow must trigger on workflow_run, run only when the build succeeded, wait at the production environment, and mint an OIDC token with id-token write. It passes when a green build triggers the deploy, the run pauses for approval, and the deploy job turns green after approval.

Build it, step by step

  • Create a build workflow that runs on push to main and runs the test suite.
  • Create a deploy workflow with on workflow_run for the build workflow, and an if line that checks the conclusion is success.
  • Add environment production to the deploy job, and set the production environment to require one reviewer.
  • Add permissions id-token write and contents read to the deploy job, and use the token in a deploy step with no stored key.
  • Validate both files with the YAML parse check, push to main, and confirm the chain fires, pauses for approval, and deploys after approval.

What success looks like

Success looks like this: both files parse with no output from the check. The build run turns green. The deploy run starts, pauses at the approval, and turns green after approval. The log shows no stored key anywhere.

If it fails, check this

If the deploy run never starts, the workflow_run trigger names the wrong workflow name. If the deploy runs after a failed build, the conclusion check is missing. If the run never pauses, the environment has no reviewers. If the token step fails, the cloud trust policy does not list this repo.

Next in this series
Lesson 50 — GitOps