Platform Engineering · Lesson 131

Ansible at Scale

Lesson 130 taught you to write a good playbook. This lesson is about running it on a fleet. Four hundred hosts do not behave like twelve. The inventory goes stale, one slow host holds up the batch, and the run that took a minute now takes an hour. The fixes are inventory, strategy, and knowing when to stop configuring and start replacing.

Difficulty: Advanced · Know this first: Lesson 130 — Ansible Playbooks and Roles

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: on a cache hit the trip is almost free, on a miss the kitchen eats the budget. Step through the lesson or switch the scenario and watch the bar change.

Extra diagram · from scratch to master

The fleet, mapped: inventory, forks, batches

Start with twelve hosts. Add one box at a time until four hundred hosts are on the map, straggler included.

7 boxes on the trip400 hosts1 straggler
Fleet
0 ms
Inventory
10 ms
Forks
20 ms
Slow host
0 ms
Check mode
15 ms
Batches
30 ms
Stalled
trip ends
Request / flowResponse · the way backFailure / dead end

Try it: change the inputs and watch the trip bend

The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the forks, the slow hosts, and the batches change in the budget bar, in the hop timings, and in the total. The numbers are a teaching example, not live measurements. The shape is the lesson: one slow host costs more than a hundred fast ones.

100

How many machines the playbook configures. More hosts means more batches, and the batch cost grows with the number of batches, not with the number of hosts directly.

20

How many hosts run at once. Twenty forks means a hundred hosts run in five batches. One fork means a hundred batches. Forks are the lever, until the control node becomes the bottleneck.

1

How many hosts are slow. Each slow host adds about 120 ms to the run, because the batch waits for it. One slow host costs 120 ms. Ten cost 1,200 ms, and the fast hosts did nothing wrong.

Try this: (1) Set hosts to 500, forks to 50, slow hosts to 3. The forks cost about 5 ms, the slow hosts about 360 ms, the batches about 30 ms, and the total is about 540 ms. (2) Set all three to the worst end: 1,000 hosts, 1 fork, 10 slow hosts. The forks cost 500 ms, the slow hosts 1,200 ms, the batches 3,000 ms, and the total is about 4,845 ms. Same playbook. Only the inputs changed.

Type your own numbers: Get the total under 600 ms. Type 500 hosts, 50 forks, 3 slow hosts, and run it. Then set the forks to 1 and watch the slow hosts take over.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

I take the same fleet four times here. First I learn the basics at any size. Then I learn dynamic inventory and strategies. Then I learn controllers, execution environments, and time. Last I ask whether the fleet should be configured at all.

Basic — One playbook, a few hosts, SSH

Definition: The basics from Lesson 130 still hold at any size: a playbook, an inventory, tasks that check first. What changes at scale is not the playbook. It is the fleet. Twelve hosts fit in your head. Four hundred do not, and the run needs help: a fresh inventory, enough forks, and a report you can read.

Types:

  • The inventory, which must be fresh, or the run aims at ghosts.
  • Forks, which decide how many hosts run at once.
  • The report, which must name the failures, not just count them.
  • Check mode, which shows the change before it is real.

Uses: I use the basics the same way at any size. I use check mode more often as the fleet grows, because the cost of a surprise grows with the fleet.

Dimensions: At low traffic everything is nearly free. What breaks first is the list: a host that no longer exists, or a new host the file never learned about. It costs almost nothing to ask the cloud. It costs an hour when the run aims at ghosts.

Try it yourself

Save this inventory with four hosts and this playbook with serial: 1. One host per batch, two hosts in the file below, so the play runs twice. Run it and count the plays.

[web]
web1 ansible_host=127.0.0.1 ansible_connection=local
web2 ansible_host=127.0.0.1 ansible_connection=local

[databases]
db1 ansible_host=127.0.0.1 ansible_connection=local
- name: Roll out in batches
  hosts: web
  gather_facts: false
  serial: 1
  tasks:
    - name: Mark this host done
      ansible.builtin.debug:
        msg: "batch host {{ inventory_hostname }} configured"
ansible-playbook -i inventory.ini serial.yml

What you should see: the play Roll out in batches runs twice, once per host, with "msg": "batch host web1 configured" in the first play and web2 in the second. The recap shows both hosts with changed=0.

Read the flags: ansible-playbook -i inventory.ini serial.yml

Serial is a playbook setting, not a flag. It runs the play on a few hosts at a time instead of all at once. The flag I pair with it is --limit, for the days I want even fewer.

  • -i inventory.ini: This names the inventory file. The playbook reads its hosts from it. Without it, Ansible uses the default inventory.
  • serial.yml (the playbook): This is the file that holds the play. Its serial setting says how many hosts run at once. The flag cannot change serial, so I edit the file when the batch size changes.
  • --limit (added): The lesson did not show this one yet. It narrows the run to hosts matching a pattern, like --limit web1. I use it to roll one host first, watch it, then roll the rest.

Try it: run ansible-playbook -i inventory.ini serial.yml. Expect the play to run in batches, with a recap per batch. Change serial in the file and run it again. Expect more batches and smaller ones.

It passes when: you count two separate plays in the output. Each batch is one host. A bad change stops after the first batch, not after the fleet.

Try it: Basic

Twelve hosts fit in my head. Four hundred do not. What does the run need at that size?

Now do it: Run the serial drill from this tier on your own machine and count the plays. One host per batch means the play runs twice, once per host.

Intermediate — Dynamic inventory and strategies

Definition: A dynamic inventory plugin asks the cloud for the hosts, groups them by tags, and builds the list at run time. Strategies decide how tasks flow: linear waits for every host at each task, free lets each host run ahead. Serial runs the play on a slice of the fleet at a time, so a bad change stops early.

Types:

  • Dynamic inventory plugins, one per cloud, which group hosts by tags.
  • Linear strategy, the default, where the slowest host sets the pace.
  • Free strategy, where fast hosts run ahead and slow hosts fall behind.
  • Serial batches, which run ten percent at a time and stop on failure.

Uses: I use dynamic inventory the moment hosts are created and destroyed by something other than me. I use free strategy when the hosts are independent. I use serial batches whenever a bad change could reach the whole fleet.

Dimensions: As the fleet grows, the strategy matters more than the tasks. What breaks is the default: linear, all at once, with no batches. It costs a strategy choice per play. The trade is that free is faster and harder to reason about, and serial is slower and safer.

Try it yourself

Save this file as fleet.py and make it executable. It is a dynamic inventory: the groups come from a list, not from a file you edit by hand.

#!/usr/bin/env python3
"""Tiny dynamic inventory: hosts come from a list, grouped by tag."""
import json
hosts = {
    "web1": {"ansible_host": "127.0.0.1", "ansible_connection": "local", "tier": "web"},
    "web2": {"ansible_host": "127.0.0.1", "ansible_connection": "local", "tier": "web"},
    "db1": {"ansible_host": "127.0.0.1", "ansible_connection": "local", "tier": "db"},
}
print(json.dumps({
    "web": {"hosts": ["web1", "web2"]},
    "databases": {"hosts": ["db1"]},
    "_meta": {"hostvars": hosts},
}))
chmod +x fleet.py
ansible-inventory -i fleet.py --list | python3 -c "import json,sys; d=json.load(sys.stdin); print('web:', d['web']['hosts']); print('databases:', d['databases']['hosts'])"

What you should see:

Read the flags: ansible-inventory -i fleet.py --list

An inventory can be a script, not a file. Ansible runs the script and reads the JSON it prints. That is how a fleet that changes every hour stays current.

  • -i fleet.py: This names the inventory source, and here the source is a Python script. Ansible runs it and reads the JSON it prints. A static file would go stale. A script asks the truth each time.
  • --list: This prints the whole inventory as JSON. It is the form a script reads. I pipe it to Python to pull out one group at a time.
  • | python3 -c: The pipe hands the JSON to Python. The -c flag runs the code string. The code prints the web hosts and the database hosts, so I can see both groups in one look.

Try it: run ansible-inventory -i fleet.py --list and pipe it to Python to print the web hosts. Expect the list to match the fleet script. Change the script and run it again. Expect the new hosts to appear.

web: ['web1', 'web2']
databases: ['db1']

It passes when: the web group lists two hosts and the databases group lists one. Add a host to the list in fleet.py and the groups change without touching any other file. That is the point of a dynamic inventory: the tags build the groups.

Try it: Intermediate

Hosts are created and destroyed by something other than me. What do I use?

Now do it: Run ansible-inventory -i fleet.py --list on your own machine and pipe it to Python to print the web hosts. Expect two web hosts and one database host.

Advanced — Controllers, execution environments, and time

Definition: A controller runs playbooks from one place, with job templates, schedules, and a history of every run. Execution environments pin the Ansible version and the collections, so a run matches everywhere it runs. Timeouts cut a hung host short, so one host cannot hold the fleet. And the run's time is measured per host, so the slow ones are visible.

Types:

  • A controller, which schedules runs and keeps the history.
  • Execution environments, which pin the version and the collections.
  • Timeouts, which stop a hung task instead of waiting forever.
  • Per-host timing, which names the slow hosts instead of hiding them.

Uses: I use a controller when more than one person runs playbooks, or when a run needs an audit trail. I use timeouts on every run that touches prod, because a hang without a timeout is a run that never ends.

Dimensions: At ten times the hosts, the controller is the bottleneck, not the tasks. What breaks is the run with no history: nobody knows what ran, when, or on which hosts. It costs a controller and a timeout on every task that can hang. The trade is that centralising the runs adds a service to look after.

Try it yourself

Run the same playbook twice, once with strategy free and once with linear. The work is uneven on purpose: web1 sleeps 4 seconds, web2 sleeps 1. Watch which host finishes task 2 first.

- name: Uneven work shows the strategy
  hosts: web
  gather_facts: false
  strategy: linear
  tasks:
    - name: Slow host works long, fast host works short
      ansible.builtin.command: "sleep {{ 4 if inventory_hostname == 'web1' else 1 }}"
    - name: Report done with a timestamp
      ansible.builtin.debug:
        msg: "{{ inventory_hostname }} finished task 2"
ansible-playbook -i inventory.ini strat2.yml
sed -i 's/strategy: free/strategy: linear/' strat2.yml
ansible-playbook -i inventory.ini strat2.yml

What you should see: with free, web2 finished task 2 prints before web1 finished task 2. With linear, web1 finished task 2 prints first.

Read the flags: sed -i

Sed edits a file in place. One command changes the strategy line, and the next run proves the change did what I meant.

  • sed -i: The -i flag edits the file in place. Without it, sed prints the changed text and the file stays as it was. I use -i only on a file I can rebuild, because there is no undo.
  • 's/strategy: free/strategy: linear/': This is the substitution. The s means substitute. The first text is what to find, the second is what to put there. The slashes separate the parts.
  • strat2.yml: This is the file being edited. Sed reads it, changes the line, and writes it back.
  • -i.bak (added): The lesson did not show this one yet. It keeps a backup copy with the .bak ending. I use it when the file matters, so a bad substitution can be undone by copying the backup back.

Try it: run ansible-playbook -i inventory.ini strat2.yml, then sed the strategy to linear, then run the playbook again. Expect web2 to finish task 2 first with free, and web1 to finish task 2 first with linear.

It passes when: the order flips between the two runs. Free lets each host run ahead. Linear waits for every host at every task. The strategy is a property of the play, and the play should say why it chose it.

Forks: the number that makes it slower. More forks is not always faster. Each fork is a connection and a process, and past a point the controller spends its time managing forks instead of running tasks. I time the run at two fork counts and pick the faster one, on my fleet, not on a guess.

Do this: time the same playbook with -f 5 and then with -f 50. Expected: one of the two is clearly faster, and the faster one is the number to use. Write it in the config file so the next run uses it without a flag.

time ansible-playbook site.yml -f 5
time ansible-playbook site.yml -f 50

What the snippet means: the wall time is the measurement. A fork count copied from a blog post is a guess. A fork count timed on your fleet is a setting.

Read the flags: time ansible-playbook -f 5 and -f 50

Forks are parallelism. Five hosts at once is the default. Fifty is a test of the control node, not just of the hosts.

  • time: This is not an Ansible flag. It is a shell word that times the command after it. It prints how long the run took, so I can compare 5 forks against 50 with numbers, not feelings.
  • -f 5: The -f flag sets forks, the number of hosts Ansible talks to at once. Five is the default. With -f 5, the sixth host waits for one of the first five to finish.
  • -f 50: Fifty forks run fifty hosts at once. It is faster until the control node runs out of memory or SSH connections. Then it is slower, and the timing proves it.
  • --forks (added): The lesson did not show this one yet. It is the long form of -f, like --forks 20. I use it in scripts, where a reader should not have to guess what f means.

Try it: run time ansible-playbook site.yml -f 5, then -f 50. Expect the 50-fork run to be faster on a small fleet and to flatten out as the fleet grows. Copy the two times into a note.

Serial batches and the failure budget. A rolling change across four hundred hosts needs a failure budget. If one host in a batch fails, I want the run to stop, not to roll the bad change to the next three hundred. Serial batches and max_fail_percentage are the budget, written down.

Do this: add the play header from the snippet, then run the playbook against a batch where one host is known to fail. Expected: the run stops after the batch, and the remaining hosts are untouched. The failure budget is the line between a bad deploy and an outage.

- name: Rolling update with a failure budget
  hosts: web
  serial: "25%"
  max_fail_percentage: 10
  tasks:
    - name: Deploy the new version
      ansible.builtin.copy:
        src: app.conf
        dest: /etc/app/app.conf
        mode: "0644"

What the snippet means: serial runs the play in batches of a quarter of the fleet, and max_fail_percentage stops the run when more than a tenth of a batch fails. The rest of the fleet keeps running the old version.

Try it: Advanced

More forks is not always faster. How do I pick the fork count?

Now do it: Time the same playbook with -f 5 and then with -f 50 on your own machine. Write the faster number in the config file so the next run uses it without a flag.

Master — Configure, or replace

Definition: At this level the honest question is whether the fleet should be configured at all. Configuring four hundred hosts takes time, and every run is a chance for drift. Replacing them from an image takes the same time once, and every host starts identical. The answer depends on how often the config changes, how fast the fleet must change, and how much drift you can tolerate.

Types:

  • Configuration, which changes hosts in place, and drifts over time.
  • Images, which bake the config in and replace the host when it changes.
  • Rolling replacement, which swaps hosts a few at a time, like a serial batch.
  • Drift detection, which compares the fleet against the playbook on a schedule.

Uses: I use configuration when the change is small and frequent. I use images when the fleet is large and the config changes slowly. And I use drift detection either way, because a fleet that drifts is a fleet that lies about its state.

Dimensions: At a hundred times the hosts the playbook is not the bottleneck, the idea is. What breaks is the fleet that is configured in place for a year and has drifted in a hundred small ways. It costs a decision: configure, or replace. The trade is that replacing is faster at scale and slower to change, and configuring is the reverse.

Try it yourself

Run the preflight script before any fleet run. Then prove the timeout works: a task that sleeps 30 seconds with a 5 second timeout must fail fast, with the host named.

#!/bin/sh
# Preflight before any fleet run: the tool, the list, the secrets.
set -e
ansible --version | head -1
ansible -i inventory.ini web --list-hosts | tail -n +2 | wc -l | xargs echo "web hosts:"
test -f .vault-pass && echo "vault password file: present"
ansible-playbook -i inventory.ini site.yml --syntax-check
echo "PREFLIGHT OK"
- name: A task that must not hang forever
  hosts: localhost
  gather_facts: false
  tasks:
    - name: Wait for a package lock that never clears
      ansible.builtin.command: sleep 30
      timeout: 5
sh preflight.sh
ansible-playbook -i localhost, -c local timeout.yml

What you should see: PREFLIGHT OK, then Task failed: Timed out after 5 second(s). in about 6 seconds.

Read the flags: ansible --version, head, tail, wc, and xargs

Small Unix tools answer small questions. Each flag here picks how much output I see, and the pipe chains them into one answer.

  • ansible --version: This prints the Ansible version and exits. It runs nothing. I check it first when a flag behaves differently than the docs say, because the version is usually the reason.
  • | head -1: The pipe hands the output to head. The -1 flag keeps only the first line. Without it, head prints ten lines, which is more than I need for a version.
  • tail -n +2: The tail flag -n +2 means start printing at line two. It drops the header line. Without it, the header would be counted as a host.
  • wc -l: The wc command counts. The -l flag counts lines. Here it counts hosts, because each host is one line after the header is dropped.
  • xargs echo "web hosts:": Xargs takes the count from the pipe and puts it after the words web hosts. Without xargs, the count would print on its own line with no label.
  • head -20 (added): The lesson did not show this one yet. It keeps the first twenty lines. I use it when a command prints pages and I only need the top of the list.

Try it: run ansible -i inventory.ini web --list-hosts | tail -n +2 | wc -l. Expect the number of web hosts. Add a host to the inventory and run it again. Expect the count to go up by one.

Read the flags: test -f and &&

Test checks a fact about a file. The double ampersand runs the next command only when the fact is true. Together they are a one-line guard.

  • test -f .vault-pass: The -f flag checks the file exists and is a regular file. It prints nothing. It exits 0 when the file is there and 1 when it is not.
  • &&: This runs the command after it only when the command before it exits 0. Without it, the echo would run even when the password file is missing, and the message would lie.
  • echo "vault password file: present": Echo prints the words. It runs only when test passed, so the message is true when it prints.

Try it: run test -f .vault-pass && echo present. Expect the message when the file exists. Delete the file in a scratch directory and run it again. Expect silence.

It passes when: the preflight prints OK and the timeout run fails in about 6 seconds instead of 30. A hang with no timeout is a run that never ends. A timeout that fires is a failure you can see, with a host you can name.

Fact caching: gather once, reuse. Gathering facts on four hundred hosts takes minutes, and most runs gather facts they already gathered an hour ago. A fact cache stores the facts and reuses them, so the run starts faster and the controller does less work.

Do this: enable the JSON file cache from the snippet, run a playbook twice, and time both runs. Expected: the second run is faster, and the cache directory holds one file per host. Delete the cache directory and run again, and expect the slow first-run time back.

[defaults]
gathering = smart
fact_caching = jsonfile
fact_caching_connection = /tmp/ansible-facts
fact_caching_timeout = 3600

What the snippet means: smart gathering skips fact collection when the cache is fresh, and the timeout says how fresh is fresh enough. An hour is a reasonable start.

When the controller is the bottleneck. At some size the controller is the bottleneck, not the hosts. Every task goes through one machine, and one machine has limits. The answer is to stop pushing from one place, and let each host pull its own config instead.

Do this: run ansible-pull from the snippet against a local repo on a test host. Expected: the host clones the repo, runs the playbook locally, and reports the result. The controller never sees the run. Pull fits a fleet that changes slowly. Push fits a fleet that needs one command to change everything at once.

ansible-pull -U /tmp/ansible-repo -d /tmp/ansible-local -i localhost, -c local site.yml

What the snippet means: ansible-pull turns the host into its own controller. The trade is control: a pull host applies the config on its own schedule, and a push run applies it now.

Read the flags: ansible-pull -U and -d

Ansible-pull flips the model. The host fetches the playbook from git and runs it on itself. No control node pushes anything.

  • ansible-pull: This checks out a repo and runs a playbook from it, on this machine. It is how laptops and lone servers keep themselves configured without a control node.
  • -U /tmp/ansible-repo: The -U flag gives the repo URL or path to pull from. Without it, ansible-pull has nothing to fetch. Here it is a local path, which is how I test the flow before I point it at real git.
  • -d /tmp/ansible-local: The -d flag gives the directory to check the repo out into. Without it, ansible-pull picks a default under the home directory, and I would have to hunt for it.
  • -i localhost,: This is the one-host inventory, localhost, with the trailing comma. Without the comma, -i would read localhost as a file name and fail.
  • -c local: This runs the tasks on this machine instead of over SSH. Ansible-pull always runs locally, and -c local says so plainly.

Try it: run ansible-pull -U /tmp/ansible-repo -d /tmp/ansible-local -i localhost, -c local site.yml. Expect the checkout in /tmp/ansible-local and the playbook to run on this machine.

Try it: Master

The fleet is large and the config changes slowly. Do I configure it in place or replace it from an image?

Now do it: Run the preflight script from this tier before your next fleet run on your own machine, then prove the timeout works with a task that sleeps 30 seconds and a 5 second timeout.

Go deeper

Dynamic inventory is a plugin, not a file. It calls the cloud API, lists the hosts, and groups them by tags. The groups are built from the tags, so a host tagged web joins the web group without anyone editing a file. The cost is the API call: about 40 ms for four hundred hosts in this lesson's example, and a dependency on the cloud API being up. If the API is down, the inventory is empty, and the run has nothing to aim at.

Strategies are about waiting. Linear waits for every host at every task, so the run is a march in step. Free lets each host run ahead, so the run is a race, and the fast hosts finish while the slow ones are still on the second task. Free is faster when the hosts are independent. It is dangerous when a task needs all hosts ready, like a rolling restart that must not take down the whole service at once. The strategy is a property of the play, and the play should say why it chose it.

Serial batches are a canary deploy for configuration. Run the play on ten percent, watch it, then run the next ten percent. A bad change hits the first batch and stops. The cost is time: ten batches take longer than one run. The saving is the incident that did not happen to ninety percent of the fleet. I use serial batches whenever the change can break the service, which is most changes worth making.

Timeouts are the fix for the hung host. A task without a timeout waits forever, and a run without a timeout waits with it. Set a timeout on every task that talks to the network, and set one on the run itself. A timeout that fires is a failure you can see, with a host you can name. A hang with no timeout is a run that never ends and a fleet that never finishes.

Execution environments solve the problem of a run that works on one machine and fails on another. The environment is a container with a pinned Ansible version and pinned collections. The controller runs the playbook inside it, and my laptop runs the same container. The run matches because the environment matches. The cost is building and storing the image. The saving is the afternoon spent finding out the collections differ.

At 10x the hosts, the shape holds and the numbers grow. The batches are more numerous, the report is longer, the slow hosts are more visible. At 100x, the design changes: the question stops being how to configure the fleet and starts being whether to configure it at all. Images replace hosts. The playbook becomes the thing that builds the image, not the thing that touches the host. The lesson never stops being about the fleet. It just moves one level up.

- name: Configure the fleet in batches
  hosts: web
  serial: 10%
  strategy: free
  tasks:
    - name: Install the web server
      package:
        name: webserver
        state: present
      async: 300
      poll: 10

What the snippet means: a play that runs on the web group ten percent at a time, with the free strategy so fast hosts run ahead, and a task that runs asynchronously with a 300-second timeout and a check every 10 seconds. Those ten lines are the whole scaling policy for this play: batches for safety, free for speed, and a timeout so one hung host cannot hold the fleet.

Try it yourself

Run the serial playbook against four hosts at 25 percent. Each batch is one host, and the output shows each batch as its own play.

[web]
w1 ansible_host=127.0.0.1 ansible_connection=local
w2 ansible_host=127.0.0.1 ansible_connection=local
w3 ansible_host=127.0.0.1 ansible_connection=local
w4 ansible_host=127.0.0.1 ansible_connection=local
- name: Roll out in batches
  hosts: web
  gather_facts: false
  serial: 25%
  tasks:
    - name: Mark this host done
      ansible.builtin.debug:
        msg: "batch host {{ inventory_hostname }} configured"
ansible-playbook -i inv4.ini serial25.yml

What you should see: four separate plays, one per host, and a recap with all four hosts at changed=0.

It passes when: you count four plays. Ten percent at a time is ten runs, not one. That is the cost of the canary, and the saving is the incident that did not happen to ninety percent of the fleet.

Measure the run before you tune it

I never tune a slow run by guessing. I turn on per-task timing, run the playbook, and read the slowest tasks. The slowest task is the work. Everything else is noise.

Do this: enable profile_tasks in ansible.cfg, run the playbook, and copy the five slowest tasks into a note. Expected: one or two tasks dominate the total, usually a download or a service restart. Fix the slowest one first, re-time the run, and expect the total to move. If it does not move, the slow task was not the bottleneck, and the timings will say what is.

time ansible-playbook site.yml
# then fix the slowest task and run it again
time ansible-playbook site.yml

What the snippet means: the two timings are the before and the after. A tuning change without a re-time is a guess. A tuning change with a faster second run is a fix.

One time it broke

What broke

A run I owned configured four hundred hosts at once, with no batches and no timeout. One host hung on a package install. The run waited. Four hundred hosts waited with it, because the strategy was linear and the run had no timeout. The run never finished. The fleet sat half-configured for an hour while I worked out which host was stuck.

Timeline

  • 09:00,the run started against four hundred hosts, all at once, no serial batches.
  • 09:12,the run stopped making progress. The output showed hosts finishing, then nothing. No error, no failure, just silence.
  • 09:25,I checked the report. Three hundred and ninety-nine hosts were done. One host was still on the second task.
  • 09:40,I logged into the hung host. Its disk was full, and the package install was waiting on a lock that would never clear.
  • 09:55,I killed the task on the hung host and the run finished. Three hundred and ninety-nine hosts had waited forty minutes for one host.
  • 10:20,I re-ran the playbook against the one hung host after clearing its disk, and confirmed the whole fleet was configured.

What changed after

Three things. First, every fleet run uses serial batches, ten percent at a time, so a bad change stops early. Second, every task that talks to the network has a timeout, so a hang becomes a failure with a name. Third, the report is read per host before the run is called done, because the slowest host is the story.

Replay it yourself

Replay the hang. Run a playbook with a 30 second sleep and no timeout, wrapped in timeout 10 so your terminal comes back. Then run the same work with a 5 second timeout.

- name: A run with no timeout anywhere
  hosts: localhost
  gather_facts: false
  tasks:
    - name: Wait on a lock that never clears
      ansible.builtin.command: sleep 30
timeout 10 ansible-playbook -i localhost, -c local hang.yml; echo $?
ansible-playbook -i localhost, -c local timeout.yml

What you should see: the first command prints 124 after 10 seconds. The wrapper killed it. The run never ended on its own. The second fails in about 6 seconds with Timed out after 5 second(s).

Read the flags: timeout 10 and echo $?

A hung run is a run with no end. Timeout gives it one. Ten seconds, then the wrapper kills it, and the exit code tells me the wrapper did the killing.

  • timeout 10: This runs the command after it and kills it after ten seconds if it is still going. Without it, a hung playbook runs until I notice it and kill it by hand.
  • ansible-playbook -i localhost, -c local hang.yml: This is the hung playbook, run on localhost over the local connection. The -i flag with the trailing comma is the one-host inventory. The -c local flag skips SSH.
  • ; echo $?: The semicolon runs the next command after the first finishes. The dollar question mark prints the exit code. Expect 124, which is the code timeout uses when it kills the command.
  • --preserve-status (added): The lesson did not show this one yet. It makes timeout exit with the command exit code instead of 124. I use it when the caller needs to know why the command ended, not just that timeout killed it.

Try it: run timeout 10 ansible-playbook -i localhost, -c local hang.yml, then echo $?. Expect 124 after ten seconds. Run the timeout.yml playbook next and expect a task failure in about six seconds.

It passes when: the first exits 124 and the second fails fast with the timeout message. Exit 124 means the wrapper gave up. The run itself never would have.

If you remember nothing else

Design exercise

Design an Ansible rollout for 1,000 hosts in four regions. The change is a config file that, if wrong, takes the service down. The rollout must finish in under an hour, a bad change must reach at most 10 percent of the fleet, and one hung host must never stop the run. Two teams share the playbooks.

Clarifying questions to ask

  • How is the fleet split into batches, and what size is the first batch?
  • What strategy does the play use, and why? What happens if a task needs all hosts ready?
  • How long may one host hang before the run gives up on it, and what happens to that host after?
  • How is the inventory built, and what happens when the cloud API is down?
  • What does the report show at the end, and who reads it before the rollout is called done?

Rough sizing

Start with the budget from this lesson. Fleet 0, inventory about a tenth of a millisecond per host from the cloud API, strategy 5, forks about half a millisecond per host per fork count, slow hosts 120 each, check mode about 15 per hundred hosts, batches about 3 per batch, report about a tenth of a millisecond per host. For 1,000 hosts, 50 forks, and 2 slow hosts that is about 100 for the inventory, 10 for the forks, 240 for the slow hosts, 60 for the batches, and 205 for the fixed hops, about 615 ms in this teaching example. The real cost is not the milliseconds, it is the hour: a run that waits on one host can eat the whole window.

Trade-offs to name

Batches buy safety and cost time: ten percent at a time is ten runs, not one. Free strategy buys speed and costs the guarantee that all hosts move together. Timeouts buy a run that ends and cost the host that gets cut off mid-task. Dynamic inventory buys a fresh list and costs a dependency on the cloud API. And the target is a run that finishes. A rollout that never ends is worse than a rollout that fails loudly.

Build it: the checkable version

Build the rollout from the exercise for ten stand-in hosts on your own machine. The inventory is generated by a script, the playbook runs in batches of 10 percent with the free strategy, every network task has a timeout, and the run stops if more than 10 percent of a batch fails.

Steps:

  • Save the generator below as gen-inventory.py and run it.
  • Save the playbook below as rollout.yml.
  • Check the syntax, run it, and count the files.
#!/usr/bin/env python3
"""Generate the 10-host stand-in inventory for the rollout drill."""
lines = ["[web]"]
for i in range(1, 11):
    lines.append(f"w{i} ansible_host=127.0.0.1 ansible_connection=local")
open("inv10.ini", "w").write("\n".join(lines) + "\n")
print("wrote inv10.ini with 10 hosts")
- name: Roll out the config in batches
  hosts: web
  gather_facts: false
  serial: 10%
  strategy: free
  max_fail_percentage: 10
  tasks:
    - name: Write the new config file
      ansible.builtin.copy:
        dest: "/tmp/pr-test/ans/rollout-{{ inventory_hostname }}.conf"
        content: "version=2\n"
        mode: "0644"
      timeout: 30
python3 gen-inventory.py
ansible-playbook -i inv10.ini rollout.yml --syntax-check
ansible-playbook -i inv10.ini rollout.yml
ls rollout-*.conf | wc -l

What you should see: playbook: rollout.yml, ten separate plays in the run, a recap with all ten hosts at failed=0, and 10 from the file count.

It passes when: the run shows ten plays, ten files, and zero failures. The first batch is one host, so a bad change reaches at most 10 percent of the fleet. That is the exercise target, proven on ten hosts: the rollout finishes, a bad change stops early, and one hung host never stops the run.

Next in this series
Lesson 111 — Jenkins