Platform Engineering · Lesson 130

Ansible Playbooks and Roles

Lesson 110 gave you Ansible. It configures machines over SSH, and the second run changes nothing. This lesson is about writing the playbook well. Tasks in order, handlers that run once, variables that lose to the right winner, and roles that turn a playbook into something a team can share.

Difficulty: Advanced · Know this first: Lesson 110 — Ansible

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: on a cache hit the trip is almost free, on a miss the kitchen eats the budget. Step through the lesson or switch the scenario and watch the bar change.

Extra diagram · from scratch to master

The playbook, mapped: inventory, tasks, handlers

Start with one playbook. Add one box at a time until the whole run is on the map, wrong value included.

7 boxes on the trip1 handler1 wrong value
Playbook
0 ms
Inventory
5 ms
Variables
5 ms
Tasks
120 ms
Handlers
20 ms
Report
5 ms
Wrong value
port 9090
Request / flowResponse · the way backFailure / dead end

Try it: change the inputs and watch the trip bend

The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the inventory, the tasks, and the handlers change in the budget bar, in the hop timings, and in the total. The numbers are a teaching example, not live measurements. The shape is the lesson: more hosts and more tasks cost more time, and forks are the lever that buys it back.

12

How many machines the playbook configures. The inventory costs about half a millisecond per host to gather, and the tasks run in batches, so more hosts means more batches.

8

How many tasks each host runs. Each task costs a little time on each host, and the handler cost grows with the tasks, because more tasks means more chances to notify a restart.

5

How many hosts Ansible configures at once. Five forks means twelve hosts run in three batches. One fork means twelve batches. Forks are the lever: more forks, fewer batches, faster run.

Try this: (1) Set hosts to 50, tasks to 20, forks to 10. The inventory is about 25 ms, the tasks about 20 ms, the handlers about 20 ms, and the total is about 90 ms. (2) Set all three to the worst end: 100 hosts, 50 tasks, 1 fork. The inventory is 50 ms, the tasks are 1,000 ms, the handlers are 35 ms, and the total is about 1,110 ms. Same playbook. Only the inputs changed.

Type your own numbers: Get the total under 120 ms. Type 50 hosts, 20 tasks, 10 forks, and run it. Then drop the forks to 1 and watch the total.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

I take the same playbook four times here. First I learn the tasks. Then I learn roles, variables, and handlers. Then I learn dynamic inventory, Vault, and scaling. Last I run it for the world, and learn where it ends.

Basic — One playbook, a few hosts, SSH

Definition: A playbook is a list of plays. A play names the hosts and the tasks. A task names a module and what it should do: install a package, write a file, start a service. Ansible connects over SSH, checks the current state, and changes only what is different. The second run changes nothing, because nothing is left to change.

Types:

  • Plays, which name the hosts and the tasks.
  • Tasks, which name a module and its arguments.
  • Modules, the small tools that do one job: copy a file, install a package, manage a service.
  • The inventory, the list of hosts, which is the blast radius.

Uses: I use a playbook the first time I configure more than one machine by hand. I use it for the boring work: installing the same package on twelve hosts, writing the same config twelve times, starting the same service twelve times.

Dimensions: At low traffic everything is nearly free. What breaks first is the hand: twelve hosts configured by hand means twelve chances to type it differently. It costs almost nothing to write the playbook once. It costs an afternoon when the hosts disagree.

Try it yourself

Save this file as inventory.ini. It lists three hosts, all on your own machine, so the runs below are real but harmless.

[web]
web1 ansible_host=127.0.0.1 ansible_connection=local
web2 ansible_host=127.0.0.1 ansible_connection=local

[databases]
db1 ansible_host=127.0.0.1 ansible_connection=local
ansible -i inventory.ini web --list-hosts
ansible -i inventory.ini all -m ping

What you should see:

Read the flags: ansible -i inventory.ini web --list-hosts and ansible -i inventory.ini all -m ping

The ansible command runs one module on a pattern, right now, with no playbook. It is the fastest way to prove the inventory and the SSH path work.

  • -i inventory.ini: This names the inventory file. Without it, Ansible reads the default inventory, which is not the file in front of me. The -i flag is how every ad-hoc command knows which hosts exist.
  • web (the pattern): This is the group the command runs on. A pattern can be one host, one group, or all. The wrong pattern runs the module on the wrong machines.
  • --list-hosts: This lists the hosts the pattern matches and runs nothing. I run it before any real command, because a pattern typo shows up here for free.
  • all (the pattern): This matches every host in the inventory. It is the widest pattern, so I check the list before I run anything against all.
  • -m ping: The -m flag picks the module. Ping is the module that answers pong when SSH works. It changes nothing. If ping fails, the problem is SSH or the inventory, not the playbook.
  • -a (added): The lesson did not show this one yet. It passes arguments to the module, like ansible all -m shell -a 'uptime'. The ping module needs no arguments, and most other modules do.

Try it: run ansible -i inventory.ini web --list-hosts, then ansible -i inventory.ini all -m ping. Expect the list to show the web hosts and every host to answer pong.

  hosts (2):
    web1
    web2

The ping prints three lines that end in SUCCESS, one per host.

It passes when: the web group lists exactly web1 and web2, and all three hosts answer the ping. If a host does not answer, the inventory is wrong, not the network.

Try it: Basic

I run the playbook a second time. What changes?

Now do it: Run ansible -i inventory.ini all -m ping on your own machine. Expect every host to answer pong, then run the playbook and expect the web group to list web1 and web2.

Intermediate — Roles, variables, and handlers

Definition: A role is a playbook broken into a reusable shape: tasks, handlers, defaults, vars, templates, files. Variables come from many places, and one wins by precedence. Handlers run once at the end, when notified. Templates render config files from variables, so one template serves twelve hosts with twelve values.

Types:

  • Roles, the reusable unit, one job per role.
  • Variables, from role defaults up to extra vars, with a fixed order of winners.
  • Handlers, which restart a service once, at the end, when notified.
  • Templates, which render a config file per host from the same source.

Uses: I use roles the moment a second playbook needs the same work. I use templates when twelve hosts need twelve configs from one source. I use handlers whenever a config change needs a restart, so the restart happens once.

Dimensions: As the playbook grows, the variables grow, and the precedence ladder gets harder to hold in your head. What breaks is the value: a variable set in the wrong place, winning quietly. It costs a check of the winning value before the run. The trade is that roles add structure, and structure is work.

Try it yourself

Make the role skeleton, then save the playbook and check its syntax. The playbook writes two files and notifies one handler.

ansible-galaxy role init webserver
- name: Configure the web servers
  hosts: web
  gather_facts: false
  tasks:
    - name: Write the welcome file
      ansible.builtin.copy:
        dest: /tmp/pr-test/ans/welcome.txt
        content: "hello from the web tier\n"
        mode: "0644"
    - name: Write the web config
      ansible.builtin.copy:
        dest: /tmp/pr-test/ans/web.conf
        content: "port=8080\n"
        mode: "0644"
      notify: Restart web server
  handlers:
    - name: Restart web server
      ansible.builtin.debug:
        msg: "web server would restart now"
ansible-playbook -i inventory.ini site.yml --syntax-check

What you should see: the init prints - Role webserver was created successfully and creates the folders tasks, defaults, handlers, vars, meta, and tests. The syntax check prints playbook: site.yml.

Read the flags: ansible-galaxy role init webserver

A role is a folder with a shape. Galaxy init builds the shape, so I do not have to remember which folders a role needs.

  • ansible-galaxy: This is the tool for roles and collections. Init is one of its jobs. It never runs a playbook.
  • role init: This creates a new role skeleton. The folders it makes are tasks, handlers, templates, files, vars, defaults, and meta. An empty folder I skip is a folder I will need later.
  • webserver (the name): This is the role name, and it becomes the folder name. The playbook calls the role by this name, so a typo here is a typo everywhere.
  • ansible-galaxy collection install (added): The lesson did not show this one yet. It installs a collection from Galaxy, like ansible-galaxy collection install community.general. Roles I write myself, collections I download.

Try it: run ansible-galaxy role init webserver and list the folders it made. Expect tasks, handlers, templates, files, vars, defaults, and meta.

Read the flags: ansible-playbook --syntax-check

A syntax check parses the playbook and runs nothing. It is the cheapest gate I have, and it belongs before every commit.

  • --syntax-check: This checks the YAML parses and the playbook structure is valid. It runs no tasks and touches no hosts. Without it, a typo is found by a failed run on real hosts.
  • -i inventory.ini: This names the inventory. The syntax check still wants it, because the playbook names hosts from it.

Try it: run ansible-playbook -i inventory.ini site.yml --syntax-check. Expect silence or a pass. Break one line of YAML on purpose and run it again. Expect an error that names the line.

It passes when: the role folders exist and the syntax check names the playbook. Run the playbook once and the recap shows changed=2 on web2.

Try it: Intermediate

A config change needs a restart. When does the handler run?

Now do it: Run ansible-galaxy role init webserver on your own machine and list the folders it made. Expect tasks, handlers, templates, files, vars, defaults, and meta.

Advanced — Dynamic inventory, Vault, and scaling

Definition: A dynamic inventory reads the hosts from the cloud, so the list is never stale. Vault encrypts secrets, so passwords never sit in plain text. Strategies change how tasks flow across hosts: linear waits for every host, free lets fast hosts run ahead. Check mode shows what would change without changing anything.

Types:

  • Dynamic inventory, which asks the cloud who exists right now.
  • Vault, which encrypts secrets and decrypts them at run time.
  • Strategies, linear and free, which decide whether hosts wait for each other.
  • Check mode and diff, which show the change before it happens.

Uses: I use dynamic inventory when hosts come and go faster than a file can be edited. I use Vault the moment a password appears in a playbook. I use check mode before every run that touches prod, so the change is visible before it is real.

Dimensions: At ten times the hosts, the inventory file is the bottleneck, not the tasks. What breaks is the stale list: a host that no longer exists, or a new host the file never learned about. It costs a dynamic inventory and a check mode run. The trade is that the cloud is now the source of truth, and the cloud can be wrong too.

Try it yourself

Make a vault password file, encrypt one string, and run the playbook. The password is a throwaway for this drill, not a real secret. The run must prove the secret loads without ever printing it.

printf 'test-password\n' > .vault-pass
ansible-vault encrypt_string --vault-password-file .vault-pass 's3cret-db-pass' --name 'db_password'
- name: Show the vault value works
  hosts: localhost
  gather_facts: false
  vars:
    db_password: !vault |
                $ANSIBLE_VAULT;1.1;AES256
                30343962353764313132383364353464623635383135363866663366323531326138396566373432
                6532316431353733313235643938633536343132643531640a636330376139333833356232626664
                66386430353539373937336537653964343738326261383335383765646132613234373233333636
                6163636637623639640a393832343833393838326561356161666336323664333037326630393733
                3363
  tasks:
    - name: Confirm the secret loads without printing it
      ansible.builtin.debug:
        msg: "password is set, length is {{ db_password | length }} characters"
ansible-playbook -i localhost, -c local vault.yml --vault-password-file .vault-pass
ansible-playbook -i localhost, -c local vault.yml --vault-password-file .vault-pass > vault-run.log 2>&1
grep -c "s3cret-db-pass" vault-run.log

What you should see: the encrypt command prints a db_password: !vault | block. The run prints "msg": "password is set, length is 14 characters". The grep prints 0.

Read the flags: ansible-vault encrypt_string

A secret in plain text in a playbook is a secret in git forever. Vault encrypts the value, and the playbook holds the encrypted blob instead.

  • ansible-vault encrypt_string: This encrypts one string and prints the encrypted block. It encrypts values, not files. The output goes into the playbook as the variable value.
  • --vault-password-file .vault-pass: This names the file holding the vault password. Without it, Ansible asks for the password interactively. In a script, the prompt would hang, so the file is the only way it works there.
  • 's3cret-db-pass': This is the secret being encrypted. The quotes keep the shell from splitting it. It is a throwaway for this drill, never a real secret.
  • --name 'db_password': This names the variable the encrypted value belongs to. The output prints as db_password with the vault block, ready to paste into the playbook.

Try it: make a vault password file, encrypt one string, and run the playbook. Expect the run to print the password length but never the password itself.

Read the flags: ansible-playbook -i localhost, -c local

Localhost with a trailing comma is a one-host inventory written on the command line. The local connection runs it on my own machine, with no SSH at all.

  • -i localhost,: This is the whole inventory: one host, localhost. The trailing comma matters. Without it, -i reads localhost as a file name and fails, because no such file exists.
  • -c local: The -c flag picks the connection. Local runs the tasks on my machine instead of over SSH. Without it, Ansible would try to SSH to localhost, which is slower and can fail for no good reason.
  • --vault-password-file .vault-pass: This names the file with the vault password, so the encrypted value in the playbook can be read. Without it, the run stops and asks for the password.
  • > vault-run.log 2>&1: The greater-than sign sends normal output to the file. The 2>&1 part sends error output to the same file. Without the second half, errors would still print on screen and the log would look clean when it is not.
  • --ask-vault-pass (added): The lesson did not show this one yet. It asks for the vault password interactively instead of reading a file. I use it by hand when no password file exists.

Try it: run the vault playbook with the password file, then run grep -c on the log for the plain secret. Expect zero matches, because the secret never appears in output.

Read the flags: grep -c and grep -q

Grep searches text. The flag picks what comes back: a count, a quiet yes or no, or the matching lines themselves.

  • grep -c: The -c flag counts matching lines instead of printing them. Here it counts how many times the plain secret appears in the log. Zero is the pass. Without -c, grep would print the secret into my terminal, which is the leak I am checking for.
  • grep -q: The -q flag is quiet. It prints nothing and exits 0 when it finds a match, 1 when it does not. The lesson uses it later to check the recap for changed=0. Quiet grep is for scripts, not for eyes.
  • grep -r (added): The lesson did not show this one yet. It searches files under a directory, not one file. I use it to find a string across a whole repo, like grep -r "old-host" .
  • grep -n (added): The lesson did not show this one yet. It prints the line number with each match. I use it when I need to open the file at the match and fix it.

Try it: run grep -c "s3cret-db-pass" vault-run.log. Expect 0. Change the search word to changed and run it again. Expect a count above zero, because the recap lines are in the log.

It passes when: the run prints length 14 and the grep prints 0. The encrypted block is different every time you run it. The length and the zero are the same every time. Keep the password file out of the repo: it is the key to every secret in it.

Find where the value came from. The same variable set in three places resolves to one value, and the wrong one is the one I set in the place I forgot about. I do not guess. I print the value and the place it came from, and I change the place, not the value.

Do this: set the same variable in role defaults, in the inventory, and with -e on the command line, then run the debug task from the snippet. Expected: the printed value is the one from the command line, because extra vars beat everything. Remove the -e flag and run again, and expect the inventory value. The ladder is real, and the debug task is how I read it.

- name: Show where the value came from
  ansible.builtin.debug:
    msg: "app_port is {{ app_port }}, from the winning source"

What the snippet means: the debug task prints the resolved value. When a value looks wrong, the question is not what the value is. It is where it came from.

The restart that never fired. A handler runs only when a task reports changed. Change nothing and the handler does not run, which is correct. The trap is the handler that should run and does not, because the task that notifies it never reported a change.

Do this: run a playbook that copies a config file and notifies a restart handler, twice. Expected: the first run restarts the service, and the second run reports zero changes and no restart. Now change the config file by hand, run the playbook, and expect the restart to fire. If it does not, the notify line is missing, and that is the bug.

- name: Place the config file
  ansible.builtin.copy:
    src: app.conf
    dest: /etc/app/app.conf
    mode: "0644"
  notify: Restart the app

- name: Restart the app
  ansible.builtin.service:
    name: app
    state: restarted
  listen: Restart the app

What the snippet means: the handler is a listener, and the copy task notifies it only when the file changes. A restart on every run is a bug. A restart on no run is also a bug.

Try it: Advanced

A password appears in my playbook. What do I do with it?

Now do it: Make a vault password file and encrypt one throwaway string on your own machine, then run the playbook and confirm it prints the password length but never the password itself.

Master — Ansible at scale, and where it ends

Definition: At this level Ansible runs from a controller, with job templates, schedules, and an audit trail of every run. Execution environments carry the exact Ansible version and collections, so a run on my laptop matches a run in the pipeline. And the honest part: Ansible configures machines. It does not build them, it does not replace images, and at some size the right answer is an image, not a playbook.

Types:

  • A controller, which schedules runs, stores the history, and limits who may run what.
  • Execution environments, which pin the Ansible version and the collections.
  • Collections, the packaged roles and modules, versioned like libraries.
  • Images, the alternative: bake the config into the image, and stop configuring at run time.

Uses: I use a controller when more than one person runs playbooks, or when a run needs an audit trail. I use execution environments when a run works on one machine and fails on another. And I use images when the fleet is large enough that configuring each host is slower than replacing it.

Dimensions: At a hundred times the hosts the playbook is not the bottleneck, the SSH fan-out is. What breaks is the run that takes an hour because one host is slow and every host waits. It costs serial batches, timeouts, and a hard look at whether the fleet should be configured or replaced. The trade is that Ansible is the right tool for configuration, and the wrong tool for everything else.

Try it yourself

Run the playbook twice. The first run changes things. The second run must change nothing. That is the whole test for idempotency.

ansible-playbook -i inventory.ini site.yml
ansible-playbook -i inventory.ini site.yml

What you should see: the first recap shows changed=2 on web2. The second recap shows changed=0 on every host, and the handler does not run again.

It passes when: the second run reports changed=0 everywhere. If it does not, a task is lying about the state, and that task is the bug. Fix the task, not the handler.

A role that fails fast on the wrong OS. A role written for one OS fails strangely on another, halfway through, after it has already changed things. I fail fast instead. The first task checks the OS family and stops with a plain message when the role does not support it.

Do this: add the guard task from the snippet to a role, then run the role against a host with the wrong OS family. Expected: the run stops on the first task with a message that names the supported OS. Run it against the right OS and expect the role to proceed. A clean stop beats a half-applied role.

- name: Fail fast on an unsupported OS
  ansible.builtin.fail:
    msg: "This role supports Debian only, not {{ ansible_facts.os_family }}."
  when: ansible_facts.os_family != "Debian"

What the snippet means: the fail task reads the gathered facts and stops the run before anything changes. The message names the OS it found, so the next person knows what to fix.

Test the role with an assert, not with hope. A role that installs a service should prove the service is there. I add an assert task at the end of the role that checks the file, the service, or the port. The assert is the test, and it runs on every run, not just in CI.

Do this: add the assert task from the snippet after a role that installs a package, and run the playbook. Expected: the run passes when the package is installed, and fails with the assert message when it is not. Remove the package by hand, run the playbook again, and expect the assert to catch it.

- name: Prove the package is installed
  ansible.builtin.assert:
    that:
      - app_package_check.rc == 0
    fail_msg: "app package is not installed after the role ran"
    success_msg: "app package is installed"

What the snippet means: the assert task turns a silent failure into a loud one. A role without an assert can report success while the service is missing.

Try it: Master

What is the whole test for idempotency?

Now do it: Run the playbook twice on your own machine. Expect changed=2 on web2 the first time and changed=0 on every host the second time, with the handler not running again.

Go deeper

Variable precedence is the part of this lesson most people learn the hard way. The ladder has about twenty rungs, from role defaults at the bottom to extra vars at the top. You do not need to memorise all twenty. You need to know the ends: role defaults lose to everything, extra vars beat everything, and the inventory sits above the role. When a value looks wrong, the question is not what the value is. It is where it came from.

Idempotency is a property of the task, not of the playbook. A task that appends a line to a file is not idempotent: run it twice and the line appears twice. A task that writes the whole file is idempotent: run it twice and the file is the same. The test is simple. Run the playbook twice. The second run should report zero changed. If it does not, a task is lying about the state, and that task is the bug.

Handlers are the reason a config change restarts a service once instead of five times. The rule is strict: a handler runs at the end of the play, once, and only if notified. A handler that runs on every run is a handler notified on every run, which means a task keeps reporting changed. Fix the task, not the handler. And a handler that never runs is a task that never notifies it, which means the config changed and the service did not restart.

Templates are where variables become files. A template renders a config per host from the same source: the port, the name, the address, all filled in from the host's variables. One template, twelve configs. The rule I use: no host-specific files in the repo. If a file differs per host, it is a template. If it does not differ, it is a plain file.

Vault encrypts values, not files. You encrypt a string, you put the encrypted string in the playbook, and Ansible decrypts it at run time with a password that comes from somewhere else: a file, a prompt, a secret store. The password that unlocks the vault is the thing to protect. A vault password in the repo is a vault with the key taped to the door.

At 10x the hosts, the shape holds and the numbers grow. The run is slower, the batches are more numerous, the report is longer. At 100x, the design changes: dynamic inventory, a controller, execution environments, and a hard question about whether the fleet should be configured at all, or replaced from an image. The playbook never stops being a list of tasks. It just gets more hosts to run them on.

- name: Configure the web servers
  hosts: web
  roles:
    - role: webserver
      vars:
        http_port: 8080
  handlers:
    - name: Restart web server
      service:
        name: webserver
        state: restarted

What the snippet means: a play that runs the webserver role on the web group, with the port set to 8080 for this play. The role's tasks install and configure the server. When a task changes the config, it notifies the handler, which restarts the service once, at the end. Those nine lines are the whole play, and the handler is the reason the service restarts once, not five times.

Try it yourself

Prove which value wins. The playbook sets app_port to 8000. Pass 9000 on the command line and watch the extra var win.

- name: Which value wins
  hosts: localhost
  gather_facts: false
  vars:
    app_port: 8000
  tasks:
    - name: Show the winning port
      ansible.builtin.debug:
        msg: "app_port is {{ app_port }}"
ansible-playbook -i localhost, -c local prec.yml -e "app_port=9000"

What you should see: "msg": "app_port is 9000"

Read the flags: ansible-playbook -e

Extra vars win. They beat the playbook, the role defaults, and the inventory. That is why they are the fastest way to test one value.

  • -e "app_port=9000": This sets the variable app_port to 9000 for this run only. The file does not change. A value set with -e beats the same variable set anywhere else, so I use it to test and never to hide a real default.
  • -e @vars.yml (added): The lesson did not show this one yet. It loads extra vars from a file, like -e @vars.yml. I use it when the run needs many overrides, because a long -e string is easy to mistype.

Try it: run the playbook with -e "app_port=9000". Expect the output to show app_port is 9000. Run it again with no -e and expect the default value back.

It passes when: the output shows 9000. Extra vars sit at the top of the ladder. Role defaults sit at the bottom. The inventory sits above the role. When a value looks wrong, the question is not what the value is. It is where it came from.

The debug task is the print statement

When a playbook behaves strangely, I add debug tasks the way I add print statements to a script. One debug task per variable I doubt, placed right before the task that uses it. I run with -v and read the values.

Do this: add three debug tasks to a playbook, one for the variable, one for the host facts, and one for the loop item, then run ansible-playbook site.yml -v. Expected: the output shows the resolved value of each one, in order. The value that surprises you is the bug. Remove the debug tasks before the playbook ships, or leave one that prints the version, which is useful in every run.

Read the flags: ansible-playbook -v and ansible-playbook --step

Verbose mode shows me what Ansible did, task by task. Step mode asks me before each task. One is for reading, the other is for stopping a bad run early.

  • -v: This turns on verbose output. It shows results from successful tasks, not just failures. Without it, a passing run prints very little. Add another v, up to -vvv, for connection detail.
  • --step: This asks me to confirm each task before it runs. I answer yes, no, or continue. It is slow on purpose. I use it when a playbook touches prod and I want a stop between every task.
  • -vv (added): The lesson did not show this one yet. Two v flags show task paths and more detail. Three show SSH commands. I climb the ladder only when -v does not explain the failure.

Try it: add three debug tasks to a playbook and run ansible-playbook site.yml -v. Expect the output to show the variable, the host facts, and the loop item. Run with --step and expect a prompt before each task.

ansible-playbook site.yml -v
ansible-playbook site.yml --step

What the snippet means: -v shows the values, and --step pauses before each task so you can watch the run one task at a time. Both are for debugging. Neither belongs in the pipeline.

One time it broke

What broke

A playbook I owned configured the wrong hosts. The inventory file had two groups with similar names, and the play was aimed at the wrong one. The tasks ran cleanly. They reported changed on every host. The hosts were the staging database servers, and the playbook installed a web server on all of them and restarted a service that should never have been touched.

Timeline

  • 10:14,I ran the playbook against the inventory file, aiming at the web group.
  • 10:15,the tasks reported changed on twelve hosts. Twelve was the right number. The hosts were the wrong twelve.
  • 10:17,the handler restarted a service on the database servers. The service came back, but the config it loaded was a web server config.
  • 10:22,a teammate asked why the database servers were answering web traffic. That was the first sign the inventory was wrong.
  • 10:30,I compared the two groups in the inventory file. The names differed by one word. The play had used the wrong word.
  • 11:05,I ran a second playbook that removed the web server, restored the original config, and confirmed all twelve hosts were back to their old state.

What changed after

Three things. First, every run starts with a list of the hosts, and the list is read out loud before the run starts. Second, group names are never similar: web and web-db became webservers and databases, so a typo cannot pick the wrong one. Third, runs against anything that holds data need a second person to confirm the inventory, because the inventory is the blast radius.

Replay it yourself

Replay the guard that would have caught the wrong hosts. Before any run, list the hosts the play will touch and read the names out loud.

ansible-playbook -i inventory.ini site.yml --list-hosts
ansible -i inventory.ini databases --list-hosts

What you should see:

Read the flags: ansible-playbook --list-hosts and ansible --list-hosts

List-hosts answers one question before anything runs: which machines will this touch. It is the cheapest safety check in Ansible.

  • --list-hosts: This prints the hosts the playbook would run on and runs nothing. I run it after any inventory change, because a wrong host list here is a wrong run everywhere.
  • ansible -i inventory.ini databases --list-hosts: The same flag works on the ad-hoc command. The pattern after the inventory is the group, so this lists only the databases group.
  • --list-tasks (added): The lesson did not show this one yet. It lists the tasks the playbook would run, without running them. I use it with --list-hosts when I want the full picture: which hosts, and what would run on them.

Try it: run ansible-playbook -i inventory.ini site.yml --list-hosts. Expect the web hosts listed and nothing run. Add a host to the inventory and run it again. Expect the new host in the list.

  play #1 (web): Configure the web servers
    pattern: ['web']
    hosts (2):
      web2
      web1

The databases list shows exactly one host: db1.

It passes when: the web list shows two hosts and the databases list shows one, and the two lists share no host. The groups in this inventory are already named web and databases, so a typo cannot pick the wrong one. That rename is the fix from the incident. Keep it that way.

If you remember nothing else

Design exercise

Design an Ansible setup for 200 hosts in three groups: web servers, application servers, and databases. Three teams share the playbooks. A run must never touch the databases without a second person confirming. Secrets must never appear in the files or the output. A run that fails halfway must be safe to run again.

Clarifying questions to ask

  • How is the inventory organised, and how do you stop a typo from aiming a play at the wrong group?
  • Where do the variables live, and which level wins when two of them disagree?
  • How are the playbooks split into roles, and what does each role own?
  • Where do the secrets live, and who can read them? What appears in the output when a task uses one?
  • What does a failed run leave behind, and what does the second run do about it?

Rough sizing

Start with the budget from this lesson. Playbook 0, inventory half a millisecond per host, variables 5, tasks about a fifth of a millisecond per task per batch, handlers 10 plus half a millisecond per task, roles 10, vault 5, report 5. For 200 hosts, 8 tasks, and 20 forks that is 10 batches, so the tasks cost about 16 ms, the inventory about 100 ms, and the total about 160 ms in this teaching example. The real cost is not the milliseconds, it is the coordination: three teams, one inventory, and a database group that must never be touched by accident.

Trade-offs to name

One inventory buys a single list and costs a single mistake reaching every group. Group vars buy shared values and cost precedence surprises. Roles buy reuse and cost the wiring between them. Vault buys secret safety and costs a password to protect. And the target is a second run that changes nothing. A playbook that cannot prove that is a playbook that cannot be trusted.

Build it: the checkable version

Build the Ansible setup from the exercise: the inventory, one role, the playbook, and the vault secret. Then run the proof script. It checks the syntax, runs the playbook twice, and greps the log for the secret.

Steps:

  • Build the inventory, the role, and the playbook from the Basic and Intermediate sections.
  • Build the vault playbook from the Advanced section.
  • Save the proof script below as run-twice.sh and run it.
#!/bin/sh
# The idempotency proof: run twice, the second run must change nothing.
set -e
ansible-playbook -i inventory.ini site.yml --syntax-check
ansible-playbook -i inventory.ini site.yml > run1.log 2>&1
ansible-playbook -i inventory.ini site.yml > run2.log 2>&1
grep -q "changed=0" run2.log && grep -qv "changed=[1-9]" run2.log && echo "second run changed=0: PASS" || { echo "second run changed something: FAIL"; exit 1; }
if grep -q "s3cret-db-pass" run2.log; then echo "secret leaked into output: FAIL"; exit 1; fi
echo "secret never appears in output: PASS"
echo "RUN-TWICE OK"
sh run-twice.sh

What you should see:

playbook: site.yml
second run changed=0: PASS
secret never appears in output: PASS
RUN-TWICE OK

It passes when: the script prints RUN-TWICE OK. That means the syntax is clean, the second run changed nothing, and the secret never appears in any output. The approval gate for the databases is the --list-hosts step from the incident above: run it before every run against data, and have a second person read the list.

Next in this series
Lesson 131 — Ansible at Scale