From manual tweaks to Ansible roles: first steps

SSH in. Tweak something. Move on to the next machine. Repeat. Sound familiar? Ansible turns those manual tweaks into configuration you can read, review and run again, on one machine or on ten.

This is the second post in my AI homelab series. The previous post ended with a plan to sandbox Claude Code. That lab will set up its sandbox with an Ansible role, and Lab 0a had also left me with two fixes I’d made by hand: an AppArmor exception and a kernel setting. Before moving on, it made sense to step back, set up Ansible properly and learn the basics.

That’s what this post is about: installing Ansible, configuring it, and taking the first steps toward managing the homelab as code. The very first run against a server I had configured by hand found exactly one difference: two spaces of indentation.

Even if you’re not following the series, you’re in the right place: you can follow along and learn the Ansible basics without reading any of the previous posts.

This one is Lab 0g. A laptop becomes the control node, the server is reached over plain SSH, the inventory holds nothing but host names, and the two fixes from Lab 0a become two roles that Ansible checks, applies and keeps in place.

• Guide: every command, step by step
• Results: versions, runs and surprises
• Entry point: site.yml
• Role common: the AppArmor sysctl, on disk and in the kernel
• Role lab_tools: the unshare copy and its profile from Lab 0a
• Configuration: ansible.cfg
• Interpreter pin: group_vars/all.yml

What “done” meant

Four checks, decided before the first run:

• From the control node over SSH, a dry run shows the drift between the server and the repo, a real run fixes it, and a second run changes nothing.
• A change I make by hand on purpose is detected by the dry run and corrected.
• The kernel setting from Lab 0a lives in a role, the bypass it closes stays closed, and the AppArmor exception still works.
• Every concept used is explained in the guide.

Setup

Two machines with two roles:

Ansible has no agent. For every task the control node packages a small Python module with its arguments, copies it to the node over SSH, runs it there with the node’s Python, reads back a JSON result and deletes the temporary files. That is why the node needs nothing installed.

Installing Ansible

Ubuntu 24.04’s archive ships ansible-core 2.16.3, from early 2024. PyPI had 2.21.5, which needs Python 3.12 or newer on the control node. I took that, with two decisions:

• ansible-core, not ansible. The ansible package is a bundle of hundreds of community collections. Everything in this lab uses only ansible.builtin.
• pipx with an exact version. pipx installs a Python application into its own virtualenv, so the system Python and its packages are left alone, and ==2.21.5 makes the lab repeatable.

On macOS there was a catch. The system Python is 3.9, too old, and Homebrew’s pipx runs on its own Python, 3.15 at the time, which ansible-core 2.21.5 does not list as supported (3.12–3.14). So Ansible’s virtualenv gets an explicit interpreter:

brew install python@3.12 pipx
pipx install --python /opt/homebrew/bin/python3.12 'ansible-core==2.21.5'
pipx ensurepath

==2.21.5 pins Ansible, not its dependencies. pipx runpip ansible-core freeze lists the exact versions; I also installed Ansible on the server for checks that need no connection, and both freezes matched package for package (Jinja2 3.1.6, PyYAML 6.0.3, cryptography 50.0.2 and the rest).

Reaching the node

SSH does the connecting

Ansible connects with the control node’s OpenSSH and honours ~/.ssh/config. So everything needed to reach the server goes there, and nowhere else:

Host z790
  HostName z790
  User <user>
  Port 22
  IdentityFile ~/.ssh/id_ed25519_lab
  IdentitiesOnly yes

IdentityFile names the key instead of relying on whatever the SSH agent happens to hold, and IdentitiesOnly offers only that key. Before Ansible ever runs, the same connection can be tested the way Ansible will use it, with prompts forbidden:

ssh -o BatchMode=yes -o ConnectTimeout=5 z790 'hostname; python3 --version'

An inventory that holds only names

With the connection details in ~/.ssh/config, the inventory is just a list of hosts. No addresses, users or ports in the repo, and adding a node later is one more line.

I did consider running Ansible on the server itself with a local connection. It would have worked, and it would have hidden exactly the parts that break in real use: SSH authentication, privilege escalation over an SSH session, and Python discovery on another machine.

ansible.cfg is read from the current directory

The whole configuration is one setting:

# Read only when Ansible runs from this directory (or via ANSIBLE_CONFIG).
[defaults]
inventory = inventory/hosts.yml

Ansible looks for ansible.cfg in $ANSIBLE_CONFIG, then the current directory, then ~/.ansible.cfg, then /etc/ansible/. A relative path inside it resolves against the file’s own directory. So all commands run from ansible/; from anywhere else there is no inventory. ansible-config dump --only-changed shows which settings are in effect and where each one came from.

First contact

ping is not ICMP

ansible z790 -m ansible.builtin.ping

Ansible’s ping module tests the whole chain: SSH with the configured key, copying a module to the node, finding Python there, running it and getting JSON back. The first one returned pong, and a warning.

Python discovery does not pick the system Python

[WARNING]: Host 'z790' is using the discovered Python interpreter at '/usr/bin/python3.12', but future installation of another Python interpreter could cause a different interpreter to be discovered.

The default discovery walks a fixed list that starts at python3.14 and goes down, and only near the end tries /usr/bin/python3. On Ubuntu 24.04 it found python3.12 first. Today that is the same binary. But a newer python3.13 installed later, from a PPA or as a dependency, would silently become Ansible’s interpreter, and modules that need the distribution’s Python packages would break: ansible.builtin.apt needs python3-apt, which is built only for the system Python. One line in group_vars/all.yml pins it for every managed host:

ansible_python_interpreter: /usr/bin/python3

After that, ping returned pong with no warning. Privilege escalation got its own test with a harmless command, ansible z790 -m ansible.builtin.command -a id --become, which printed uid=0(root).

Two spaces of drift

The first role, lab_tools, holds the AppArmor exception from Lab 0a: a dedicated copy of unshare and a profile that lets it create user namespaces. I had installed both by hand. The first run was a dry run, --check --diff, which runs every check as root and applies nothing:

TASK [lab_tools : Copy unshare to a dedicated binary for Lab 0a] *****
ok: [z790]

TASK [lab_tools : Install AppArmor profile that allows user namespaces for the copy] *****
--- before: /etc/apparmor.d/lab0a-unshare
+++ after: <repo>/ansible/roles/lab_tools/files/lab0a-unshare
@@ -2,5 +2,5 @@
 include <tunables/global>

 profile lab0a-unshare /usr/local/bin/lab0a-unshare flags=(unconfined) {
-  userns,
+userns,
 }

changed: [z790]

RUNNING HANDLER [lab_tools : Reload lab0a-unshare AppArmor profile] *****
skipping: [z790]

PLAY RECAP *****
z790 : ok=3 changed=1 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0

The binary was identical. The profile on the server was indented, the one in the repo was not. The repo is the source of truth, so Ansible reported a change.

The handler line is worth a second look. A handler runs only when a task that notifies it reports changed, so here it started. Its task is a command (apparmor_parser -r), and command cannot know what an arbitrary command would do, so in check mode it does not run it and reports skipped. Ansible does not even evaluate changed_when for a skipped task: I checked that in the ansible-core source before trusting the recap.

The real run replaced the profile and reloaded it (ok=4 changed=2). The file was now byte-for-byte the repo’s, 124 bytes instead of 126, and an unprivileged user still got root in a user namespace through the copy, while the system unshare stayed blocked:

uid=0(root) gid=0(root) groups=0(root),65534(nogroup)
unshare: write failed /proc/self/uid_map: Operation not permitted

The second run: ok=3 changed=0, no diff and no handler. That is the whole promise of Ansible: once the machine is in the described state, running it again changes nothing.

One thing about that first change: AppArmor does not care about indentation. The policy the kernel loaded was the same before and after; the profile’s entries in /sys/kernel/security/apparmor/ still carried the boot time. Ansible compares text, not meaning. A whitespace-only change is a change, and it fires the handler.

Breaking it on purpose

Next I changed something by hand, on purpose:

sudo chmod 0775 /usr/local/bin/lab0a-unshare

Group-write is not a random choice. Git does not store it, which is why every file task in the role sets mode explicitly. And on this binary it is dangerous: AppArmor lets this one file create user namespaces, and if its group can write to it, any member of that group can replace it and the replacement inherits the exception.

The dry run caught it: the binary task changed, ok=3 changed=1, no handler. But there was no diff. The copy module’s diff covers file content only; when only the owner, group or mode differ, it reports changed and shows nothing else. A changed without a diff from copy means attributes.

The real run fixed it with the same output, and the next run was changed=0 again. On the server, ctime was new and mtime unchanged: Ansible had reset the mode without rewriting the file, because the checksums matched.

A setting that lives in two places

The second fix from Lab 0a is a kernel setting. With kernel.apparmor_restrict_unprivileged_unconfined = 1, an unprivileged user can no longer use aa-exec to slip into a permissive AppArmor profile and get a user namespace with full capabilities, one of the bypasses Qualys published in 2025. I had put it in /etc/sysctl.d/ by hand.

It became a new role, common, rather than a task in lab_tools: lab_tools is meant to be removed when the labs no longer need it, and this protection must stay.

A file in /etc/sysctl.d/ takes effect at boot. The obvious design, writing the file and reloading it from a handler when the file changes, misses one case: someone runs sysctl -w at runtime. The file stays right, Ansible says ok, and the protection is off until the next reboot. With only ansible.builtin there is no sysctl module, so the role does it in three tasks: install the file, read the running value, and load the file when the value differs.

- name: Read the running value of kernel.apparmor_restrict_unprivileged_unconfined
  ansible.builtin.command: sysctl -n kernel.apparmor_restrict_unprivileged_unconfined
  register: common_restrict_unconfined
  changed_when: false
  check_mode: false

- name: Apply the drop-in when the running value differs
  ansible.builtin.command:
    cmd: sysctl -p /etc/sysctl.d/60-apparmor-restrict-unconfined.conf
    # With removes, check mode reports "would run" as changed instead of skipping.
    removes: /etc/sysctl.d/60-apparmor-restrict-unconfined.conf
  when: common_restrict_unconfined.stdout != "1"

check_mode: false lets the read run during a dry run too; it only reads. The removes line is the interesting one. As the handler showed earlier, command in check mode reports skipped whether it would run or not, so a dry run would never show this drift. With creates or removes set, the module can tell, and reports changed when it would run.

I tested exactly that case. On the server:

sudo sysctl -w kernel.apparmor_restrict_unprivileged_unconfined=0

For a minute, the aa-exec bypass from Lab 0a was open again. The dry run reported the apply task as changed (ok=7 changed=1) and left the kernel at 0; the real run loaded the file and set it back to 1; the next run was changed=0. Then the check that matters:

$ aa-exec -p trinity -- unshare --user --map-root-user id -u
unshare: write failed /proc/self/uid_map: Operation not permitted
$ lab0a-unshare --user --map-root-user sh -c 'id -u; grep CapEff /proc/self/status'
0
CapEff:	000001ffffffffff

The detour through a foreign profile is closed; the exception I granted on purpose still works.

Guard rails

• One entry point. site.yml only imports: common.yml (the baseline for every host) and lab0a.yml. ansible-playbook site.yml brings everything to the described state; --limit narrows it to one host.
• Lint from day one. ansible-lint 26.9.0, installed with pipx inject into the same virtualenv, so it lints against the same ansible-core that runs the playbooks. It passed at production, its strictest profile: 0 failures, 0 warnings. It also creates a cache directory, .ansible/, at the repo root, which now sits in .gitignore.

Every run, from the first ping to the last:

What surprised me

1. Python discovery skips the system Python whenever a versioned python3.X is found first; it has to be pinned.
2. A whitespace-only change is a change, and it fires handlers, even when the result is semantically identical.
3. copy shows no diff when only the mode changes.
4. In check mode, command stays silent (skipped) unless creates or removes gives it a way to tell.
5. Homebrew’s pipx runs on a Python that ansible-core does not support yet; the interpreter for Ansible has to be chosen explicitly.

What’s next

The next lab is the one Lab 0a promised: sandboxing Claude Code, the AI coding agent I use in this lab. Its sandbox needs an AppArmor profile and settings on the server, and now they will arrive as a role instead of another manual tweak. Later, these roles get automated tests, and a new node in the homelab becomes one more line in the inventory.

Everything from this post, in full:

• Guide: every command, step by step
• Results: versions, runs and surprises
• Entry point: site.yml
• Role common: the AppArmor sysctl, on disk and in the kernel
• Role lab_tools: the unshare copy and its profile from Lab 0a
• Configuration: ansible.cfg
• Interpreter pin: group_vars/all.yml