feat(roles/bootloader): add role

.ansible-lint-ignore

@@ -8,6 +8,7 @@ roles/at/defaults/main.yml jinja[spacing]
 roles/audit/defaults/main.yml jinja[spacing]
 roles/bind/defaults/main.yml jinja[spacing]
 roles/blocky/defaults/main.yml jinja[spacing]
+roles/bootloader/defaults/main.yml jinja[spacing]
 roles/borg_local/defaults/main.yml jinja[spacing]
 roles/chrony/defaults/main.yml jinja[spacing]
 roles/clamav/defaults/main.yml jinja[spacing]

CHANGELOG.md

@@ -8,7 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
-**Highlights:** On RHEL 8, a MariaDB package upgrade no longer cuts applications on the same host off from their database. Apache no longer loads `mod_info`, which served the complete configuration including other modules' credentials. A broken PHP-FPM configuration aborts the run instead of taking the service down on the restart. Sudo rules deployed by `freeipa_server` can carry their commands again. The Bitwarden lookup can be told to abort instead of silently generating a new password, for runs against hosts whose credentials must already exist. The Grafana graph configuration for the Monitoring Plugins is no longer deployed on every ordinary run and has to be requested explicitly by its tag.
+**Highlights:** Kernel parameters that only take effect at boot time, `psi=1` among them, can be deployed from the inventory with the new `bootloader` role. On RHEL 8, a MariaDB package upgrade no longer cuts applications on the same host off from their database. Apache no longer loads `mod_info`, which served the complete configuration including other modules' credentials. A broken PHP-FPM configuration aborts the run instead of taking the service down on the restart. Sudo rules deployed by `freeipa_server` can carry their commands again. The Bitwarden lookup can be told to abort instead of silently generating a new password, for runs against hosts whose credentials must already exist. The Grafana graph configuration for the Monitoring Plugins is no longer deployed on every ordinary run and has to be requested explicitly by its tag.
 
 ### Breaking Changes
 
@@ -20,6 +20,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+* **role:bootloader**: New role that manages the kernel command line, for parameters that only take effect at boot time such as `psi=1`. Options are applied to every boot entry of the host, on the Red Hat family with `grubby` and on Debian and Ubuntu through a GRUB drop-in of its own. A changed command line requests a reboot at the maintenance window instead of rebooting right away, and a `--check` run reports what it would change without touching the host.
 * **role:files**: A file can opt out of the backup copy that is written before it is overwritten, via the `backup` subkey of `files__files__*_var`.
 * **role:collabora**: The `collabora:configure` tag deploys `coolwsd.xml` and the logrotate configuration without touching the packages.
 * **role:docker**: The address pools docker assigns container network subnets from (`default-address-pools`) can be configured.

COMPATIBILITY.md

@@ -15,6 +15,7 @@ Which Ansible role is proven to run on which OS?
 | audit                                 |        |        |   x    |   x    |   (x)   |           |           |           |                                              |
 | bind                                  |        |        |   x    |   x    |    x    |           |           |           |                                              |
 | blocky                                |        |        |   x    |   x    |   (x)   |           |           |           |                                              |
+| bootloader                            |   x    |   x    |   x    |   x    |    x    |     x     |     x     |     x     | Fedora 44                                    |
 | borg_local                            |        |        |   x    |  (x)   |   (x)   |           |           |           |                                              |
 | chromium_headless                     |   x    |  (x)   |   x    |   x    |    x    |           |           |           |                                              |
 | chrony                                |        |        |   x    |   x    |    x    |           |           |           |                                              |

CONTRIBUTING.md

@@ -219,6 +219,7 @@ When creating a new role, make sure to deliver:
 * Update `playbooks/all.yml`.
 * Update `COMPATIBILITY.md`.
 * Update `CHANGELOG.md`.
+* Update `.ansible-lint-ignore` if the role defines a `__combined_var`.
 
 
 ### OS Coverage

extensions/molecule/bootloader/install/converge.yml

@@ -0,0 +1,2 @@
+- name: 'Converge bootloader playbook'
+  ansible.builtin.import_playbook: 'linuxfabrik.lfops.bootloader'

extensions/molecule/bootloader/install/inventory/group_vars/systems_under_test.yml

@@ -0,0 +1,19 @@
+# The option under test. psi is a good candidate: it takes effect at boot time only, and the
+# kernel reports whether it acted on it (/proc/pressure/cpu), so verify.yml can check the effect
+# instead of just the string.
+bootloader__cmdline_options__group_var:
+  - name: 'psi'
+    value: 1
+
+# Inputs for the roles the bootloader playbook pulls in. mailto_root is mandatory: it needs a
+# from and a list of recipients. A local address is enough, the test never sends real mail.
+mailto_root__from: 'root@localhost'
+mailto_root__to:
+  - 'root@localhost'
+
+# postfix needs a relayhost; an unreachable example host is fine, the test does not deliver mail.
+postfix__relayhost: 'mail.example.com'
+
+# Reboot window far from the test run, so the schedule_reboot timer never fires mid-test. The
+# reboot this scenario needs is performed by verify.yml itself.
+schedule_reboot__reboot_time__group_var: '03:00'

extensions/molecule/bootloader/install/inventory/hosts.yml

@@ -0,0 +1,15 @@
+# yamllint disable rule:empty-values
+lfops_bootloader:
+  children:
+    systems_under_test:
+
+systems_under_test:
+  hosts:
+    debian12-vm:
+    debian13-vm:
+    rocky8-vm:
+    rocky9-vm:
+    rocky10-vm:
+    ubuntu2204-vm:
+    ubuntu2404-vm:
+    ubuntu2604-vm:

extensions/molecule/bootloader/install/molecule.yml

@@ -0,0 +1,10 @@
+# Molecule scenario marker for the 'install' sub-scenario, run with
+# `molecule test --scenario-name bootloader/install`.
+#
+# The prepare playbook is overridden to work around a property of the Rocky 8 cloud image that an
+# installed host does not have; see the scenario-local prepare.yml. It imports the shared
+# vm-prepare.yml first, so the systems under test are still waited for and their facts still
+# gathered.
+provisioner:
+  playbooks:
+    prepare: '${MOLECULE_SCENARIO_DIRECTORY}/prepare.yml'

extensions/molecule/bootloader/install/prepare.yml

@@ -0,0 +1,37 @@
+# Wait for the systems under test and gather their facts, exactly as every other scenario does.
+- name: 'Prepare VMs for Ansible'
+  ansible.builtin.import_playbook: '../../playbooks/vm-prepare.yml'
+
+
+# Bring the EL8 test image into the shape an installed EL8 host has, so this scenario tests the
+# role rather than a property of the cloud image.
+#
+# The GenericCloud image carries both boot paths, so grub2-efi-x64 is installed and ships
+# /boot/grub2/grubenv as a symlink onto the EFI System Partition. These VMs boot BIOS, where GRUB
+# cannot read across into the ESP, which is the combination the role rejects (see the role's
+# "Known Limitations"). An installed BIOS host has a regular file there, so the scenario creates
+# one. EL9 and later neither ship the symlink nor keep the command line in that file.
+- name: 'Make the EL8 grub environment block readable from a BIOS boot'
+  hosts: 'systems_under_test'
+
+  tasks:
+
+    - name: 'stat /boot/grub2/grubenv'
+      ansible.builtin.stat:
+        path: '/boot/grub2/grubenv'
+        follow: false
+      register: '__molecule__grubenv_stat_result'
+      when:
+        - 'ansible_facts["os_family"] == "RedHat"'
+        - 'ansible_facts["distribution_major_version"] | int == 8'
+
+    - name: 'Replace the grubenv symlink with a regular file'
+      ansible.builtin.command:
+        argv:
+          - 'cp'
+          - '--remove-destination'
+          - '{{ __molecule__grubenv_stat_result["stat"]["lnk_source"] | d("") }}'
+          - '/boot/grub2/grubenv'
+      changed_when: true
+      when:
+        - '__molecule__grubenv_stat_result["stat"]["islnk"] | d(false) | bool'

extensions/molecule/bootloader/install/verify.yml

@@ -0,0 +1,92 @@
+# The kernel command line can only be observed on the running kernel, and it only changes on a
+# reboot. The scenario therefore reboots the host once, on the run where the option is still
+# missing from /proc/cmdline, and asserts the running kernel came up with it. verify runs twice
+# (after converge and after idempotence, see the test_sequence in config.yml); on the second run
+# the host already carries the option, so it neither expects a reboot request nor reboots again.
+- name: 'Verify the kernel command line carries the configured option'
+  hosts: 'systems_under_test'
+  tasks:
+
+    # Persistent state, visible before the reboot: on the Red Hat family the role writes the
+    # boot entries with grubby, on the Debian family update-grub regenerates grub.cfg.
+    - name: 'grubby --info=ALL'
+      ansible.builtin.command: 'grubby --info=ALL'
+      register: '__molecule__grubby_info_result'
+      changed_when: false
+      when:
+        - 'ansible_facts["os_family"] == "RedHat"'
+
+    - name: 'Assert that every boot entry has psi=1 on its command line'
+      ansible.builtin.assert:
+        that: '__molecule__grubby_info_result["stdout_lines"]
+            | select("match", "^args=")
+            | reject("search", "psi=1")
+            | list | length == 0'
+      when:
+        - 'ansible_facts["os_family"] == "RedHat"'
+
+    - name: 'grep "^[[:space:]]*linux" /boot/grub/grub.cfg'
+      ansible.builtin.command: 'grep --extended-regexp "^[[:space:]]*linux" /boot/grub/grub.cfg'
+      register: '__molecule__grub_cfg_linux_result'
+      changed_when: false
+      when:
+        - 'ansible_facts["os_family"] == "Debian"'
+
+    - name: 'Assert that every menu entry has psi=1 on its command line'
+      ansible.builtin.assert:
+        that: '__molecule__grub_cfg_linux_result["stdout_lines"]
+            | reject("search", "psi=1")
+            | list | length == 0'
+      when:
+        - 'ansible_facts["os_family"] == "Debian"'
+
+    - name: 'slurp /proc/cmdline'
+      ansible.builtin.slurp:
+        src: '/proc/cmdline'
+      register: '__molecule__proc_cmdline_result'
+
+    # Only true right after converge changed the boot loader configuration. The role requests the
+    # reboot instead of performing it, so the request has to be waiting in the schedule_reboot spool.
+    - name: 'Verify the pending reboot and apply it'
+      when: '"psi=1" not in (__molecule__proc_cmdline_result["content"] | ansible.builtin.b64decode)'
+      block:
+
+        - name: 'stat /run/schedule-reboot/bootloader'
+          ansible.builtin.stat:
+            path: '/run/schedule-reboot/bootloader'
+          register: '__molecule__reboot_request_stat_result'
+
+        - name: 'Assert that the role requested a reboot'
+          ansible.builtin.assert:
+            that: '__molecule__reboot_request_stat_result["stat"]["exists"] | bool'
+
+        - name: 'systemctl reboot'
+          ansible.builtin.reboot:  # yamllint disable-line rule:empty-values
+
+    # Read again, after the reboot above if there was one. This has to sit outside the block: a
+    # `register` on a skipped task overwrites the variable with the skip result, so re-registering
+    # inside the block would leave the assertion below without content on the second verify run,
+    # where the host already carries the option and the block does not run.
+    - name: 'slurp /proc/cmdline'
+      ansible.builtin.slurp:
+        src: '/proc/cmdline'
+      register: '__molecule__proc_cmdline_result'
+
+    - name: 'Assert that the running kernel booted with psi=1'
+      ansible.builtin.assert:
+        that: '"psi=1" in (__molecule__proc_cmdline_result["content"] | ansible.builtin.b64decode)'
+
+    # The option is only worth setting if the kernel acts on it. RHEL 8 ships kernels without
+    # pressure stall information, so this is checked everywhere else.
+    - name: 'slurp /proc/pressure/cpu'
+      ansible.builtin.slurp:
+        src: '/proc/pressure/cpu'
+      register: '__molecule__proc_pressure_cpu_result'
+      when:
+        - 'not (ansible_facts["os_family"] == "RedHat" and ansible_facts["distribution_major_version"] | int < 9)'
+
+    - name: 'Assert that the kernel reports pressure stall information'
+      ansible.builtin.assert:
+        that: '(__molecule__proc_pressure_cpu_result["content"] | ansible.builtin.b64decode) is search("^some ")'
+      when:
+        - 'not (ansible_facts["os_family"] == "RedHat" and ansible_facts["distribution_major_version"] | int < 9)'

extensions/molecule/bootloader/remove/converge.yml

@@ -0,0 +1,2 @@
+- name: 'Converge bootloader playbook'
+  ansible.builtin.import_playbook: 'linuxfabrik.lfops.bootloader'

extensions/molecule/bootloader/remove/inventory/group_vars/systems_under_test.yml

@@ -0,0 +1,17 @@
+# prepare.yml has put the option on the kernel command line, this run removes it again.
+bootloader__cmdline_options__group_var:
+  - name: 'psi'
+    state: 'absent'
+
+# Inputs for the roles the bootloader playbook pulls in. mailto_root is mandatory: it needs a
+# from and a list of recipients. A local address is enough, the test never sends real mail.
+mailto_root__from: 'root@localhost'
+mailto_root__to:
+  - 'root@localhost'
+
+# postfix needs a relayhost; an unreachable example host is fine, the test does not deliver mail.
+postfix__relayhost: 'mail.example.com'
+
+# Reboot window far from the test run, so the schedule_reboot timer never fires mid-test. This
+# scenario never reboots: it asserts on the boot entries, which change right away.
+schedule_reboot__reboot_time__group_var: '03:00'

extensions/molecule/bootloader/remove/inventory/hosts.yml

@@ -0,0 +1,15 @@
+# yamllint disable rule:empty-values
+lfops_bootloader:
+  children:
+    systems_under_test:
+
+systems_under_test:
+  hosts:
+    debian12-vm:
+    debian13-vm:
+    rocky8-vm:
+    rocky9-vm:
+    rocky10-vm:
+    ubuntu2204-vm:
+    ubuntu2404-vm:
+    ubuntu2604-vm:

extensions/molecule/bootloader/remove/molecule.yml

@@ -0,0 +1,11 @@
+# Molecule scenario marker for the 'remove' sub-scenario, run with
+# `molecule test --scenario-name bootloader/remove`.
+#
+# The prepare playbook is overridden so the option is on the kernel command line before converge
+# runs. Without it, converge would remove an option that was never there and every assertion in
+# verify.yml would pass on a host the role never touched. The scenario-local prepare.yml imports
+# the shared vm-prepare.yml first, so the systems under test are still waited for and their facts
+# still gathered.
+provisioner:
+  playbooks:
+    prepare: '${MOLECULE_SCENARIO_DIRECTORY}/prepare.yml'

extensions/molecule/bootloader/remove/prepare.yml

@@ -0,0 +1,53 @@
+# Wait for the systems under test and gather their facts, exactly as every other scenario does.
+- name: 'Prepare VMs for Ansible'
+  ansible.builtin.import_playbook: '../../playbooks/vm-prepare.yml'
+
+
+# Bring the EL8 test image into the shape an installed EL8 host has, so this scenario tests the
+# role rather than a property of the cloud image.
+#
+# The GenericCloud image carries both boot paths, so grub2-efi-x64 is installed and ships
+# /boot/grub2/grubenv as a symlink onto the EFI System Partition. These VMs boot BIOS, where GRUB
+# cannot read across into the ESP, which is the combination the role rejects (see the role's
+# "Known Limitations"). An installed BIOS host has a regular file there, so the scenario creates
+# one. EL9 and later neither ship the symlink nor keep the command line in that file.
+- name: 'Make the EL8 grub environment block readable from a BIOS boot'
+  hosts: 'systems_under_test'
+
+  tasks:
+
+    - name: 'stat /boot/grub2/grubenv'
+      ansible.builtin.stat:
+        path: '/boot/grub2/grubenv'
+        follow: false
+      register: '__molecule__grubenv_stat_result'
+      when:
+        - 'ansible_facts["os_family"] == "RedHat"'
+        - 'ansible_facts["distribution_major_version"] | int == 8'
+
+    - name: 'Replace the grubenv symlink with a regular file'
+      ansible.builtin.command:
+        argv:
+          - 'cp'
+          - '--remove-destination'
+          - '{{ __molecule__grubenv_stat_result["stat"]["lnk_source"] | d("") }}'
+          - '/boot/grub2/grubenv'
+      changed_when: true
+      when:
+        - '__molecule__grubenv_stat_result["stat"]["islnk"] | d(false) | bool'
+
+
+# Put the option on the kernel command line, so the converge run has something to remove. The
+# role is called directly rather than through the playbook: schedule_reboot is deliberately not
+# deployed here, so the role only reports that a reboot would be needed and the boot entries stay
+# the single thing this play changes. The role parameter beats the scenario group_vars, which set
+# the same variable to state: 'absent' for the converge run.
+- name: 'Put psi=1 on the kernel command line'
+  hosts: 'systems_under_test'
+
+  roles:
+
+    - role: 'linuxfabrik.lfops.bootloader'
+      bootloader__cmdline_options__group_var:
+        - name: 'psi'
+          value: 1