chore/delete dead ldap playbook - #377
Merged
Merged
Conversation
swares
commented
Aug 9, 2026
swares
commented
Owner
- docs: one backlog, and collapse the five competing lists into pointers
- verify-lab: fix two faults found by the first honest alert-metrics run
- docs: close two resolved items, point HARDWARE.md at the backlog
- docs: close Authelia and sudo items; record the ansible.cfg working-dir trap
- docs: sweep no_log registers; close Authelia and sudo; record the ansible.cfg trap
- ansible: delete dead ldap.yml; sweep no_log registers; close three items
Open work lived in six places that disagreed with each other. BACKLOG.md is now the only list; README, docs/OVERVIEW.md and docs/services.md point at it, and the three dated TODO-*.md files are marked as historical session notes to be kept for their reasoning rather than their tasks. The disagreements were not cosmetic. Offsite backup was marked DONE in three documents while the unit had never copied a byte -- it existed, but offsite_restic_repo was never set, so it exited 0 nightly and reported PASSED. The Vault root token is described as a required safeguarded credential in three more, after being deliberately eliminated on 08-07. Two files end mid-sentence. A task list that contradicts itself is worse than none, because it gets believed during an incident. Remaining contradictions are catalogued in BACKLOG.md section 6 rather than silently corrected, so each can be checked before anything is deleted. Also fixes two entries that were actively misleading rather than merely stale: CLAUDE.md gave 192.168.1.160 as the *.apps wildcard target when the inventory says .201 and records the 2026-07-27 move away from it; and scripts/verify-lab.py suppressed node_systemd_unit_state on a diagnosis retracted on 08-02, telling the verifier to ignore the metric that the LabBackupUnitFailed alert depends on.
Emptying KNOWN_ABSENT earned its keep immediately. The systemd collectors
reported healthy -- confirming the suppression had been hiding a working metric
-- and the check then failed on two things, one a bug and one a modelling gap.
job_name reported ABSENT. False positive: it is a label in LabBackupJobFailed's
"and on(namespace, job_name)" clause. It carries an underscore and is not
followed by '(', so it passed every filter in _metric_names. Grouping clauses
name labels, not metrics, so on/ignoring/by/without/group_left/group_right
contents are now stripped before identifiers are read. A false positive is
expensive here: it fails the whole check and buries any genuine finding.
kube_job_failed reported ABSENT. Not a fault. kube-state-metrics emits condition
series only for conditions a resource actually carries, so a metric describing
failure has no series while nothing has failed. The alert fires correctly --
proven 2026-08-07 with a deliberately failing Job that paged via ntfy and
resolved on deletion.
That needed a new concept rather than a suppression. ABSENT_WHEN_HEALTHY is kept
separate from KNOWN_ABSENT because the semantics are opposite: KNOWN_ABSENT means
the alert cannot fire, this means it works and the absence is the good outcome.
Merging them would either paint a healthy lab red forever or teach someone to
silence a genuinely broken metric by excusing it. Only add an entry if you have
seen that alert fire.
Authelia stuck Progressing in ArgoCD was fixed in an earlier session, and the leftover PVC has been removed. The item survived only because README.md and docs/HARDWARE.md were never updated -- and in HARDWARE.md it survived as a truncated line, the file ending mid-word at "- investigate(2". A half-written sentence kept a closed issue alive across two audits. bootstrap.yml needing -K for n150-1/n150-2 is also closed: both hosts now have passwordless sudo, and bootstrap.yml:43-44 already codifies it via /etc/sudoers.d/ansible-<user>, so a rebuild reproduces it rather than depending on the change having been made by hand. Also adds a pointer from HARDWARE.md's Pending/TODO section to BACKLOG.md. Its completed entries stay as a hardware changelog, which is useful; its open items do not, because that was the sixth list.
…ir trap Authelia stuck Progressing was fixed in an earlier session and the leftover PVC removed. The item survived only because README.md and docs/HARDWARE.md were never updated -- and in HARDWARE.md it survived as a truncated line, the file ending mid-word at "- investigate(2". A half-written sentence kept a closed issue alive across two audits. HARDWARE.md now points at BACKLOG.md; its completed entries stay as a hardware changelog. bootstrap.yml / n150-1 / n150-2 is closed, but not the way it looked. The --check run did not confirm the codified and hand-made states agreed -- it found /etc/sudoers.d/ansible-swares did not exist on either host, so the working passwordless sudo came from an uncodified edit somewhere else. Both worked, and differed. The hand edit was removed and bootstrap.yml applied, leaving one source of the grant rather than two. Records ansible.cfg only being auto-loaded from the current working directory. Running playbooks from the repo root with an explicit -i means vault_password_file and pipelining silently do nothing. Every playbook run this session was made that way and succeeded only because none needed to decrypt a vaulted variable; bootstrap.yml was the first that did, and failed with "Attempting to decrypt but no vault secrets found". Correctness should not depend on which directory you are standing in. Also defers a decision on the control node's RSA key, which bootstrap.yml pushes to every host alongside the ed25519 keys.
…ible.cfg trap Swept all 12 Ansible files carrying no_log for the shape that broke healthchecks.yml under --check: a register consumed by a later expression, with no_log censoring the failure. Results recorded in BACKLOG so it need not be repeated. Eleven are safe; one is not. ldap.yml:23-28 is the same signature -- command + no_log, with changed_when and failed_when both reading .stdout and .rc, which do not exist when the task skips under --check. It is a write (ldapadd), so the fix is defaults on the register rather than check_mode: false. But the prior question is whether the playbook should exist at all: it installs slapd, and lldap replaced it as a k3s Deployment when ldap-1 was decommissioned on 2026-07-04. Its admin password also falls back to a literal CHANGEME string. Deleting is likely better than repairing. Two generalisations worth keeping: slurp supports check mode and runs, so only command/shell skip; and the robust shape is a guarded consumer, not just check_mode: false on the producer. rotate-passwords.yml and sync-secrets-to-vault.yml already do this correctly and are the models. Also closes two items. Authelia stuck Progressing was fixed in an earlier session and survived only in README.md and a truncated line in docs/HARDWARE.md, which ended mid-word at "- investigate(2". And bootstrap.yml / n150-1 / n150-2 closed after --check revealed the sudoers drop-in did not exist on either host -- the working passwordless sudo came from an uncodified edit, so live and git both worked and differed. The hand edit was removed and the playbook applied. Records that ansible.cfg is only auto-loaded from the current working directory, so running from the repo root with an explicit -i silently disables vault_password_file and pipelining.
Swept all 12 Ansible files carrying no_log for the shape that broke
healthchecks.yml under --check: a register consumed by a later expression, with
no_log censoring the failure. Eleven are safe. Results are recorded in BACKLOG so
the sweep need not be repeated.
The twelfth was ldap.yml, and it is deleted rather than repaired. It targeted
"hosts: ldap" -- a group removed from the inventory on 2026-07-18 when ldap-1 was
decommissioned and lldap moved into k3s -- so it matched no hosts and could only
no-op. But Makefile:54 still offered `make ldap`, and site.yml:14 described it as
"lldap VM provisioning", which it never was: it installed OpenLDAP/slapd. A dead
playbook is harmless right up until someone runs it.
Removing it closed three items together: the --check bug (command + no_log with
changed_when/failed_when reading .stdout and .rc, which do not exist when the task
skips), the admin password falling back to the literal CHANGEME-set-via-vault, and
the dead code itself. Makefile target and site.yml comment removed with it.
Two generalisations from the sweep, recorded for next time: slurp supports check
mode and runs, so only command/shell skip -- k3s-agent.yml and k3s-ha-join.yml
look like the bad pattern and are not. And the robust shape is a guarded consumer
("when: x is defined", or "| default(...)"), not just check_mode: false on the
producer. rotate-passwords.yml and sync-secrets-to-vault.yml already do this and
are the models to copy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.