fix/k3s registry match deployed - #378
Merged
Merged
Conversation
swares
commented
Aug 9, 2026
swares
commented
Owner
- docs: one backlog, and collapse the five competing lists into pointers
- verify-lab: fix two faults found by the first honest alert-metrics run
- docs: close two resolved items, point HARDWARE.md at the backlog
- docs: close Authelia and sudo items; record the ansible.cfg working-dir trap
- docs: sweep no_log registers; close Authelia and sudo; record the ansible.cfg trap
- ansible: delete dead ldap.yml; sweep no_log registers; close three items
- k3s-registry: match the deployed config, add serial, drop the phantom credential
Open work lived in six places that disagreed with each other. BACKLOG.md is now the only list; README, docs/OVERVIEW.md and docs/services.md point at it, and the three dated TODO-*.md files are marked as historical session notes to be kept for their reasoning rather than their tasks. The disagreements were not cosmetic. Offsite backup was marked DONE in three documents while the unit had never copied a byte -- it existed, but offsite_restic_repo was never set, so it exited 0 nightly and reported PASSED. The Vault root token is described as a required safeguarded credential in three more, after being deliberately eliminated on 08-07. Two files end mid-sentence. A task list that contradicts itself is worse than none, because it gets believed during an incident. Remaining contradictions are catalogued in BACKLOG.md section 6 rather than silently corrected, so each can be checked before anything is deleted. Also fixes two entries that were actively misleading rather than merely stale: CLAUDE.md gave 192.168.1.160 as the *.apps wildcard target when the inventory says .201 and records the 2026-07-27 move away from it; and scripts/verify-lab.py suppressed node_systemd_unit_state on a diagnosis retracted on 08-02, telling the verifier to ignore the metric that the LabBackupUnitFailed alert depends on.
Emptying KNOWN_ABSENT earned its keep immediately. The systemd collectors
reported healthy -- confirming the suppression had been hiding a working metric
-- and the check then failed on two things, one a bug and one a modelling gap.
job_name reported ABSENT. False positive: it is a label in LabBackupJobFailed's
"and on(namespace, job_name)" clause. It carries an underscore and is not
followed by '(', so it passed every filter in _metric_names. Grouping clauses
name labels, not metrics, so on/ignoring/by/without/group_left/group_right
contents are now stripped before identifiers are read. A false positive is
expensive here: it fails the whole check and buries any genuine finding.
kube_job_failed reported ABSENT. Not a fault. kube-state-metrics emits condition
series only for conditions a resource actually carries, so a metric describing
failure has no series while nothing has failed. The alert fires correctly --
proven 2026-08-07 with a deliberately failing Job that paged via ntfy and
resolved on deletion.
That needed a new concept rather than a suppression. ABSENT_WHEN_HEALTHY is kept
separate from KNOWN_ABSENT because the semantics are opposite: KNOWN_ABSENT means
the alert cannot fire, this means it works and the absence is the good outcome.
Merging them would either paint a healthy lab red forever or teach someone to
silence a genuinely broken metric by excusing it. Only add an entry if you have
seen that alert fire.
Authelia stuck Progressing in ArgoCD was fixed in an earlier session, and the leftover PVC has been removed. The item survived only because README.md and docs/HARDWARE.md were never updated -- and in HARDWARE.md it survived as a truncated line, the file ending mid-word at "- investigate(2". A half-written sentence kept a closed issue alive across two audits. bootstrap.yml needing -K for n150-1/n150-2 is also closed: both hosts now have passwordless sudo, and bootstrap.yml:43-44 already codifies it via /etc/sudoers.d/ansible-<user>, so a rebuild reproduces it rather than depending on the change having been made by hand. Also adds a pointer from HARDWARE.md's Pending/TODO section to BACKLOG.md. Its completed entries stay as a hardware changelog, which is useful; its open items do not, because that was the sixth list.
…ir trap Authelia stuck Progressing was fixed in an earlier session and the leftover PVC removed. The item survived only because README.md and docs/HARDWARE.md were never updated -- and in HARDWARE.md it survived as a truncated line, the file ending mid-word at "- investigate(2". A half-written sentence kept a closed issue alive across two audits. HARDWARE.md now points at BACKLOG.md; its completed entries stay as a hardware changelog. bootstrap.yml / n150-1 / n150-2 is closed, but not the way it looked. The --check run did not confirm the codified and hand-made states agreed -- it found /etc/sudoers.d/ansible-swares did not exist on either host, so the working passwordless sudo came from an uncodified edit somewhere else. Both worked, and differed. The hand edit was removed and bootstrap.yml applied, leaving one source of the grant rather than two. Records ansible.cfg only being auto-loaded from the current working directory. Running playbooks from the repo root with an explicit -i means vault_password_file and pipelining silently do nothing. Every playbook run this session was made that way and succeeded only because none needed to decrypt a vaulted variable; bootstrap.yml was the first that did, and failed with "Attempting to decrypt but no vault secrets found". Correctness should not depend on which directory you are standing in. Also defers a decision on the control node's RSA key, which bootstrap.yml pushes to every host alongside the ed25519 keys.
…ible.cfg trap Swept all 12 Ansible files carrying no_log for the shape that broke healthchecks.yml under --check: a register consumed by a later expression, with no_log censoring the failure. Results recorded in BACKLOG so it need not be repeated. Eleven are safe; one is not. ldap.yml:23-28 is the same signature -- command + no_log, with changed_when and failed_when both reading .stdout and .rc, which do not exist when the task skips under --check. It is a write (ldapadd), so the fix is defaults on the register rather than check_mode: false. But the prior question is whether the playbook should exist at all: it installs slapd, and lldap replaced it as a k3s Deployment when ldap-1 was decommissioned on 2026-07-04. Its admin password also falls back to a literal CHANGEME string. Deleting is likely better than repairing. Two generalisations worth keeping: slurp supports check mode and runs, so only command/shell skip; and the robust shape is a guarded consumer, not just check_mode: false on the producer. rotate-passwords.yml and sync-secrets-to-vault.yml already do this correctly and are the models. Also closes two items. Authelia stuck Progressing was fixed in an earlier session and survived only in README.md and a truncated line in docs/HARDWARE.md, which ended mid-word at "- investigate(2". And bootstrap.yml / n150-1 / n150-2 closed after --check revealed the sudoers drop-in did not exist on either host -- the working passwordless sudo came from an uncodified edit, so live and git both worked and differed. The hand edit was removed and the playbook applied. Records that ansible.cfg is only auto-loaded from the current working directory, so running from the repo root with an explicit -i silently disables vault_password_file and pipelining.
Swept all 12 Ansible files carrying no_log for the shape that broke
healthchecks.yml under --check: a register consumed by a later expression, with
no_log censoring the failure. Eleven are safe. Results are recorded in BACKLOG so
the sweep need not be repeated.
The twelfth was ldap.yml, and it is deleted rather than repaired. It targeted
"hosts: ldap" -- a group removed from the inventory on 2026-07-18 when ldap-1 was
decommissioned and lldap moved into k3s -- so it matched no hosts and could only
no-op. But Makefile:54 still offered `make ldap`, and site.yml:14 described it as
"lldap VM provisioning", which it never was: it installed OpenLDAP/slapd. A dead
playbook is harmless right up until someone runs it.
Removing it closed three items together: the --check bug (command + no_log with
changed_when/failed_when reading .stdout and .rc, which do not exist when the task
skips), the admin password falling back to the literal CHANGEME-set-via-vault, and
the dead code itself. Makefile target and site.yml comment removed with it.
Two generalisations from the sweep, recorded for next time: slurp supports check
mode and runs, so only command/shell skip -- k3s-agent.yml and k3s-ha-join.yml
look like the bad pattern and are not. And the robust shape is a guarded consumer
("when: x is defined", or "| default(...)"), not just check_mode: false on the
producer. rotate-passwords.yml and sync-secrets-to-vault.yml already do this and
are the models to copy.
… credential zot_admin_password was referenced at k3s-registry.yml:48 and defined nowhere in the repo, so the play could only fail on an undefined variable -- which means it had never been run. Checking the live nodes explained why nobody noticed: h4-core, opi5pro-2, n150-1, n150-2 registry mirror only, no auth block opi5pro-1 file absent entirely The template was ahead of reality in a risky direction. It also mirrored docker.io and ghcr.io through Zot, which runs inside the cluster it serves -- so a cold cluster, a new node, or a node after image GC would need Zot to start Zot. And the auth block guarded a credential nothing uses: Zot serves the only lab-registry image in production, registry.apps.lab.home.arpa/m5stack-adapter:0.1.1, without authentication. So the template is reduced to match the deployed file rather than minting the password. No rotation, no new secret, and the cold-start dependency is never introduced. Zot as a pull-through cache can be a deliberate change later, with its own rollback story, instead of a side effect of running this playbook. More dangerous than the missing password: the play had no serial, so its "restart k3s" handler would have restarted all three control-plane servers in parallel and lost the embedded etcd quorum. Now serial: 1. make k3s-registry was one working variable away from being a cluster outage. Remaining: run it once so opi5pro-1 gets the file. With the template matching what is deployed, only that node changes, so only its k3s-agent restarts.
swares
added a commit
that referenced
this pull request
Aug 9, 2026
Follow-up to #378, from running --check against the full fleet. mode 0600 -> 0644. Content matched on all four configured nodes; the only diff was the mode. Since the copy notifies "restart k3s", applying it would have rolled every node in the cluster to tighten permissions on a file that holds no credentials. If an auth block is ever added here, tighten it in that same commit and accept the restart then -- permissions are worth a restart when there is a secret to protect, and not before. The Docker play could not succeed and would have been harmful if it had. h4-core runs docker-ce 29.7.1, which depends on containerd.io; the task installed Ubuntu's docker.io, which depends on containerd, and the two conflict: containerd.io : Conflicts: containerd So the task failed every run, and had it succeeded it would have torn out a working Docker install. Now guarded by a docker --version probe. xu3-1 dropped from that play. It runs Ubuntu 16.04, whose Python is too old for ansible-core: every module dies with a SyntaxError on an f-string inside module_utils/basic.py, before any task logic runs. ignore_unreachable does not help because the host is reachable -- the module is what breaks. It is already excluded from Ansible auto-updates; configure it by hand or retire it rather than having every run report a failure. opi5pro-1 has been brought into line separately: file created, k3s-agent restarted, nothing else touched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.