diff --git a/.github/workflows/mesh-integration.yml b/.github/workflows/mesh-integration.yml index a8216377..b0eef224 100644 --- a/.github/workflows/mesh-integration.yml +++ b/.github/workflows/mesh-integration.yml @@ -1,6 +1,6 @@ name: Mesh integration -# Brings up three k3s nodes in Docker and joins them through Headscale with the cluster role (tests/mesh/run.sh), once per scenario: headscale_hosted, headscale_in_cluster, and the headscale_existing policy check, plus the role's refusal of a Docker Compose release that recreates the containers it has just built. headscale_in_cluster also runs with github_runner_cluster_docker_context pinned while the current Docker context points at nothing, which covers the cluster role's own docker calls and the recovery playbook's, and context_checks checks the role refuses a pin it cannot honour. regrade turns memory-pressure regrading on and steps a server down to etcd-only under (simulated) pressure and back up, checking that it keeps its etcd seat and vote and that no other server steps meanwhile. native_server joins a native host (a systemd container) as an agent beside three Docker servers, then promotes it to a server in place: it must join etcd as a learner, be promoted to a voter, and keep the cluster writable throughout. rejoin removes the bootstrap Docker server's etcd member and checks the server idles on its tombstone, refuses to start a second cluster from its moved-aside datastore with no server to join, joins the existing cluster from a join URL in its role directory with the same cluster ID, and idles as a server whose role file names the agent. role_controller adds a standby agent, runs the role controller's probe and controller from the image built from this checkout, makes a voter slow with tc netem and pauses another, and checks every probe reports the link medium the cluster role labelled its node with, the controller reports the voter unfit with the swap named, reports that it would not act with a voter paused, and changes nothing. Only on changes that can affect how nodes join the mesh. +# Brings up three k3s nodes in Docker and joins them through Headscale with the cluster role (tests/mesh/run.sh), once per scenario: headscale_hosted, headscale_in_cluster, and the headscale_existing policy check, plus the role's refusal of a Docker Compose release that recreates the containers it has just built. headscale_in_cluster also runs with github_runner_cluster_docker_context pinned while the current Docker context points at nothing, which covers the cluster role's own docker calls and the recovery playbook's, and context_checks checks the role refuses a pin it cannot honour. regrade turns memory-pressure regrading on and steps a server down to etcd-only under (simulated) pressure and back up, checking that it keeps its etcd seat and vote and that no other server steps meanwhile. native_server joins a native host (a systemd container) as an agent beside three Docker servers, then promotes it to a server in place: it must join etcd as a learner, be promoted to a voter, and keep the cluster writable throughout. rejoin removes the bootstrap Docker server's etcd member and checks the server idles on its tombstone, refuses to start a second cluster from its moved-aside datastore with no server to join, joins the existing cluster from a join URL in its role directory with the same cluster ID, and idles as a server whose role file names the agent. role_controller adds a standby agent, runs the role controller's probe and controller from the image built from this checkout, makes a voter slow with tc netem and pauses another, and checks every probe reports the link medium the cluster role labelled its node with, the controller reports the voter unfit with the swap named, reports that it would not act with a voter paused, and changes nothing. role_actuator acts as the role controller by hand on three Docker servers and an eligible native standby: the native host's role actuator promotes it to a voter with writes succeeding throughout, steps it down to etcd-only with no pod's shim left, and once its member is removed through k3s's annotation finishes it as an agent with the same node password and Tailscale address and its datastore moved aside, the two k3s units never running together. Only on changes that can affect how nodes join the mesh. # # Runs on GitHub-hosted ubuntu-latest, whose Docker daemon runs the privileged containers the test needs (k3s, and Tailscale inside it). The work directory sits under runner.temp, which the job always has write access to. @@ -45,6 +45,7 @@ jobs: - {scenario: native_server, compose: v5.5.1} - {scenario: role_controller, compose: v5.5.1} - {scenario: rejoin, compose: v5.5.1} + - {scenario: role_actuator, compose: v5.5.1} - {scenario: hosted, compose: v2.39.0} - {scenario: compose_faulty, compose: v2.38.2} - {scenario: in_cluster, compose: v5.5.1, pin: true} diff --git a/roles/github_runner_cluster/README.md b/roles/github_runner_cluster/README.md index dab16c21..7cf1e69d 100644 --- a/roles/github_runner_cluster/README.md +++ b/roles/github_runner_cluster/README.md @@ -103,7 +103,7 @@ A native server is configured as every Docker server is, because k3s refuses a s **The restart guard.** Native servers take turns, one at a time, as recreated Docker servers do ([Recreating servers one at a time](#recreating-servers-one-at-a-time)). A restart, a first start beside an existing cluster, or a promotion goes ahead only while every other etcd voter is Ready, the voters being read from each node's `node-role.kubernetes.io/etcd` label as the cluster reports it now, never from the inventory. Under `--check` the guard still reads and refuses, and the change is reported, not made. -**Switching role.** The node keeps its name, its node password (`/etc/rancher/node/password`, which neither unit is told to change) and its Tailscale device. Both directions cordon the node, wait up to `github_runner_cluster_native_runner_wait_timeout` seconds for its ARC runner pods to finish, and drain it; a node that cannot be emptied is uncordoned and the run fails with the units as they were, so a planned switch never cuts a job. A promotion then writes `server` to the role file, stops and disables the agent unit and starts the server unit. A demotion is make-before-break: it goes ahead only while at least `github_runner_cluster_native_demotion_min_voters` (3) other voters, every one Ready, remain, so the cluster gains a voter before it loses one (etcd calls a two-member cluster unsafe). It writes `agent` to the role file first, so that neither `Restart=always` nor a reboot can start the server again (k3s marks a removed member's datastore with a tombstone and would rejoin it as a learner), then removes the etcd member through k3s's own controller (`etcd.k3s.cattle.io/remove=true` on the Node through another server, waiting for k3s to record `etcd.k3s.cattle.io/removed-node-name`; `pkg/etcd/member_controller.go`), stops and disables the server unit, moves `server/db` aside to `server/db.demoted` (replacing an older copy, and keeping the snapshots in it), because k3s reuses any datastore it finds and would start the old member again on a later promotion, takes the server role labels and k3s's etcd annotations off the Node, and starts the agent unit. Moving votes between hosts on its own is not this role's: it only makes a switch safe when the inventory or the role file asks for one. +**Switching role.** The node keeps its name, its node password (`/etc/rancher/node/password`, which neither unit is told to change) and its Tailscale device. Both directions cordon the node, wait up to `github_runner_cluster_native_runner_wait_timeout` seconds for its ARC runner pods to finish, and drain it; a node that cannot be emptied is uncordoned and the run fails with the units as they were, so a planned switch never cuts a job. A promotion then writes `server` to the role file, stops and disables the agent unit and starts the server unit, through the role actuator's host-local promotion (`fleet-role-actuator promote-local`, see [Role actuator](#role-actuator)), so that the role and the controller's swaps switch a host one way. A demotion is make-before-break: it goes ahead only while at least `github_runner_cluster_native_demotion_min_voters` (3) other voters, every one Ready, remain, so the cluster gains a voter before it loses one (etcd calls a two-member cluster unsafe). It writes `agent` to the role file first, so that neither `Restart=always` nor a reboot can start the server again (k3s marks a removed member's datastore with a tombstone and would rejoin it as a learner), then removes the etcd member through k3s's own controller (`etcd.k3s.cattle.io/remove=true` on the Node through another server, waiting for k3s to record `etcd.k3s.cattle.io/removed-node-name`; `pkg/etcd/member_controller.go`), then, through the role actuator's host-local demotion (`fleet-role-actuator demote-local`), stops and disables the server unit, forgets its regrading level, moves `server/db` aside to `server/db.demoted` (replacing an older copy, and keeping the snapshots in it), because k3s reuses any datastore it finds and would start the old member again on a later promotion, and starts the agent unit, and then takes the server role labels and k3s's etcd annotations off the Node, which the agent's k3s puts none of back. Moving votes between hosts on its own is not this role's: it only makes a switch safe when the inventory or the role file asks for one. **Firewall.** The role opens nothing. Every peer reaches a native server over `tailscale0`; a host whose firewall limits the API (`github_runner_cluster_api_port`), etcd (2379 and 2380) and kubelet (10250) ports to `tailscale0` and loopback keeps them off the LAN. @@ -311,7 +311,7 @@ The decision logic lives in `files/regrade/fleet-regrade-core.sh` and calls only ## Role controller -`github_runner_cluster_role_controller: true` turns on the ARC role's role controller, which measures which hosts are fit to hold an etcd vote and reports the swap of an unfit voter for a fit standby that it would make (see the ARC role's README, Role controller). It reports and changes nothing. It is off by default. This role holds the controller's inventory: +`github_runner_cluster_role_controller: true` turns on the ARC role's role controller, which measures which hosts are fit to hold an etcd vote and reports the swap of an unfit voter for a fit standby that it would make (see the ARC role's README, Role controller). It reports and changes nothing. On a native server or eligible native host the role also runs the role actuator that a swap switches the host through ([Role actuator](#role-actuator)). It is off by default. This role holds the controller's inventory: - `github_runner_cluster_voter_eligible` (per host, default false) marks a host the controller may promote to a voter or demote from one. The role prepares no host for promotion by marking it, so mark only a host that is, or has been staged as, a server. - `github_runner_cluster_voter_pinned` (per host, default false) marks a voter the controller must never demote. Set each pin in the inventory with a comment saying why. @@ -335,6 +335,27 @@ Whatever the directory holds, a server also idles while its datastore holds a to A server never bootstraps a second cluster from a former member's host: with no join URL from either source, it passes `--cluster-init` when it has a datastore (`server/db`, which k3s reuses, ignoring the flag) or none at all (a first bootstrap), and when its datastore was moved aside by a demotion (`server/db.demoted` and no `server/db`) it does not start k3s and exits with a message, and its restart policy repeats the refusal until `join-url` names a current server. An idle service ends on `docker stop`'s signal like any other; idling never exits by itself, since the restart policy would start it again. +### Role actuator + +On a native host the controller may switch or take a snapshot on, a server or a host marked `github_runner_cluster_voter_eligible`, the role runs the role actuator's watcher while `github_runner_cluster_role_controller` is true: the `fleet-role-actuator` unit, which runs `files/role/fleet-role-actuator.sh watch` as root from `/usr/local/lib/github-runner-k3s/role` (`tasks/role_actuator.yml`). It switches the host between its server and agent units when the controller asks through the host's Node, and does nothing on the API but read: the controller cordons, waits for runner pods, drains, taints, labels the level, removes the etcd member, cleans the Node's labels and annotations and uncordons, because an agent host has no credential that can evict pods and its own k3s is down while it switches. Turning the controller off, or a host's eligibility, stops and removes the unit, its kubeconfig and its join URLs; the scripts stay on every native host, since the role's own promotion and demotion run them. + +**What the controller writes on the Node**, all under `github-runner.exadev.com/`: the label `desired-role` (`server` or `agent`), and the annotations `role-swap` (the swap's id, `--`), `role-step` (`promote`, `step-down` or `rollback`), `role-join-node` (the node of the voter a promotion joins) and `snapshot-request` (a swap whose snapshot a remaining voter is to take). The watcher reads its Node at the pace regrading waits for a restarted server at (`POLL_SECONDS` in `files/regrade/fleet-regrade-k3s.sh`) and takes each step once: a step reported `done`, `failed` or `refused` for the swap is not taken again (what follows a failure is the controller's to decide), and a step whose `desired-role` does not go with it is refused. Finishing a demotion is taken until it is done. + +**What it reports**, in the ConfigMap `fleet-role-actuators` in the platform namespace (the ARC role's `github_runner_arc_platform_namespace`), under the node's name, as JSON: `{"swap", "step", "phase", "role", "started_at", "at", "message", "snapshot"}`, `phase` being `running`, `done`, `failed` or `refused`, `role` what the role file names afterwards and `started_at` the host's time just before a unit was started, which the controller waits for a later heartbeat than. A report the API does not take is made again on the next pass, since every step can be taken again. + +**Authority.** `promote`, `step-down`, `rollback` and `snapshot` are taken only while the regrade Lease (`kube-system/fleet-regrade`) is held, unexpired, by `fleet-role` with the swap's id in its `github-runner.exadev.com/swap-id` annotation (see [Runs during a voter swap](#runs-during-a-voter-swap)). Finishing a demotion needs no Lease: it runs once the role file names the agent and the member is gone (a tombstone in the datastore, `etcd.k3s.cattle.io/removed-node-name` naming the datastore's member, or no etcd label on the Node), which can never cost quorum, so a host that was down while its member was removed finishes when it returns. + +**The steps.** + +- `promote`: refused unless the host is provisioned to be a server (its `server.yaml` is there) and holds no datastore a demotion did not move aside. It writes `server` to the role file, writes `server.yaml.d/20-fleet-role-join.yaml` (`server:` the join node's URL, from the `join-urls` file the role writes for every other server and eligible host, and `cluster-init: false`) and a persistent drop-in for the server unit setting `GITHUB_RUNNER_JOIN_HOST` to that URL's host, since the Tailscale wrapper otherwise waits for the inventory's join host, which may by then be an agent, then stops and disables the agent unit and enables and starts the server unit. +- `step-down`: takes the running server to `etcd-only` through regrading's own level (`write_level` and `restart_into` from `files/regrade/fleet-regrade-k3s.sh`) and stops the pods left without a kubelet (`stop_pod_shims`), then writes `agent` to the role file, before the controller asks for the member's removal, so that a restart of the server unit afterwards (its `Restart=always`, a reboot) cannot rejoin the cluster as a new learner, and stops the `fleet-regrade` unit, whose own condition is now the server unit's (`fleet-role-is server`), so that its `Restart=always` cannot start a watcher that would step the server back up. +- `finish-demote` and `rollback`: write `agent` to the role file, stop and disable the server unit, forget its regrading level, move `server/db` aside to `server/db.demoted` (one copy), remove the join drop-ins and enable and start the agent unit. A rollback is the controller's for a candidate that did not become a voter. +- `snapshot`: `k3s etcd-snapshot save --name fleet-role-` through the server's own supervisor, with the server's token from its data directory, reporting the names of the files k3s created. + +A switch the role makes and one the actuator makes never run together: both hold the host's switch lock (`flock` on `/var/lib/rancher/k3s/fleet-role.lock`). `fleet-role-actuator` is on the PATH as root for a step by hand (`promote ID`, `step-down ID`, `rollback ID`, `snapshot ID`, `finish-demote`) and for `status`, which prints the role file, both units, the datastore, the joins and the last report. Its exit status is 0 when the step is done, 1 when it failed (the host is left with at most one role's unit running and a role file naming that role), 2 when it was refused with nothing changed (the Lease, a host that is not provisioned, a role file that names no role, a step it does not take, or the step's own precondition, each named), and 3 when a demotion cannot be finished yet. + +**Its credential.** The role applies, through another server's API, the ServiceAccount `fleet-role-actuator` in the platform namespace with a ClusterRole that can `get`, `list` and `watch` Nodes, a Role that can `get` the regrade Lease by name, a Role that can `get` and `patch` the report ConfigMap by name, and a token Secret the token controller fills (`templates/role-actuator-rbac.yaml.j2`); it creates the ConfigMap when there is none, and writes a kubeconfig holding the token to `/etc/rancher/k3s/fleet-role-actuator.kubeconfig`, mode 0600, by a task that logs nothing. The watcher reaches the cluster through the first of the other servers and eligible hosts whose `/readyz` answers (`KUBE_SERVERS`, as regrading does), never through its own host's API. It needs `jq`, `ps` and `flock`, which the role installs on Debian and Ubuntu when the host lacks them and refuses the run for otherwise. + ## Networking Every node's k3s uses its mesh address for everything: its node IP, etcd's peer URLs, the API server and pod traffic. Which physical path the mesh takes between two nodes is Tailscale's choice, among the endpoints each node's Tailscale client can offer its peers. On Compose's default bridge network (`github_runner_cluster_network_mode: bridge`, the default) the client sits behind the Docker host's NAT and, inside a Docker VM, behind the VM's too, so it has no address its peers can reach and every pair of nodes is relayed, even on one LAN. @@ -466,6 +487,7 @@ The role controller holds the regrade Lease (`kube-system/fleet-regrade`, the Le - `github_runner_cluster_native_tailscale_install` (default `true`): whether the role installs the pinned Tailscale package from Tailscale's apt repository (Debian and Ubuntu). `false` leaves the host's client as it is and only checks its login. - `github_runner_cluster_native_tailscale_repository_url` (default `https://pkgs.tailscale.com/stable`): where that repository's signing key and list file are fetched from, when the host has none. - `github_runner_cluster_native_tailscale_leave_other_tailnet` (default `false`): whether the role logs a client out of another tailnet so that k3s can join the cluster's; `false` stops the run instead. +- `github_runner_cluster_role_actuator_dir`, `github_runner_cluster_role_actuator_service`, `github_runner_cluster_role_actuator_name`, `github_runner_cluster_role_actuator_configmap`, `github_runner_cluster_role_actuator_kubeconfig`, `github_runner_cluster_role_actuator_namespace` and `github_runner_cluster_role_actuator_commands` are derived: where the role actuator's scripts and settings go, its unit, its ServiceAccount, its report ConfigMap, its kubeconfig, the platform namespace it reports in and the commands its watcher needs with the package of each (see [Role actuator](#role-actuator)); `github_runner_cluster_api_candidates` is derived too, the other servers and eligible hosts whose API the regrade service and the role actuator reach the cluster through. - `github_runner_cluster_docker_role_file`, `github_runner_cluster_role_lease_holder` and `github_runner_cluster_role_lease_swap_id_annotation` are derived: the file that records an eligible Docker host's role, and the role controller's holder name on the regrade Lease and the annotation naming its swap (see [A Docker host's recorded role](#a-docker-hosts-recorded-role) and [Runs during a voter swap](#runs-during-a-voter-swap)). - `github_runner_cluster_native_agent_service`, `github_runner_cluster_native_server_service`, `github_runner_cluster_native_service`, `github_runner_cluster_native_join_url`, `github_runner_cluster_native_join_host`, `github_runner_cluster_native_role_file` and `github_runner_cluster_native_restart_marker` are derived: the units' names (`k3s-agent` and `k3s`), the unit of the host's role, the URL the node joins and the host in it, which the Tailscale wrapper waits for, the role file both units' conditions read, and the prefix of the per-unit file that says a restart is waiting; `github_runner_cluster_native_regrade_level_file`, `github_runner_cluster_native_regrade_started_file`, `github_runner_cluster_native_regrade_dir`, `github_runner_cluster_native_regrade_service`, `github_runner_cluster_native_regrade_commands`, `github_runner_cluster_native_agent_runtime_dir` and `github_runner_cluster_native_server_runtime_dir` are derived too: a native server's requested and started regrading levels, where its regrade scripts and settings go, the unit that runs them, the commands regrading needs with the package of each, and each unit's own runtime directory under `/run` (one per unit, since systemd removes a unit's runtime directory when its condition skips a start, which on a shared one would take the running unit's resolver file). `github_runner_cluster_native_drain_for` is derived as well: what `tasks/native/drain.yml` says was not done when it cannot empty the node, a role switch's by default, which a Colima VM resize passes as its own. `github_runner_cluster_native_bin_dir`, `github_runner_cluster_native_lib_dir`, `github_runner_cluster_native_config_dir`, `github_runner_cluster_native_unit_dir`, `github_runner_cluster_native_data_dir`, `github_runner_cluster_native_tailscale_cli`, `github_runner_cluster_native_apt_keyring`, `github_runner_cluster_native_apt_source`, `github_runner_cluster_native_run_dir` and `github_runner_cluster_native_psi_file` are where the native node's files live, which are not settings, and are variables only so the tests can point them at a stand-in. @@ -517,7 +539,7 @@ or read from 1Password by adding `exadev.github_runner.github_runner_secrets_one `tests/unit/test_cluster_wait_nodes.py` runs the wait for nodes to be Ready and its rescue against a stand-in for `kubectl`: a node outside the inventory that is not Ready is reported and never waited for, and an inventory node that stays not Ready still fails the run. The mesh test does not cover it, since its nodes publish no API port for the host's `kubectl` to reach. -`tests/unit/test_node_entrypoint.py` runs the entrypoint's guards in a shell with a stand-in for `sleep` ([The role directory and a former member](#the-role-directory-and-a-former-member)): the join URL's precedence, `--cluster-init` with a datastore, with one moved aside and with none, the idling of a service its role file does not name and of a server on a tombstone, a host with no role file starting as before, and the override that mounts the directory. `tests/unit/test_native_server.py` runs the native server unit's condition on a tombstone. The `rejoin` mesh scenario does the same with real k3s and etcd: it removes the bootstrap Docker server's member through another server and checks the restarted server idles on its tombstone, that with its datastore moved aside and nothing to join it refuses to start, that a `join-url` naming another server makes it join the existing cluster as a learner that is promoted, with the same etcd cluster ID before and after, and that a role file naming the agent leaves the server container running nothing but its idle wait. +`tests/unit/test_node_entrypoint.py` runs the entrypoint's guards in a shell with a stand-in for `sleep` ([The role directory and a former member](#the-role-directory-and-a-former-member)): the join URL's precedence, `--cluster-init` with a datastore, with one moved aside and with none, the idling of a service its role file does not name and of a server on a tombstone, a host with no role file starting as before, and the override that mounts the directory. `tests/unit/test_native_server.py` runs the native server unit's condition on a tombstone. `tests/unit/test_role_actuator.py` runs the [role actuator](#role-actuator) against stand-ins for `systemctl`, `kubectl` and `k3s` (`tests/role_actuator/`, over regrading's own in `tests/regrade/`) and the real `files/native/fleet-role-is.sh` as both units' condition: each step's success, failure, refusal (no Lease, a Lease for another swap, a host that is not provisioned, a role file that names no role, a desired role that does not go with the step) and report, a demotion finished without the Lease once a tombstone, k3s's removal annotation or a missing etcd label says the member is gone and not before, one moved-aside datastore kept, the join drop-ins written and removed, the switch lock, `status`, the watcher taking a swap through each step once, and, through the role, the watcher's unit, settings and kubeconfig on an eligible host, its credential scoped to reading Nodes and the Lease and patching its report, its token kept out of everything the role prints, nothing set up on a host the controller may not switch or under `--check`, and the regrade unit's condition. The `role_actuator` mesh scenario acts as the controller by hand against real k3s and etcd: the native standby's actuator promotes it (a learner, then a voter, with writes succeeding throughout), steps it down (etcd serving at etcd-only, no pod's shim left), and once its member is removed through k3s's annotation finishes it as an agent with the same node password and Tailscale address and its datastore moved aside, the two k3s units never running together. The `rejoin` mesh scenario does the same with real k3s and etcd: it removes the bootstrap Docker server's member through another server and checks the restarted server idles on its tombstone, that with its datastore moved aside and nothing to join it refuses to start, that a `join-url` naming another server makes it join the existing cluster as a learner that is promoted, with the same etcd cluster ID before and after, and that a role file naming the agent leaves the server container running nothing but its idle wait. `tests/unit/test_docker_role.py` runs the Docker host's role decision, its record and the guard for a voter swap through the role against the stand-ins for `docker` and the Compose module ([A Docker host's recorded role](#a-docker-hosts-recorded-role), [Runs during a voter swap](#runs-during-a-voter-swap)): an eligible host keeps its recorded role over the inventory's, the role is recorded only when absent and only for an eligible host, the servers' turns and the regrade service follow it, a host whose containers or record differ from the role it is to run is refused, a run is refused while the controller holds the Lease and goes on while regrading holds it or it has run out, `--check` reports the same refusals, and the one `.env` gives each service the configuration hash it had before. `tests/unit/test_native_node.py` runs the same guard on a native host. diff --git a/roles/github_runner_cluster/defaults/main.yml b/roles/github_runner_cluster/defaults/main.yml index 15390852..08c17a2b 100644 --- a/roles/github_runner_cluster/defaults/main.yml +++ b/roles/github_runner_cluster/defaults/main.yml @@ -207,6 +207,21 @@ github_runner_cluster_docker_role_file: "{{ github_runner_cluster_dir }}/fleet-r # Derived, never set directly: the holderIdentity the role controller takes the regrade Lease (github_runner_cluster_regrade_lease_namespace and github_runner_cluster_regrade_lease_name) as for the whole of a voter swap, and the annotation on it that names the swap. While the controller holds it unexpired, every run of the role refuses before it changes a node (tasks/role_lease_guard.yml). github_runner_cluster_role_lease_holder: fleet-role github_runner_cluster_role_lease_swap_id_annotation: github-runner.exadev.com/swap-id +# Derived, never set directly: the other hosts of the inventory whose API this host's cluster-side helpers (the regrade service, the role actuator) may reach the cluster through, as the topology describes them: every other server, then every other host that may become one (github_runner_cluster_voter_eligible), each in inventory order. Never this host: a server stepped down to etcd-only has no API of its own, and the servers change at run time, so a helper uses the first of them whose /readyz answers, by the address the servers' certificates carry. +github_runner_cluster_api_candidates: >- + {{ (github_runner_cluster_topology.hosts | dict2items | rejectattr('key', 'equalto', inventory_hostname) | selectattr('value.role', 'equalto', 'server') | list) + + (github_runner_cluster_topology.hosts | dict2items | rejectattr('key', 'equalto', inventory_hostname) | rejectattr('value.role', 'equalto', 'server') + | selectattr('value.eligible') | list) }} +# Derived, never set directly: the role actuator (files/role/fleet-role-actuator.sh), which switches a native host between its server and agent units for the role controller's voter swap, as the host's Node asks, and holds the host-local part of the role's own promotion and demotion. Its scripts and settings go in github_runner_cluster_role_actuator_dir on every native host; its watcher, the github_runner_cluster_role_actuator_service unit, runs on every server and every eligible host while github_runner_cluster_role_controller is true (tasks/role_actuator.yml), with a credential of its own (the ServiceAccount github_runner_cluster_role_actuator_name, which can read Nodes and the regrade Lease and patch its report, the ConfigMap github_runner_cluster_role_actuator_configmap in github_runner_cluster_role_actuator_namespace) in a kubeconfig only root reads. +github_runner_cluster_role_actuator_dir: "{{ github_runner_cluster_native_lib_dir }}/role" +github_runner_cluster_role_actuator_service: fleet-role-actuator +github_runner_cluster_role_actuator_name: fleet-role-actuator +github_runner_cluster_role_actuator_configmap: fleet-role-actuators +github_runner_cluster_role_actuator_kubeconfig: "{{ github_runner_cluster_native_config_dir }}/fleet-role-actuator.kubeconfig" +# The ARC role's platform namespace, where the role controller runs and reads the actuators' reports: its github_runner_arc_platform_namespace where the inventory sets that, and that variable's default otherwise (tests/unit/test_role_actuator.py holds the two together). +github_runner_cluster_role_actuator_namespace: "{{ github_runner_arc_platform_namespace | default('github-runner-platform') }}" +# Not a setting: the commands the actuator's watcher runs that a minimal host may lack, each with the Debian package that installs it. jq reads the cluster's answers, ps finds the pods left at etcd-only (as the regrade service does) and flock holds the host's switch lock. +github_runner_cluster_role_actuator_commands: {jq: jq, ps: procps, flock: util-linux} # The node's k3s container as this host runs it: the server's, or the agent's, by the role tasks/docker_role.yml decided this host runs. Derived, never set directly. github_runner_cluster_node_container: "{{ github_runner_cluster_container_name if github_runner_cluster_docker_role == 'server' else github_runner_cluster_agent_container_name }}" # Derived, never set directly: the Compose service and the named volume that holds /etc/rancher/node (the node's password) for this host's role, as docker-compose.yml names them, and the volume's name in Docker, which Compose forms from the project and the volume's key. The role seeds the volume from the running container before the first recreate that mounts it (tasks/node_password.yml). diff --git a/roles/github_runner_cluster/files/role/fleet-role-actuator-systemd.sh b/roles/github_runner_cluster/files/role/fleet-role-actuator-systemd.sh new file mode 100644 index 00000000..81170313 --- /dev/null +++ b/roles/github_runner_cluster/files/role/fleet-role-actuator-systemd.sh @@ -0,0 +1,199 @@ +# The systemd platform of fleet-role-actuator, for a native host (github_runner_cluster_node_mode: native), whose k3s runs on the host as one of two systemd units, the server's K3S_SERVICE or the agent's K3S_AGENT_SERVICE, each started only while the role file ROLE_FILE names its role (files/native/fleet-role-is.sh, the units' ExecCondition). fleet-role-actuator.sh sources fleet-regrade-core.sh, fleet-regrade-k3s.sh and fleet-regrade-systemd.sh ahead of this file, which supply kubectl_, the Lease reader, write_level, restart_into, current_level and stop_pod_shims, and then defines the commands, the authority rules and the reports, which call only the functions below; a platform that runs k3s elsewhere (a container under Compose) supplies the same functions. +# +# THE PLATFORM CONTRACT. Each function returns non-zero when it could not do its part, and leaves the host as the part found it or as far as it got; none prints anything but its result, save take_snapshot, which logs what k3s said when the snapshot failed. +# +# read_role prints the role ROLE_FILE names, or nothing when there is none. +# +# write_role ROLE makes ROLE_FILE name ROLE, in one rename, so a unit's condition never reads half a name. +# +# provisioned succeeds when the host has the server's configuration a promotion starts k3s with. +# +# server_active succeeds while the server's k3s runs, and agent_active while the agent's does. +# +# datastore_present succeeds while the server's datastore (SERVER_DIR/db) is in place. +# +# datastore_tombstoned succeeds when k3s has written the tombstone that says etcd removed this server's member. +# +# datastore_member prints the name of the etcd member the datastore holds, or nothing. +# +# datastore_moved_aside succeeds while a datastore a demotion moved aside is kept. +# +# join_set URL makes the server join URL when it starts, whatever the configuration the role wrote says, and makes the unit wait for that URL's host on the mesh. +# +# join_clear takes join_set's files away again. +# +# join_current prints the URL join_set last set, or says that the join configuration the role wrote decides. +# +# start_server_unit stops and disables the agent's unit, then enables and starts the server's, setting STARTED_AT to the time just before the start. +# +# stop_server_unit stops and disables the server's unit. +# +# start_agent_unit enables and starts the agent's unit, setting STARTED_AT to the time just before the start. +# +# last_start prints the STARTED_AT of the last start either function made, so a step taken again reports the time of the start it found. +# +# host_now prints the time by the host's clock, as RFC 3339 in UTC to the second. +# +# move_datastore_aside moves SERVER_DIR/db to SERVER_DIR/db.demoted, replacing an older copy, when the datastore is in place. +# +# forget_level removes the regrading level the server was stepped to. +# +# enter_etcd_only takes the running server to the regrading level etcd-only and leaves nothing of its pods running. +# +# stop_regrade stops the fleet-regrade watcher, when it runs. +# +# take_snapshot NAME takes an on-demand etcd snapshot named NAME through the server's own supervisor and prints the names k3s gave the files it created, one per line. +# +# lock_switch / unlock_switch hold and release the host's role switch lock, so that a switch the role makes (tasks/native/demote.yml, server_turn.yml) and one the actuator makes never run together. +# +# It needs systemctl, flock and date, besides what fleet-regrade-systemd.sh needs, and the settings named below, which the role writes to the actuator's environment file (templates/role-actuator.env.j2). + +: "${ROLE_FILE:?ROLE_FILE must be set}" "${CONFIG_DIR:?CONFIG_DIR must be set}" "${UNIT_DIR:?UNIT_DIR must be set}" +: "${K3S_SERVICE:?K3S_SERVICE must be set}" "${K3S_AGENT_SERVICE:?K3S_AGENT_SERVICE must be set}" "${REGRADE_SERVICE:?REGRADE_SERVICE must be set}" + +SERVER_DIR=$K3S_DATA_DIR/server +# The drop-ins join_set writes. The server reads every *.yaml in its configuration's drop-in directory after server.yaml, in name order (dotDFiles in pkg/configfilearg/parser.go of k3s), so this one, named after the role's 10-join.yaml and before the 50-regrade.yaml write-regrade-level.sh writes, replaces 10-join.yaml's keys. systemd reads a unit's drop-ins after the unit file, so the Environment= of this one replaces the unit's own GITHUB_RUNNER_JOIN_HOST, which files/native/tailscale-up-wait.sh waits for. +JOIN_CONFIG=$CONFIG_DIR/server.yaml.d/20-fleet-role-join.yaml +JOIN_UNIT_DROPIN=$UNIT_DIR/$K3S_SERVICE.service.d/20-fleet-role-join.conf +SWITCH_LOCK=$K3S_DATA_DIR/fleet-role.lock +LAST_START_FILE=$K3S_DATA_DIR/fleet-role-started-at + +read_role() { + cat "$ROLE_FILE" 2> /dev/null || true +} + +write_role() { + printf '%s' "$1" > "$ROLE_FILE.new" && mv "$ROLE_FILE.new" "$ROLE_FILE" +} + +provisioned() { + [ -f "$CONFIG_DIR/server.yaml" ] +} + +server_active() { + systemctl is-active --quiet "$K3S_SERVICE" +} + +agent_active() { + systemctl is-active --quiet "$K3S_AGENT_SERVICE" +} + +datastore_present() { + [ -d "$SERVER_DIR/db" ] +} + +datastore_tombstoned() { + [ -e "$SERVER_DIR/db/etcd/tombstone" ] +} + +datastore_member() { + cat "$SERVER_DIR/db/etcd/name" 2> /dev/null || true +} + +datastore_moved_aside() { + [ -d "$SERVER_DIR/db.demoted" ] +} + +# The time by the host's clock, which the kubelet stamps its heartbeat with, so a heartbeat later than it is from the k3s started after it. +host_now() { + date -u +%Y-%m-%dT%H:%M:%SZ +} + +# Writes $2, and a line break, to the file $1 in one rename when the file does not already hold exactly that, and sets WROTE to 1 when it wrote and to nothing when it did not, so a caller knows whether systemd must reload. +write_if_changed() { + local file=$1 content=$2 + WROTE='' + [ "$(cat "$file" 2> /dev/null)" != "$content" ] || return 0 + mkdir -p "$(dirname "$file")" && printf '%s\n' "$content" > "$file.new" && mv "$file.new" "$file" || return 1 + WROTE=1 +} + +join_set() { + local url=$1 host + host=${url#*://} + host=${host%%[:/]*} + # cluster-init: false as well as server, since 10-join.yaml says cluster-init: true on a bootstrap host that was never a member, and a joining server must not start a cluster of its own. + write_if_changed "$JOIN_CONFIG" "$(printf '# Written by fleet-role-actuator for a promotion: the voter this server joins, whatever 10-join.yaml says.\ncluster-init: false\nserver: "%s"' "$url")" || return 1 + write_if_changed "$JOIN_UNIT_DROPIN" "$(printf '# Written by fleet-role-actuator for a promotion: the host files/native/tailscale-up-wait.sh waits for, the host of the URL in %s.\n[Service]\nEnvironment="GITHUB_RUNNER_JOIN_HOST=%s"' "$JOIN_CONFIG" "$host")" || return 1 + [ -z "$WROTE" ] || systemctl daemon-reload +} + +join_current() { + local url + url=$(sed -n 's/^server: "\(.*\)"$/\1/p' "$JOIN_CONFIG" 2> /dev/null || true) + if [ -n "$url" ]; then echo "$url"; else echo "the join configuration the role wrote"; fi +} + +join_clear() { + local reload='' + [ ! -e "$JOIN_UNIT_DROPIN" ] || reload=1 + rm -f "$JOIN_CONFIG" "$JOIN_UNIT_DROPIN" || return 1 + [ -z "$reload" ] || systemctl daemon-reload +} + +start_server_unit() { + systemctl disable --now "$K3S_AGENT_SERVICE" || return 1 + record_start || return 1 + systemctl enable --now "$K3S_SERVICE" +} + +stop_server_unit() { + systemctl disable --now "$K3S_SERVICE" +} + +start_agent_unit() { + record_start || return 1 + systemctl enable --now "$K3S_AGENT_SERVICE" +} + +record_start() { + STARTED_AT=$(host_now) + printf '%s\n' "$STARTED_AT" > "$LAST_START_FILE" +} + +last_start() { + cat "$LAST_START_FILE" 2> /dev/null || true +} + +# A copy, not a deletion: it holds the snapshots this server took (server/db/snapshots). One copy only, the one from the latest demotion, so repeated switches do not fill the disk. +move_datastore_aside() { + datastore_present || return 0 + rm -rf "$SERVER_DIR/db.demoted" && mv "$SERVER_DIR/db" "$SERVER_DIR/db.demoted" +} + +# The level is a server's, and the agent unit never reads it; a later promotion must start the server whole, not at a level it was stepped to in an earlier term as a server. +forget_level() { + rm -f "$LEVEL_FILE" +} + +# k3s at etcd-only starts no kubelet and so no containerd to stop the pods that survived the drain; their shims are stopped as fleet-regrade-systemd.sh's apply_level stops them. +enter_etcd_only() { + write_level "$LEVEL_FILE" etcd-only || return 1 + restart_into etcd-only || return 1 + stop_pod_shims +} + +stop_regrade() { + systemctl is-active --quiet "$REGRADE_SERVICE" || return 0 + systemctl stop "$REGRADE_SERVICE" +} + +# k3s etcd-snapshot save asks the server's supervisor (--etcd-server, this host's own API port) with the server's token, which it reads from its data directory when none is given (commandSetup in pkg/cli/etcdsnapshot/etcd_snapshot.go of k3s), and logs "Snapshot saved." for each file the server created; the token is never on the command line. +take_snapshot() { + local output + output=$(k3s etcd-snapshot save --data-dir "$K3S_DATA_DIR" --etcd-server "https://127.0.0.1:${API_PORT:?API_PORT must be set}" --name "$1" 2>&1) || { + log "$output" >&2 + return 1 + } + sed -n 's/.*Snapshot \([^ ]*\) saved\..*/\1/p' <<< "$output" +} + +# Waits for the lock for no longer than one operation, which is what a switch holding it takes at most. +lock_switch() { + exec 9> "$SWITCH_LOCK" || return 1 + flock -w "$OPERATION_TIMEOUT_SECONDS" 9 +} + +unlock_switch() { + exec 9>&- +} diff --git a/roles/github_runner_cluster/files/role/fleet-role-actuator.sh b/roles/github_runner_cluster/files/role/fleet-role-actuator.sh new file mode 100644 index 00000000..d52295a9 --- /dev/null +++ b/roles/github_runner_cluster/files/role/fleet-role-actuator.sh @@ -0,0 +1,425 @@ +#!/usr/bin/env bash +# fleet-role-actuator: switches this host between a k3s server and a k3s agent, for the role controller's voter swap (scripts/fleet-role.sh), as its Node asks. It does only what has to happen on the host: the controller does everything on the API (the cordon, the wait for runner pods, the drain, the taint, the regrading level label, the removal of the etcd member, the clean-up of the Node's labels and annotations, the uncordon), and the actuator's own credential can read Nodes and the regrade Lease and patch its report, nothing more. +# +# THE NODE. Only the controller sets these, on this host's Node; all are under github-runner.exadev.com/. The label desired-role (server or agent) is the role the host is to run. The annotation role-swap names the swap (--), role-step the step this host is to take in it (promote, step-down or rollback), role-join-node the node of the voter a promotion joins, and snapshot-request a swap whose snapshot this host, a voter that stays, is to take. +# +# THE REPORT. Each step's outcome goes to the ConfigMap ACTUATOR_CONFIGMAP in ACTUATOR_NAMESPACE, under this node's name, as JSON: {swap, step, phase, role, started_at, at, message, snapshot}, phase being running, done, failed or refused, role what the role file names afterwards, started_at the host's time just before a unit was started (what the controller waits for a later heartbeat than), at the time of the report and snapshot the names of the files a snapshot created. A report the API does not take is made again by the watch, which runs a step again until its outcome is reported; every step can be run again. +# +# AUTHORITY. Every step that could add a member (promote), and every other step a swap asks for (step-down, rollback, snapshot), is taken only while the regrade Lease (LEASE_NAMESPACE/LEASE_NAME) is held, unexpired, by ROLE_LEASE_HOLDER with the swap's id in its SWAP_ID_ANNOTATION: the controller holds it for the whole swap, and a regrade step or a run of the role cannot hold it meanwhile. finish-demote needs no Lease: it runs only once the role file names the agent and the server's etcd member is gone (a tombstone in the datastore, the Node's etcd.k3s.cattle.io/removed-node-name naming the datastore's member, or no etcd label on the Node), and completing the demotion of a member that is no longer one can never cost quorum, so a host that was down while it was removed finishes on its return. +# +# COMMANDS. watch runs for ever, reading this host's Node every POLL_SECONDS and taking the step it asks for, a snapshot it asks for, and the finish of a demotion whose member is gone. status prints the host's role, units and datastore and its last report. promote ID, step-down ID, rollback ID, snapshot ID and finish-demote take one step by hand and report it as the watch does. promote-local [JOIN_URL] and demote-local are the host-local part of a switch with no Lease and no report, for the role's own promotion and demotion (tasks/native/server_turn.yml and demote.yml), so that Ansible and the actuator switch a host the same way. +# +# EXIT STATUS. 0: done. 1: failed; the host is left with at most one role's unit running and a role file naming that role. 2: refused, with nothing changed: the Lease is not held by ROLE_LEASE_HOLDER for this swap (or cannot be read), the host is not provisioned to be a server, the role file names something that is not a role, the step is not one this actuator takes, or the step's own precondition does not hold (each says which). 3: not yet (finish-demote before the member is gone). +# +# The platform's file, fleet-role-actuator-$FLEET_ROLE_ACTUATOR.sh, says how each part is done on the host (its header lists the functions), after fleet-regrade-core.sh (kubectl_, select_api_server, lease_state, now_epoch, log), fleet-regrade-k3s.sh (write_level, restart_into, current_level, POLL_SECONDS) and the regrading actuator of the same platform, all from this script's directory. The settings are in fleet-role-actuator.env beside it, which the role templates (templates/role-actuator.env.j2). It needs bash, jq and kubectl besides what the platform needs. +set -euo pipefail + +lib=$(dirname "$(readlink -f "$0")") +# The core reads its settings from FLEET_REGRADE_ENV as it is sourced, so it is pointed at the actuator's. +FLEET_REGRADE_ENV=${FLEET_ROLE_ACTUATOR_ENV:-$lib/fleet-role-actuator.env} + +EXIT_DONE=0 +EXIT_FAILED=1 +EXIT_REFUSED=2 +EXIT_NOT_YET=3 + +# shellcheck source=/dev/null +. "$lib/fleet-regrade-core.sh" +case ${FLEET_ROLE_ACTUATOR:-} in + systemd) ;; + *) + echo "FLEET_ROLE_ACTUATOR is '${FLEET_ROLE_ACTUATOR:-}' in $FLEET_REGRADE_ENV; it must be systemd." >&2 + exit "$EXIT_REFUSED" + ;; +esac +# shellcheck source=/dev/null +. "$lib/fleet-regrade-k3s.sh" +# shellcheck source=/dev/null +. "$lib/fleet-regrade-$FLEET_ROLE_ACTUATOR.sh" +# shellcheck source=/dev/null +. "$lib/fleet-role-actuator-$FLEET_ROLE_ACTUATOR.sh" + +: "${NODE_NAME:?NODE_NAME must be set}" "${ROLE_LEASE_HOLDER:?ROLE_LEASE_HOLDER must be set}" "${SWAP_ID_ANNOTATION:?SWAP_ID_ANNOTATION must be set}" +: "${ACTUATOR_NAMESPACE:?ACTUATOR_NAMESPACE must be set}" "${ACTUATOR_CONFIGMAP:?ACTUATOR_CONFIGMAP must be set}" "${JOIN_URLS_FILE:?JOIN_URLS_FILE must be set}" + +# The Node's interface, fixed here: the controller writes these and the actuator reads them. +DESIRED_ROLE_LABEL=github-runner.exadev.com/desired-role +SWAP_ANNOTATION=github-runner.exadev.com/role-swap +STEP_ANNOTATION=github-runner.exadev.com/role-step +JOIN_NODE_ANNOTATION=github-runner.exadev.com/role-join-node +SNAPSHOT_ANNOTATION=github-runner.exadev.com/snapshot-request +# What k3s writes: the etcd role label it puts on a member's Node, and the annotation its member controller records once etcd has removed the member named in it (pkg/etcd/member_controller.go). +ETCD_LABEL=node-role.kubernetes.io/etcd +REMOVED_ANNOTATION=etcd.k3s.cattle.io/removed-node-name + +# What the step in hand says and sets, for its report: why it ended as it did, the host's time just before it started a unit, and the snapshot files it created. +MESSAGE='' +STARTED_AT='' +SNAPSHOT='' +# The swap and step run_step is taking, which report() reports against. +CURRENT_SWAP='' +CURRENT_STEP='' +# This host's Node, as JSON, read by the watch pass or the step that needs it. +NODE_JSON='{}' + +# Succeeds when $1 has the shape of a swap id: the UTC time to the second, then the two node names, which are DNS labels. +valid_swap_id() { + [[ $1 =~ ^[0-9]{8}T[0-9]{6}Z-[a-z0-9]([-a-z0-9.]*[a-z0-9])?$ ]] +} + +read_node() { + NODE_JSON=$(kubectl_ get node "$NODE_NAME" --output json) +} + +node_annotation() { + jq -r --arg key "$1" '.metadata.annotations[$key] // ""' <<< "$NODE_JSON" +} + +node_label() { + jq -r --arg key "$1" '.metadata.labels[$key] // ""' <<< "$NODE_JSON" +} + +# This node's last report, as JSON, or {} when it has none; fails when the ConfigMap cannot be read. +own_entry() { + local configmap + configmap=$(kubectl_ get configmap "$ACTUATOR_CONFIGMAP" --namespace "$ACTUATOR_NAMESPACE" --output json) || return 1 + jq -r --arg key "$NODE_NAME" '.data[$key] // "{}"' <<< "$configmap" +} + +# Succeeds when the report $1 is for swap $2 and step $3 and its phase is one of the rest, so the watch takes that step no more. +settled() { + local entry=$1 swap=$2 step=$3 + shift 3 + jq -e --arg swap "$swap" --arg step "$step" --args '.swap == $swap and .step == $step and (.phase as $phase | $ARGS.positional | index($phase) != null)' "$@" <<< "$entry" > /dev/null +} + +# Reports the step in hand with the phase $1, and logs it. A report that says what the last one said, but for its time, is not made again, so a step that waits does not write on every pass. +report() { + local phase=$1 entry current + log "${CURRENT_STEP} ${CURRENT_SWAP:-(no swap)}: ${phase}${MESSAGE:+: $MESSAGE}" + entry=$(jq -cn --arg swap "$CURRENT_SWAP" --arg step "$CURRENT_STEP" --arg phase "$phase" --arg role "$(read_role)" --arg started "$STARTED_AT" \ + --arg at "$(host_now)" --arg message "$MESSAGE" --arg snapshot "$SNAPSHOT" \ + '{swap: $swap, step: $step, phase: $phase, role: $role, started_at: $started, at: $at, message: $message, snapshot: $snapshot}') + if select_api_server && current=$(own_entry) && [ "$(jq -c 'del(.at)' <<< "$current")" = "$(jq -c 'del(.at)' <<< "$entry")" ]; then + return 0 + fi + if ! { select_api_server && kubectl_ patch configmap "$ACTUATOR_CONFIGMAP" --namespace "$ACTUATOR_NAMESPACE" --type merge \ + --patch "$(jq -cn --arg key "$NODE_NAME" --arg value "$entry" '{data: {($key): $value}}')" > /dev/null; }; then + log "the outcome could not be reported in $ACTUATOR_NAMESPACE/$ACTUATOR_CONFIGMAP; a watching actuator takes the step again and reports it" + fi +} + +# Succeeds when the regrade Lease is held, unexpired, by ROLE_LEASE_HOLDER for the swap $1; otherwise says why in MESSAGE. A Lease that cannot be read authorises nothing, since a step must not start on a guess. +authorised() { + local swap=$1 lease state holder held_for + if ! select_api_server; then + MESSAGE="no API server answers (tried: $(api_candidates_text)), so whether the regrade Lease is held for swap $swap cannot be told" + return 1 + fi + if ! lease=$(kubectl_ get lease "$LEASE_NAME" --namespace "$LEASE_NAMESPACE" --output json 2>&1); then + MESSAGE="the regrade Lease cannot be read: $lease" + return 1 + fi + state=$(lease_state "$lease" "$ROLE_LEASE_HOLDER" "$(now_epoch)") + if [ "$state" != mine ]; then + holder=$(jq -r '.spec.holderIdentity // ""' <<< "$lease") + case $state in + free) MESSAGE="the regrade Lease is free, not held by $ROLE_LEASE_HOLDER" ;; + expired) MESSAGE="the regrade Lease held by ${holder} has expired" ;; + *) MESSAGE="the regrade Lease is held by ${holder}, not $ROLE_LEASE_HOLDER" ;; + esac + return 1 + fi + held_for=$(jq -r --arg key "$SWAP_ID_ANNOTATION" '.metadata.annotations[$key] // ""' <<< "$lease") + if [ "$held_for" != "$swap" ]; then + MESSAGE="the regrade Lease is held for swap '${held_for}', not $swap" + return 1 + fi +} + +# The URL a promotion joins for the node $1, from the file the role writes with one line per server and eligible host, " ". +join_url_of() { + awk -v node="$1" '$1 == node { print $2; exit }' "$JOIN_URLS_FILE" +} + +# The host-local part of a promotion: the role file first, so that neither unit's Restart=always can start the agent again from here on, then the join drop-ins (or none, which leaves the role's own join configuration to decide), then the agent unit stopped and the server unit started. +promote_local() { + local url=${1:-} + write_role server || { MESSAGE="the role file could not be written"; return "$EXIT_FAILED"; } + if [ -n "$url" ]; then + join_set "$url" || { MESSAGE="the join drop-ins could not be written"; return "$EXIT_FAILED"; } + else + join_clear || { MESSAGE="the join drop-ins could not be removed"; return "$EXIT_FAILED"; } + fi + start_server_unit || { MESSAGE="the server unit did not start"; return "$EXIT_FAILED"; } + MESSAGE="the server unit started${url:+, joining $url}" +} + +# The host-local part of finishing a demotion or a rollback: the role file names the agent (it already does after a step-down), the server unit is stopped, its regrading level forgotten, its datastore moved aside and its join drop-ins removed, and the agent unit started. The order is the role's demotion's: nothing of the server is touched while its unit could still be running. +demote_local() { + write_role agent || { MESSAGE="the role file could not be written"; return "$EXIT_FAILED"; } + stop_server_unit || { MESSAGE="the server unit did not stop"; return "$EXIT_FAILED"; } + forget_level || { MESSAGE="the regrading level could not be removed"; return "$EXIT_FAILED"; } + move_datastore_aside || { MESSAGE="the datastore could not be moved aside"; return "$EXIT_FAILED"; } + join_clear || { MESSAGE="the join drop-ins could not be removed"; return "$EXIT_FAILED"; } + start_agent_unit || { MESSAGE="the agent unit did not start"; return "$EXIT_FAILED"; } + MESSAGE="the agent unit started, with the server's datastore moved aside" +} + +# Runs $@ while holding the host's switch lock, and returns its status. +locked() { + local status=0 + lock_switch || { MESSAGE="another role switch on this host held its lock for longer than ${OPERATION_TIMEOUT_SECONDS}s"; return "$EXIT_FAILED"; } + "$@" || status=$? + unlock_switch + return "$status" +} + +cmd_promote() { + local swap=$1 join_node url + authorised "$swap" || return "$EXIT_REFUSED" + if [ "$(read_role)" = server ] && server_active; then + STARTED_AT=$(last_start) + MESSAGE="already runs as a server" + return "$EXIT_DONE" + fi + provisioned || { MESSAGE="this host has no server configuration, so the role has not provisioned it to be a server"; return "$EXIT_REFUSED"; } + if datastore_present; then + MESSAGE="this host holds a datastore no demotion moved aside, and k3s would start that old member's etcd instead of joining as a new one" + return "$EXIT_REFUSED" + fi + read_node || { MESSAGE="this host's Node cannot be read"; return "$EXIT_REFUSED"; } + join_node=$(node_annotation "$JOIN_NODE_ANNOTATION") + url=$(join_url_of "$join_node") + [ -n "$url" ] || { MESSAGE="the node to join, '${join_node}', is not one $JOIN_URLS_FILE gives a URL for"; return "$EXIT_REFUSED"; } + report running + locked promote_local "$url" +} + +# Named apart from the core's own step_down, cmd_watch and cmd_status, as watch_node and print_status are, so that sourcing the core never puts its functions in the place of the actuator's or the reverse. +# At etcd-only first, through regrading's own levels, so that the server keeps its vote and nothing else while the controller asks for its removal; then the role file names the agent, before the removal, so that a restart of the server unit after it (Restart=always, a reboot) cannot rejoin the cluster as a new learner; then the regrade watcher stops, which could otherwise step the server back up. A server whose role file already names the agent has stepped down already and is not restarted, since its unit's condition would now keep it down. +step_down_local() { + local role + role=$(read_role) + case $role in + agent) + stop_regrade || { MESSAGE="the regrade watcher did not stop"; return "$EXIT_FAILED"; } + MESSAGE="the role file already names the agent" + return "$EXIT_DONE" + ;; + server) ;; + *) + MESSAGE="the role file names '${role}', which is not a role" + return "$EXIT_REFUSED" + ;; + esac + server_active || { MESSAGE="the server unit is not running, so there is no member here to step down"; return "$EXIT_FAILED"; } + if [ "$(current_level)" != etcd-only ]; then + enter_etcd_only || { MESSAGE="the server did not come back at etcd-only within ${OPERATION_TIMEOUT_SECONDS}s"; return "$EXIT_FAILED"; } + fi + write_role agent || { MESSAGE="the role file could not be written"; return "$EXIT_FAILED"; } + stop_regrade || { MESSAGE="the regrade watcher did not stop"; return "$EXIT_FAILED"; } + MESSAGE="at etcd-only, with the role file naming the agent" +} + +cmd_step_down() { + authorised "$1" || return "$EXIT_REFUSED" + report running + locked step_down_local +} + +# Succeeds once this host's etcd member is gone, and otherwise says why not in MESSAGE. The tombstone is read first, since it needs no API: k3s writes it when etcd tells the member it was removed, which a member that was down when it was removed never hears, so the Node is asked then. +member_gone() { + local member + if ! datastore_present; then + MESSAGE="no datastore is left" + return 0 + fi + if datastore_tombstoned; then + MESSAGE="the datastore holds k3s's tombstone" + return 0 + fi + member=$(datastore_member) + if ! { select_api_server && read_node; }; then + MESSAGE="no API server answers and the datastore holds no tombstone, so whether the member was removed cannot be told yet" + return 1 + fi + if [ -n "$member" ] && [ "$(node_annotation "$REMOVED_ANNOTATION")" = "$member" ]; then + MESSAGE="k3s recorded the removal of member $member" + return 0 + fi + if [ -z "$(node_label "$ETCD_LABEL")" ]; then + MESSAGE="the Node carries no etcd label" + return 0 + fi + MESSAGE="etcd member ${member:-of this host} has not been removed yet" + return 1 +} + +cmd_finish_demote() { + local role + role=$(read_role) + [ "$role" = agent ] || { MESSAGE="the role file names '${role}', not agent, so no demotion of this host is under way"; return "$EXIT_REFUSED"; } + member_gone || return "$EXIT_NOT_YET" + report running + locked demote_local +} + +cmd_rollback() { + authorised "$1" || return "$EXIT_REFUSED" + report running + locked demote_local +} + +take_named_snapshot() { + SNAPSHOT=$(take_snapshot "fleet-role-$1" | paste -s -d , -) || { MESSAGE="k3s etcd-snapshot save failed"; return "$EXIT_FAILED"; } + [ -n "$SNAPSHOT" ] || { MESSAGE="k3s etcd-snapshot save reported no snapshot file"; return "$EXIT_FAILED"; } + MESSAGE="snapshot taken" +} + +cmd_snapshot() { + authorised "$1" || return "$EXIT_REFUSED" + if [ "$(read_role)" != server ] || ! server_active; then + MESSAGE="this host is not a running server, so it has no etcd to take a snapshot of" + return "$EXIT_REFUSED" + fi + report running + locked take_named_snapshot "$1" +} + +# Takes the step $1 of the swap $2, reports how it ended unless it is not yet due, and returns its exit status. +run_step() { + local step=$1 swap=$2 status=0 + CURRENT_STEP=$step + CURRENT_SWAP=$swap + MESSAGE='' + STARTED_AT='' + SNAPSHOT='' + case $step in + promote | step-down | rollback | snapshot) + if ! valid_swap_id "$swap"; then + MESSAGE="'${swap}' is not a swap id (--)" + status=$EXIT_REFUSED + fi + ;; + finish-demote) ;; + *) + MESSAGE="'${step}' is not a step this actuator takes" + status=$EXIT_REFUSED + ;; + esac + if [ "$status" -eq 0 ]; then + "cmd_${step//-/_}" "$swap" || status=$? + fi + case $status in + "$EXIT_DONE") report "done" ;; + "$EXIT_FAILED") report failed ;; + "$EXIT_REFUSED") report refused ;; + "$EXIT_NOT_YET") say "${step} ${swap:-(no swap)}: not yet: ${MESSAGE}" ;; + esac + return "$status" +} + +# Logs $1 unless it is what the last call logged, so a watch that waits says so once. +LAST_SAID='' +say() { + [ "$1" = "$LAST_SAID" ] || log "$1" + LAST_SAID=$1 +} + +# The role a step asks the host to run, which the Node's desired-role label must agree with: a label and an annotation that disagree are a controller's mistake, and the step is refused rather than guessed at. +role_for_step() { + case $1 in + promote) echo server ;; + step-down | rollback) echo agent ;; + esac +} + +# One look at the Node: a snapshot it asks for, the step it asks for, then the finish of a demotion whose role file names the agent while something of the server is left. A step reported done, failed or refused for the swap is not taken again (the controller decides what follows a failure); a finish is taken until it is done, since it can never cost quorum. +watch_pass() { + local entry request swap step desired + if ! select_api_server; then + say "no API server answers (tried: $(api_candidates_text)); waiting" + return 1 + fi + read_node || return 1 + entry=$(own_entry) || return 1 + request=$(node_annotation "$SNAPSHOT_ANNOTATION") + if [ -n "$request" ] && ! settled "$entry" "$request" snapshot "done" failed refused; then + run_step snapshot "$request" || true + read_node || return 1 + entry=$(own_entry) || return 1 + fi + swap=$(node_annotation "$SWAP_ANNOTATION") + step=$(node_annotation "$STEP_ANNOTATION") + if [ -n "$swap" ] && [ -n "$step" ] && ! settled "$entry" "$swap" "$step" "done" failed refused; then + desired=$(node_label "$DESIRED_ROLE_LABEL") + if [ -n "$(role_for_step "$step")" ] && [ "$desired" != "$(role_for_step "$step")" ]; then + CURRENT_STEP=$step CURRENT_SWAP=$swap STARTED_AT='' SNAPSHOT='' + MESSAGE="the Node's $DESIRED_ROLE_LABEL is '${desired}', which a $step does not go with" + report refused + else + run_step "$step" "$swap" || true + fi + entry=$(own_entry) || return 1 + fi + if [ "$(read_role)" = agent ] && { server_active || datastore_present; } && ! settled "$entry" "$swap" finish-demote "done"; then + run_step finish-demote "$swap" || true + fi +} + +watch_node() { + log "watching node $NODE_NAME every ${POLL_SECONDS}s" + while true; do + watch_pass || true + sleep "$POLL_SECONDS" + done +} + +print_status() { + local entry + log "role file: $(read_role)" + log "server unit: $(server_active && echo running || echo stopped); agent unit: $(agent_active && echo running || echo stopped)" + log "datastore: $(datastore_present && echo "in place, member $(datastore_member)$(datastore_tombstoned && echo ', tombstoned')" || echo none); moved aside: $(datastore_moved_aside && echo yes || echo no)" + log "joins: $(join_current)" + if select_api_server && entry=$(own_entry); then + log "last report: $entry" + else + log "last report: no API server answers (tried: $(api_candidates_text))" + fi +} + +usage() { + echo "usage: fleet-role-actuator watch|status|promote ID|step-down ID|rollback ID|snapshot ID|finish-demote|promote-local [JOIN_URL]|demote-local" >&2 + exit "$EXIT_REFUSED" +} + +actuator_main() { + local status=0 + case ${1:-} in + watch) watch_node ;; + status) print_status ;; + promote | step-down | rollback | snapshot) + [ $# -eq 2 ] || usage + run_step "$1" "$2" || status=$? + ;; + finish-demote) + [ $# -eq 1 ] || usage + run_step finish-demote "" || status=$? + ;; + promote-local) + [ $# -le 2 ] || usage + locked promote_local "${2:-}" || status=$? + log "promote-local: ${MESSAGE}" + ;; + demote-local) + [ $# -eq 1 ] || usage + locked demote_local || status=$? + log "demote-local: ${MESSAGE}" + ;; + *) usage ;; + esac + return "$status" +} + +actuator_main "$@" diff --git a/roles/github_runner_cluster/tasks/native/config.yml b/roles/github_runner_cluster/tasks/native/config.yml index 3e87e4de..1a04ea41 100644 --- a/roles/github_runner_cluster/tasks/native/config.yml +++ b/roles/github_runner_cluster/tasks/native/config.yml @@ -32,6 +32,47 @@ label: "{{ item.dest }}" register: github_runner_cluster_native_helpers +- name: Create the directory the role actuator's scripts and settings go in + become: true + ansible.builtin.file: + path: "{{ github_runner_cluster_role_actuator_dir }}" + state: directory + mode: "0755" + +- name: Place the role actuator's scripts + # On every native host, whether or not its watcher runs: the role's own promotion and demotion (server_turn.yml, demote.yml) switch the host through its promote-local and demote-local, so that the role and the actuator switch a host one way. It runs on regrading's core and systemd actuator, copied beside it. Copied on every run, so a collection upgrade reaches the host; none of the k3s units reads them, so a change restarts only the actuator's own unit (tasks/role_actuator.yml). + become: true + ansible.builtin.copy: + src: "{{ item.src }}" + dest: "{{ github_runner_cluster_role_actuator_dir }}/{{ item.src | basename }}" + mode: "{{ item.mode }}" + loop: + - {src: role/fleet-role-actuator.sh, mode: "0755"} + - {src: role/fleet-role-actuator-systemd.sh, mode: "0644"} + - {src: regrade/fleet-regrade-core.sh, mode: "0644"} + - {src: regrade/fleet-regrade-k3s.sh, mode: "0644"} + - {src: regrade/fleet-regrade-systemd.sh, mode: "0644"} + loop_control: + label: "{{ item.src | basename }}" + register: github_runner_cluster_role_actuator_scripts + +- name: Template the role actuator's settings + become: true + ansible.builtin.template: + src: role-actuator.env.j2 + dest: "{{ github_runner_cluster_role_actuator_dir }}/fleet-role-actuator.env" + mode: "0644" + register: github_runner_cluster_role_actuator_settings + +- name: Put fleet-role-actuator on the PATH, for a switch or a status by hand + become: true + ansible.builtin.file: + src: "{{ github_runner_cluster_role_actuator_dir }}/fleet-role-actuator.sh" + dest: "{{ github_runner_cluster_native_bin_dir }}/fleet-role-actuator" + state: link + # Under --check the script above has not been placed, so the link's target may not exist yet. + force: true + - name: Record the role a host set up before the role file existed runs # Before either unit is written: the new units start only while this file names their role, so a host that runs the agent unit from an earlier release keeps running it across the restart below. A host that has the file keeps it; changing it is a role switch (service.yml), never a run's side effect. A first install records the role it is to run, which nothing runs yet. become: true diff --git a/roles/github_runner_cluster/tasks/native/demote.yml b/roles/github_runner_cluster/tasks/native/demote.yml index eb2fa733..99bf27a9 100644 --- a/roles/github_runner_cluster/tasks/native/demote.yml +++ b/roles/github_runner_cluster/tasks/native/demote.yml @@ -1,5 +1,5 @@ --- -# Switches a native server to the agent unit, keeping its node name, node password and Tailscale device. Make-before-break: the server's etcd member is removed only while at least three other voters, every one Ready, remain to hold quorum without it (etcd calls a two-member cluster unsafe, https://etcd.io/docs/v3.6/op-guide/runtime-configuration/), so a cluster grows a voter before it loses one. In order: the node is emptied (drain.yml); the role file is switched to agent, so that neither Restart=always nor a reboot can start the server unit again (k3s marks a removed member's datastore with a tombstone and, started again, would rejoin it as a learner; Register in pkg/etcd/etcd.go at v1.34.1+k3s1); the member is removed through k3s's own controller, which acts on the etcd.k3s.cattle.io/remove annotation and records etcd.k3s.cattle.io/removed-node-name once etcd has removed it (pkg/etcd/member_controller.go); the server unit is stopped and disabled; the server's datastore is moved aside, because k3s reuses any datastore it finds and would otherwise start the old member's etcd again on a later promotion (https://docs.k3s.io/datastore/ha-embedded); the server role labels and k3s's etcd annotations are taken off the Node, which k3s does not do for a server that runs as an agent, so the cluster reads the host's live role right; and the agent unit is enabled and started, the node waited for and uncordoned. Needs service.yml's unit state. Under --check every read runs, the refusal included, and the switch is reported, not made. +# Switches a native server to the agent unit, keeping its node name, node password and Tailscale device. Make-before-break: the server's etcd member is removed only while at least three other voters, every one Ready, remain to hold quorum without it (etcd calls a two-member cluster unsafe, https://etcd.io/docs/v3.6/op-guide/runtime-configuration/), so a cluster grows a voter before it loses one. In order: the node is emptied (drain.yml); the role file is switched to agent, so that neither Restart=always nor a reboot can start the server unit again (k3s marks a removed member's datastore with a tombstone and, started again, would rejoin it as a learner; Register in pkg/etcd/etcd.go at v1.34.1+k3s1); the member is removed through k3s's own controller, which acts on the etcd.k3s.cattle.io/remove annotation and records etcd.k3s.cattle.io/removed-node-name once etcd has removed it (pkg/etcd/member_controller.go); the role actuator's host-local demotion (files/role/fleet-role-actuator.sh demote-local) stops and disables the server unit, moves the server's datastore aside, because k3s reuses any datastore it finds and would otherwise start the old member's etcd again on a later promotion (https://docs.k3s.io/datastore/ha-embedded), and enables and starts the agent unit; the server role labels and k3s's etcd annotations are taken off the Node, which k3s does not do for a server that runs as an agent, so the cluster reads the host's live role right; and the node is waited for and uncordoned. Needs service.yml's unit state. Under --check every read runs, the refusal included, and the switch is reported, not made. - name: Reach the cluster through another server, since this one is to stop ansible.builtin.include_tasks: native/api.yml @@ -92,33 +92,27 @@ delay: "{{ github_runner_cluster_native_poll }}" changed_when: false - - name: Stop and disable the server unit + - name: Read the time on the host, before the agent is started + ansible.builtin.command: + argv: [date, -u, "+%Y-%m-%dT%H:%M:%SZ"] + register: github_runner_cluster_native_started + changed_when: false + + - name: Switch the host to the agent unit + # The role actuator's host-local demotion (files/role/fleet-role-actuator.sh), which finishing a swap's demotion takes too: the server unit is stopped and disabled; its regrading level is forgotten, since the level is a server's and a later promotion must start the server whole; its datastore is moved aside, keeping only the newest copy, because k3s reuses any datastore it finds and would otherwise start the old member's etcd again on a later promotion, and a copy rather than a deletion since it holds the snapshots this server took; any join drop-ins a swap left are removed; and the agent unit is enabled and started. become: true ansible.builtin.command: - argv: [systemctl, disable, --now, "{{ github_runner_cluster_native_server_service }}"] + argv: ["{{ github_runner_cluster_role_actuator_dir }}/fleet-role-actuator.sh", demote-local] changed_when: true - - name: Forget the regrading level the server was left at - # The level is a server's, and the agent unit never reads it; a later promotion must start the server whole, not at a level it was stepped to in an earlier term as a server. + - name: Clear the agent's waiting restart, since it has started on its files become: true ansible.builtin.file: - path: "{{ github_runner_cluster_native_regrade_level_file }}" + path: "{{ github_runner_cluster_native_restart_marker }}-{{ github_runner_cluster_native_agent_service }}" state: absent - - name: Move the server's datastore aside, keeping only the newest copy - # A copy, not a deletion: it holds the snapshots this server took (server/db/snapshots). One copy only, the one from this demotion, so repeated switches do not fill the disk. - become: true - ansible.builtin.shell: | - set -eu - rm -rf "$server/db.demoted" - mv "$server/db" "$server/db.demoted" - environment: - server: "{{ github_runner_cluster_native_data_dir }}/server" - changed_when: true - when: github_runner_cluster_native_has_datastore | bool - - name: Take the server role labels and k3s's etcd annotations off the Node - # A trailing "-" removes a key; one that is not there is left alone. + # A trailing "-" removes a key; one that is not there is left alone. After the agent has started, as the role controller's swap takes them off: the agent's k3s puts none of them back. become: "{{ github_runner_cluster_native_api_become }}" ansible.builtin.command: argv: "{{ github_runner_cluster_native_api_argv + [item, 'node', github_runner_cluster_effective_node_name] + github_runner_cluster_native_keys[item] }}" @@ -131,24 +125,6 @@ delegate_to: "{{ github_runner_cluster_native_api_host }}" changed_when: true - - name: Read the time on the host, before the agent is started - ansible.builtin.command: - argv: [date, -u, "+%Y-%m-%dT%H:%M:%SZ"] - register: github_runner_cluster_native_started - changed_when: false - - - name: Enable and start the agent unit - become: true - ansible.builtin.command: - argv: [systemctl, enable, --now, "{{ github_runner_cluster_native_agent_service }}"] - changed_when: true - - - name: Clear the agent's waiting restart, since it has started on its files - become: true - ansible.builtin.file: - path: "{{ github_runner_cluster_native_restart_marker }}-{{ github_runner_cluster_native_agent_service }}" - state: absent - - name: Wait for a server to see this node Ready on a heartbeat from the agent just started ansible.builtin.include_tasks: native/server_ready.yml diff --git a/roles/github_runner_cluster/tasks/native/main.yml b/roles/github_runner_cluster/tasks/native/main.yml index b71c0318..8a444703 100644 --- a/roles/github_runner_cluster/tasks/native/main.yml +++ b/roles/github_runner_cluster/tasks/native/main.yml @@ -1,5 +1,5 @@ --- -# Brings this host up as a native k3s node: k3s runs directly on the host as the systemd service k3s-agent (an agent) or k3s (a server), with no Docker or Compose layer, and the host's own Tailscale client is the node's device on the cluster's tailnet (k3s's --vpn-auth runs `tailscale up` against it). Selected by github_runner_cluster_node_mode: native, from tasks/main.yml, which ends the role after this. The topology, the versions, the API port, the token and the mesh are worked out by the same task files as the Docker path, so a native node joins the same cluster with the same secrets, node name rules and API port, and a native server is configured as every Docker server is. Safe to rerun: each step converges, a unit is restarted only when its binary, configuration or secrets changed, and the host switches between its server and agent units only when its role changed (role.yml). Every task that changes the host uses become (sudo) on its own; under --check nothing is changed and each change a real run would make is reported. +# Brings this host up as a native k3s node: k3s runs directly on the host as the systemd service k3s-agent (an agent) or k3s (a server), with no Docker or Compose layer, and the host's own Tailscale client is the node's device on the cluster's tailnet (k3s's --vpn-auth runs `tailscale up` against it). Selected by github_runner_cluster_node_mode: native, from tasks/main.yml, which ends the role after this. The topology, the versions, the API port, the token and the mesh are worked out by the same task files as the Docker path, so a native node joins the same cluster with the same secrets, node name rules and API port, and a native server is configured as every Docker server is. Safe to rerun: each step converges, a unit is restarted only when its binary, configuration or secrets changed, and the host switches between its server and agent units only when its role changed (role.yml) or, with the role controller on, when the controller asks through the host's Node (role_actuator.yml). Every task that changes the host uses become (sudo) on its own; under --check nothing is changed and each change a real run would make is reported. - name: Read the platform the native node needs # Only the subsets that hold the kernel, architecture, distribution and init system, and only when the caller did not gather them: a play with gather_facts: false leaves ansible_facts without them. Read-only, so it runs under --check too. @@ -111,3 +111,6 @@ - name: Run the fleet-regrade unit beside this server, or remove it ansible.builtin.include_tasks: native/regrade_service.yml + +- name: Run the role actuator's watcher on a host the role controller may switch, or remove it + ansible.builtin.include_tasks: role_actuator.yml diff --git a/roles/github_runner_cluster/tasks/native/server_turn.yml b/roles/github_runner_cluster/tasks/native/server_turn.yml index 5cec7151..906ab961 100644 --- a/roles/github_runner_cluster/tasks/native/server_turn.yml +++ b/roles/github_runner_cluster/tasks/native/server_turn.yml @@ -1,7 +1,7 @@ --- # One native server's turn (see service.yml, which gives each native server of the play its turn, one at a time, as tasks/start_servers.yml does for the Docker servers): works out what the server unit needs, refuses when that would take quorum away or add a member beside an unhealthy one, does it, and waits for the node to be Ready on a heartbeat from the k3s just started and a voting member of etcd. What a turn does: # -# - promote: the host runs the agent unit and is to be a server. Its node is emptied (drain.yml), the role file is switched to server, the agent unit is stopped and disabled and the server unit enabled and started. k3s joins it as an etcd learner (MemberAddAsLearner in join, pkg/etcd/etcd.go), which the etcd leader promotes once it has caught up; a learner does not count towards quorum, so the add costs none. Same node name, node password (/etc/rancher/node/password, which neither unit's k3s is told to change) and Tailscale device, so the cluster's node-password Secret stays valid. +# - promote: the host runs the agent unit and is to be a server. Its node is emptied (drain.yml), then the role actuator's host-local promotion (files/role/fleet-role-actuator.sh promote-local) switches the role file to server, stops and disables the agent unit and enables and starts the server unit. k3s joins it as an etcd learner (MemberAddAsLearner in join, pkg/etcd/etcd.go), which the etcd leader promotes once it has caught up; a learner does not count towards quorum, so the add costs none. Same node name, node password (/etc/rancher/node/password, which neither unit's k3s is told to change) and Tailscale device, so the cluster's node-password Secret stays valid. # - first start: nothing runs on the host yet; the server unit is enabled and started, bootstrapping a new cluster or joining one as 10-join.yaml says. # - restart: the server runs and something it reads changed. Its etcd member is away while k3s restarts. # - start: the server unit is installed and stopped; it is started, and rejoins with its own datastore. @@ -71,22 +71,6 @@ ansible.builtin.include_tasks: native/drain.yml when: github_runner_cluster_native_turn == 'promote' - - name: Record that the host is now a server - # Before the agent unit stops, so that Restart=always cannot start it again: from here on only the server unit's condition holds. - become: true - ansible.builtin.copy: - content: server - dest: "{{ github_runner_cluster_native_role_file }}" - mode: "0644" - when: github_runner_cluster_native_turn == 'promote' - - - name: Stop and disable the agent unit - become: true - ansible.builtin.command: - argv: [systemctl, disable, --now, "{{ github_runner_cluster_native_agent_service }}"] - changed_when: true - when: github_runner_cluster_native_turn == 'promote' - - name: Read the time on the host, before the server is started or restarted # The node's Ready condition outlives a stopped k3s for a while, so the wait below is for a heartbeat later than this. The host's own clock, which the kubelet's heartbeat is stamped with. ansible.builtin.command: @@ -94,12 +78,22 @@ register: github_runner_cluster_native_started changed_when: false + - name: Switch the host to the server unit + # The role actuator's host-local promotion (files/role/fleet-role-actuator.sh), which the role controller's swaps take too: the role file is switched to server before the agent unit stops, so that Restart=always cannot start the agent again, any join drop-ins a swap left are removed, so that 10-join.yaml decides, and the agent unit is stopped and disabled and the server unit enabled and started. + become: true + ansible.builtin.command: + argv: ["{{ github_runner_cluster_role_actuator_dir }}/fleet-role-actuator.sh", promote-local] + changed_when: true + when: github_runner_cluster_native_turn == 'promote' + - name: Enable the server unit become: true ansible.builtin.command: argv: [systemctl, enable, "{{ github_runner_cluster_native_server_service }}"] changed_when: true - when: not github_runner_cluster_native_server_enabled | bool + when: + - github_runner_cluster_native_turn != 'promote' + - not github_runner_cluster_native_server_enabled | bool - name: Start or restart the server unit # A restart leaves the pods running: the unit's KillMode=process leaves their containerd shims alone. @@ -107,6 +101,7 @@ ansible.builtin.command: argv: [systemctl, "{{ 'restart' if github_runner_cluster_native_turn == 'restart' else 'start' }}", "{{ github_runner_cluster_native_server_service }}"] changed_when: true + when: github_runner_cluster_native_turn != 'promote' - name: Clear the server's waiting restart, since it has started on its files become: true diff --git a/roles/github_runner_cluster/tasks/role_actuator.yml b/roles/github_runner_cluster/tasks/role_actuator.yml new file mode 100644 index 00000000..50485f83 --- /dev/null +++ b/roles/github_runner_cluster/tasks/role_actuator.yml @@ -0,0 +1,44 @@ +--- +# Runs the role actuator's watcher (files/role/fleet-role-actuator.sh watch, as the github_runner_cluster_role_actuator_service unit) on a native host the role controller may switch or take a snapshot on, a server or a host marked eligible, while github_runner_cluster_role_controller is true, and removes it otherwise; native/config.yml has placed its scripts and settings on every native host. The watcher gets a credential of its own: the role applies its ServiceAccount, its scoped rules and its token Secret (templates/role-actuator-rbac.yaml.j2) through another server's API, creates the report ConfigMap the controller reads when there is none, and writes a kubeconfig holding the token, mode 0600, by a task that logs nothing. A file maps every other server and eligible host's node name to the URL a promotion joins it at. Run after native/service.yml has run the host's unit. While github_runner_cluster_native_start is false the unit is left as it is found, running or not, as the k3s units are, and only its files are written. Under --check nothing is changed and each change a real run would make is reported. + +- name: Work out whether the role actuator's watcher runs on this host + ansible.builtin.set_fact: + github_runner_cluster_role_actuator_on: "{{ github_runner_cluster_role_controller | bool and github_runner_cluster_native_stage_server | bool }}" + +- name: Run the role actuator's watcher + ansible.builtin.include_tasks: role_actuator_watcher.yml + when: github_runner_cluster_role_actuator_on | bool + +- name: Remove the role actuator's watcher, its credential and its join URLs from the host + # The scripts and settings stay: the role's own promotion and demotion run them. What the role applied to the cluster stays too, shared by every host's watcher, and grants nothing to anyone without its token. + when: not github_runner_cluster_role_actuator_on | bool + block: + - name: Read whether the watcher is installed + ansible.builtin.stat: + path: "{{ github_runner_cluster_native_unit_dir }}/{{ github_runner_cluster_role_actuator_service }}.service" + register: github_runner_cluster_role_actuator_installed + + - name: Remove the watcher + when: github_runner_cluster_role_actuator_installed.stat.exists + block: + - name: Stop and disable the watcher + become: true + ansible.builtin.command: + argv: [systemctl, disable, --now, "{{ github_runner_cluster_role_actuator_service }}"] + changed_when: true + + - name: Remove the watcher's unit, kubeconfig and join URLs + become: true + ansible.builtin.file: + path: "{{ item }}" + state: absent + loop: + - "{{ github_runner_cluster_native_unit_dir }}/{{ github_runner_cluster_role_actuator_service }}.service" + - "{{ github_runner_cluster_role_actuator_kubeconfig }}" + - "{{ github_runner_cluster_role_actuator_dir }}/join-urls" + + - name: Reload systemd, which still knows the removed unit + become: true + ansible.builtin.command: + argv: [systemctl, daemon-reload] + changed_when: true diff --git a/roles/github_runner_cluster/tasks/role_actuator_watcher.yml b/roles/github_runner_cluster/tasks/role_actuator_watcher.yml new file mode 100644 index 00000000..bef32517 --- /dev/null +++ b/roles/github_runner_cluster/tasks/role_actuator_watcher.yml @@ -0,0 +1,205 @@ +--- +# Runs the role actuator's watcher on a native host the role controller may switch, for tasks/role_actuator.yml, which includes this file only on such a host: an include skipped by its condition loads none of its tasks, whereas a task in a block skipped by its condition still has its delegate_to templated (ansible-core resolves delegate_to before when, _calculate_delegate_to in TaskExecutor._execute), and the delegated tasks here name github_runner_cluster_native_api_host, which only native/api.yml below sets. Installs the commands the watcher needs, applies its credential through another server's API, writes its kubeconfig, join URLs and unit, and starts or restarts it. Under --check nothing is changed and each change a real run would make is reported. + +- name: Read which of the commands the watcher needs are installed + # Read-only, so it runs under --check too. + ansible.builtin.shell: | + set -u + for command in $commands; do + if command -v "$command" >/dev/null; then echo "$command"; fi + done + environment: + commands: "{{ github_runner_cluster_role_actuator_commands.keys() | join(' ') }}" + register: github_runner_cluster_role_actuator_host + changed_when: false + check_mode: false + +- name: Record the packages the watcher needs that are not installed + ansible.builtin.set_fact: + github_runner_cluster_role_actuator_packages: >- + {{ github_runner_cluster_role_actuator_commands | dict2items + | rejectattr('key', 'in', github_runner_cluster_role_actuator_host.stdout_lines) | map(attribute='value') | unique | list }} + +- name: Fail if a command the watcher needs is missing on a host the role cannot install it on + ansible.builtin.fail: + msg: >- + github_runner_cluster_role_controller is true, but {{ inventory_hostname }} lacks the packages + {{ github_runner_cluster_role_actuator_packages | join(', ') }}, which the role actuator needs, and the role installs them from + the distribution's packages on Debian and Ubuntu only. Install them and run again. Nothing was changed. + when: + - github_runner_cluster_role_actuator_packages | length > 0 + - ansible_facts['os_family'] != 'Debian' + +- name: Install the packages the watcher needs that the host lacks + # The index is refreshed first, since a host's lists can be missing or stale. + when: github_runner_cluster_role_actuator_packages | length > 0 + block: + - name: Refresh the package index + become: true + ansible.builtin.command: + argv: [apt-get, update] + environment: + DEBIAN_FRONTEND: noninteractive + changed_when: true + + - name: Install the packages + become: true + ansible.builtin.command: + argv: "{{ ['apt-get', 'install', '--yes', '--no-install-recommends'] + github_runner_cluster_role_actuator_packages }}" + environment: + DEBIAN_FRONTEND: noninteractive + changed_when: true + +- name: Reach the cluster through another server, since this host's own may be the one a swap stops + ansible.builtin.include_tasks: native/api.yml + vars: + github_runner_cluster_native_api_self: false + +- name: Fail if no other server of the inventory can apply the watcher's credential + ansible.builtin.fail: + msg: >- + github_runner_cluster_role_controller is true, but the inventory names no server other than {{ inventory_hostname }} to apply the + role actuator's credential through. Nothing was changed. + when: github_runner_cluster_native_api_host | length == 0 + +- name: Write the watcher's unit + # Under --check too, as the regrade unit is (native/regrade_service.yml), so a check run reports a changed unit. + become: true + ansible.builtin.template: + src: native/fleet-role-actuator.service.j2 + dest: "{{ github_runner_cluster_native_unit_dir }}/{{ github_runner_cluster_role_actuator_service }}.service" + mode: "0644" + register: github_runner_cluster_role_actuator_unit + +- name: Reload systemd when the unit changed + become: true + ansible.builtin.command: + argv: [systemctl, daemon-reload] + changed_when: true + when: + - github_runner_cluster_role_actuator_unit is changed + - not ansible_check_mode + +- name: Report the credential and the watcher a real run would set up + ansible.builtin.debug: + msg: >- + apply the role actuator's credential ({{ github_runner_cluster_role_actuator_name }} in {{ github_runner_cluster_role_actuator_namespace }}), + write {{ github_runner_cluster_role_actuator_kubeconfig }} and run {{ github_runner_cluster_role_actuator_service }} + changed_when: true + when: ansible_check_mode + +- name: Set up the watcher's credential and run it + when: not ansible_check_mode + block: + - name: Apply the watcher's ServiceAccount, its rules and its token Secret + become: "{{ github_runner_cluster_native_api_become }}" + ansible.builtin.command: + # docker exec passes its standard input to kubectl only with --interactive, as tasks/colima_size_turn.yml gives it; k3s kubectl reads its own. + argv: >- + {{ ((github_runner_cluster_native_api_argv[:2] + ['--interactive'] + github_runner_cluster_native_api_argv[2:]) + if github_runner_cluster_native_api_argv[0] == 'docker' else github_runner_cluster_native_api_argv) + + ['apply', '--filename', '-'] }} + stdin: "{{ lookup('ansible.builtin.template', 'role-actuator-rbac.yaml.j2') }}" + environment: "{{ github_runner_cluster_native_api_environment }}" + delegate_to: "{{ github_runner_cluster_native_api_host }}" + register: github_runner_cluster_role_actuator_applied + changed_when: github_runner_cluster_role_actuator_applied.stdout_lines | reject('search', ' unchanged$') | list | length > 0 + + - name: Create the report ConfigMap the controller reads, when there is none + # Created rather than applied, since applying an empty one again would wipe the reports. + become: "{{ github_runner_cluster_native_api_become }}" + ansible.builtin.command: + argv: >- + {{ github_runner_cluster_native_api_argv + ['create', 'configmap', github_runner_cluster_role_actuator_configmap, + '--namespace', github_runner_cluster_role_actuator_namespace] }} + environment: "{{ github_runner_cluster_native_api_environment }}" + delegate_to: "{{ github_runner_cluster_native_api_host }}" + register: github_runner_cluster_role_actuator_configmap_created + changed_when: github_runner_cluster_role_actuator_configmap_created.rc == 0 + failed_when: + - github_runner_cluster_role_actuator_configmap_created.rc != 0 + - "'AlreadyExists' not in github_runner_cluster_role_actuator_configmap_created.stderr" + + - name: Read the watcher's token and the cluster's CA, once the token controller has filled them + # Both base64, as the Secret holds them; the token is a secret, so nothing of this task is logged. + become: "{{ github_runner_cluster_native_api_become }}" + ansible.builtin.command: + argv: >- + {{ github_runner_cluster_native_api_argv + ['get', 'secret', github_runner_cluster_role_actuator_name ~ '-token', + '--namespace', github_runner_cluster_role_actuator_namespace, '--request-timeout=' ~ github_runner_cluster_cluster_read_timeout ~ 's', + '--output', 'jsonpath={.data.token} {.data.ca\.crt}'] }} + environment: "{{ github_runner_cluster_native_api_environment }}" + delegate_to: "{{ github_runner_cluster_native_api_host }}" + register: github_runner_cluster_role_actuator_secret + until: + - github_runner_cluster_role_actuator_secret.rc == 0 + - github_runner_cluster_role_actuator_secret.stdout | trim | split(' ') | length == 2 + retries: "{{ ((github_runner_cluster_cluster_read_timeout | int) / github_runner_cluster_native_poll) | round(0, 'ceil') | int }}" + delay: "{{ github_runner_cluster_native_poll }}" + vars: + # Seconds between reads, the pace every native wait polls at (tasks/native/server_ready.yml). + github_runner_cluster_native_poll: 5 + changed_when: false + no_log: true + + - name: Write the watcher's kubeconfig + # Its server is the first API candidate; the watcher passes the one that answers (KUBE_SERVERS in its settings) on every call. + become: true + ansible.builtin.copy: + content: >- + {{ {'apiVersion': 'v1', 'kind': 'Config', 'current-context': 'fleet', + 'clusters': [{'name': 'fleet', 'cluster': {'server': 'https://' ~ (github_runner_cluster_api_candidates | first).value.address ~ ':' ~ github_runner_cluster_api_port, + 'certificate-authority-data': (github_runner_cluster_role_actuator_secret.stdout | trim | split(' '))[1]}}], + 'users': [{'name': github_runner_cluster_role_actuator_name, 'user': {'token': (github_runner_cluster_role_actuator_secret.stdout | trim | split(' '))[0] | b64decode}}], + 'contexts': [{'name': 'fleet', 'context': {'cluster': 'fleet', 'user': github_runner_cluster_role_actuator_name}}]} | to_nice_yaml }} + dest: "{{ github_runner_cluster_role_actuator_kubeconfig }}" + mode: "0600" + no_log: true + register: github_runner_cluster_role_actuator_kubeconfig_file + + - name: Write the URL a promotion joins each other server and eligible host at + become: true + ansible.builtin.copy: + content: | + {% for candidate in github_runner_cluster_api_candidates %} + {{ candidate.value.node_name }} https://{{ candidate.value.address }}:{{ github_runner_cluster_api_port }} + {% endfor %} + dest: "{{ github_runner_cluster_role_actuator_dir }}/join-urls" + mode: "0644" + register: github_runner_cluster_role_actuator_join_urls + + - name: Read whether the watcher is enabled and running + ansible.builtin.command: + argv: [systemctl, "{{ item }}", "{{ github_runner_cluster_role_actuator_service }}"] + loop: [is-enabled, is-active] + register: github_runner_cluster_role_actuator_state + changed_when: false + failed_when: false + + - name: Enable the watcher + become: true + ansible.builtin.command: + argv: [systemctl, enable, "{{ github_runner_cluster_role_actuator_service }}"] + changed_when: true + when: + - github_runner_cluster_native_start | bool + - github_runner_cluster_role_actuator_state.results[0].rc != 0 + + - name: Start the watcher, or restart it since something it reads changed + # The watcher reads its scripts and settings at its start, and its report and the host hold everything else, so a restart costs nothing; a watcher nothing changed for is left running. + become: true + ansible.builtin.command: + argv: + - systemctl + - "{{ 'restart' if github_runner_cluster_role_actuator_state.results[1].rc == 0 else 'start' }}" + - "{{ github_runner_cluster_role_actuator_service }}" + changed_when: true + when: + - github_runner_cluster_native_start | bool + - >- + github_runner_cluster_role_actuator_state.results[1].rc != 0 + or github_runner_cluster_role_actuator_scripts is changed + or github_runner_cluster_role_actuator_settings is changed + or github_runner_cluster_role_actuator_kubeconfig_file is changed + or github_runner_cluster_role_actuator_join_urls is changed + or github_runner_cluster_role_actuator_unit is changed diff --git a/roles/github_runner_cluster/templates/native/fleet-regrade.service.j2 b/roles/github_runner_cluster/templates/native/fleet-regrade.service.j2 index d423e492..6c21c0d8 100644 --- a/roles/github_runner_cluster/templates/native/fleet-regrade.service.j2 +++ b/roles/github_runner_cluster/templates/native/fleet-regrade.service.j2 @@ -12,6 +12,8 @@ User=root Environment="FLEET_REGRADE_ENV={{ github_runner_cluster_native_regrade_dir }}/fleet-regrade.env" # k3s, for `k3s kubectl`, is in the role's binary directory. Environment="PATH={{ github_runner_cluster_native_bin_dir }}:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" +# Starts only while the role file names this host a server and k3s has not marked its datastore with a tombstone, the server unit's own condition (files/native/fleet-role-is.sh): a host the role actuator has stepped down for a voter swap names the agent from then on, and Restart=always must not start a watcher that would step its server back up. +ExecCondition={{ github_runner_cluster_native_lib_dir }}/fleet-role-is server {{ github_runner_cluster_native_role_file }} {{ github_runner_cluster_native_data_dir }}/server ExecStart={{ github_runner_cluster_native_regrade_dir }}/fleet-regrade.sh watch # The watcher keeps its streaks in memory, so a restart only starts them again; a kernel without memory pressure stops it at once, with the reason in the journal, and the role refuses to turn regrading on there in the first place. It is restarted after one sample period, the pace it reads at in any case. Restart=always diff --git a/roles/github_runner_cluster/templates/native/fleet-role-actuator.service.j2 b/roles/github_runner_cluster/templates/native/fleet-role-actuator.service.j2 new file mode 100644 index 00000000..1a68b2f7 --- /dev/null +++ b/roles/github_runner_cluster/templates/native/fleet-role-actuator.service.j2 @@ -0,0 +1,22 @@ +# Managed by the exadev.github_runner collection's github_runner_cluster role (node mode native, github_runner_cluster_role_controller: true). Do not edit by hand: change this host's variables and run the role again. +# Switches this host between the {{ github_runner_cluster_native_server_service }} and {{ github_runner_cluster_native_agent_service }} units when the role controller asks through the host's Node, for a voter swap; see the role README's Role actuator. It runs fleet-role-actuator with the systemd platform (files/role/fleet-role-actuator.sh and fleet-role-actuator-systemd.sh), which starts and stops those units and writes the role file their conditions read, so it runs as root, as they do. +[Unit] +Description=fleet-role-actuator: switches this native k3s node between server and agent for the role controller +Documentation=https://github.com/ExaDev/github-runner/tree/main/roles/github_runner_cluster#role-actuator +# Ordered after the network and the Tailscale client it reaches the cluster's API through, and bound to neither k3s unit: it stops and starts them itself, and must keep running while it does. +Wants=network-online.target tailscaled.service +After=network-online.target tailscaled.service + +[Service] +Type=exec +User=root +Environment="FLEET_ROLE_ACTUATOR_ENV={{ github_runner_cluster_role_actuator_dir }}/fleet-role-actuator.env" +# k3s, for `k3s kubectl` and `k3s etcd-snapshot`, is in the role's binary directory. +Environment="PATH={{ github_runner_cluster_native_bin_dir }}:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" +ExecStart={{ github_runner_cluster_role_actuator_dir }}/fleet-role-actuator.sh watch +# The watcher keeps nothing in memory that its report and the host do not hold, so a restart takes up a step where it was: a step it had not reported is taken again, and every step can be. +Restart=always +RestartSec=5s + +[Install] +WantedBy=multi-user.target diff --git a/roles/github_runner_cluster/templates/regrade.env.j2 b/roles/github_runner_cluster/templates/regrade.env.j2 index 719b94f1..291a0b44 100644 --- a/roles/github_runner_cluster/templates/regrade.env.j2 +++ b/roles/github_runner_cluster/templates/regrade.env.j2 @@ -25,8 +25,5 @@ NODE_CONTAINER={{ github_runner_cluster_node_container }} KUBECTL="docker exec -i {{ github_runner_cluster_node_container }} kubectl" KUBECONFIG_PATH=/etc/rancher/k3s/k3s.yaml {% endif %} -{# The API servers the service may reach the cluster through, never this one's: at etcd-only this server has none, and its own answers "apiserver disabled" on some servers. Every other server, then every other host that may become one (github_runner_cluster_voter_eligible), each in inventory order and by the address the servers' certificates carry; the core uses the first whose /readyz answers, since the servers can change at run time. #} -{% set github_runner_cluster_regrade_others = github_runner_cluster_topology.hosts | dict2items | rejectattr('key', 'equalto', inventory_hostname) | list %} -{% set github_runner_cluster_regrade_candidates = (github_runner_cluster_regrade_others | selectattr('value.role', 'equalto', 'server') | list) - + (github_runner_cluster_regrade_others | rejectattr('value.role', 'equalto', 'server') | selectattr('value.eligible') | list) %} -KUBE_SERVERS="{% for candidate in github_runner_cluster_regrade_candidates %}{{ ' ' if not loop.first else '' }}https://{{ candidate.value.address }}:{{ github_runner_cluster_api_port }}{% endfor %}" +{# The API servers the service may reach the cluster through, never this one's: at etcd-only this server has none, and its own answers "apiserver disabled" on some servers. github_runner_cluster_api_candidates says which; the core uses the first whose /readyz answers. #} +KUBE_SERVERS="{% for candidate in github_runner_cluster_api_candidates %}{{ ' ' if not loop.first else '' }}https://{{ candidate.value.address }}:{{ github_runner_cluster_api_port }}{% endfor %}" diff --git a/roles/github_runner_cluster/templates/role-actuator-rbac.yaml.j2 b/roles/github_runner_cluster/templates/role-actuator-rbac.yaml.j2 new file mode 100644 index 00000000..d3888a69 --- /dev/null +++ b/roles/github_runner_cluster/templates/role-actuator-rbac.yaml.j2 @@ -0,0 +1,81 @@ +{# The role actuator's credential (tasks/role_actuator.yml applies it): its ServiceAccount, which can read Nodes (a ClusterRole, since a Node is cluster-wide), read the regrade Lease (a Role in its namespace, by name) and read and patch its report ConfigMap (a Role in the platform namespace, by name), and the token Secret the token controller fills for it. Nothing here can write a Node or take the Lease: the controller does every write on the API. The namespace is the ARC role's platform namespace, applied here too since a cluster can run the actuators before the ARC role has run. The ConfigMap is created separately, since applying an empty one again would wipe the reports. #} +{% set name = github_runner_cluster_role_actuator_name %} +{% set namespace = github_runner_cluster_role_actuator_namespace %} +{% set labels = {'app.kubernetes.io/part-of': 'github-runner', 'app.kubernetes.io/name': name} %} +apiVersion: v1 +kind: Namespace +metadata: + name: {{ namespace | to_json }} +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ name | to_json }} + namespace: {{ namespace | to_json }} + labels: {{ labels | to_json }} +--- +apiVersion: v1 +kind: Secret +type: kubernetes.io/service-account-token +metadata: + name: {{ (name ~ '-token') | to_json }} + namespace: {{ namespace | to_json }} + labels: {{ labels | to_json }} + annotations: + kubernetes.io/service-account.name: {{ name | to_json }} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ ('github-runner-' ~ name) | to_json }} + labels: {{ labels | to_json }} +rules: + - apiGroups: [""] + resources: ["nodes"] + verbs: ["get", "list", "watch"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: {{ ('github-runner-' ~ name) | to_json }} + labels: {{ labels | to_json }} +subjects: + - kind: ServiceAccount + name: {{ name | to_json }} + namespace: {{ namespace | to_json }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ ('github-runner-' ~ name) | to_json }} +{% for scope in [ + {'namespace': namespace, 'group': '', 'resource': 'configmaps', 'object': github_runner_cluster_role_actuator_configmap, 'verbs': ['get', 'patch']}, + {'namespace': github_runner_cluster_regrade_lease_namespace, 'group': 'coordination.k8s.io', 'resource': 'leases', 'object': github_runner_cluster_regrade_lease_name, 'verbs': ['get']}, + ] %} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: {{ ('github-runner-' ~ name) | to_json }} + namespace: {{ scope.namespace | to_json }} + labels: {{ labels | to_json }} +rules: + - apiGroups: {{ [scope.group] | to_json }} + resources: {{ [scope.resource] | to_json }} + resourceNames: {{ [scope.object] | to_json }} + verbs: {{ scope.verbs | to_json }} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: {{ ('github-runner-' ~ name) | to_json }} + namespace: {{ scope.namespace | to_json }} + labels: {{ labels | to_json }} +subjects: + - kind: ServiceAccount + name: {{ name | to_json }} + namespace: {{ namespace | to_json }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: {{ ('github-runner-' ~ name) | to_json }} +{% endfor %} diff --git a/roles/github_runner_cluster/templates/role-actuator.env.j2 b/roles/github_runner_cluster/templates/role-actuator.env.j2 new file mode 100644 index 00000000..97826309 --- /dev/null +++ b/roles/github_runner_cluster/templates/role-actuator.env.j2 @@ -0,0 +1,26 @@ +# Generated by the exadev.github_runner collection's github_runner_cluster role on every run. Do not edit by hand: change this host's ansible/host_vars file and rerun ansible-playbook. Read by the role actuator (files/role/fleet-role-actuator.sh), by its watcher unit and by the role's own promotion and demotion of this host; see the role README's Role actuator. It holds no secret: the actuator's credential is in its kubeconfig, which only root reads. +FLEET_ROLE_ACTUATOR=systemd +NODE_NAME={{ github_runner_cluster_effective_node_name }} +ROLE_FILE={{ github_runner_cluster_native_role_file }} +K3S_SERVICE={{ github_runner_cluster_native_server_service }} +K3S_AGENT_SERVICE={{ github_runner_cluster_native_agent_service }} +REGRADE_SERVICE={{ github_runner_cluster_native_regrade_service }} +K3S_DATA_DIR={{ github_runner_cluster_native_data_dir }} +CONFIG_DIR={{ github_runner_cluster_native_config_dir }} +UNIT_DIR={{ github_runner_cluster_native_unit_dir }} +LEVEL_FILE={{ github_runner_cluster_native_regrade_level_file }} +STARTED_FILE={{ github_runner_cluster_native_regrade_started_file }} +{# A step's waits (a restart into etcd-only, the switch lock) are bounded by regrading's, which they are made of. #} +OPERATION_TIMEOUT_SECONDS={{ github_runner_cluster_regrade_operation_timeout_seconds }} +API_PORT={{ github_runner_cluster_api_port }} +LEASE_NAMESPACE={{ github_runner_cluster_regrade_lease_namespace }} +LEASE_NAME={{ github_runner_cluster_regrade_lease_name }} +ROLE_LEASE_HOLDER={{ github_runner_cluster_role_lease_holder }} +SWAP_ID_ANNOTATION={{ github_runner_cluster_role_lease_swap_id_annotation }} +ACTUATOR_NAMESPACE={{ github_runner_cluster_role_actuator_namespace }} +ACTUATOR_CONFIGMAP={{ github_runner_cluster_role_actuator_configmap }} +JOIN_URLS_FILE={{ github_runner_cluster_role_actuator_dir }}/join-urls +KUBECTL="{{ github_runner_cluster_native_bin_dir }}/k3s kubectl" +KUBECONFIG_PATH={{ github_runner_cluster_role_actuator_kubeconfig }} +{# The API servers the actuator reaches the cluster through: never this host's own, which a server stepped down to etcd-only does not have and an agent never has. #} +KUBE_SERVERS="{% for candidate in github_runner_cluster_api_candidates %}{{ ' ' if not loop.first else '' }}https://{{ candidate.value.address }}:{{ github_runner_cluster_api_port }}{% endfor %}" diff --git a/tests/mesh/run.sh b/tests/mesh/run.sh index e5c0372d..8ff979bd 100755 --- a/tests/mesh/run.sh +++ b/tests/mesh/run.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# Mesh integration test for the github_runner_cluster role: stands up three k3s nodes in Docker on one machine, each in its own copy of the Compose project, and has the role join them through Headscale. Scenarios: hosted headscale_hosted: the role runs Headscale under Compose beside the first node. in_cluster headscale_in_cluster: the role runs Headscale inside the cluster, on the bootstrap server. existing headscale_existing: the policy check fails, with the entry to add, against a server whose policy lacks autoApprovers, and passes once it has one. Every scenario uses automatic server selection (three hosts, no overrides, so three servers). native_server three Docker servers and a native host (a privileged systemd container, tests/mesh/native/Dockerfile, managed over Ansible's docker connection) that joins as an agent, is then made a server by the inventory and promoted in place: it must join etcd as a learner, be promoted to a voter with its mesh address as its peer URL, and keep the cluster writable throughout, and a rerun must not restart it. role_controller four hosts (three servers and a standby agent, with etcd's data on memory so the machine's disk does not decide): the role controller's probe and controller run in the cluster from the image built from this checkout, with the manifests the ARC role renders; every host is reported fit from a full window of real samples, with the link the cluster role labelled its node with, a voter made slow with tc netem on its mesh interface is reported unfit with the swap named and every unfit verdict attributed to it, a paused voter makes the controller report that it would not act, and the controller changes nothing (not in the default set; needs jq, and takes a window of 100 samples before it can judge). rejoin three Docker servers whose hosts the inventory marks eligible, so the role mounts each one's role directory: the bootstrap server's etcd member is removed through another server, and restarted on its tombstone it idles instead of joining again; with its datastore moved aside and no server to join it refuses to start rather than bootstrap a second cluster; given a join URL naming another server in its role directory it joins the existing cluster as a learner and is promoted, with the same cluster ID before and after; and a role file naming the agent leaves its server container idle with no k3s. compose_faulty the role refuses a Docker Compose release that recreates the containers it has just built, before writing anything (run it under such a release; it is not in the default set). host_network each node runs in a Docker daemon of its own (a docker:dind container standing in for a VM with an address on the LAN, on the shared test network) with github_runner_cluster_network_mode host, joined through a Headscale server the test runs (headscale_existing); the nodes' Tailscale clients must then reach each other directly at those daemons' addresses, and the role must not publish any port; then the role must refuse to move two running nodes to the bridge network in one run, and move one there and back through its guard, the node keeping its mesh address and etcd membership. Linux only, since the control node reaches each daemon at its address on the test network. The cluster scenarios assert that etcd has three members, that every node is Ready, and that pods on different nodes reach each other over the mesh. +# Mesh integration test for the github_runner_cluster role: stands up three k3s nodes in Docker on one machine, each in its own copy of the Compose project, and has the role join them through Headscale. Scenarios: hosted headscale_hosted: the role runs Headscale under Compose beside the first node. in_cluster headscale_in_cluster: the role runs Headscale inside the cluster, on the bootstrap server. existing headscale_existing: the policy check fails, with the entry to add, against a server whose policy lacks autoApprovers, and passes once it has one. Every scenario uses automatic server selection (three hosts, no overrides, so three servers). native_server three Docker servers and a native host (a privileged systemd container, tests/mesh/native/Dockerfile, managed over Ansible's docker connection) that joins as an agent, is then made a server by the inventory and promoted in place: it must join etcd as a learner, be promoted to a voter with its mesh address as its peer URL, and keep the cluster writable throughout, and a rerun must not restart it. role_controller four hosts (three servers and a standby agent, with etcd's data on memory so the machine's disk does not decide): the role controller's probe and controller run in the cluster from the image built from this checkout, with the manifests the ARC role renders; every host is reported fit from a full window of real samples, with the link the cluster role labelled its node with, a voter made slow with tc netem on its mesh interface is reported unfit with the swap named and every unfit verdict attributed to it, a paused voter makes the controller report that it would not act, and the controller changes nothing (not in the default set; needs jq, and takes a window of 100 samples before it can judge). role_actuator three Docker servers and a native standby, every host eligible and the role controller on, with the test acting as the controller (holding the regrade Lease and writing the Node's desired-role label and swap annotations by hand): the native host's role actuator promotes it (a learner, then a voter, with writes succeeding throughout), steps it down (its etcd serving at etcd-only with no pod's shim left), and, once its member is removed through k3s's annotation, finishes it as an agent with the same node password and Tailscale address and its datastore moved aside, the two k3s units never running together. rejoin three Docker servers whose hosts the inventory marks eligible, so the role mounts each one's role directory: the bootstrap server's etcd member is removed through another server, and restarted on its tombstone it idles instead of joining again; with its datastore moved aside and no server to join it refuses to start rather than bootstrap a second cluster; given a join URL naming another server in its role directory it joins the existing cluster as a learner and is promoted, with the same cluster ID before and after; and a role file naming the agent leaves its server container idle with no k3s. compose_faulty the role refuses a Docker Compose release that recreates the containers it has just built, before writing anything (run it under such a release; it is not in the default set). host_network each node runs in a Docker daemon of its own (a docker:dind container standing in for a VM with an address on the LAN, on the shared test network) with github_runner_cluster_network_mode host, joined through a Headscale server the test runs (headscale_existing); the nodes' Tailscale clients must then reach each other directly at those daemons' addresses, and the role must not publish any port; then the role must refuse to move two running nodes to the bridge network in one run, and move one there and back through its guard, the node keeping its mesh address and etcd membership. Linux only, since the control node reaches each daemon at its address on the test network. The cluster scenarios assert that etcd has three members, that every node is Ready, and that pods on different nodes reach each other over the mesh. # # With GRTEST_PIN_CONTEXT=1, the cluster scenarios run the role, and the in_cluster recovery playbook, with github_runner_cluster_docker_context pinned while the current Docker context they see points at nothing (tests/lib/docker_context.sh), so any docker call that ignored the pin fails the run; afterwards the current context must be unchanged. context_checks the role refuses a pinned context that does not exist, and a pinned context alongside DOCKER_HOST, before writing anything (not in the default set). # @@ -258,6 +258,8 @@ EOF echo " github_runner_cluster_node_mode: native" echo " github_runner_cluster_node_role: ${native_role}" echo " github_runner_cluster_regrade: ${native_regrade}" + # The role_actuator scenario marks the native host a candidate for a vote too, which stages it as a standby server and runs its role actuator. + if [ "${native_eligible:-0}" = 1 ]; then echo " github_runner_cluster_voter_eligible: true"; fi fi } > "${work}/inventory.yml" } @@ -1534,6 +1536,169 @@ scenario_rejoin() { [ "$(docker inspect --format '{{.State.StartedAt}} {{.RestartCount}}' "$(container 1)")" = "$started_before" ] || fail "the idle container restarted" } +# --- The role actuator switching a native host (the role_actuator scenario) --- + +# The most seconds each step of the role_actuator scenario is waited for, the native_server scenario's bound for a learner's catch-up and promotion, and how long the scenario's Lease lasts: one such wait for each of the steps it holds the Lease across (the promotion, the vote, the step-down, the removal and the finish), so it never runs out while the test acts as the controller and renews nothing. +actuator_wait_seconds=600 +actuator_lease_seconds=$((5 * actuator_wait_seconds)) +actuator_prefix=github-runner.exadev.com + +# The native host's last report in the actuators' ConfigMap, or {}. +actuator_report() { + kubectl_on 1 --namespace "$platform_namespace" get configmap fleet-role-actuators -o json 2>/dev/null /dev/null)" = "$1 $2" ]; } +# Whether the native host's Node reports the Ready condition True and its kubelet has renewed the node's Lease (kube-node-lease/) at or after the time $1 (RFC 3339, UTC), as tasks/native/server_ready.yml judges a restart: the API keeps a stopped kubelet's Ready condition until the node controller's grace period runs out, so the condition alone could be the server's from before its step-down. Read from the condition, not the STATUS column, which says Ready,SchedulingDisabled while the node is cordoned. Both times are cut to whole seconds before they are compared, as strings, since RFC 3339 in UTC sorts as it reads. +native_node_ready_since() { + local renewed + [ "$(kubectl_on 1 get node "$native_host" -o 'jsonpath={.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null /dev/null /dev/null /dev/null | grep -c ', started, ' || true; } + +# Holds the regrade Lease as the role controller does for the whole of the swap $1 of the nodes $2. +hold_swap_lease() { + local now + now=$(date -u +%Y-%m-%dT%H:%M:%S.000000Z) + kubectl_on 1 apply -f - >/dev/null < "$log" + until [ -e "$stop" ]; do + if native_unit_is k3s active && native_unit_is k3s-agent active; then echo "$(date -u +%H:%M:%S) k3s and k3s-agent both active" >> "$log"; fi + sleep 1 + done +} + +# The role actuator's native part: three Docker servers and a native standby, every host eligible with the role controller on, and the test acting as the controller, holding the Lease and writing the Node's label and annotations by hand. The actuator promotes the standby (a learner, then a voter, with writes succeeding throughout), steps it down (its etcd serving at etcd-only and no pod's shim left), it is removed through k3s's annotation, and the actuator finishes it as an agent, keeping its node password and Tailscale address and moving its datastore aside, with the two units never running together. +scenario_role_actuator() { + local swap password address writes members stop units writer watcher unit_watcher list ok failed + role_controller=1 + native_eligible=1 + reset_environment + prepare_nodes hosted + start_native_host + native_role=agent + write_inventory hosted + log "Bringing up three Docker servers, then the native host as an eligible agent, all with the role controller on" + run_role --limit "$(node_name 1),$(node_name 2),$(node_name 3)" + wait_for "every Docker server's API to serve and every Docker node to be Ready" 600 docker_servers_serving + run_role --limit "$native_host" + wait_for "${native_host} to be Ready as an agent" 300 native_node_ready + wait_for "${native_host}'s role actuator to run" 120 native_unit_is fleet-role-actuator active + native_exec test -f /etc/rancher/k3s/server.yaml /dev/null + kubectl_on 1 drain "$native_host" --ignore-daemonsets --delete-emptydir-data --timeout="${actuator_wait_seconds}s" >/dev/null + write_throughout "$writes" "$stop" & + writer=$! + watch_native_member "$members" "$stop" & + watcher=$! + kubectl_on 1 label --overwrite node "$native_host" "${actuator_prefix}/desired-role=server" >/dev/null + kubectl_on 1 annotate --overwrite node "$native_host" "${actuator_prefix}/role-swap=${swap}" "${actuator_prefix}/role-step=promote" "${actuator_prefix}/role-join-node=$(node_name 1)" >/dev/null + wait_for "${native_host}'s actuator to report the promotion done" "$actuator_wait_seconds" actuator_reported promote "done" + wait_for "${native_host} to be a voting member of etcd" "$actuator_wait_seconds" native_is_voter + touch "$stop" + wait "$writer" "$watcher" + actuator_report + list=$(etcd_members) + echo "$list" + [ "$(echo "$list" | grep -c ', started, ')" = 4 ] || fail "expected four started etcd members after the promotion" + echo "$list" | grep -F ", ${native_host}-" | grep -Eq ', started, .*, false$' || fail "${native_host}'s member is not a started voter" + local promoted=0 index + for index in 1 2 3; do + if docker logs "$(container "$index")" 2>&1 | grep -F "Promoted learner ${native_host}-" >/dev/null; then promoted=1; fi + done + [ "$promoted" = 1 ] || fail "no server's k3s logged promoting ${native_host}'s learner" + if grep -Eq ', true$' "$members"; then log "The member watch saw ${native_host} as a learner"; else log "The member watch did not catch ${native_host} as a learner (promoted between reads); the promotion is in the servers' logs"; fi + ok=$(grep -c '^ok$' "$writes" || true) + failed=$(grep -c '^failed' "$writes" || true) + echo " ${ok} writes succeeded, ${failed} failed" + [ "$ok" -gt 0 ] || fail "no write was made during the promotion" + [ "$failed" = 0 ] || { grep '^failed' "$writes"; fail "writes failed during the promotion"; } + native_unit_is k3s active || fail "${native_host} does not run k3s" + native_unit_is k3s-agent inactive || fail "${native_host} still runs k3s-agent" + [ "$(native_exec cat /var/lib/rancher/k3s/fleet-role /dev/null || fail "the join drop-in does not name $(node_name 1)" + native_exec grep -Fx "Environment=\"GITHUB_RUNNER_JOIN_HOST=$(node_name 1).mesh.grtest.internal\"" /etc/systemd/system/k3s.service.d/20-fleet-role-join.conf /dev/null || fail "the unit's join drop-in does not name $(node_name 1)" + kubectl_on 1 uncordon "$native_host" >/dev/null + + log "Stepping ${native_host} down, as the controller asks: drained, labelled etcd-only, desired-role agent" + kubectl_on 1 taint --overwrite node "$native_host" node-role.kubernetes.io/control-plane=true:NoSchedule >/dev/null + kubectl_on 1 drain "$native_host" --ignore-daemonsets --delete-emptydir-data --timeout="${actuator_wait_seconds}s" >/dev/null + kubectl_on 1 label --overwrite node "$native_host" "${actuator_prefix}/regrade-level=etcd-only" "${actuator_prefix}/desired-role=agent" >/dev/null + kubectl_on 1 annotate --overwrite node "$native_host" "${actuator_prefix}/role-step=step-down" >/dev/null + wait_for "${native_host}'s actuator to report the step-down done" "$actuator_wait_seconds" actuator_reported step-down "done" + actuator_report + [ "$(native_exec cat /run/github-runner-k3s-server/regrade-level /dev/null + wait_for "k3s to record the removal of ${native_host}'s member" "$actuator_wait_seconds" native_member_removed + wait_for "${native_host}'s actuator to report the demotion finished" "$actuator_wait_seconds" actuator_reported finish-demote "done" + actuator_report + [ "$(started_member_count)" = 3 ] || { etcd_members; fail "expected three started etcd members after the removal"; } + [ "$(etcd_members | wc -l | tr -d ' ')" = 3 ] || { etcd_members; fail "the removed member is still listed"; } + native_unit_is k3s inactive || fail "${native_host} still runs k3s" + native_unit_is k3s-agent active || fail "${native_host} does not run k3s-agent" + native_exec test -d /var/lib/rancher/k3s/server/db.demoted /dev/null + kubectl_on 1 annotate node "$native_host" etcd.k3s.cattle.io/remove- etcd.k3s.cattle.io/removed-node-name- etcd.k3s.cattle.io/node-name- etcd.k3s.cattle.io/node-address- \ + "${actuator_prefix}/role-swap-" "${actuator_prefix}/role-step-" "${actuator_prefix}/role-join-node-" >/dev/null + kubectl_on 1 taint node "$native_host" node-role.kubernetes.io/control-plane=true:NoSchedule- >/dev/null + wait_for "${native_host} to be Ready as an agent again" "$actuator_wait_seconds" native_node_ready_since "$(actuator_report | jq -r .started_at)" + kubectl_on 1 uncordon "$native_host" >/dev/null + kubectl_on 1 delete lease fleet-regrade --namespace kube-system >/dev/null + touch "$stop" + wait "$unit_watcher" + [ ! -s "$units" ] || { cat "$units"; fail "${native_host} ran both k3s units at once"; } + [ "$(kubectl_on 1 get nodes --no-headers | grep -c "^${native_host} ")" = 1 ] || fail "${native_host} is not exactly one Node" +} + scenarios=("$@") [ "${#scenarios[@]}" -gt 0 ] || scenarios=(existing hosted in_cluster) for scenario in "${scenarios[@]}"; do @@ -1545,6 +1710,7 @@ for scenario in "${scenarios[@]}"; do native_server) scenario_native_server ;; role_controller) scenario_role_controller ;; rejoin) scenario_rejoin ;; + role_actuator) scenario_role_actuator ;; host_network) scenario_host_network ;; compose_faulty) scenario_compose_faulty ;; context_checks) scenario_context_checks ;; diff --git a/tests/role_actuator/fake-k3s.sh b/tests/role_actuator/fake-k3s.sh new file mode 100644 index 00000000..f371f0c5 --- /dev/null +++ b/tests/role_actuator/fake-k3s.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# A stand-in for k3s for fleet-role-actuator's tests: `k3s kubectl` is the kubectl on the PATH (the test makes it tests/role_actuator/fake-kubectl.sh), and `k3s etcd-snapshot save` records its arguments in FAKE_HOST/snapshots and logs, as k3s logs through logrus, the one file the server created, named as k3s names an on-demand snapshot (the given name, the node and a time). FAKE_SNAPSHOT_FAILS makes the save fail as k3s does when the server refuses it. +set -u + +host=${FAKE_HOST:?FAKE_HOST must be set} +case "${1:-} ${2:-}" in + "kubectl "*) + shift + exec kubectl "$@" + ;; + "etcd-snapshot save") + shift 2 + echo "$*" >> "$host/snapshots" + if [ -n "${FAKE_SNAPSHOT_FAILS:-}" ]; then + echo 'time="2026-01-01T00:00:00Z" level=fatal msg="see server log for details: Internal Server Error"' >&2 + exit 1 + fi + name='' + while [ $# -gt 0 ]; do + if [ "$1" = --name ]; then name=$2; fi + shift + done + echo "time=\"2026-01-01T00:00:00Z\" level=info msg=\"Snapshot ${name}-testnode-1767225600 saved.\"" >&2 + ;; + *) + echo "unexpected k3s call: $*" >&2 + exit 2 + ;; +esac diff --git a/tests/role_actuator/fake-kubectl.sh b/tests/role_actuator/fake-kubectl.sh new file mode 100644 index 00000000..e07b3649 --- /dev/null +++ b/tests/role_actuator/fake-kubectl.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +# A stand-in for kubectl for fleet-role-actuator's tests, keeping a cluster in the directory FAKE_DIR: node.json (this host's Node, as `kubectl get node -o json` prints it) and configmap.json (the actuators' report ConfigMap, absent when there is none), beside what tests/regrade/fake-kubectl.sh keeps (lease.json, unready), to which every other call goes (FAKE_REGRADE_KUBECTL), so the Lease and the API servers' readiness are answered as they are for the regrade core. A patch merges into configmap.json as the API server does for --type merge. Every call is recorded, as that stand-in records them, in FAKE_DIR/calls. FAKE_PATCH_FAILS makes every patch fail. +set -u + +dir=${FAKE_DIR:?FAKE_DIR must be set} +original=("$@") +args=() +while [ $# -gt 0 ]; do + case $1 in + --kubeconfig | --server) shift 2 ;; + --request-timeout=*) shift ;; + *) args+=("$1"); shift ;; + esac +done + +case "${args[0]:-} ${args[1]:-}" in + "get node") + echo "${args[*]}" >> "$dir/calls" + cat "$dir/node.json" + ;; + "get configmap") + echo "${args[*]}" >> "$dir/calls" + if [ -f "$dir/configmap.json" ]; then + cat "$dir/configmap.json" + else + echo "Error from server (NotFound): configmaps \"${args[2]}\" not found" >&2 + exit 1 + fi + ;; + "patch configmap") + echo "${args[*]}" >> "$dir/calls" + [ -z "${FAKE_PATCH_FAILS:-}" ] || { echo "Unable to connect to the server" >&2; exit 1; } + [ -f "$dir/configmap.json" ] || { echo "Error from server (NotFound): configmaps \"${args[2]}\" not found" >&2; exit 1; } + patch='' + for index in "${!args[@]}"; do + if [ "${args[$index]}" = --patch ]; then patch=${args[$((index + 1))]}; fi + done + jq --argjson patch "$patch" '.data = ((.data // {}) + ($patch.data // {}))' "$dir/configmap.json" > "$dir/configmap.json.new" + mv "$dir/configmap.json.new" "$dir/configmap.json" + echo "configmap/${args[2]} patched" + ;; + *) exec "${FAKE_REGRADE_KUBECTL:?}" "${original[@]}" ;; +esac diff --git a/tests/role_actuator/fake-systemctl.sh b/tests/role_actuator/fake-systemctl.sh new file mode 100644 index 00000000..b87bad4f --- /dev/null +++ b/tests/role_actuator/fake-systemctl.sh @@ -0,0 +1,98 @@ +#!/usr/bin/env bash +# A stand-in for systemctl for fleet-role-actuator's tests, keeping a native host's units in the directory FAKE_HOST. The k3s server unit is tests/regrade/fake-systemctl.sh's (FAKE_REGRADE_SYSTEMCTL, which this file runs for it): `active` while it runs, its invocations and journal, and its ExecStartPre (FAKE_PRESTART) run at every start. Every other unit is `active-` while it runs and `enabled-` while it is enabled. A start of either k3s unit first runs its ExecCondition, the real files/native/fleet-role-is.sh (FAKE_ROLE_IS) with the role file FAKE_ROLE_FILE and the server directory FAKE_SERVER_DIR as the units pass them, and is skipped, as systemd skips it, when that fails; a start that would leave both k3s units running is recorded in FAKE_HOST/both-running. Each call is recorded in FAKE_HOST/units, and each start as ` started` or ` skipped by its condition`. +set -u + +host=${FAKE_HOST:?FAKE_HOST must be set} +echo "$*" >> "$host/units" + +active() { + if [ "$1" = k3s ]; then [ -f "$host/active" ]; else [ -f "$host/active-$1" ]; fi +} + +# What systemd does for a start of the unit $1: nothing for one already running, its condition, then the unit. +start() { + local unit=$1 other + active "$unit" && return 0 + case $unit in + k3s) + if ! "${FAKE_ROLE_IS:?}" server "${FAKE_ROLE_FILE:?}" "${FAKE_SERVER_DIR:?}" 2> /dev/null; then + echo "$unit skipped by its condition" >> "$host/units" + return 0 + fi + other=k3s-agent + ;; + k3s-agent) + if ! "$FAKE_ROLE_IS" agent "$FAKE_ROLE_FILE" 2> /dev/null; then + echo "$unit skipped by its condition" >> "$host/units" + return 0 + fi + other=k3s + ;; + *) other='' ;; + esac + if [ -n "$other" ] && active "$other"; then echo "$unit started while $other runs" >> "$host/both-running"; fi + if [ "$unit" = k3s ]; then + "${FAKE_REGRADE_SYSTEMCTL:?}" start k3s || return + else + touch "$host/active-$unit" + fi + echo "$unit started" >> "$host/units" +} + +stop() { + if [ "$1" = k3s ]; then + active k3s || return 0 + "$FAKE_REGRADE_SYSTEMCTL" stop k3s + else + rm -f "$host/active-$1" + fi +} + +command=${1:-} +shift +now='' +quiet='' +units=() +for argument in "$@"; do + case $argument in + --now) now=1 ;; + --quiet) quiet=1 ;; + *) units+=("$argument") ;; + esac +done +unit=${units[0]:-} + +case $command in + is-active) + if active "$unit"; then [ -n "$quiet" ] || echo active; exit 0; fi + [ -n "$quiet" ] || echo inactive + exit 3 + ;; + is-enabled) [ -f "$host/enabled-$unit" ] ;; + enable) + touch "$host/enabled-$unit" + [ -z "$now" ] || start "$unit" + ;; + disable) + rm -f "$host/enabled-$unit" + [ -z "$now" ] || stop "$unit" + ;; + start) start "$unit" ;; + stop) stop "$unit" ;; + restart) + if [ "$unit" = k3s ]; then + # systemd runs the condition of a restart's start too: a server whose role file no longer names it stops and stays down. + stop k3s + start k3s + else + stop "$unit" + start "$unit" + fi + ;; + daemon-reload) ;; + show) exec "$FAKE_REGRADE_SYSTEMCTL" show "$@" ;; + *) + echo "unexpected systemctl call: $command $*" >&2 + exit 2 + ;; +esac diff --git a/tests/unit/native_stand_ins.py b/tests/unit/native_stand_ins.py index a76d48e7..596a64ce 100644 --- a/tests/unit/native_stand_ins.py +++ b/tests/unit/native_stand_ins.py @@ -30,6 +30,8 @@ K3S_TAG = "v1.34.1-k3s1" K3S_RELEASE = "v1.34.1+k3s1" TAILSCALE_VERSION = "1.102.4" +# The token the role actuator's token Secret holds, which nothing the role prints may show. +ACTUATOR_TOKEN = "fleet-role-actuator-token-example-0123456789" FACTS = {"system": "Linux", "architecture": "x86_64", "os_family": "Debian", "distribution": "Ubuntu", "distribution_release": "plucky", "service_mgr": "systemd"} @@ -196,10 +198,11 @@ def make_stand_ins(self) -> None: esac done """) - # `docker exec kubectl ...` reaches the same kubectl stand-in as `k3s kubectl ...`. + # `docker exec kubectl ...` reaches the same kubectl stand-in as `k3s kubectl ...`. As Docker does, it passes its own standard input to the command only with --interactive, and the command otherwise reads an empty one. stand_in(bin_dir / "docker", log + r"""case "$*" in "ps "*) if [ -e "$STUB/state/docker-stopped" ]; then echo exited; else echo running; fi ;; - "exec "*" kubectl "*) shift 3; exec "$STUB/bin/kubectl" "$@" ;; + "exec --interactive "*" kubectl "*) shift 4; exec "$STUB/bin/kubectl" "$@" ;; + "exec "*" kubectl "*) shift 3; exec "$STUB/bin/kubectl" "$@" < /dev/null ;; *) echo "docker stand-in: unexpected $*" >&2; exit 1 ;; esac """) @@ -218,9 +221,18 @@ def make_stand_ins(self) -> None: "get lease "*) if [ -e "$state/lease" ]; then cat "$state/lease"; else echo 2099-01-01T00:00:00Z; fi ;; "get pods "*) [ -e "$state/runner-pods" ] && cat "$state/runner-pods"; exit 0 ;; cordon*|uncordon*|drain*|label*|annotate*) ;; + # The role actuator's credential: what is applied is kept for the test to read, the report ConfigMap exists once created, and its token Secret, filled as the token controller fills one, holds ACTUATOR_TOKEN. + "apply --filename -"*) + cat > "$state/applied" + # As kubectl refuses a manifest with no objects in it. + if ! grep -q '[^[:space:]]' "$state/applied"; then echo "error: no objects passed to apply" >&2; exit 1; fi ;; + "create configmap "*) + if [ -e "$state/configmap-created" ]; then echo "Error from server (AlreadyExists): configmaps \"$3\" already exists" >&2; exit 1; fi + touch "$state/configmap-created" ;; + "get secret "*) printf '%s %s' "$(printf '%s' "{ACTUATOR_TOKEN}" | base64)" "$(printf 'cluster-ca' | base64)" ;; *) echo "kubectl stand-in: unexpected $*" >&2; exit 1 ;; esac -""") +""".replace("{ACTUATOR_TOKEN}", ACTUATOR_TOKEN)) def set_tailscale(self, backend: str | None, domain: str | None) -> None: path = self.state / "tailscale.json" @@ -241,7 +253,7 @@ def set_package(self, version: str | None) -> None: (self.state / "enabled-tailscaled").touch() (self.state / "active-tailscaled").touch() - def inventory(self, only_the_node: bool = False, second_server: bool = False, **host_vars: Any) -> Path: + def inventory(self, only_the_node: bool = False, second_server: bool = False, group_vars: dict[str, Any] | None = None, **host_vars: Any) -> Path: shared = { "ansible_connection": "local", "ansible_become": False, @@ -254,6 +266,7 @@ def inventory(self, only_the_node: bool = False, second_server: bool = False, ** "github_runner_cluster_native_tailscale_cli": str(self.stubs / "bin" / "tailscale"), "github_runner_cluster_native_ready_timeout": 20, **{name: str(path) for name, path in self.paths.items()}, + **(group_vars or {}), } node = {**shared, "github_runner_cluster_node_mode": "native", **host_vars} hosts = {"laptop": node} if only_the_node else {"server1": dict(shared), **({"server2": dict(shared)} if second_server else {}), "laptop": node} @@ -262,7 +275,7 @@ def inventory(self, only_the_node: bool = False, second_server: bool = False, ** path.write_text(yaml.safe_dump(inventory)) return path - def run_role(self, *args: str, host_vars: dict[str, Any] | None = None, check: bool = False, only_the_node: bool = False, second_server: bool = False, **extra: Any) -> subprocess.CompletedProcess[str]: + def run_role(self, *args: str, host_vars: dict[str, Any] | None = None, check: bool = False, only_the_node: bool = False, second_server: bool = False, group_vars: dict[str, Any] | None = None, **extra: Any) -> subprocess.CompletedProcess[str]: env = { **os.environ, "PATH": f"{self.stubs / 'bin'}:{os.environ['PATH']}", @@ -277,7 +290,7 @@ def run_role(self, *args: str, host_vars: dict[str, Any] | None = None, check: b "ANSIBLE_RETRY_FILES_ENABLED": "false", **self.network, } - command = [ANSIBLE_PLAYBOOK, "-i", str(self.inventory(only_the_node, second_server, **(host_vars or {}))), str(RUN), "-e", json.dumps({"host_facts": self.facts, **extra}), *args] + command = [ANSIBLE_PLAYBOOK, "-i", str(self.inventory(only_the_node, second_server, group_vars, **(host_vars or {}))), str(RUN), "-e", json.dumps({"host_facts": self.facts, **extra}), *args] if check: command.append("--check") return subprocess.run(command, env=env, capture_output=True, text=True, timeout=300, check=False, cwd=ROOT / "ansible") diff --git a/tests/unit/test_native_node.py b/tests/unit/test_native_node.py index e5fef792..41324c49 100644 --- a/tests/unit/test_native_node.py +++ b/tests/unit/test_native_node.py @@ -371,7 +371,7 @@ class TemplateTest(unittest.TestCase): def test_the_units_and_configurations_are_templates_the_role_has(self) -> None: templates = sorted(path.name for path in (ROLE / "templates" / "native").iterdir()) - self.assertEqual(templates, ["agent.yaml.j2", "fleet-regrade.service.j2", "join.yaml.j2", "k3s-agent.service.j2", "k3s.service.j2", "server.yaml.j2"]) + self.assertEqual(templates, ["agent.yaml.j2", "fleet-regrade.service.j2", "fleet-role-actuator.service.j2", "join.yaml.j2", "k3s-agent.service.j2", "k3s.service.j2", "server.yaml.j2"]) config = (ROLE / "tasks" / "native" / "config.yml").read_text() for name in ("agent.yaml.j2", "server.yaml.j2", "join.yaml.j2", "native/{{ item }}.service.j2"): self.assertIn(name, config) diff --git a/tests/unit/test_native_server.py b/tests/unit/test_native_server.py index 0a49aa5d..961502ce 100644 --- a/tests/unit/test_native_server.py +++ b/tests/unit/test_native_server.py @@ -98,12 +98,13 @@ def test_an_agent_is_promoted_in_place_once_its_node_is_empty(self) -> None: next(call for call in calls if call.startswith("kubectl get pods --all-namespaces --selector actions-ephemeral-runner=True --field-selector spec.nodeName=laptop,status.phase!=Succeeded,status.phase!=Failed")), next(call for call in calls if call.startswith("kubectl drain laptop --ignore-daemonsets --delete-emptydir-data")), "systemctl disable --now k3s-agent", - "systemctl start k3s", + # The actuator's promote-local starts the server unit, as a swap's promotion does. + "systemctl enable --now k3s", "kubectl uncordon laptop", ] self.assertEqual([call for call in calls if call in order], order) self.assertEqual(self.role_file(), "server") - self.assertEqual(self.starts()[-1], "start k3s server") + self.assertEqual(self.starts()[-1], "enable k3s server") self.assertFalse((self.state / "active-k3s-agent").exists()) self.assertFalse((self.state / "enabled-k3s-agent").exists()) self.assertEqual(password.read_text(), "node-password") @@ -179,9 +180,10 @@ def test_a_server_is_demoted_once_its_member_is_removed_with_its_datastore_moved "kubectl cordon laptop", "kubectl annotate --overwrite node laptop etcd.k3s.cattle.io/remove=true", "systemctl disable --now k3s", + # The actuator's demote-local starts the agent before the Node is cleaned, as a swap's controller cleans it after the finish; the agent's k3s puts none of the keys back. + "systemctl enable --now k3s-agent", "kubectl label node laptop node-role.kubernetes.io/control-plane- node-role.kubernetes.io/etcd- node-role.kubernetes.io/master-", "kubectl annotate node laptop etcd.k3s.cattle.io/remove- etcd.k3s.cattle.io/removed-node-name- etcd.k3s.cattle.io/node-name- etcd.k3s.cattle.io/node-address-", - "systemctl enable --now k3s-agent", "kubectl uncordon laptop", ] self.assertEqual([call for call in calls if call in order], order) diff --git a/tests/unit/test_role_actuator.py b/tests/unit/test_role_actuator.py new file mode 100644 index 00000000..45b4d8ba --- /dev/null +++ b/tests/unit/test_role_actuator.py @@ -0,0 +1,637 @@ +"""Tests the role actuator (roles/github_runner_cluster/files/role): its commands with the systemd platform against stand-ins that keep a native host's units and a cluster in directories (tests/role_actuator, over tests/regrade's), and the role's setup of it on a native host (tasks/role_actuator.yml) against the stand-ins in native_stand_ins.py. The role's own promotion and demotion, which run the actuator's promote-local and demote-local, are tested in test_native_server.py and test_native_regrade.py.""" + +from __future__ import annotations + +import base64 +import fcntl +import json +import os +import re +import shutil +import stat +import subprocess +import tempfile +import time +import unittest +from datetime import UTC, datetime +from pathlib import Path +from typing import Any + +import yaml +from native_stand_ins import ACTUATOR_TOKEN, DOMAIN, ROLE, ROOT, NativeRoleHarness + +FILES = ROLE / "files" +FAKES = ROOT / "tests" / "role_actuator" +REGRADE_FAKES = ROOT / "tests" / "regrade" +NODE = "testnode" +SWAP = "20260101T000000Z-server3-testnode" +HOLDER = "fleet-role" +SWAP_ID_ANNOTATION = "github-runner.exadev.com/swap-id" +NAMESPACE = "github-runner-platform" +CONFIGMAP = "fleet-role-actuators" +JOIN_URL = f"https://server1.{DOMAIN}:6443" +# The exit statuses the actuator documents. +DONE, FAILED, REFUSED, NOT_YET = 0, 1, 2, 3 +# Long enough that the Lease a test holds never runs out during it. +LEASE_SECONDS = 600 +# How often the watcher reads its Node: POLL_SECONDS in fleet-regrade-k3s.sh, which the actuator sources. +_POLL = re.search(r"^POLL_SECONDS=(\d+)$", (FILES / "regrade" / "fleet-regrade-k3s.sh").read_text(), re.MULTILINE) +if _POLL is None: + raise AssertionError("fleet-regrade-k3s.sh sets no POLL_SECONDS") +WATCH_POLL_SECONDS = int(_POLL.group(1)) + + +class ActuatorHarness(unittest.TestCase): + """Runs the actuator's commands on a stand-in native host. Holds no tests of its own.""" + + def setUp(self) -> None: + self.tmp = Path(tempfile.mkdtemp(prefix="role-actuator-")) + self.addCleanup(shutil.rmtree, self.tmp, True) + self.lib = self.tmp / "lib" + self.bin = self.tmp / "bin" + self.host = self.tmp / "host" + self.cluster = self.tmp / "cluster" + self.data = self.tmp / "data" + self.config = self.tmp / "config" + self.units = self.tmp / "units" + self.run_dir = self.tmp / "run" + for path in (self.lib, self.bin, self.host, self.cluster, self.data, self.config / "server.yaml.d", self.units): + path.mkdir(parents=True) + # The scripts as the role places them, side by side (tasks/native/config.yml). + for source in ("role/fleet-role-actuator.sh", "role/fleet-role-actuator-systemd.sh", "regrade/fleet-regrade-core.sh", "regrade/fleet-regrade-k3s.sh", "regrade/fleet-regrade-systemd.sh"): + shutil.copy(FILES / source, self.lib / Path(source).name) + for name, source in {"systemctl": FAKES / "fake-systemctl.sh", "kubectl": FAKES / "fake-kubectl.sh", "k3s": FAKES / "fake-k3s.sh", "journalctl": REGRADE_FAKES / "fake-journalctl.sh", "date": REGRADE_FAKES / "fake-date.sh"}.items(): + shutil.copy(source, self.bin / name) + (self.bin / name).chmod(0o755) + self.level_file = self.data / "regrade-level" + self.started_file = self.run_dir / "regrade-level" + self.role_file = self.data / "fleet-role" + settings = { + "FLEET_ROLE_ACTUATOR": "systemd", + "NODE_NAME": NODE, + "ROLE_FILE": self.role_file, + "K3S_SERVICE": "k3s", + "K3S_AGENT_SERVICE": "k3s-agent", + "REGRADE_SERVICE": "fleet-regrade", + "K3S_DATA_DIR": self.data, + "CONFIG_DIR": self.config, + "UNIT_DIR": self.units, + "LEVEL_FILE": self.level_file, + "STARTED_FILE": self.started_file, + "OPERATION_TIMEOUT_SECONDS": 2, + "API_PORT": 6443, + "LEASE_NAMESPACE": "kube-system", + "LEASE_NAME": "fleet-regrade", + "ROLE_LEASE_HOLDER": HOLDER, + "SWAP_ID_ANNOTATION": SWAP_ID_ANNOTATION, + "ACTUATOR_NAMESPACE": NAMESPACE, + "ACTUATOR_CONFIGMAP": CONFIGMAP, + "JOIN_URLS_FILE": self.lib / "join-urls", + "KUBECTL": '"k3s kubectl"', + "KUBECONFIG_PATH": self.config / "fleet-role-actuator.kubeconfig", + "KUBE_SERVERS": f'"https://server1.{DOMAIN}:6443 https://server2.{DOMAIN}:6443"', + } + (self.lib / "fleet-role-actuator.env").write_text("".join(f"{key}={value}\n" for key, value in settings.items())) + (self.lib / "join-urls").write_text(f"server1 {JOIN_URL}\nserver2 https://server2.{DOMAIN}:6443\n") + self.env = { + **os.environ, + "PATH": f"{self.bin}:{os.environ['PATH']}", + "FAKE_HOST": str(self.host), + "FAKE_DIR": str(self.cluster), + "FAKE_REGRADE_SYSTEMCTL": str(REGRADE_FAKES / "fake-systemctl.sh"), + "FAKE_REGRADE_KUBECTL": str(REGRADE_FAKES / "fake-kubectl.sh"), + "FAKE_ROLE_IS": str(FILES / "native" / "fleet-role-is.sh"), + "FAKE_ROLE_FILE": str(self.role_file), + "FAKE_SERVER_DIR": str(self.data / "server"), + "FAKE_RUNTIME_DIR": str(self.run_dir), + "FAKE_SHIM_PREFIX": str(self.data / "data") + "/", + # What the server unit's ExecStartPre runs, as templates/native/k3s.service.j2 writes it. + "FAKE_PRESTART": f"{FILES / 'native' / 'write-regrade-level.sh'} {self.level_file} {self.config / 'server.yaml.d' / '50-regrade.yaml'} {self.started_file}", + } + (self.config / "server.yaml").write_text("token-file: token\n") + (self.cluster / "configmap.json").write_text(json.dumps({"metadata": {"name": CONFIGMAP}, "data": {}})) + self.set_node() + + # --- The host --- + + def running_agent(self) -> None: + self.role_file.write_text("agent") + (self.host / "active-k3s-agent").touch() + (self.host / "enabled-k3s-agent").touch() + + def running_server(self, level: str = "full") -> None: + """A server running at ``level`` through its real ExecStartPre, with a datastore whose member is named, as k3s leaves one.""" + self.role_file.write_text("server") + self.level_file.write_text(level + "\n") + (self.data / "server" / "db" / "etcd").mkdir(parents=True) + (self.data / "server" / "db" / "etcd" / "name").write_text(f"{NODE}-0a1b2c3d") + (self.data / "server" / "db" / "etcd" / "member").mkdir() + (self.host / "enabled-k3s").touch() + self.systemctl("start", "k3s") + + def systemctl(self, *args: str) -> None: + subprocess.run([str(self.bin / "systemctl"), *args], env=self.env, check=True, capture_output=True) + + def active(self, unit: str) -> bool: + return (self.host / ("active" if unit == "k3s" else f"active-{unit}")).exists() + + def assert_never_both_running(self) -> None: + both = self.host / "both-running" + self.assertFalse(both.exists(), both.read_text() if both.exists() else "") + + # --- The cluster --- + + def set_node(self, labels: dict[str, str] | None = None, annotations: dict[str, str] | None = None) -> None: + (self.cluster / "node.json").write_text(json.dumps({"metadata": {"name": NODE, "labels": labels or {}, "annotations": annotations or {}}})) + + def hold_lease(self, holder: str = HOLDER, swap: str = SWAP, renewed: float | None = None) -> None: + stamp = datetime.fromtimestamp(time.time() if renewed is None else renewed, UTC).strftime("%Y-%m-%dT%H:%M:%S.000000Z") + lease = {"metadata": {"name": "fleet-regrade", "namespace": "kube-system", "resourceVersion": "1", "annotations": {SWAP_ID_ANNOTATION: swap}}, "spec": {"holderIdentity": holder, "leaseDurationSeconds": LEASE_SECONDS, "acquireTime": stamp, "renewTime": stamp}} + (self.cluster / "lease.json").write_text(json.dumps(lease)) + + def no_api(self) -> None: + (self.cluster / "unready").write_text(f"https://server1.{DOMAIN}:6443\nhttps://server2.{DOMAIN}:6443\n") + + def report(self) -> dict[str, Any]: + data = json.loads((self.cluster / "configmap.json").read_text())["data"] + return json.loads(data[NODE]) if NODE in data else {} + + def patches(self) -> int: + calls = self.cluster / "calls" + return sum(1 for line in calls.read_text().splitlines() if line.startswith("patch configmap")) if calls.exists() else 0 + + # --- Running it --- + + def actuator(self, *args: str, **env: str) -> subprocess.CompletedProcess[str]: + return subprocess.run(["bash", str(self.lib / "fleet-role-actuator.sh"), *args], env={**self.env, **env}, capture_output=True, text=True, timeout=120, check=False) + + def assert_status(self, result: subprocess.CompletedProcess[str], status: int) -> None: + self.assertEqual(result.returncode, status, result.stdout + result.stderr) + + +class PromoteTest(ActuatorHarness): + def test_a_promotion_joins_the_named_voter_and_starts_the_server_unit_in_place_of_the_agent(self) -> None: + self.running_agent() + self.hold_lease() + self.set_node({"github-runner.exadev.com/desired-role": "server"}, {"github-runner.exadev.com/role-join-node": "server1"}) + result = self.actuator("promote", SWAP) + self.assert_status(result, DONE) + self.assertEqual(self.role_file.read_text(), "server") + self.assertEqual(yaml.safe_load((self.config / "server.yaml.d" / "20-fleet-role-join.yaml").read_text()), {"cluster-init": False, "server": JOIN_URL}) + dropin = (self.units / "k3s.service.d" / "20-fleet-role-join.conf").read_text() + self.assertIn(f'Environment="GITHUB_RUNNER_JOIN_HOST=server1.{DOMAIN}"', dropin.splitlines()) + self.assertTrue(self.active("k3s")) + self.assertFalse(self.active("k3s-agent")) + self.assertFalse((self.host / "enabled-k3s-agent").exists()) + units = (self.host / "units").read_text().splitlines() + # The agent stops before the server starts, and systemd reloads the unit's new drop-in before the start. + self.assertLess(units.index("disable --now k3s-agent"), units.index("k3s started")) + self.assertLess(units.index("daemon-reload"), units.index("k3s started")) + self.assert_never_both_running() + report = self.report() + self.assertEqual({key: report[key] for key in ("swap", "step", "phase", "role")}, {"swap": SWAP, "step": "promote", "phase": "done", "role": "server"}) + self.assertRegex(report["started_at"], r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$") + + def test_a_promotion_taken_again_starts_nothing_and_reports_the_start_it_found(self) -> None: + self.running_agent() + self.hold_lease() + self.set_node({"github-runner.exadev.com/desired-role": "server"}, {"github-runner.exadev.com/role-join-node": "server1"}) + self.assert_status(self.actuator("promote", SWAP), DONE) + first = self.report()["started_at"] + before = (self.host / "units").read_text() + self.assert_status(self.actuator("promote", SWAP), DONE) + self.assertNotIn("start", (self.host / "units").read_text()[len(before):].replace("is-active", "")) + self.assertEqual(self.report()["started_at"], first) + + def test_a_promotion_without_the_lease_for_its_swap_is_refused_and_changes_nothing(self) -> None: + cases = { + "no lease": (None, "the regrade Lease cannot be read"), + "free": ({"holder": ""}, "the regrade Lease is free, not held by fleet-role"), + "another holder": ({"holder": "server2"}, "the regrade Lease is held by server2, not fleet-role"), + "expired": ({"renewed": time.time() - 2 * LEASE_SECONDS}, "the regrade Lease held by fleet-role has expired"), + "a stale swap id": ({"swap": "20250101T000000Z-server2-testnode"}, f"the regrade Lease is held for swap '20250101T000000Z-server2-testnode', not {SWAP}"), + } + for case, (lease, message) in cases.items(): + with self.subTest(case=case): + self.setUp() + self.running_agent() + if lease is not None: + self.hold_lease(**lease) + self.set_node({"github-runner.exadev.com/desired-role": "server"}, {"github-runner.exadev.com/role-join-node": "server1"}) + result = self.actuator("promote", SWAP) + self.assert_status(result, REFUSED) + self.assertIn(message, self.report()["message"]) + self.assertEqual(self.report()["phase"], "refused") + self.assertEqual(self.role_file.read_text(), "agent") + self.assertTrue(self.active("k3s-agent")) + self.assertFalse((self.units / "k3s.service.d").exists()) + self.assertEqual([line for line in (self.host / "units").read_text().splitlines() if not line.startswith("is-active")] if (self.host / "units").exists() else [], []) + + def test_a_promotion_of_a_host_that_cannot_join_cleanly_is_refused(self) -> None: + cases = { + "not provisioned": ("no-config", "has no server configuration"), + "a datastore no demotion moved aside": ("datastore", "holds a datastore no demotion moved aside"), + "a join node with no URL": ("unknown-join", "the node to join, 'server9', is not one"), + "a step that is not a swap's": ("bad-id", "'not-a-swap' is not a swap id"), + } + for case, (fault, message) in cases.items(): + with self.subTest(case=case): + self.setUp() + self.running_agent() + self.hold_lease(swap="not-a-swap" if fault == "bad-id" else SWAP) + self.set_node({"github-runner.exadev.com/desired-role": "server"}, {"github-runner.exadev.com/role-join-node": "server9" if fault == "unknown-join" else "server1"}) + if fault == "no-config": + (self.config / "server.yaml").unlink() + if fault == "datastore": + (self.data / "server" / "db").mkdir(parents=True) + result = self.actuator("promote", "not-a-swap" if fault == "bad-id" else SWAP) + self.assert_status(result, REFUSED) + self.assertIn(message, result.stdout) + self.assertEqual(self.role_file.read_text(), "agent") + self.assertTrue(self.active("k3s-agent")) + + def test_a_promotion_whose_server_does_not_start_fails_with_one_role_running_at_most(self) -> None: + self.running_agent() + self.hold_lease() + self.set_node({"github-runner.exadev.com/desired-role": "server"}, {"github-runner.exadev.com/role-join-node": "server1"}) + result = self.actuator("promote", SWAP, FAKE_RESTART_FAILS="1") + self.assert_status(result, FAILED) + self.assertEqual(self.report()["phase"], "failed") + self.assertEqual(self.report()["message"], "the server unit did not start") + self.assertEqual(self.role_file.read_text(), "server") + self.assertFalse(self.active("k3s-agent")) + self.assertFalse(self.active("k3s")) + + +class StepDownTest(ActuatorHarness): + def test_a_step_down_runs_the_server_at_etcd_only_then_names_the_agent_and_stops_regrading(self) -> None: + self.running_server() + (self.host / "active-fleet-regrade").touch() + self.hold_lease() + result = self.actuator("step-down", SWAP) + self.assert_status(result, DONE) + # Restarted while the role file still named the server, so its condition let it start, at etcd-only. + self.assertTrue(self.active("k3s")) + self.assertEqual(self.started_file.read_text().strip(), "etcd-only") + self.assertIn("disable-apiserver: true", (self.config / "server.yaml.d" / "50-regrade.yaml").read_text()) + self.assertNotIn("k3s skipped by its condition", (self.host / "units").read_text()) + self.assertEqual(self.role_file.read_text(), "agent") + self.assertFalse(self.active("fleet-regrade")) + self.assertEqual({key: self.report()[key] for key in ("step", "phase", "role")}, {"step": "step-down", "phase": "done", "role": "agent"}) + # The member is the controller's to remove: the server keeps running at etcd-only with its datastore. + self.assertTrue((self.data / "server" / "db").is_dir()) + + def test_a_step_down_whose_etcd_does_not_come_back_fails_with_the_role_file_still_naming_the_server(self) -> None: + self.running_server() + self.hold_lease() + result = self.actuator("step-down", SWAP, FAKE_NO_ETCD="1") + self.assert_status(result, FAILED) + self.assertEqual(self.role_file.read_text(), "server") + self.assertEqual(self.report()["phase"], "failed") + + def test_a_step_down_without_the_lease_is_refused(self) -> None: + self.running_server() + result = self.actuator("step-down", SWAP) + self.assert_status(result, REFUSED) + self.assertEqual(self.role_file.read_text(), "server") + self.assertEqual(self.started_file.read_text().strip(), "full") + + def test_a_role_file_that_names_no_role_is_refused(self) -> None: + self.running_server() + self.role_file.write_text("voter") + self.hold_lease() + result = self.actuator("step-down", SWAP) + self.assert_status(result, REFUSED) + self.assertIn("the role file names 'voter', which is not a role", result.stdout) + + +class FinishDemoteTest(ActuatorHarness): + def stepped_down(self) -> None: + """A server the actuator has stepped down: at etcd-only, its role file naming the agent, a level recorded and a swap's join drop-ins in place.""" + self.running_server(level="etcd-only") + self.role_file.write_text("agent") + (self.config / "server.yaml.d" / "20-fleet-role-join.yaml").write_text("server: x\n") + (self.units / "k3s.service.d").mkdir() + (self.units / "k3s.service.d" / "20-fleet-role-join.conf").write_text("[Service]\n") + + def assert_finished(self) -> None: + self.assertFalse(self.active("k3s")) + self.assertFalse((self.host / "enabled-k3s").exists()) + self.assertTrue(self.active("k3s-agent")) + self.assertFalse((self.data / "server" / "db").exists()) + self.assertEqual((self.data / "server" / "db.demoted" / "etcd" / "name").read_text(), f"{NODE}-0a1b2c3d") + self.assertFalse(self.level_file.exists()) + self.assertFalse((self.config / "server.yaml.d" / "20-fleet-role-join.yaml").exists()) + self.assertFalse((self.units / "k3s.service.d" / "20-fleet-role-join.conf").exists()) + self.assert_never_both_running() + + def test_a_demotion_is_finished_without_the_lease_once_k3s_records_the_members_removal(self) -> None: + self.stepped_down() + self.set_node({"node-role.kubernetes.io/etcd": "true"}, {"etcd.k3s.cattle.io/removed-node-name": f"{NODE}-0a1b2c3d"}) + result = self.actuator("finish-demote") + self.assert_status(result, DONE) + self.assert_finished() + report = self.report() + self.assertEqual({key: report[key] for key in ("step", "phase", "role")}, {"step": "finish-demote", "phase": "done", "role": "agent"}) + self.assertTrue(report["started_at"]) + + def test_a_tombstone_counts_as_removed_with_no_api_to_ask(self) -> None: + self.stepped_down() + (self.data / "server" / "db" / "etcd" / "tombstone").write_text("") + self.no_api() + self.assert_status(self.actuator("finish-demote"), DONE) + self.assert_finished() + + def test_a_node_with_no_etcd_label_counts_as_removed(self) -> None: + self.stepped_down() + self.set_node({}, {}) + self.assert_status(self.actuator("finish-demote"), DONE) + self.assert_finished() + + def test_a_demotion_is_not_finished_while_the_member_stays_and_nothing_is_reported(self) -> None: + self.stepped_down() + cases = { + "a member still listed": lambda: self.set_node({"node-role.kubernetes.io/etcd": "true"}, {"etcd.k3s.cattle.io/removed-node-name": "another-member"}), + "no API to ask": self.no_api, + } + for case, arrange in cases.items(): + with self.subTest(case=case): + arrange() + result = self.actuator("finish-demote") + self.assert_status(result, NOT_YET) + self.assertTrue(self.active("k3s")) + self.assertTrue((self.data / "server" / "db").is_dir()) + self.assertEqual(self.patches(), 0) + + def test_finishing_keeps_only_the_newest_datastore_copy(self) -> None: + self.stepped_down() + older = self.data / "server" / "db.demoted" / "etcd" / "name" + older.parent.mkdir(parents=True) + older.write_text("from-an-earlier-demotion") + (self.data / "server" / "db" / "etcd" / "tombstone").write_text("") + self.assert_status(self.actuator("finish-demote"), DONE) + self.assertEqual(older.read_text(), f"{NODE}-0a1b2c3d") + self.assertEqual(sorted(path.name for path in (self.data / "server").iterdir()), ["db.demoted"]) + + def test_finishing_a_host_whose_role_file_names_the_server_is_refused(self) -> None: + self.running_server() + (self.data / "server" / "db" / "etcd" / "tombstone").write_text("") + result = self.actuator("finish-demote") + self.assert_status(result, REFUSED) + self.assertTrue((self.data / "server" / "db").is_dir()) + + +class RollbackAndSnapshotTest(ActuatorHarness): + def test_a_rollback_takes_a_learner_back_to_the_agent(self) -> None: + self.running_server() + (self.config / "server.yaml.d" / "20-fleet-role-join.yaml").write_text("server: x\n") + self.hold_lease() + self.assert_status(self.actuator("rollback", SWAP), DONE) + self.assertEqual(self.role_file.read_text(), "agent") + self.assertFalse(self.active("k3s")) + self.assertTrue(self.active("k3s-agent")) + self.assertTrue((self.data / "server" / "db.demoted").is_dir()) + self.assertFalse((self.config / "server.yaml.d" / "20-fleet-role-join.yaml").exists()) + self.assertEqual(self.report()["step"], "rollback") + + def test_a_rollback_without_the_lease_is_refused(self) -> None: + self.running_server() + self.assert_status(self.actuator("rollback", SWAP), REFUSED) + self.assertTrue(self.active("k3s")) + self.assertEqual(self.role_file.read_text(), "server") + + def test_a_voter_takes_the_snapshot_named_for_the_swap_and_reports_its_file(self) -> None: + self.running_server() + self.hold_lease() + self.assert_status(self.actuator("snapshot", SWAP), DONE) + self.assertEqual((self.host / "snapshots").read_text().split(), ["--data-dir", str(self.data), "--etcd-server", "https://127.0.0.1:6443", "--name", f"fleet-role-{SWAP}"]) + self.assertEqual(self.report()["snapshot"], f"fleet-role-{SWAP}-testnode-1767225600") + self.assertEqual(self.report()["phase"], "done") + + def test_a_snapshot_the_server_refuses_fails_and_one_asked_of_an_agent_is_refused(self) -> None: + self.running_server() + self.hold_lease() + result = self.actuator("snapshot", SWAP, FAKE_SNAPSHOT_FAILS="1") + self.assert_status(result, FAILED) + self.assertIn("Internal Server Error", result.stderr) + self.assertEqual(self.report()["snapshot"], "") + self.setUp() + self.running_agent() + self.hold_lease() + self.assert_status(self.actuator("snapshot", SWAP), REFUSED) + self.assertFalse((self.host / "snapshots").exists()) + + +class CommandLineTest(ActuatorHarness): + def test_status_prints_the_role_the_units_the_datastore_and_the_last_report(self) -> None: + self.running_server() + self.hold_lease() + self.assert_status(self.actuator("snapshot", SWAP), DONE) + result = self.actuator("status") + self.assert_status(result, DONE) + self.assertIn("role file: server", result.stdout) + self.assertIn("server unit: running; agent unit: stopped", result.stdout) + self.assertIn(f"datastore: in place, member {NODE}-0a1b2c3d; moved aside: no", result.stdout) + self.assertIn("joins: the join configuration the role wrote", result.stdout) + self.assertIn('"step":"snapshot"', result.stdout) + + def test_a_command_it_does_not_have_and_a_missing_argument_are_refused(self) -> None: + for args in (("leave",), ("promote",), ("finish-demote", SWAP), ()): + with self.subTest(args=args): + result = self.actuator(*args) + self.assert_status(result, REFUSED) + self.assertIn("usage: fleet-role-actuator", result.stderr) + + def test_a_switch_waits_for_the_hosts_lock_and_fails_when_another_holds_it_too_long(self) -> None: + self.running_agent() + with (self.data / "fleet-role.lock").open("w") as lock: + fcntl.flock(lock, fcntl.LOCK_EX) + result = self.actuator("promote-local") + self.assert_status(result, FAILED) + self.assertIn("another role switch on this host held its lock for longer than 2s", result.stdout) + self.assertEqual(self.role_file.read_text(), "agent") + self.assertTrue(self.active("k3s-agent")) + + def test_the_local_promotion_without_a_join_url_leaves_the_roles_own_join_configuration_to_decide(self) -> None: + self.running_agent() + (self.config / "server.yaml.d" / "20-fleet-role-join.yaml").write_text("server: x\n") + (self.units / "k3s.service.d").mkdir() + (self.units / "k3s.service.d" / "20-fleet-role-join.conf").write_text("[Service]\n") + self.assert_status(self.actuator("promote-local"), DONE) + self.assertFalse((self.config / "server.yaml.d" / "20-fleet-role-join.yaml").exists()) + self.assertFalse((self.units / "k3s.service.d" / "20-fleet-role-join.conf").exists()) + self.assertTrue(self.active("k3s")) + # No report: the role's own switch is not a swap's step. + self.assertEqual(self.patches(), 0) + + +class WatchTest(ActuatorHarness): + """The watcher, run as its unit runs it, against a Node the test changes as the controller would.""" + + def watch(self) -> subprocess.Popen[str]: + process = subprocess.Popen(["bash", str(self.lib / "fleet-role-actuator.sh"), "watch"], env=self.env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) + + def stop() -> None: + process.kill() + process.wait() + assert process.stdout is not None + process.stdout.close() + + self.addCleanup(stop) + return process + + def wait_for_report(self, step: str, phase: str) -> dict[str, Any]: + # The watcher looks every POLL_SECONDS (2, fleet-regrade-k3s.sh); a step takes a few looks at most against the stand-ins. + deadline = time.monotonic() + 60 + while time.monotonic() < deadline: + report = self.report() + if report.get("step") == step and report.get("phase") == phase: + return report + time.sleep(0.2) + self.fail(f"no {step} {phase} report; last: {self.report()}") + + def test_the_watcher_takes_a_swap_through_promotion_step_down_and_finish_once_each(self) -> None: + self.running_agent() + self.hold_lease() + self.watch() + self.set_node({"github-runner.exadev.com/desired-role": "server"}, {"github-runner.exadev.com/role-swap": SWAP, "github-runner.exadev.com/role-step": "promote", "github-runner.exadev.com/role-join-node": "server1"}) + self.wait_for_report("promote", "done") + self.assertTrue(self.active("k3s")) + # k3s leaves a datastore once it has joined. + (self.data / "server" / "db" / "etcd").mkdir(parents=True) + (self.data / "server" / "db" / "etcd" / "name").write_text(f"{NODE}-0a1b2c3d") + self.set_node({"github-runner.exadev.com/desired-role": "agent", "node-role.kubernetes.io/etcd": "true"}, {"github-runner.exadev.com/role-swap": SWAP, "github-runner.exadev.com/role-step": "step-down"}) + self.wait_for_report("step-down", "done") + self.assertEqual(self.started_file.read_text().strip(), "etcd-only") + # Nothing more happens until the member is gone: the report stays the step-down's. + time.sleep(5) + self.assertEqual(self.report()["step"], "step-down") + self.set_node({"github-runner.exadev.com/desired-role": "agent", "node-role.kubernetes.io/etcd": "true"}, {"github-runner.exadev.com/role-swap": SWAP, "github-runner.exadev.com/role-step": "step-down", "etcd.k3s.cattle.io/removed-node-name": f"{NODE}-0a1b2c3d"}) + self.wait_for_report("finish-demote", "done") + self.assertTrue(self.active("k3s-agent")) + self.assertFalse(self.active("k3s")) + self.assertTrue((self.data / "server" / "db.demoted").is_dir()) + self.assert_never_both_running() + units = (self.host / "units").read_text().splitlines() + self.assertEqual([line for line in units if line.endswith(" started")], ["k3s started", "k3s started", "k3s-agent started"]) + + def test_the_watcher_leaves_a_refused_step_to_the_controller_once_its_reason_is_gone(self) -> None: + self.running_agent() + self.watch() + self.set_node({"github-runner.exadev.com/desired-role": "server"}, {"github-runner.exadev.com/role-swap": SWAP, "github-runner.exadev.com/role-step": "promote", "github-runner.exadev.com/role-join-node": "server1"}) + self.wait_for_report("promote", "refused") + # The Lease the step lacked is now held, but what follows a refusal is the controller's to decide: a new swap or step, never the watcher taking the old one again. + self.hold_lease() + time.sleep(3 * WATCH_POLL_SECONDS) + self.assertEqual(self.report()["phase"], "refused") + self.assertEqual(self.role_file.read_text(), "agent") + self.assertTrue(self.active("k3s-agent")) + self.assertFalse(self.active("k3s")) + + def test_the_watcher_refuses_a_step_its_desired_role_does_not_go_with(self) -> None: + self.running_agent() + self.hold_lease() + self.watch() + self.set_node({"github-runner.exadev.com/desired-role": "agent"}, {"github-runner.exadev.com/role-swap": SWAP, "github-runner.exadev.com/role-step": "promote", "github-runner.exadev.com/role-join-node": "server1"}) + report = self.wait_for_report("promote", "refused") + self.assertIn("desired-role is 'agent', which a promote does not go with", report["message"]) + self.assertTrue(self.active("k3s-agent")) + self.assertEqual(self.role_file.read_text(), "agent") + + +class RoleSetupTest(NativeRoleHarness): + """The role's setup of the actuator on a native host, with the role controller on and every host eligible.""" + + ON = {"github_runner_cluster_role_controller": True, "github_runner_cluster_voter_eligible": True} + + def run_eligible(self, *args: str, check: bool = False, **laptop: Any) -> subprocess.CompletedProcess[str]: + """Run laptop as an eligible native agent beside the Docker servers server1 and server2, every host eligible.""" + return self.run_role(*args, host_vars={"github_runner_cluster_node_role": "agent", **self.ON, **laptop}, check=check, second_server=True, group_vars=self.ON) + + def lib(self, name: str) -> Path: + return self.paths["github_runner_cluster_native_lib_dir"] / "role" / name + + def test_an_eligible_host_runs_the_watcher_with_its_own_credential_and_settings(self) -> None: + result = self.run_eligible() + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + unit = self.file("github_runner_cluster_native_unit_dir", "fleet-role-actuator.service").read_text() + self.assertIn(f"ExecStart={self.lib('fleet-role-actuator.sh')} watch", unit) + self.assertIn(f'Environment="FLEET_ROLE_ACTUATOR_ENV={self.lib("fleet-role-actuator.env")}"', unit) + self.assertIn("systemctl start fleet-role-actuator", self.calls()) + settings = dict(line.split("=", 1) for line in self.lib("fleet-role-actuator.env").read_text().splitlines() if line and not line.startswith("#")) + self.assertEqual(settings["NODE_NAME"], "laptop") + self.assertEqual(settings["KUBE_SERVERS"], f'"https://server1.{DOMAIN}:6443 https://server2.{DOMAIN}:6443"') + self.assertEqual(settings["ACTUATOR_NAMESPACE"], "github-runner-platform") + self.assertEqual(self.lib("join-urls").read_text(), f"server1 https://server1.{DOMAIN}:6443\nserver2 https://server2.{DOMAIN}:6443\n") + for name in ("fleet-role-actuator.sh", "fleet-role-actuator-systemd.sh"): + self.assertEqual(self.lib(name).read_text(), (FILES / "role" / name).read_text()) + for name in ("fleet-regrade-core.sh", "fleet-regrade-k3s.sh", "fleet-regrade-systemd.sh"): + self.assertEqual(self.lib(name).read_text(), (FILES / "regrade" / name).read_text()) + kubeconfig_path = self.file("github_runner_cluster_native_config_dir", "fleet-role-actuator.kubeconfig") + self.assertEqual(stat.S_IMODE(kubeconfig_path.stat().st_mode), 0o600) + kubeconfig = yaml.safe_load(kubeconfig_path.read_text()) + self.assertEqual(kubeconfig["users"], [{"name": "fleet-role-actuator", "user": {"token": ACTUATOR_TOKEN}}]) + self.assertEqual(kubeconfig["clusters"][0]["cluster"], {"server": f"https://server1.{DOMAIN}:6443", "certificate-authority-data": base64.b64encode(b"cluster-ca").decode()}) + # The credential was applied, and the ConfigMap created, through a Docker server's container, since this host's own k3s may be the one a swap stops. + self.assertIn("docker exec --interactive github-runner-k3s kubectl apply --filename -", self.calls()) + self.assertIn("docker exec github-runner-k3s kubectl create configmap fleet-role-actuators --namespace github-runner-platform", self.calls()) + + def test_the_token_stays_out_of_everything_the_role_prints(self) -> None: + result = self.run_eligible("-vvv") + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + encoded = base64.b64encode(ACTUATOR_TOKEN.encode()).decode() + for text in (result.stdout, result.stderr, (self.stubs / "calls.log").read_text()): + self.assertNotIn(ACTUATOR_TOKEN, text) + self.assertNotIn(encoded, text) + + def test_the_credential_reads_nodes_and_the_lease_and_patches_its_report_and_nothing_more(self) -> None: + self.assertEqual(self.run_eligible().returncode, 0) + documents = [document for document in yaml.safe_load_all((self.state / "applied").read_text()) if document] + kinds = {(document["kind"], document["metadata"].get("namespace", "")): document for document in documents} + self.assertEqual(sorted(kinds), sorted([("Namespace", ""), ("ServiceAccount", "github-runner-platform"), ("Secret", "github-runner-platform"), ("ClusterRole", ""), ("ClusterRoleBinding", ""), ("Role", "github-runner-platform"), ("RoleBinding", "github-runner-platform"), ("Role", "kube-system"), ("RoleBinding", "kube-system")])) + self.assertEqual(kinds[("ClusterRole", "")]["rules"], [{"apiGroups": [""], "resources": ["nodes"], "verbs": ["get", "list", "watch"]}]) + self.assertEqual(kinds[("Role", "github-runner-platform")]["rules"], [{"apiGroups": [""], "resources": ["configmaps"], "resourceNames": ["fleet-role-actuators"], "verbs": ["get", "patch"]}]) + self.assertEqual(kinds[("Role", "kube-system")]["rules"], [{"apiGroups": ["coordination.k8s.io"], "resources": ["leases"], "resourceNames": ["fleet-regrade"], "verbs": ["get"]}]) + secret = kinds[("Secret", "github-runner-platform")] + self.assertEqual((secret["type"], secret["metadata"]["annotations"]), ("kubernetes.io/service-account-token", {"kubernetes.io/service-account.name": "fleet-role-actuator"})) + for binding in ("ClusterRoleBinding", "RoleBinding"): + for document in documents: + if document["kind"] == binding: + self.assertEqual(document["subjects"], [{"kind": "ServiceAccount", "name": "fleet-role-actuator", "namespace": "github-runner-platform"}]) + + def test_a_host_the_controller_may_not_switch_keeps_the_scripts_and_runs_no_watcher(self) -> None: + self.assertEqual(self.run_eligible().returncode, 0) + result = self.run_role(host_vars={"github_runner_cluster_node_role": "agent"}, second_server=True) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn("systemctl disable --now fleet-role-actuator", self.calls()) + self.assertFalse(self.file("github_runner_cluster_native_unit_dir", "fleet-role-actuator.service").exists()) + self.assertFalse(self.file("github_runner_cluster_native_config_dir", "fleet-role-actuator.kubeconfig").exists()) + self.assertFalse(self.lib("join-urls").exists()) + self.assertTrue(self.lib("fleet-role-actuator.sh").exists()) + + def test_a_check_run_reports_the_watcher_and_sets_up_nothing(self) -> None: + result = self.run_eligible(check=True) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn("apply the role actuator's credential (fleet-role-actuator in github-runner-platform)", re.sub(r"\s+", " ", result.stdout)) + self.assertFalse((self.state / "applied").exists()) + self.assertFalse(self.file("github_runner_cluster_native_config_dir", "fleet-role-actuator.kubeconfig").exists()) + + def test_the_regrade_unit_starts_only_on_a_server_whose_datastore_holds_no_tombstone(self) -> None: + (self.state / "voters").write_text("server1,True,true\nserver2,True,true\nlaptop,True,true\n") + result = self.run_role(host_vars={"github_runner_cluster_node_role": "server", "github_runner_cluster_regrade": True}, second_server=True) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + unit = self.file("github_runner_cluster_native_unit_dir", "fleet-regrade.service").read_text() + lib, data = self.paths["github_runner_cluster_native_lib_dir"], self.paths["github_runner_cluster_native_data_dir"] + self.assertIn(f"ExecCondition={lib}/fleet-role-is server {data}/fleet-role {data}/server", unit.splitlines()) + # The same condition as the server unit's own. + server_unit = self.file("github_runner_cluster_native_unit_dir", "k3s.service").read_text() + self.assertIn(f"ExecCondition={lib}/fleet-role-is server {data}/fleet-role {data}/server", server_unit.splitlines()) + + def test_the_actuators_namespace_is_the_arc_roles_platform_namespace(self) -> None: + arc = yaml.safe_load((ROOT / "roles" / "github_runner_arc" / "defaults" / "main.yml").read_text()) + cluster = yaml.safe_load((ROLE / "defaults" / "main.yml").read_text()) + self.assertEqual(cluster["github_runner_cluster_role_actuator_namespace"], "{{ github_runner_arc_platform_namespace | default('" + arc["github_runner_arc_platform_namespace"] + "') }}") + + +if __name__ == "__main__": + unittest.main()