Repository navigation
feat(cluster): switch a native host's role from a host-side actuator driven by its Node - #216
Merged
Merged
Conversation
…cluster through once The regrade service's settings worked out, in their own template, which other servers and eligible hosts it may reach the cluster through. That list is now the derived default github_runner_cluster_api_candidates, so the role actuator's settings and its join URLs can take the same hosts in the same order rather than repeating the expression.
…driven by its Node The role controller's voter swap needs something on each host that can start one k3s unit in place of the other, and an agent host has no credential that can act on the cluster. fleet-role-actuator does only the host-local part of a switch, as the host's Node asks through the desired-role label and the role-swap, role-step, role-join-node and snapshot-request annotations, and reports each step in the fleet-role-actuators ConfigMap; the controller keeps every write on the API. Every step a swap asks for is taken only while the regrade Lease is held by fleet-role for that swap. Finishing a demotion needs no Lease once the member is gone, since completing it can never cost quorum. A promotion writes join drop-ins for both the server's configuration and its unit, so the Tailscale wrapper waits for the voter it joins rather than the inventory's join host. A step-down goes to etcd-only through regrading's own level and names the agent in the role file before the member is removed. The role's own promotion and demotion now call promote-local and demote-local, so Ansible and the actuator switch a host one way, and the fleet-regrade unit gains the server unit's condition so that a stepped-down host cannot restart its watcher. On a server or eligible native host with the role controller on, the role runs the watcher with a ServiceAccount that can read Nodes and the regrade Lease and patch its own report, nothing more.
…al k3s The role_actuator scenario acts as the role controller by hand: it holds the regrade Lease for a swap and writes the native host's desired-role label and swap annotations itself. The host's actuator must promote it to a learner and then a voter with writes succeeding throughout, step it down to etcd-only with no pod's shim left, and, once its member is removed through k3s's annotation, finish it as an agent with its node password and Tailscale address unchanged, its datastore moved aside and the two k3s units never running together.
…grade unit is A check run now reports a changed watcher unit, and systemd is reloaded for it only in a real run, as tasks/native/regrade_service.yml does.
…ontroller A step reported refused is not taken again by the watcher once the reason for the refusal is gone; the controller decides what follows by writing a new swap or step.
…hat runs it ansible-core resolves a task's delegate_to before its when, so the delegated credential tasks failed on a host whose watcher is off and whose run had never included native/api.yml. They move to tasks/role_actuator_watcher.yml, included only where the watcher runs.
…e role actuator The role's promotion and demotion now run the actuator's promote-local and demote-local, which enable and start the server or agent unit in one call, and a demotion starts the agent before the Node's role labels and etcd annotations are removed.
…docker exec --interactive docker exec gives the command its standard input only with --interactive, so applying the credential through a Docker server read an empty manifest and kubectl refused it. The apply now adds the flag as tasks/colima_size_turn.yml does, and the native stand-in for docker passes standard input only with it, with its kubectl refusing an empty manifest as kubectl does.
…resh node lease The node is still cordoned from the step-down when the agent comes up, so kubectl's STATUS column reads Ready,SchedulingDisabled and never Ready. The wait now reads the Ready condition, and requires the kubelet to have renewed the node's Lease since the finish started, as tasks/native/server_ready.yml judges a restart.
Mearman
marked this pull request as ready for review
November 4, 2026 09:01
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins. |
|
🎉 This PR is included in version 2.37.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #183
Adds
fleet-role-actuator, the host-side half of a voter swap on a native host. It switches the host between itsk3sandk3s-agentunits as its Node asks (thegithub-runner.exadev.com/desired-rolelabel and therole-swap,role-step,role-join-nodeandsnapshot-requestannotations) and reports each step in thefleet-role-actuatorsConfigMap in the platform namespace. The controller keeps every write on the API; the actuator only reads Nodes and the regrade Lease and patches its own report. The role's own promotion and demotion (server_turn.yml,demote.yml) now run the actuator'spromote-localanddemote-local, so both paths switch a host the same way. Thefleet-regradeunit gains the server unit'sExecCondition, so a host stepped down for a swap cannot restart a watcher that would step it back up.The controller side (writing the label and annotations, renewing the Lease) is PR E2 in the plan, and the Docker host's switch is D2.
Tests
tests/unit/test_role_actuator.pyruns every command against stand-ins forsystemctl,kubectlandk3swith the realfleet-role-is.shas both units' condition, and the role's setup of the watcher through the native stand-ins. Therole_actuatormesh scenario (added to the mesh matrix) acts as the controller by hand against real k3s: promote the native standby (learner, then voter, writes succeed throughout), step it down (etcd serving at etcd-only, no shim left), remove its member through k3s's annotation, and check it finishes as an agent with the same node password hash and Tailscale IP,db.demotedin place and the two units never active together. The existing native server tests pass against the refactoreddemote.ymlandserver_turn.ymlwith two expectations updated, both listed under deviations: a promotion now starts the server withsystemctl enable --now k3s, and a demotion starts the agent before the Node's role labels and etcd annotations are removed.Each unit test below was shown to discriminate by breaking the code in a throwaway clone and watching it fail, then passing on this branch:
test_a_promotion_without_the_lease_for_its_swap_is_refused_and_changes_nothingtest_a_tombstone_counts_as_removed_with_no_api_to_askdb.demotedkept (norm -rfbeforemv)test_finishing_keeps_only_the_newest_datastore_copytest_a_promotion_joins_the_named_voter_and_starts_the_server_unit_in_place_of_the_agenttest_a_node_with_no_etcd_label_counts_as_removedtest_a_node_with_no_etcd_label_counts_as_removedtest_a_step_down_whose_etcd_does_not_come_back_fails_with_the_role_file_still_naming_the_servertest_a_demotion_is_not_finished_while_the_member_stays_and_nothing_is_reportedtest_the_watcher_leaves_a_refused_step_to_the_controller_once_its_reason_is_gonedesired-roleignoredtest_the_watcher_refuses_a_step_its_desired_role_does_not_go_withfleet-regradeunit withoutExecConditiontest_the_regrade_unit_starts_only_on_a_server_whose_datastore_holds_no_tombstonepatchtest_the_credential_reads_nodes_and_the_lease_and_patches_its_report_and_nothing_moreno_logdropped from the token readtest_the_token_stays_out_of_everything_the_role_printstest_a_voter_takes_the_snapshot_named_for_the_swap_and_reports_its_file--interactivedropped from the credential'sdocker exec ... apply --filename -test_the_credential_reads_nodes_and_the_lease_and_patches_its_report_and_nothing_more(the docker stand-in now passes stdin only with--interactive, as Docker does)test_a_switch_waits_for_the_hosts_lock_and_fails_when_another_holds_it_too_longThe first version of the watcher test (
..._once_each) did not fail when the settled check was removed, since every step is idempotent, so the refused-step test was added for that break.Deviations from the plan
getthe regrade Lease by name (a Role inkube-system), since it checks the Lease's holder and swap id itself before any step.desired-rolelabel does not go with it (apromoteneedsserver, astep-downorrollbackneedsagent), so a controller that writes one and not the other gets a refusal rather than a guess.demote.ymlnow removes the server role labels and k3s's etcd annotations after the agent has started rather than before, since the agent's start is part of the shareddemote-local; the agent's k3s puts none of them back.flockon/var/lib/rancher/k3s/fleet-role.lock) keeps a switch the role makes and one the actuator makes from running together.step_down,cmd_watchandcmd_status, which it sources, so neither replaces the other's.github_runner_cluster_api_candidates, shared by the regrade service's settings, the actuator's settings and its join URLs, instead of an expression repeated in each template.meta/argument_specs.ymlis unchanged: every new variable is derived, and the argument specs hold only settings.--check).tasks/role_actuator_watcher.yml, included only on a host that runs the watcher: ansible-core resolves a task'sdelegate_tobefore itswhen, so the delegated credential tasks inside a skipped block failed on a host that had never setgithub_runner_cluster_native_api_host.enable --now) where it used to enable and then start, since that is the sharedpromote-local.