Repository navigation
feat(role): carry out a voter swap, by decision or by command - #228
Merged
Merged
Conversation
…otations acquire_lease and release_lease take the holder as an optional argument, defaulting to the node, and acquire_lease writes a JSON object of annotations with the Lease. release_lease clears the annotations with the holder, so the next holder does not read a finished swap's.
…3s's tombstone A host that was down while it was stepped down and removed never wrote the agent's role file, and its units idle on the tombstone, so nothing finished it. The actuator now finishes a server whose datastore holds the tombstone as an agent, whatever its role file says.
The role controller now swaps an unfit voter for an eligible standby under the regrade Lease, which it renews on every pass. scripts/fleet-role-swap.jq is the pure next-action function: it reads the step a swap is at from the Nodes, the actuators' reports and the Lease, so a pod that is rescheduled resumes, and each pass takes at most one action. scripts/fleet-role.sh carries the action out: the snapshot request, the cordon and runner wait, the drain, the promotion, the gates with one more voter, the step-down, the member removal, the clean-up and the rollback of a candidate that does not vote. `fleet-role.sh swap OUT IN` records a manual request, which is carried out in a dry run too and honours every gate but the verdicts and the cooldown; with the dry run off the controller carries out its own decisions as well, and a swap under way is always finished or rolled back. The decision program treats a Node whose member k3s removed as no voter, takes a voter that a swap is removing out of the gates, and reports the members each voter lists. The controller's ClusterRole and Roles gain only what a swap does, and the probe stays read-only.
… etcd The role_swap scenarios run the controller's own command on three Docker servers and a native standby: a swap in and out with writes throughout and the bootstrap host promoted again into the same cluster, the outgoing voter paused, the candidate's peer port dropped, and a swap refused while a voter is paused. role_controller also turns the dry run off and checks the controller swaps the slow voter for the Docker standby. The image is loaded into each node container as it comes up, since the two services keep separate containerd data.
…the image smoke test The smoke test checks the swap module is copied beside the other jq programs and compiles under the image's jq, answers the Lease and the actuators' reports as an empty cluster does, and gives the controller the settings it now requires.
…nchanged step start A failing cordon is named with its node, since the stand-in matches the first two words of a call, and the step-since test reads the start it set from the status it wrote.
…ot by lacking the voter labels
The bootstrap host has no server URL of its own, so K3S_AGENT_SERVER_URL was empty and the agent of a bootstrap host that a role switch had demoted restarted for ever, unable to join. The agent now joins the first other server of the inventory, as a native agent already did, from one derived github_runner_cluster_agent_server_url that the native join URL defaults to.
…ing the removal A server at etcd-only runs no kubelet, so its Node stops saying EtcdIsVoter while etcd still lists its member. The gates read that as a voter count one short and as an unknown member, and held the removal for ever. A voter a swap is taking out now counts as a member until k3s removes it, and is held to none of the gates on its own health.
…it was taken with
Stepping down the member that leads makes the others elect another, and a write in flight is answered "request timed out, possibly due to previous leader failure". A failed write now has to be followed by a success within the time an election takes (twice etcd's election timeout), which quorum lost never meets, and a server stopped after its member was removed may be failed, not inactive, so the swap scenarios ask only that it is not active.
… by its steps A swap holds its steps while any voter's probe is silent for three periods, and probes sampling every 5 seconds on a runner with this many k3s nodes missed that often enough to stall the role_controller swap past its wait. The swap scenarios deploy at the ARC role's default period, and role_controller sets it before turning the dry run off. The wait for a swap is the sum of the steps' own bounds, not a rounded figure.
… the default period A window of samples at the ARC role's default period takes as long to fill as the controller says, and a swap's steps have their own bounds on top. Together they do not fit in one job beside the other role_controller checks, so the swap ran out of wait while its last step was still inside the swap's own bound. role_controller keeps its decision, label and pause checks at the short period and ends with the controller having changed nothing. role_controller_swap runs the controller at the default period, makes a voter slow, and turns the dry run off, waiting for the window the controller reads from its own module. The bring-up and the slow-voter step are shared.
…o fails After a swap the former server's host runs an agent container that the dump of each node's own container does not cover, so a failure that left that agent not Ready showed only its server's log. The dump now adds the agent's state, its Tailscale status and its last log lines.
The default probe window alone takes twenty-five minutes. The induced unfitness wait, swap budget and setup budget can exceed the old limit, so GitHub cancelled the scenario before its own diagnostics could say which controller step was blocked.
A step-down that fails before the role file names the agent leaves it naming the server, and on a node that is not Ready the controller removes the member anyway. The actuator refused to finish such a host unless its datastore held k3s's tombstone, so the server unit kept restarting on the host after its member was gone, put the Node's etcd label back once the controller had cleaned it, and the swap returned to the removal and waited for ever. A server whose Node asks it to step down is now finished once its member is gone, as member_gone decides, by the command and by the watch.
…er has gone The gates counted a voter being taken out as a member for as long as its Node carried the etcd label without k3s's removed-node-name annotation. A server still running on the host sets that label again after the clean-up, so once k3s had removed the member every remaining voter reported three members against four expected, and the swap held its removal for ever. When every voter the gates judge reports, fresh, that etcd lists as many members as there are voters, the outgoing node has departed: it is no longer expected, the swap goes on to the finish of its demotion, and past the finish's deadline with the clean-up done it ends completed and leaves the host to node recovery.
…ember The actuator finishes a demotion only once it can see that the host's member is gone, and on the API it reads k3s's removed-node-name, which names the member k3s last removed. When that names another member than the host's datastore holds, the actuator waited while every remaining voter reported that etcd listed only the voters, and the swap ended at the finish's deadline with the host left to node recovery. Once the remaining voters show the member gone, the controller annotates the Node with role-member-gone naming the swap, and the actuator takes that as the member gone; the annotation is cleared with the swap's others when the swap ends. The controller-swap scenario now fails when a swap leaves its outgoing host pending, since the slow voter stays up and its demotion must finish inside the swap.
…pfs over its datastore The controller-swap scenario mounted a tmpfs over server/db in the k3s service alone. A tmpfs mount is emptied at every restart of the container, so a server restarted into etcd-only, or by its restart policy after its member was removed, came back with no datastore and no tombstone and joined etcd again as a new member under a new name. The role actuator mounts the data volume, not the tmpfs, so it read an empty server/db: no member name to match k3s's record and no tombstone, and it moved the wrong directory aside. The whole data volume is now a tmpfs volume, which the actuator mounts too and which outlives the server's restarts while the actuator holds it.
…its cluster ID A member that has just joined or been promoted can be a voter to the others, and its Node can say EtcdIsVoter, before its own etcd serves its clients. The scenarios read its cluster ID at once and took the empty answer of a timed-out endpoint status as another cluster's ID. They now wait for the member's etcd to answer, then compare the ID.
… a step is taken again The controller uncordons a demoted node once it is Ready on a kubelet heartbeat later than the start its actuator reports. A Docker host's first report of the finish is often lost, since its agent has no Tailscale yet, and the watch takes the finish again; start_agent_unit then found the agent running and left STARTED_AT empty, so the report said no start and no heartbeat could ever follow it. The swap waited past the cleanup's deadline and left a Ready agent to node recovery. Both start functions of the Docker platform now report the last start they recorded when the container already runs, as the systemd platform's last_start does for a promotion.
|
🎉 This PR is included in version 2.40.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the dynamic regrading epic (Refs #183, which F completes).
The role controller now carries out a voter swap, by command (
fleet-role.sh swap OUT IN, run withkubectl exec deploy/fleet-role) or, withgithub_runner_arc_role_controller_dry_run: false, by its own decision. The dry run stays the default everywhere: nothing here turns it off in a default, template, inventory or live object. A manual request is carried out in a dry run too, and a swap under way is always finished or rolled back.The next action is a pure function (
scripts/fleet-role-swap.jq) over the Nodes, the actuators' reports and the regrade Lease, so a rescheduled pod resumes where the swap is; each pass renews the Lease and takes at most one action.Deviations from the plan in #183, each needed by what the code does:
The controller gets
getas well asliston pods:kubectl drainreads each evicted pod while it waits for it to go.The controller also reads the kubelets' Leases (
kube-node-lease,get), since a restarted node is judged Ready by its kubelet's heartbeat, astasks/native/server_ready.ymldoes.A manual request is kept under its own key (
request.json) infleet-role-status, so the command and the controller's pass never overwrite each other;status.jsonmirrors it asrequest.A Node whose member k3s removed (
removed-node-name) is no voter, whatever its stale label and condition say; the gates leave out the voter a swap is removing.The outgoing voter's runner pods are waited for (up to the runner wait) before it is drained, as the candidate's are.
The actuator finishes a server whose datastore holds k3s's tombstone as an agent whatever its role file says, so a host that was down while it was stepped down and removed does not idle for ever (the
role_swap_pausedscenario).acquire_leaseandrelease_leasein the regrade core take a holder and annotations.A Docker bootstrap host's agent had no server to join (
K3S_AGENT_SERVER_URLwas empty), so one demoted by a swap restarted for ever; it now joins the first other server, fromgithub_runner_cluster_agent_server_url, which the native join URL defaults to. The mesh run found it.The gates counted a stepped-down member (no kubelet, so no
EtcdIsVoter) as a missing voter and held the removal for ever; a voter a swap is taking out now counts as a member until k3s removes it. The mesh run found it.An eligible host the cluster role did not stage as a standby is refused by its actuator, which abandons the swap with nothing changed on the host.
Break checks, each made in a clone outside the repository with the named tests then run and failing:
scripts/fleet-role-swap.jqtest_the_gates_are_held_with_four_voters_and_the_outgoing_voter_is_not_touched_while_one_fails; test_the_removal_is_held_while_a_remaining_voter_is_unhealthyscripts/fleet-role-swap.jqtest_the_swap_waits_while_the_snapshot_is_not_ready_even_when_the_actuator_says_donescripts/fleet-role.shtest_every_pass_renews_the_lease_the_swap_holds_and_keeps_its_annotationsscripts/fleet-role-swap.jqtest_past_the_runner_wait_the_swap_is_abandoned_with_the_candidate_uncordoned_the_lease_released_and_no_cooldownscripts/fleet-role-swap.jqtest_a_finished_rollback_waits_until_every_voter_lists_the_target_members_and_no_morescripts/fleet-role-swap.jqtest_a_candidate_that_is_not_a_voter_by_the_deadline_is_rolled_back_asking_k3s_to_remove_its_learner_when_it_joinedscripts/fleet-role-swap.jqtest_a_lease_that_regrade_took_mid_swap_is_never_acted_under_and_never_renewedscripts/fleet-role-swap.jqtest_a_manual_request_is_carried_out_inside_the_cooldown_and_whatever_the_verdicts_sayscripts/fleet-role-swap.jqtest_dry_run_carries_out_a_manual_request_and_reports_an_automatic_decision_onlyscripts/fleet-role.jqtest_the_removed_member_is_no_voter_so_the_membership_the_controller_publishes_has_three_votersscripts/fleet-role-swap.jqtest_the_promotion_waits_for_a_voter_that_is_ready_on_a_heartbeat_after_the_server_startedscripts/fleet-role.shtest_the_swap_ends_completed_the_annotations_cleared_the_lease_released; test_an_actuator_that_refuses_the_promotion_abandons_the_swap_since_nothing_changed_on_the_hostscripts/fleet-role-swap.jqtest_the_removal_waits_for_the_step_down_while_the_outgoing_voter_is_ready_and_has_not_reportedscripts/fleet-role-swap.jqtest_a_swap_that_ended_with_the_annotations_cleared_before_the_lease_was_released_is_ended_not_started_againscripts/fleet-role-swap.jqtest_a_host_that_does_not_come_back_in_the_operation_timeout_is_left_to_node_recovery_with_the_swap_recorded_pendingscripts/fleet-role.shtest_a_demotion_the_actuator_finished_is_cleaned_up_the_role_labels_etcd_annotations_level_label_and_taintscripts/fleet-role.shtest_an_action_that_fails_is_repeated_and_leaves_the_swap_where_it_wasscripts/fleet-role-swap.jqtest_a_snapshot_is_never_asked_of_the_outgoing_voter_even_when_it_sorts_firstroles/github_runner_cluster/files/role/fleet-role-actuator.shtest_the_watcher_finishes_a_server_whose_role_file_still_names_the_server_once_its_datastore_holds_the_tombstoneroles/github_runner_cluster/files/regrade/fleet-regrade-core.shFAIL and the annotations it wroteroles/github_runner_cluster/templates/env.j2test_the_bootstrap_hosts_agent_joins_the_first_other_server_since_it_has_no_bootstrap_host_to_dialscripts/fleet-role.jqtest_a_stepped_down_member_whose_node_no_longer_says_it_votes_is_still_counted_as_a_member_and_removedMesh scenarios
role_swap,role_swap_paused,role_swap_blocked_peer,role_swap_refusedand the extendedrole_controllerare in the matrix.