feat(role): carry out a voter swap, by decision or by command

.github/workflows/mesh-integration.yml

@@ -1,6 +1,6 @@
 name: Mesh integration
 
-# Brings up three k3s nodes in Docker and joins them through Headscale with the cluster role (tests/mesh/run.sh), once per scenario: headscale_hosted, headscale_in_cluster, and the headscale_existing policy check, plus the role's refusal of a Docker Compose release that recreates the containers it has just built. headscale_in_cluster also runs with github_runner_cluster_docker_context pinned while the current Docker context points at nothing, which covers the cluster role's own docker calls and the recovery playbook's, and context_checks checks the role refuses a pin it cannot honour. regrade turns memory-pressure regrading on and steps a server down to etcd-only under (simulated) pressure and back up, checking that it keeps its etcd seat and vote and that no other server steps meanwhile. native_server joins a native host (a systemd container) as an agent beside three Docker servers, then promotes it to a server in place: it must join etcd as a learner, be promoted to a voter, and keep the cluster writable throughout. rejoin removes the bootstrap Docker server's etcd member and checks the server idles on its tombstone, refuses to start a second cluster from its moved-aside datastore with no server to join, joins the existing cluster from a join URL in its role directory with the same cluster ID, and idles as a server whose role file names the agent. role_controller adds a standby agent, runs the role controller's probe and controller from the image built from this checkout, makes a voter slow with tc netem and pauses another, and checks every probe reports the link medium the cluster role labelled its node with, the controller reports the voter unfit with the swap named, reports that it would not act with a voter paused, and changes nothing. role_actuator acts as the role controller by hand on three Docker servers and an eligible native standby: the native host's role actuator promotes it to a voter with writes succeeding throughout, steps it down to etcd-only with no pod's shim left, and once its member is removed through k3s's annotation finishes it as an agent with the same node password and Tailscale address and its datastore moved aside, the two k3s units never running together; then a Docker server's role-actuator service demotes it the same way, its agent keeping the server's node password and Tailscale address as one Node with the datastore moved aside, and promotes it again into the same etcd cluster. Only on changes that can affect how nodes join the mesh.
+# Brings up three k3s nodes in Docker and joins them through Headscale with the cluster role (tests/mesh/run.sh), once per scenario: headscale_hosted, headscale_in_cluster, and the headscale_existing policy check, plus the role's refusal of a Docker Compose release that recreates the containers it has just built. headscale_in_cluster also runs with github_runner_cluster_docker_context pinned while the current Docker context points at nothing, which covers the cluster role's own docker calls and the recovery playbook's, and context_checks checks the role refuses a pin it cannot honour. regrade turns memory-pressure regrading on and steps a server down to etcd-only under (simulated) pressure and back up, checking that it keeps its etcd seat and vote and that no other server steps meanwhile. native_server joins a native host (a systemd container) as an agent beside three Docker servers, then promotes it to a server in place: it must join etcd as a learner, be promoted to a voter, and keep the cluster writable throughout. rejoin removes the bootstrap Docker server's etcd member and checks the server idles on its tombstone, refuses to start a second cluster from its moved-aside datastore with no server to join, joins the existing cluster from a join URL in its role directory with the same cluster ID, and idles as a server whose role file names the agent. role_controller adds a standby agent, runs the role controller's probe and controller from the image built from this checkout, makes a voter slow with tc netem and pauses another, and checks every probe reports the link medium the cluster role labelled its node with, the controller reports the voter unfit with the swap named, reports that it would not act with a voter paused, and changes nothing. role_actuator acts as the role controller by hand on three Docker servers and an eligible native standby: the native host's role actuator promotes it to a voter with writes succeeding throughout, steps it down to etcd-only with no pod's shim left, and once its member is removed through k3s's annotation finishes it as an agent with the same node password and Tailscale address and its datastore moved aside, the two k3s units never running together; then a Docker server's role-actuator service demotes it the same way, its agent keeping the server's node password and Tailscale address as one Node with the datastore moved aside, and promotes it again into the same etcd cluster. role_swap, role_swap_paused, role_swap_blocked_peer and role_swap_refused run the role controller's own voter swap by command, with the controller's image built from this checkout: a native standby in and the Docker bootstrap host out with writes throughout and the bootstrap host back in a second swap; the outgoing voter paused (the removal still completes, and the tombstone keeps it out on its return); the candidate's peer port dropped (the promotion is rolled back); and a swap refused while a voter is paused. role_controller_swap runs the controller at the ARC role's default period, makes a voter slow, and turns the dry run off, checking the controller swaps the slow voter for the Docker standby on its own decision; it is a scenario of its own because filling a window of samples at that period takes as long as the controller says. Only on changes that can affect how nodes join the mesh.
 #
 # Runs on GitHub-hosted ubuntu-latest, whose Docker daemon runs the privileged containers the test needs (k3s, and Tailscale inside it). The work directory sits under runner.temp, which the job always has write access to.
 
@@ -32,7 +32,8 @@ jobs:
   mesh:
     name: Mesh (${{ matrix.scenario }}, Compose ${{ matrix.compose }}${{ matrix.pin && ', context pinned' || '' }})
     runs-on: ubuntu-latest
-    timeout-minutes: 45
+    # role_controller_swap first fills 100 samples at the default 15-second period (25 minutes), waits up to 15 minutes for the induced unfitness and then allows the controller's own swap budget: seven 5-minute operations, a 10-minute runner wait and a 5-minute learner stall (50 minutes). Those bounds total 90 minutes before cluster setup, so CI allows the 15-minute setup and teardown budget as well.
+    timeout-minutes: 105
     strategy:
       fail-fast: false
       matrix:
@@ -44,8 +45,13 @@ jobs:
           - {scenario: regrade, compose: v5.5.1}
           - {scenario: native_server, compose: v5.5.1}
           - {scenario: role_controller, compose: v5.5.1}
+          - {scenario: role_controller_swap, compose: v5.5.1}
           - {scenario: rejoin, compose: v5.5.1}
           - {scenario: role_actuator, compose: v5.5.1}
+          - {scenario: role_swap, compose: v5.5.1}
+          - {scenario: role_swap_paused, compose: v5.5.1}
+          - {scenario: role_swap_blocked_peer, compose: v5.5.1}
+          - {scenario: role_swap_refused, compose: v5.5.1}
           - {scenario: hosted, compose: v2.39.0}
           - {scenario: compose_faulty, compose: v2.38.2}
           - {scenario: in_cluster, compose: v5.5.1, pin: true}