doc: design for removing the legacy cluster scheduling and staged reconfig paths - #38078
Closed
aljoscha wants to merge 5 commits into
Closed
doc: design for removing the legacy cluster scheduling and staged reconfig paths#38078aljoscha wants to merge 5 commits into
aljoscha wants to merge 5 commits into
Conversation
aljoscha
force-pushed
the
aljoscha/cluster-legacy-00-design
branch
from
August 7, 2026 05:27
18abe8d to
26a81a2
Compare
…duler The cluster controller has been default-on since v26.29 and owns the replica set of every user managed cluster. The break-glass dyncfg kept two legacy paths reachable: the REFRESH scheduler in `cluster_scheduling.rs`, whose two entry points already returned before doing any work while the gate was on, and the legacy branches in the ALTER sequencer. Delete the gate and everything only it kept alive: the scheduler module, its coordinator plumbing (two messages, the timer, the select-loop tick, the `cluster_scheduling_decisions` state), the two scheduler metrics, the `cluster_check_scheduling_policies_interval` system var, and the `ReplicaCreateDropReason::ClusterScheduling` variant. The persisted audit vocabulary stays: `SchedulingDecisionsWithReasonsV2` and friends are written by the controller's on-refresh path too, and old events must remain decodable.
`ManagedClusterIds` and the sequencer's two ownership tests excluded system/builtin clusters, so the sequencer kept a second, complete replica-materialization implementation alive just for them. The exception was originally load-bearing (the boot-time builtin replica migration and the controller would have been two conflicting writers), but `reconcile_builtin_cluster_replicas` now converges a builtin cluster's replica set on the cluster's own managed config, the same config the controller derives its targets from, so the two converge by construction. Drop the `is_user()` conjuncts. Runtime ALTERs of system clusters now flow through the controller like any other managed cluster: a config-shape change reshapes into a durable reconfiguration record, a factor change updates the config and the controller converges the replica set within a tick. The boot migration and the controller compose in both directions. The controller matches replicas by shape and count, never by name, so the migration-created `r1..rN` satisfy its baseline. The migration converges by canonical name, so a boot after a reshape renames or re-creates replicas the controller materialized under generator names, which is harmless churn on processes that are cold at boot anyway.
With the controller owning every managed cluster's replica set, `NeedsFinalization::Yes` has no producer. Delete the machine it drove: the `WaitForHydrated` and `Finalize` stages, the `-pending` overlap replicas, the `pending_cluster_alters` connection state and its retire paths, and the `AlterClusterWhilePendingReplicas` error. The direct reshape path is deliberately kept, as the synchronous cut-over. It is now routed to by an explicitly zero-timeout commit strategy (`WITH (WAIT FOR '0s')`, or `WAIT UNTIL READY (TIMEOUT '0s', ON TIMEOUT 'COMMIT')`) rather than by the absence of a `WAIT` clause. Two reasons. It is the escape hatch: every other reshape depends on the controller ticking and applying, and this is the one that still works when the controller itself is the problem, while also unsticking a wedged reconfiguration by retiring its record. And the semantics are honest: a zero timeout with commit already means "cut over now, hydrated or not", so doing it synchronously in the ALTER is the same outcome minus a tick. "The same outcome" has to be true, so the cut-over does not improvise. It folds its target onto an in-flight one exactly as the reshape path does, and it converges the replica set with the controller's own reconcile kernel, so a replica that already has the target shape is kept rather than bounced. Forcing a stuck-but-hydrating resize to commit therefore keeps the replica that was already up, and lands the record on `finalized` (forced) rather than `cancelled`, since the cut-over reached the record's own target. Creating a replica from the controller's `ReplicaShape` also drops a lossy round-trip through the planner's `ComputeReplicaConfig`, which cannot represent `INTROSPECTION DEBUGGING` without an interval. A cluster in that state used to get a replica whose logging disagreed with the config that called for it.
Graceful cluster reconfiguration has been behind the `enable_zero_downtime_cluster_reconfiguration` feature flag, default off, so the `WITH (WAIT ...)` surface was rejected at plan time unless a deployment turned it on. That is now the only thing standing between an operator and the synchronous cut-over, which is the escape hatch for a reshape when the cluster controller itself is the problem. A break-glass path behind a default-off flag is not a break-glass path. Remove the flag and the planner gate. The two rejections that share that code path stay: a `WAIT` without a replica-shape change, and a `WAIT` on an unmanaged cluster. Every test that used the surface enabled the flag itself, so those statements go. In a mixed-version run some phases execute against a released binary that still enforces the gate (platform-checks' graceful-reconfiguration check runs its first manipulate phase there), so `get_minimal_system_parameters` pins the flag on below v26.37 instead. The docs drop the private-preview badges: the surface is generally available once nothing gates it.
…onfig paths Records the design the four changes below this one implement, and what the implementation settled that the design left open (the folded cut-over target, the reconcile-kernel reuse, and how a cut-over settles a carried record). Reference only, not intended to merge.
aljoscha
force-pushed
the
aljoscha/cluster-legacy-00-design
branch
from
August 7, 2026 07:37
26a81a2 to
fe7b182
Compare
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Design for removing the three legacy cluster paths the cluster controller has
superseded: the REFRESH scheduler in
cluster_scheduling.rs, the staged(
WaitForHydrated/Finalize/-pending) reconfiguration machine, and thesequencer's direct replica create/drop path for system clusters.
The design is in three parts, each a separate PR stacked on this one:
ENABLE_CLUSTER_CONTROLLERdyncfg and the legacy REFRESHscheduler (38079)
The direct whole-set recreate path is deliberately kept, re-keyed from "no
WAITclause" to "an explicitly zero-timeout commitWAIT". It is the escapehatch that still works when the controller itself is the problem, and it
doubles as the cleanup tool for a wedged reconfiguration. See the Alternatives
section for why deleting it entirely was rejected.