adapter: let the cluster controller own system clusters - #38102
Open
aljoscha wants to merge 1 commit into
Open
Conversation
aljoscha
force-pushed
the
aljoscha/cluster-legacy-02-system-clusters
branch
from
August 7, 2026 08:01
45f74e0 to
17bdfda
Compare
aljoscha
marked this pull request as ready for review
August 7, 2026 08:14
aljoscha
commented
Aug 7, 2026
| ClusterControllerRequest::ManagedClusterIds { tx } => { | ||
| let ids = if active { | ||
| // Every managed cluster, system ones included. | ||
| // `reconcile_builtin_cluster_replicas` materializes a builtin |
Contributor
Author
There was a problem hiding this comment.
I don't think we need to document this reconciliation behavior here
Contributor
Author
There was a problem hiding this comment.
Agreed, dropped it. The module doc already states the ownership contract and the convergence with reconcile_builtin_cluster_replicas, which is the right home for it, so the match arm now just loses the is_user() conjunct with no commentary.
`ManagedClusterIds` and the sequencer's two ownership tests excluded system/builtin clusters, so the sequencer kept a second, complete replica-materialization implementation alive just for them. The exception was originally load-bearing (the boot-time builtin replica migration and the controller would have been two conflicting writers), but `reconcile_builtin_cluster_replicas` now converges a builtin cluster's replica set on the cluster's own managed config, the same config the controller derives its targets from, so the two converge by construction. Drop the `is_user()` conjuncts. Runtime ALTERs of system clusters now flow through the controller like any other managed cluster: a config-shape change reshapes into a durable reconfiguration record, a factor change updates the config and the controller converges the replica set within a tick. The boot migration and the controller compose in both directions. The controller matches replicas by shape and count, never by name, so the migration-created `r1..rN` satisfy its baseline. The migration converges by canonical name, so a boot after a reshape renames or re-creates replicas the controller materialized under generator names, which is harmless churn on processes that are cold at boot anyway.
aljoscha
force-pushed
the
aljoscha/cluster-legacy-02-system-clusters
branch
from
August 7, 2026 08:28
17bdfda to
e7ffa90
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 2 of 3 of the design for removing the legacy cluster paths.
Stacked on #38101. Part 3 stacks on this one.
Why
System/builtin clusters were excluded from the controller's ownership
(
ManagedClusterIdsand the sequencer's two ownership tests all requiredis_user()), so the sequencer had to keep a second, completereplica-materialization implementation alive just for them.
The exception was originally load-bearing: the boot-time builtin replica
migration and the controller would have been two conflicting writers of one
replica set. #37929 removed that conflict.
reconcile_builtin_cluster_replicasnow converges a builtin cluster's replica set on the cluster's own managed
config, the same config the controller derives its targets from, so the two
converge on the same set by construction.
What lands
Drop the three
is_user()conjuncts. Runtime ALTERs of system clusters nowflow through the controller like any other managed cluster: a config-shape
change reshapes into a durable reconfiguration record, a factor change updates
the config and the controller converges the replica set within a tick.
Ownership and the boot migration compose in both directions. The controller
matches replicas by shape and count, never by name, so the migration-created
r1..rNsatisfy its baseline and a steady system cluster reconciles to nodecisions. The migration converges by canonical name, so a boot after a reshape
(or with a reconfiguration in flight) renames or re-creates replicas the
controller materialized under generator names. That is harmless churn: every
replica is a cold process at boot anyway, so the cost is replica-id and
audit-log noise, and an in-flight record is durable, so the controller picks the
reconfiguration back up on its first tick.
Boot ordering and 0dt read-only behavior are unchanged.
reconcile_builtin_cluster_replicasstill runs at catalog open, which is whatgets system clusters up and hydrated before the serve loop starts and before a
0dt cutover. The controller cannot cover either window:
Coordinator::bootstrapruns before the controller task is spawned, and the controller is inactive while
read-only.
Also rewrites the comment in
validate_reconfiguration_resource_limits. Theearly return for non-user clusters stays, since system clusters are exempt from
max_replicas_per_clusterand credit accounting everywhere else, but the reasonit gave ("a system cluster never reshapes into a record") is now false.
Behavior changes
ALTER CLUSTER mz_system SET (SIZE ...)goes from a synchronous whole-setrecreation to a background graceful reconfiguration (default deadline 24h,
ROLLBACKon timeout). Strictly more capable: the graceful path did not existfor system clusters before, and part 3 adds
WITH (WAIT FOR '0s')foroperators who cannot afford the overlap set.
ALTER CLUSTER mz_support SET (REPLICATION FACTOR 1)updates the config andreturns; the controller materializes the replica within a tick. Same as user
clusters today.
Tests
test/sqllogictest/system-cluster.slt: the block asserting a replica countright after a factor flip is reworked. slt does not retry, so it now asserts
the declared factor (synchronous) and phrases the system-id property as "no
replica that isn't system-id'd", which holds regardless of convergence.
test/testdrive/system-cluster.td(which does retry) picks up theconvergence coverage: factor up materializes exactly one system-id replica,
a
SIZEreshape cuts the realized config over and leaves exactly one replicaat the new size, and factor back to 0 retires it with no activity left behind.
src/environmentd/tests/bootstrap_builtin_clusters.rs: the three assertionsthat immediately follow a runtime
ALTER CLUSTERnow poll(
await_declared_and_actual). The bootstrap and post-restart assertions stayexact, since the catalog-open migration converges synchronously.
test/cluster/resources/resource-limits.tdalready uses retrying queries forits
mz_analyticsfactor flips, so it survives unchanged and pins thesystem-cluster limit exemption end to end.
mz_catalog_serverresizewalkthrough gets a note that the resize is a graceful reconfiguration and
SHOW CLUSTERSsettles rather than flips.