Skip to content

adapter: let the cluster controller own system clusters - #38080

Merged
aljoscha merged 1 commit into
aljoscha/cluster-legacy-00-designfrom
aljoscha/cluster-legacy-02-system-clusters
Aug 7, 2026
Merged

adapter: let the cluster controller own system clusters#38080
aljoscha merged 1 commit into
aljoscha/cluster-legacy-00-designfrom
aljoscha/cluster-legacy-02-system-clusters

Conversation

@aljoscha

@aljoscha aljoscha commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Part 2 of 3 of the design for removing the legacy cluster paths.
Stacked on #38079. Part 3 stacks on this one.

Why

System/builtin clusters were excluded from the controller's ownership
(ManagedClusterIds and the sequencer's two ownership tests all required
is_user()), so the sequencer had to keep a second, complete
replica-materialization implementation alive just for them.

The exception was originally load-bearing: the boot-time builtin replica
migration and the controller would have been two conflicting writers of one
replica set. #37929 removed that conflict. reconcile_builtin_cluster_replicas
now converges a builtin cluster's replica set on the cluster's own managed
config, the same config the controller derives its targets from, so the two
converge on the same set by construction.

What lands

Drop the three is_user() conjuncts. Runtime ALTERs of system clusters now
flow through the controller like any other managed cluster: a config-shape
change reshapes into a durable reconfiguration record, a factor change updates
the config and the controller converges the replica set within a tick.

Ownership and the boot migration compose in both directions. The controller
matches replicas by shape and count, never by name, so the migration-created
r1..rN satisfy its baseline and a steady system cluster reconciles to no
decisions. The migration converges by canonical name, so a boot after a reshape
(or with a reconfiguration in flight) renames or re-creates replicas the
controller materialized under generator names. That is harmless churn: every
replica is a cold process at boot anyway, so the cost is replica-id and
audit-log noise, and an in-flight record is durable, so the controller picks the
reconfiguration back up on its first tick.

Boot ordering and 0dt read-only behavior are unchanged.
reconcile_builtin_cluster_replicas still runs at catalog open, which is what
gets system clusters up and hydrated before the serve loop starts and before a
0dt cutover. The controller cannot cover either window: Coordinator::bootstrap
runs before the controller task is spawned, and the controller is inactive while
read-only.

Also rewrites the comment in validate_reconfiguration_resource_limits. The
early return for non-user clusters stays, since system clusters are exempt from
max_replicas_per_cluster and credit accounting everywhere else, but the reason
it gave ("a system cluster never reshapes into a record") is now false.

Behavior changes

  • ALTER CLUSTER mz_system SET (SIZE ...) goes from a synchronous whole-set
    recreation to a background graceful reconfiguration (default deadline 24h,
    ROLLBACK on timeout). Strictly more capable: the graceful path did not exist
    for system clusters before, and part 3 adds WITH (WAIT FOR '0s') for
    operators who cannot afford the overlap set.
  • ALTER CLUSTER mz_support SET (REPLICATION FACTOR 1) updates the config and
    returns; the controller materializes the replica within a tick. Same as user
    clusters today.

Tests

  • test/sqllogictest/system-cluster.slt: the block asserting a replica count
    right after a factor flip is reworked. slt does not retry, so it now asserts
    the declared factor (synchronous) and phrases the system-id property as "no
    replica that isn't system-id'd", which holds regardless of convergence.
  • test/testdrive/system-cluster.td (which does retry) picks up the
    convergence coverage: factor up materializes exactly one system-id replica,
    a SIZE reshape cuts the realized config over and leaves exactly one replica
    at the new size, and factor back to 0 retires it with no activity left behind.
  • src/environmentd/tests/bootstrap_builtin_clusters.rs: the three assertions
    that immediately follow a runtime ALTER CLUSTER now poll
    (await_declared_and_actual). The bootstrap and post-restart assertions stay
    exact, since the catalog-open migration converges synchronously.
  • test/cluster/resources/resource-limits.td already uses retrying queries for
    its mz_analytics factor flips, so it survives unchanged and pins the
    system-cluster limit exemption end to end.
  • Docs: the self-managed troubleshooting guide's mz_catalog_server resize
    walkthrough gets a note that the resize is a graceful reconfiguration and
    SHOW CLUSTERS settles rather than flips.

@aljoscha
aljoscha force-pushed the aljoscha/cluster-legacy-02-system-clusters branch from 3f40ada to 6b9f124 Compare August 6, 2026 12:43
@aljoscha
aljoscha force-pushed the aljoscha/cluster-legacy-02-system-clusters branch from 6b9f124 to 5816b58 Compare August 7, 2026 05:27
`ManagedClusterIds` and the sequencer's two ownership tests excluded
system/builtin clusters, so the sequencer kept a second, complete
replica-materialization implementation alive just for them. The exception was
originally load-bearing (the boot-time builtin replica migration and the
controller would have been two conflicting writers), but
`reconcile_builtin_cluster_replicas` now converges a builtin cluster's replica
set on the cluster's own managed config, the same config the controller derives
its targets from, so the two converge by construction.

Drop the `is_user()` conjuncts. Runtime ALTERs of system clusters now flow
through the controller like any other managed cluster: a config-shape change
reshapes into a durable reconfiguration record, a factor change updates the
config and the controller converges the replica set within a tick.

The boot migration and the controller compose in both directions. The
controller matches replicas by shape and count, never by name, so the
migration-created `r1..rN` satisfy its baseline. The migration converges by
canonical name, so a boot after a reshape renames or re-creates replicas the
controller materialized under generator names, which is harmless churn on
processes that are cold at boot anyway.
@aljoscha
aljoscha force-pushed the aljoscha/cluster-legacy-02-system-clusters branch from 5816b58 to 45f74e0 Compare August 7, 2026 07:37
Base automatically changed from aljoscha/cluster-legacy-01-scheduler to aljoscha/cluster-legacy-00-design August 7, 2026 07:37
@aljoscha
aljoscha merged commit 45f74e0 into main Aug 7, 2026
@aljoscha
aljoscha deleted the aljoscha/cluster-legacy-02-system-clusters branch August 7, 2026 07:37
@aljoscha
aljoscha restored the aljoscha/cluster-legacy-02-system-clusters branch August 7, 2026 07:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant