Skip to content

adapter: delete the staged cluster reconfiguration machine - #38081

Merged
aljoscha merged 1 commit into
aljoscha/cluster-legacy-00-designfrom
aljoscha/cluster-legacy-03-staged-machine
Aug 7, 2026
Merged

adapter: delete the staged cluster reconfiguration machine#38081
aljoscha merged 1 commit into
aljoscha/cluster-legacy-00-designfrom
aljoscha/cluster-legacy-03-staged-machine

Conversation

@aljoscha

@aljoscha aljoscha commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Part 3 of 3 of the design for removing the legacy cluster paths.
Stacked on #38080.

Why

With the controller owning the replica set of every managed cluster (parts 1
and 2), NeedsFinalization::Yes has no producer left, so the staged
reconfiguration machine it drove is unreachable.

What is deleted

  • ClusterStage::WaitForHydrated and ClusterStage::Finalize, their structs,
    dispatch arms, and handlers (check_if_pending_replicas_hydrated_stage,
    finalize_alter_cluster_stage).
  • NeedsFinalization and PENDING_REPLICA_SUFFIX, and the pending parameter
    of create_managed_cluster_replica_op.
  • The pending_cluster_alters connection state and its retire paths:
    drop_reconfiguration_replicas, retire_cluster_reconfigurations_for_conn,
    cancel_cluster_reconfigurations_for_conn, and their call sites in
    connection cleanup and cancellation.
  • The AlterClusterWhilePendingReplicas error (its raise site goes with the
    machine, its catch site went with the scheduler in part 1).
  • In sequence_alter_cluster_managed_to_managed: the pending-replica arm and
    the separate scale-up/down branches. A factor-only change is now a
    config-only write the controller converges, like a user cluster's today.

What is kept, and why

The whole-set recreate branch stays, as the direct cut-over path: retire any
carried reconfiguration record as cancelled, write the target config, and
synchronously drop the observed owned replica set and recreate it at the target
shape and factor, all in one catalog transaction with no controller
involvement.

It is now routed to by an explicitly zero-timeout commit strategy
(WITH (WAIT FOR '0s'), or WAIT UNTIL READY (TIMEOUT '0s', ON TIMEOUT 'COMMIT')) instead of by AlterClusterPlanStrategy::None. Two reasons:

  • Escape hatch. Once the gate is gone there is no break-glass flag, and
    every other reshape depends on the controller ticking and applying. The
    direct path is the one reshape that still works when the controller itself is
    the problem, and it simultaneously unsticks a wedged reconfiguration by
    retiring the record. Requesting it under a live controller stays safe: the
    config write invalidates any in-flight tick's compare-and-append witness, so a
    stale controller batch is rejected, the same as any user DDL landing mid-tick.
  • Honest semantics. A zero timeout with commit already means "cut over now,
    hydrated or not". Doing it synchronously in the ALTER instead of one
    controller tick later is the same outcome, minus the tick.

A zero timeout that rolls back is not this: it asks for the reconfiguration
to be abandoned at once, which is the record path's job, so it keeps writing a
record.

AlterClusterPlanStrategy and ClusterStage::AwaitReconfiguration stay (the
controller reshape path and its foreground wait-shim consume them, as does
cluster_alter_check_ready_interval), as do
remove_pending_cluster_replicas_migration (now the only remaining cleaner for
replicas stranded durably pending by a crash on a pre-deletion version, so it
retires later together with the durable field) and
reconcile_builtin_cluster_replicas.

Behavior change

WITH (WAIT FOR '0s') used to write a record the controller committed on its
next tick. It now returns with the cut-over already transacted. Same outcome,
one tick sooner, and it works even when the controller does not.

Tests

  • test/testdrive/cluster-controller.td: a new "Direct cut-over: the
    zero-timeout commit escape hatch" section. A sleeping materialized view pins
    hydration so the controller can never cut over, leaving a wedged
    reconfiguration with a stranded overlap replica; WITH (WAIT FOR '0s') then
    returns with the realized config advanced, exactly one replica at the target
    size, the record cancelled, and a started -> cancelled audit trail. A
    second case covers the WAIT UNTIL READY (TIMEOUT '0s', ON TIMEOUT 'COMMIT')
    spelling with a factor change riding along. This replaces the cc_strand
    break-glass section part 1 removed.
  • The two existing controller-forced cut-over scenarios (cc_force_existing,
    cc_rollback) move from TIMEOUT '0s' to TIMEOUT '1ms', so they keep
    exercising the controller's past-deadline commit rather than being rerouted
    to the direct path.
  • test/sqllogictest/managed_cluster.slt: the zero-timeout WAIT cases now
    read back the size (and, where a factor rides along, the replica count)
    without retrying, which is exactly what the synchronous path buys.

@aljoscha
aljoscha force-pushed the aljoscha/cluster-legacy-03-staged-machine branch 2 times, most recently from 2e87368 to c351566 Compare August 6, 2026 15:49
@aljoscha aljoscha closed this Aug 7, 2026
@aljoscha aljoscha reopened this Aug 7, 2026
@aljoscha
aljoscha force-pushed the aljoscha/cluster-legacy-03-staged-machine branch from c351566 to c420239 Compare August 7, 2026 05:27
With the controller owning every managed cluster's replica set,
`NeedsFinalization::Yes` has no producer. Delete the machine it drove: the
`WaitForHydrated` and `Finalize` stages, the `-pending` overlap replicas, the
`pending_cluster_alters` connection state and its retire paths, and the
`AlterClusterWhilePendingReplicas` error.

The direct reshape path is deliberately kept, as the synchronous cut-over. It
is now routed to by an explicitly zero-timeout commit strategy
(`WITH (WAIT FOR '0s')`, or `WAIT UNTIL READY (TIMEOUT '0s', ON TIMEOUT
'COMMIT')`) rather than by the absence of a `WAIT` clause. Two reasons. It is
the escape hatch: every other reshape depends on the controller ticking and
applying, and this is the one that still works when the controller itself is
the problem, while also unsticking a wedged reconfiguration by retiring its
record. And the semantics are honest: a zero timeout with commit already means
"cut over now, hydrated or not", so doing it synchronously in the ALTER is the
same outcome minus a tick.

"The same outcome" has to be true, so the cut-over does not improvise. It
folds its target onto an in-flight one exactly as the reshape path does, and it
converges the replica set with the controller's own reconcile kernel, so a
replica that already has the target shape is kept rather than bounced. Forcing
a stuck-but-hydrating resize to commit therefore keeps the replica that was
already up, and lands the record on `finalized` (forced) rather than
`cancelled`, since the cut-over reached the record's own target.

Creating a replica from the controller's `ReplicaShape` also drops a lossy
round-trip through the planner's `ComputeReplicaConfig`, which cannot represent
`INTROSPECTION DEBUGGING` without an interval. A cluster in that state used to
get a replica whose logging disagreed with the config that called for it.
@aljoscha
aljoscha force-pushed the aljoscha/cluster-legacy-03-staged-machine branch from c420239 to 385e39c Compare August 7, 2026 07:37
Base automatically changed from aljoscha/cluster-legacy-02-system-clusters to aljoscha/cluster-legacy-00-design August 7, 2026 07:37
@aljoscha
aljoscha merged commit 385e39c into main Aug 7, 2026
@aljoscha
aljoscha deleted the aljoscha/cluster-legacy-03-staged-machine branch August 7, 2026 07:37
@aljoscha
aljoscha restored the aljoscha/cluster-legacy-03-staged-machine branch August 7, 2026 07:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant