Skip to content

adapter: don't leak arrangement compression into the realized config - #38112

Draft
aljoscha wants to merge 1 commit into
aljoscha/cluster-legacy-04-zdt-flagfrom
aljoscha/cluster-legacy-06-compression-fix
Draft

adapter: don't leak arrangement compression into the realized config#38112
aljoscha wants to merge 1 commit into
aljoscha/cluster-legacy-04-zdt-flagfrom
aljoscha/cluster-legacy-06-compression-fix

Conversation

@aljoscha

@aljoscha aljoscha commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Stacked on #38104.

Arrangement compression is a replica config shape dimension: it is part of ReconfigurationTarget, the reconcile kernel distinguishes replicas by it, and the controller transitions it through the hydrate-overlap like SIZE. Several hand-maintained dimension lists forgot it.

The reshape leak. reshape_alter_cluster_managed writes the reconfiguration record while resetting the config shape carried by new_config back to the realized values, deferring the transition to the record's cut-over. The reset enumerated size, replication factor, availability zones, and logging, but missed arrangement compression. A compression-carrying ALTER thus flipped the realized config immediately: the baseline strategy then desired the new shape and bounced the baseline replicas mid-reconfiguration, defeating the graceful overlap, and the cut-over later wrote the same value again. The reset now goes through realized_reconfiguration_target / apply_reconfiguration_target instead of hand-listing dimensions, so a target dimension can no longer be missed at this site.

The planner gate. The WAIT gate rejected a compression-only ALTER as having nothing to wait on, even though such an ALTER transitions through a hydrate-overlap like any other shape change. It is now allowed (and the error message names the option).

The observability surface. mz_cluster_reconfigurations.changes diffed only the other four dimensions, so a compression-only record reported an empty diff while in progress, and the SHOW CLUSTERS activity summary showed such a cluster as steady mid-reconfiguration. Both now report the dimension.

Tests. The compression ALTERs in managed_cluster.slt relied on the leak: their SHOW CREATE readback only saw the new value because the reshape wrote it early. They now request the synchronous cut-over (WAIT FOR '0s'), which advances the realized config in the ALTER's own transaction. ALTER ... RESET takes no WITH clause, so the RESET spelling is exercised for acceptance without a readback. cluster-controller.td gains regression coverage: the cc_preserve section asserts that a folding compression ALTER lands in the record's target and changes while the realized config stays untouched and only the overlap replica is bounced, and a new section runs a compression-only ALTER through the record path end to end.

Known gap, left for a follow-up: the audit event's reconfiguration details intentionally omit compression, and a shape-dimension RESET cannot carry a WAIT clause (the parser only accepts WITH on SET).

Motivation

Fixes a bug in a feature that has yet to be released.

Checklist

  • This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
  • This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
  • If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
  • If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
  • If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

@aljoscha
aljoscha requested review from a team as code owners August 7, 2026 11:44
@aljoscha
aljoscha marked this pull request as draft August 7, 2026 11:49
@aljoscha
aljoscha force-pushed the aljoscha/cluster-legacy-06-compression-fix branch 2 times, most recently from 6ce5344 to cf73bc0 Compare August 7, 2026 12:28
Arrangement compression is a replica config shape dimension: it is part of
`ReconfigurationTarget`, the reconcile kernel distinguishes replicas by it,
and the controller transitions it through the hydrate-overlap like SIZE.
Several hand-maintained dimension lists forgot it.

The reshape path writes the reconfiguration record while resetting the
config shape carried by `new_config` back to the realized values, deferring
the transition to the record's cut-over. The reset enumerated size,
replication factor, availability zones, and logging, but missed arrangement
compression. A compression-carrying ALTER thus flipped the realized config
immediately: the baseline strategy then desired the new shape and bounced
the baseline replicas mid-reconfiguration, defeating the graceful overlap,
and the cut-over later wrote the same value again. Reset via
`realized_reconfiguration_target` and `apply_reconfiguration_target`
instead of hand-listing dimensions, so a target dimension can no longer be
missed here.

The planner's WAIT gate had the same gap: it rejected a `WAIT` clause on a
compression-only ALTER as having nothing to wait on, even though such an
ALTER transitions through a hydrate-overlap like any other shape change.
Allow it.

The observability surface had it too: `mz_cluster_reconfigurations.changes`
diffed only the other four dimensions, so a compression-only record
reported an empty diff while in progress, and the SHOW CLUSTERS activity
summary showed such a cluster as steady. Both now report the dimension.

The compression ALTERs in managed_cluster.slt relied on the realized-config
leak: their SHOW CREATE readback only saw the new value because the reshape
wrote it early. They now request the synchronous cut-over (WAIT FOR '0s'),
which advances the realized config in the ALTER's own transaction. ALTER
... RESET takes no WITH clause, so the RESET spelling is exercised for
acceptance without a readback. cluster-controller.td gains regression
coverage: the cc_preserve section asserts that a folding compression ALTER
lands in the record's target and changes while the realized config stays
untouched and only the overlap replica is bounced, and a new section runs a
compression-only ALTER through the record path end to end.

Changing the MV's definition changes its builtin fingerprint, so the
builtin schema migration registers a replacement step for
`mz_cluster_reconfigurations`, and the catalog-docs golden and the
mz_show_clusters plan golden are rewritten.
@aljoscha
aljoscha force-pushed the aljoscha/cluster-legacy-06-compression-fix branch from cf73bc0 to 8f11dd9 Compare August 7, 2026 12:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant