adapter: remove ENABLE_CLUSTER_CONTROLLER and the legacy REFRESH scheduler

[aljoscha]

Part 1 of 3 of the design for removing the legacy cluster paths.
Stacked on #38078 (the design doc). Parts 2 and 3 stack on this one.

Why

The cluster controller has been default-on since v26.29 and owns the replica
set of every user managed cluster in production. It still lives behind the
break-glass dyncfg ENABLE_CLUSTER_CONTROLLER, and that flag is the only thing
keeping the legacy REFRESH scheduler reachable: both of its entry points return
before doing any work while the gate is on, so it is a strict no-op in
production today.

Removing the gate means reverting to the legacy paths requires a binary
rollback. The gate has several releases of burn-in behind it, and the direct
reshape path (see part 3) remains as the operational escape hatch.

What lands

• The ENABLE_CLUSTER_CONTROLLER dyncfg: definition, registration, its five
  read sites, and the sqllogictest binary's force-on entry.
• src/adapter/src/coord/cluster_scheduling.rs wholesale:
  check_scheduling_policies, check_refresh_policy,
  handle_scheduling_decisions, and the SchedulingDecision /
  RefreshDecision types.
• Its coordinator plumbing: Message::CheckSchedulingPolicies and
  Message::SchedulingDecisions with their handlers, the timer, the
  select-loop tick, and the cluster_scheduling_decisions state.
• The metrics mz_check_scheduling_policies_seconds and
  mz_handle_scheduling_decisions_seconds (and their rows in the generated
  doc/user/data/metrics.yml).
• The system var cluster_check_scheduling_policies_interval, whose sole
  consumer was the timer.
• ReplicaCreateDropReason::ClusterScheduling, constructed only by the
  scheduler.

The persisted audit vocabulary stays: SchedulingDecisionsWithReasonsV2 and
friends are written by the controller's on-refresh path too
(refresh_window_decision_to_audit_log), and old events must remain decodable
regardless.

Tests

• test/sqllogictest/mz_cluster_schedules.slt: the
  cluster_check_scheduling_policies_interval validation block goes with the
  var.
• test/testdrive/cluster-controller.td: the cc_handoff* legacy-handoff
  scenarios and the cc_strand break-glass section are deleted (both are
  gate-off scenarios). The freeze-the-controller technique in the
  unmanaged-conversion refusal block is replaced by cranking
  cluster_controller_tick_interval up. Part 3 adds a replacement for
  cc_strand that exercises the direct cut-over escape hatch instead.
• test/pg-cdc/cluster-graceful-reconfiguration.td: the explicit legacy
  foreground section goes, the controller section stays.
• misc/python/materialize/mzcompose/__init__.py pins the flag only for
  versions below v26.38, so mixed-version runs against older binaries still
  exercise the same path. parallel_workload/action.py drops it from the
  do-not-flip list, and the two launchdarkly-flag-consistency allowlist entries
  go.

[aljoscha]

hmm, does our new controller actually run on a similar interval to where we had this before? Now I'm curious.

[aljoscha]

Close, but slower, and the difference is absorbed by a knob that was always the real one.

The old poll was cluster_check_scheduling_policies_interval, default 3s. The controller's is cluster_controller_tick_interval, default 5s. So worst-case detection latency for a scheduling decision goes 3s -> 5s.

The controller's interval is only a fallback cadence, reconcile_now cuts the sleep short after a catalog transaction changes durable cluster state. That doesn't help here though: an ON REFRESH window opening is driven by wall clock and frontier movement, not by a catalog write, so refresh scheduling really does ride the 5s tick.

Why the 2s doesn't worry me: the window decision is

write_frontier < read_ts + hydration_time_estimate

so the turn-on already leads the refresh by hydration_time_estimate. The tick only has to be small relative to that lead, and REFRESH EVERY periods are minutes to days. The one case where the 2s is visible is hydration_time_estimate unset, which defaults to 0: the window then opens exactly when the refresh is due and the cluster comes up up to 5s late instead of up to 3s late, delaying that refresh by the same amount. No correctness consequence, and if that latency matters you want a non-zero estimate anyway, which is what the option is for.

Worth saying explicitly: both numbers are the polling interval, not a bound on how long a refresh takes. Turning the cluster on is the cheap part.

Happy to drop cluster_controller_tick_interval to 3s if you'd rather keep the cadence identical. I left it at 5s because it's the controller's own tuned default and it now covers reconfiguration and autoscaling too, not just scheduling, so it's a broader tradeoff than the old scheduling-only poll.

[mtabebe]

Change looks good for me. Thanks for updating all the comments too

[mtabebe]

🥳

[aljoscha]

tyty! 🙇‍♂️

you probably saw, and I don't want to hurry you, but there are some more cleanups left in this stack 😅