compute: run MV sink correction maintenance on a blocking thread

[antiguru]

Follow-up to #38074, which disabled ENABLE_SYNC_MV_SINK by default after incident 1175.

The sync MV sink's write Tokio task performed correction-buffer maintenance inline. Inserts merge chains spanning the whole buffer and consolidation sorts it, neither with an await point in between, so a pass over a large buffer occupied a Tokio worker thread for its entire duration and stopped it polling everything else scheduled there. Persist lease heartbeats were among the casualties: readers lost their leases and the process halted with batch fetcher could not fetch batch part. The async sink never hit this, because its operators run on the Timely thread.

Under memory pressure the pass is not merely slow. CorrectionV2 pages its chunks out through the column pager, and consolidation calls Chunk::column on every chunk it merges, so the chunks come back one blocking read at a time from inside the sorts and merges. The thread is blocked in the kernel rather than computing. That rules out fixing this by yielding: there is no program point between the faults at which a yield could be placed, and a cooperative fix is only ever as strong as the worst un-yielded path. Running the work on a blocking thread lets the OS preempt it, which holds regardless of what the buffer does.

Correction::updates_before is split into consolidate_before, which does the expensive part, and consolidated_updates_before, which reads the result back. updates_before becomes the composition of the two, so the async sink and the benchmarks are unaffected. The CorrectionV2 since guard moves from updates_before into consolidate_before; consolidate_at_since always passes an upper beyond the since, so it is unaffected.

The write task runs the correction work on a blocking thread and then feeds the updates to the persist batch builder from the async task, where the loop awaits at every part flush and so stays bounded.

The cost is a thread hop per command, one or two per Timely activation. Note that persist-client's IsolatedRuntime documents a preference against spawn_blocking for CPU-bound work, on thread-count and shutdown grounds. It is used here deliberately: the failure mode is a thread blocked in the kernel, and OS preemption is what addresses that.

This does not re-flip the ENABLE_SYNC_MV_SINK default, which stays false pending gradual rollout via LaunchDarkly. mzcompose continues to set the parameter to true, so CI exercises the path.

Adds write::tests::correction_work_does_not_stall_the_runtime, which stalls every worker thread of a two-worker runtime inside the correction path via a new mv_sink_correction failpoint and asserts an unrelated task keeps getting polled. Moving the apply_batch body back out of spawn_blocking fails it. The failpoint also drives the end-to-end lease-expiry scenario when set through the FAILPOINTS environment variable on a clusterd process; the failpoints session variable only reaches environmentd, never a cluster.

Release notes

This release will not include user-visible changes.

🤖 Generated with Claude Code