refactor(disttae): redesign CN transaction workspace

[gouhongshen]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

Related to #26796.

The issue remains open as the benchmark record for transaction performance versus transaction size. This PR removes the positional workspace architecture that caused current-state operations to depend on accumulated transaction history; it does not claim to close every transaction-performance problem recorded in the issue.

What this PR does / why we need it

Problem

The CN transaction workspace used physical positions in a global write slice as statement boundaries, read visibility, rollback markers, and commit inputs. Readers and transaction operators depended on txnOffset, snapshotWriteOffset, offsets, and direct access to txn.writes.

That representation coupled logical transaction semantics to mutable slice layout. Statement retry, compaction, spilling, DDL rollback, CN transfer, and payload reclamation had to scan or rewrite shared positional state. As retired history grew, current-state operations could become proportional to the transaction's total history instead of the active working set.

Architecture

This PR replaces the positional model with explicit owners and opaque identities:

• StatementJournal owns statement identity, retry attempts, rollback resources, RC boundaries, DDL operations, and LOAD-created files.
• TableOverlay owns each table's active mutations, PK candidates, row/block tombstones, uncommitted objects, and DDL visibility.
• PayloadStore owns memory batches and spilled objects through generation-based leases. Readers pin payload generations instead of retaining unowned raw batch pointers.
• WorkspaceReadView is an immutable logical visibility boundary. WorkspaceWriteMark identifies one write scope without exposing physical workspace positions.
• CommitBuilder freezes logical mutations, pins referenced payloads, and builds the existing TN commit requests without mutating live workspace state.

The workspace maintains ordered active indexes. Current-state reads, commit preparation, object-delete lookup, DDL lookup, and rollback use active indexes rather than scanning retired mutation history. Historical access is limited to an explicit statement-scoped read view.

Transaction semantics

The implementation preserves and tests:

• statement begin/end, active write-scope ownership, out-of-order branch completion, exactly-once closure, and retry identity;
• statement rollback and RC snapshot advancement as one workspace boundary transition;
• reader visibility across local, combined, delegated, and sharded data sources;
• CN transfer and table-meta readers using the exact read view captured by their execution;
• three-phase spill (capture/pin -> I/O -> validate/publish) so stale or rolled-back attempts cannot publish;
• DDL create/drop/rename visibility and exact rollback, including one active catalog name per table ID;
• LOAD object ownership and cleanup across statement retries;
• payload generation reclamation only after all leases close;
• construction of the existing TN commit protocol through CommitBuilder.

Protocol-order correctness

Logical visibility order and TN precommit protocol order are separate concerns. This PR now models the latter with a commitEpoch advanced by ordinary Compile read-view publication:

• write scopes are active ownership tokens rather than a LIFO stack, because UNION branches and internal Data Branch SQL may complete in a different order from their start order;
• internal SQL inherits the caller's epoch, allowing later-produced catalog mutations to precede user-table mutations within one Compile;
• a new ordinary Compile advances the epoch, preventing a later statement's catalog mutation from crossing an earlier statement boundary;
• same-statement physical rewrites preserve the source mutation's immutable commit order;
• a later statement rewriting an older mutation receives the current epoch, so recreated catalog metadata is sent before rewritten table data.

This fixes the DDL/DML regression exposed by delete_workspace.sql without reintroducing physical workspace offsets.

API changes

The internal engine workspace contract no longer exposes positional write offsets:

• PublishReadView and CurrentReadView replace snapshot/write offset accessors.
• BeginWriteAttempt and Adjust(WorkspaceWriteMark) replace positional adjustment.
• Ranges, CollectTombstones, and reader construction accept WorkspaceReadView instead of transaction offsets.

All in-repository callers and mocks are migrated. No compatibility wrapper or fallback positional implementation is retained.

Performance

Benchmarks cover retired histories of 0, 1,000, and 10,000 mutations. Current-state operations keep identical allocation counts and show no growth trend with retired history:

• table entries, commit entries, compaction candidates, PK candidates, dump scope, dropped-table lookup, and object references remain sub-microsecond at these sizes;
• object-delete snapshots and payload reclamation remain insensitive to unrelated retired history;
• RC boundary state remains allocation-free;
• ordered mutation snapshot cost is linear only in active mutations, as intended.

Validation

Passed locally on a clean MO instance at port 6002 (port 6001 was already owned by Docker and was not disturbed):

• go test ./pkg/vm/engine/disttae -count=1
• make build
• DDL BVT: 7,815 total, 7,695 passed, 120 ignored, 0 failed, 100%
• DML BVT excluding CDC: 5,855 total, 5,735 passed, 120 ignored, 0 failed, 100%
• Snapshot BVT: 17,716 total, 17,716 passed, 0 failed, 100%
• CDC syntax: 4/4 passed
• CDC behavior: 11/11 passed using an uncommitted temporary copy with only the hard-coded endpoint changed from 127.0.0.1:6001 to the test instance at 127.0.0.1:6002
• git diff --check

The ownership, visibility, lock-order, payload-lifecycle, spill, rollback, and commit contracts are documented in docs/design/cn-transaction-workspace.md.