feat(control): thread transaction id through cross-node dispatch (F1)

[emanzx]

What

Implements F1 — transaction-id threading through cross-node dispatch from the locked cross-shard transaction design.

An explicit transaction's staged writes live in per-core overlays keyed by the session's transaction id. Before this change the id was threaded only on local/single-shard paths — the gateway's cross-node hop dropped it. So when a coordinator was not the leader for a collection's shard:

• in-transaction reads could not see the transaction's own staged rows (broken read-your-own-writes), and
• a forwarded StageWrite was stranded (rejected: "StageWrite dispatched without a txn_id").

Changes

• ExecuteRequest carries an Option<u64> txn_id; QueryContext, DispatchRouteStreamParams, and RemoteDispatchArgs carry Option<TxnId>. Autocommit call sites pass None; every transactional path threads the session id through the gateway, remote executor, and all-cores fan.
• gather_all_vshards stops dropping txn_id at its QueryContext boundary — the cross-node in-transaction read now reaches the owning leader correctly keyed.
• dispatch_local routes in-transaction plans via a new dispatch_to_data_plane_in_txn so the id reaches the Data Plane overlay.
• The pgwire whole-batch gateway-forward fast path is now skipped inside a transaction block. It is an autocommit optimization; an in-transaction write forwarded through it reached the leader as a bare Raft-committed write, bypassing the staging overlay — this broke read-your-own-writes and left ROLLBACK unable to undo it. In-block tasks fall through to the staging gate, which stages writes and forwards reads/stage-ops to the leader.
• Transaction ids are made globally unique (node id in the high 16 bits) so two coordinators' overlays never collide on a shard that hosts both; the overlay-drop meta-op now carries the id so ROLLBACK's DropTxnOverlay routes to the owning leader, not a local replica.

Testing

New 3-node e2e cluster_txn_cross_node_ryow.rs runs, from every node, BEGIN / staged INSERT / point + scan RYOW / ROLLBACK / post-rollback-empty — so at least two iterations drive the non-leader forward path.

Green: the new e2e, sql_transactions_*_overlay + native_transactions_staging (single-node overlay regressions), calvin_cluster_pgwire_e2e + calvin_ollp_cross_node + single_node_calvin, rpc_codec roundtrips, session unit tests. clippy clean on nodedb + nodedb-cluster.

Scope / not in this PR

• U7 (session lease / idle GC) is intentionally left out: it hits a wiring fork — the lease sweeper must reach the per-handler SessionStores, which are not reachable from SharedState where the spawn_loop sweeper pattern lives. Flagging for a design decision rather than guessing the wiring.
• U4 / U4a (validation barrier + write-admission gate) are out of scope by design.

[farhan-syah]

F1-only scope is fine — F10 and U7 are correctly out. Two blocking defects inside F1's own diff, plus a test gap. Please continue on this branch.

Blocker 1 — cross-node COMMIT applies staged writes on the wrong node

Removing the pgwire whole-batch fast path (routing/execute.rs:206-208) is the right call and is intrinsic to F1 — without it, in-block writes never reach the staging gate and there is nothing to thread. But it redirects in-block writes into the commit-apply path, and that path was only ever correct when the coordinator is the shard leader.

• dispatch_single_shard (shared/session/commit.rs:224-232) builds the batch task as txn_id: None, wal_lsn: Some(..).
• The new pgwire reroute (pgwire/handler/dispatch.rs:384-388) fires only on wal_lsn.is_none() && task.txn_id.is_some(). Both false here → falls through to submit_to_data_plane, executing locally.
• A few lines earlier, state.wal.append_transaction(tenant_id, vshard_id, ..) writes a WAL record on the coordinator for a vShard it does not lead.
• drop_txn_overlay (commit.rs:314) does now carry txn_id: Some(..), so it reroutes to the remote leader and discards the staged rows.

Net: BEGIN; INSERT; COMMIT from a non-leader coordinator over pgwire lands the row only in the coordinator's local data plane + WAL, outside Raft. The leader never receives it; the overlay holding the real copy is dropped. Silent data loss + replica divergence.

This is not the "still rejects cross-shard COMMIT" carve-out — that guards write sets spanning ≥2 vShards. This is a single-shard txn whose one shard happens to be led elsewhere. It classifies as SingleShard and sails past the rejection.

Not a regression on the native transport: NativeTxnDp::dispatch_no_wal (native/dispatch/transaction.rs:50-71) routes through the gateway unconditionally. That asymmetry is why native_transactions_staging stayed green.

Fix: route the TransactionBatch apply to the vShard leader and stamp the WAL there, symmetrically with the DropTxnOverlay reroute already added. The teardown leg knows how to find the leader; the apply leg doesn't. Scoped to dispatch_single_shard + the dispatch_task_no_wal guard — does not pull in F10 or U4.

Blocker 2 — node_id << 48 silently truncates

shared/session/transaction.rs:53-55 mints (node_id << 48) | (counter & 0xFFFF_FFFF_FFFF) and the comment promises global uniqueness. But node_id is a u64 from NODEDB_NODE_ID (config/server/cluster.rs:22) and validate() rejects only zero (cluster.rs:158). 65536u64 << 48 == 0 silently, in debug and release — shift-overflow checks only cover the shift amount, not lost bits. So nodes 1 and 65537 mint identical TxnIds and their overlays collide on any shard hosting both: exactly the failure this change exists to prevent.

Fix: reject node_id >= 1 << 16 in ClusterSettings::validate() with a typed Error::Config. A silent mask is not acceptable here (see CLAUDE.md on silent truncation at hard caps).

Also: the NEXT_TXN_ID doc comment (transaction.rs:16-19, "unique per shard … scoped to a single shard's in-memory state") is now stale and contradicts the new invariant. Update it.

Test gap

cluster_txn_cross_node_ryow.rs ends every one of its three iterations in ROLLBACK. ROLLBACK exercises only DropTxnOverlay — the one meta-op whose routing this PR fixed. The path the PR actually altered (in-block writes now reaching the commit-apply leg) has no coverage, which is why Blocker 1 shipped green.

Please add:

1. A COMMIT iteration asserting the row is visible from a different node afterward. On current main+this branch that fails.
2. Two concurrent transactions from two different coordinators staging to the same shard. This is the only test that exercises the node-id-in-TxnId change; nothing today does.

Note gh pr checks reports no checks ran on this branch at all.

Verified correct — no action

• watermark_lsn: Lsn::new(0) conservatism holds. commit.rs:58 aborts iff current > read_lsn && current > snapshot_lsn; a zero read version makes the first term trivially true, so it can only widen aborts. Matches the design doc's §U2 note.
• payloads.into_iter().next().unwrap_or_default() does not drop rows — Gateway::execute fuses multi-route payloads into one before returning (gateway/core.rs:326-332), so the vec is never longer than 1. It mirrors the native leg exactly. Fragile (correctness rests on a non-local invariant) but correct.
• The resolve/exchange.rs "guarded above" comment is accurate: line 166's txn_id.is_none() genuinely guards that branch.
• Deferring U7 matches the design doc's unit ordering.

Unrelated churn

Cargo.lock gains source/checksum for the fluxbench crates. Orthogonal to this change — almost certainly local [patch.crates-io] state. Split it out.

[emanzx]

Pushed 30c7916 addressing Blocker 2, the test gap's config half, and the churn:

• Blocker 2 — ClusterSettings::validate() now rejects node_id >= 1 << 16 with a typed Error::Config (no mask). Boundary tests: 65535 ok, 65536/65537/2^20 rejected. Stale NEXT_TXN_ID doc comment updated to the low-48 / node-id-high-16 split.
• Cargo.lock — reverted the fluxbench source/checksum churn; the lock is now byte-identical to origin/main.

Blocker 1 — need your call on the mechanism before I write it. Confirmed the diagnosis: non-leader single-shard COMMIT local-WALs + applies the TransactionBatch outside Raft, and DropTxnOverlay discards the real copy. The reroute you point at is the right shape, but "stamp the WAL there" has no in-scope mechanism today:

• append_transaction (wal/manager/append.rs:81) is local-WAL only — it does not replicate.
• The Data-Plane batch apply (handlers/transaction/batch.rs) deliberately does not WAL; it relies on the CP having written the record. So a batch merely forwarded to the leader via the gateway applies to the leader's base with no WAL on the leader — a leader-crash loss, just relocated.
• ReplicatedEntry carries a single ReplicatedWrite; there is no atomic transaction-batch Raft entry.

So the two ways to make the leader durably own the commit both have a tradeoff I won't pick unilaterally on the txn core:

1. N replicated proposals — replay each buffered write through the leader's group via the existing to_replicated_entry path (autocommit-symmetric, leader WALs+applies+replicates each). Minimal, fully in scope — but it drops the single-TransactionBatch atomicity: a crash between entries 3 and 4 leaves a partial commit on a single shard.
2. New atomic batch entry — a ReplicatedWrite/ReplicatedEntry variant carrying the whole batch, proposed as one Raft entry so the leader applies it atomically. Preserves single-shard commit atomicity, but adds a wire type + an apply-path arm (beyond dispatch_single_shard + the guard).

Which do you want? I lean (2) to keep single-shard COMMIT atomic, but it's your durability contract. Once you confirm, I'll land it with the COMMIT-visible-from-another-node e2e (fails on main+this branch today) and the two-coordinator-same-shard concurrency test.

(Re CI: no workflow is configured for PRs from the fork — happy to add one, or run whatever suite you point me at.)