feat(table): stable partition transition on the rewrite commit path

[LuQQiu]

Stack

	PR
2	👉 #8978 — transition + commit path
1	#8972 — row map format (base of this stack)

---

What

Part 2 of reordered-rewrite support, stacked on #8972 (the first four commits are that PR; review the last commit here). This wires the row map format into the commit path: a reordered rewrite now commits through the ordinary Operation::Rewrite, carrying a new stable partition transition that lands in the manifest as its own system index entry.

Design

The transaction carries nothing new on disk

Operation::Rewrite gains an in-memory stable_partition field following the frag_reuse_index precedent exactly: it is dropped when the transaction serializes and None when one is read back. Other writers' conflict decisions only need the fragment sets in groups, which are exact either way — so a reader or writer that predates this feature sees a perfectly ordinary rewrite transaction, not an unknown field.

The manifest carries a new system index entry

__lance_stable_partition (separate from __lance_frag_reuse, whose proto is untouched) accumulates one transition per reordered rewrite:

• sources — ordered fragment digests; their physical row counts are the row map's addressing base (prefix sums position each fragment's rows in the concatenated label sequence),
• destinations — ordered fragment ids; a row map label is an index into this list,
• the row map file reference ({indices dir}/{row_map_id}/row_map.lance).

Inline below 200KB, spilled to an external details.binpb above it, mirroring the fragment reuse entry. The entry's fragment bitmap is the union of all transition source ids — provenance, deliberately keeping retired fragments (IndexMetadata::fragment_bitmap documents that ids may no longer exist).

What the commit path does differently for reordered groups

For an ordinary (compaction) group, manifest build swaps index fragment bitmaps from old to new fragments — legal only because row order is preserved. A reordered group redistributes rows, so its groups skip bitmap maintenance entirely, for every index kind: the bitmaps keep the retired source ids. That single choice is what makes existing readers degrade instead of lie — a bitmap of retired ids intersects the live fragments as empty, so the index is skipped for those rows and queries fall back to correct scans. A group straddling reordered and order-preserving sources is rejected.

Coverage-based index retention now exempts all system indices (previously only the fragment reuse entry), since their bitmaps are provenance and an empty intersection with live fragments does not mean the entry is obsolete.

Conservation is validated before the transaction exists

build_stable_partition_rewrite checks the rewrite against the counts the row map file itself carries — no extra IO:

• every destination's physical rows equal its label total,
• the row map covers exactly the sources' physical rows,
• labeled (live) rows equal the sources' live rows (physical − deleted).

Concurrent reordered rewrites

Because the transaction field is never serialized, one reordered rewrite cannot detect another through the transaction file. Instead the commit records which entry version it appended onto (base_entry_version), and manifest build rejects the splice when the manifest's entry no longer matches — the alternative would silently drop the concurrent transition. The rejected caller rebuilds against the latest entry and retries.

Compatibility

• Old readers: the entry is an unrecognized system index name — carried through manifests losslessly, never opened, queries degrade to scans for rows whose only index coverage was through retired sources. Verified in the end-to-end test.
• Old writers: the transaction file contains a plain Rewrite; nothing new to drop or misread.

Tests

• End-to-end reordered rewrite through apply_commit: entry installed and round-trips (sources, destinations, row map ref), the scalar index bitmap keeps the retired source ids instead of following the rewrite, a filtered query on the now-uncovered column still answers correctly (degrade path), and a second rewrite built against a stale entry version is rejected with the concurrent-rewrite error.
• Conservation rejections: destination row count vs label total, destination count vs the row map's m, source rows vs row map rows.
• ordered_rewrite_groups: partitioning, no-op without a stable partition, and the mixed-group error.
• Details proto round trip.

cargo test -p lance-table (382) and the new -p lance tests pass; clippy clean for the new code.

Follow-ups (separate PRs)

1. Read integration: coverage derivation and decode-time translation through the existing RowIdRemapper seams; row map file lifecycle (cleanup integration).
2. Conflict handling: deletion-vector fold on rebase via the sweep translator; combining disjoint concurrent rewrites instead of rejecting them.

🤖 Generated with Claude Code