Skip to content

feat: integrate stable partitions with FRI V1 - #9007

Draft
LuQQiu wants to merge 2 commits into
lance-format:lu/friv2from
LuQQiu:lu/friv2-mixed-design
Draft

feat: integrate stable partitions with FRI V1#9007
LuQQiu wants to merge 2 commits into
lance-format:lu/friv2from
LuQQiu:lu/friv2-mixed-design

Conversation

@LuQQiu

@LuQQiu LuQQiu commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Stable partitions can retain existing B-tree segments while compaction continues writing FRI V1. This implements the mixed lifecycle using #8972's RowMapWriter and RowMapReader, without duplicating the codec or migrating V1 history.

  • Publish prepared stable partitions and immutable _row_maps/<uuid>/row_map.lance references atomically through Operation::Rewrite, with paired reader/writer capability flags.
  • Keep stored segment bitmaps as source provenance. Derive coverage from the union of a logical index's segments, probe every contributor, and give directly indexed destinations priority over translated entries.
  • Compose V1 and V2 by fragment lineage. Translate B-tree pages on cold, cached, and streamed reads with bounded row-map block reads; route Python remap_row_addrs through the same Rust translator.
  • Preserve mappings across append/delete and subsequent eager or deferred V1 compaction. Reject changed sources and unresolved repartitioning, allow disjoint partitions to rebase, and protect committed mapping files during cleanup. The prepared-commit API requires an explicit AllowUnindexed policy to accept incomplete source coverage.

Experimental scope: physical row addresses, one storage base, and B-tree query/maintenance integration. Other affected index types fall back to scanning. Automatic clustering, coordinated draining/pruning, and cloning mixed datasets remain outside this implementation. This is a draft; the formal format-change process is deferred until the design is agreed.

Stack: #8972this PR, with base branch lu/friv2. Replaces closed #8978. GitHub's native stack API rejects fork PRs, so the dependency is represented by the PR base branch.

Validation: 17 FRI tests and 6 table-format/transaction/flag tests passed; 2 targeted Python regression tests passed. Workspace cargo fmt --all and cargo clippy --all --tests --benches -- -D warnings, Python uv run make lint, JNI Clippy, and commit hooks passed. The Python extension was rebuilt with make build.

@LuQQiu LuQQiu changed the title docs: propose mixed FRI V1 and V2 lifecycle feat: integrate stable partitions with FRI V1 Sep 4, 2026
@LuQQiu LuQQiu reopened this Sep 4, 2026
@github-actions github-actions Bot added A-python Python bindings A-java Java bindings + JNI A-format On-disk format: protos and format spec docs format-change A change to the format spec, which requires a vote. Remove if minor (e.g. fixing typo). enhancement New feature or request labels Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-docs Documentation A-format On-disk format: protos and format spec docs A-java Java bindings + JNI A-python Python bindings documentation Improvements or additions to documentation enhancement New feature or request format-change A change to the format spec, which requires a vote. Remove if minor (e.g. fixing typo).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant