feat(java): scan fragment physical row slices

[majin1102]

What

Add a Java/JNI scanner API for selecting physical row slices within Lance fragments:

FragmentSlice slice = new FragmentSlice(fragmentId, rowOffset, rowCount);

ScanOptions options =
    new ScanOptions.Builder()
        .fragmentSlices(Collections.singletonList(slice))
        .filter("price > 100")
        .columns(Arrays.asList("id", "price"))
        .build();

try (LanceScanner scanner = dataset.newScan(options)) {
  // scan
}

A slice covers [rowOffset, rowOffset + rowCount) in the fragment's physical row domain. Deleted rows do not compact offsets and are omitted by normal scans.

Semantics

• fragmentIds and fragmentSlices may be combined; both restrictions apply.
• When only slices are provided, the scanner is restricted to the referenced fragments.
• Overlapping or duplicate slices have set semantics and do not duplicate rows.
• Result order follows Lance fragment / physical-row scan order, not slice input order.
• rowCount == 0 is allowed and represents an empty slice.
• Slices must be used with the dataset snapshot from which they were planned.
• V2 storage is supported. Legacy V1 masked scans return the existing NotSupported error.

How

• Add serializable Java FragmentSlice and ScanOptions.Builder.fragmentSlices(...).
• Pass slices through both blocking and async scanner JNI entry points.
• Parse, validate, and convert slices in their shared native scanner builder.
• Validate fragment existence, physical bounds, arithmetic overflow, row-address limits, and stable row-id sequence length.
• For non-stable row IDs, insert physical address ranges directly into RowAddrTreeMap.
• For stable row IDs, load each fragment's row-id sequence once and translate the requested physical positions.
• Feed the result into the existing:

  scanner.with_row_addr_prefilter(
      RowAddrMask::from_allowed(selected_row_ids)
  );

• Keep deletion handling, refine filters, projection, and limit in the existing FilteredReadExec path.
• Remove the earlier Rust-only with_physical_row_addr_prefilter API; no second public Rust Scanner API or new file-reading path is added.

Tests

Coverage includes:

• stable and non-stable row IDs
• multiple fragments and overlapping slices
• deletions without physical offset compaction
• filter, projection, and limit composition
• empty-projection count-style scans used by Spark COUNT(*) readers
• fragmentIds intersection
• Fragment.newScan(options)
• blocking and async scanners
• empty slices, invalid bounds, missing fragments, negative values, and overflow
• legacy V1 rejection

Verified with:

• cargo fmt --manifest-path ./java/lance-jni/Cargo.toml --all --check
• cargo clippy --manifest-path ./java/lance-jni/Cargo.toml --all-targets
• ./mvnw spotless:check
• ./mvnw test -Dtest=ScannerTest,AsyncScannerTest (58 Java tests + 18 JNI Rust tests)
• ./mvnw test -Dtest='!RestNamespaceTest,!DynamicContextProviderTest' (465 Java tests + 18 JNI Rust tests)

The unfiltered Maven suite ran 485 Java tests; 13 existing REST namespace tests failed because their external endpoint returned HTTP 403. All other tests passed.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-04T04:57:49.633798Z	b8afdc1	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b8afdc1af7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve compressed stable row-id ranges

When a slice covers a large fragment with stable row IDs, this loop expands the compressed RowIdSequence and performs one tree insertion per physical row, making scanner creation O(rowCount) even when the sequence is represented by a single range; overlapping slices repeat the work. Whole-fragment slices over large ML datasets can therefore spend substantial time synchronously enumerating millions or billions of IDs before scanning. Convert each sliced sequence through its range-aware From<&RowIdSequence> implementation and union the resulting maps (ideally after merging overlapping physical ranges) so compressed ranges remain compressed.

Useful? React with 👍 / 👎.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

Fragment slicing needs to remain in physical (fragment, offset) space until it is intersected with deletion state and any fragment scope. Converting tombstoned slots to stable logical IDs lets a slice follow an updated row into another fragment. Keeping a physical selection through scan planning is the cleanest fix and also provides a path to address the separate compressed-range planning cost in the existing discussion.

[lance-gatekeeper]

The stable-ID conversion loses the physical fragment/offset boundary here. A stable row ID is preserved when an update tombstones the old physical slot and writes the live row into another fragment, so adding the deleted slot's ID to this dataset-wide mask can select the replacement row outside the requested slice. This violates the documented fragmentSlices and fragmentIds intersection.

Keep the restriction in physical-address space until it has been intersected with fragment scope and deletion state; a logical-ID-only mask cannot distinguish a deleted incarnation from its live replacement when includeDeletedRows is involved.

Reproducer run against this head

I added this regression to ScannerTest:

@Test
void testFragmentSliceDoesNotFollowUpdatedStableRowId(@TempDir Path tempDir) throws Exception {
  String path = tempDir.resolve("fragment_slice_stable_row_id_update").toString();
  WriteParams params =
      new WriteParams.Builder()
          .withDataStorageVersion(LanceConstants.FILE_FORMAT_VERSION_STABLE)
          .withEnableStableRowIds(true)
          .build();
  try (BufferAllocator allocator = new RootAllocator()) {
    TestUtils.SimpleTestDataset fixture =
        new TestUtils.SimpleTestDataset(allocator, path);
    fixture.createDatasetWithWriteParams(params).close();
    List<FragmentMetadata> metadata = fixture.createNewFragment(8, params);
    try (Dataset original =
        Dataset.commit(
            allocator, path, new FragmentOperation.Append(metadata), Optional.of(1L))) {
      int oldFragment = original.getFragments().get(0).getId();
      org.lance.update.UpdateResult result =
          original.update(
              new org.lance.update.UpdateParams(
                      Collections.singletonMap("name", "'updated'"))
                  .withWhere("id = 2"));
      try (Dataset updated = result.getDataset()) {
        int replacementFragment =
            updated.getFragments().stream()
                .map(Fragment::getId)
                .filter(id -> id != oldFragment)
                .findFirst()
                .orElseThrow();
        ScanOptions options =
            new ScanOptions.Builder()
                .fragmentSlices(
                    Collections.singletonList(new FragmentSlice(oldFragment, 2, 1)))
                .fragmentIds(Collections.singletonList(replacementFragment))
                .columns(Collections.singletonList("id"))
                .build();
        try (LanceScanner scanner = updated.newScan(options)) {
          assertEquals(Collections.emptyList(), readIds(scanner));
        }
      }
    }
  }
}

Run with:

cd java && ./mvnw -Djava.io.tmpdir=/home/agent/tmp -Dtest=ScannerTest#testFragmentSliceDoesNotFollowUpdatedStableRowId test

Expected []; observed [2].