Skip to content

feat(java): expose aggregate index fragment coverage - #8961

Open
everySympathy wants to merge 3 commits into
lance-format:mainfrom
everySympathy:codex/java-index-coverage
Open

feat(java): expose aggregate index fragment coverage#8961
everySympathy wants to merge 3 commits into
lance-format:mainfrom
everySympathy:codex/java-index-coverage

Conversation

@everySympathy

Copy link
Copy Markdown
Contributor

Why

Java callers can inspect each segment fragment list today, but must materialize every fragment id in the JVM to calculate logical index coverage. This is expensive for large datasets and does not expose the serialized bitmap size.

What

  • add an aggregate IndexFragmentCoverage result to Rust IndexDescription and Java IndexDescription
  • union all physical segment bitmaps before calculating covered, current, missing, and stale fragment counts
  • report the exact run-optimized manifest serialization size of all segment bitmaps
  • expose a Java coverage ratio without copying the raw bitmap
  • preserve compatibility for external Rust IndexDescription implementations and existing Java constructors

Tests

  • cargo test -p lance-table test_fragment_bitmap_serialized_run_optimized
  • cargo test -p lance test_describe_indices_rows_indexed_stale_bitmap_fragment
  • cd java && ./mvnw test -Dtest=DatasetTest#testDescribeIndicesByName
  • cargo clippy --all --tests --benches -- -D warnings

@github-actions github-actions Bot added enhancement New feature or request A-index Vector index, linalg, tokenizer A-java Java bindings + JNI labels Sep 3, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 3, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 3, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 3, 2026
} else {
JObject::null()
};
let fragment_coverage = if let Some(coverage) = description.fragment_coverage() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to exclude system indexes

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. System indexes should remain visible in describeIndices for compatibility, but their bitmaps do not represent normal data-index coverage (__lance_frag_reuse tracks remapping participation, while __mem_wal may have no bitmap). Fixed in e6c513d at the Rust core layer: all system indexes now return no fragment coverage, so JNI exposes Optional.empty(). Added regression coverage for both bitmap-bearing frag reuse and bitmap-less mem-wal indexes.

@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 3, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

System indexes remain discoverable without reporting misleading data-fragment coverage, and ordinary index summaries retain their aggregate coverage and no-expansion behavior.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer A-java Java bindings + JNI enhancement New feature or request K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants