Skip to content

feat(python): add Bitmap binding for RoaringBitmap#7837

Draft
wjones127 wants to merge 1 commit into
lance-format:mainfrom
wjones127:feat/python-bitmap-7695
Draft

feat(python): add Bitmap binding for RoaringBitmap#7837
wjones127 wants to merge 1 commit into
lance-format:mainfrom
wjones127:feat/python-bitmap-7695

Conversation

@wjones127

Copy link
Copy Markdown
Contributor

Two sites in python/src/transaction.rs copy a RoaringBitmap through a Python set/list on every Rust↔Python crossing:

This adds a Bitmap type (lance.bitmap.bitmap) wrapping Arc<RoaringBitmap>:

  • Constructible from a Python iterable of ints, a range, or an integer pyarrow Array/ChunkedArray (read via a zero-copy buffer, not per-value Python objects).
  • __len__/__contains__/__iter__/__eq__, add/discard/update (copy-on-write via Arc::make_mut), and pickling.

Both transaction.rs sites now emit Bitmap on output and accept it directly on input (an Arc clone instead of a Python-set round trip), while still accepting a plain set/list on input for backward compatibility. The same set-expansion pattern existed in two more places — IndexSegment.fragment_ids and IndexSegmentDescription.fragment_ids in indices.rs (used by list_indices()/describe_indices()) — so those were switched to Bitmap too.

FragmentMetadata.to_json() normalizes Bitmap/List[Bitmap] offsets back to plain (nested) lists of ints for JSON serialization.

Fixes #7695

Test plan

  • New python/tests/test_bitmap.py: construction (list/range/pyarrow array of various int widths/chunked array), len/iter/in, equality, pickle round-trip, copy-on-write mutation isolation, error cases (negative values, out-of-range values, nulls).
  • Extended test_dataset.py overlay tests to cover Bitmap-typed offsets (dense + sparse) and assert the output type; updated the invalid-offsets error-message assertion.
  • Extended test_indices.py/test_scalar_index.py to assert Index.fragment_ids/IndexInformation.fragment_ids are Bitmap.
  • cargo fmt --all, cargo clippy -p pylance --lib --tests -- -D warnings, uv run ruff format/ruff check, uv run pyright all clean.

Two sites in python/src/transaction.rs (Index.fragment_ids, and
DataOverlayFile.offsets added in lance-format#7540) copy a RoaringBitmap through a
Python set/list on every Rust<->Python crossing. This adds a Bitmap
type wrapping Arc<RoaringBitmap>, constructible from a Python iterable,
range, or pyarrow integer Array/ChunkedArray, with copy-on-write
mutation. Both sites now emit/accept Bitmap directly (falling back to
plain sets/lists on input for backward compatibility), and the same
fix is applied to the two IndexSegment(Description) getters in
indices.rs that had the identical set-expansion pattern.

Fixes lance-format#7695
@github-actions github-actions Bot added A-python Python bindings enhancement New feature or request labels Jul 18, 2026
@coderabbitai

coderabbitai Bot commented Jul 18, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Pro Plus

Run ID: 20a2c896-b4dc-4133-801b-31ca4f5f6288

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-python Python bindings enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(python): make Python binding for Roaring Bitmap

1 participant