feat(python): expose DataOverlay commit operation

[wjones127]

Exposes the DataOverlay commit operation to Python so overlays can be created and committed from Python (needed to benchmark and use data overlay files without dropping into Rust). Mirrors the existing DataReplacement binding.

What's here

• LanceOperation.DataOverlay with DataOverlayFile and DataOverlayGroup. A DataOverlayFile carries the value DataFile plus exactly one of shared_offsets (dense coverage shared by every field) or field_offsets (sparse, one offset set per field). The commit stamps committed_version; passing both/neither coverage is rejected with a clear error.
• PyO3 conversions (both directions) in python/src/transaction.rs.
• Fills in the overlays field on the Python FragmentMetadata -> Fragment conversion, which OSS-1322 left unset (Python metadata doesn't carry overlays — they're committed via DataOverlay).

Tests

test_data_overlay_* in test_dataset.py: dense round-trip resolves on read; newest overlay wins; sparse per-field coverage resolves fields independently; and the exactly-one-coverage validation rejects both/neither.

Stacking

Stacked on #7536 (OSS-1324 read path) — base retargets automatically when it lands. Next PR (benchmarks) stacks on this.

Note: the python/src/fragment.rs one-liner arguably belongs in the OSS-1322 PR (#7535), which added Fragment.overlays without updating the binding; included here so the stack compiles.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Added cell-level DataOverlay operations to append fragment overlays without rewriting entire fragments.
  • Supports dense and sparse overlays, with newest overlapping values taking precedence.
  • Persists overlay metadata on fragments and carries optional committed version stamps.
• Bug Fixes
  • Preserves overlay metadata correctly through JSON serialization and Python↔Rust round-trips.
• Tests
  • Added end-to-end coverage for dense/sparse overlays, precedence behavior, metadata round-tripping, and offset validation.
• Documentation/Chores
  • Updated fragment metadata repr expectations and improved default max_bytes_per_file for fragment writing.

[wjones127]

note: this could certainly be more efficient. I filed a follow up: #7695

[wjones127]

suggestion: it might be more pythonic if we had one field and accepted either type signature. Then we differentiate which it is based on the type.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds cell-level data overlay operations, fragment metadata persistence, Python-Rust interoperability, and end-to-end coverage for dense, sparse, precedence, round-trip, and validation behavior.

Changes

Data Overlay Support

Layer / File(s)	Summary
Overlay contracts and fragment metadata python/python/lance/dataset.py, python/python/lance/fragment.py	Defines overlay files, groups, and operations; stores overlay metadata in fragment JSON; adds the default fragment file-size constant.
Rust overlay interoperability python/src/transaction.rs, python/src/fragment.rs	Converts overlay coverage, groups, operations, and fragment metadata between Python and Rust.
Overlay behavior validation python/python/tests/test_dataset.py, python/python/tests/test_fragment.py	Tests dense and sparse overlays, newest-wins precedence, metadata round trips, invalid offsets, and updated fragment metadata representation.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Dataset
  participant DataOverlay
  participant PyO3Bindings
  participant RustOperation
  participant FragmentMetadata
  Dataset->>DataOverlay: create overlay groups
  DataOverlay->>PyO3Bindings: convert groups and coverage
  PyO3Bindings->>RustOperation: construct DataOverlay operation
  RustOperation->>FragmentMetadata: preserve overlay metadata
  FragmentMetadata->>PyO3Bindings: serialize fragment overlays
  PyO3Bindings->>Dataset: reconstruct overlay metadata

Loading

Possibly related PRs

• lance-format/lance#7772: Uses overlay counts and committed versions in compaction and transaction logic.

Suggested reviewers: xuanwo, yanghua, jackye1995

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: exposing the Python DataOverlay commit operation.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch will/overlay-python-commit

---

_{Comment @coderabbitai help to get the list of available commands.}

[westonpace]

Just some minor nits

[westonpace]

Wouldn't we still want it to round-trip? Most of our functions don't take Fragment as input so it's hard to come up with a good example.

Probably the only example I could think of would be a Merge operation. If I built a "deleting" Merge by taking a set of fragments and dropping one of them then the resulting commit would drop all the overlays from the fragments I keep too?

[wjones127]

I've got it so it can roundtrip now.

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@python/src/transaction.rs`:
- Around line 210-249: The PyLance<DataOverlayFile>::extract implementation must
reject unsorted offsets before constructing OverlayCoverage, rather than
allowing RoaringBitmap::from_iter to reorder them. Validate that the dense
offsets list and every sparse per-field list are strictly ascending, return a
clear ValueError on failure, and preserve the existing dense/sparse parsing
behavior otherwise. The corresponding dataset.py site requires no direct change;
it is covered by this Rust-side validation.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 0c7ed629-7e5b-4d27-bb03-c03381753098

📥 Commits

Reviewing files that changed from the base of the PR and between 9effdb1 and 968d024.

📒 Files selected for processing (5)

• python/python/lance/dataset.py
• python/python/lance/fragment.py
• python/python/tests/test_dataset.py
• python/src/fragment.rs
• python/src/transaction.rs

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)

python/python/lance/dataset.py (1)

5994-6062: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Add examples and API links for the new public overlay operations.

DataOverlayFile, DataOverlayGroup, and DataOverlay are public APIs but lack usage examples and links to related types or LanceDataset.commit.

As per coding guidelines, “Document all public APIs with examples and links to relevant structs and methods.”

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/python/lance/dataset.py` around lines 5994 - 6062, Add documentation
examples and cross-references to the public APIs DataOverlayFile,
DataOverlayGroup, and DataOverlay. Show how to construct and use overlay
operations, link related types such as DataFile and FragmentMetadata, and
reference LanceDataset.commit for applying DataOverlay, while preserving the
existing attribute and behavior descriptions.

Source: Coding guidelines

python/src/transaction.rs (1)

234-247: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Report out-of-range offsets as a value error. offsets=[-1] and values above u32::MAX fall through both extraction branches and end up with the generic “must be a list of ints” message. Reject them with a bounds-specific error and add coverage for those cases.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/src/transaction.rs` around lines 234 - 247, Update the offsets parsing
flow around the dense and sparse `offsets.extract` branches to detect integer
offsets outside the u32 range and return a PyValueError with a bounds-specific
message instead of the generic type error. Preserve valid dense and sparse
coverage handling, and add tests covering negative offsets and values above
u32::MAX.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@python/python/lance/dataset.py`:
- Around line 5994-6062: Add documentation examples and cross-references to the
public APIs DataOverlayFile, DataOverlayGroup, and DataOverlay. Show how to
construct and use overlay operations, link related types such as DataFile and
FragmentMetadata, and reference LanceDataset.commit for applying DataOverlay,
while preserving the existing attribute and behavior descriptions.

In `@python/src/transaction.rs`:
- Around line 234-247: Update the offsets parsing flow around the dense and
sparse `offsets.extract` branches to detect integer offsets outside the u32
range and return a PyValueError with a bounds-specific message instead of the
generic type error. Preserve valid dense and sparse coverage handling, and add
tests covering negative offsets and values above u32::MAX.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 1afbc6a3-a664-4ce5-91d1-6167dff20438

📥 Commits

Reviewing files that changed from the base of the PR and between 968d024 and de8c63c.

📒 Files selected for processing (4)

• python/python/lance/dataset.py
• python/python/tests/test_dataset.py
• python/python/tests/test_fragment.py
• python/src/transaction.rs