feat: resolve data overlay files on the take (and scan) read path by wjones127 · Pull Request #7409 · lance-format/lance

wjones127 · 2026-06-23T04:00:11Z

Resolves the take random-access path against data overlay files, replacing the temporary "overlays not supported" error from #7407 (OSS-1322). Implements OSS-1324.

Because take and scan share FragmentReader::new_read_impl, the merge is wired there once: each row is addressed by its physical offset (from ReadBatchParams::to_offsets_total) and resolved against the overlays that cover its field. This necessarily also enables the scan-path merge that #7407 stubbed out.

How it works

FileFragment::open loads, for each projected field, the overlay value columns that cover it, ordered newest-first (by committed_version, list-position tiebreak).
The merge runs on physical rows in read order, before deletion filtering, so:
- deletions take precedence (an overlay value computed for a deleted row is dropped with the row),
- NULL overrides apply (a covered offset with a NULL value resolves to NULL, distinct from fall-through),
- fields resolve independently.
The resolution core now takes explicit per-row physical offsets instead of a contiguous start, giving one code path for the contiguous scan range and arbitrary take indices.
Sparse per-field overlays read each field's value column independently, so unequal-length value columns (feat(file): v2 writer/reader support columns of unequal length #7406 / OSS-1323) need no rectangular batch. Rank-based addressing only (rank on the coverage bitmap + a value fetch; no offset key column, no binary search).

Overlays on nested (non-top-level) fields are not yet matched and are left for follow-up.

Tests

take covered/uncovered offsets; multiple overlays (newest wins); per-field coverage with unequal-length columns; NULL override; overlay on a deleted row (inert); multi-fragment scan — each over v2.0 and v2.1. Plus unit tests for the offset-based core and the batch merge.

Stacking

Stacked on #7406 (OSS-1323) and #7407 (OSS-1322). Until those merge, this PR's diff against main includes their commits; review only the final commit ("resolve data overlay files on the take and scan read paths").

🤖 Generated with Claude Code

Add a specification for data overlay files: small files attached to a fragment that supply new values for a subset of (row offset, field) cells without rewriting the base data files, for cheap cell-level updates. - protos/table.proto: rework DataOverlayFile with a dense/sparse coverage oneof (shared_offset_bitmap vs new FieldCoverage), rename read_version to committed_version (effective, commit-stamped), and document rank-based addressing with no offset column. Document reader feature flag 64. - docs: add data_overlay_file.md (full spec, worked example, guidance stub) and link it from the table format overview. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add the `DataOverlay` operation (and `DataOverlayGroup`) to attach overlay files to fragments without rewriting their base data. Mirrors the `DataReplacement` batch shape, appends to each fragment's `overlays` list, and documents permissive conflict semantics: concurrent overlays, appends, deletes, and column rewrites are compatible; row-rewrites, compaction, and overlay->base folds conflict. committed_version is left 0 by the writer and stamped at commit time. Proto only — Rust/Python bindings deferred. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The table/transaction proto changes generate new fields and an Operation variant. This wires the minimum needed to compile without implementing overlay support: - Emit empty `overlays` when converting fragments to proto. - Reject the `DataOverlay` transaction operation with NotSupported on read. Datasets that use overlays set reader feature flag 64, which already falls in the unknown-flag range rejected by `can_read_dataset`, so the library refuses them at the feature-flag layer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Adds the in-memory + commit machinery for data overlay files (per the spec in lance-format#7381), the foundation the scanner/take/index/compaction work builds on. - `DataOverlayFile` / `OverlayCoverage` (dense `shared_offset_bitmap` and sparse per-field) with protobuf round-trip, attached to `Fragment.overlays`. - Reader feature flag 64 (`FLAG_DATA_OVERLAY_FILES`): set whenever any fragment carries overlays, so a reader that does not understand them refuses the dataset instead of returning stale base values. - `Operation::DataOverlay` transaction op: appends overlays to a fragment's list (preserving concurrently-written overlays) and stamps each overlay's `committed_version` to the new dataset version at commit time (re-stamped on retry). Conflict rules mirror DataReplacement — permissive against appends, deletes, column rewrites, index builds, and other overlays; conflicts only with row-rewriting compaction of the same fragment. Scan-side merge, take, and end-to-end write+read tests follow in the same PR branch. Part of the Data Overlay Files feature (OSS-1322). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Now that overlays can be committed, a scan or take over a fragment that has overlays would silently return stale base values, since the read-path merge is not implemented yet. Refuse such reads at `FileFragment::open` with a clear error instead of serving incorrect data. Lifted once the scan/take merge lands (rest of OSS-1322 / OSS-1324). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Adds `dataset::overlay`, the tested heart of reading overlays: given a base column for a physical row range and the overlays covering a field (newest first), `resolve_overlay_column` produces the merged column. An offset is resolved to the newest covering overlay's value at the offset's rank in the coverage bitmap; an uncovered offset falls through to the base; a covered offset whose value is NULL overrides the cell to NULL. `overlay_indices_newest_first` orders a fragment's overlays by `committed_version` then list position. Deletion precedence needs no handling here: the merge runs before the deletion filter, so an overlay value for a deleted offset is computed and dropped with the row. Wiring this into the scan stream and `take` follows on this branch. Unit tests cover rank addressing, multi-overlay precedence, NULL override vs. fall-through, physical-offset base, string columns, and ordering. Part of OSS-1322. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The v2 file writer advanced every column from a single global row counter, so a single file could only hold columns of equal length. Sparse data overlay files need columns whose item counts differ within one file (each field covers a different set of rows). Add `FileWriter::write_columns`, which writes a set of `(field, array)` pairs and advances each field's row counter independently, leaving other fields untouched. A field never written ends up as a zero-length column. `write_batch` is unchanged: it still advances all fields together, so ordinary rectangular files round-trip exactly as before. Per-column lengths were already derivable from page metadata; expose them via `FileReader::column_num_rows`. The reader already schedules each column from its own pages, so reading a column at its own length and random access within it work without further changes. Part of the Data Overlay Files feature (OSS-1323). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Wire the overlay cell-resolution core into reads so `take` (and scan) return merged values. `FileFragment::open` loads each projected field's overlay value columns newest-first, and `new_read_impl`/`read_ranges` merge them into base batches by physical offset before deletion filtering — so resolution is identical for take and scan, NULL overrides apply, deletions take precedence (an overlay value for a deleted row is dropped with the row), and fields resolve independently. The merge addresses each row by its physical offset (via `ReadBatchParams::to_offsets_total`), so the resolution core now takes explicit per-row offsets instead of a contiguous start — a single code path for the contiguous scan range and arbitrary take indices. Sparse per-field overlays read each field's value column independently, so unequal-length columns (OSS-1323) are handled without materializing a rectangular batch. Removes the temporary "overlays not supported" guard from OSS-1322. Tests: take of covered/uncovered offsets, multiple overlays (newest-wins), per-field coverage with unequal-length columns, NULL override, overlay on a deleted row (inert), and multi-fragment scan — all over v2.0 and v2.1 files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-06-23T04:00:20Z

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

Start a vote following the Lance community voting process.
Format specification modifications need 3 binding +1 votes (excluding the
proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
Once the vote passes, link the completed vote in this PR. It should not be
merged until the vote is linked.

codecov · 2026-06-23T04:40:57Z

Codecov Report

❌ Patch coverage is 89.85177% with 89 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/io/commit/conflict_resolver.rs	35.71%	27 Missing ⚠️
rust/lance-file/src/writer.rs	84.39%	14 Missing and 8 partials ⚠️
rust/lance-table/src/format/fragment.rs	87.03%	8 Missing and 6 partials ⚠️
rust/lance/src/dataset/fragment.rs	94.31%	3 Missing and 9 partials ⚠️
rust/lance/src/dataset/transaction.rs	91.53%	8 Missing and 3 partials ⚠️
rust/lance/src/dataset/overlay.rs	98.60%	1 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

Overlay value columns were read in full at FileFragment::open, so a take of a few rows read the whole column (O(coverage)) regardless of how many rows were requested, and serially after the base read. Defer the value read to merge time: route each batch's physical offsets against the coverage bitmaps (cheap, no IO), fetch only the needed ranks via the existing take primitive, and run base + overlay reads concurrently. Each overlay file is now opened once (not once per field). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

wjones127 · 2026-06-30T17:28:20Z

Superseded by #7536, relocated into lance-format/lance and properly stacked on the OSS-1322 branch (#7535) so the diff is exactly this change. Closing in favor of the stacked PR.

) > **Supersedes #7409.** Relocated into `lance-format/lance` and **properly stacked on the OSS-1322 branch** so the diff is now exactly this PR's change (no spec/1322/1323 commits to scroll past). Resolves the `take` random-access path against data overlay files, replacing the temporary "overlays not supported" error from OSS-1322. Implements OSS-1324. Because `take` and scan share `FragmentReader`'s read path, the merge is wired there once: each row is addressed by its physical offset (from `ReadBatchParams::to_offsets_total`) and resolved against the overlays that cover its field. This also enables the scan-path merge that the OSS-1322 PR stubbed out. ### How it works - **Lazy, rank-pushed reads.** Each contributing overlay file is opened once (projected to the covered ∩ requested fields) and **no value bytes are read up front**. Requested offsets are routed to the newest covering overlay at their coverage rank, and only the touched ranks are fetched — via the existing `take` primitive — so overlay reads are O(requested rows), not O(coverage). - **Concurrent IO.** Base and overlay reads are issued together, then the column is assembled with `interleave` (rank → fetch-position remap). - The merge runs on physical rows in read order, **before** deletion filtering, so: - deletions take precedence (an overlay value computed for a deleted row is dropped with the row), - NULL overrides apply (a covered offset with a NULL value resolves to NULL, distinct from fall-through), - fields resolve independently. - Sparse per-field overlays read each field's value column independently, so unequal-length value columns (OSS-1323) need no rectangular batch. Rank-based addressing only (rank on the coverage bitmap + a value fetch; no offset key column, no binary search). Overlays on nested (non-top-level) fields are not yet matched and are left for follow-up. There's a `TODO(overlay perf)` on reader priority to settle with a benchmark. ### Tests take covered/uncovered offsets; multiple overlays (newest wins); per-field coverage with unequal-length columns; NULL override; overlay on a deleted row (inert); multi-fragment scan — each over v2.0 and v2.1. Plus unit tests for the routing/assembly core, and an IO guard (`test_take_reads_only_needed_overlay_ranks`) asserting a 2-row take over a fully-covering 100k-row overlay reads ~one miniblock, not the whole value column. ### Stacking Stacked on the **OSS-1322** branch (`will/oss-1322-write-data-overlay-and-scan-data`). Merge that first; this PR's base retargets to `main` automatically when it lands. 🤖 Generated with [Claude Code](https://claude.com/claude-code)  ## Summary by CodeRabbit * **New Features** * Added support for reading dataset fragments that include overlay files. * Overlay values are resolved and merged into results during both scans and targeted row reads, honoring newest-wins semantics across nested fields. * Optimizes reads by fetching only overlay data needed for the requested rows and projections. * **Bug Fixes** * Fragments with overlays are no longer rejected, and overlay resolution remains correct when overlays overlap deleted rows. * **Tests** * Added end-to-end overlay read coverage for scan/take, including multi-batch slicing and edge cases.  --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

wjones127 and others added 8 commits June 22, 2026 18:07

github-actions Bot added A-encoding Encoding, IO, file reader/writer A-format On-disk format: protos and format spec docs enhancement New feature or request labels Jun 23, 2026

This was referenced Jun 23, 2026

feat: mask data overlay files in scalar and vector index queries #7411

Closed

feat: overlay-aware compaction scheduler and rewrite #7421

Closed

wjones127 mentioned this pull request Jun 30, 2026

feat: resolve data overlay files on the take (and scan) read path #7536

Merged

wjones127 closed this Jun 30, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: resolve data overlay files on the take (and scan) read path#7409

feat: resolve data overlay files on the take (and scan) read path#7409
wjones127 wants to merge 9 commits into
lance-format:mainfrom
wjones127:will/oss-1324-take-can-read-overlays

wjones127 commented Jun 23, 2026

Uh oh!

github-actions Bot commented Jun 23, 2026

Uh oh!

codecov Bot commented Jun 23, 2026

Uh oh!

wjones127 commented Jun 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

wjones127 commented Jun 23, 2026

How it works

Tests

Stacking

Uh oh!

github-actions Bot commented Jun 23, 2026

Uh oh!

codecov Bot commented Jun 23, 2026

Codecov Report

Uh oh!

wjones127 commented Jun 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant