feat: overlay-aware compaction scheduler and rewrite#7421
Closed
wjones127 wants to merge 9 commits into
Closed
Conversation
Add a specification for data overlay files: small files attached to a fragment that supply new values for a subset of (row offset, field) cells without rewriting the base data files, for cheap cell-level updates. - protos/table.proto: rework DataOverlayFile with a dense/sparse coverage oneof (shared_offset_bitmap vs new FieldCoverage), rename read_version to committed_version (effective, commit-stamped), and document rank-based addressing with no offset column. Document reader feature flag 64. - docs: add data_overlay_file.md (full spec, worked example, guidance stub) and link it from the table format overview. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the `DataOverlay` operation (and `DataOverlayGroup`) to attach overlay files to fragments without rewriting their base data. Mirrors the `DataReplacement` batch shape, appends to each fragment's `overlays` list, and documents permissive conflict semantics: concurrent overlays, appends, deletes, and column rewrites are compatible; row-rewrites, compaction, and overlay->base folds conflict. committed_version is left 0 by the writer and stamped at commit time. Proto only — Rust/Python bindings deferred. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The table/transaction proto changes generate new fields and an Operation variant. This wires the minimum needed to compile without implementing overlay support: - Emit empty `overlays` when converting fragments to proto. - Reject the `DataOverlay` transaction operation with NotSupported on read. Datasets that use overlays set reader feature flag 64, which already falls in the unknown-flag range rejected by `can_read_dataset`, so the library refuses them at the feature-flag layer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the in-memory + commit machinery for data overlay files (per the spec in lance-format#7381), the foundation the scanner/take/index/compaction work builds on. - `DataOverlayFile` / `OverlayCoverage` (dense `shared_offset_bitmap` and sparse per-field) with protobuf round-trip, attached to `Fragment.overlays`. - Reader feature flag 64 (`FLAG_DATA_OVERLAY_FILES`): set whenever any fragment carries overlays, so a reader that does not understand them refuses the dataset instead of returning stale base values. - `Operation::DataOverlay` transaction op: appends overlays to a fragment's list (preserving concurrently-written overlays) and stamps each overlay's `committed_version` to the new dataset version at commit time (re-stamped on retry). Conflict rules mirror DataReplacement — permissive against appends, deletes, column rewrites, index builds, and other overlays; conflicts only with row-rewriting compaction of the same fragment. Scan-side merge, take, and end-to-end write+read tests follow in the same PR branch. Part of the Data Overlay Files feature (OSS-1322). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Now that overlays can be committed, a scan or take over a fragment that has overlays would silently return stale base values, since the read-path merge is not implemented yet. Refuse such reads at `FileFragment::open` with a clear error instead of serving incorrect data. Lifted once the scan/take merge lands (rest of OSS-1322 / OSS-1324). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds `dataset::overlay`, the tested heart of reading overlays: given a base column for a physical row range and the overlays covering a field (newest first), `resolve_overlay_column` produces the merged column. An offset is resolved to the newest covering overlay's value at the offset's rank in the coverage bitmap; an uncovered offset falls through to the base; a covered offset whose value is NULL overrides the cell to NULL. `overlay_indices_newest_first` orders a fragment's overlays by `committed_version` then list position. Deletion precedence needs no handling here: the merge runs before the deletion filter, so an overlay value for a deleted offset is computed and dropped with the row. Wiring this into the scan stream and `take` follows on this branch. Unit tests cover rank addressing, multi-overlay precedence, NULL override vs. fall-through, physical-offset base, string columns, and ordering. Part of OSS-1322. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The v2 file writer advanced every column from a single global row counter, so a single file could only hold columns of equal length. Sparse data overlay files need columns whose item counts differ within one file (each field covers a different set of rows). Add `FileWriter::write_columns`, which writes a set of `(field, array)` pairs and advances each field's row counter independently, leaving other fields untouched. A field never written ends up as a zero-length column. `write_batch` is unchanged: it still advances all fields together, so ordinary rectangular files round-trip exactly as before. Per-column lengths were already derivable from page metadata; expose them via `FileReader::column_num_rows`. The reader already schedules each column from its own pages, so reading a column at its own length and random access within it work without further changes. Part of the Data Overlay Files feature (OSS-1323). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire the overlay cell-resolution core into reads so `take` (and scan) return merged values. `FileFragment::open` loads each projected field's overlay value columns newest-first, and `new_read_impl`/`read_ranges` merge them into base batches by physical offset before deletion filtering — so resolution is identical for take and scan, NULL overrides apply, deletions take precedence (an overlay value for a deleted row is dropped with the row), and fields resolve independently. The merge addresses each row by its physical offset (via `ReadBatchParams::to_offsets_total`), so the resolution core now takes explicit per-row offsets instead of a contiguous start — a single code path for the contiguous scan range and arbitrary take indices. Sparse per-field overlays read each field's value column independently, so unequal-length columns (OSS-1323) are handled without materializing a rectangular batch. Removes the temporary "overlays not supported" guard from OSS-1322. Tests: take of covered/uncovered offsets, multiple overlays (newest-wins), per-field coverage with unequal-length columns, NULL override, overlay on a deleted row (inert), and multi-fragment scan — all over v2.0 and v2.1 files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add overlay-aware compaction (the "Compaction" section of the Data Overlay Files spec). A scheduler reads each fragment's overlay state — overlay count, covered cells, and the version gap between an overlay's committed_version and the base and each covering index's dataset_version — and picks a mode from the staleness signal: - Overlay -> overlay merge: collapse a fragment's overlays into one smaller overlay carrying the per-(offset, field) post-image, stamped with the maximum input committed_version. Base and indexes are untouched. - Overlay -> base fold: materialize the post-image of every covered cell into a fresh base data file, tombstone the folded fields in the old files, and clear the fragment's overlays. Row addresses are preserved (a column rewrite). The same commit drops the folded fragment from any index on a folded field, so an index is never left serving stale values. Both modes read each base/overlay cell at most once: overlay value columns are read only at the ranks that win (a cell a newer overlay supplies is never fetched from an older overlay), via FileFragment::read_overlay_field_winners. Base columns are read whole, the cheaper coalesced read the spec permits. The rewrite is committed via the existing Operation::Update (RewriteColumns) path, so no new persisted operation or conflict-matrix change is needed. Tests cover scheduler mode selection (field-aware, version-gap driven), fold materialization + index reconciliation + NULL override + multi-fragment, merge collapse preserving the max committed_version, and an assert_io check that overlapping coverage reads strictly fewer value bytes than a disjoint control. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
|
Important This PR touches the Lance format specification. Substantive changes to the format specification — the If this is a meaningful format change:
|
github-actions
Bot
added
enhancement
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements OSS-1326. Overlay-aware compaction: a scheduler that reads per-fragment overlay state and decides when and in which mode to compact, plus the rewrite executor for both modes.
What it does
Awareness & scheduling (
OverlayFragmentState,plan_overlay_compaction)committed_versionand the base (current dataset version) and each covering index'sdataset_version.choose_mode): fold when an index on an overlaid field is stale, else merge once overlays accumulate, else skip. Field-aware — an overlay on an unrelated column never makes an index look stale.Execution (
compact_overlays)(offset, field)post-image, stamped with the maximum inputcommitted_version. Base and indexes are untouched.overlays. Row addresses are preserved (a column rewrite, not a row rewrite). The same commit drops the folded fragment from any index on a folded field, so an index is never left serving stale values.Read efficiency (the core optimization)
FileFragment::read_overlay_field_winnersusing ranktakes. Base columns are read whole — the cheaper coalesced read the spec permits.The rewrite commits through the existing
Operation::Update(RewriteColumns) path, so there's no new persisted operation or conflict-matrix change. Index reconciliation uses the drop-from-coverage option (the cheaper of the two the spec allows); rebuild-in-place is a possible follow-up.Tests
committed_version; union coverage; read still correct.assert_iocheck that overlapping coverage reads strictly fewer value bytes than a disjoint negative control of the same per-overlay size.🤖 Generated with Claude Code