Skip to content

feat: overlay-aware compaction scheduler and rewrite#7567

Closed
wjones127 wants to merge 13 commits into
will/oss-1324-take-can-read-overlaysfrom
will/oss-1326-compaction-scheduler-thats-aware-of-overlays
Closed

feat: overlay-aware compaction scheduler and rewrite#7567
wjones127 wants to merge 13 commits into
will/oss-1324-take-can-read-overlaysfrom
will/oss-1326-compaction-scheduler-thats-aware-of-overlays

Conversation

@wjones127

@wjones127 wjones127 commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Implements OSS-1326. Overlay-aware compaction: a scheduler that reads per-fragment overlay state and decides when and in which mode to compact, plus the rewrite executor for both modes.

Stacked on #7536 (OSS-1324). Branches off the 1324 branch — the compaction executor must read overlay cell values, a capability that lands in 1324. This PR is now a proper stack, so its diff is exactly this change (three files); review against the 1324 base branch.

What it does

Awareness & scheduling (OverlayFragmentState, plan_overlay_compaction)

  • Reads each overlaid fragment's state: overlay count, covered offsets/cells, and the version gap between an overlay's committed_version and the base (current dataset version) and each covering index's dataset_version.
  • Picks a mode from the version-gap staleness signal (choose_mode): fold when an index on an overlaid field is stale, else merge once overlays accumulate, else skip. Field-aware — an overlay on an unrelated column never makes an index look stale.

Execution (compact_overlays)

  • Overlay → overlay merge: walks the overlays newest-first and writes a single smaller overlay carrying the per-(offset, field) post-image, stamped with the maximum input committed_version. Base and indexes are untouched.
  • Overlay → base fold: materializes the post-image of every covered cell into a fresh base data file, tombstones the folded fields in the old files, and clears the fragment's overlays. Row addresses are preserved (a column rewrite, not a row rewrite). The same commit drops the folded fragment from any index on a folded field, so an index is never left serving stale values.

Read efficiency (the core optimization)

  • Each base/overlay cell is read at most once: overlay value columns are read only at the ranks that win (a cell a newer overlay supplies is never fetched from an older overlay), via FileFragment::read_overlay_field_winners using rank takes. Base columns are read whole — the cheaper coalesced read the spec permits.

The rewrite commits through the existing Operation::Update (RewriteColumns) path, so there's no new persisted operation or conflict-matrix change. Index reconciliation uses the drop-from-coverage option (the cheaper of the two the spec allows); rebuild-in-place is a possible follow-up.

Tests

  • Scheduler mode selection: version-gap computation, field-awareness, fold-on-stale-index, merge-on-count, skip.
  • Fold: materialization + overlays cleared; index reconciliation (folded fragment removed from the stale index's coverage, query still correct); NULL override; multi-fragment / multi-overlay (newest wins).
  • Merge: collapse to one overlay preserving the max committed_version; union coverage; read still correct.
  • Read-each-cell-once: assert_io check that overlapping coverage reads strictly fewer value bytes than a disjoint negative control of the same per-overlay size.

🤖 Generated with Claude Code

wjones127 and others added 13 commits June 22, 2026 18:07
@wjones127 @claude
docs: specify data overlay files for the table format
fbd230a 
Add a specification for data overlay files: small files attached to a
fragment that supply new values for a subset of (row offset, field) cells
without rewriting the base data files, for cheap cell-level updates.

- protos/table.proto: rework DataOverlayFile with a dense/sparse coverage
  oneof (shared_offset_bitmap vs new FieldCoverage), rename read_version to
  committed_version (effective, commit-stamped), and document rank-based
  addressing with no offset column. Document reader feature flag 64.
- docs: add data_overlay_file.md (full spec, worked example, guidance stub)
  and link it from the table format overview.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wjones127 @claude
feat: add DataOverlay transaction operation
6aaca5e 
Add the `DataOverlay` operation (and `DataOverlayGroup`) to attach overlay
files to fragments without rewriting their base data. Mirrors the
`DataReplacement` batch shape, appends to each fragment's `overlays` list, and
documents permissive conflict semantics: concurrent overlays, appends, deletes,
and column rewrites are compatible; row-rewrites, compaction, and overlay->base
folds conflict.

committed_version is left 0 by the writer and stamped at commit time.

Proto only — Rust/Python bindings deferred.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wjones127 @claude
chore: compile against DataOverlay proto, reject unsupported overlays
a88fbe2 
The table/transaction proto changes generate new fields and an Operation
variant. This wires the minimum needed to compile without implementing overlay
support:

- Emit empty `overlays` when converting fragments to proto.
- Reject the `DataOverlay` transaction operation with NotSupported on read.

Datasets that use overlays set reader feature flag 64, which already falls in
the unknown-flag range rejected by `can_read_dataset`, so the library refuses
them at the feature-flag layer.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wjones127 @claude
feat(file): v2 writer/reader support columns of unequal length
7243137 
The v2 file writer advanced every column from a single global row counter,
so a single file could only hold columns of equal length. Sparse data
overlay files need columns whose item counts differ within one file (each
field covers a different set of rows).

Add `FileWriter::write_columns`, which writes a set of `(field, array)`
pairs and advances each field's row counter independently, leaving other
fields untouched. A field never written ends up as a zero-length column.
`write_batch` is unchanged: it still advances all fields together, so
ordinary rectangular files round-trip exactly as before.

Per-column lengths were already derivable from page metadata; expose them
via `FileReader::column_num_rows`. The reader already schedules each column
from its own pages, so reading a column at its own length and random access
within it work without further changes.

Part of the Data Overlay Files feature (OSS-1323).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wjones127
refactor(file): write_column singular API + reader length validation
1dba164 
Address review on the unequal-length-columns work:

- Rename `FileWriter::write_columns(Vec<(usize, ArrayRef)>)` to the simpler
  `write_column(column_index, array)`; callers loop instead of batching.
- Drop doc comments from internal-only helpers.
- Reader: validate that a projection's top-level columns share a length and
  reject mismatches up front with a descriptive error, instead of panicking in
  the decoder. `RangeFull`/`RangeFrom` now resolve to the projected common
  length rather than the file's longest column, so a single short column reads
  back at its own length. Rectangular files are unchanged.

Adds `test_read_unequal_length_projection` (V2_0 + V2_1).
@wjones127 @claude
feat: data overlay file model, feature flag, and DataOverlay transaction
34a19af 
Adds the in-memory + commit machinery for data overlay files (per the spec in
#7381), the foundation the scanner/take/index/compaction work builds on.

- `DataOverlayFile` / `OverlayCoverage` (dense `shared_offset_bitmap` and sparse
  per-field) with protobuf round-trip, attached to `Fragment.overlays`.
- Reader feature flag 64 (`FLAG_DATA_OVERLAY_FILES`): set whenever any fragment
  carries overlays, so a reader that does not understand them refuses the
  dataset instead of returning stale base values.
- `Operation::DataOverlay` transaction op: appends overlays to a fragment's
  list (preserving concurrently-written overlays) and stamps each overlay's
  `committed_version` to the new dataset version at commit time (re-stamped on
  retry). Conflict rules mirror DataReplacement — permissive against appends,
  deletes, column rewrites, index builds, and other overlays; conflicts only
  with row-rewriting compaction of the same fragment.

Scan-side merge, take, and end-to-end write+read tests follow in the same PR
branch.

Part of the Data Overlay Files feature (OSS-1322).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wjones127 @claude
feat: refuse reads of fragments with overlays until merge lands
107587e 
Now that overlays can be committed, a scan or take over a fragment that has
overlays would silently return stale base values, since the read-path merge is
not implemented yet. Refuse such reads at `FileFragment::open` with a clear
error instead of serving incorrect data. Lifted once the scan/take merge lands
(rest of OSS-1322 / OSS-1324).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wjones127 @claude
refactor(table): parse overlay coverage on load and gate the overlay …
a1157a9 
…flag

OverlayCoverage now holds parsed Arc<RoaringBitmap>s, deserialized once when a
fragment is loaded and shared cheaply on clone, instead of re-deserializing the
serialized bytes on every coverage_for_field call.

Treat the data overlay feature flag (64) as unknown in release builds unless
LANCE_ENABLE_DATA_OVERLAY_FILES is set, so the unreleased feature is neither
emitted by release writers nor accepted by release readers; debug builds
understand it so tests exercise the path. Stable-sort a fragment's overlays by
committed_version (newest last) on load so resolution can assume the ordering.

Adds tests for the missing-coverage/data_file errors, the release gating, and
the load-time sort.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wjones127 @claude
fix(table): correct DataOverlay commit/conflict edge cases
e2f98f5 
Three correctness fixes in the data-overlay write/commit path, each with tests
(this path had ~0% coverage):

- build_manifest dropped overlays when two DataOverlayGroups targeted the same
  fragment (HashMap collect kept only the last). Merge groups by fragment_id so
  all overlays are appended in order.
- check_data_overlay_txn did not conflict when a concurrent Update/Delete
  removed an overlaid fragment, leaving the overlay orphaned and hard-erroring
  on retry. Mirror check_data_replacement_txn: retryable conflict when
  removed/deleted_fragment_ids intersect our fragments; fragments merely updated
  in place stay compatible.
- impl PartialEq for Operation had the DataOverlay arm backwards: it reported
  DataOverlay == Rewrite as true and DataOverlay == DataOverlay as false
  (conflict semantics copied into the equality impl). Compare groups for the
  same variant; not equal to other variants.

Tests: build_manifest append/stamp, duplicate-group merge, unknown-fragment
error; the conflict matrix incl. remove-fragment conflict; Operation equality;
OverlayCoverage serde JSON round-trip; coverage_for_field out-of-bounds;
apply_feature_flags overlay arm.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wjones127 @claude
feat: resolve data overlay files on the take and scan read path
8052f87 
Wire overlay resolution into FragmentReader so `take` and scan reads merge
data overlay cells over the base data, removing the temporary guard that
refused reads of fragments carrying overlays.

Reads are lazy and rank-pushed: each overlay file is opened once (no value
bytes read up front), requested offsets are routed to the newest covering
overlay at their coverage rank, and only the touched ranks are fetched via
the existing `take` primitive. Base and overlay IO run concurrently, then
the column is assembled with `interleave`. This makes overlay reads
O(requested rows) rather than O(coverage).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wjones127 @claude
test: cover data overlay read-path edge cases
0a984a0 
Adds tests for combinations the initial read-path suite missed: row-id
pass-through alongside an overlay, an older overlay routed zero ranks
(empty-ranks fetch branch), a newest NULL value shadowing an older
non-null, per-field newest-wins across two sparse overlays, a plan
present but all requested offsets uncovered (all-fall-through), and a
dataset-level take spanning multiple overlaid fragments.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wjones127 @claude
perf: route overlays bitmap-major on contiguous scans
3ba338b 
`route_overlays` was offset-major: for every physical offset it probed
each coverage bitmap newest-first. On a full scan that is
`O(N_rows * K_overlays)` roaring `contains` probes — ~16M for a 1M-row,
16-overlay scan — of which 99% only confirm a fall-through to base. It
dominated scan CPU (~87% of the scan thread) and made scan cost grow
linearly with overlay count, badly so for scattered (stride) coverage.

A scan reads a contiguous physical range, so offset `o` is output row
`o - base`. The fast path now intersects each coverage with the batch's
offset range (a container-level op that drops a non-overlapping batch in
`O(containers)`) and routes only the in-range bits directly to rows.
Those bits carry consecutive coverage ranks starting at the count of bits
below `base`, so ranks need a single `rank` lookup rather than a probe per
cell. `take` still supplies arbitrary offsets and keeps the general path.

Microbenchmark of `route_overlays` over a 1M-row scan (ms/scan):

    overlays   stride old -> new      contiguous old -> new
    1          13.3  -> 0.67          2.8  -> 0.55
    4          46.3  -> 0.86          3.5  -> 0.66
    16        178.5  -> 1.48          9.4  -> 0.96
    64        829.0  -> 4.07         33.3  -> 2.09

A fuzz test asserts the fast path produces byte-identical routing to the
general path across random bases, lengths, overlay counts, and densities.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wjones127 @claude
feat: overlay-aware compaction scheduler and rewrite
a2e997f 
Add overlay-aware compaction (the "Compaction" section of the Data Overlay
Files spec). A scheduler reads each fragment's overlay state — overlay count,
covered cells, and the version gap between an overlay's committed_version and
the base and each covering index's dataset_version — and picks a mode from the
staleness signal:

- Overlay -> overlay merge: collapse a fragment's overlays into one smaller
  overlay carrying the per-(offset, field) post-image, stamped with the maximum
  input committed_version. Base and indexes are untouched.
- Overlay -> base fold: materialize the post-image of every covered cell into a
  fresh base data file, tombstone the folded fields in the old files, and clear
  the fragment's overlays. Row addresses are preserved (a column rewrite). The
  same commit drops the folded fragment from any index on a folded field, so an
  index is never left serving stale values.

Both modes read each base/overlay cell at most once: overlay value columns are
read only at the ranks that win (a cell a newer overlay supplies is never
fetched from an older overlay), via FileFragment::read_overlay_field_winners.
Base columns are read whole, the cheaper coalesced read the spec permits. The
rewrite is committed via the existing Operation::Update (RewriteColumns) path,
so no new persisted operation or conflict-matrix change is needed.

Tests cover scheduler mode selection (field-aware, version-gap driven), fold
materialization + index reconciliation + NULL override + multi-fragment, merge
collapse preserving the max committed_version, and an assert_io check that
overlapping coverage reads strictly fewer value bytes than a disjoint control.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added the enhancement New feature or request label Jul 1, 2026
@wjones127 wjones127 mentioned this pull request Jul 1, 2026
Closed
@wjones127
wjones127 force-pushed the will/oss-1324-take-can-read-overlays branch 5 times, most recently from 4bb5e39 to 5ff87ce Compare July 13, 2026 19:54
@wjones127 wjones127 closed this Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@wjones127