docs: specify data overlay files for the table format

[wjones127]

Adds a specification for data overlay files: small files attached to a fragment that supply new values for a subset of (row offset, field) cells without rewriting the base data files. They make cell-level updates cheap when only a small fraction of rows and/or columns change.

This PR is spec + proto only — no read/write implementation yet. It is also explicitly experimental. The released libraries will not produce tables with this feature enabled. Once the implementation is done in the library, we will vote on the final design before releasing. This is similar to how we have done file format updates.

Changes

• protos/table.proto
  • Rework DataOverlayFile: a oneof coverage { bytes shared_offset_bitmap | FieldCoverage field_coverage } to support both dense (rectangular) and sparse overlays; add the FieldCoverage message.
  • Rename read_version → committed_version (uint64), with effective/commit-stamped semantics so overlay-vs-index ordering is correct.
  • Drop the in-file offset key column in favor of rank-based addressing off the coverage bitmap.
  • Document reader feature flag 64 (and previously-undocumented 16/32).
• docs/src/format/table/data_overlay_file.md (new): full specification — coverage/resolution, deletion precedence, NULL-override, layout + rank addressing, dense vs. sparse, versioning, field-aware index exclusion with flat re-evaluation, the correctness invariant, both compaction modes, row lineage, a worked example (write → read → index query → sparse write → read → compaction), and a guidance stub with open questions.
• docs/src/format/table/index.md: concise overview + link to the new spec (replacing the earlier inline sketch).

Out of scope / follow-ups

• Write transaction shape (new Operation variant in transaction.proto + Rust).
• Writer support for unequal-length columns (needed for single-file sparse overlays).
• Coverage bitmap external spill for very large coverage.
• Per-fragment vs. per-table overlays / LSM analogy (open question in the doc).

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added documentation for experimental Data Overlay Files, including storage, versioning, querying, compaction, and transaction behavior.
  • Added format and transaction schema support for describing data overlays.
  • Added navigation links to the new documentation.

• Bug Fixes

  • Datasets using unsupported data overlays are now explicitly rejected instead of risking stale results.

• Documentation

  • Expanded guidance on invalidated index results and overlay-related filtering scenarios.

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.

[wjones127]

See discussion: #7401

[codecov]

Codecov Report

❌ Patch coverage is 85.71429% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/transaction.rs	82.35%	2 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[hamersaw]

IIUC this may not be enough to ensure accurate results in case of overlay updates -- if that is indeed the goal here. This covers the case where an query matching row was updated to no longer match, but what about one where the row previously didn't match and the updated value does? I think the only way to ensure 100% correctness is to brute force every update in the overlay.

[wjones127]

Yeah I see. We need to brute force a full exclusion set, not just the intersection between the candidate rows and the full exclusion set. Good point, I will update.

[wjones127]

I've prototyped the basic write, scan, and take paths.

• Write: feat: data overlay files — model, feature flag, and write/commit path #7535 (comment)
  • Overlay files have a significant impact on manifest size when > 5% of rows have overlays and they are sparse rather than contiguous.
  • Overlays are significantly faster and more IO efficient for targeted updates than update or merge insert.
• Scan & Take: feat: resolve data overlay files on the take (and scan) read path #7536 (comment)
  • Overlays have negligible impact on take performance
  • Overlays can have a large impact on scan performance. Having 16 overlays can make scanning 3.5x slower.

[westonpace]

Looks like a cool feature. I have some questions, some of which I think are probably blocking (but maybe I'm misunderstanding) but only because they need to be clarified. I don't see anything insurmountable.

[westonpace]

Does this mean a single commit can add multiple overlays that target the same field? Is there a reason for this? Or are we just being exhaustive?

[wjones127]

It's possible, but not encouraged. This is mostly just trying to be exhaustive in the spec.

[westonpace]

I'm not sure I believe that rank is O(1).

[westonpace]

This is already in data_overlay_file.md. Do we really want it here too?

[westonpace]

What operation is used to merge overlays (overlay->overlay compaction)? This operation or a different one?

[wjones127]

It can't be this one, as this one appends overlays and doesn't remove any. We'll create another operation for that.

[westonpace]

What if I have a column with an overlay file. This has been the stable state for some time. Later I want to replace the column with DataReplacement. Doesn't this conflict rule mean that the overlay will still sit on top? In other words, there is no way to overwrite an overlay with a DataReplacment until I've folded it in with a rewrite?

[wjones127]

I think this is talking what if I try to add a data overlay but a concurrent operation already made a data replacement or update. In those cases it is compatible.

But I think you are right the other direction needs care: if I"m trying to do a data replacement but another operation just did an overlay, I either need to drop/rewrite that overlay or raise a conflict error.

[westonpace]

I'm not actually worried about "just did an overlay". The overlay could be days or months old.

This isn't a "conflict error and retry the operation" failure but a more lasting "there is no way to paste over an overlay" problem.

If we had one overlay file per field (which I'm not recommending) we could say "as part of a DataReplacement commit you should remove any overlay files for that field" but I don't know how this works when there is an overlay file that covers multiple fields (how to say that overlay file is only valid for a subset of fields)

[wjones127]

I see. This is actually something I addressed in #7535

how to say that overlay file is only valid for a subset of fields

It's actually the same as when you need to invalidate a field in a datafile: you change the field id to -2 to mask it out. If all fields at -2 in the file, then you can drop the data file or overlay.

[westonpace]

Got it, and I see that in 7535. I think that's probably worth mentioning in this specification doc.

[westonpace]

Conflict semantics should probably be in a spec doc somewhere (transaction.md) and not here

[westonpace]

I don't have any further blocking questions

[wkalt]

I think we can state that merged overlays must be contiguous in the version record?

[wkalt]

nitpick: I think the single-cell case makes the choice of sparse vs dense a little arbitrary. We might be able to strengthen this example by updating more than one row/col.

[wkalt]

@wjones127 thanks for pushing this through! Very excited for this feature. Just two minor comments.

[coderabbitai]

📝 Walkthrough

Walkthrough

The change defines experimental data overlay schemas and specifications, documents transaction compatibility, index invalidation, resolution, and compaction rules, and rejects overlay transactions in the current Rust implementation while initializing overlay fields explicitly.

Changes

Data Overlay Support

Layer / File(s)	Summary
Overlay schema and format semantics protos/table.proto, docs/src/format/table/data_overlay_file.md, docs/src/format/table/index.md, docs/src/format/table/.pages	Adds overlay metadata, dense and sparse coverage definitions, precedence rules, feature-flag documentation, and navigation for the new specification.
Overlay transaction contracts protos/transaction.proto, docs/src/format/table/transaction.md	Adds overlay transaction messages and documents fragment attachment, concurrent overlay preservation, version stamping, and compatibility rules.
Query and compaction behavior docs/src/format/index/index.md, docs/src/format/table/data_overlay_file.md	Documents field-aware index exclusion, row re-evaluation, overlay resolution, lineage, compaction modes, and index coverage updates.
Unsupported runtime handling rust/lance/src/dataset/transaction.rs, rust/lance-table/src/format/fragment.rs, rust/lance-table/benches/manifest_intern.rs	Rejects DataOverlay transactions as unsupported, tests the rejection, and initializes serialized overlay lists as empty in existing paths.

Estimated code review effort: 3 (Moderate) | ~30 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: adding data overlay file support/specification for the table format.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

docs/src/format/table/data_overlay_file.md (1)

214-311: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy lift

Keep this format reference text-only and use an executable schema definition.

The worked example adds SQL/protobuf code blocks and a markdown schema table. Replace it with concise prose and express the shown schema as a pyarrow schema definition.

As per coding guidelines, format docs must be concise and text-only, must not include code examples, and must express file schemas as pyarrow schema definitions rather than markdown tables.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/src/format/table/data_overlay_file.md` around lines 214 - 311, Replace
the worked example in the overlay lifecycle section with concise prose only:
remove the SQL statements, protobuf-like DataOverlayFile blocks, and markdown
schema table. Define the shown table and overlay file schemas using executable
pyarrow schema definitions, while retaining the essential lifecycle, coverage,
versioning, read, and index-query behavior in text.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rust/lance/src/dataset/transaction.rs`:
- Around line 6200-6216: Update test_data_overlay_operation_rejected to capture
the returned error and assert it is Error::NotSupported with a message
containing either “data overlay” or the reader feature flag 64 identifier,
ensuring the test validates the specific rejection reason.

---

Nitpick comments:
In `@docs/src/format/table/data_overlay_file.md`:
- Around line 214-311: Replace the worked example in the overlay lifecycle
section with concise prose only: remove the SQL statements, protobuf-like
DataOverlayFile blocks, and markdown schema table. Define the shown table and
overlay file schemas using executable pyarrow schema definitions, while
retaining the essential lifecycle, coverage, versioning, read, and index-query
behavior in text.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: b5a0d66f-9dae-48e7-ab92-26bb5d071685

📥 Commits

Reviewing files that changed from the base of the PR and between bf4f32b and f079f39.

📒 Files selected for processing (10)

• docs/src/format/index/index.md
• docs/src/format/table/.pages
• docs/src/format/table/data_overlay_file.md
• docs/src/format/table/index.md
• docs/src/format/table/transaction.md
• protos/table.proto
• protos/transaction.proto
• rust/lance-table/benches/manifest_intern.rs
• rust/lance-table/src/format/fragment.rs
• rust/lance/src/dataset/transaction.rs

[coderabbitai]

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Assert the expected error message as well as the variant.

The test currently verifies only Error::NotSupported, so an unrelated NotSupported error could pass. Assert that the message identifies unsupported data overlays or reader feature flag 64.

As per coding guidelines: “Assert on both the error variant and the message content in tests; do not check only is_err().”

Proposed test assertion

-        let result = Transaction::try_from(message);
-        assert!(matches!(result, Err(Error::NotSupported { .. })));
+        let err = match Transaction::try_from(message) {
+            Err(err) => err,
+            Ok(_) => panic!("expected data overlay transaction to be rejected"),
+        };
+        assert!(matches!(&err, Error::NotSupported { .. }));
+        assert!(err
+            .to_string()
+            .contains("data overlay files are not supported"));

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	#[test]
	fn test_data_overlay_operation_rejected() {
	// Overlay files are not supported by this version of the library. A
	// transaction carrying the DataOverlay operation must be rejected rather
	// than silently ignored, mirroring the feature-flag-64 rejection.
	let message = pb::Transaction {
	read_version: 1,
	uuid: Uuid::new_v4().to_string(),
	operation: Some(pb::transaction::Operation::DataOverlay(
	pb::transaction::DataOverlay { groups: vec![] },
	)),
	..Default::default()
	};
	let result = Transaction::try_from(message);
	assert!(matches!(result, Err(Error::NotSupported { .. })));
	}
	#[test]
	fn test_data_overlay_operation_rejected() {
	// Overlay files are not supported by this version of the library. A
	// transaction carrying the DataOverlay operation must be rejected rather
	// than silently ignored, mirroring the feature-flag-64 rejection.
	let message = pb::Transaction {
	read_version: 1,
	uuid: Uuid::new_v4().to_string(),
	operation: Some(pb::transaction::Operation::DataOverlay(
	pb::transaction::DataOverlay { groups: vec![] },
	)),
	..Default::default()
	};
	let err = match Transaction::try_from(message) {
	Err(err) => err,
	Ok(_) => panic!("expected data overlay transaction to be rejected"),
	};
	assert!(matches!(&err, Error::NotSupported { .. }));
	assert!(err
	.to_string()
	.contains("data overlay files are not supported"));
	}

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rust/lance/src/dataset/transaction.rs` around lines 6200 - 6216, Update
test_data_overlay_operation_rejected to capture the returned error and assert it
is Error::NotSupported with a message containing either “data overlay” or the
reader feature flag 64 identifier, ensuring the test validates the specific
rejection reason.

Source: Coding guidelines