feat(transaction): composite transaction foundation (UpdateBases + DataReplacement + Merge)#7834
Draft
wjones127 wants to merge 2 commits into
Conversation
…geSchema/RefreshRowVersionMetadata actions Extends the experimental Action enum with the actions needed to decompose the legacy UpdateBases, DataReplacement, and Merge operations: - AddBases: mint new base paths from the manifest's base-id counter. - ReplaceFragmentColumns (+ ColumnReplacement): swap or wholesale-replace a fragment's data files. - AddFields / ChangeSchema: append new fields, or replace the schema wholesale (for column recasts). - RefreshRowVersionMetadata: bump row-version metadata for stable-row-id datasets. - BaseRef: a tagged reference (Committed/Local) so a same-operation ReplaceFragmentColumns/AddFields can land files on a base minted earlier in the same operation, resolved at apply time. No apply/translation/conflict wiring yet; that follows in the next commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ataReplacement/Merge as composite actions Wires the new actions through the existing commit path as a feature-gated Operation::UserOperation variant (opaque wire payload, mirroring the Append+AddIndex slice's ExperimentalUserOperation hook): - apply_user_operation resolves AddBases up front (so same-operation BaseRef::Local references can be minted before the manifest exists), applies ReplaceFragmentColumns (with covering-index invalidation), AddFields, ChangeSchema (with dead-data-file pruning), and RefreshRowVersionMetadata. - actions_from_update_bases/actions_from_data_replacement/actions_from_merge translate the three legacy operations into their action-based equivalent; Merge diffs the prior manifest to recover a pure-column-add or recast delta instead of replacing the fragment list wholesale. - Conflict resolution derives a composite's footprint (added bases / replaced fragment-column pairs / merge-class) from its constituent actions and checks it against every other operation and against another composite, reproducing the legacy UpdateBases/DataReplacement/Merge rules per action rather than treating the whole operation as one opaque unit. Round-trip parity tests assert the composite path produces the same fragments/schema/indices as the legacy build_manifest for each operation (including all-NULL backfill, multi-fragment, and stable-row-id cases), and conflict tests cover composite-vs-legacy and composite-vs-composite. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
|
Important This PR touches the Lance format specification. Substantive changes to the format specification — the If this is a meaningful format change:
|
github-actions
Bot
added
A-format
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Vertical slice for OSS-1490 (Composite Transactions milestone), sibling to
#7674 (Append + AddIndex) rather than stacked on it: this one decomposes
UpdateBases,DataReplacement, andMergeinto action-based transactions,stacked on the bare skeleton (#7671).
AddBases,ReplaceFragmentColumns(+ColumnReplacement),AddFields,ChangeSchema,RefreshRowVersionMetadata, plus aBaseReftagged reference for same-operation base late-binding.
apply_user_operationapplies them to the in-progress fragment/indexlists, including covering-index invalidation and dead-data-file pruning.
Transaction::actions_from_update_bases/_data_replacement/_mergetranslate the three legacy operations into their action-based equivalent
(Merge diffs the prior manifest to recover a pure-column-add or recast
delta).
actions and reproduces the legacy per-operation rules against every other
operation kind and against another composite.
Everything is behind the non-default
unstable-action-transactionsCargofeature, as with the rest of this milestone.
Test plan
apply, and assert the resulting manifest matches legacy
build_manifest(covers all-NULL backfill, multi-fragment, andstable-row-id refresh cases).
composite vs. composite.
cargo fmt --allandcargo clippy --all --tests --benches -- -D warnings, both with and withoutunstable-action-transactions.🤖 Generated with Claude Code