Repository navigation
[core] Keep files referenced by other branches during snapshot expiration - #10897
Open
henriquejsza wants to merge 1 commit into
Open
henriquejsza wants to merge 1 commit into
henriquejsza wants to merge 1 commit into
Conversation
…tion All branches of a table share the same physical data, manifest, index and statistics files; only snapshot, tag and schema metadata are branch scoped. Snapshot expiration on a branch built its skipping set only from that branch's own snapshots and tags, so after a branch created from a tag drops its copy of the tag, expiring it deletes files that main still references and main can no longer be read. Before deleting files, collect the data files and manifest/index/statistics files referenced by every other branch's live snapshots and tags, restricted to the current expiration batch, and skip any file still referenced. A failure to read another branch keeps the batch rather than risking deletion of in-use files. The common single-branch path is unchanged. Signed-off-by: henriquejsza <henriquejsza@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Closes #10883.
All branches of a table share the same physical data, manifest, index and
statistics files; only snapshot, tag and schema metadata are branch scoped
(
FileStorePathFactoryroots data and manifest paths at the table root, whileSnapshotManager/TagManagerprepend the branch path). Snapshot expiration ona branch built its skipping set only from that branch's own snapshots and tags
(
ExpireSnapshotsImpl#collectTaggedSnapshotsandmanifestSkippingSet), sowhen a branch created from a tag drops its own copy of that tag and then
expires, it deletes files that the main branch still references. After that,
reading main fails with
FileNotFoundExceptionon a manifest list or datafile, exactly as reported in the issue.
This change makes expiration aware of the other branches. Before deleting
files,
AbstractFileStoreTable#newExpireSnapshotscollects the data files andthe manifest/manifest-list/index/statistics files referenced by every other
branch's live snapshots and tags, and
ExpireSnapshotsImplskips any file thatis still referenced. The data-file lookup is restricted to the current
expiration batch so memory stays proportional to the batch, mirroring the
existing per-tag skipper. Reading another branch is best effort: if it fails,
the batch is kept rather than risking deletion of in-use files. When the table
has no other branch, the collector is
nulland the common path is unchanged.Tests
AppendOnlySimpleTableTest#testExpireOnBranchKeepsFilesReferencedByMain,which reproduces the issue: tag main's snapshot, create a branch from the tag,
write on the branch, delete the branch's copy of the tag, expire the branch to
one snapshot, then assert main is still fully readable. It fails on master
(
FileNotFoundExceptionon main's manifest list) and passes with this change.mvn -pl paimon-core -Dtest='AppendOnlySimpleTableTest,ExpireSnapshotsTest,FileDeletionTest' testpasses.
mvn -pl paimon-core spotless:check checkstyle:checkpasses.