Skip to content

[core] Keep files referenced by other branches during snapshot expiration - #10897

Open
henriquejsza wants to merge 1 commit into
apache:masterfrom
henriquejsza:core-10883-branch-expire-shared-files
Open

henriquejsza wants to merge 1 commit into
apache:masterfrom
henriquejsza:core-10883-branch-expire-shared-files

Conversation

@henriquejsza

Copy link
Copy Markdown

Purpose

Closes #10883.

All branches of a table share the same physical data, manifest, index and
statistics files; only snapshot, tag and schema metadata are branch scoped
(FileStorePathFactory roots data and manifest paths at the table root, while
SnapshotManager/TagManager prepend the branch path). Snapshot expiration on
a branch built its skipping set only from that branch's own snapshots and tags
(ExpireSnapshotsImpl#collectTaggedSnapshots and manifestSkippingSet), so
when a branch created from a tag drops its own copy of that tag and then
expires, it deletes files that the main branch still references. After that,
reading main fails with FileNotFoundException on a manifest list or data
file, exactly as reported in the issue.

This change makes expiration aware of the other branches. Before deleting
files, AbstractFileStoreTable#newExpireSnapshots collects the data files and
the manifest/manifest-list/index/statistics files referenced by every other
branch's live snapshots and tags, and ExpireSnapshotsImpl skips any file that
is still referenced. The data-file lookup is restricted to the current
expiration batch so memory stays proportional to the batch, mirroring the
existing per-tag skipper. Reading another branch is best effort: if it fails,
the batch is kept rather than risking deletion of in-use files. When the table
has no other branch, the collector is null and the common path is unchanged.

Tests

  • Added AppendOnlySimpleTableTest#testExpireOnBranchKeepsFilesReferencedByMain,
    which reproduces the issue: tag main's snapshot, create a branch from the tag,
    write on the branch, delete the branch's copy of the tag, expire the branch to
    one snapshot, then assert main is still fully readable. It fails on master
    (FileNotFoundException on main's manifest list) and passes with this change.
  • mvn -pl paimon-core -Dtest='AppendOnlySimpleTableTest,ExpireSnapshotsTest,FileDeletionTest' test
    passes.
  • mvn -pl paimon-core spotless:check checkstyle:check passes.

…tion

All branches of a table share the same physical data, manifest, index and
statistics files; only snapshot, tag and schema metadata are branch scoped.
Snapshot expiration on a branch built its skipping set only from that branch's
own snapshots and tags, so after a branch created from a tag drops its copy of
the tag, expiring it deletes files that main still references and main can no
longer be read.

Before deleting files, collect the data files and manifest/index/statistics
files referenced by every other branch's live snapshots and tags, restricted to
the current expiration batch, and skip any file still referenced. A failure to
read another branch keeps the batch rather than risking deletion of in-use
files. The common single-branch path is unchanged.

Signed-off-by: henriquejsza <henriquejsza@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Deleting a branch's copy of a tag lets its expiration delete main's files

1 participant