Skip to content

fix: collect unreferenced data newer than retained manifests - #8944

Open
lance-gatefixer[bot] wants to merge 1 commit into
mainfrom
gatekeeper/fix-8942-1
Open

fix: collect unreferenced data newer than retained manifests#8944
lance-gatefixer[bot] wants to merge 1 commit into
mainfrom
gatekeeper/fix-8942-1

Conversation

@lance-gatefixer

Copy link
Copy Markdown
Contributor

Summary

  • List data files through the moving unverified-file age threshold during normal cleanup.
  • Scan the complete managed data subtree when delete_unverified is enabled.
  • Extend the failed-commit regression test for aged and explicitly unverified orphan files newer than every retained manifest.

Root cause

cleanup_old_versions capped the data subtree listing at the earliest retained manifest timestamp. Abandoned writes are not owned by any manifest and can be newer than that timestamp, so they never reached the existing age and delete_unverified checks.

Validation

  • cargo test -p lance dataset::cleanup::tests (45 passed)
  • cargo fmt --all
  • cargo clippy --all --tests --benches -- -D warnings
  • git diff --check

Fixes #8942

@github-actions github-actions Bot added the bug Something isn't working label Sep 2, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The data listing now covers the union of manifest-proven stale files and files past the moving unverified threshold, while retained-manifest references and the existing recent-write guard continue to protect live data. This addresses the leak at the candidate-visibility boundary without changing the cleanup API or format.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cleanup_old_versions never lists unreferenced data files newer than the oldest retained manifest

0 participants