Repository navigation
fix: prevent CleanOrphanUploadFiles from skipping records during offset pagination - #1642
Asthenia0412 wants to merge 1 commit into
Conversation
…et pagination CleanOrphanUploadFiles queried file records with offset-based pagination (page=1,2,3,...) while simultaneously deleting orphan records that dropped out of the Status=Available filter. After a batch deletes N records, the remaining result set shrinks and records that were at positions page*pageSize and above slide into earlier slots. Advancing the page counter then skips past those shifted rows, leaving eligible orphans unprocessed for a later cleanup run. Track deletions in the current batch. When any deletions occur the result set has changed, so reset page to 1 and re-scan the front of the result. When no deletions occur the Available set is stable and incrementing the page counter is safe. Closes: apache#1631 Signed-off-by: yancy <yancy@users.noreply.github.com>
|
@LinkinStars @kumfo @sy-records 各位大佬好,我是 yancy (Asthenia0412),来贡献一个修复 PR。 问题issue #1631 — 修复在每批处理中跟踪
验证
麻烦帮忙 review 一下,感谢 🙏 |
|
Thanks for the contribution. Resetting to page 1 after a deletion and advancing normally when a page is unchanged avoids the early-exit problem that affects PR #1634, and it addresses the original offset-shift bug. That said, this approach can repeatedly rescan large prefixes of retained records whenever later pages contain deletions. It also does not include a regression test in the patch, which makes the pagination edge cases harder to protect going forward. PR #1635 resolves the underlying issue more directly with an ordered ID cursor ( |
Summary
Fixes issue #1631
CleanOrphanUploadFilesqueries file records with offset-based pagination (page=1,2,3,...) while simultaneously deleting orphan records that drop out of theStatus = Availablefilter. After a batch deletes N records, the remaining result set shrinks and records at positionspage*pageSizeand above slide into earlier slots. Advancing the page counter then skips past those shifted rows, leaving eligible orphans unprocessed until the next cleanup run.Changes
deletedInBatchcounter during each page iterationpageto1so the next query rescans the front of the current Available setpagenormallyVerification
go build ./...— full project builds cleanlygo test ./internal/service/file_record/...— package has no test filesAuthor
Signed-off-by: yancy yancy@users.noreply.github.com
Co-authored by: Asthenia0412
Closes: #1631