HDDS-15828. Fix flaky TestDirectoryDeletingServiceWithFSO snapshot cleanup. - #10726
Open
jojochuang wants to merge 2 commits into
Open
HDDS-15828. Fix flaky TestDirectoryDeletingServiceWithFSO snapshot cleanup.#10726jojochuang wants to merge 2 commits into
jojochuang wants to merge 2 commits into
Conversation
…eanup. Wait for SnapshotDeletingService to purge deleted snapshots before snapshot tests return on the shared mini-cluster, so later tests do not chain off stale global snapshot entries. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I57ec596cd5870b68f21ff9d9a89850ea9832f56e
Contributor
There was a problem hiding this comment.
Pull request overview
This PR targets integration-test flakiness in TestDirectoryDeletingServiceWithFSO by ensuring snapshot deletions are fully purged from OM metadata before tests return, avoiding cross-test interference on the shared mini-cluster.
Changes:
- Replace SnapshotDeletingService shutdown/restart logic with suspend/resume and a dedicated purge-wait helper.
- Add a
waitForSnapshotsPurged(...)helper that drives SnapshotDeletingService and waits forsnapshotInfoTableto reach an expected row count. - Ensure snapshot purge is awaited after snapshot deletion in another test to avoid leaving global snapshot entries behind.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+574
to
577
| cluster.getOzoneManager().getKeyManager().getSnapshotDeletingService().suspend(); | ||
| DirectoryDeletingService dirDeletingService = cluster.getOzoneManager().getKeyManager().getDirDeletingService(); | ||
| // Suspend KeyDeletingService | ||
| dirDeletingService.suspend(); |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Comment on lines
+820
to
+832
| snapshotDeletingService.resume(); | ||
| GenericTestUtils.waitFor(() -> { | ||
| try { | ||
| snapshotDeletingService.runPeriodicalTaskNow(); | ||
| cluster.getOzoneManager().awaitDoubleBufferFlush(); | ||
| return cluster.getOzoneManager().getMetadataManager() | ||
| .countRowsInTable(snapshotInfoTable) == expectedCount; | ||
| } catch (Exception e) { | ||
| throw new RuntimeException("Failed to run SnapshotDeletingService purge task", e); | ||
| } | ||
| }, 1000, 120000); | ||
| assertTableRowCount(snapshotInfoTable, expectedCount); | ||
| } |
Comment on lines
+574
to
577
| cluster.getOzoneManager().getKeyManager().getSnapshotDeletingService().suspend(); | ||
| DirectoryDeletingService dirDeletingService = cluster.getOzoneManager().getKeyManager().getDirDeletingService(); | ||
| // Suspend KeyDeletingService | ||
| dirDeletingService.suspend(); |
Comment on lines
+662
to
663
| waitForSnapshotsPurged(snapshotInfoTable, initialSnapshotCount); | ||
| dirDeletingService.resume(); |
Contributor
There was a problem hiding this comment.
Thanks for the patch @jojochuang . It might be better to move waitForSnapshotsPurged and dirDeletingService.resume() if suspended to the test teardown method cleanup
| // Manual cleanup deletedDirTable for next tests | ||
| client.getObjectStore().deleteSnapshot(volumeName, bucketName, "snap1"); | ||
| cluster.getOzoneManager().awaitDoubleBufferFlush(); | ||
| waitForSnapshotsPurged(snapshotInfoTable, 0); |
Contributor
There was a problem hiding this comment.
Same comment as above. It's better to move waitForSnapshotsPurged and cleanupTables() if suspended to the test teardown method cleanup.
|
|
||
| // Manual cleanup deletedDirTable for next tests | ||
| client.getObjectStore().deleteSnapshot(volumeName, bucketName, "snap1"); | ||
| cluster.getOzoneManager().awaitDoubleBufferFlush(); |
Contributor
There was a problem hiding this comment.
Suggested change
| cluster.getOzoneManager().awaitDoubleBufferFlush(); |
|
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days. |
|
Thank you for your contribution. This PR is being closed due to inactivity. Please contact a maintainer if you would like to reopen it. |
Contributor
|
Hi @SaketaChalamchala @jojochuang I've addressed all the review comments in a follow-up PR: #11212 Changes made on top of this patch:
Flaky Test run after the fix : https://github.com/arunsarin85/ozone/actions/runs/33992835360 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Previously failed in https://github.com/apache/ozone/actions/runs/29127546564/job/86477855080
Wait for SnapshotDeletingService to purge deleted snapshots before snapshot tests return on the shared mini-cluster, so later tests do not chain off stale global snapshot entries.
What is the link to the Apache JIRA
HDDS-15828. Fix flaky TestDirectoryDeletingServiceWithFSO snapshot cleanup.
How was this patch tested?
flaky-test-check 10x10 all passed: https://github.com/jojochuang/ozone/actions/runs/29134805647