HDDS-8714. Fix flaky TestScmHAFinalization#testFinalizationWithRestart#10790
HDDS-8714. Fix flaky TestScmHAFinalization#testFinalizationWithRestart#10790jojochuang wants to merge 3 commits into
Conversation
Set hdds.scm.safemode.rule.refresh.interval=0s in the test mini-cluster so follower SCMs re-evaluate safe-mode rules synchronously after Ratis catch-up, instead of waiting for the default 5s background poll. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I2283d7ddbe4b78797f89107c54db4420819c6e1b
Add fast heartbeat, pipeline, and transaction buffer flush configs and wait for leader finalization before polling the client after SCM restart. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I37b16aa8492f1b1f9ffb7fe28ff915759d5b501b
There was a problem hiding this comment.
Pull request overview
Addresses flakiness in the SCM HA upgrade finalization integration test (TestScmHAFinalization#testFinalizationWithRestart) by making the mini-cluster progress faster and by explicitly waiting for the SCM leader to reach FINALIZATION_COMPLETE after restart before validating finalization status via the client.
Changes:
- Tune mini-cluster timing/interval configuration in
init()(heartbeat, reports, pipeline creation, DB transaction buffer flush, safemode refresh) to reduce timeouts and improve determinism. - After restarting all SCMs, wait for the elected leader SCM to reach
FINALIZATION_COMPLETEbefore polling the client for finalization completion. - Remove the
@Flaky("HDDS-8714")annotation from the restart test.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Validate config-only fix without waitForScmToFinalize before client status. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: Ic14b2d278fd9256e01cc31560ac5d1a36d1aa4c9
|
Fix makes sense at a high level, but according to this Jira comment 100 runs may not be enough to verify the problem is actually fixed. |
|
Good to know. It was failing 11 out of 100 times for me before https://github.com/jojochuang/ozone/actions/runs/29468880808 Let me trigger more test runs. |
|
Triggered extended flaky-test-check runs (20×20 = 400 iterations each, 800 total):
Previous validation was 10×10 (100 iterations): https://github.com/jojochuang/ozone/actions/runs/29605746378 |
Summary
TestScmHAFinalization#testFinalizationWithRestart[2](AFTER_POST_FINALIZE_UPGRADE), which timed out inwaitForFinalizationFromClient()after restarting all SCMs.init()with fast heartbeat, pipeline creation, transaction buffer flush, and report intervals (matching other upgrade finalization tests), plushdds.scm.safemode.rule.refresh.interval=0sandhdds.scm.wait.time.after.safe.mode.exit=0s.@Flaky("HDDS-8714").At
[2], finalization is nearly complete before the triple SCM restart. Slow test defaults (especially the 60s HA transaction buffer flush interval and 120s pipeline creation interval) let the finalizing mark stay in memory across restart and forced a slow resume path that could exceed the test's 60s client poll timeout.Jira
https://issues.apache.org/jira/browse/HDDS-8714
Test plan
./hadoop-ozone/dev-support/checks/checkstyle.shGenerated-by: Cursor (Composer)