HDDS-13173. Clean up temporary checkpoint directories on snapshot defrag failure

[cchung100m]

What changes were proposed in this pull request?

Root Cause: When snapshot defragmentation fails due to transient errors, the temporary checkpoint directories are not cleaned up. Over 3+ days of continuous operations, these orphaned directories accumulate and exhaust disk space, causing OM startup failures.

Solution: Implement multi-layer exception handling with an explicit defragSuccessful flag.

1. Add success flag defragSuccessful: Initialize to false, set to true only after atomicSwitchSnampshotDB() succeeds.
2. Protect close(): Wrap in try-catch to prevent cleanup bypass
3. Protect old checkpoint deletion: Wrap in try-catch but continue with flag set
4. Add cleanup in finally block: Delete checkpoint directory if !defragSuccessful
5. Add metrics: Track cleanup failures via incNumSnapshotDefragFails()

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13173

How was this patch tested?

• Existing unit and integration tests validate the current change cause unexpected impacts
• Add new test case testCheckpointCleanupOnDefragFailure to validate the exception handling

[adoroszlai]

Thanks @cchung100m for working on this.

• Stack trace in the Jira issue indicates root cause of OM startup failure was No space left on device. Can you please explain how a change in SnapshotDefragService solves that problem?
• I'm not sure allowing OM to start with No space left on device is useful/desired.
• Please describe testing in more detail. "Existing tests" does not cut it, since this is supposed to fix a bug, but existing tests are already passing without the patch.

[cchung100m]

Hi @adoroszlai

Sorry for the late reply and the incomplete PR before. I think the No space left on device is caused by checkpoint directory accumulation. Therefore, I added cleanup code that deletes the checkpoint directory on failure. I implemented multi-layer exception handling with an explicit defragSuccessful flag detailed in the PR description.

[adoroszlai]

I think the No space left on device is caused by checkpoint directory accumulation. Therefore, I added cleanup code that deletes the checkpoint directory on failure.

Thanks for the explanation, makes sense.