OCPBUGS-75913: Fix VolumeSnapshot race condition in backup processing

[jparrill]

What this PR does / why we need it

This PR fixes a race condition in the CheckVolumeSnapshot() function where VolumeSnapshots deleted between the List and Get calls would cause the entire backup to fail. The fix adds graceful handling of NotFound errors, allowing the plugin to skip deleted snapshots and continue processing remaining ones.

Problem: When iterating over VolumeSnapshots from a List call, if a snapshot is deleted between the List (line 286) and subsequent Get call (line 292), the 404 NotFound error was treated as fatal, aborting processing of subsequent snapshots and marking backups as PartiallyFailed.

Solution: Add apierrors.IsNotFound(err) handling in CheckVolumeSnapshot() to match the existing pattern in CheckVolumeSnapshotContent(), allowing deleted snapshots to be skipped gracefully.

Impact: Reduces backup failure rate from 25/100 (observed with ARO-HCP PremiumV2 disks) to near 0% by eliminating this specific race condition.

Which issue(s) this PR fixes

• Fixes https://issues.redhat.com/browse/OCPBUGS-75913
• Depending on NO_JIRA: chore(dependencies): Update dependencies to be aligned with Hypershift repo #184 to fix the tests

Checklist

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

---

🤖 Generated with Claude Code via /jira:solve OCPBUGS-75913 origin

Summary by CodeRabbit

• Bug Fixes
  • Improved volume snapshot validation to handle unavailable snapshots more gracefully. When a snapshot cannot be found during processing, the system now continues to the next item instead of terminating the entire check. Other validation errors are still properly reported.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci-robot]

@jparrill: This pull request references Jira Issue OCPBUGS-75913, which is invalid:

• expected the bug to target the "4.22.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

What this PR does / why we need it

This PR fixes a race condition in the CheckVolumeSnapshot() function where VolumeSnapshots deleted between the List and Get calls would cause the entire backup to fail. The fix adds graceful handling of NotFound errors, allowing the plugin to skip deleted snapshots and continue processing remaining ones.

Problem: When iterating over VolumeSnapshots from a List call, if a snapshot is deleted between the List (line 286) and subsequent Get call (line 292), the 404 NotFound error was treated as fatal, aborting processing of subsequent snapshots and marking backups as PartiallyFailed.

Solution: Add apierrors.IsNotFound(err) handling in CheckVolumeSnapshot() to match the existing pattern in CheckVolumeSnapshotContent(), allowing deleted snapshots to be skipped gracefully.

Impact: Reduces backup failure rate from 25/100 (observed with ARO-HCP PremiumV2 disks) to near 0% by eliminating this specific race condition.

Which issue(s) this PR fixes

Fixes https://issues.redhat.com/browse/OCPBUGS-75913

Checklist

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

---

🤖 Generated with Claude Code via /jira:solve OCPBUGS-75913 origin

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro

Cache: Disabled due to data retention organization setting

Knowledge base: Disabled due to data retention organization setting

📥 Commits

Reviewing files that changed from the base of the PR and between ce5f520 and 981d4ff.

📒 Files selected for processing (1)

• pkg/common/utils.go

🚧 Files skipped from review as they are similar to previous changes (1)

• pkg/common/utils.go

---

Walkthrough

The CheckVolumeSnapshot function in pkg/common/utils.go is modified to handle NotFound errors by skipping missing VolumeSnapshot objects during iteration instead of returning an error. Other error types still return an error as before.

Changes

Cohort / File(s)	Summary
Error Handling in VolumeSnapshot Check pkg/common/utils.go	Modified CheckVolumeSnapshot to continue iteration when a NotFound error occurs during VolumeSnapshot retrieval, while maintaining error returns for other error types.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: fixing a race condition in VolumeSnapshot handling during backup processing by gracefully skipping deleted snapshots.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Stable And Deterministic Test Names	✅ Passed	All test names in the repository are stable and deterministic, using standard Go testing framework with static, descriptive test names that clearly indicate what each test validates.
Test Structure And Quality	✅ Passed	No test code was added in this PR. Per PR objectives, tests are excluded and depend on PR 184.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jparrill

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [jparrill]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jparrill]

/jira refresh

[openshift-ci-robot]

@jparrill: This pull request references Jira Issue OCPBUGS-75913, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira (yli2@redhat.com), skipping review request.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[jparrill]

/auto-cc

[enxebre]

seems like you might want to add a test case and a log for this case.
I'm also curious about the success ratio, why do those volumes are deleted?
/lgtm

[michalskrivanek]

LGTM, I observed the problem with a similar probability on BM as well, cherry-picked this fix to latest oadp-1.5 and it fixes the problem indeed

@enxebre IIUC in my case at least the VolumeSnapshot is processed and deleted by the time plugin gets to it

[jparrill]

Separating the deps bump in other commit:

• NO_JIRA: chore(dependencies): Update dependencies to be aligned with Hypershift repo #184

[openshift-ci-robot]

@jparrill: This pull request references Jira Issue OCPBUGS-75913, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira (jiezhao@redhat.com), skipping review request.

Details

In response to this:

What this PR does / why we need it

This PR fixes a race condition in the CheckVolumeSnapshot() function where VolumeSnapshots deleted between the List and Get calls would cause the entire backup to fail. The fix adds graceful handling of NotFound errors, allowing the plugin to skip deleted snapshots and continue processing remaining ones.

Problem: When iterating over VolumeSnapshots from a List call, if a snapshot is deleted between the List (line 286) and subsequent Get call (line 292), the 404 NotFound error was treated as fatal, aborting processing of subsequent snapshots and marking backups as PartiallyFailed.

Solution: Add apierrors.IsNotFound(err) handling in CheckVolumeSnapshot() to match the existing pattern in CheckVolumeSnapshotContent(), allowing deleted snapshots to be skipped gracefully.

Impact: Reduces backup failure rate from 25/100 (observed with ARO-HCP PremiumV2 disks) to near 0% by eliminating this specific race condition.

Which issue(s) this PR fixes

• Fixes https://issues.redhat.com/browse/OCPBUGS-75913
• Depending on NO_JIRA: chore(dependencies): Update dependencies to be aligned with Hypershift repo #184 to fix the tests

Checklist

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

---

🤖 Generated with Claude Code via /jira:solve OCPBUGS-75913 origin

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

@jparrill: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[enxebre]

/lgtm

[openshift-ci-robot]

@jparrill: Jira Issue OCPBUGS-75913: All pull requests linked via external trackers have merged:

• openshift/hypershift-oadp-plugin#182

Jira Issue OCPBUGS-75913 has been moved to the MODIFIED state.

Details

In response to this:

What this PR does / why we need it

This PR fixes a race condition in the CheckVolumeSnapshot() function where VolumeSnapshots deleted between the List and Get calls would cause the entire backup to fail. The fix adds graceful handling of NotFound errors, allowing the plugin to skip deleted snapshots and continue processing remaining ones.

Problem: When iterating over VolumeSnapshots from a List call, if a snapshot is deleted between the List (line 286) and subsequent Get call (line 292), the 404 NotFound error was treated as fatal, aborting processing of subsequent snapshots and marking backups as PartiallyFailed.

Solution: Add apierrors.IsNotFound(err) handling in CheckVolumeSnapshot() to match the existing pattern in CheckVolumeSnapshotContent(), allowing deleted snapshots to be skipped gracefully.

Impact: Reduces backup failure rate from 25/100 (observed with ARO-HCP PremiumV2 disks) to near 0% by eliminating this specific race condition.

Which issue(s) this PR fixes

• Fixes https://issues.redhat.com/browse/OCPBUGS-75913
• Depending on NO_JIRA: chore(dependencies): Update dependencies to be aligned with Hypershift repo #184 to fix the tests

Checklist

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

---

🤖 Generated with Claude Code via /jira:solve OCPBUGS-75913 origin

Summary by CodeRabbit

• Bug Fixes
• Improved volume snapshot validation to handle unavailable snapshots more gracefully. When a snapshot cannot be found during processing, the system now continues to the next item instead of terminating the entire check. Other validation errors are still properly reported.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.