Skip to content

CNTRLPLANE-3201: ci(hypershift): add self-managed Azure OADP backup/restore E2E test - #82796

Open
mgencur wants to merge 7 commits into
openshift:mainfrom
mgencur:CNTRLPLANE-3201_backup_restore_self_managed_azure
Open

CNTRLPLANE-3201: ci(hypershift): add self-managed Azure OADP backup/restore E2E test#82796
mgencur wants to merge 7 commits into
openshift:mainfrom
mgencur:CNTRLPLANE-3201_backup_restore_self_managed_azure

Conversation

@mgencur

@mgencur mgencur commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

https://redhat.atlassian.net/browse/CNTRLPLANE-3201

Summary by CodeRabbit

  • Adds optional and weekly Azure CI jobs for HyperShift self-managed OADP backup and restore tests on main, 4.23, and 5.0.
  • Adds a workflow that provisions management and guest clusters, configures Azure Blob Storage and federated workload identity, runs backup, restore, and conformance tests, and cleans up resources.
  • Adds Azure OADP setup and destroy steps with credential handling, managed identity configuration, retry logic, and resource cleanup.
  • Enables tech-preview settings for the related AWS backup and restore periodic jobs.

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 3, 2026
@openshift-ci
openshift-ci Bot requested review from bryan-cox and enxebre August 3, 2026 10:06
@openshift-ci

openshift-ci Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mgencur
Once this PR has been reviewed and has the lgtm label, please assign jparrill for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

The pull request adds Azure self-managed backup and restore workflows for HyperShift. It provisions OADP resources, runs optional and scheduled CI jobs, and cleans up Azure storage and identities.

Changes

Azure self-managed backup and restore

Layer / File(s) Summary
Provision Azure OADP resources
ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/*
Creates Azure storage, managed identity federation, workload identity credentials, OADP resources, and backup and snapshot locations.
Wire the backup and restore workflow
ci-operator/step-registry/hypershift/self-managed-azure/e2e-backuprestore/*, ci-operator/step-registry/hypershift/self-managed-azure/OWNERS
Adds the workflow, metadata, ownership links, cluster setup, OADP testing, backup and restore testing, conformance testing, and cleanup.
Clean up Azure OADP resources
ci-operator/step-registry/hypershift/self-managed-azure/oadp-destroy/*
Adds retrying deletion for marked storage accounts and managed identities, with aggregate failure reporting.
Register Azure CI jobs
ci-operator/config/openshift/hypershift/*
Adds optional and weekly Azure backup and restore jobs. Enables TechPreview mode for existing AWS periodic jobs.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CIJob
  participant AzureWorkflow
  participant OADPSetup
  participant GuestCluster
  participant Azure
  participant OADPDestroy
  CIJob->>AzureWorkflow: Start backup and restore test
  AzureWorkflow->>OADPSetup: Provision OADP prerequisites
  OADPSetup->>Azure: Create storage and managed identity
  OADPSetup->>GuestCluster: Configure OADP and workload identity
  AzureWorkflow->>GuestCluster: Run backup, restore, and conformance tests
  AzureWorkflow->>OADPDestroy: Run cleanup
  OADPDestroy->>Azure: Delete storage and managed identity
Loading

Suggested labels: rehearsals-ack

Suggested reviewers: enxebre


Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error, 1 warning)

Check name Status Explanation Resolution
No-Weak-Crypto ❌ Error The PR adds md5sum in the Azure OADP setup script to derive JOB_SUFFIX from PROW_JOB_ID (line 30). The check flags MD5 usage. Replace md5sum with an approved modern digest such as sha256sum, or use a non-cryptographic unique-name mechanism that does not invoke weak crypto.
Ipv6 And Disconnected Network Test Compatibility ⚠️ Warning The new Ginkgo backup/restore job invokes Azure APIs and directly uses quay.io/konveyor/hypershift-oadp-plugin:latest; no mirror or disconnected skip is configured. Apply the IPv6/disconnected notice: run periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-ipv6, use GetIPAddressFamily(), mirror quay.io, or mark the test [Skipped:Disconnected].
✅ Passed checks (13 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the HyperShift self-managed Azure OADP backup/restore E2E test added by the pull request.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The complete branch diff adds no Ginkgo It, Describe, Context, or When declarations; the workflow only invokes the existing hypershift-e2e-backuprestore chain.
Test Structure And Quality ✅ Passed The PR adds YAML, metadata, OWNERS, and Bash files only. The comparison contains zero Go files and no Ginkgo test constructs, so these Ginkgo-specific checks are not applicable.
Microshift Test Compatibility ✅ Passed The PR adds CI workflow/configuration and Azure setup/cleanup scripts, but no new Ginkgo test declarations or test source; this MicroShift test check is not applicable.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The PR adds no Go files or Ginkgo declarations; it adds CI YAML, shell setup, and invokes the existing hypershift-e2e-backuprestore chain.
Topology-Aware Scheduling Compatibility ✅ Passed The added workflow and OADP resources introduce no topology-sensitive scheduling fields: no node selectors, affinity, spread constraints, replica/PDB settings, tolerations, or control-plane labels.
Ote Binary Stdout Contract ✅ Passed The PR changes only YAML and shell files; it adds no OTE binary code or process-level Go entry points. The test-backuprestore invocation remains in the existing chain.
Container-Privileges ✅ Passed The complete PR diff contains no privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, allowPrivilegeEscalation, or explicit root container settings.
No-Sensitive-Data-In-Logs ✅ Passed New scripts do not log passwords or tokens; Azure login and resource creation suppress output, while logs contain only resource names, client IDs, and the public OIDC issuer.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.23__periodics.yaml`:
- Around line 406-413: Set NODEPOOL_N1_VERSION to "4.22" in both periodic job
definitions:
ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.23__periodics.yaml
lines 406-413 and
ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics.yaml
lines 437-444. Add the value under each job’s env section while preserving the
existing job configuration.

In
`@ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/hypershift-self-managed-azure-oadp-setup-commands.sh`:
- Around line 113-115: Update the Velero readiness check in the setup command so
oc wait --for=condition=Available deployment/velero returns a nonzero status
when the deployment is missing or unavailable. Remove the trailing || true,
preserving the existing timeout and namespace settings so setup fails
immediately with the readiness error.
- Around line 22-23: Update the STORAGE_ACCOUNT_NAME generation to retain a
valid lowercase alphanumeric cluster stem and append a job-unique suffix derived
from PROW_JOB_ID, ensuring the complete Azure storage account name is no longer
than 24 characters. Preserve writing the generated name to SHARED_DIR so cleanup
continues to work.

In
`@ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/hypershift-self-managed-azure-oadp-setup-ref.yaml`:
- Around line 16-18: Update the documentation block for the Azure Blob Storage
container name to state that an empty value defaults to a container named using
the hypershift-oadp-${CLUSTER_NAME} pattern, matching the command script
behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: e190f3be-245f-405c-b8c9-3fa5684dc5eb

📥 Commits

Reviewing files that changed from the base of the PR and between a996379 and a15b3ae.

⛔ Files ignored due to path filters (5)
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-main-presubmits.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.23-periodics.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.23-presubmits.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-5.0-periodics.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-5.0-presubmits.yaml is excluded by !ci-operator/jobs/**
📒 Files selected for processing (17)
  • ci-operator/config/openshift/hypershift/openshift-hypershift-main.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.23.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.23__periodics.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics.yaml
  • ci-operator/step-registry/hypershift/self-managed-azure/OWNERS
  • ci-operator/step-registry/hypershift/self-managed-azure/e2e-backuprestore/OWNERS
  • ci-operator/step-registry/hypershift/self-managed-azure/e2e-backuprestore/hypershift-self-managed-azure-e2e-backuprestore-workflow.metadata.json
  • ci-operator/step-registry/hypershift/self-managed-azure/e2e-backuprestore/hypershift-self-managed-azure-e2e-backuprestore-workflow.yaml
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-destroy/OWNERS
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-destroy/hypershift-self-managed-azure-oadp-destroy-commands.sh
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-destroy/hypershift-self-managed-azure-oadp-destroy-ref.metadata.json
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-destroy/hypershift-self-managed-azure-oadp-destroy-ref.yaml
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/OWNERS
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/hypershift-self-managed-azure-oadp-setup-commands.sh
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/hypershift-self-managed-azure-oadp-setup-ref.metadata.json
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/hypershift-self-managed-azure-oadp-setup-ref.yaml

@mgencur

mgencur commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-v2-azure-self-managed-backuprestore

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@bryan-cox

Copy link
Copy Markdown
Member

/uncc @bryan-cox

@mgencur feel free to add me when its ready for review.

@openshift-ci
openshift-ci Bot removed the request for review from bryan-cox August 3, 2026 12:54
@mgencur
mgencur force-pushed the CNTRLPLANE-3201_backup_restore_self_managed_azure branch from a15b3ae to 97f100e Compare August 4, 2026 08:05
@mgencur mgencur changed the title [WIP] ci(hypershift): add self-managed Azure OADP backup/restore E2E test CNTRLPLANE-3201: ci(hypershift): add self-managed Azure OADP backup/restore E2E test Aug 4, 2026
@openshift-ci-robot

openshift-ci-robot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

@mgencur: This pull request references CNTRLPLANE-3201 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

https://redhat.atlassian.net/browse/CNTRLPLANE-3201

Summary by CodeRabbit

  • Adds optional and weekly periodic Azure CI jobs for HyperShift self-managed OADP backup and restore testing across supported release configurations.
  • Adds a workflow that provisions management and guest clusters, configures Azure Blob Storage and OADP, runs backup/restore and conformance tests, and cleans up resources.
  • Adds Azure OADP setup and destroy steps, including service-principal authentication, storage-account management, retry logic, and cleanup metadata.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Aug 4, 2026
@openshift-ci openshift-ci Bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Aug 4, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@mgencur, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

couldn't prepare candidate: couldn't rebase candidate onto 82a39e70663333d086deffc9a9007c5059301571 due to conflicts
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

mgencur added 3 commits August 4, 2026 10:44
Add hypershift-self-managed-azure-oadp-setup and -oadp-destroy steps
that discover the public self-managed-Azure guest cluster and create/
tear down an Azure Storage Account and Blob container for OADP, along
with a DataProtectionApplication, BackupStorageLocation, and
VolumeSnapshotLocation, mirroring the existing AWS OADP setup.

Wire these into a new hypershift-self-managed-azure-e2e-backuprestore
workflow (reusing e2e-v2-azure-self-managed's pre-steps) and a new
e2e-v2-azure-self-managed-backuprestore test running the minimal
conformance suite after backup/restore.
Branch the e2e-v2-azure-self-managed-backuprestore presubmit to
release-4.23 and release-5.0, and add a weekly periodic version of it
to both release branches' __periodics.yaml files.
…dp-setup

Make the target guest cluster name filter configurable via
CLUSTER_PREFIX (default "public") and remove the stray [default]
header from the Azure credentials file.
@mgencur
mgencur force-pushed the CNTRLPLANE-3201_backup_restore_self_managed_azure branch from 97f100e to b3e25be Compare August 4, 2026 08:51
@openshift-ci openshift-ci Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 4, 2026
mgencur added 2 commits August 4, 2026 11:31
…pgrade

Add TECH_PREVIEW_NO_UPGRADE: "true" to the e2e-v2-aws-backuprestore
periodic in the release-4.23 and release-5.0 configs.
…ewNoUpgrade

Add TECH_PREVIEW_NO_UPGRADE: "true" to
hypershift-self-managed-azure-e2e-backuprestore in main, release-4.23,
and release-5.0 configs.
@mgencur

mgencur commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-v2-azure-self-managed-backuprestore

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@mgencur

mgencur commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-v2-azure-self-managed-backuprestore

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@mgencur

mgencur commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-v2-azure-self-managed-backuprestore

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@mgencur

mgencur commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Hitting this when running the BackupRestoreEtcdSnapshot test:

oc logs etcd-backup-oadp-public-31ab62b058-clusters-95cvcv-s24v-54qmgl6 -n hypershift
Defaulted container "upload" out of: upload, fetch-certs (init), snapshot (init)
Error: failed to upload snapshot: failed to upload to Azure Blob hypershift-oadp-public-31ab62b058/backup-objects/backups/public-31ab62b058-clusters-95cvcv/etcd-backup/1785854592.db: DefaultAzureCredential: failed to acquire a token.
Attempted credentials:
	EnvironmentCredential: incomplete environment variable configuration. Only AZURE_TENANT_ID and AZURE_CLIENT_ID are set
	WorkloadIdentityCredential authentication failed. 
		POST https://login.microsoftonline.com/520cf09d-78ff-44ed-a731-abd623e73b09/oauth2/v2.0/token
		--------------------------------------------------------------------------------
		RESPONSE 401: 401 Unauthorized
		--------------------------------------------------------------------------------
		{
		  "error": "invalid_client",
		  "error_description": "AADSTS70025: The client '76129d62-06c8-4398-a34f-32255af836e8'(self-managed-azure-root-ci) has no configured federated identity credentials. Trace ID: 047cdc9e-12bd-4b48-bd58-b4a803445a00 Correlation ID: ecc99210-b9bf-4e2d-944b-783c41d15200 Timestamp: 2026-08-04 14:43:12Z",
		  "error_codes": [
		    70025
		  ],
		  "timestamp": "2026-08-04 14:43:12Z",
		  "trace_id": "047cdc9e-12bd-4b48-bd58-b4a803445a00",
		  "correlation_id": "ecc99210-b9bf-4e2d-944b-783c41d15200"
		}
		--------------------------------------------------------------------------------
		To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#workload
failed to upload snapshot: failed to upload to Azure Blob hypershift-oadp-public-31ab62b058/backup-objects/backups/public-31ab62b058-clusters-95cvcv/etcd-backup/1785854592.db: DefaultAzureCredential: failed to acquire a token.
Attempted credentials:
	EnvironmentCredential: incomplete environment variable configuration. Only AZURE_TENANT_ID and AZURE_CLIENT_ID are set
	WorkloadIdentityCredential authentication failed. 
		POST https://login.microsoftonline.com/520cf09d-78ff-44ed-a731-abd623e73b09/oauth2/v2.0/token
		--------------------------------------------------------------------------------
		RESPONSE 401: 401 Unauthorized
		--------------------------------------------------------------------------------
		{
		  "error": "invalid_client",
		  "error_description": "AADSTS70025: The client '76129d62-06c8-4398-a34f-32255af836e8'(self-managed-azure-root-ci) has no configured federated identity credentials. Trace ID: 047cdc9e-12bd-4b48-bd58-b4a803445a00 Correlation ID: ecc99210-b9bf-4e2d-944b-783c41d15200 Timestamp: 2026-08-04 14:43:12Z",
		  "error_codes": [
		    70025
		  ],
		  "timestamp": "2026-08-04 14:43:12Z",
		  "trace_id": "047cdc9e-12bd-4b48-bd58-b4a803445a00",
		  "correlation_id": "ecc99210-b9bf-4e2d-944b-783c41d15200"
		}
		--------------------------------------------------------------------------------
		To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#workload

Will need to investigate this.

Will need to follow https://redhat.atlassian.net/browse/CNTRLPLANE-3167

EDIT: Fixed by using a shared dedicated workload identity for OADP Velero and etcd-backup on self-managed Azure

@coderabbitai

coderabbitai Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@mgencur

mgencur commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-v2-azure-self-managed-backuprestore

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (2)
ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/hypershift-self-managed-azure-oadp-setup-commands.sh (2)

186-187: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Make the secret creation idempotent.

oc create secret fails if cloud-credentials already exists in openshift-adp. Under set -e the step then aborts. The rest of the script uses oc apply, which is idempotent.

oc create secret generic cloud-credentials -n openshift-adp \
  --from-file cloud="${AZURE_CREDS_FILE}" \
  --dry-run=client -o yaml | oc apply -f -
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/hypershift-self-managed-azure-oadp-setup-commands.sh`
around lines 186 - 187, Update the cloud-credentials creation command in the
OADP setup script to use a client-side dry-run piped into oc apply, preserving
the existing secret name, namespace, and cloud credentials file while making
repeated runs succeed.

55-60: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Add retry to the blob container creation.

--auth-mode login requires a data-plane RBAC role for the service principal on the newly created account. Azure RBAC propagation is delayed for tens of seconds after account creation. This single attempt can fail with AuthorizationPermissionMismatch and abort the step. The script already retries the role assignment and identity lookups for the same reason.

Wrap this call in the same retry pattern used at Lines 158-170.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/hypershift-self-managed-azure-oadp-setup-commands.sh`
around lines 55 - 60, Update the blob container creation command near the
“Creating blob container” message to use the same retry pattern as the
role-assignment and identity-lookup logic around the referenced retry block.
Retry Azure failures caused by delayed RBAC propagation while preserving the
existing container name, storage account, login authentication, and output
settings.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@ci-operator/step-registry/hypershift/self-managed-azure/oadp-destroy/hypershift-self-managed-azure-oadp-destroy-commands.sh`:
- Around line 30-32: Guard both marker-file reads in the cleanup script: before
reading STORAGE_RESOURCEGROUP_MARKER and OADP_MI_RG_MARKER, verify each file
exists; when either is missing, record the failure in OVERALL_RESULT and
continue cleanup so set -e does not abort before managed identity removal.
Preserve the existing reads when the markers are present.

In
`@ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/hypershift-self-managed-azure-oadp-setup-commands.sh`:
- Around line 16-21: Update the CLUSTER_NAME assignment to tolerate a
non-matching grep by appending || true within the command substitution, allowing
the subsequent empty-result guard to execute and print diagnostics. Preserve the
existing HostedCluster lookup and failure handling.
- Around line 99-107: Update the retry loop around the OADP_MI_CLIENT_ID and
OADP_MI_PRINCIPAL_ID assignments so transient az identity show failures do not
terminate the script under set -e. Suppress or otherwise safely capture
command-substitution failures, allowing the loop to reach its retry message and
continue until both IDs are populated or the existing retry limit is exhausted.

---

Nitpick comments:
In
`@ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/hypershift-self-managed-azure-oadp-setup-commands.sh`:
- Around line 186-187: Update the cloud-credentials creation command in the OADP
setup script to use a client-side dry-run piped into oc apply, preserving the
existing secret name, namespace, and cloud credentials file while making
repeated runs succeed.
- Around line 55-60: Update the blob container creation command near the
“Creating blob container” message to use the same retry pattern as the
role-assignment and identity-lookup logic around the referenced retry block.
Retry Azure failures caused by delayed RBAC propagation while preserving the
existing container name, storage account, login authentication, and output
settings.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 9e3f3ac1-98c3-4959-a023-68f83d85e8a9

📥 Commits

Reviewing files that changed from the base of the PR and between abc3bd3 and 97b8ff3.

⛔ Files ignored due to path filters (5)
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-main-presubmits.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.23-periodics.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.23-presubmits.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-5.0-periodics.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-5.0-presubmits.yaml is excluded by !ci-operator/jobs/**
📒 Files selected for processing (17)
  • ci-operator/config/openshift/hypershift/openshift-hypershift-main.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.23.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.23__periodics.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics.yaml
  • ci-operator/step-registry/hypershift/self-managed-azure/OWNERS
  • ci-operator/step-registry/hypershift/self-managed-azure/e2e-backuprestore/OWNERS
  • ci-operator/step-registry/hypershift/self-managed-azure/e2e-backuprestore/hypershift-self-managed-azure-e2e-backuprestore-workflow.metadata.json
  • ci-operator/step-registry/hypershift/self-managed-azure/e2e-backuprestore/hypershift-self-managed-azure-e2e-backuprestore-workflow.yaml
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-destroy/OWNERS
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-destroy/hypershift-self-managed-azure-oadp-destroy-commands.sh
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-destroy/hypershift-self-managed-azure-oadp-destroy-ref.metadata.json
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-destroy/hypershift-self-managed-azure-oadp-destroy-ref.yaml
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/OWNERS
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/hypershift-self-managed-azure-oadp-setup-commands.sh
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/hypershift-self-managed-azure-oadp-setup-ref.metadata.json
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/hypershift-self-managed-azure-oadp-setup-ref.yaml
🚧 Files skipped from review as they are similar to previous changes (14)
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/hypershift-self-managed-azure-oadp-setup-ref.metadata.json
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-destroy/hypershift-self-managed-azure-oadp-destroy-ref.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-main.yaml
  • ci-operator/step-registry/hypershift/self-managed-azure/e2e-backuprestore/OWNERS
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-destroy/OWNERS
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-destroy/hypershift-self-managed-azure-oadp-destroy-ref.metadata.json
  • ci-operator/step-registry/hypershift/self-managed-azure/e2e-backuprestore/hypershift-self-managed-azure-e2e-backuprestore-workflow.yaml
  • ci-operator/step-registry/hypershift/self-managed-azure/e2e-backuprestore/hypershift-self-managed-azure-e2e-backuprestore-workflow.metadata.json
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.23.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.23__periodics.yaml
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/OWNERS
  • ci-operator/step-registry/hypershift/self-managed-azure/OWNERS

…lero and etcd-backup on self-managed Azure

oadp-setup now provisions one managed identity federated to both the
velero and etcd-backup-job ServiceAccounts, so both Velero's object-store
plugin and the HCPEtcdBackup controller authenticate via Workload
Identity instead of the shared root Service Principal, which had no
federated credential and caused AADSTS70025 failures. Adds the
Workload-Identity pod labels/SA annotation OADP needs, sets useAAD on the
BSL, and drops the now-dead standalone etcd-backup credential secret.
oadp-destroy cleans up the renamed identity marker files.
@mgencur
mgencur force-pushed the CNTRLPLANE-3201_backup_restore_self_managed_azure branch from 97b8ff3 to f2b1daa Compare August 6, 2026 06:27
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/hypershift-self-managed-azure-oadp-setup-commands.sh`:
- Line 25: Update the default construction of CONTAINER_NAME to normalize
CLUSTER_NAME to lowercase, collapse consecutive hyphens, remove invalid boundary
characters, and truncate the generated value to Azure’s 63-character limit while
preserving a valid 3-character minimum. Before az storage container create,
validate an explicitly provided OADP_AZURE_CONTAINER_NAME against Azure’s
lowercase alphanumeric/hyphen rules and fail clearly when invalid.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 2f399a1f-450e-4e8c-b393-6ef3273431a1

📥 Commits

Reviewing files that changed from the base of the PR and between f518d40 and f2b1daa.

⛔ Files ignored due to path filters (5)
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-main-presubmits.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.23-periodics.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.23-presubmits.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-5.0-periodics.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-5.0-presubmits.yaml is excluded by !ci-operator/jobs/**
📒 Files selected for processing (17)
  • ci-operator/config/openshift/hypershift/openshift-hypershift-main.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.23.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.23__periodics.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics.yaml
  • ci-operator/step-registry/hypershift/self-managed-azure/OWNERS
  • ci-operator/step-registry/hypershift/self-managed-azure/e2e-backuprestore/OWNERS
  • ci-operator/step-registry/hypershift/self-managed-azure/e2e-backuprestore/hypershift-self-managed-azure-e2e-backuprestore-workflow.metadata.json
  • ci-operator/step-registry/hypershift/self-managed-azure/e2e-backuprestore/hypershift-self-managed-azure-e2e-backuprestore-workflow.yaml
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-destroy/OWNERS
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-destroy/hypershift-self-managed-azure-oadp-destroy-commands.sh
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-destroy/hypershift-self-managed-azure-oadp-destroy-ref.metadata.json
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-destroy/hypershift-self-managed-azure-oadp-destroy-ref.yaml
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/OWNERS
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/hypershift-self-managed-azure-oadp-setup-commands.sh
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/hypershift-self-managed-azure-oadp-setup-ref.metadata.json
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/hypershift-self-managed-azure-oadp-setup-ref.yaml
🚧 Files skipped from review as they are similar to previous changes (15)
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/hypershift-self-managed-azure-oadp-setup-ref.yaml
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/OWNERS
  • ci-operator/step-registry/hypershift/self-managed-azure/e2e-backuprestore/OWNERS
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.23.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0.yaml
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-destroy/hypershift-self-managed-azure-oadp-destroy-ref.yaml
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-destroy/hypershift-self-managed-azure-oadp-destroy-ref.metadata.json
  • ci-operator/step-registry/hypershift/self-managed-azure/e2e-backuprestore/hypershift-self-managed-azure-e2e-backuprestore-workflow.metadata.json
  • ci-operator/step-registry/hypershift/self-managed-azure/OWNERS
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-destroy/OWNERS
  • ci-operator/config/openshift/hypershift/openshift-hypershift-main.yaml
  • ci-operator/step-registry/hypershift/self-managed-azure/e2e-backuprestore/hypershift-self-managed-azure-e2e-backuprestore-workflow.yaml
  • ci-operator/step-registry/hypershift/self-managed-azure/oadp-setup/hypershift-self-managed-azure-oadp-setup-ref.metadata.json
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.23__periodics.yaml

…s check

Use PROW_JOB_ID-derived suffix for Azure storage account uniqueness,
remove || true from Velero wait, and align backuprestore CI configs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@mgencur
mgencur force-pushed the CNTRLPLANE-3201_backup_restore_self_managed_azure branch from f2b1daa to e3fc66b Compare August 6, 2026 06:51
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@mgencur: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-hypershift-main-e2e-v2-azure-self-managed-backuprestore openshift/hypershift presubmit Presubmit changed
pull-ci-openshift-hypershift-release-4.23-e2e-v2-azure-self-managed-backuprestore openshift/hypershift presubmit Presubmit changed
pull-ci-openshift-hypershift-release-5.0-e2e-v2-azure-self-managed-backuprestore openshift/hypershift presubmit Presubmit changed
periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-v2-azure-self-managed-backuprestore N/A periodic Periodic changed
periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-v2-aws-backuprestore N/A periodic Ci-operator config changed
periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-v2-aws-backuprestore N/A periodic Ci-operator config changed
periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-v2-azure-self-managed-backuprestore N/A periodic Periodic changed
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@openshift-ci

openshift-ci Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

@mgencur: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-v2-azure-self-managed-backuprestore 15ec390 link unknown /pj-rehearse periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-v2-azure-self-managed-backuprestore
ci/rehearse/periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-v2-azure-self-managed-backuprestore 97b8ff3 link unknown /pj-rehearse periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-v2-azure-self-managed-backuprestore

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@mgencur

mgencur commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

/hold

Depends on openshift/hypershift#9226 . We need to re-run tests here after merging the PR.

@openshift-ci openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants