Skip to content

Delete driver pod in update-clusterpolicy #2136

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
JunAr7112 wants to merge 1 commit into
base: main
Choose a base branch
from
+8 −0
Open

Delete driver pod in update-clusterpolicy #2136

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions tests/scripts/update-clusterpolicy.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,14 @@ test_custom_labels_override() {

operands="nvidia-driver-daemonset nvidia-container-toolkit-daemonset nvidia-operator-validator gpu-feature-discovery nvidia-dcgm-exporter nvidia-device-plugin-daemonset"

# Wait for the operator to update the DaemonSet pod templates with new labels (driver daemonset as representative).
echo "Waiting for DaemonSet pod template to be updated with new labels..."
kubectl wait daemonset -n "$TEST_NAMESPACE" nvidia-driver-daemonset --for=jsonpath='{.spec.template.metadata.labels.cloudprovider}'=aws --timeout=120s

# Delete driver pod to force recreation with updated labels. Existing pods are not automatically restarted due to the DaemonSet OnDelete updateStrategy.
echo "Deleting driver pod to trigger recreation with updated labels..."
kubectl delete pod -n "$TEST_NAMESPACE" -l app=nvidia-driver-daemonset --ignore-not-found
Copy link
Contributor

@rahulait rahulait Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Upgrade-controller is responsible for deleting the pod. We shouldn't be deleting manually in the tests. When the labels for daemonset changes, daemonset hash changes. Upgrade controller then detects that hash of daemonset has changed and then it labels the node as "upgrade-required". Then upgrade-controller gracefully terminates the driver pod and new one gets created after that.

If its failing to delete sometimes, we need to gather more logs as to why and maybe it takes some extra time to catch up. So I would try to understand and see if increasing time for the test(like timing out after 10 mins to timeout after 15 mins) helps or not.

Copy link
Contributor Author

@JunAr7112 JunAr7112 Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I see. I'll try looking at the logs here and verifying why it isn't deleting.


# The labels override triggers a rollout of all gpu-operator operands, so we wait for the driver upgrade to transition to "upgrade-done" state.
wait_for_driver_upgrade_done

Expand Down