Fix EphemeralRunnerSet metadata drift check comparing against the wrong value - #4634
Conversation
|
Hello! Thank you for your contribution. Please review our contribution guidelines to understand the project's testing and code conventions. |
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The new envtest has a timing-dependent listener Get/Delete sequence that can flake unless it waits for the listener to exist before deleting it.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Lite
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
controllers/actions.github.com/autoscalingrunnerset_controller_test.go — This test assumes the AutoscalingListener already exists and does an immediate Get before deleting… |
What changed in this PR
Fixes a reconciliation bug in AutoscalingRunnerSetReconciler where EphemeralRunnerSet label/annotation drift detection compared against the unmerged desired metadata, causing perpetual “modified” detection when third-party metadata exists and preventing later reconciliation steps (notably listener reconciliation) from running.
Changes:
- Compute merged labels/annotations first, compare against the merged result, and then assign the same merged values when patching EphemeralRunnerSet metadata drift.
- Add an envtest that injects “foreign” labels/annotations on the EphemeralRunnerSet and verifies listener recreation still happens and foreign metadata is preserved.
| File | Description |
|---|---|
| controllers/actions.github.com/autoscalingrunnerset_controller.go | Aligns metadata drift comparison with the merged (preserving) labels/annotations that will actually be applied, avoiding perpetual drift and early-return stalls. |
| controllers/actions.github.com/autoscalingrunnerset_controller_test.go | Adds coverage for preserving third-party metadata and ensuring reconciliation proceeds far enough to recreate a deleted listener. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
536975948763
humbertohlgl
El sept 9, 2026, 03:48 -0600, Copilot ***@***.***>, escribió:
|
|
536975948763
humbertohlgl
El sept 9, 2026, 03:44 -0600, Nikola Jokic ***@***.***>, escribió:
|
|
536975948763
humbertohlgl
El sept 9, 2026, 03:44 -0600, github-actions[bot] ***@***.***>, escribió:
|
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
The listener assertion does not yet prove that recreation completed.
Review tier: Lite
Findings: None
Issues resolved since last review (1)
| Severity | Finding |
|---|---|
controllers/actions.github.com/autoscalingrunnerset_controller_test.go — This test assumes the AutoscalingListener already exists and does an immediate Get before deleting… View resolved comment |
Suppressed comments (1)
controllers/actions.github.com/autoscalingrunnerset_controller_test.go:725
- This assertion only checks that the listener object is readable, not that it was recreated. AutoscalingListener has a finalizer and remains GET-able with a non-zero
DeletionTimestampwhile its pod/secrets/RBAC resources are cleaned up (controllers/actions.github.com/autoscalinglistener_controller.go:83-109,587-690), so the pre-fix controller can satisfy thisEventuallybefore the listener is deleted and never recreated. Assert thatrecreated.DeletionTimestamp.IsZero()and that its UID differs from the UID captured before deletion.
recreated := new(v1alpha1.AutoscalingListener)
g.Expect(k8sClient.Get(ctx, client.ObjectKey{Name: scaleSetListenerName(autoscalingRunnerSet), Namespace: autoscalingRunnerSet.Namespace}, recreated)).To(Succeed())
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The fix preserves foreign metadata, restores listener reconciliation, and includes targeted test coverage.
Review tier: Lite
Findings: None
Issues resolved since last review (1)
| Severity | Finding |
|---|---|
controllers/actions.github.com/autoscalingrunnerset_controller_test.go — This test assumes the AutoscalingListener already exists and does an immediate Get before deleting… View resolved comment |
2e48517 to
5c66247
Compare
…ng value The AutoscalingRunnerSet controller decides whether the EphemeralRunnerSet's labels and annotations need updating by comparing the live object against desired.Labels and desired.Annotations, but when it acts on that decision it assigns the result of filterAndMergeLabels and mergeAnnotations instead. Those two values only agree when the EphemeralRunnerSet carries nothing beyond what this controller puts there. As soon as anything else writes to the object -- a workload identity injector, a service mesh, an admission webhook, or a person with kubectl -- the merged result keeps the foreign key while the desired map does not, so the comparison reports "modified" forever. Every reconcile then takes that branch, issues a patch that changes nothing, and returns early. The early return is the damaging part: everything after the block, including listener reconciliation, stops running entirely. A listener that is deleted or evicted at that point is never recreated, because the controller can no longer get past the metadata check. Compute the merged labels and annotations first and compare against those, then assign the same values. Foreign metadata is still preserved, and the check reports "modified" only when the merge would actually change something. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
5c66247 to
86c974a
Compare

This is layer 1 of a stack splitting #4575. It is a pre-existing bug on master, independent of the update-detection rework in that PR, so it can be reviewed and merged on its own.
AutoscalingRunnerSetReconciler.Reconciledecides whether the EphemeralRunnerSet's labels and annotations need updating by comparing the live object againstdesired.Labelsanddesired.Annotations, but when it acts on that decision it assigns the result offilterAndMergeLabelsandmergeAnnotations. Those two values only agree when the EphemeralRunnerSet carries nothing beyond what this controller puts there. As soon as anything else writes metadata to the object — a workload identity injector, a service mesh, an admission webhook, or a person with kubectl — the merged result keeps the foreign key while the desired map does not, so the comparison reports "modified" on every reconcile, forever.The wasted patch is not the real cost, since a merge patch that changes nothing does not even bump the resourceVersion. The damage is the early
returnthat follows it. Everything after that block stops running, including listener reconciliation. A listener that is deleted or evicted while a foreign annotation is present is never recreated, because the controller can no longer get past the metadata check.The fix computes the merged labels and annotations first, compares against those, and then assigns the same values. Foreign metadata is still preserved, and the check reports "modified" only when the merge would actually change something.
The added envtest spec injects a foreign annotation and label on the EphemeralRunnerSet, drives the controller through the metadata path, then deletes the listener. Without the fix the listener is never recreated and the spec fails on a 404; with it the listener comes back and the foreign keys survive.
Validation:
go build ./...,go vet ./...,gofmt -l apis/ controllers/, and the fullcontrollers/actions.github.comenvtest suite all pass.