Skip to content

Fix EphemeralRunnerSet metadata drift check comparing against the wrong value - #4634

Merged
nikola-jokic merged 2 commits into
masterfrom
nikola-jokic-fix-ers-metadata-merge-comparison
Sep 10, 2026
Merged

Fix EphemeralRunnerSet metadata drift check comparing against the wrong value#4634
nikola-jokic merged 2 commits into
masterfrom
nikola-jokic-fix-ers-metadata-merge-comparison

Conversation

@nikola-jokic

Copy link
Copy Markdown
Collaborator

This is layer 1 of a stack splitting #4575. It is a pre-existing bug on master, independent of the update-detection rework in that PR, so it can be reviewed and merged on its own.

AutoscalingRunnerSetReconciler.Reconcile decides whether the EphemeralRunnerSet's labels and annotations need updating by comparing the live object against desired.Labels and desired.Annotations, but when it acts on that decision it assigns the result of filterAndMergeLabels and mergeAnnotations. Those two values only agree when the EphemeralRunnerSet carries nothing beyond what this controller puts there. As soon as anything else writes metadata to the object — a workload identity injector, a service mesh, an admission webhook, or a person with kubectl — the merged result keeps the foreign key while the desired map does not, so the comparison reports "modified" on every reconcile, forever.

The wasted patch is not the real cost, since a merge patch that changes nothing does not even bump the resourceVersion. The damage is the early return that follows it. Everything after that block stops running, including listener reconciliation. A listener that is deleted or evicted while a foreign annotation is present is never recreated, because the controller can no longer get past the metadata check.

The fix computes the merged labels and annotations first, compares against those, and then assigns the same values. Foreign metadata is still preserved, and the check reports "modified" only when the merge would actually change something.

The added envtest spec injects a foreign annotation and label on the EphemeralRunnerSet, drives the controller through the metadata path, then deletes the listener. Without the fix the listener is never recreated and the spec fails on a 404; with it the listener comes back and the foreign keys survive.

Validation: go build ./..., go vet ./..., gofmt -l apis/ controllers/, and the full controllers/actions.github.com envtest suite all pass.

Copilot AI lite review requested due to automatic review settings September 9, 2026 09:43
@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Hello! Thank you for your contribution.

Please review our contribution guidelines to understand the project's testing and code conventions.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The new envtest has a timing-dependent listener Get/Delete sequence that can flake unless it waits for the listener to exist before deleting it.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Lite
Findings: 1 Medium severity

New issues introduced by this change (1)
Severity Finding
Medium severity controllers/​actions.github.com/​autoscalingrunnerset_controller_test.go — This test assumes the AutoscalingListener already exists and does an immediate Get before deleting…
What changed in this PR

Fixes a reconciliation bug in AutoscalingRunnerSetReconciler where EphemeralRunnerSet label/annotation drift detection compared against the unmerged desired metadata, causing perpetual “modified” detection when third-party metadata exists and preventing later reconciliation steps (notably listener reconciliation) from running.

Changes:

  • Compute merged labels/annotations first, compare against the merged result, and then assign the same merged values when patching EphemeralRunnerSet metadata drift.
  • Add an envtest that injects “foreign” labels/annotations on the EphemeralRunnerSet and verifies listener recreation still happens and foreign metadata is preserved.
File Description
controllers/​actions.github.com/​autoscalingrunnerset_controller.go Aligns metadata drift comparison with the merged (preserving) labels/annotations that will actually be applied, avoiding perpetual drift and early-return stalls.
controllers/​actions.github.com/​autoscalingrunnerset_controller_test.go Adds coverage for preserving third-party metadata and ensuring reconciliation proceeds far enough to recreate a deleted listener.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread controllers/actions.github.com/autoscalingrunnerset_controller_test.go Outdated
@humh25

humh25 commented Sep 9, 2026 via email

Copy link
Copy Markdown

@humh25

humh25 commented Sep 9, 2026 via email

Copy link
Copy Markdown

@humh25

humh25 commented Sep 9, 2026 via email

Copy link
Copy Markdown

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The listener assertion does not yet prove that recreation completed.

Review tier: Lite
Findings: None

Issues resolved since last review (1)
Severity Finding
Medium severity controllers/​actions.github.com/​autoscalingrunnerset_controller_test.go — This test assumes the AutoscalingListener already exists and does an immediate Get before deleting… View resolved comment
Suppressed comments (1)

controllers/actions.github.com/autoscalingrunnerset_controller_test.go:725

  • This assertion only checks that the listener object is readable, not that it was recreated. AutoscalingListener has a finalizer and remains GET-able with a non-zero DeletionTimestamp while its pod/secrets/RBAC resources are cleaned up (controllers/actions.github.com/autoscalinglistener_controller.go:83-109,587-690), so the pre-fix controller can satisfy this Eventually before the listener is deleted and never recreated. Assert that recreated.DeletionTimestamp.IsZero() and that its UID differs from the UID captured before deletion.
					recreated := new(v1alpha1.AutoscalingListener)
					g.Expect(k8sClient.Get(ctx, client.ObjectKey{Name: scaleSetListenerName(autoscalingRunnerSet), Namespace: autoscalingRunnerSet.Namespace}, recreated)).To(Succeed())

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The fix preserves foreign metadata, restores listener reconciliation, and includes targeted test coverage.

Review tier: Lite
Findings: None

Issues resolved since last review (1)
Severity Finding
Medium severity controllers/​actions.github.com/​autoscalingrunnerset_controller_test.go — This test assumes the AutoscalingListener already exists and does an immediate Get before deleting… View resolved comment

@nikola-jokic
nikola-jokic force-pushed the nikola-jokic-fix-ers-metadata-merge-comparison branch from 2e48517 to 5c66247 Compare September 10, 2026 09:12
nikola-jokic and others added 2 commits September 10, 2026 11:20
…ng value

The AutoscalingRunnerSet controller decides whether the EphemeralRunnerSet's
labels and annotations need updating by comparing the live object against
desired.Labels and desired.Annotations, but when it acts on that decision it
assigns the result of filterAndMergeLabels and mergeAnnotations instead. Those
two values only agree when the EphemeralRunnerSet carries nothing beyond what
this controller puts there. As soon as anything else writes to the object -- a
workload identity injector, a service mesh, an admission webhook, or a person
with kubectl -- the merged result keeps the foreign key while the desired map
does not, so the comparison reports "modified" forever.

Every reconcile then takes that branch, issues a patch that changes nothing,
and returns early. The early return is the damaging part: everything after the
block, including listener reconciliation, stops running entirely. A listener
that is deleted or evicted at that point is never recreated, because the
controller can no longer get past the metadata check.

Compute the merged labels and annotations first and compare against those, then
assign the same values. Foreign metadata is still preserved, and the check
reports "modified" only when the merge would actually change something.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@nikola-jokic
nikola-jokic force-pushed the nikola-jokic-fix-ers-metadata-merge-comparison branch from 5c66247 to 86c974a Compare September 10, 2026 09:20
@nikola-jokic
nikola-jokic merged commit c475023 into master Sep 10, 2026
32 of 33 checks passed
@nikola-jokic
nikola-jokic deleted the nikola-jokic-fix-ers-metadata-merge-comparison branch September 10, 2026 09:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants