Make the Outdated phase revision-aware - #4644
Conversation
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
patchAppliedActionableRevisionStatus can regress Status.AppliedActionableRevision under a stale reconcile, which can reintroduce incorrect Outdated/teardown behavior.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Lite
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
controllers/actions.github.com/ephemeralrunnerset_controller.go — patchAppliedActionableRevisionStatus now unconditionally sets… |
What changed in this PR
This PR makes the EphemeralRunnerSet “Outdated” phase revision-aware so that Outdated reports from runners built from a superseded runner spec don’t keep the set switched off after the user has already updated the spec. It does this by stamping runners with the set’s actionable revision and splitting “Outdated” runners into “current” vs “stale” based on the applied revision being judged, and it tightens the AutoscalingRunnerSet teardown condition to avoid acting on transient/stale Outdated status.
Changes:
- Stamp each
EphemeralRunnerwithactions.github.com/actionable-revisionand classify Outdated runners asoutdatedvsstaleOutdatedbased on the revision being applied/judged. - Recompute
EphemeralRunnerSetphase duringpatchAppliedActionableRevisionStatusagainst the revision being applied, so a spec update can clear Outdated immediately and stale phase values don’t persist. - Update controller logic and tests to delete/replace stale-outdated runners and to gate listener teardown on “outdated for the applied revision”.
| File | Description |
|---|---|
| controllers/actions.github.com/resourcebuilder.go | Stamp new runners with the set’s actionable revision annotation. |
| controllers/actions.github.com/helpers.go | Add ephemeralRunnerSetOutdatedForAppliedRevision guard for safe teardown decisions. |
| controllers/actions.github.com/helpers_outdated_test.go | Add focused unit tests for revision-scoped outdated classification and teardown guard. |
| controllers/actions.github.com/ephemeralrunnerset_controller.go | Revision-aware runner classification, stale-outdated deletion path, phase recomputation in status patching, and safer terminated() concatenation. |
| controllers/actions.github.com/ephemeralrunnerset_controller_test.go | Add envtest coverage for deleting stale-outdated runners and ensuring set doesn’t go Outdated because of them; update existing spec to annotate revision. |
| controllers/actions.github.com/constants.go | Introduce AnnotationKeyActionableRevision. |
| controllers/actions.github.com/autoscalingrunnerset_controller.go | Use the new “outdated for applied revision” guard instead of a bare phase check. |
| controllers/actions.github.com/autoscalingrunnerset_controller_test.go | Add/extend coverage around re-registration propagation behavior (incl. EphemeralRunnerSet scale set ID + revision bump). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
536975948763
humbertohlgl
El sept 9, 2026, 04:50 -0600, Copilot ***@***.***>, escribió:
|
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
One or more issues must be addressed before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Lite
Findings: 3
New issues introduced by this change (2)
| Severity | Finding |
|---|---|
controllers/actions.github.com/helpers_outdated_test.go — This test does not exercise the aliasing bug it claims to guard: with one finished and one… |
|
controllers/actions.github.com/resourcebuilder.go — The new revision-scoping behavior depends on this annotation, but the existing ResourceBuilder test… |
Pre-existing issues (1)
| Severity | Finding |
|---|---|
controllers/actions.github.com/ephemeralrunnerset_controller.go — patchAppliedActionableRevisionStatus now unconditionally sets… View comment |
Suppressed comments (1)
controllers/actions.github.com/ephemeralrunnerset_controller.go:327
- This assignment can regress the applied marker under a stale/concurrent reconcile. The function re-fetches inside a retry, and
WithMaxConcurrentReconcilesallows multiple reconciles, so another invocation may already have advancedlatest.Status.AppliedActionableRevisionbeyondtargetAppliedRevisionbefore this block runs. Writing the older target back—and classifying the runners below against that older revision—can make the controller redo cleanup or treat a superseded Outdated report as current. Keep the applied revision monotonic and use the effective latest revision for the classification.
latest.Status.AppliedActionableRevision = targetAppliedRevision
|
536975948763
humbertohlgl
El sept 9, 2026, 04:40 -0600, Nikola Jokic ***@***.***>, escribió:
|
The merge-base changed after approval.
eb57c9f to
a68ee9e
Compare
|
Note for reviewers — cross-layer coupling with #4642 that is not visible in either diff on its own. #4642 clears This PR deliberately removes that early return, because the phase recompute has to run unconditionally for a spec update to clear The failure mode is worth noting in itself: git auto-merges the two cleanly, there is no compile error, and neither layer has a test that fails in its own tree. It surfaces only when the combined tree is executed — and even then the package prints Resolution in Two things a reviewer should check rather than take on trust:
Residual, documented in the code rather than glossed: a listener restart without a spec change keeps the marker and still renumbers from 0, so a collision remains possible. It costs one suppressed reconcile, not an outage, because the listener re-enters scaling on every long-poll timeout and an idle set at |
a68ee9e to
201c5ea
Compare
aaf895f to
2140ac5
Compare
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
An unresolved moderate finding requests regression coverage for the actionable-revision annotation, with additional phase and replacement coverage gaps.
Review tier: Lite
Findings: 2
Pre-existing issues (2)
| Severity | Finding |
|---|---|
controllers/actions.github.com/resourcebuilder.go — The new revision-scoping behavior depends on this annotation, but the existing ResourceBuilder test… View comment |
|
controllers/actions.github.com/helpers_outdated_test.go — This test does not exercise the aliasing bug it claims to guard: with one finished and one… View comment |
Issues resolved since last review (1)
| Severity | Finding |
|---|---|
controllers/actions.github.com/ephemeralrunnerset_controller.go — patchAppliedActionableRevisionStatus now unconditionally sets… View resolved comment |
Suppressed comments (3)
controllers/actions.github.com/ephemeralrunnerset_controller.go:377
- This branch is responsible for clearing a previously
Outdatedset when a newer revision is applied, but the added envtest starts withRunningand the revision already applied, so it never exercises that transition. Please add a regression case that starts with an Outdated status, advances the spec revision while the old runner remains, and verifies the status returns to Running; otherwise a later change could leave the scale set switched off after the fix is applied.
if len(state.outdated) > 0 {
latest.Status.Phase = v1alpha1.EphemeralRunnerSetPhaseOutdated
} else {
latest.Status.Phase = v1alpha1.EphemeralRunnerSetPhaseRunning
controllers/actions.github.com/ephemeralrunnerset_controller_test.go:2856
- The stale-outdated branch returns immediately after deleting the runner, so the following reconcile's scale-up path is what must create its replacement. This fixture leaves
Spec.Replicasat its zero default and never asserts a new runner, so the test would still pass if this return path permanently lost capacity; use a positive desired replica count and verify that a replacement is created from the current revision.
// The stale runner is removed rather than being treated as a verdict on the
// current spec.
Eventually(func(g Gomega) {
controllers/actions.github.com/resourcebuilder.go:833
- This new annotation is the only stamp used by
ephemeralRunnerActionableRevisionto decide whether an Outdated report belongs to the current spec, but the existingnewEphemeralRunnertest never asserts it. A regression here would make every newly created runner look like revision 0 and be discarded as stale after the first update; please add a resource-builder assertion that the annotation equalsSpec.ActionableRevision(and remains controller-owned when runner metadata supplies annotations).
annotations[AnnotationKeyActionableRevision] = strconv.FormatInt(ephemeralRunnerSet.Spec.ActionableRevision, 10)
Layer 5 clears Status.FinishedRunnerCleanupPatchID beneath an early return that this layer removes, so a straight integration left the marker being cleared on every call. Guard the clear on the same advance it belongs to, and keep the phase recompute unconditional, which is what lets a spec update clear the Outdated phase immediately. Also register the resource owner index on the fake client, since the phase recompute lists the child runners the way the real manager does. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
2140ac5 to
d099740
Compare
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
patchAppliedActionableRevisionStatus wrapped retry.RetryOnConflict around a plain client.MergeFrom status patch. A merge patch carries no resourceVersion precondition, so the API server can never reject the write as conflicting and the retry wrapper could never fire. Worse, the monotonicity check inside the retry was unsound. Both the target revision and the re-fetched object come from the cache-backed client, so a stale reconcile could compare a stale target against an equally stale read, pass the guard, and patch AppliedActionableRevision backwards. That re-satisfies Spec.ActionableRevision > Status.AppliedActionableRevision and sends the controller through the idle and pending runner cleanup again. MergeFromWithOptimisticLock stamps the re-fetched resourceVersion into the patch, so the server accepts it only if that object is still live. A successful patch therefore proves the guard was evaluated against live data, which is what makes the applied marker monotonic. A stale target can now only ever under-advance the marker, never regress it, and a later reconcile with fresh data completes the advance. At this layer the re-fetch inside the retry still uses the cached client, so a genuine conflict may refetch stale data, exhaust the backoff and return the conflict error. That requeues rather than writing a bad value, so it fails closed; it is only noisier than necessary. A later change makes the refetch authoritative so the retry can resolve the conflict in place. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The optimistic lock and the monotonicity guard in
patchAppliedActionableRevisionStatus both prevent the applied revision
marker from moving backwards, and neither was covered.
Reproducing the lost update needs two reconciles racing on one object with
a stale cached read, which is awkward to provoke deterministically. The
observable property is cheaper: the emitted status patch must carry a
resourceVersion precondition, since that is what lets the API server reject
a stale write. The test intercepts SubResourcePatch and asserts on the bytes
the reconciler actually emits, so it cannot pass while the patch is built
some other way.
Both assertions were mutation checked. Reverting the patch option to
client.MergeFrom fails the precondition test, reporting the emitted patch
as {"status":{"appliedActionableRevision":7}} with no resourceVersion.
Disabling the >= guard fails the backwards test.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…sion' into nikola-jokic-ers-scale-up-after-cleanup # Conflicts: # controllers/actions.github.com/ephemeralrunnerset_controller.go
patchFinishedRunnerCleanupPatchIDStatus wrapped a plain merge patch in retry.RetryOnConflict. A plain merge patch carries no resourceVersion precondition, so the API server has nothing to reject: the write always succeeds, the retry can never fire, and the surrounding machinery reads as protection while providing none. The exposure is worse here than for the applied revision, which at least refuses to move backwards. This helper compares for equality and then writes whatever patch ID the reconcile is carrying, so it is willing to lower the marker. A reconcile serving an older patch ID can therefore overwrite a marker recorded for a newer one, and the scale-up guard then stops suppressing for the patch it actually serviced -- creating the replacement runners this layer exists to prevent. Re-fetching through the API reader narrows that window to the gap between the read and the patch, but it cannot close it, because the decision is only as fresh as the moment it was taken. The optimistic lock is what makes the write conditional on that decision still holding; a stale attempt now conflicts and is retried rather than silently recording the wrong patch ID. The test asserts on the patch bytes the reconciler actually emits rather than on an independently constructed patch, so it cannot pass while the production code builds its patch some other way. A second test covers the early return, since an unconditional write would churn the status on every reconcile for the same patch. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…cleanup' into nikola-jokic-remove-integrity-hash-annotation
…sh-annotation' into nikola-jokic-revision-aware-outdated
The neighbouring applied-revision helper refuses to move its marker backwards, so the obvious tidy-up is to make this one match. That would break it. Applied revisions derive from metadata.generation and only ever climb. Listener patch IDs do not. setDesiredWorkerState publishes 0 whenever the set is idle at MinRunners with nothing dirty, restarts its sequence from 0 when the listener restarts, and wraps explicitly at MaxInt32, so Spec.PatchID legitimately moves down. The marker only means "the gap below Spec.Replicas was created by cleaning up for exactly this patch ID", so it has to follow the patch ID wherever it goes. A marker that refused to descend would sit above every value the listener went on to publish, and since the guard suppresses only on an exact match, suppression would never fire again -- silently disabling the behaviour this layer adds. This also settles what the optimistic lock is and is not for. It cannot make an older patch ID unwritable, because the retry re-reads and re-applies the same argument, and recording the patch ID whose cleanup actually happened is a true statement regardless of ordering. What it prevents is a write decided against state that has since changed. The test pins the descending case, so a future reader who reaches for >= gets a failure rather than a silently inert guard. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…cleanup' into nikola-jokic-remove-integrity-hash-annotation
patchAppliedActionableRevisionStatus reads the EphemeralRunnerSet through the uncached reader precisely because a cached read cannot be trusted there, then derived Status.Phase from a cached list of the child runners. The optimistic lock on the patch covers the EphemeralRunnerSet object alone, so it could not vouch for that list, while a successful write made the result look consistent. The list also sits inside RetryOnConflict, where a cached read can return the same stale data on every attempt. The ownership filter moves into the process, because resourceOwnerKey is a client-side index on the manager's cache and the API server rejects .metadata.controller as an unsupported field label. The predicate lives next to the indexer so the two do not drift apart. Add the index to the revision patch test fakes, which need it now that the function lists children, and a test that separates the two reads: it fails when the list goes through the cache and passes when it does not. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…sh-annotation' into nikola-jokic-revision-aware-outdated
…ion stamp The aliasing test passed against the implementation it was written to catch. With one finished runner the classifier leaves no spare capacity, so the old nested append was forced to allocate and could not corrupt anything, and the returned slice was discarded rather than inspected. Give finished room to spare, retain the result, then append to state.finished so the reused slot overwrites a runner in the slice already handed out. Verified against the original implementation, where it now fails with the failed runner replaced. The spare capacity is asserted rather than assumed, so the test reports rather than hides it if the classifier ever stops leaving room. The actionable revision annotation had no builder-level assertion, though every test that mentions it supplies it as input. A runner that lost it parses as revision 0 and is treated as stale, so it would be deleted and replaced instead of holding the set Outdated. Cover the stamp, the zero revision, which has to be explicit because a missing annotation parses the same way, and the merge order that stops user metadata forging a revision. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved critical and moderate findings remain in the controller.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Lite
Findings: 1
New issues introduced by this change (2)
| Severity | Finding |
|---|---|
controllers/actions.github.com/ephemeralrunnerset_controller.go — Use the live applied revision when classifying runners |
|
controllers/actions.github.com/ephemeralrunnerset_controller.go — Avoid namespace-wide API reads during revision cleanup |
Issues resolved since last review (2)
| Severity | Finding |
|---|---|
controllers/actions.github.com/resourcebuilder.go — The new revision-scoping behavior depends on this annotation, but the existing ResourceBuilder test… View resolved comment |
|
controllers/actions.github.com/helpers_outdated_test.go — This test does not exercise the aliasing bug it claims to guard: with one finished and one… View resolved comment |
Layers 2-4 removed every reader of the actions.github.com/integrity-hash annotation, so what remained were write-only dead paths. Delete the annotation constant, all remaining hash helpers that computed it, the call sites that stamped it onto objects, the integrity-hash fallback in newResourceCacheObjectRef, and the now-unused AutoscalingListenerSpec.Hash and EphemeralRunnerSpec.Hash methods. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Code review raised two upgrade consequences of no longer stamping actions.github.com/integrity-hash. Both are real, and neither is covered by a test, so cover them. The AutoscalingRunnerSet reconciler compares a live listener's annotations against the desired ones exactly, so a listener created by an older controller is replaced on the first reconcile after an upgrade. That rollout is one-time rather than a loop, because the replacement is built by the same path and carries no annotation. It is not additional either: the listener runs the manager's own image, so a controller upgrade already forces the same recreation through the Spec comparison. Reconcilers that update objects in place merge live annotations under the desired ones, so the annotation survives on objects that already carry it. Nothing reads it any more, so it is inert metadata; stripping it would mean patching every managed object on upgrade, which is a separate decision. Both tests were mutation-checked: re-stamping the annotation fails the first, and stripping the key in mergeAnnotations fails the second. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The merge-base changed after approval.
…sh-annotation' into nikola-jokic-revision-aware-outdated # Conflicts: # controllers/actions.github.com/ephemeralrunnerset_controller.go # controllers/actions.github.com/ephemeralrunnerset_controller_test.go # controllers/actions.github.com/ephemeralrunnerset_revision_patch_test.go # controllers/actions.github.com/ephemeralrunnerset_scaleup_suppression_test.go # controllers/actions.github.com/helpers.go
Reconcile reads the EphemeralRunnerSet through the cache and passes Spec.ActionableRevision to patchAppliedActionableRevisionStatus, which re-reads the status through APIReader. When another reconcile has already advanced the live marker, a lagging reconcile arrives with a target behind it. The monotonicity guard keeps the marker from regressing, but the phase is derived separately and had no such protection. Classifying against the stale target rated a runner left over from the superseded revision as current rather than staleOutdated, flipping a set that had already moved on back to Outdated. That does not self-correct: Reconcile's Outdated branch returns before updateStatus, and this function only runs while spec > applied, so the set would stay switched off until the next spec change. After the monotonicity guard the status field already holds max(live, target), so deriving the state from it is a no-op whenever the target is ahead and uses the live revision when it is behind. Also correct the comment on the child listing. A namespace can hold more than one scale set, so the read is wider than claimed. Narrowing it with a server-side label selector is still not safe, because label propagation is operator-configurable through --exclude-label-propagation-prefix, so a selector could match no runners and derive the phase from an empty list rather than fail. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>




Layer 7/7 — the final layer of the stacked split of #4575. Base:
nikola-jokic-remove-integrity-hash-annotation.The problem
An
EphemeralRunnerSetgoesOutdatedwhen a child runner reports that the Actions service rejected its runner spec, and theAutoscalingRunnerSetreacts by tearing the listener down so the scale set stops taking jobs. That is the intended behaviour — a broken runner spec should switch the scale set off rather than churn runners.But nothing recorded which runner spec a given
Outdatedreport was about. A runner that was busy with a job while the spec was updated survives the revision cleanup and only exits — with the outdated exit code — afterwards. Its verdict is about a spec that no longer exists, yet it kept the whole set inOutdated, discarding the fix the user had just applied and leaving the scale set switched off until something else nudged it.The fix
newEphemeralRunnernow stamps each runner with the set's actionable revision (actions.github.com/actionable-revision), andnewEphemeralRunnersByStatestakes the revision the runners are being judged against, splittingOutdatedrunners into two groups:staleOutdated— actionable revision older than the applied revision. Their report says nothing about the current spec, so they must not hold the set inOutdated. They are deleted and replaced by runners built from the current spec.outdated— actionable revision at or after the applied revision. Their report is about the current spec, so they drive the set intoOutdated. These are deliberately not delete-and-replaced: a fresh runner at the same revision would be rejected the same way and reportOutdatedagain, looping forever. Leaving them in place is what makes the phase stick until the spec actually changes.Runners created before the annotation existed parse to
0, which matches the zero value ofStatus.AppliedActionableRevision, so an in-place upgrade behaves exactly as today until the spec is updated for the first time.patchAppliedActionableRevisionStatusnow lists the child runners and recomputes the phase in both directions against the revision being applied. It has to: that path returns fromReconcilewithout reachingupdateStatus, so a staleRunningwould hide genuinely outdated runners from the cleanup path, and a staleOutdatedwould keep the set switched off after the spec that caused it was replaced. Judging against the revision being applied (rather than the one in status) is what lets a spec update clearOutdatedimmediately instead of waiting for the pre-update runners to be collected.Finally, the
AutoscalingRunnerSetteardown guard moves from a bare phase check toephemeralRunnerSetOutdatedForAppliedRevision, which additionally requires the applied revision to have caught up with the spec revision. That closes the window between theAutoscalingRunnerSetpatching a new runner spec onto the set and theEphemeralRunnerSetcontroller processing it, during which the set still reportsOutdatedfor a spec that is already gone.Also here
terminated()allocated its result by appending intos.finished, which could write through that slice's backing array. It now allocates a fresh, correctly-sized slice, and includesstaleOutdated.Tests
helpers_outdated_test.gocoveringephemeralRunnerSetOutdatedForAppliedRevision.Outdatedagainst a set running revision 2 is deleted, and the set never reportsOutdatedbecause of it.go build ./...,go vet ./...,gofmtclean;make manifestsproduces no diff and the chart CRD copies are byte-identical toconfig/crd/bases; the fullcontrollers/actions.github.com+apissuite passes.