Let the listener own the EphemeralRunner Running phase transition - #4646
Let the listener own the EphemeralRunner Running phase transition#4646nikola-jokic wants to merge 1 commit into
Conversation
0f307dd to
cc67813
Compare
cc67813 to
82a24f7
Compare
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Address the phase-transition race conditions and update the Running phase documentation.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Lite
Findings: 1
New issues introduced by this change (2)
| Severity | Finding |
|---|---|
cmd/ghalistener/scaler/scaler.go — The GET and this status merge patch are not tied by a resourceVersion precondition. If the runner… |
|
cmd/ghalistener/scaler/scaler.go — This changes the public meaning of Running to indicate an assigned/busy runner, but the API type… |
What changed in this PR
This PR moves EphemeralRunner’s Running phase transition from pod reconciliation to listener job assignment.
Changes:
- Adds listener reads and phase-aware status updates.
- Keeps pod reconciliation from promoting runners to
Running. - Updates RBAC and test coverage.
| File | Review summary |
|---|---|
controllers/actions.github.com/resourcebuilder.go |
Adds listener read access for ephemeral runners. |
controllers/actions.github.com/ephemeralrunnerset_controller_test.go |
Covers listener-owned phase transitions. |
controllers/actions.github.com/ephemeralrunner_controller.go |
Retains Pending initialization; its phase write needs resource-version protection. |
controllers/actions.github.com/ephemeralrunner_controller_test.go |
Tests pod phase and readiness behavior. |
cmd/ghalistener/scaler/scaler.go |
Sets Running on assignment; requires conditional updates to prevent terminal-state races and updated phase documentation. |
cmd/ghalistener/scaler/scaler_test.go |
Tests listener phase transitions and job updates. |
Suppressed comments (1)
controllers/actions.github.com/ephemeralrunner_controller.go:858
- This guard only observes the reconciler's cached object. If that read sees an empty phase, the listener can patch Running before this pod reconcile writes its status; the subsequent merge patch then includes Pending and can overwrite the listener-owned Running phase because it is not an optimistic-lock patch. Protect this initial phase write with a resource-version/refetch strategy so the controller cannot regress a phase that the listener has already advanced.
phase := ephemeralRunner.Status.Phase
if pod.Status.Phase == corev1.PodPending && phase == "" {
phase = v1alpha1.EphemeralRunnerPhasePending
}
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| if currentRunner.DeletionTimestamp == nil && | ||
| currentRunner.Status.Phase != v1alpha1.EphemeralRunnerPhaseFailed && | ||
| currentRunner.Status.Phase != v1alpha1.EphemeralRunnerPhaseSucceeded && | ||
| currentRunner.Status.Phase != v1alpha1.EphemeralRunnerPhaseOutdated { | ||
| patchRunner.Status.Phase = v1alpha1.EphemeralRunnerPhaseRunning |
| if currentRunner.DeletionTimestamp == nil && | ||
| currentRunner.Status.Phase != v1alpha1.EphemeralRunnerPhaseFailed && | ||
| currentRunner.Status.Phase != v1alpha1.EphemeralRunnerPhaseSucceeded && | ||
| currentRunner.Status.Phase != v1alpha1.EphemeralRunnerPhaseOutdated { | ||
| patchRunner.Status.Phase = v1alpha1.EphemeralRunnerPhaseRunning |
82a24f7 to
3cad613
Compare
3cad613 to
4afa7e1
Compare
The controller derived Status.Phase directly from the pod phase, so a runner became Running as soon as its pod started, whether or not it had picked up a job. That made Running mean "the pod is up" instead of "the runner is busy", and it left the EphemeralRunnerSet scale-down path unable to tell an idle runner from one that is executing a job. The listener already knows when a job is assigned to a specific runner, so move the transition there. HandleJobStarted now reads the runner first and only promotes it to Running when it is not terminal (Failed, Succeeded or Outdated) and not being deleted, then patches the phase alongside the job fields it already writes. The listener role gains "get" on ephemeralrunners for that read. On the controller side updateRunStatusFromPod keeps publishing the initial Pending phase while the pod is starting, and no longer promotes to Running. Runners waiting for work now stay Pending, so scale-down picks them before runners that are actually executing a job. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
4afa7e1 to
9c0ca96
Compare
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Critical and moderate correctness and concurrency issues remain unresolved.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Lite
Findings: 3
New issues introduced by this change (2)
| Severity | Finding |
|---|---|
cmd/ghalistener/scaler/scaler.go — Use the lowercase EphemeralRunner resource name |
|
controllers/actions.github.com/ephemeralrunner_controller.go — Stale pod reconciliation can reset the listener-owned phase |
Pre-existing issues (2)
| Severity | Finding |
|---|---|
cmd/ghalistener/scaler/scaler.go — The GET and this status merge patch are not tied by a resourceVersion precondition. If the runner… View comment |
|
cmd/ghalistener/scaler/scaler.go — This changes the public meaning of Running to indicate an assigned/busy runner, but the API type… View comment |
| Get(). | ||
| Prefix("apis", v1alpha1.GroupVersion.Group, v1alpha1.GroupVersion.Version). | ||
| Namespace(w.config.EphemeralRunnerSetNamespace). | ||
| Resource("EphemeralRunners"). |
| phase := ephemeralRunner.Status.Phase | ||
| if pod.Status.Phase == corev1.PodPending && phase == "" { | ||
| phase = v1alpha1.EphemeralRunnerPhasePending | ||
| } |


Stacked on #4580. First of two PRs that split #4582 apart.
af55427fin #4582 bundled two unrelated things: a behavioural change to whoowns the
Runningphase, and a workqueue optimisation. This PR is thebehavioural half. The predicates follow separately so that an optimisation is
never reviewed together with a semantic change.
The problem
updateRunStatusFromPodderivedEphemeralRunner.Status.Phasestraight fromthe pod phase, so a runner flipped to
Runningthe moment its pod started,whether or not it had picked up a job.
Runningtherefore meant "the pod isup", not "the runner is busy".
That matters for scale-down.
newEphemeralRunnersByStatesgroups runners intopendingandrunning, anddeleteIdleEphemeralRunnerswalkspendingfirstand
runningsecond. Because every started runner wasRunning, the twobuckets carried no information about which runners were safe to remove.
The change
The listener already knows exactly when a job is assigned to a named runner, so
the transition moves there:
HandleJobStartedreads the runner first and only promotes it toRunningwhen it is not
Failed,SucceededorOutdatedand is not being deleted.The phase goes out in the same status merge patch as the job fields it
already writes.
getonephemeralrunnersfor that read, split outfrom the existing
ephemeralrunners/statusrule sostatuskeeps onlypatch.updateRunStatusFromPodstill publishes the initialPendingphase while thepod starts, and no longer promotes to
Running.Runners waiting for work now stay
Pending, so scale-down drains genuinely idlerunners before ones executing a job.
Note for review
The read and the patch are two calls, so a runner can reach a terminal phase in
between and still receive
Running. The window is small and the ephemeralrunner set reaps terminal runners regardless, but it is worth a look if you want
this tightened with a resource version precondition.
Testing
go test ./controllers/... ./cmd/...with envtest, all green. New coverage:TestHandleJobStartedin the scaler, plus envtest cases asserting thecontroller does not set
Runningfrom pod status and that readiness stilltracks the pod.