Skip to content

Filter owned-resource events in the workqueue - #4647

Open
nikola-jokic wants to merge 1 commit into
nikola-jokic/listener-owns-running-phasefrom
nikola-jokic/workqueue-predicates
Open

Filter owned-resource events in the workqueue#4647
nikola-jokic wants to merge 1 commit into
nikola-jokic/listener-owns-running-phasefrom
nikola-jokic/workqueue-predicates

Conversation

@nikola-jokic

@nikola-jokic nikola-jokic commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #4646, which is stacked on #4580. Second of two PRs that split #4582
apart. This is the optimisation half, and it is meant to be behaviour
preserving.

The problem

Each controller woke up on every update of the resources it owns, including
updates that only touch fields it never reads. An EphemeralRunner status
carries readiness, failure bookkeeping and the job details the listener writes.
A pod reports IPs, its node, a start time and several conditions. An
EphemeralRunnerSet rewrites Status.FinishedRunnerCleanupPatchID for every
listener patch id. None of that is input to the owner, and all of it enqueued a
reconcile.

The change

An update predicate on each owned watch. A predicate must only ever be an
optimisation, so each one is written as the projection of the fields its
reconciler actually reads, and drops an update only when all of them are equal:

Watch Fields that wake the reconciler
ARS -> EphemeralRunnerSet object metadata, the whole spec, Status.Phase and Status.AppliedActionableRevision
ERS -> EphemeralRunner object metadata, spec, Status.Phase, Status.RunnerID
ER -> Pod UID, deletion timestamp, pod phase/reason/message, container and init container statuses, Ready condition

Object metadata here means generation, deletion timestamp, finalizers, labels,
annotations and owner references. Annotations matter because the runner set
reads AnnotationKeyPatchID and AnnotationKeyActionableRevision off runners,
and because the autoscaling runner set patches runner set labels and
annotations without touching the spec.

The ARS predicate has to keep Status.Phase and
Status.AppliedActionableRevision because ephemeralRunnerSetOutdatedForAppliedRevision
reads exactly that pair to decide whether the autoscaling runner set becomes
Outdated. Status.FinishedRunnerCleanupPatchID is the only runner set status
field it never reads, and it is the one that churns.

podReady is extracted so the pod predicate and updateRunStatusFromPod cannot
disagree about what readiness means.

Why this does not change behaviour

The version of these predicates in #4582 did change behaviour, in five ways.
This PR does not carry any of them:

  1. It filtered EphemeralRunner status on the primary watch, keyed on
    generation. markAsFailed and markAsOutdated patch status and return; the
    pod and secret are cleaned up on the next reconcile via IsDone(). That
    reconcile would never have fired. Here the primary watches are untouched,
    so a reconciler handing work to its own next pass still re-enqueues.
  2. Same predicate dropped the Status.Failures patch from deletePodAsFailed,
    which drives the maxFailures delete path. Same fix.
  3. It filtered the EphemeralRunnerSet primary watch on generation, but
    autoscalingrunnerset_controller.go patches runner set labels and
    annotations without touching the spec, so generation does not move. Those
    would never have propagated to the runners. Same fix.
  4. Its EphemeralRunner predicate ignored annotations, which the runner set
    reads for patch id and actionable revision. Annotations are compared here.
  5. Its pod predicate ignored Status.Conditions, the source of
    Status.Ready, and Status.Reason/Message, which are copied into the
    runner status. All three are compared here.

Two smaller hardenings: an event whose type does not match is now delivered
rather than dropped, and create, delete and generic events are untouched.

Testing

go test ./controllers/... ./cmd/... with envtest, all green, no changes needed
to existing tests. predicates_test.go adds 40 cases that pin the contract from
both sides: one per field that must wake each reconciler, plus cases asserting
the specific noise fields stay filtered.

@nikola-jokic
nikola-jokic force-pushed the nikola-jokic/workqueue-predicates branch from 197555c to d48fc8d Compare September 10, 2026 12:07
@nikola-jokic
nikola-jokic added this pull request to stack #4645 September 10, 2026 12:07
@nikola-jokic
nikola-jokic force-pushed the nikola-jokic/workqueue-predicates branch from d48fc8d to 6e48a61 Compare September 10, 2026 12:52
@nikola-jokic
nikola-jokic requested a lite review from Copilot September 10, 2026 14:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The critical Status.JobID projection issue must be addressed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Lite
Findings: 1 High severity · 1 Low severity

New issues introduced by this change (2)
Severity Finding
High severity controllers/​actions.github.com/​predicates.go — The EphemeralRunnerSet reconciler uses HasJob() (which is based on Status.JobID) to skip busy…
Low severity controllers/​actions.github.com/​predicates_test.go — The ARS predicate adds Status.Phase and Status.AppliedActionableRevision as wake-up fields, but…
What changed in this PR

Optimizes controller workqueues by filtering irrelevant owned-resource updates while preserving reconciliation behavior.

Changes:

  • Adds field-aware predicates for owned resource watches.
  • Extracts shared pod readiness logic.
  • Adds predicate tests and wires predicates into three controllers.
File Summary
controllers/​actions.github.com/​predicates.go Defines owned-resource predicates. Critical (3 votes): include Status.JobID in the runner projection.
controllers/​actions.github.com/​predicates_test.go Tests predicate behavior. Nit (3 votes): add positive cases for ARS phase and actionable revision.
controllers/​actions.github.com/​ephemeralrunnerset_controller.go Applies the runner predicate.
controllers/​actions.github.com/​ephemeralrunner_controller.go Applies the pod predicate and shared readiness helper.
controllers/​actions.github.com/​autoscalingrunnerset_controller.go Applies the runner-set predicate.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +79 to +80
return oldRunner.Status.Phase != newRunner.Status.Phase ||
oldRunner.Status.RunnerID != newRunner.Status.RunnerID
Comment on lines +39 to +45
for name, mutate := range map[string]func(*v1alpha1.EphemeralRunnerSet){
"replicas": func(s *v1alpha1.EphemeralRunnerSet) { s.Spec.Replicas = 5 },
"patch id": func(s *v1alpha1.EphemeralRunnerSet) { s.Spec.PatchID = 9 },
"actionable revision": func(s *v1alpha1.EphemeralRunnerSet) { s.Spec.ActionableRevision = 7 },
"ephemeral runner spec": func(s *v1alpha1.EphemeralRunnerSet) {
s.Spec.EphemeralRunnerSpec.GitHubConfigURL = "https://github.com/org"
},
Every controller in this package woke up on every update of the resources it
owns, including the ones that only touch fields it never reads. An
EphemeralRunner status carries readiness, failure bookkeeping and the job
details written by the listener, a pod reports IPs, its node, a start time and
several conditions, and an EphemeralRunnerSet rewrites
Status.FinishedRunnerCleanupPatchID for every listener patch id. None of that is
input to the owner, yet all of it enqueued a reconcile.

Add an update predicate to each owned watch. A predicate is only allowed to be
an optimisation, so each one is written as the projection of the fields its
reconciler actually reads and drops an update only when all of them are equal:

  - AutoscalingRunnerSet -> EphemeralRunnerSet: object metadata, the whole spec,
    and the Status.Phase and Status.AppliedActionableRevision pair that
    ephemeralRunnerSetOutdatedForAppliedRevision consults.
  - EphemeralRunnerSet -> EphemeralRunner: object metadata, spec, Status.Phase
    and Status.RunnerID.
  - EphemeralRunner -> Pod: UID, deletion timestamp, pod phase, reason and
    message, the container and init container statuses, and the Ready
    condition.

An event of an unexpected type is always delivered, and create, delete and
generic events are untouched. The primary watches keep seeing every update, so
the status patches a reconciler makes to hand work to its own next pass, such
as marking a runner Failed or Outdated before cleaning up its resources, still
re-enqueue.

The Ready condition lookup is extracted into podReady so that the predicate and
updateRunStatusFromPod cannot disagree about what readiness means.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@nikola-jokic
nikola-jokic force-pushed the nikola-jokic/workqueue-predicates branch from 790f2bb to b3e44a1 Compare September 11, 2026 09:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants