Exit ephemeral runner when its job assignment is lost

[LouisCuvelier]

Fixes #4617

Problem

When GetJobMessageAsync fails with 404 / 409 / 422, Runner.cs logs Skipping message Job. and continues back into the message loop. That is correct for a persistent runner.

For an ephemeral runner it is not, on the two statuses that mean the assignment is gone: the service consumes the registration when it assigns the job, so there is no session left to listen on. The runner stays alive, deregistered, and is never assigned work again.

Nothing surfaces the failure. The process keeps printing Listening for Jobs, the container reports healthy, and the runner is simply absent from GET /orgs/{org}/actions/runners. On a pool of ephemeral runners each occurrence permanently removes one, so the pool drains to zero over hours and jobs queue with no error anywhere.

Observed on runner 2.334.0 (still present on main):

[11:55:53Z INFO Terminal] WRITE LINE: 2026-08-07 11:55:53Z: Listening for Jobs
[11:59:45Z INFO BrokerMessageListener] Acknowledging runner request '5f7c2a52-****'.
[11:59:46Z ERR  GitHubActionsService] POST .../acquirejob failed. HTTP Status: Conflict
[11:59:46Z INFO Runner] Skipping message Job. Job message already acquired '5f7c2a52-****'. job assignment is invalid: MissingKey

Nothing after that line. The process was still running 12 minutes later.

Change

On 404 and 409 for an ephemeral runner, exit instead of skipping, mirroring the handling #4540 already put in place ~40 lines above for a lost acknowledge. Setting runOnceJobCompleted lets the existing finally delete the local config, so the supervising process (Docker restart policy, systemd, ARC) registers a fresh runner.

422 is deliberately excluded. Unprocessable job says the job cannot be processed, not that the assignment moved to another runner. Exiting there would delete .runner and .credentials (ConfigurationManager.cs:496) on a runner that could still serve, and run-helper.sh reads exit 0 as final — the host would never come back, and could not re-register if it did.

skipSessionDeletion = true is set before returning: the registration is already consumed, so DeleteSessionAsync would block up to 30 s (MessageListener.cs:206) on a call the service is bound to reject. This mirrors the registration-gone handler at line 838. It also removes a failure mode — runOnce (line 331) and settings.Ephemeral are read from different objects and can disagree after a server-pushed config migration, in which case the catch (…) when (runOnce) at line 878 would not swallow the resulting 401 and the exception would escape the finally.

The exit also writes to the terminal. A trace-only exit shows the operator Removed .credentials, Removed .runner and a bare exit 0, indistinguishable from a completed job.

Why settings.Ephemeral and not runOnce. runOnce also covers --once and returnJobResultForHosted. --once is a client-side flag — the service does not consume the registration, so skip-and-retry stays correct there. #4540 scoped its fix the same way. A test row pins this.

Not covered here

Three adjacent gaps found while reviewing, each needing its own change:

• The legacy IActionsRunServer acquire path (empty RunServiceUrl, Runner.cs:713-722) has no handling at all; a lost assignment there escapes to Program.MainAsync and run.sh relaunches against a consumed registration.
• RunServiceHttpClient.cs:100-121 only maps the three typed exceptions when TryParseErrorBody succeeds; a 409 with an unparsable body still falls through to the generic catch and the old skip.
• Exiting Success on a hosted runner reports 0 rather than a translated job result, so lost assignments raise no failure-code spike. The pre-existing acknowledge path at line 703 has the same shape.

Tests

• TestEphemeralRunnerJobRequestMessageFromRunServiceExitsOnLostJobAssignment — a [Theory] over 404 and 409. Asserts the runner exits Success, never dispatches a job, never touches the acquire throttler, does not delete the session, and does delete the local config.
• TestRunnerJobRequestMessageFromRunServiceContinuesOnLostJobAssignment — the counterpart, over four rows: ephemeral + 422, persistent + 404, persistent + 409, and --once + 409. Each asserts the runner keeps listening, throttles once, drains the message, and keeps its local config.

Both are built on a shared arrange helper. Its message-loop mock parks on an empty queue instead of dequeuing, so a regression surfaces as the intended assertion rather than InvalidOperationException: Queue empty, and each theory row names its own TestHostContext so rows stop deleting each other's trace file.

Verified the tests fail against the regressions they guard:

# exit on all three statuses  -> the 422 row fails
# predicate widened to runOnce -> the --once row fails
# unpatched Runner.cs          -> both exit rows fail

./dev.sh build is clean and the full ./dev.sh test suite passes, 1131/1131.

🤖 Generated with Claude Code