Claude harness: failure classifiers match text inside tool results, so a run that read a document saying "not logged in" is never retried

[mwpastore]

Analysis

actions/setup/js/claude_harness.cjs classifies a failed attempt by running regexes over the whole of Claude Code's stdout (result.output, lines 493–506 on main at v0.88.7). In --output-format stream-json that stdout carries every tool result the model received, verbatim, in user events. Three classifiers match ordinary prose, and three of them are checked in handleFailure before the signal-termination rule that would otherwise retry as a fresh run:

Classifier	Pattern	Matched by	Rule (line)	Effect
AUTHENTICATION_FAILED_PATTERNS (harness_retry_guard.cjs:38)	/not logged in/i	a repository document the agent Read: "gh auth status reports not logged into any GitHub hosts"	attempt === 0 && isAuthenticationFailed (561)	stop, non-retryable
PERMISSION_DENIED_PATTERN (permission_denied_helpers.cjs:5, threshold 3)	permission denied|EACCES|EPERM	a note the agent Read, recording an EACCES from an unrelated upload step	hasNumerousPermissionDenied (571)	stop, "missing tool/permission issue"
RATE_LIMIT_ERROR_PATTERN (claude_harness.cjs:76)	/rate limit/i	a spec the agent Read ("rate limiting")	logged as isRateLimitError=true; changes the retry reason	misreport

Observed on three consecutive runs in mwpastore/bejazzler, each stopped by a different classifier on text from a file the agent had read. All three exited 143 on purpose: a Stop hook sends SIGTERM after the agent records a resume pointer, relying on the harness's documented behaviour for signal exits ("signal-style termination … will retry with fresh run"). The fourth run, with a file list screened against every pattern, retried as designed and completed on attempt 4.

# run 34544392806
attempt 1 failed: exitCode=143 isOverloadedError=false isRateLimitError=true isAuthenticationFailedError=true ...
attempt 1: authentication failed — not retrying (first-attempt auth failure is non-retryable)
done: exitCode=143

# run 34546395191
attempt 1 failed: exitCode=143 isOverloadedError=false isRateLimitError=false isAuthenticationFailedError=false ...
attempt 1: detected numerous permission-denied issues — not retrying (classified as missing tool/permission issue)
done: exitCode=143

# run 34547827539, files without the phrases
attempt 3: signal-style termination exitCode=143 (failure_reason=cancelled_or_timed_out) — will retry with fresh run (--continue disabled permanently)
success on attempt 4

(The first of the three, 34538322580, matched not logged in the same way and was also stopped.)

The comment above AUTHENTICATION_FAILED_PATTERNS names the intended sources: the stream-JSON error field and Claude Code's own "not logged in" message. Tool results are neither. The same class of false positive was fixed for the Copilot harness in #49792 ("agent-emitted tool output contained benign auth text such as gh auth login") by widening the post-result watchdog rescue; that rescue is Copilot-only and covers completed runs, so it does not reach a Claude run that exits by signal before its safe outputs are written.

Implementation Plan

1. Classify from the harness-visible error surface, not from tool results (actions/setup/js/claude_harness.cjs):

   • Add a helper, e.g. classifiableOutput(output), that walks output line by line, parses each line that starts with { as JSON, and drops lines whose type is "user" (tool results) — keeping result, system, assistant and every non-JSON line (harness log lines, plain-text errors) unchanged.
   • Feed classifiableOutput(result.output) to isOverloadedError, isRateLimitError, isAuthenticationFailedError, countPermissionDeniedIssues / hasNumerousPermissionDeniedIssues, and detectNonRetryableHarnessGuard. Leave hasClaudeSessionProgress and extractDeniedCommands on the full output; they read assistant/tool events on purpose.
   • Keep the patterns themselves as they are.

2. Let a signal exit win over the text classifiers (same file, handleFailure): evaluate isSignalTerminationExitCode(result.exitCode) || isCrashSignalExitCode(result.exitCode) before the attempt === 0 && isAuthenticationFailed and hasNumerousPermissionDenied rules, so that an exit of 137/143 is retried as a fresh run regardless of what the output contained. A process that was killed did not fail authentication.

3. Tests (actions/setup/js/claude_harness.test.cjs, alongside the existing isAuthenticationFailedError and isRateLimitError cases at lines 185–210):

   • classifiableOutput drops a {"type":"user",…,"tool_result":…} line containing not logged in and keeps a {"type":"result",…} line containing it.
   • isAuthenticationFailedError(classifiableOutput(x)) is false when not logged in appears only inside a tool result, true when it appears in a result line or plain text.
   • hasNumerousPermissionDeniedIssues(classifiableOutput(x)) is false when three EACCES sit inside tool results.
   • handleFailure with exitCode: 143 and output containing not logged in inside a tool result returns { action: "retry" } with the fresh-run reason.

4. Follow guidelines: make agent-finish before completing.

Related

hasNoopInSafeOutputs (line 527) is also checked before the exit code, so a noop emitted before a signal exit ends the run as "work complete". That is a separate ordering question; happy to file it on its own if wanted.

---

🤖 Drafted with Claude Code; investigated, reviewed, and submitted by a human.