feat(webAgent): diagnose why a generation returned no tool call - #637
Conversation
When the model returns zero tool calls, the agent recovers — the
ToolExecutionError is fed back and the next turn usually succeeds — but the
turn is spent. In eval runs this is a significant iteration-budget drain that
left no trace anywhere: 36% of iterations on affected tasks, after which the
agent aborts and confabulates an infrastructure cause ("browser connection
lost") that appears in no error event.
The two underlying causes need opposite fixes and are only distinguishable by
finishReason: "stop" means the model wrote prose instead of calling a tool,
"length" means it was truncated mid-call. Neither was recorded — finishReason
was awaited but unused here, and streamResult.text was never read at all.
Add streamResult.text to ProcessedAIResponse and emit
SYSTEM_DEBUG_NO_TOOL_CALL with finishReason, textLength, and a 500-char prose
preview before throwing, mirroring the existing SYSTEM_DEBUG_TOOL_DROP handling
of the >1-tool-call case.
The corrective message itself is deliberately unchanged. It reads oddly as a
label ("exactly one" when the condition is zero), but it is fed back to the
model as the instruction to retry, where it is accurate advice — and downstream
eval tooling classifies on that exact string. Diagnosis belongs in the event,
not in the model-facing text.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds diagnostics to distinguish why an action-generation turn produced zero tool calls in WebAgent, so eval runs can attribute failures correctly (prose response vs truncation) without changing the recovery behavior.
Changes:
- Extend the processed
streamTextresponse handling to capture the model’s generatedtextand includefinishReason. - Emit a new
SYSTEM_DEBUG_NO_TOOL_CALLevent withfinishReason,textLength, and a preview when no tool calls are returned. - Add/extend tests and update the webagent event schema/type exhaustiveness list.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| packages/core/src/webAgent.ts | Captures generation text/finishReason and emits SYSTEM_DEBUG_NO_TOOL_CALL before throwing on empty tool calls. |
| packages/core/src/events.ts | Adds a new event type + strongly-typed payload for the no-tool-call diagnostic event. |
| packages/core/test/webAgent.test.ts | Extends coverage for “no tools called” to assert the diagnostic event for both stop and length finish reasons. |
| packages/core/test/events.test.ts | Registers the new event type in the exhaustive event type list. |
| packages/core/schemas/webagent-event.json | Regenerates the JSON schema to include the new event type/data shape. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| streamResult.usage, | ||
| streamResult.warnings, | ||
| streamResult.providerMetadata, | ||
| streamResult.text, |
There was a problem hiding this comment.
Fixed in 9151748 — you're right that the code contradicted its own comment.
One correction on the stated cost, for the record: by the time we reach the Promise.all, fullStream has already been fully consumed by the for await loop above, so streamResult.text is an already-accumulated string rather than deferred work. The overhead was holding a large string in ProcessedAIResponse on every iteration, not extra materialization.
The fix is still right regardless, so I've made it: text now moves out of the Promise.all and resolves only when toolResults.length === 0, which is what the comment claimed all along. pnpm run check still green (950/229/103/273).
…all path Addresses review: the Pick comment said text was "only read when no tool was called" but the Promise.all awaited it unconditionally, so the code contradicted its own comment on every generation. Move it out of the Promise.all and resolve it only when toolResults is empty. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Problem
When the model returns zero tool calls,
webAgent.tsthrows aToolExecutionErrorthat gets fed back to the model. The agent recovers — the next turn usually succeeds — but the turn is spent.Auditing 82 archived eval tasks from the scheduled WebVoyagerX runs (
pilo-cli+tabstack-stage, every 5th run since 2026-06-13):AAMAMAMA). The agent recovers each time and then runs out of budget.pilo-clitasks vs 0/30tabstack-stage.The two underlying causes need opposite fixes and are only distinguishable by
finishReason—"stop"means the model wrote prose instead of calling a tool,"length"means it was truncated mid-call. Neither was recorded:finishReasonwas awaited but unused on this path, andstreamResult.textwas never read at all.Change
Diagnostics only — no behaviour change to the recovery path.
"text"toProcessedAIResponseand thePromise.allSYSTEM_DEBUG_NO_TOOL_CALLwithfinishReason,textLength, and a 500-char prose preview before throwing, mirroring how the existingSYSTEM_DEBUG_TOOL_DROPhandles the >1-tool-call casefinishReason+textLengthin the existingconsole.errorOn the unchanged message
I deliberately left
"You must use exactly one tool. Please use one of the available tools."alone, even though the trigger is zero tool calls, not more than one.It reads oddly as a label, but it is fed back to the model as the instruction to retry — where telling a model that returned no call to use exactly one is accurate advice. It is also the exact string downstream eval tooling classifies on. Changing agent-facing text and the classifier contract in a diagnostics PR seemed like the wrong trade. The diagnosis belongs in the event.
Why diagnostics before a retry
A retry on empty
toolResultsis the obvious fix, but it's premature: if this is truncation (finishReason: "length"), retrying the same oversized request likely reproduces it, and the real fix is budget. A day of scheduled runs with this event will say which case dominates.Testing
should handle missing tool resultsto assert the new event,finishReason: "stop",textLength, and previewfinishReason: "length") with a recovery turnevents.test.tsenum listschemas/webagent-event.json(check:schemasgate)pnpm run checkgreen: 950 core / 229 cli / 103 server / 273 extension🤖 Generated with Claude Code