Repository navigation
fix(session): end at the step cap with a toolless answer, and report empty task results - #1632
RaghwendraSingh wants to merge 1 commit into
Conversation
…empty task results - At an agent's `steps` cap the max-steps instruction was sent as a trailing assistant message with every tool still offered. The Altimate gateway rejected that request, so the session ended with no answer. It is now a user turn with no tools and toolChoice "none", the same toolless shape the answer echo, compaction and title requests use. - The cap counts model requests only. Loop iterations that ran a compaction or a subtask and continued were counted too, so capped agents stopped early. - A child task that ends on an error or without text no longer reaches the parent as an empty <task_result>: the result says why it stopped and carries the child's last written text when there is one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
|
Thanks for your contribution! This PR doesn't have a linked issue. All PRs must reference an existing issue. Please:
See CONTRIBUTING.md for details. |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
|
Thanks for updating your PR! It now meets our contributing guidelines. 👍 |
Gateway run: the step cap now ends with an answerFour short headless
The The |
sahrizvi
left a comment
There was a problem hiding this comment.
Approved. No critical or major finding is open, and the fix does what it claims on the main paths: the request at the cap is a trailing user message with no tools, only model requests count toward the cap, and an empty child result now says why.
Five minors are posted inline. None of them blocks:
lastTaskTextloses text after a compaction in the child: edge input, the error reason still reaches the parent.- A tool call at the cap keeps the loop going with no bound: edge input, and main loops the same way while actually running the tools.
- An errored child with a closed text block reads as an answer: pre-existing, safe in a follow-up.
- A request queued after the cap gets no tools: edge input, the model can still answer.
- Test gaps on those paths: test-only.
Tests at this head against its base: test/session, test/tool and test/agent show the same 2 failures on both sides (scratch-store tests), and the only difference in counts is this PR's 5 new tests. See the PR comment for nits, sibling PR notes and what was checked.
| for await (const message of MessageV2.stream(sessionID)) { | ||
| if (message.info.role === "user") break | ||
| if (message.info.role !== "assistant") continue | ||
| if (parentID && message.info.parentID !== parentID) continue |
There was a problem hiding this comment.
Minor: after a compaction in the child this loses all text, and the parent is told "It wrote no text".
After a compaction the newest user message is the compaction continuation. Post-compaction assistants carry parentID = continuation, so this filter skips them, and line 30 then breaks at that continuation, so text under the original prompt is never reached.
I ran it through the real prompt loop on a mock provider: the child writes "Found 3 candidates so far.", a usage-triggered compaction runs, and the next request returns 400. lastTaskText returns undefined, so emptyTaskStatus says "It wrote no text". The same run without the compaction returns the text. The PR says the result "carries the child's last written text if there is one".
Suggested fix: walk past user messages that carry a compaction part or are synthetic, stop only at the task's prompt id, and drop the parentID equality check.
Not blocking: it needs a compaction, then an error, with no later text. The error reason still reaches the parent, and the wording errs toward "unfinished".
| format.type === "text" && agent.name === "analyst" && committedAnswer !== undefined | ||
| // altimate_change start — at the step cap a text answer is produced with tools off. A | ||
| // json_schema request keeps its tools: its answer is the StructuredOutput tool call. | ||
| const textFinalStep = isLastStep && !echoCommittedAnswer && format.type !== "json_schema" |
There was a problem hiding this comment.
Minor: a tool call in answer to the toolless request keeps the loop going, with no bound.
If the provider returns a tool call anyway, the repair path maps it to invalid, the tool part errors with "Model tried to call unavailable tool 'invalid'", finish stays tool-calls, and the loop sends the same toolless request again. With steps: 3 on a mock that always returns a tool call, I saw 2 normal requests and then 10 capped requests, until the mock stopped. Repeating the identical call gives the same 12 requests, so the doom-loop guard does not end it.
Base also makes 12 requests, and there the tools really run, so this PR is an improvement, but the cap still does not end the run.
Suggested fix: after a textFinalStep request, break whatever the finish reason, and record an error if there is no text.
Not blocking: edge input (a provider returning tool calls with none declared), and main has the same loop in a worse form.
| const text = result?.parts.findLast((x) => x.type === "text")?.text ?? "" | ||
| // altimate_change start — a child that ends on an error or with no text must not reach the | ||
| // parent as an empty result, which reads as "done, nothing found". | ||
| const status = text.trim() || ctx.abort.aborted ? [] : await emptyTaskStatus(session.id, messageID, result) |
There was a problem hiding this comment.
Minor: an errored child with a closed text block still reads as an answer.
status depends only on text.trim(). With a scripted stream of text-start, text-delta, text-end, error, result.info.error is set and the parts hold "Let me check the remaining tables.". The parent gets that narration inside <task_result> with no mention of the error. On an OpenAI-compatible stream that dies before the text block closes, the text is blanked to "", so that case does get the new status. This needs a provider that closes blocks before the failure.
Suggested fix: emit the reason whenever result.info.error is set, and keep text below it.
Not blocking: pre-existing on main, and the PR's claim is about empty results, so this is safe in a follow-up.
| const maxSteps = agent.steps ?? Infinity | ||
| const isLastStep = step >= maxSteps | ||
| // altimate_change start — see `modelSteps` | ||
| modelSteps++ |
There was a problem hiding this comment.
Minor: a user request that joins the running loop after the cap gets no tools.
modelSteps lives for the generation and does not reset when a new request joins (where currentTurn changes). With steps: 2 and a request queued during the capped request, the follow-up at this head goes out with 0 tools and MAX_STEPS as the trailing user message, after the real question. At base it went out with all 94 tools and the assistant MAX_STEPS.
MAX_STEPS says "Tools are disabled until next user input", and a follow-up sent after idle gets a fresh count, so the outcome depends on timing. The cap text on that request is pre-existing; losing the tools is new.
Suggested fix: reset modelSteps where currentTurn moves to a new request.
Not blocking: edge input (a capped agent plus a follow-up queued in that window), and the model can still answer.
| } as never), | ||
| // Newest first, like the real stream: the failed final message, then an earlier one with text. | ||
| spyOn(MessageV2, "stream").mockImplementation(async function* () { | ||
| yield { info: { role: "assistant", parentID: childPromptID }, parts: [] } as never |
There was a problem hiding this comment.
Minor: the new tests miss the paths above, and this mock hides the compaction case.
The new tests do what they claim. Run against base, the 4 failing cases fail for the stated reasons (role assistant, 3 requests instead of 4, empty <task_result>), and the normal-answer case passes on both.
This mock gives every assistant the prompt's parentID, which real history does not do after a compaction, so lastTaskText never sees that shape. Not covered: a tool call at the cap, json_schema at the cap, a compaction followed by lastTaskText, a queued follow-up after the cap, and an error alongside text.
Not blocking: test-only.
|
Notes from the review that don't belong on a single diff line, plus the nits. Nits
Sibling PRs
Checked, no finding
Test runs (local, at this head and its base)
Not verified
|
Issue for this PR
Closes #1634
Type of change
What does this PR do?
An agent with a
stepscap is meant to end with a text answer. On the Altimate gateway it ended with no answer instead, and a capped subagent handed its parent an empty result.1. The request at the cap failed (
session/prompt.ts).At step N the loop appended
MAX_STEPSas a trailing assistant message and still sent every tool, with notoolChoice. On the gateway that request failed withUnknownError "Could not get a response from the agent"and 0 tokens, so the session ended without an answer. In our headless runs these were the only two gateway errors in about 1,000 requests, and the only two requests sent at the cap. The answer-echo path a few lines above already notes that a conversation must end on a user turn.Fix: at the cap the instruction is a user turn, sent with no tools and
toolChoice: "none", the same toolless shape the answer echo, compaction and title requests use;llm.tsthen flattens historical tool calls, which the gateway requires for a toolless request. Ajson_schemarequest keeps its tools at the cap, because its answer is theStructuredOutputtool call.2. The cap counted iterations, not requests (
session/prompt.ts).stepalso increments on loop iterations that run a compaction or a subtask andcontinue. A capped agent that compacted twice reached its cap several requests early. The cap now uses a separatemodelStepscounter that advances only on the normal model-call path.stepis unchanged for everything else that reads it (title, telemetry, plan-mode checks).3. An errored or silent child returned an empty result (
tool/task.ts,session/task-output.ts).The parent received only the child's last text part, so a child that failed at its cap produced
<task_result>\n\n</task_result>with statuscompleted, which reads as "done, nothing found". Now, when the child produced no final text, the result says why ("Task stopped with an error before giving a final answer (UnknownError: …)" or "Task ended without a final text answer") and carries the child's last written text if there is one. The<task_result>tag itself is unchanged, sincecli/cmd/run/tool.tsparses it.Not in this PR:
task_id. Making it persist per session would also cap interactive sessions for good after N steps in total, so that belongs with a per-run budget that counts child sessions.run --max-turns: passing the remaining turns into the loop would make the last allowed turn the toolless answer, but it changes the documented--max-turnscontract (N work turns, then an abort and one toolless finalization reply, exit 1) thatbudget-finalization.test.tspins, so it is left for a separate change.How did you verify your code works?
prompt.test.tscases (the step cap asks for a final text answer …), plain and with a usage-triggered compaction, on an OpenAI-compatible mock: the last request ends with a user message carryingMAXIMUM STEPS REACHEDand declares no tools; no earlier request carries it; with a compaction the agent still gets its full step count. Both fail onmain(roleassistant; one request short with the compaction).test/tool/task-empty-result.test.ts: an errored child with earlier text, a child with no text, and a normal answer. The first two fail onmain.bun test test/session test/tool test/agent: 2,131 pass, 0 fail.test/cli: the same 2--time-budgetfailures injson-output.test.tsasmain.budget-finalization.test.ts: 46/46. Anapply_patchcase timed out once under load and passes alone (27/27).script/check-tracker-leaks.tsandanalyze.ts --markers --strictpass.Cost side effects
None added. The toolless final request sends no tool schemas, so it does not reuse the cached tools prefix; it is one request per capped session, and it replaces a request that failed and a parent turn that continued without the child's findings.
Screenshots / recordings
N/A
Checklist
🤖 Generated with Claude Code