Skip to content

fix(agent-loop): one invariant for the five ways a turn dies silently (supersedes #869 #872 #873 #874 #876 #879) - #880

Open
leonsong09 wants to merge 6 commits into
lsdefine:mainfrom
leonsong09:fix/silent-truncation-umbrella
Open

leonsong09 wants to merge 6 commits into
lsdefine:mainfrom
leonsong09:fix/silent-truncation-umbrella

Conversation

@leonsong09

Copy link
Copy Markdown

Closes #863, #860, #866, #875, #877.

Why one PR

These five reports looked unrelated when I filed them, but they are one failure mode wearing five masks: the loop treats "nothing happened" as "the model finished". A reply that hit a wall — truncated mid-stream, empty, cut off before the tool call, or a tool call emitted as plain text — all end up in the same place: no error, no retry, task quietly over.

Splitting them into #869 / #872 / #873 / #874 / #876 / #879 made each diff small, but it also meant a reviewer had to hold five interlocking invariants in their head to judge any one of them. This branch merges all six in that order, and the tests for the merged result are the deliverable: every shape from #863 plus the retry paths play through one code state.

Merged cleanly, no conflicts. pytest tests/test_text_tool_call.py tests/test_stream_retry.py -q → 8 passed.

The invariant

A turn may only end without a tool call if the model produced usable output and nothing is left dangling.

_has_usable_output() and textual_tool_residue() are two halves of that sentence. Everything else follows.

What's in it

1. llmcore.py — pseudo-XML tool calls are reconstructed, never half-executed

_parse_text_tool_calls only understood a trailing tool_use JSON array plus <tool_use>/<tool_call> wrappers. Every spelling that upstreams actually emit fell through to do_no_tool, where no guard matched, so the turn ended silently.

shape before after
A <code_run><script>… task ends executed
B <parameter name="script"> task ends executed
C <file_read"> (stray quote) task ends executed with the right arg name
D <type>-wrapped parameters task ends retry
E ||DSML||invoke retry (tail heuristic) executed
F <function=code_run><parameter=script> task ends executed
G </code_run><code_run></code_run> task ends retry
H cut mid-stream task ends / executes half a script retry, never executed

H is the reason the guard sits inside _parse_text_tool_calls rather than in do_no_tool: once a truncated call becomes a real tool_call, the normal response path executes it and do_no_tool is never reached. Two cases pinned — the cut that leaves a syntax error, and the cut that happens to leave valid Python.

Argument names are parsed independently of the tool tag (<parameter name="k">, <parameter=k>, ||DSML|| prefix all share one expression), so shape C no longer produces dsml_parameter.

textual_tool_residue() returns unclosed / empty / orphan / invalid; do_no_tool retries on any of them.

This supersedes #853 (open since 2026-10-26, same root cause). The review on #863 by @aron-intframe measured three gaps in #853 — F still ending the task, H being executed, C getting the wrong key — and all three are closed here by construction rather than by a second pass.

2. ga.py — truncation markers only match at the tail

do_no_tool() fired on a marker appearing anywhere in the trailing window. So a reply that merely quoted [!!! 流异常中断 ...] — which happens whenever the model debugs this check, documents it, or writes a regression test for it — was discarded wholesale. The failure is self-triggering: the more thoroughly the model explains the mechanism, the more reliably it trips.

llmcore always appends these markers at the very end, so anchoring to $ is both sufficient and precise. Two markers that llmcore does emit were also missing from the check.

3. llmcore.py — never replay what the caller already saw

_stream_with_retry retried on Timeout / ConnectionError / ChunkedEncodingError regardless of whether it had already yielded, so the consumer received AAABBB three times. MixinSession did the same at channel level. Now: if streamed, don't retry — append the interruption marker (same convention as the existing generic branch) and leave the next call pointed at a different channel.

4. llmcore.py — empty / thinking-only replies are failures

The emptiness test was not e.value and not streamed, which only requires blocks to be non-empty. A response with a thinking block and no text and no tool_call is non-empty — so it was reported as success, and the user got a blank screen with the system insisting all was well. _has_usable_output() now requires non-blank text or a tool_use (a pure tool-call response still counts, so nothing is over-rejected).

5. llmcore.py — empty tool_use_id results keep their order

MockToolCall reconstructs from text with id='', and that branch prepends, so three results arrived as R3, R2, R1 — the opposite of the append-based path used when ids exist. Collected in order into orphan_tails, appended after the loop. Byte-identical when all ids are present — that path is untouched.

6. ga.py — code_run no longer sleeps a flat second

time.sleep(1) before joining the reader thread. Replaced with t.join(0.05), plus a process.wait(timeout=5) fallback — because the reader exiting does not mean the process was reaped, and poll() returns None, losing the exit code. That regression is in the diff because the first version of this patch had it.

Medians from 5,187 recorded response gaps: 1.000s exactly, 2,012 of them, code_run responsible for 1,912.

7. agentmain.py — --no-reply-wait

--task waits 600s for a follow-up round, which is right for supervised sessions and pure idle for schedulers. Report written at 20:41, process killed at 21:00 with rc=-999, three days running. Opt-in flag; default unchanged.

Scope

agentmain.py +9, ga.py +64/-8, llmcore.py +324, and two new test files (351 lines). No frontend, no channel config, no schedule changes.

Nothing here touches files outside the agent loop, so it can be reviewed and landed independently of the TUI perf series (#870 / #871, merged separately).

Verification

pytest tests/test_text_tool_call.py tests/test_stream_retry.py -q
8 passed

Both suites are offline and deterministic — test_stream_retry.py drives a local ThreadingHTTPServer fake SSE, no network.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment