fix: stop submitting truncated reasoning as the browsecomp-plus answer

[shehabyasser-scale]

Stacked on pr3-harness-bench (#46). One file: the BrowseComp+ baseline agent.

The scoring bug

When a completion came back with finish_reason == "length", the agent submitted message.content verbatim as its final answer:

if not calls:
    if (message.content or "").strip():
        await self._submit(environment, message.content)

This target writes its chain-of-thought into ordinary content (it has no separate reasoning_content channel) and routinely runs past the token cap. So what got submitted was not an answer, it was 42k to 52k characters of truncated, mid-sentence reasoning. The judge scores that 0 every time, and the trial is indistinguishable from a genuine wrong answer.

Measured across the smoke runs carrying this change: 6 length-truncated turns over 3 trials, content length min 42,253 / median 46,793 / max 52,104 chars. The pre-fix baseline traces (198 trials over 3 rounds) contain no finish_reason field at all, which is exactly why this went unnoticed. The instrumentation added here is what surfaced it.

Truncated turns are now recoverable rather than final: the model is asked for only the three required labeled lines instead of having its unfinished thinking submitted on its behalf.

Supporting changes

All four target the same failure mode of losing a trial that had real retrieved evidence behind it.

Wall-clock deadline with a final-answer reserve. Turn count is not this agent's binding constraint. Most of a turn is sequential search.py invocations, each booting a fresh pyserini/Lucene JVM against the BM25 index (~13s apiece) against ~3-9s of inference. Trials ran out of clock long before MAX_TURNS, so the existing turn-budget fallback never fired and harbor killed the trial mid-search with no answer recorded. There is now a deadline (BROWSECOMP_AGENT_BUDGET_SEC, default 900) with a reserve (BROWSECOMP_FINAL_RESERVE_SEC, default 120) so the best-effort answer is written before harbor's AgentTimeoutError. Harbor does not tell an agent its own timeout, hence the env var; keep it in step with the run's --agent-timeout-multiplier.

Tool calls past the deadline are stubbed, never skipped, because every tool_call still needs a matching tool message or the forced-final request is rejected for an unanswered call.

Completion shape validation. A gateway can answer 200 with a body the SDK does not turn into a completion, observed once as 'str' object has no attribute 'usage', which crashed the trial on an attribute access two lines later. _complete() now rejects anything without usable .choices in one place.

Empty turns nudge instead of raising. raise RuntimeError("model returned neither a response nor a tool call") became a bounded retry, giving up only after MAX_EMPTY_RETRIES consecutive empty turns. An assistant message carrying neither content nor tool calls is dropped rather than appended, since strict OpenAI-compatible servers reject it.

Bounded forced-final. The last-resort call uses max_retries=1 (the SDK default of 8 could outlast the reserve on its own) and ANSWER_MAX_TOKENS (so the model cannot spend the whole reserve on chain-of-thought and get truncated a second time), and degrades to a stub answer on failure instead of propagating.

Extra trace fields. finish_reason, per-turn llm_sec, left_sec, and a forced_final_reason of deadline vs turns, which lets a post-hoc conservative score treat exactly the trials the pre-fix agent would have lost as zeros.

Overlap with #54

This diff includes the _is_reasoning_model capability gate for this agent, which is also in #54. It is not separable here: the same _completion_kwargs line that #54 rewrites is the one this PR parameterizes to take max_tokens for the bounded forced-final call.

Whichever lands second will need that hunk dropped. Flagging rather than pre-emptively rewriting #54's scope.

Verification

pytest tests/test_v05_benchmark_configs.py on this branch: 10 passed, 6 skipped, no failures. Agent compiles clean. Exercised live in the smoke-browsecomp-plus-qwen fix1/fix2 runs cited above.

🤖 Generated with Claude Code

Greptile Summary

This PR fixes a scoring bug where finish_reason == \"length\" completions had their 42–52k character truncated chain-of-thought submitted as the final answer (scoring 0 every time), and adds complementary hardening so trials that ran out of wall-clock time now record a best-effort answer instead of being killed mid-search with no answer at all.

• Primary fix: when finish_reason == \"length\", content is no longer submitted; the model is nudged to produce only the three required labeled lines, with up to MAX_EMPTY_RETRIES attempts before falling through to the forced-final path.
• Deadline management: a AGENT_BUDGET_SEC wall-clock budget with a FINAL_RESERVE_SEC reserve breaks the search loop before harbor's AgentTimeoutError fires; tool calls past the deadline are stubbed (never skipped) to keep the OpenAI message history valid.
• Resilience: _complete() validates completion shape in one place, the forced-final call is bounded by max_retries=1 and ANSWER_MAX_TOKENS, and a bare except Exception degrades to a stub answer rather than losing the trial entirely. Extra trace fields (finish_reason, llm_sec, left_sec, forced_final_reason) make the failure modes observable post-hoc.

Confidence Score: 4/5

Safe to merge; the primary truncation fix is correct and the deadline logic is sound. The two observations are diagnostic quality issues in the trace output, not correctness bugs.

The core fix—blocking submission of length-truncated content and nudging for a proper answer—is implemented correctly. The deadline management, message-history invariants (stubbing rather than skipping tool calls, dropping empty assistant messages), and forced-final fallback all look sound. The two flagged items are both about trace completeness: stubbed tool calls emit no individual trace events, and the forced_final_reason field conflates turn-budget exhaustion with empty-retry exhaustion. Neither affects the submitted answer or trial scoring, but both limit the usefulness of the post-hoc forensics the PR explicitly aims to enable.

Files Needing Attention: The single changed file agent.py is worth a second read around the deadline-drain tool-loop (lines 377–434) and the forced-final block (lines 436–492) to confirm the trace gaps are acceptable for the audit use-case described in the PR.

Important Files Changed

Filename	Overview
harness-engineering-bench/browsecomp-plus/baseline/target/src/browsecomp_plus_agent/agent.py	Major rework of the main agent loop: adds wall-clock deadline management, fixes truncated-reasoning submissions, validates completion shape, and replaces the RuntimeError on empty turns with bounded retries. Logic is well-structured and the primary bug fix is correct; one diagnostic gap in deadline-drain tracing.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Start run] --> B[Initialize: messages, deadline, empty_turns=0, completed=False]
    B --> C{remaining <= FINAL_RESERVE_SEC?}
    C -- yes --> D[out_of_time=True, trace 'stopping: deadline', break]
    C -- no --> E[_complete: LLM call with timeout=remaining - FINAL_RESERVE_SEC/2]
    E --> F{choices valid?}
    F -- no --> G[raise RuntimeError: malformed completion]
    F -- yes --> H[Extract message, finish_reason, tool_calls]
    H --> I{tool_calls?}
    I -- no --> J{content present AND finish_reason != length?}
    J -- yes --> K[_submit answer, completed=True, break]
    J -- no --> L[empty_turns++ trace truncated/empty]
    L --> M{empty_turns >= MAX_EMPTY_RETRIES?}
    M -- yes --> N[break to forced-final]
    M -- no --> O[Append nudge user message, continue]
    I -- yes --> P[empty_turns = 0]
    P --> Q{for each tool_call}
    Q --> R{out_of_time OR remaining <= FINAL_RESERVE_SEC?}
    R -- yes --> S[stub result: search budget exhausted]
    R -- no --> T[Execute: search / get_document / submit_response]
    T --> U{submit_response called?}
    U -- yes --> V[submitted=True, completed=True, break]
    U -- no --> W[Append tool message]
    S --> W
    W --> Q
    Q -- done --> X{submitted?}
    X -- yes --> V
    X -- no --> Y{out_of_time?}
    Y -- yes --> N
    Y -- no --> C
    D --> N
    N --> Z{completed?}
    Z -- yes --> END[Update token counts, return]
    Z -- no --> AA[Append forced-final user message]
    AA --> BB[_complete: tools=False, max_retries=1, max_tokens=ANSWER_MAX_TOKENS]
    BB --> CC{Exception?}
    CC -- yes --> DD[trace error, use stub answer]
    CC -- no --> EE[answer = final content or stub]
    DD --> FF[_submit answer]
    EE --> FF
    FF --> END

Loading

Prompt To Fix All With AI

Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
harness-engineering-bench/browsecomp-plus/baseline/target/src/browsecomp_plus_agent/agent.py:377-414
**Stubbed tool calls produce no trace events during deadline drain**

The `_trace` call for individual tool results sits inside the `else` branch, so once `out_of_time` is True every subsequent tool call in the same turn is silently stubbed with `{"error": "search budget exhausted"}` without any corresponding trace event. The first stall is captured by the `"stopping": "deadline-midturn"` event, but if the model issued three tool calls at that point only the trigger is visible; the other two stubs are invisible in the JSONL trace. This makes post-hoc reconstruction of which searches were abandoned harder, which matters when using `forced_final_reason` to audit the trials the pre-fix agent would have lost.

### Issue 2 of 2
harness-engineering-bench/browsecomp-plus/baseline/target/src/browsecomp_plus_agent/agent.py:466-469
**`forced_final_reason` is `"turns"` for empty-retry exhaustion, not just turn-count exhaustion**

When the loop exits because `empty_turns >= MAX_EMPTY_RETRIES` (three consecutive truncated/empty responses) with `out_of_time=False`, the trace and `context.metadata` both record `"forced_final_reason": "turns"`. That conflates two distinct causes: MAX_TURNS exhaustion and MAX_EMPTY_RETRIES exhaustion. A conservative post-hoc scorer using `forced_final_reason` to identify pre-fix losses would not be able to distinguish a trial that hit turn 32 from one the model kept returning truncated responses on turn 5. Consider adding a third reason string (e.g. `"empty_retries"`) or including `"empty_turns": empty_turns` in the forced-final trace event.

_{Reviews (1): Last reviewed commit: "fix: stop submitting truncated reasoning..." | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

[greptile-apps]

forced_final_reason is "turns" for empty-retry exhaustion, not just turn-count exhaustion

When the loop exits because empty_turns >= MAX_EMPTY_RETRIES (three consecutive truncated/empty responses) with out_of_time=False, the trace and context.metadata both record "forced_final_reason": "turns". That conflates two distinct causes: MAX_TURNS exhaustion and MAX_EMPTY_RETRIES exhaustion. A conservative post-hoc scorer using forced_final_reason to identify pre-fix losses would not be able to distinguish a trial that hit turn 32 from one the model kept returning truncated responses on turn 5. Consider adding a third reason string (e.g. "empty_retries") or including "empty_turns": empty_turns in the forced-final trace event.

Prompt To Fix With AI

This is a comment left during a code review.
Path: harness-engineering-bench/browsecomp-plus/baseline/target/src/browsecomp_plus_agent/agent.py
Line: 466-469

Comment:
**`forced_final_reason` is `"turns"` for empty-retry exhaustion, not just turn-count exhaustion**

When the loop exits because `empty_turns >= MAX_EMPTY_RETRIES` (three consecutive truncated/empty responses) with `out_of_time=False`, the trace and `context.metadata` both record `"forced_final_reason": "turns"`. That conflates two distinct causes: MAX_TURNS exhaustion and MAX_EMPTY_RETRIES exhaustion. A conservative post-hoc scorer using `forced_final_reason` to identify pre-fix losses would not be able to distinguish a trial that hit turn 32 from one the model kept returning truncated responses on turn 5. Consider adding a third reason string (e.g. `"empty_retries"`) or including `"empty_turns": empty_turns` in the forced-final trace event.

How can I resolve this? If you propose a fix, please make it concise.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!