fix: only send reasoning.effort to models that support it

[shehabyasser-scale]

Why this is now load-bearing, not defensive

This started as cleanup for a 400 seen during a local gpt-4o swap. It is now a prerequisite, because gpt-4o is the only usable eval model on the configured Azure resource.

Probing the endpoint directly, only two deployments answer: gpt-5.3-codex and gpt-4o. Everything else, including the committed gpt-5.4-mini-2026-03-17 that four of the five benchmarks point at, returns 404 DeploymentNotFound (details in #58). So moving swe-atlas-qna to gpt-4o is the only way to get it running at all, and without this guard every one of its agent calls trades a 404 for:

400 unsupported_parameter: 'reasoning.effort' is not supported with this model

The change

Each of the five bench agents built an unconditional "reasoning": {"effort": ...} into its Responses request. Now it is sent only when the model supports it:

_model = self._api_model.lower()
if _model.startswith(("gpt-5", "o1", "o3", "o4")) or "codex" in _model:
    request["reasoning"] = {"effort": "high"}

The predicate runs on _api_model, which is model_name.removeprefix("openai/") (e.g. atlas_agent/agent.py:80), so the openai/ prefix carried in build.yaml never defeats the startswith.

Both branches matter and both are covered by the run below:

• gpt-4o → no reasoning key → the 400 cannot occur.
• gpt-5.3-codex (producer) → matches on "codex" → still receives effort. gpt-5.4-mini-2026-03-17 → matches on gpt-5. So no reasoning model silently loses its effort setting.

Verification

Live combined run in flight on swe-atlas-qna with a gpt-4o agent, on a branch carrying this plus #51/#52/#53. The check is session/artifacts/inference/requests/*.jsonl: zero records with status 400 / unsupported_parameter, against 322/322 upstream failures in the last run. Posting the counts here before merge.

Update: rebased on the refreshed base, swe-bench-pro hunk dropped

The base branch (#50) now applies this exact capability gate to the swe-bench-pro agent itself, as part of fixing five seed-agent defects there. That file therefore conflicted.

Resolved by merging the refreshed base and taking its version of swebench_pro_agent/agent.py verbatim. The gate is still applied to swe-bench-pro, just by the base commit rather than by this PR, so the merged result is unchanged and this PR's diff drops from six files to five. The content changes to the five surviving agents are byte-identical to what was reviewed.

This also resolves Greptile's note about swe-bench-pro inlining the predicate: the base's version uses the shared _is_reasoning_model helper.

Verified: all five remaining hunks apply cleanly to the current pr3-harness-bench tip.

Note on officeqa, do not collapse the two guards

A hand-reapplied copy of this gate currently running in a local worktree collapses both settings under one capability check:

if _is_reasoning_model(self._api_model):
    kwargs["reasoning_effort"] = "medium"
    kwargs["parallel_tool_calls"] = False   # regression

That is a behavior regression and this PR deliberately does not do it. parallel_tool_calls is a separate axis: Fireworks-served models reject it but gpt-4o supports it, so it stays a provider check while reasoning_effort moves to a capability check. Collapsing them silently drops parallel_tool_calls=False for gpt-4o.

Overlap with #61

#61 (browsecomp-plus truncated-answer fix) also touches this agent's _completion_kwargs and necessarily carries the same gate, because it parameterizes the same line. Whichever lands second needs that hunk dropped.

Greptile Summary

Replaces the previous provider-based guard ("fireworks" not in model) with a capability-based predicate (_is_reasoning_model) that correctly identifies which models support reasoning.effort and max_completion_tokens. This fixes HTTP 400 errors when running benchmark agents against Azure gpt-4o, which rejects both parameters.

• Four agents (browsecomp-plus, officeqa, atlas, tau3) define an identical _is_reasoning_model helper and use it to gate both reasoning_effort and the max_tokens/max_completion_tokens key; gaia applies equivalent inline logic since it uses the Responses API (max_output_tokens is already correct there).
• officeqa cleanly separates the capability check for reasoning_effort from the provider check for parallel_tool_calls, with a comment explaining the distinction.

Confidence Score: 5/5

Safe to merge — the change is a targeted guard that prevents known 400 errors on Azure gpt-4o and does not alter behavior for any model the predicate already matches.

All five agents apply logically equivalent and correct capability detection. The predicate is consistent with the stated model inventory, and the max_tokens/max_completion_tokens switch keeps non-reasoning models from hitting parameter-rejection errors. No existing reasoning model silently loses its effort setting.

Files Needing Attention: gaia/agent.py duplicates the capability predicate inline rather than using the shared helper; worth unifying if the model list ever needs to change.

Important Files Changed

Filename	Overview
harness-engineering-bench/browsecomp-plus/baseline/target/src/browsecomp_plus_agent/agent.py	Adds _is_reasoning_model helper and uses it to gate reasoning_effort and select max_tokens vs max_completion_tokens. Logic is correct; helper is duplicated across 4 files.
harness-engineering-bench/gaia/baseline/target/src/gaia_agent/agent.py	Moves reasoning.effort behind the same capability predicate as other agents, but uses inline logic instead of the shared _is_reasoning_model helper — creating a silent maintenance divergence.
harness-engineering-bench/officeqa/baseline/target/src/officeqa_agent/agent.py	Correctly separates the reasoning_effort gate (capability-based) from the parallel_tool_calls gate (provider-based, Fireworks check retained). Clean split with a clear comment.
harness-engineering-bench/swe-atlas-qna/baseline/target/src/atlas_agent/agent.py	Uses _is_reasoning_model for both reasoning_effort and the max_tokens/max_completion_tokens key selection. Straightforward and consistent with browsecomp and officeqa.
harness-engineering-bench/tau3/baseline/target/src/tau3_agent/agent.py	Applies the same capability-based gating correctly, but _is_reasoning_model is called twice per iteration inside a MAX_TURNS=80 loop; hoisting the result above the loop would be cleaner.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Agent._completion_kwargs / request build] --> B{_is_reasoning_model?}
    B -- "model starts with gpt-5 / o1 / o3 / o4\nor contains 'codex'" --> C[Use max_completion_tokens]
    B -- else --> D[Use max_tokens]
    C --> E{Responses API?\ne.g. gaia}
    D --> E
    E -- Yes --> F[max_output_tokens already used\nno key switch needed]
    E -- No --> G[Set chosen token key in kwargs]
    G --> H{_is_reasoning_model?}
    F --> I{_is_reasoning_model?\ninline predicate}
    H -- Yes --> J[kwargs reasoning_effort = medium/high]
    H -- No --> K[No reasoning_effort]
    I -- Yes --> L[request reasoning = effort medium]
    I -- No --> M[No reasoning key]
    J --> N{fireworks in model?}
    K --> N
    N -- No --> O[kwargs parallel_tool_calls = False\nofficeqa only]
    N -- Yes --> P[Skip parallel_tool_calls]
    O --> Q[API call]
    P --> Q
    L --> Q
    M --> Q

Loading

_{Reviews (5): Last reviewed commit: "Merge the refreshed swe-bench-pro scaffo..." | Re-trigger Greptile}

[shehabyasser-scale]

Validated live, with a same-model A/B

Both halves of the guard are now confirmed against real runs, using the same agent model on both sides so the only variable is this patch.

Without the guard — GAIA, 2026-07-24, gpt-4o agent

runs/gaia-full/jobs/2026-07-24__18-09-28, request-log census:

1274 × (evaluation,   gpt-4o, 200)
 514 × (evaluation,   gpt-4o, 400)   <- unsupported_parameter: reasoning.effort
  79 × (evaluation,   gpt-4o, 500)
 296 × (finalization, gpt-4o, 200)
 201 × (finalization, gpt-4o, 400)   <- same
  12 × (finalization, gpt-4o, 500)
 152 × (producer, gpt-5.3-codex, 200)

715 HTTP 400s — roughly 30% of every inference call that run made, thrown away on a parameter the model does not accept.

With the guard — swe-atlas-qna, 2026-07-25, gpt-4o agent

runs/atlas-all-fixes/jobs/2026-07-25__08-58-16, same census:

284 × (evaluation, gpt-4o,        200)
147 × (producer,   gpt-5.3-codex, 200)

Zero 400s. Zero non-200 responses of any kind. usage.json agrees: evaluation: 284 requests / 0 upstream_errors / 245,524 tokens.

The negative case is confirmed too

The predicate must not quietly strip effort from models that want it. Two independent confirmations:

• Live: the 2026-07-25 07:33 run was already running guarded candidate code at gpt-5.4-mini-2026-03-17, and all 322 of its evaluation requests still carried reasoning: {effort: medium} — startswith("gpt-5") matches after the prefix strip.
• In this run: the producer is gpt-5.3-codex, which matches on "codex", and its 147 requests all succeeded.

So: non-reasoning models lose the parameter, reasoning models keep it.

One thing worth flagging for review

The guard is five copy-pasted lines in five separate bench agents, with no shared helper and no unit test — the evidence above is all end-to-end. That is a fair reviewer objection. I did not refactor it into a helper because these files are the candidate programs under optimization: the optimizer edits them, and a shared import outside the candidate repo would change what the candidate can see and modify. Worth a deliberate decision rather than an accident, though.

This is also no longer optional. Per #58, gpt-4o and gpt-5.3-codex are the only two deployments that answer on the configured Azure resource, so every benchmark that gets un-blocked has to run a non-reasoning agent, and each one 400s on every call without this.

[shehabyasser-scale]

Restacked and narrowed from five agents to two.

b7a5b80 added a gate to officeqa, tau3, swe-atlas-qna and browsecomp-plus, so three of my five hunks conflicted with it. I have dropped those three rather than rewriting them, and kept only gaia and swe-bench-pro, which have no gate at all. gaia still sends "reasoning": {"effort": "medium"} unconditionally at gaia_agent/agent.py:188.

One thing to flag rather than change unilaterally, since it is your code and a judgement call:

if "fireworks" not in self._api_model:
    kwargs["reasoning_effort"] = "medium"

That gate is provider-shaped, not capability-shaped. It correctly spares Fireworks-served open models, but it still sends reasoning_effort to any non-Fireworks model that is not a reasoning model, Azure gpt-4o being the case I measured. The capability form is what my remaining hunks use:

_model = self._api_model.lower()
if _model.startswith(("gpt-5", "o1", "o3", "o4")) or "codex" in _model:

which also excludes fireworks_ai/*, so it is a strict superset of the current behaviour. If you want, the clean merge is to keep your provider gate for parallel_tool_calls (gpt-4o does support that, so the capability gate would wrongly drop it) and use the capability gate for reasoning_effort alone. I did not make that change here because it is your gate and either reading is defensible.

Live measurement behind the PR, unchanged: gpt-4o without a guard on gaia burned 715 HTTP 400s of 2376 calls (514 evaluation + 201 finalization); gpt-4o with the guard on swe-atlas-qna logged 284 x (evaluation, gpt-4o, 200) and 147 x (producer, gpt-5.3-codex, 200), zero non-200s.

Related and separate, worth its own look: gaia is the one benchmark now pointed at a Fireworks model while still on the Responses API with a hosted web_search tool, parallel_tool_calls: False, and a bare AsyncOpenAI() with no max_retries=8. That is the combination 7d57386's message says does not translate via litellm.

Full suite matches base: 11 pre-existing failures, none new.

[shehabyasser-scale]

Restoring the three hunks I dropped, because the gap turned out to be load-bearing rather than stylistic. I said above that the provider gate "will bite whoever points this at OpenAI or Azure". It bit today, and it is worse than I described: it makes three of the five benchmarks unrunnable on any non-Fireworks upstream.

Measured against the configured Azure endpoint, running the swe-atlas-qna seed agent over its full 50-case held-out split:

target model	gate	result
gpt-5.3-codex	provider	50/50 BadRequestError, 400 The requested operation is unsupported
gpt-4o	provider	50/50 BadRequestError, 400 Unrecognized request argument supplied: reasoning_effort
gpt-4o	capability	0 BadRequestError, agent inference succeeded

Two separate things conspire, and the second is the one worth knowing:

1. gpt-4o is not Fireworks, so the guard sends it reasoning_effort, and it 400s.
2. gpt-5.3-codex cannot be the escape hatch, because it has no /chat/completions surface on this resource at all. Probed directly: 400 for every parameter combination, including a bare request with only max_completion_tokens. It answers on /responses only. Since 1b5ee0c / 7d57386 ported these three agents to Chat Completions, no reasoning-capable model on this endpoint can drive them.

So the intersection of "model this endpoint serves on Chat Completions" and "model the provider gate does not break" is empty. The capability gate is the only thing that makes swe-atlas-qna, tau3 and officeqa runnable here.

The change is a strict superset of current behaviour: fireworks_ai/* names do not match gpt-5/o1/o3/o4/codex either, so nothing changes for the default configuration.

officeqa needed care. It gates two arguments on the one condition. Only reasoning_effort moves to the capability check; parallel_tool_calls stays provider-gated, because that one really is a Fireworks-vs-OpenAI distinction and gpt-4o supports it. Collapsing both would have silently changed gpt-4o's tool-calling behaviour.

Verified after the change: all three files compile, and the follow-on run reached the verifier stage with zero BadRequestError, so the remaining failure there is a separate issue (the in-container rubric judge needs upstream credentials, which task_services_use_upstream supplies under vero and --ve supplies under raw harbor).

[varunursekar]

is this something we should include in all the target agent designs? the only task for which we are going to use a reasoning model as the target is gaia