fix: bundle harness eval flat-zero fixes (taxonomy + reasoning.effort + tau3 retries)

[shehabyasser-scale]

What

A single branch stacking the three harness eval fixes so both benchmarks can be re-run with all of them applied at once:

• fix: classify Modal sandbox / tests-dir loss as transient_infra #53 classify Modal sandbox / tests-dir loss as transient_infra (unmask the flat-zero)
• fix: only send reasoning.effort to models that support it #54 only send reasoning.effort to models that support it (kill the gpt-4o 400s)
• feat: let a build configure its outer optimizer trial's harbor flags #55 raise tau3 optimizer trial max_retries 1 -> 3 (survive transient Modal stream drops)

Commits are cherry-picked from those PRs (disjoint files, no conflicts), based on feat/swe-bench-pro-baseline-scaffold.

Purpose

One branch to re-run tau3 and swe-atlas-qna end to end with the full fix set. The individual PRs (#53, #54, #55) remain open for granular review.

Caveats for the rerun

• fix: classify Modal sandbox / tests-dir loss as transient_infra #53 unmasks the swe-atlas-qna environment failures; it does not fix the underlying Modal sandbox / tests-dir provisioning (tracked in swe-atlas-qna: eval cases fail on Modal sandbox / tests-dir provisioning (flat-zero) #51). Expect swe-atlas-qna to now report an invalid aggregate (high error_rate) rather than a fake 0.0, until swe-atlas-qna: eval cases fail on Modal sandbox / tests-dir provisioning (flat-zero) #51 is resolved.
• W&B trace logging to scaleai.wandb.io is still not landing (W&B Weave traces not landing on scaleai.wandb.io from Modal runs #52); the rerun's observability will be local-only until that is fixed.
• Model config: the committed build.yaml uses gpt-5.4-mini-2026-03-17. If the run targets gpt-4o, fix: only send reasoning.effort to models that support it #54 is what keeps it from 400ing.

🤖 Generated with Claude Code

Greptile Summary

This PR stacks three harness eval fixes from PRs #53, #54, and #55 to enable a clean combined re-run of tau3 and swe-atlas-qna. The individual changes are well-scoped and address real problems (a flat-zero masking bug, HTTP 400s from unconditional reasoning.effort, and transient Modal stream drops).

• Taxonomy fix (fix: classify Modal sandbox / tests-dir loss as transient_infra #53): Adds a new _SIGNAL_PATTERNS entry to classify Modal sandbox/container/tests-dir failures as TRANSIENT_INFRA, with clear tests covering the known failure messages.
• Reasoning guard (fix: only send reasoning.effort to models that support it #54): Removes the always-on reasoning.effort field from all five agent request builders and gates it on model-name prefix detection, preventing HTTP 400s from non-reasoning models like gpt-4o.
• Retry bump (feat: let a build configure its outer optimizer trial's harbor flags #55): Raises max_retries from 1 to 3 in tau3/baseline/build.yaml to survive transient Modal stream drops.

Confidence Score: 4/5

Safe to merge for the re-run; the three fixes are independently correct and the taxonomy change ships with good test coverage.

The taxonomy addition resolves the flat-zero masking problem with clear tests, the reasoning guard correctly prevents HTTP 400s on non-reasoning models, and the retry bump is a straightforward config change. Two minor concerns exist: the bare sandbox keyword in the new regex is the only term in the entry not anchored to Modal-specific context, risking occasional misclassification of task failures in non-Modal harnesses; and the model-detection block is copied verbatim into five agent files, making future model-list updates error-prone. Neither concern blocks the intended re-run.

Files Needing Attention: vero/src/vero/evaluation/error_taxonomy.py — review the sandbox keyword breadth before this taxonomy is used with harnesses that don't run in Modal.

Important Files Changed

Filename	Overview
vero/src/vero/evaluation/error_taxonomy.py	New SIGNAL_PATTERNS entry classifies Modal sandbox/container/tests-dir errors as TRANSIENT_INFRA. The bare sandbox keyword is the broadest term and could match non-infrastructure errors in other harnesses, but the risk is low in practice.
vero/tests/test_v05_error_taxonomy.py	Two new test functions cover the added taxonomy patterns with concrete Modal error message fixtures and a regression guard for the flat-zero scenario; coverage is thorough.
harness-engineering-bench/tau3/baseline/target/src/tau3_agent/agent.py	Removes always-on reasoning.effort and adds a model-name prefix guard. Same pattern duplicated across all five agent files; logic is correct for the listed prefixes.
harness-engineering-bench/gaia/baseline/target/src/gaia_agent/agent.py	Identical reasoning-guard fix as tau3_agent; no additional issues.
harness-engineering-bench/officeqa/baseline/target/src/officeqa_agent/agent.py	Identical reasoning-guard fix; no additional issues.
harness-engineering-bench/swe-atlas-qna/baseline/target/src/atlas_agent/agent.py	Identical reasoning-guard fix but with effort: high (versus medium in the others); correct per the original value for this agent.
harness-engineering-bench/swe-bench-pro/baseline/target/src/swebench_pro_agent/agent.py	Identical reasoning-guard fix with effort: high; no additional issues.
harness-engineering-bench/tau3/baseline/build.yaml	max_retries raised from 1 to 3; straightforward configuration change to survive transient Modal stream drops.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Agent receives model name] --> B{Reasoning-capable model?\ngpt-5 / o1 / o3 / o4 / codex}
    B -- Yes --> C[Add reasoning.effort to request]
    B -- No --> D[Omit reasoning.effort\ne.g. gpt-4o avoids HTTP 400]
    C --> E[client.responses.create]
    D --> E

    F[Exception signal from harness run] --> G[classify_signal]
    G --> H{Budget pattern?}
    H -- Yes --> I[INFERENCE_BUDGET_EXHAUSTED\nTerminating]
    H -- No --> J{Auth pattern?}
    J -- Yes --> K[AUTH_FAILURE\nTerminating]
    J -- No --> L{Transient pattern?\nrate-limit OR sandbox/streamterminated\naddtestsdirerror/fetchspec added PR53}
    L -- Yes --> M[TRANSIENT_INFRA\nRetryable - excluded from aggregate]
    L -- No --> N[TASK_FAILURE\nInformative sample - scored at 0]

Loading

Prompt To Fix All With AI

Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
vero/src/vero/evaluation/error_taxonomy.py:170-176
**Bare `sandbox` term is over-broad in a global taxonomy**

`sandbox` alone will match any exception message or type name that contains the word across all harnesses, not just Modal-specific ones. If a task in gaia or officeqa involves agent code that raises an error from a library with "sandbox" in its name or message (e.g., a Python security sandbox, a custom execution environment, any tool named `*Sandbox*`), the case would silently be reclassified from `TASK_FAILURE` to `TRANSIENT_INFRA` — excluded from the aggregate and counted toward invalidity instead of being scored. The other terms in this entry (`streamterminated`, `addtestsdirerror`, `loading.?container`, `fetchspec`) are Modal-specific enough to be safe. Consider anchoring the bare `sandbox` match more tightly, e.g. `modal.*sandbox|sandbox.*unavailable|sandbox.*not.?found`, to reduce false-positive risk for harnesses that don't use Modal sandboxes.

### Issue 2 of 2
harness-engineering-bench/tau3/baseline/target/src/tau3_agent/agent.py:301-304
**Model-detection logic duplicated across five agents**

The identical three-line block (`_model = …`, `startswith(…)`, `request["reasoning"] = …`) is copy-pasted into `gaia_agent`, `officeqa_agent`, `atlas_agent`, `swebench_pro_agent`, and `tau3_agent`. When the set of reasoning-capable models changes (e.g., a new `o5` or `gpt-6` prefix), all five files must be updated in sync. A shared helper or constant in a common `base_agent` module would make this a single-point update.

_{Reviews (1): Last reviewed commit: "fix: raise tau3 optimizer trial max_retr..." | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

[greptile-apps]

Bare sandbox term is over-broad in a global taxonomy

sandbox alone will match any exception message or type name that contains the word across all harnesses, not just Modal-specific ones. If a task in gaia or officeqa involves agent code that raises an error from a library with "sandbox" in its name or message (e.g., a Python security sandbox, a custom execution environment, any tool named *Sandbox*), the case would silently be reclassified from TASK_FAILURE to TRANSIENT_INFRA — excluded from the aggregate and counted toward invalidity instead of being scored. The other terms in this entry (streamterminated, addtestsdirerror, loading.?container, fetchspec) are Modal-specific enough to be safe. Consider anchoring the bare sandbox match more tightly, e.g. modal.*sandbox|sandbox.*unavailable|sandbox.*not.?found, to reduce false-positive risk for harnesses that don't use Modal sandboxes.

Prompt To Fix With AI

This is a comment left during a code review.
Path: vero/src/vero/evaluation/error_taxonomy.py
Line: 170-176

Comment:
**Bare `sandbox` term is over-broad in a global taxonomy**

`sandbox` alone will match any exception message or type name that contains the word across all harnesses, not just Modal-specific ones. If a task in gaia or officeqa involves agent code that raises an error from a library with "sandbox" in its name or message (e.g., a Python security sandbox, a custom execution environment, any tool named `*Sandbox*`), the case would silently be reclassified from `TASK_FAILURE` to `TRANSIENT_INFRA` — excluded from the aggregate and counted toward invalidity instead of being scored. The other terms in this entry (`streamterminated`, `addtestsdirerror`, `loading.?container`, `fetchspec`) are Modal-specific enough to be safe. Consider anchoring the bare `sandbox` match more tightly, e.g. `modal.*sandbox|sandbox.*unavailable|sandbox.*not.?found`, to reduce false-positive risk for harnesses that don't use Modal sandboxes.

How can I resolve this? If you propose a fix, please make it concise.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[shehabyasser-scale]

This bundle is now stale — superseded by the PR set below. Recommend closing it once the individual PRs are reviewed.

It carries #53 + #54 + the original #55, and that third one has since been proven wrong (max_retries is the nested per-case retry, not the outer optimizer trial, which is where the stream actually drops). #55 has been re-scoped, and the investigation it enabled found two further root causes that were not in this bundle at all.

Current state, all based on feat/swe-bench-pro-baseline-scaffold:

PR	what	status
#53	classify Modal sandbox / tests-dir loss as transient_infra	validated live 2026-07-25
#54	send reasoning.effort only to models that support it	now load-bearing (gpt-4o is the only usable eval model); live A/B in flight
#55	optimizer_harbor_args → tau3 --ek modal_vm_runtime=true	re-scoped; live A/B in flight
#57	scheme-less WANDB_BASE_URL + record swallowed sink failures	root cause reproduced at $0
#58	swe-atlas-qna → gpt-4o, off the undeployed model	model half solid; infra half under test
#59	preflight allow-listed models before launching a trial	verified live against the endpoint
#60	new UPSTREAM_MODEL_UNAVAILABLE category (stacked on #53)	unit-tested against the run's literal strings

The two things this bundle was built without knowing:

1. gpt-5.4-mini-2026-03-17 is not deployed on the configured Azure resource. Only gpt-5.3-codex and gpt-4o answer. 145 of swe-atlas-qna's 469 cases died on 404 DeploymentNotFound and were recorded as candidate task failures. gaia, officeqa and tau3 still point at it.
2. WANDB_BASE_URL was missing its scheme, so wandb.init() raised before any network call and every run's reporting was silently disabled.

Neither was visible from a diff, and both were cheap to find once the exported session.tar.gz was read instead of result.json.

[shehabyasser-scale]

Closing as superseded. This was a rollup of #53 + #54 + a tau3 max_retries bump. #53 is restacked and open on its own; #54 is restacked and narrowed to the two agents that still lack a reasoning-effort gate; the max_retries bump was the wrong layer and was reverted out of #55. Nothing here is left unrepresented.