feat: make the SWE-bench-Pro baseline runnable for the vero optimizer

[shehabyasser-scale]

Adds a SWE-bench-Pro baseline under harness-engineering-bench/swe-bench-pro/, mirroring the gaia baseline layout, so the vero optimizer can hill-climb a patch-style SWE agent.

This is no longer a stub. The two blockers called out in the original draft (an unpinned task_source and empty partitions) are resolved, and the seed agent has been fixed to the point where it actually completes a trial.

Verification

A single-case smoke run against the real dataset, instance_ansible__ansible-0ea40e with a gpt-4o agent on Modal:

reward	1.0
required tests	16 / 16 PASSED
errored trials	0
retries	0

pytest tests/test_v05_benchmark_configs.py -k swe-bench-pro passes 3/3. The suite has 6 pre-existing failures on officeqa and browsecomp-plus (registry task_source must include an explicit version) that are present on this branch's base with or without this PR; they are fixed by newer commits on pr3-harness-bench that this branch has not absorbed yet.

Pinning the task source

SWE-bench-Pro is not a package dataset. It ships as swebenchpro in Harbor's default registry (731 instances over 11 projects), so the pin is a bare name@version, not an <org>/<name>@sha256:<digest>:

task_source: swebenchpro@1.0

Its tasks resolve to git-backed ids under laude-institute/harbor-datasets at commit c8e8f3fac7097accaacf261d74c3d6f441de45b1. Grading is fully offline (the task's own run_script.sh + parser.py write reward.txt), so no judge model and no verifier credentials are needed.

Consequences that shaped the config:

• expose_case_resources: false on the development partition. VeRO materializes case resources through PackageDatasetClient, which only accepts <org>/<name> refs, so a registry dataset cannot be exposed that way.
• scripts/partition_swe_bench_pro.py now reads the stratification key (repo) from each task's tests/config.json rather than [metadata].repository, because these task.toml files carry no [task].name and Harbor derives the canonical name from the task directory.
• task_agent_timeout_seconds 3600 to 3000, mirroring the task package's own [agent] timeout_sec, which makes the runner's --agent-timeout-multiplier exactly 1800/3000 = 0.6.

Partitions

Regenerated and stratified by repo: 146 / 292 / 293 (development / validation / test), replacing three empty files and a TOTAL_TASKS = 0 stub guard. 731 splits 20/40/40 as 146.2/292.4/292.4; the leftover goes by largest remainder and the .4/.4 tie breaks toward test, matching how officeqa (246 to 49/98/99) and swe-atlas-qna (124 to 25/49/50) resolved the same tie.

Five seed-agent defects

Each of these silently biases the measurement rather than failing loudly, so the optimizer would have hill-climbed against a broken baseline.

#	Defect	Effect
1	REPO_DIR was /app/repo	The images set WORKDIR /app and reset the checkout in place there. Every shell command ran in a directory that does not exist.
2	reasoning.effort sent unconditionally	Hard 400 on gpt-4o at turn 1, so not one trial could complete. Now gated on capability, matching the sibling agents.
3	Absolute path in _write_file	Path(logs_dir) / "staged" / "/app/x.py" discards the left side, so the write landed on the host root and killed the trial with OSError [Errno 30] Read-only file system: '/app'. Seen on 2 of the first 12 held-out trials. Paths are normalized repo-relative; traversal is refused.
4	raise RuntimeError on turn-budget exhaustion	Reward comes from the hidden suite run against whatever the agent left in the repo, so raising discarded real edits, errored the trial, forced a full harbor re-run (9 of the first 49 held-out trials) and scored a hard 0 for a fix that may have been correct. Now recorded in metadata and left for the verifier to grade, exactly as the "model stopped calling tools" branch already did.
5	Unhandled UnicodeDecodeError	Harbor decodes exec output as strict UTF-8. A model that cats or greps a binary killed the trial: 5 of the first 49 held-out trials. Now a tool error the model can react to.

Note on the base

This branch is behind pr3-harness-bench and GitHub currently reports it as conflicting. That predates this PR and is not addressed here; the branch needs a merge from its base before it can land.

How to run

cd vero
uv run vero harbor build --config ../harness-engineering-bench/swe-bench-pro/baseline/build.yaml \
  --output ../harness-engineering-bench/swe-bench-pro/baseline/compiled
cp ../harness-engineering-bench/swe-bench-pro/baseline/secrets.env.example \
   ../harness-engineering-bench/swe-bench-pro/baseline/secrets.env   # fill Azure OPENAI_*, MODAL_*, WANDB_*
uv run vero harbor run --config ../harness-engineering-bench/swe-bench-pro/baseline/build.yaml \
  --env-file ../harness-engineering-bench/swe-bench-pro/baseline/secrets.env \
  --environment modal --agent codex --model gpt-5.3-codex --yes -o ../runs/swe-bench-pro/jobs

--environment modal needs a Modal-reachable model endpoint (e.g. the Azure …openai.azure.com/openai/v1 endpoint), not a VPN-internal one.

🤖 Generated with Claude Code

Greptile Summary

This PR scaffolds a fully functional SWE-bench-Pro baseline under harness-engineering-bench/swe-bench-pro/, fixes five agent defects that silently biased measurements in the held-out run, and introduces a shared error taxonomy module (error_taxonomy.py) that gives all pipeline layers a single vocabulary for classifying failures.

• Agent fixes: resolves five production defects — wrong REPO_DIR, unconditional reasoning.effort crashing gpt-4o, absolute-path staging landing on the host root, raise RuntimeError on turn-budget exhaustion discarding edits, and unhandled UnicodeDecodeError on binary output.
• Partition & config: 731 tasks stratified by repo into 146/292/293 with a SHA-256-keyed deterministic split; task_source: swebenchpro@1.0 correctly pins the registry dataset; expose_case_resources: false justified by PackageDatasetClient limitation.
• Infrastructure: error_taxonomy.py consolidates error classification with UPSTREAM_MODEL_UNAVAILABLE specifically added to prevent misclassifying Azure 404s as informative task failures; harbor/cli.py adds pre-flight model probing and a _redact_trace_text path for session export.

Confidence Score: 4/5

Safe to merge with two minor quality issues that do not affect correctness or reward integrity.

The agent's path-traversal guard and error taxonomy precedence table are both sound. The two findings are: full response.output_text written verbatim to the trace on every turn, and when both git apply passes fail the model only sees the C1 retry error. Neither breaks the reward pipeline or introduces data loss.

Files Needing Attention: swebench_pro_agent/agent.py for trace logging volume and apply-patch error-message retention; test_agent.py would benefit from parametrised tests for _repo_relative boundary conditions.

Important Files Changed

Filename	Overview
harness-engineering-bench/swe-bench-pro/baseline/target/src/swebench_pro_agent/agent.py	Core SWE-bench-Pro agent: fixes all 5 documented defects; two minor issues — original git-apply error lost on C1 retry, and full response.output_text logged per turn.
harness-engineering-bench/swe-bench-pro/baseline/build.yaml	Build config correctly pins task_source as swebenchpro@1.0, sets expose_case_resources: false on both partitions with clear justification, and matches the task package's own agent timeout (3000s).
harness-engineering-bench/swe-bench-pro/scripts/partition_swe_bench_pro.py	Deterministic SHA-256-keyed stratified split; reads stratification key from tests/config.json correctly; TARGET_COUNTS sum to TOTAL_TASKS; --check mode enables CI verification.
vero/src/vero/evaluation/scoring/error_taxonomy.py	New single-source-of-truth error taxonomy with correct precedence ordering and a specific UPSTREAM_MODEL_UNAVAILABLE pattern that avoids false-positives on container load failures.
vero/src/vero/harbor/cli.py	New Harbor CLI module; preflight probing distinguishes route-level 404s from model-level 404s; _redact_trace_text covers Bearer tokens, env var values, and sk- keys.
harness-engineering-bench/swe-bench-pro/baseline/target/tests/test_agent.py	Tests cover happy-path submit, transient retry, and missing-model guard; missing coverage for _repo_relative path normalisation and traversal rejection.
vero/tests/test_v05_benchmark_configs.py	Adds swe-bench-pro to parametrised benchmark config tests; verifies gateway scoping, credential isolation, and model allow-list invariants.
vero/tests/test_v05_error_taxonomy.py	Comprehensive tests for the new error taxonomy: signal classification, case precedence, harness environment loss, and the swe-atlas-qna deployment-not-found regression case.
vero/src/vero/sandbox.py	Cosmetic-only change: removes two section-header comment blocks. No functional changes.

Sequence Diagram

sequenceDiagram
    participant VeRO as VeRO Optimizer
    participant CLI as harbor CLI
    participant GW as Inference Gateway
    participant Agent as SweBenchProAgent
    participant Env as Harbor Environment (Modal)
    participant Verifier as Task Verifier

    VeRO->>CLI: harbor run --config build.yaml
    CLI->>CLI: _preflight_models()
    CLI->>GW: POST /responses (1 token probe)
    GW-->>CLI: 200 OK
    CLI->>Env: uvx harbor run compiled task

    loop Each turn (max 50)
        Agent->>GW: responses.create(model, tools, input)
        GW->>GW: enforce producer scope budget
        GW-->>Agent: response (function_calls)
        alt tool call
            Agent->>Env: "environment.exec(command, cwd=/app)"
            Env-->>Agent: ExecResult
            Agent->>Agent: _trace(turn, tool, result)
        else submit
            Agent->>Agent: record metadata, break
        end
    end

    Agent->>Verifier: repo state in /app
    Verifier->>Verifier: run hidden test suite
    Verifier-->>VeRO: reward.txt (0.0-1.0)

Loading

Prompt To Fix All With AI

Fix the following 3 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 3
harness-engineering-bench/swe-bench-pro/baseline/target/src/swebench_pro_agent/agent.py:480-490
**Full response body logged per turn**

`response.output_text` is the model's complete text output and is serialised verbatim into the JSONL trace on every turn. Over a 50-turn session with a reasoning model this can easily run to hundreds of kilobytes per trial, which conflicts with the codebase's rule against logging full response bodies in production code. Consider logging a truncated summary (e.g. character count or first 200 chars) rather than the full text.

### Issue 2 of 3
harness-engineering-bench/swe-bench-pro/baseline/target/src/swebench_pro_agent/agent.py:415-427
**Original diagnostic discarded on C1 retry failure**

When both `git apply` attempts fail the function returns the `stderr` from the C1 retry only. The original attempt usually gives a more informative hunk-mismatch message (exact offsets, context lines), while the `-C1` output is often just "patch failed". A model trying to fix the patch only sees the less precise C1 error, making self-correction harder. Preserving the first attempt's `stderr` (e.g. under a separate key like `"original_stderr"`) would give the model the full picture.

### Issue 3 of 3
harness-engineering-bench/swe-bench-pro/baseline/target/tests/test_agent.py:1-10
**No tests for `_repo_relative` path-traversal guard**

`_repo_relative` is the sole barrier that prevents the model from writing files outside `/app` (defect #3 in the PR description). It handles absolute paths, `..` components, and traversal via `/app/../…`. The existing test suite exercises the happy path (submit, retry) but does not test any of these normalisation or rejection cases. A targeted parametrised test would be cheap to add and protects against regressions if the VeRO optimizer rewrites the tool dispatch loop.

_{Reviews (1): Last reviewed commit: "test: cover swe-bench-pro in the benchma..." | Re-trigger Greptile}

Greptile also left 2 inline comments on this PR.

Context used:

• Rule used - What: Never log full response bodies, request bodi... (source)

[shehabyasser-scale]

Rebased onto the current pr3-harness-bench. This is now a one-commit PR: 12 of the 13 commits were already patch-equivalent upstream (0072a5c -> 87d7fd9, badc42c -> 5766013, and so on), so git rebase dropped them as already-applied and replayed only the scaffold itself.

Added one commit on top, fixing a real defect the new invariants exposed. The config had:

model: openai/gpt-4o
producer:   allowed_models: ["gpt-5.3-codex", "openai/gpt-5.3-codex"]
evaluation: allowed_models: ["gpt-4o", "openai/gpt-4o"]

The dual-spelling lists were a workaround for a mismatch that should not exist. Agents strip only openai/ (removeprefix("openai/")) and the gateway matches exactly (model not in scope.allowed_models -> 403 model_denied), so model: openai/gpt-4o puts gpt-4o on the wire and the prefixed entry can never match. It also fails 20dd166's invariant that the evaluation scope allow exactly the target model, and hard-coding the producer list fails the params-override test. Now unprefixed on both sides with the producer templated as ["${optimizer_model:-gpt-5.3-codex}"], matching every other benchmark.

Two things a reviewer should know:

1. This config still cannot be swept by tests/test_v05_benchmark_configs.py. Its partitions are deliberately empty pending real task ids, so load_harbor_build_config rejects it before any invariant is checked. It should join BENCHMARKS there once the partitions and the task_source digest are pinned; I did not add it now because it would fail on load.

2. That test currently fails 6 of 16 on a clean checkout, independent of this PR. officeqa and browsecomp-plus both set task_source: ../tasks and tasks/ is gitignored in both, so validate_references raises registry task_source must include an explicit version. Verified on pr3-harness-bench at 20dd166: 10 passed, 6 failed. Dropping the skip guard in d131412 was the right call, but those two are now green only on a machine that has already run officeqa/scripts/vendor_tasks.sh and browsecomp-plus/scripts/build_tasks.py.

Full suite matches pr3-harness-bench exactly: 11 pre-existing failures, none new.

[greptile-apps]

Original diagnostic discarded on C1 retry failure

When both git apply attempts fail the function returns the stderr from the C1 retry only. The original attempt usually gives a more informative hunk-mismatch message (exact offsets, context lines), while the -C1 output is often just "patch failed". A model trying to fix the patch only sees the less precise C1 error, making self-correction harder. Preserving the first attempt's stderr (e.g. under a separate key like "original_stderr") would give the model the full picture.

Prompt To Fix With AI

This is a comment left during a code review.
Path: harness-engineering-bench/swe-bench-pro/baseline/target/src/swebench_pro_agent/agent.py
Line: 415-427

Comment:
**Original diagnostic discarded on C1 retry failure**

When both `git apply` attempts fail the function returns the `stderr` from the C1 retry only. The original attempt usually gives a more informative hunk-mismatch message (exact offsets, context lines), while the `-C1` output is often just "patch failed". A model trying to fix the patch only sees the less precise C1 error, making self-correction harder. Preserving the first attempt's `stderr` (e.g. under a separate key like `"original_stderr"`) would give the model the full picture.

How can I resolve this? If you propose a fix, please make it concise.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

No tests for _repo_relative path-traversal guard

_repo_relative is the sole barrier that prevents the model from writing files outside /app (defect #3 in the PR description). It handles absolute paths, .. components, and traversal via /app/../…. The existing test suite exercises the happy path (submit, retry) but does not test any of these normalisation or rejection cases. A targeted parametrised test would be cheap to add and protects against regressions if the VeRO optimizer rewrites the tool dispatch loop.

Prompt To Fix With AI

This is a comment left during a code review.
Path: harness-engineering-bench/swe-bench-pro/baseline/target/tests/test_agent.py
Line: 1-10

Comment:
**No tests for `_repo_relative` path-traversal guard**

`_repo_relative` is the sole barrier that prevents the model from writing files outside `/app` (defect #3 in the PR description). It handles absolute paths, `..` components, and traversal via `/app/../…`. The existing test suite exercises the happy path (submit, retry) but does not test any of these normalisation or rejection cases. A targeted parametrised test would be cheap to add and protects against regressions if the VeRO optimizer rewrites the tool dispatch loop.

How can I resolve this? If you propose a fix, please make it concise.