fix: stop submitting truncated reasoning as the browsecomp-plus answer

.gitignore

@@ -15,6 +15,3 @@ wheels/
 secrets.env
 *.secrets.env
 !*.example
-
-# Local benchmark scoping notes (paper planning; not tracked)
-harness-engineering-bench/benchmark-scoping.md

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "browsecomp-plus-upstream"]
+	path = harness-engineering-bench/browsecomp-plus/upstream
+	url = https://github.com/texttron/BrowseComp-Plus.git

harness-engineering-bench/CONFIGURATION.md

@@ -0,0 +1,181 @@
+# Benchmark configuration
+
+Every benchmark compiles from its `*/baseline/build.yaml`, which is the source
+of truth. This page records the normalized settings shared across the suite,
+the per-benchmark values, and the conventions behind them, so a change to one
+benchmark can be checked against the others at a glance.
+
+## Shared settings (all benchmarks)
+
+- **Objective**: maximize `score`; downstream task accuracy is the only input
+  to selection and reward. Cost and latency are reported as evaluation metrics
+  (`inference_*_tokens`, `wall_seconds`, `mean/max_case_wall_seconds`) and in
+  `finalize.json`'s `reward_metrics`, but never scored.
+- **Partitions**: development (full disclosure, task resources exposed),
+  validation (aggregate-only, `min_aggregate_cases: 5`), test (held out;
+  single target, `reward_key: reward`, `failure_value: 0`, `max_attempts: 1`).
+- **Selection**: `reward_mode: submit` — the agent nominates its candidate;
+  auto-best over validation and then last-candidate are fallbacks only.
+  `baseline_floor: false` (a floor would gate on validation while the reward
+  is on test; opt-in only). `score_baseline: true`, `rescore_top_k: 3`,
+  `rescore_attempts: 1`.
+- **Budgets**: 100 runs and 4 full passes of case budget on each agent
+  partition (development and validation).
+- **Budget disclosure**: `disclose_budget: true` (the default; no benchmark
+  overrides it) — the agent sees remaining runs/cases and inference-scope
+  usage through `evals status` and `plan.json`, plus per-evaluation
+  `inference_*` token metrics. Setting `disclose_budget: false` is the
+  budget-blind ablation: enforcement is unchanged but all budget signal is
+  hidden from the agent, including a redaction of `inference_*` metrics from
+  agent-facing receipts and context (latency metrics stay visible).
+- **Per-trial token/latency accounting**: the trusted gateway meters each
+  evaluation's usage as an input/cached/output/total split
+  (`inference_input_tokens`, `inference_cached_input_tokens`,
+  `inference_output_tokens`, `inference_total_tokens`, `inference_requests`) in
+  `reward_metrics`; each trial's `result.json` also carries agent-self-reported
+  tokens (`agent_reported_*`) and its wall window. Cost is reported per case the
+  same way latency is, and because token and latency distributions are unbounded
+  and heavy-tailed (a few cases carry much of the total) each carries a **mean, a
+  median, and a max** — `mean/median/max_case_wall_seconds` and
+  `mean/median/max_case_agent_reported_{input,cached_input,output}_tokens`.
+  Accuracy, being bounded, keeps mean plus stddev. The trusted gateway figure is
+  metered per evaluation rather than per case, so it contributes the sum plus a
+  derived `mean_case_inference_*`; its median and max come from post-hoc
+  attribution (`scripts/per_trial_tokens.py`, which reports mean/median/max over
+  trials for every token and latency measure). All of these are budget signal and
+  are redacted from the agent under `disclose_budget: false`; latency is not. With
+  `inference_gateway.request_log_attribution: true` the gateway stamps every
+  request-log record with a `thread_id`, so `scripts/per_trial_tokens.py`
+  attributes gateway tokens to individual trials (trusted, versus a content-match
+  fallback when off) and rolls a run — or a grid of runs — up to a flat CSV.
+  Enabled on officeqa; extended across the suite as the grid rolls out. Because
+  each scope's target model is fixed, **dollar cost is a linear function of the
+  (input, cached, output) token triple** with a per-model rate vector — computed
+  downstream, deliberately not stored anywhere in the run.
+- **Target model**: `fireworks_ai/deepseek-v4-flash` by default (see the
+  per-benchmark table — a benchmark may pin a different evaluated model), fixed
+  per run by the gateway's evaluation scope (the `evaluation.allowed_models`
+  allow-list, so a candidate cannot swap it). The optimizer uses a separate
+  producer scope bound to `${optimizer_model:-openai/gpt-5.4}`. The gateway
+  matches the requested model against the allow-list as an exact string, so the
+  `-m` the outer trial is launched with has to be spelled the same way as this
+  default (or as whatever `--param optimizer_model=` overrides it with);
+  the router resolves both `gpt-5.4` and `openai/gpt-5.4`, so the prefix is a
+  convention rather than a requirement. deepseek-v4-flash was
+  chosen over gpt-oss-120b and gpt-5.4-mini from a 10-trial per-benchmark probe:
+  it matches or beats both on tau3 (0.875) and is ~2–3× gpt-oss on the
+  grounded-reasoning benchmarks (officeqa/browsecomp 0.60) at roughly gpt-oss
+  cost — far cheaper than mini. `swe-atlas-qna` is pinned to
+  `fireworks_ai/gpt-oss-120b`, the one benchmark where deepseek is weaker
+  (0.30 vs 0.59 mean rubric) and gpt-oss is both cheaper and stronger. `gaia` is
+  pinned to `gpt-5.4-mini` for a different reason: it is the one multimodal
+  benchmark, 5 of its 66 held-out tasks send image inputs, and deepseek-v4-flash
+  rejects those outright (`This model does not support image inputs`), capping
+  achievable reward near 0.92 and disguising the shortfall as agent failure.
+- **Execution**: `harbor[modal]==0.20.0`, python 3.12, `n_attempts: 1`,
+  `max_retries: 1`, 3 infrastructure attempts at 5s, `aggregate_attempts:
+  best`, `max_concurrency: 8`, `error_rate_threshold: 0.1`,
+  `feedback_transcripts: true` with `feedback_max_bytes: 16000`,
+  `environment_name: ${inner_env:-modal}` (pass `--param inner_env=docker`
+  for local shakedowns).
+- **Telemetry**: W&B project `vero-<benchmark>` with trace uploads; inner
+  sandboxes grouped under the dedicated `harness-engineering-bench` Modal app
+  with a 1h idle timeout; the gateway records a per-request log.
+
+## Per-benchmark values
+
+| | gaia | officeqa | swe-atlas-qna | tau3 | browsecomp-plus |
+|---|---|---|---|---|---|
+| target model | gpt-5.4-mini ◇ | deepseek-v4-flash | gpt-oss-120b | deepseek-v4-flash | deepseek-v4-flash |
+| held-out baseline (K=3) ◆ | 0.574 ±0.010 | 0.360 ±0.042 | 0.097 ±0.011 (agg 0.632) | 0.611 ±0.021 | 0.449 ±0.007 |
+| split dev/val/test | 33/66/66 | 49/98/99 | 25/49/50 | 75/150/150 | 33/66/66 |
+| dev budget (runs / cases) | 100 / 132 | 100 / 196 | 100 / 100 | 100 / 300 | 100 / 132 |
+| val budget (runs / cases) | 100 / 264 | 100 / 392 | 100 / 196 | 100 / 600 | 100 / 264 |
+| timeout_seconds (per eval) | 3600 | 7200 | 14400 | 14400 | 28800 |
+| case_timeout_seconds (enforced) † | 900 | 1200 | 1800 | 1200 | 2100 |
+| task_agent_timeout_seconds (declared) | 600 | 1800 | 10800 | 3600 | 3600 |
+| verifier_timeout_seconds ‖ | 10800 | 25200 | 28800 | 43200 | 64800 |
+| harness_user | harness | harness | null ‡ | null ‡ | null ‡ |
+| task_services_use_upstream | false | false | true (rubric judge) | true (user-sim + grader) | true (answer judge) |
+| task-specific extras | — | `--no-force-build` (prebuilt corpus image) | `keepalive` --ek (ENTRYPOINT images) | `TAU2_*` model pins | pinned 2.2 GB BM25 index |
+
+## Conventions
+
+- **Timeouts are per-phase, not one shared wall.** Harbor runs the optimizer
+  agent phase and the verifier (finalization) phase with independent clocks, so
+  a long search does not eat into finalization's budget and vice versa. The
+  **optimizer agent phase is unbounded** (vero sets no `[agent] timeout_sec`);
+  the search is governed by the gateway token budget, not wall time.
+- **`case_timeout_seconds` is the enforced per-case wall cap**;
+  `task_agent_timeout_seconds` is the task-declared agent timeout used only as
+  the rescale denominator (Harbor's per-case timeout = declared ×
+  `case_timeout/task_agent_timeout`). `case_timeout` may exceed
+  `task_agent_timeout`. Set both explicitly — omitting them silently applies the
+  180/600 defaults regardless of what the tasks declare.
+- **`verifier_timeout_seconds`** bounds only finalization (the held-out
+  evaluation of the selected candidate), sized for `n_attempts` × held-out +
+  rescore headroom (‖). A verifier timeout yields no reward (the score is lost,
+  only agent artifacts are salvaged), so it is sized generously.
+- **Case budgets** are 4× the partition size, i.e. four full passes.
+- **Optimizer `agent_env`** (currently on officeqa; propagate to all): inner
+  evals take 15–30 min, but Claude Code caps a single Bash call at
+  `BASH_MAX_TIMEOUT_MS` (default 600000=10min), which forces the agent into
+  `--detach` + background-poll + end-turn — and in headless `--print` mode,
+  ending the turn ends the run. Set `BASH_MAX_TIMEOUT_MS`/`BASH_DEFAULT_TIMEOUT_MS`
+  high and `ENABLE_BACKGROUND_TASKS`/`FORCE_AUTO_BACKGROUND_TASKS=0` so the agent
+  blocks on a whole eval in one call and stays in-loop.
+- **`infrastructure_max_attempts: 3`** applies only to trusted finalization
+  re-scores. For competitive (agent) evaluations, whole-sub-run infrastructure
+  retry is disabled and a within-trial transient-infra failure is scored at the
+  failure value rather than excluded — a candidate cannot inflate its mean by
+  emitting a timeout/connection error. Coverage gaps (no trial produced) and
+  gateway budget/auth exhaustion remain excluded/terminating for both.
+
+◆ Held-out baseline of the seed harness on the **test** partition, mean over
+K=3 independent rounds; ± is the stdev across the three round means. Pinned
+into each target's `baseline_reward` with `score_baseline: false`, so runs use
+this number instead of re-scoring the seed every finalization. swe-atlas's
+`reward` is a binary pass/fail over a rubric and sits near the floor (0.097);
+the continuous `agg_score` (0.632, sd 0.011) is the far more informative
+signal — a candidate `reward_key` switch, pending the verifier emitting
+`agg_score` as a selectable key. Measured with each benchmark's target model
+(deepseek-v4-flash; gpt-oss-120b on swe-atlas; gpt-5.4-mini on gaia): the three
+deepseek benchmarks logged zero exceptions over 945 trials, swe-atlas lost
+5/150 to gpt-oss 128k context overflow, and gaia lost 4/198 (infra) after the
+agent's reason/search-only-turn crash was fixed.
+
+◇ gaia is the exception to the deepseek-v4-flash default: it is multimodal and
+that model is text-only. Verified against the same litellm endpoint the gateway
+proxies to — gpt-5.4-mini returns 200 for every request shape the gaia agent
+sends (image input, hosted `web_search`, `reasoning.effort`,
+`parallel_tool_calls`), while deepseek-v4-flash returns
+`400 This model does not support image inputs`. Written unprefixed on purpose:
+each agent sends `model_name.removeprefix("openai/")`, so an `openai/`-prefixed
+name would be allow-listed in one form and requested in another and the gateway
+would deny it.
+
+† Sized from the **held-out baseline** per-case wall-time distributions (the seed
+target agent on each benchmark's target model), set at or above each benchmark's
+observed max: gaia p99≈608/max≈609 → 900; officeqa p99≈640/max≈1076 → 1200;
+swe-atlas p99≈1163/max≈1278 → 1800 (unchanged); tau3 p99≈643/max≈1122 → 1200;
+browsecomp p99≈1479/max≈1771 → 2100. This replaces the earlier codex-probe
+sizing, which was too tight for the real target agents — the prior caps
+(180/300/900) would have killed ~9/13/26% of gaia/officeqa/browsecomp candidate
+cases, scoring candidates far harsher than the leniently-measured baselines.
+
+‖ `verifier_timeout_seconds` sized to ~1.5× a with-rescore finalize
+(`n_attempts=3` × held-out eval + `rescore_top_k=3` × validation eval), for
+salvage headroom even though the validation rescore does not fire in the
+`submit` selection path. Because phases are independent, this covers
+finalization only — not the search.
+
+‡ Exception to the harness-isolation default: these tasks run LLM services
+(rubric judge, user-simulator/grader, or answer judge) inside their task
+containers, which
+cannot reach the compose-internal gateway, so the build hands the real
+upstream credential to the task environment via `task_services_use_upstream`.
+That credential path is incompatible with `harness_user` isolation (the key
+would sit in the isolated harness's environment), so these benchmarks run
+unisolated under a non-adversarial-optimizer assumption, with a post-run
+leakage audit. Restore isolation once task-service credentials are delivered
+off the harness env (scoped judge key or per-role egress isolation).

harness-engineering-bench/README.md

@@ -0,0 +1,38 @@
+# Harness engineering benchmarks
+
+`harness-engineering-bench` contains end-to-end benchmarks for automatically
+improving the harness of an agent or the code of a component used to build
+agents. Each leaf directory pairs one editable target program with one immutable
+Harbor dataset and compiles them into an outer Harbor optimization task.
+
+The benchmark definitions intentionally keep three boundaries visible:
+
+- `target/` is the program the optimization agent may edit.
+- `partitions/` pins the cases and the development/validation/test split.
+- `build.yaml` is trusted configuration: model, evaluator, access policy,
+  budgets, and final scoring.
+
+In each benchmark, the complete development tasks and attachments are mounted
+read-only for the optimization agent. Development evaluations expose per-case
+results and complete Harbor trial records, including exact failures and
+target-agent logs. Validation remains aggregate-only, and
+test is reachable only by the trusted final verifier.
+
+The paper-era benchmark stack remains available on the `paper/v1` branch and
+the `paper-v1` tag. New benchmarks should use this Harbor-native layout.
+
+## Benchmarks
+
+Promoted benchmarks live at the top level. Task sets still under review live in
+`candidates/`; we work through the list in the paper's `benchmark-scoping.md` and
+promote a task set to the top level once it is ready.
+
+### Promoted
+
+| Benchmark | Editable target | Dataset | Split |
+| --- | --- | --- | --- |
+| [GAIA baseline](gaia/baseline/) | Tool-using Responses API agent | Harbor `gaia/gaia` | 20% / 40% / 40% |
+| [OfficeQA baseline](officeqa/baseline/) | Grounded document-QA agent | Treasury Bulletin corpus | 20% / 40% / 40% |
+| [SWE-Atlas-QnA baseline](swe-atlas-qna/baseline/) | Codebase investigation agent | Harbor `scale-ai/swe-atlas-qna` | 20% / 40% / 40% |
+| [tau3 baseline](tau3/baseline/) | MCP customer-service agent | Harbor `sierra-research/tau3-bench` | 20% / 40% / 40% |
+| [BrowseComp-Plus baseline](browsecomp-plus/baseline/) | Fixed-corpus deep-research agent | Pinned local Harbor tasks | 20% / 40% / 40% |

harness-engineering-bench/browsecomp-plus/.gitignore

@@ -0,0 +1,6 @@
+# Generated from the obfuscated upstream dataset. These files contain decrypted
+# benchmark questions and answers and must not be committed.
+tasks/
+baseline/target/.venv/
+**/__pycache__/
+**/*.pyc

harness-engineering-bench/browsecomp-plus/README.md

@@ -0,0 +1,53 @@
+# BrowseComp-Plus
+
+This benchmark turns all 830 queries from
+[BrowseComp-Plus](https://github.com/texttron/BrowseComp-Plus) into local Harbor
+tasks. It evaluates a deep-research agent against the benchmark's fixed corpus
+and canonical BM25 index rather than the live web.
+
+## Pinned sources
+
+- Upstream repository submodule: `046949032b0328319cc9a02663a759ec601d9402`
+- Query dataset `Tevatron/browsecomp-plus`:
+  `144cff8e35b5eaef7e526346aa60774a9deb941f`
+- BM25 index `Tevatron/browsecomp-plus-indexes`:
+  `b3f37f70c33829eb09d04784a54277a31871fd63`
+
+The submodule is the Git pointer requested for the integration. The generator
+refuses to run if it is checked out at another commit. Hugging Face revisions
+are full immutable commit ids as well.
+
+## Generate Harbor tasks
+
+Initialize the submodule and run the builder from this directory:
+
+```bash
+git submodule update --init harness-engineering-bench/browsecomp-plus/upstream
+
+cd harness-engineering-bench/browsecomp-plus
+uv run --no-project --python 3.12 --with datasets==4.0.0 -- \
+  python scripts/build_tasks.py
+```
+
+The builder downloads and decrypts the pinned query dataset using the pinned
+upstream implementation, then writes one complete Harbor task per query under
+the ignored `tasks/` directory. It also regenerates the committed deterministic
+166/332/332 development, validation, and test split. Use `--force` to replace
+an existing generated tree or `--check` to verify it byte-for-byte.
+
+The first Harbor image build downloads the pinned BM25 index (about 2.2 GB).
+Every task has an identical environment, so subsequent tasks reuse that image.
+
+## Scoring and trust boundary
+
+Answers use BrowseComp-Plus's required Explanation / Exact Answer / Confidence
+format. The verifier follows the official upstream OpenAI evaluator and its
+default `gpt-4.1` judge. The upstream primary leaderboard instead runs the same
+judge prompt with Qwen3-32B; this Harbor integration deliberately uses the
+repository's supported API evaluator so tasks do not require a local GPU judge.
+
+The judge runs inside the task environment and therefore receives the real
+upstream credential. As with the SWE-Atlas-QnA rubric judge and tau3 task-owned
+LLM services, this disables uid isolation and assumes a non-adversarial
+optimizer. The editable baseline itself exposes no live-web or shell tool and
+uses only the fixed local index.

harness-engineering-bench/browsecomp-plus/baseline/.gitignore

@@ -0,0 +1,3 @@
+compiled/
+target/.venv/
+target/.pytest_cache/

harness-engineering-bench/browsecomp-plus/baseline/README.md

@@ -0,0 +1,20 @@
+# BrowseComp-Plus baseline
+
+This editable target is a Responses API deep-research agent with three tools:
+search the pinned BM25 index, open a document, and submit a formatted response.
+The optimization agent may change its prompts, control flow, tool use, or
+dependencies, but not the dataset, index, split, evaluated model, or verifier.
+
+Build the generated tasks first as described in the parent
+[`README.md`](../README.md), then compile from the repository root:
+
+```bash
+cd vero
+VERO_SKIP_SECRET_CHECK=1 uv run vero harbor build \
+  --config ../harness-engineering-bench/browsecomp-plus/baseline/build.yaml \
+  --output ../harness-engineering-bench/browsecomp-plus/baseline/compiled
+```
+
+For a real run, copy `secrets.env.example` to the ignored `secrets.env`, fill it
+in, and use `vero harbor run` in the same way as the other harness-engineering
+benchmarks.