3/3 Add harness-engineering-bench (GAIA + benchmarks) - #46
Open
varunursekar wants to merge 49 commits into
Open
Conversation
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
varunursekar
force-pushed
the
pr3-harness-bench
branch
3 times, most recently
from
July 23, 2026 20:00
e0a1efb to
ab4c52f
Compare
varunursekar
force-pushed
the
pr3-harness-bench
branch
2 times, most recently
from
July 23, 2026 22:58
573c1ac to
c86351e
Compare
varunursekar
force-pushed
the
pr3-harness-bench
branch
3 times, most recently
from
July 24, 2026 03:49
745ed8e to
6788d76
Compare
varunursekar
force-pushed
the
pr3-harness-bench
branch
4 times, most recently
from
July 24, 2026 06:41
214f1aa to
a2fcd19
Compare
shehabyasser-scale
approved these changes
Jul 24, 2026
Collaborator
|
Approve with nits. The split is the part I care most about for experiment validity, and it checks out. GAIA split is genuinely leak-safe (verified):
One real defect (baseline quality, not integrity): the tau3 baseline drops conversation history after every plain-text turn. One thing to note for readers: every build.yaml ships |
varunursekar
force-pushed
the
pr3-harness-bench
branch
4 times, most recently
from
July 24, 2026 22:28
768de08 to
ad2c081
Compare
varunursekar
force-pushed
the
pr3-harness-bench
branch
from
July 25, 2026 19:52
741bfa6 to
95fed89
Compare
varunursekar
force-pushed
the
pr3-harness-bench
branch
from
July 26, 2026 04:26
51c152d to
ced19f9
Compare
Same port as swe-atlas: stateless chat messages + tool_calls, works across providers (validated cross-provider on gpt-5.4-mini and fireworks_ai/gpt-oss-120b). officeqa drops the hosted web_search tool (closed-corpus benchmark, and chat/completions has no hosted-tool equivalent) and moves read_image feedback to chat image_url content. tau3 keeps its host-side MCP loop; the stateless history also fixes the previous_response_id reset that dropped context after plain-text turns, and the MCP session id is now charset-validated before shell interpolation (greptile injection finding).
Mirrors the swe-atlas fix: at MAX_TURNS force one tool-free completion for the best answer and submit it, so a budget-exhausted case scores best-effort rather than being lost with no answer recorded.
A fifth harness-engineering benchmark: optimize a retrieval + reasoning agent over BrowseComp-Plus's fixed corpus and canonical BM25 index, graded by a semantic answer judge (gpt-4.1). Task packages are generated from the encrypted upstream dataset by scripts/build_tasks.py and are gitignored (they contain decrypted questions/answers); the upstream repo is pinned as a submodule and the 2.2 GB index is pulled at image-build time. Wired consistently with the normalized suite: 33/66/66 stratified split, 4-pass budgets, submit-mode selection, harness_user null + task_services_use_upstream for the in-container judge. Baseline agent uses Chat Completions (works across providers) and forces a best-effort answer at the turn cap. The task image installs a full JDK with JAVA_HOME resolved from it, so pyserini's Lucene BM25 searcher works (a headless JRE with JAVA_HOME unset left jnius unable to locate the JVM).
… succeeds pyserini 1.2.0 constructs an openai.OpenAI() client at import time for its unused OpenAI encoders; BM25/Lucene search never calls OpenAI, so set a placeholder key in search.py before the import.
The task.toml had no verifier.env, so the gpt-4.1 answer judge ran without OPENAI_API_KEY and scored every case 0 (Missing credentials). Template the upstream creds and judge model into the verifier phase.
…las) From a 10-trial per-benchmark probe of gpt-5.4-mini vs gpt-oss-120b vs deepseek-v4-flash: deepseek matches/beats both on tau3 (0.875) and is ~2-3x gpt-oss on the grounded-reasoning benchmarks (officeqa/browsecomp 0.60) at roughly gpt-oss cost and far below mini. It is weaker only on swe-atlas (0.30 vs gpt-oss 0.59 mean rubric), which is pinned to gpt-oss-120b — cheaper and stronger there. Updates each benchmark's target `model` and the evaluation-scope `allowed_models` (kept with the fireworks_ai/ prefix, since agents strip only the openai/ prefix and the gateway matches the sent model exactly). Producer scopes and tau3's user/assertion models are unchanged. Also hardens the swe-atlas atlas agent: a shell command emitting non-UTF-8 (binary) output raised UnicodeDecodeError and killed the whole trial; it now returns a tool error so the model can retry. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… relock - .gitignore: remove the paper-planning benchmark-scoping.md ignore - README.md: correct candidate paths (candidates/ -> top level), add browsecomp-plus, drop the stale nested table - vero/uv.lock: relock Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fireworks' shared-org rate limit surfaces as a RateLimitError inside the agent's OpenAI client; harbor only retries infra/env failures, not LLM 429s, so a burst lost trials. The SDK's default 2 retries were exhausted under sustained load. Bump to 8 so transient 429s back off in-place instead of failing the trial (durable fix is gateway-level retry, tracked separately). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…line Full held-out (test) pass of the seed harness, 3 independent rounds, under the new targets (deepseek-v4-flash; gpt-oss-120b on swe-atlas): swe-atlas 0.097 ±0.011 (agg_score 0.632) [gpt-oss-120b] tau3 0.611 ±0.021 [deepseek-v4-flash] officeqa 0.360 ±0.042 [deepseek-v4-flash] browsecomp 0.449 ±0.007 [deepseek-v4-flash] Each pinned into its target's baseline_reward with score_baseline: false, so finalization uses the number instead of re-scoring the seed every run. The three deepseek benchmarks logged zero exceptions over 945 trials (max_retries absorbed the shared-quota 429s); swe-atlas lost 5/150 to gpt-oss 128k overflow. CONFIGURATION.md documents the numbers, method, and the swe-atlas binary-vs- agg_score note. gaia deferred. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
It was the suite's only Family B task and the only one that bypassed the compiler: a hand-written task directory with no build.yaml, so it got none of the build-time validation the other five benchmarks get. Its optimizer was also unmetered -- no gateway container, raw upstream credentials in main, no model allow-list -- which is a fair-play hole in a benchmark whose whole point is controlled measurement. Its sidecar factory had additionally gone stale against SidecarEvaluationPolicy and would have died at startup. The nested-backend refactor now supports `evaluation_backend: command`, so a replacement Family B task can be built through the compiler and get the gateway for free. examples/harbor-circle-packing carries a worked version of exactly that shape, verified end to end. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
These tests moved here from pr2-add-vero, where they could only ever skip. They had been failing for a while against a pinned literal -- evaluation allowed_models == ["gpt-5.4-mini-2026-03-17"] -- that stopped matching when the benchmarks switched to fireworks-served targets. Rewrite the assertions so they cannot drift that way again: the evaluation scope must allow exactly [config.model], derived rather than pinned. That is the real invariant, and it now matches the build-time validator that rejects a model outside its scope's allow-list. Also drop the skip guard, so a missing harness-engineering-bench fails rather than passing vacuously, and extend coverage from three benchmarks to all five. 415 passed, 5 skipped -- the three long-standing failures are gone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six findings from the reviews on #45/#46 that survived checking against the current tree. Each was verified against the live litellm endpoint the gateway proxies to, not inferred from a commit message. gaia moves to gpt-5.4-mini. It is the one multimodal benchmark and deepseek-v4-flash is text-only: 5 of the 66 held-out tasks send images and get 400 "This model does not support image inputs", which caps achievable reward near 0.92 and reads as ordinary agent failure. gpt-5.4-mini returns 200 for every request shape the gaia agent sends -- image input, hosted web_search, reasoning.effort, parallel_tool_calls -- confirmed through the SDK path the agent actually uses. The name is unprefixed on purpose: every agent sends model_name.removeprefix("openai/"), so an openai/-prefixed name would be allow-listed in one form and requested in another and the gateway would deny it. The gaia held-out baseline now needs re-measuring. tau3 executes every tool call the model returns instead of only the first. deepseek-v4-flash does return two in a turn, and the usual guard is not available: litellm rejects parallel_tool_calls for fireworks_ai with UnsupportedParamsError, so the reviewer's suggested one-liner would have 400'd every tau3 call. Dropping calls[1:] silently skipped actions -- for a customer-service agent, a verified identity with no message sent. officeqa and swe-atlas-qna run their tool dispatch inside the try rather than an else, and catch KeyError/TypeError/ValueError alongside JSONDecodeError, matching browsecomp-plus. Neither tool schema is strict, so the model can return valid JSON missing a required key; that ended the trial instead of feeding the model an error. Widening the except alone would have caught nothing, since the lookups were in the else. gaia's client gets max_retries=8, the only one of the five without it. A within-trial transient failure scores at the failure value for competitive evaluations, so an unretried 429 cost a candidate a 0.0. tau3 quotes the MCP session id before interpolating it into a shell command, for the same reason the payload beside it is base64-encoded. per_trial_tokens buckets unattributed records under a name rather than None, which crashed sorted() as soon as one appeared beside an attributed record. Not actioned: the browsecomp-plus grader was reported as vulnerable to format-string injection from curly braces in the agent's response. str.format does not rescan substituted values, so there is nothing to fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
These tests fail 6 of 16 on a clean checkout. officeqa and browsecomp-plus point task_source at a gitignored tasks/ tree built by their own scripts/, and the loader only absolutizes a relative task_source when it exists, so on a fresh clone the path stays literal and is rejected as an unpinned registry reference. They pass here only because this machine has already run the vendor scripts. Dropping the blanket skip guard in d131412 was the right call -- that guard is what let the earlier version rot unnoticed -- but it was aimed at the wrong condition. A missing harness-engineering-bench still errors loudly; what is skipped now is one benchmark whose tasks are not fetched yet, and the skip names the directory and points at the script. That is a precondition of the machine, not a property of the config. Verified by moving both tasks/ directories aside: 10 passed, 6 skipped with the reason attached, where before it was 6 failed. Reported by shehabyasser-scale on #46. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gateway matches a requested model against the scope allow-list as an
exact string, so the producer default and the -m the outer trial is
launched with have to agree character for character. Both gpt-5.4 and
openai/gpt-5.4 resolve on the router, so this is a convention choice, not a
correctness fix: prefixed matches how every evaluation scope names its
model and how harbor's own uploader keeps the two spellings apart.
Note the resulting asymmetry in gaia, which is deliberate. Its producer is
openai/gpt-5.4 while its target is bare gpt-5.4-mini, because the baseline
agents send model_name.removeprefix("openai/") and the optimizer agent does
not. The allow-list has to match what each actually sends.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gaia still had the try/except/else shape the last round removed from officeqa and swe-atlas-qna: only json.loads sat inside the try, so every argument lookup and every tool call ran unguarded. It matters more here than it did there, because gaia is the one agent with read_image, which downloads a model-supplied path and then stats and reads it -- a hallucinated path fails in the filesystem, not in json.loads, and took the whole trial with it. gaia is also the benchmark about to be re-baselined on a multimodal model, so that path is now exercised rather than dormant. OSError joins the caught set in all four agents, not just gaia. Every one of them writes a file in _submit, and three download or upload through the environment, so the previous tuple of JSONDecodeError/KeyError/TypeError/ ValueError left the I/O half of the dispatch uncovered everywhere. A failed tool call should reach the model as a tool error it can react to, not end the trial. Reported by greptile on #46 (gaia); extended to the other three for the same reason rather than waiting to be told about each in turn. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gaia was never ported off the Responses API and lacked two robustness
guards the other four baseline agents have, causing ~27% of held-out trials
to error out (and silently inflate any reward computed over survivors):
- A turn where the model only reasoned, ran a hosted web_search, or was
truncated at max_output_tokens returns no function_call and no message.
The loop treated that as fatal ("neither answer nor tool call"); it now
carries the response chain forward with a nudge. MAX_TURNS bounds it.
- Exceeding MAX_TURNS raised RuntimeError; it now forces one final tool-free
answer so the case scores best-effort.
Held-out crash rate drops 27% -> 2% (189 OK / 5 forced / 4 infra of 198);
clean baseline_reward 0.5736 (sd 0.010 across 3 rounds).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Clean K=3 held-out pass with gpt-5.4-mini after the gaia agent crash fix: baseline_reward 0.5736 (sd 0.010 across rounds 0.576/0.561/0.585), 197/198 trials scored (189 OK, 5 forced-final, 4 infra). Pins the number and disables per-run baseline re-scoring, matching the other four benchmarks. CONFIGURATION.md baseline row + footnote updated (gaia now measured). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Set the test target's per-target n_attempts=3 + aggregate_attempts=mean so the selected candidate's held-out reward is a 3x-per-case mean (comparable to the K=3 pinned baseline); search/validation keep n_attempts=1. Point W&B at the shared harness-engineering-bench project, grouped by benchmark, with a per-launch run name. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A provider-agnostic runbook for launching one benchmark's optimization end-to-end (compile -> gateway -> optimizer -> sandboxed evals -> finalize), with the preflight to confirm before launching and the health checks after. No secrets or environment-specific paths; points at CONFIGURATION.md and each build.yaml for the per-benchmark specifics. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…mizer loop Audited the held-out baseline per-case wall times: the codex-probe-sized case_timeouts were too tight for the real target agents, so ~9/13/26% of gaia/officeqa/browsecomp candidate cases would be killed (scored failure) against leniently-measured baselines. Raise case_timeout to >= each benchmark's baseline max: gaia 180->900, officeqa 300->1200, tau3 900->1200, browsecomp 900->2100 (swe-atlas 1800 already fine). The pinned baselines themselves ran lenient (near-zero timeout exceptions), so they stay valid. Harbor runs the agent and verifier phases on independent clocks, and the optimizer agent phase is unbounded (governed by the gateway token budget), so verifier_timeout must cover only finalization. A verifier timeout loses the score, so size generously for n_attempts x held-out + rescore headroom: gaia 7200->10800, officeqa 14400->25200, tau3 28800->43200, browsecomp 57600->64800. Per-eval timeouts already have ample headroom; unchanged. officeqa also gets agent_env raising BASH_MAX_TIMEOUT_MS + disabling forced background tasks, so the Claude Code optimizer blocks on a whole 15-30min eval in one Bash call instead of detach/poll/end-turn (which ended the headless run after a single eval). CONFIGURATION.md updated with the new values + rationale. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…n officeqa Tokens (not dollars) are the reported unit: with a fixed target model, dollars are a downstream linear function of the (input, cached, output) triple, so they are not stored. Turn on the gateway's per-trial thread attribution for officeqa (request_log_attribution: true) so per_trial_tokens.py attributes gateway tokens to trials via stamped thread_ids instead of content-matching. Extend the aggregator with a multi-session grid rollup (flat CSV keyed by run/eval/task), per-trial wall time, and a gateway-vs-attributed-vs-agent-reported reconciliation. Document the token surfaces in CONFIGURATION.md and the run-benchmark skill. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Give every token and latency measure a mean, median, and max over trials: these distributions are heavy-tailed, so a mean alone hides that a few trials carry much of an evaluation's spend. Fix the run label for a session passed as "." (an empty name blanked the run column) and key the single-session JSON payload off the grid rather than the argument name. Document the per-case statistics in CONFIGURATION.md and the run-benchmark skill. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tokens officeqa's first full run finished with reward 0.0 and `inference_budget_exhausted`. No benchmark set the reserved `finalization` scope, so it silently inherited `evaluation`'s 100M ceiling, and finalizing 99 held-out cases at n_attempts=3 needs roughly 400M. The same ceiling had already killed both validation jobs earlier in that run: six development evaluations consumed 80.7M of the 100M, leaving the two 30-case validation runs to die partway with `evaluation terminated`. An optimizer exhausting its own evaluation budget is a legitimate result and stays possible. Trusted finalization failing on budget is not, so every benchmark now sets `finalization` explicitly. Size both caps so the agent's case budget and the fixed held-out set are what bind, since a token cap that bites first only aborts work those budgets already authorized -- raising it does not license more spend. ~3M tokens per case-run is ~2.3x the worst measured cost: 1.33M/case-run for an optimized officeqa candidate, itself ~3x its own baseline, because more turns and larger contexts are precisely what the optimizer buys. Note ~90% of the metered tokens are cache reads, counted at full weight, so the cap bites far earlier than real spend. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ncy to 24
A verifier timeout yields no reward at all -- the score is lost and only agent
artifacts are salvaged -- which makes it exactly as destructive as the
finalization budget exhaustion fixed in the previous commit. So these clocks
should never be the thing that fails, and they no longer are: because
case_timeout_seconds caps every case, there is a hard bound available,
worst-case eval = ceil(trials / max_concurrency) x case_timeout_seconds
that is, every trial timing out. timeout_seconds now sits above that and
verifier_timeout_seconds above worst-case finalize + worst-case rescore, so both
are unreachable rather than tuned to an expected duration.
officeqa's run #3 was killed once this was derived. Its finalize is 297 trials
(99 held-out x n_attempts=3); at ~420s measured mean case wall and concurrency 8
that is ~4.3h against a 7200s per-eval timeout, so it would have died about 46%
of the way through -- a third distinct finalize failure mode after the killed
agent and the exhausted budget. Run #2 would have hit it too; budget exhaustion
just reached it first. gaia and tau3 were undersized on the same arithmetic.
Raise max_concurrency 8 -> 24 everywhere. It is the throughput lever (finalize
wall = waves x mean case wall, so officeqa goes ~4.1h -> ~1.4h) and the headroom
is measured: run #2 sustained 16 case slots at ~181k metered TPM/slot with
upstream_errors=0, making 24 about 1.5x a proven configuration. Sizing assumes
concurrency 24 and n_attempts=3 on every held-out target, so both lowering
concurrency and adopting a 3x finalize elsewhere are covered.
Also propagate agent_env to the four benchmarks that lacked it, with
BASH_MAX_TIMEOUT_MS above a worst-case full validation eval. Its
background-task variables are documented as defence in depth only: they gate
automatic backgrounding and do not remove the Bash tool's run_in_background
parameter, which run #2's optimizer chose deliberately. The instruction is what
forbids that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every benchmark's case_timeout_seconds is now exactly its dataset's declared [agent] timeout_sec, so vero's derived --agent-timeout-multiplier is 1.0 and the target agent gets precisely the clock the benchmark intends. case_timeout_seconds was never an independently enforced wall: vero's only use of it is computing case_timeout / task_agent_timeout as harbor's --agent-timeout-multiplier. It is a scale factor on the dataset's own clock, and we had been scaling it hard in both directions -- swe-atlas-qna ran at 0.17x its declared 10800s while gaia ran at 1.5x its declared 600s. task_agent_timeout was already correct everywhere, so the denominators were never fictional; only the numerators were ours. Declared values come from each task.toml: the hub packages cached under ~/.cache/harbor/tasks/packages for gaia, swe-atlas-qna and tau3, and the vendored task dirs for officeqa and browsecomp-plus. All five declare uniformly across their tasks, so one value per benchmark loses nothing. No buffer above 1.0, because harbor runs four independent clocks per trial, each with its own multiplier: agent phase, agent setup (_AGENT_SETUP_TIMEOUT_SEC=360), environment build, and verification. Container start, venv install, image build and scoring cannot eat into the agent's budget, so there is no translation loss for a 1.1 to absorb -- it would only be 10% more lenient than declared. This also means mean_case_wall_seconds, which is whole-case wall, is not directly comparable to this cap. gaia gets tighter as a result (900 -> 600), newly clipping ~1% of cases whose p99 sat at 608s. That is the benchmark's intent, and the gaia agent's own MAX_TURNS cap lands at ~608s, so it appears to have been written against the declared 600. Recompute every dependent ceiling off the new per-case clocks, keeping them unreachable: per-eval timeout above ceil(trials/24) x case_timeout, verifier above worst-case finalize + worst-case rescore, and the optimizer's Bash cap above a full validation pass. swe-atlas-qna and tau3 land at 49h and 44h verifier ceilings; those are insurance that should never fire, but note swe-atlas's 3h declared per-case clock means one pathological case adds 3h to a finalize. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
varunursekar
force-pushed
the
pr3-harness-bench
branch
from
July 27, 2026 16:50
5723247 to
20d2bc5
Compare
The shared-settings bullet still advertised max_concurrency: 8 and a bare n_attempts: 1. Both are now actively misleading: concurrency is 24 everywhere and every timeout in the doc is explicitly derived from that number, and officeqa's test target overrides n_attempts to 3 with aggregate_attempts: mean. Found by review of 1487d6d. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each pinned baseline_reward was pooled over 3 measurement rounds (n = 197/297/ 450/198/145 = cases x3), but only officeqa's test target had n_attempts: 3, so on the other four a submitted candidate was scored in a single pass against a 3x-averaged floor. That gives the candidate ~sqrt(3) more standard error than the number it is judged against, in an experiment whose whole claim is that the candidate beat the baseline. Single-pass sd is 0.0099 (gaia), 0.0206 (tau3), 0.0071 (browsecomp) and 0.0106 (swe-atlas) against pooled baseline SEs of 0.0057, 0.0119, 0.0041 and 0.0061 -- tau3 matters most, where plausible gains sit inside the noise. n_attempts: 3 with aggregate_attempts: mean yields 3 observations per case, which is how the baselines were built, so the comparison is like-for-like again. Search and validation keep the global n_attempts: 1. Timeouts and gateway budgets were already sized for a 3x finalize everywhere, so nothing else changes. Also correct two doc claims that contradicted every config: score_baseline is false (the seed's score is pinned, not re-measured), and the execution defaults no longer describe the n_attempts override as officeqa-only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third of three stacked PRs splitting #41. Base:
pr2-add-vero.Adds
harness-engineering-bench/with each benchmark as a top-level sibling (gaia,ale-bench,officeqa,swe-atlas-qna,tau3) + scripts. Pure adds (71 files).Union of 1/3 + 2/3 + 3/3 ==
v0.5plus alegacy/archive (verified).🤖 Generated with Claude Code
Greptile Summary
This PR adds the final five benchmarks (
gaia,officeqa,swe-atlas-qna,tau3,browsecomp-plus) toharness-engineering-bench/, along with shared accounting and partitioning scripts, and a newvero/tests/test_v05_benchmark_configs.pythat enforces gateway-routing and scope-isolation invariants across all five configs. It is the third of three stacked PRs that together composev0.5.context.metadatais now set unconditionally in the tau3 agent,shlex.quotewraps the MCP session ID in the shell command, the officeqaextra_harbor_argsduplicate key is merged,web_searchis removed from the OfficeQA tool list, and every dispatch loop now catchesKeyError/OSErrorin addition toJSONDecodeError.GRADER_TEMPLATE.format()call inbrowsecomp-plus/task-template/tests/evaluate.pysilently scores 0 when the agent response contains curly braces, andtest_build_tasks.pyhas an assertion that will always fail (len(flattened) == 165 == 830).Confidence Score: 4/5
Safe to merge with the understanding that two issues from prior rounds remain unresolved.
Every issue flagged in previous review rounds that touched this PR's files has been addressed — session ID shell injection fixed with shlex.quote, duplicate YAML key merged, missing exception handling added across all agents, and None attribution key replaced. Two prior-round findings still apply: the GRADER_TEMPLATE.format() call in the BrowseComp-Plus verifier silently scores any task whose agent response contains curly braces as 0, and test_build_tasks.py has an assertion that will always fail (165 != 830). Both affect benchmark scoring reliability.
Files Needing Attention: harness-engineering-bench/browsecomp-plus/task-template/tests/evaluate.py (format-string injection) and harness-engineering-bench/browsecomp-plus/scripts/test_build_tasks.py (always-failing assertion).
Important Files Changed
Reviews (50): Last reviewed commit: "Average the held-out score over 3 on eve..." | Re-trigger Greptile