Feature/n agent support#75
Open
Raandom wants to merge 3 commits into
Open
Conversation
Raandom
marked this pull request as ready for review
July 2, 2026 20:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
It includes all the functionality to run full n agent benchmarks, using the already existing codebase for that setup in the repository.
Changes
src/cooperbench/eval/sandbox.py
Adds test_merged_n(), the N-agent counterpart to the existing test_merged(), which sets up N git branches and folds them together sequentially (agent2 into agent1, then agent3, etc.) before running each feature's test suite. Also adds the supporting helpers _setup_branches_n(), _merge_fold(), _build_merged_n_result(), and _merged_n_error_result() — all existing 2-agent functions are untouched.
src/cooperbench/eval/evaluate.py
Adds a team branch in _evaluate_single() that loads one patch per feature and routes to test_merged_n(), with dual-write of the legacy feature1/feature2/both_passed keys when N=2 for backward compatibility. Also updates the single-run display to handle the new features_result dict, and guards the GCP batch path against N>2 team runs.
src/cooperbench/runner/tasks.py
load_subset() now reads a "groups" key (N-element arrays) from subset JSON files alongside the existing "pairs" key, and discover_tasks() no longer hardcodes a 2-element destructure when iterating subset groups, making N-agent task definitions possible without touching existing subset files.
src/cooperbench/runner/core.py
All both_passed checks in _process_eval_result() and the inline eval progress display are updated to also check all_passed, so team eval results are correctly counted in run summaries alongside legacy coop results.
scripts/check_gold_conflicts.py
Generalises the existing pairwise gold conflict checker to support arbitrary N-tuple groups via a new --group-size N flag (default 2, fully backwards compatible). Adds discover_all_groups(), check_one_group(), _build_merge_script(), and write_subset_json(), and fixes two pre-existing Modal bugs: containers now start with sleep infinity to stay alive, and sb.terminate() is wrapped to tolerate already-terminated sandboxes.
tests/eval/test_sandbox.py
Adds TestMergedNErrorResult (schema correctness for the N-agent error result, including dual-write legacy keys for N=2) and TestTestMergedNSignature (source-level invariants verifying the sequential fold strategy, identical-patch short-circuit, lead-alone fallback, and N-branch loop are all present).
tests/eval/test_evaluate.py
Adds TestEvaluateSingleTeamRouting with five cases covering: team setting routes to test_merged_n and not test_merged; 2-agent team runs dual-write legacy keys; 3-agent team runs omit legacy keys; feature_ids are forwarded correctly; and coop runs are unaffected.
tests/runner/test_tasks.py
Adds TestGroupsSubsetKey with four cases verifying: load_subset() reads the "groups" key; "groups" takes precedence over "pairs" when both are present; discover_tasks() emits N-element feature lists from a groups subset; and the existing "pairs" key still works unchanged.
Checklist