Skip to content

Feature/n agent support#75

Open
Raandom wants to merge 3 commits into
mainfrom
feature/n-agent-support
Open

Feature/n agent support#75
Raandom wants to merge 3 commits into
mainfrom
feature/n-agent-support

Conversation

@Raandom

@Raandom Raandom commented Jun 30, 2026

Copy link
Copy Markdown

Description

It includes all the functionality to run full n agent benchmarks, using the already existing codebase for that setup in the repository.

Changes

src/cooperbench/eval/sandbox.py
Adds test_merged_n(), the N-agent counterpart to the existing test_merged(), which sets up N git branches and folds them together sequentially (agent2 into agent1, then agent3, etc.) before running each feature's test suite. Also adds the supporting helpers _setup_branches_n(), _merge_fold(), _build_merged_n_result(), and _merged_n_error_result() — all existing 2-agent functions are untouched.

src/cooperbench/eval/evaluate.py
Adds a team branch in _evaluate_single() that loads one patch per feature and routes to test_merged_n(), with dual-write of the legacy feature1/feature2/both_passed keys when N=2 for backward compatibility. Also updates the single-run display to handle the new features_result dict, and guards the GCP batch path against N>2 team runs.

src/cooperbench/runner/tasks.py
load_subset() now reads a "groups" key (N-element arrays) from subset JSON files alongside the existing "pairs" key, and discover_tasks() no longer hardcodes a 2-element destructure when iterating subset groups, making N-agent task definitions possible without touching existing subset files.

src/cooperbench/runner/core.py
All both_passed checks in _process_eval_result() and the inline eval progress display are updated to also check all_passed, so team eval results are correctly counted in run summaries alongside legacy coop results.

scripts/check_gold_conflicts.py
Generalises the existing pairwise gold conflict checker to support arbitrary N-tuple groups via a new --group-size N flag (default 2, fully backwards compatible). Adds discover_all_groups(), check_one_group(), _build_merge_script(), and write_subset_json(), and fixes two pre-existing Modal bugs: containers now start with sleep infinity to stay alive, and sb.terminate() is wrapped to tolerate already-terminated sandboxes.

tests/eval/test_sandbox.py
Adds TestMergedNErrorResult (schema correctness for the N-agent error result, including dual-write legacy keys for N=2) and TestTestMergedNSignature (source-level invariants verifying the sequential fold strategy, identical-patch short-circuit, lead-alone fallback, and N-branch loop are all present).

tests/eval/test_evaluate.py
Adds TestEvaluateSingleTeamRouting with five cases covering: team setting routes to test_merged_n and not test_merged; 2-agent team runs dual-write legacy keys; 3-agent team runs omit legacy keys; feature_ids are forwarded correctly; and coop runs are unaffected.

tests/runner/test_tasks.py
Adds TestGroupsSubsetKey with four cases verifying: load_subset() reads the "groups" key; "groups" takes precedence over "pairs" when both are present; discover_tasks() emits N-element feature lists from a groups subset; and the existing "pairs" key still works unchanged.

Checklist

  • Code follows project style
  • Tests pass locally
  • Documentation updated (if needed)

@Raandom Raandom changed the title Untested, do not met yerge. Feature/n agent support Feature/n agent support Jun 30, 2026
@Raandom
Raandom marked this pull request as ready for review July 2, 2026 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant