exp: v1 runtime benchmark matrix

[mikasenghaas]

Summary

Reshapes the v1 benchmark to compare runtimes instead of env-server worker modes, writes one results file per matrix cell, adds a plot script, and commits a single-turn results set (subprocess / prime / modal).

• Swept axis is the runtime: benchmark.sh (gsm8k-v1), agentic_benchmark.sh (terminal-bench-2-v1).
• Rollouts {64, 512, 1024} (single-turn); concurrency unbounded by default (MAX_CONCURRENT knob to cap it); agentic turns capped at 32; gsm8k generation unbounded.
• Per-cell results: bench/results/<suite>/<runtime>-r<rollouts>.json. bench_aggregate.py writes one file per cell (records max_concurrent, reward/errors, and the sorted per-rollout duration lists for setup / generation / scoring / total); the scripts write into the committed results dir without wiping it, so re-running a subset (e.g. just modal) refreshes only those cells. bench/plot.py renders the committed plot below.
• Base-compat fixes: --retry.attempts → --retries.*.max_retries 0 (feat: per-call model + runtime retries #1632), and a guard on the ~/.env source so a detached run doesn't die re-reading the 1Password FIFO.

Results (bench/results/single_turn/)

gsm8k-v1 on deepseek-v4-flash, unbounded concurrency. Each stage column is p50 (p10–p90) seconds per rollout; e2e is the cell wall clock. (subprocess runs rollouts on the eval host; prime/modal offload to remote sandboxes.)

cell	conc	e2e	err	rew	setup	generation	scoring	total
subprocess r64	∞	12s	0	1.0	0.0 (0.0–0.0)	6.3 (3.9–7.6)	0.3 (0.2–0.3)	6.5 (4.2–7.8)
subprocess r512	∞	33s	0	0.994	0.0 (0.0–0.0)	18.6 (11.8–21.6)	5.3 (4.3–6.9)	24.6 (17.1–26.9)
subprocess r1024	∞	56s	0	0.997	0.0 (0.0–0.0)	30.3 (15.9–45.5)	14.1 (4.7–16.7)	45.0 (31.7–50.3)
prime r64	∞	32s	0	1.0	10.9 (8.7–11.8)	9.6 (9.6–12.6)	3.5 (3.5–3.5)	24.6 (21.8–27.2)
prime r512	∞	49s	0	0.996	14.2 (10.2–17.2)	12.8 (12.6–16.0)	3.5 (3.5–3.7)	31.9 (26.7–35.8)
prime r1024	∞	84s	0	0.999	20.4 (12.6–40.6)	16.0 (12.8–19.6)	6.8 (3.5–9.9)	45.7 (34.9–57.4)
modal r64	∞	41s	0	1.0	8.0 (2.5–12.5)	10.3 (8.7–12.7)	3.5 (2.7–4.4)	22.0 (15.6–28.0)
modal r512	∞	173s	0	1.0	51.1 (11.8–91.4)	10.7 (8.5–13.2)	3.4 (2.6–4.5)	64.0 (26.1–105.8)
modal r1024	∞	283s	0	0.998	101.7 (21.7–183.7)	10.0 (8.2–13.3)	3.5 (2.6–4.5)	115.7 (35.8–197.7)

Results (bench/results/agentic/)

terminal-bench-2-v1 fix-git on deepseek-v4-flash, 32-turn agent. docker capped at 128 (local containers), prime/modal unbounded. subprocess is excluded — agentic needs a container. Stage columns are p50 (p10–p90) seconds per rollout.

cell	conc	e2e	err	rew	setup	generation	scoring	total
docker r64	128	165s	0	0.984	0.5 (0.4–0.6)	68.3 (62.0–85.1)	4.5 (4.2–5.4)	74.0 (66.9–89.8)
docker r512	128	485s	0	0.992	0.9 (0.3–2.0)	90.0 (58.6–111.2)	7.5 (3.5–18.8)	99.0 (65.5–131.6)
prime r64	∞	233s	0	0.984	10.4 (8.4–12.3)	60.6 (50.5–75.1)	14.7 (8.1–16.3)	87.4 (69.7–94.3)
prime r512	∞	289s	0	0.996	14.1 (10.2–17.3)	105.3 (81.0–117.9)	6.8 (6.7–14.5)	126.0 (108.6–140.0)
modal r64	∞	136s	0	1.0	8.4 (2.8–13.1)	64.0 (50.9–84.5)	6.6 (4.8–7.9)	80.3 (62.7–100.4)
modal r512	∞	234s	0	0.977	51.3 (11.3–92.8)	60.6 (52.0–73.4)	6.4 (4.6–8.3)	121.9 (79.4–161.2)

Depends on

• feat: record per-rollout setup timing as a distinct phase #1631 (setup span) — merged.
• The Trace.to_wire timing-exclusion fix needed to run over the env-server path — merged into the base (generic computed-field exclusion); fix: exclude timing.setup.duration from Trace.to_wire #1634 closed as superseded.