perf-report silently skips three bench groups: tableau_surface_bench aborts with exit 101 mid-run

[Roger-luo]

Summary

On codex/traits-2-impl (the ppvm-*-2 branch, #204 — ppvm-conformance-2 does not exist on main), tableau_surface_bench panics partway through every run. mise run perf-report treats this as a warning and continues, so three bench groups have never produced a single measurement and the perf gate silently covers less than it appears to.

Symptom

$ mise run perf-report -- --bench tableau_surface_bench --launches 4
...
warning: tableau_surface_bench exited 101      (×4, once per launch)

The panic:

crates/ppvm-conformance-2/benches/tableau_surface/mod.rs:88:5
assert_bare_eq  — old-vs-new to_string() mismatch on a 96-qubit tableau

reached from the channel! macro at crates/ppvm-conformance-2/benches/tableau_surface/noise.rs:18.

Impact

Groups are registered in order in tableau_surface_bench.rs; the abort happens in noise, so everything after it never runs:

group	runs today
construction, display, clifford, rotation, measurement	yes (registered before noise)
noise	aborts here
observation, projection, sparse_amplitudes	never

So there is no perf coverage at all for observation, projection, or sparse amplitudes, and no coverage for the part of noise after the failing fixture. Anyone reading a green-ish perf-report would reasonably assume otherwise.

Two things to fix, and they're independent:

1. The fixture. assert_bare_eq compares the old and new engines' to_string() on a 96-qubit tableau inside a benchmark setup path. Either it's a real old-vs-new divergence worth investigating on its own, or the assertion is too strict for a bench fixture and should be narrowed or dropped.
2. The harness. benchmarks/perf_regression_report.py should treat a non-zero bench exit as a failure, not a warning. A benchmark binary that dies halfway through is not a partial result to summarize; silently degrading coverage is worse than a red run, because it looks like a pass.

Provenance

Pre-existing, not caused by any in-flight change: reproduced in a pristine git archive HEAD | tar -x checkout of the branch tip, so it predates the current working-tree changeset. Two separate investigations reached different conclusions about the cause (one reverting a single file and concluding "it must be one of the others"), which is itself a reason to fix the harness — a warning-level abort invites exactly that kind of misattribution.

🤖 Generated with Claude Code