From db954bd059bc8c6727930fa75368d6dc670e1ca4 Mon Sep 17 00:00:00 2001 From: Branimir Georgiev Date: Thu, 24 Sep 2026 19:01:59 +0300 Subject: [PATCH] docs(design): the hidden suite's count and validations after round 3's extension MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Design ยง11 still gave round 1's 377 checks and five mutations. It now gives the measured round-3 figures: 534 build checks and 74 change checks. The validations are the reference, the reference with the change made, twelve caught mutations, and the no-sign-in control. Part of #1767. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/design/efficacy-benchmark.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/docs/design/efficacy-benchmark.md b/docs/design/efficacy-benchmark.md index 0f7c6b4b..2029ebf1 100644 --- a/docs/design/efficacy-benchmark.md +++ b/docs/design/efficacy-benchmark.md @@ -891,15 +891,16 @@ two-hour trial stays under the cap. **How the hidden suite was validated.** - -377 checks across 16 modules in `braboj/tariff-hidden-suite`, validated two -ways before any arm existed. + +534 build checks across 18 modules and 74 change checks in +`braboj/tariff-hidden-suite`, validated before round 3's first trial. - A reference implementation written from the spec alone passes every - check, so a correct implementation is not marked wrong. -- A mutation control plants five spec violations one at a time, and the - suite catches all five. - -Round 3 extends the suite to sign-in and customers, and both validations -run again before its first trial (#1767). + build check, so a correct implementation is not marked wrong. +- A second agent made the change task on that reference from the change + prompt alone. It passes every change check, and the build suite still. +- A mutation control plants twelve spec violations one at a time, and the + suite catches all twelve. +- The pre-round-3 reference, which has no sign-in, fails the sign-in + module, so its checks cannot pass without a gate.