From e1ad9cacf8562d17d9744a3ee902119774195aec Mon Sep 17 00:00:00 2001 From: Branimir Georgiev Date: Thu, 24 Sep 2026 13:03:39 +0300 Subject: [PATCH] docs(efficacy): rewrite the benchmark README for its operator The README gets the same review the design had. Wrong claims removed: - "the same turn limit": there is none; trials are bounded by budget and timeout - "the evidence check is the only check on it": the control fixture checks the judge too - "Still to come: nothing": trials have run Missing parts added: - the control fixture and the second judge - `security.py`, `reuse.py`, `withdrawn.json`, `control/`, the arms' `generation.json`, and the brief and change-prompt files, in the path table Outdated for round 3: - `--arms` examples run all five arms - the "two arms are reused" reason for `--arms` is gone, in the README and in `harness.py`'s help - reuse shrinks to its own short subsection - security notes it becomes primary from round 3 Structure: - the scoring section splits into scoring, judging, security, reporting, one run at a time, and reuse - "How a trial ends" becomes a section, not a lone subsection - dense paragraphs become rule, bullets and a one-line why - a judging diagram - "What the arms receive" and the hidden suite's validation history point to the design instead of repeating it "Pre-registered" is replaced in the current docs and code comments that still used it. The dev journal keeps it as written. Part of #1767. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/SPEC.md | 2 +- docs/design/design-notes.md | 4 +- docs/design/testing-ai-assets.md | 2 +- tests/efficacy/README.md | 767 +++++++++++++++---------------- tests/efficacy/harness.py | 14 +- tests/efficacy/security.py | 4 +- 6 files changed, 387 insertions(+), 406 deletions(-) diff --git a/docs/SPEC.md b/docs/SPEC.md index 3c9cd793..f612d27b 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -341,7 +341,7 @@ Current design documents: keep the template library in shape so it doesn't dilute the agent's attention, with implications for code review, smoke tests, structure audit, and 360 -- `efficacy-benchmark.md` — the pre-registered experiment that +- `efficacy-benchmark.md` — the experiment, its rules fixed in advance, that measures whether a generated context file improves an agent's output, and the harness a template change is measured with - `testing-ai-assets.md` — the generic method the benchmark applies: diff --git a/docs/design/design-notes.md b/docs/design/design-notes.md index 57f6369f..3f40bd52 100644 --- a/docs/design/design-notes.md +++ b/docs/design/design-notes.md @@ -1210,8 +1210,8 @@ work outside accepted scope, not every possible improvement. ## 16. Outcome evaluation -The concrete, pre-registered form of this section is -`docs/design/efficacy-benchmark.md`: one application, three arms, paired +The concrete form of this section is +`docs/design/efficacy-benchmark.md`: one application, five arms, paired trials, a layered judge and a verdict rule fixed before the first run. The sketch below is the direction it was derived from. diff --git a/docs/design/testing-ai-assets.md b/docs/design/testing-ai-assets.md index f5ffe196..6a1acd03 100644 --- a/docs/design/testing-ai-assets.md +++ b/docs/design/testing-ai-assets.md @@ -309,7 +309,7 @@ checklist above and the calibration rules that keep it honest. ## Hand-off - the solid-ai-templates measurement The concrete application of this method to the template library itself is -`efficacy-benchmark.md` in this directory, the pre-registered experiment +`efficacy-benchmark.md` in this directory, the experiment that was blocked until this generic method concluded. This document hands it the design, and the benchmark supplies the parts this one leaves generic. diff --git a/tests/efficacy/README.md b/tests/efficacy/README.md index 674f1e55..1022f7a0 100644 --- a/tests/efficacy/README.md +++ b/tests/efficacy/README.md @@ -1,42 +1,39 @@ # Efficacy benchmark -Does a generated context file improve the result? The design, the -pre-registered question and the verdict rule live in -`docs/design/efficacy-benchmark.md`. This directory holds the parts that -run. +Does a generated context file improve the result? The design, its rules and +how a result is read live in `docs/design/efficacy-benchmark.md`. This +directory holds the parts that run. | Path | What it is | |---|---| -| `SPEC.md` | the application every arm is asked to build; the harness copies it into each workspace under this name | -| `harness.py` | sets up a workspace, runs one trial under an isolated configuration, freezes the result | +| `SPEC.md` | the application every arm is asked to build; copied into each workspace | | `brief.txt` | the project brief the generated arms' interview reads, the only copy | | `change-prompt.txt` | the change task's prompt, the only copy | -| `arms/hand/CLAUDE.md` | arm `hand`: the hand-written reference context file, the arm that asks whether the effect is the templates or merely having a file | -| `arms/full/CLAUDE.md` | arm `full`: the generated context file, produced once by `generate_arm.py` and never hand-edited | -| `arms/short/CLAUDE.md` | arm `short`: the same generation under a 40-line budget | -| `arms/hybrid/CLAUDE.md` | arm `hybrid`: the same generation in the interview's hybrid model, read beside the templates the harness vendors into the workspace | -| `generate_arm.py` | produces each generated arm's file: resolves the chain at the recorded release, builds the prompt from the interview, the pinned brief and the arm's output model and budget, scans for a specification leak, refuses a file over its budget | -| `score.py` | takes one frozen trial to a JSON score: clean install, boot, source discovery, the hidden suite, the static battery, the adherence checklist, scope and cost | -| `probes.py` | the measurements that must run inside the trial's own interpreter — the structural design probes, and the web-quality probes | -| `scoring-requirements.txt`, `toolconfig/` | the one ruler: the tools, and the lint and type configuration every arm is measured under | -| `judge.py` | builds a blind bundle per trial and runs the model judge over it, with the rubric as a JSON Schema | -| `report.py` | the paired contrasts, the bootstrap intervals, the verdict vector, and the report under `docs/audits/` | - -Still to come: nothing in this directory. The first attended trial is what -tells us which of the battery's command lines need correcting, because a tool -whose flags moved records a missing metric rather than a wrong one. +| `arms//CLAUDE.md` | each arm's context file: `full`, `short`, `hybrid` generated, `hand` hand-written | +| `arms//generation.json` | the record of each generated file's generation | +| `generate_arm.py` | generates one arm's file through the interview | +| `harness.py` | sets up a workspace, runs one trial in isolation, freezes the result | +| `score.py` | scores one frozen trial: install, boot, the hidden suite, the static tools, adherence, scope, cost | +| `probes.py` | the structural and web-quality probes that run inside the trial's own interpreter | +| `scoring-requirements.txt`, `toolconfig/` | the one ruler: the tools, and the configuration every arm is measured under | +| `judge.py` | builds a blind bundle per trial and runs the model judge over it | +| `security.py` | reads each trial for security | +| `report.py` | the contrasts, intervals, verdicts and the report under `docs/audits/` | +| `withdrawn.json` | metrics withdrawn after grading, keyed by the grader's revision | +| `reuse.py` | copies an earlier round's trials into a new run root | +| `control/` | the rubric control: six trees that check the judge | ## Before a run -The harness, the scorer and the judge reach outside Python. Each missing -piece shows up as a refusal or a metric recorded missing, not as a crash: +The harness, scorer and judge reach outside Python. A missing piece shows up +as a refusal or a missing metric, not a crash. | Needs | For | Without it | |---|---|---| -| `claude` on `PATH`, logged in | every trial and arm `full`'s generation | the preflight refuses the run | +| `claude` on `PATH`, logged in | every trial and every arm's generation | the preflight refuses the run | | `gh`, logged in with access to `braboj/tariff-hidden-suite` | scoring clones the hidden suite | scoring refuses | -| a Java runtime on `PATH` | `html5validator`, the HTML validity metric | that metric is recorded missing | -| a Playwright browser | the suite's browser flows and the accessibility probe | scoring installs one for the Playwright it resolves, and refuses the trial where it cannot | +| a Java runtime on `PATH` | `html5validator`, the HTML validity metric | that metric is missing | +| a Playwright browser | the suite's browser flows and the accessibility probe | scoring installs one, and refuses the trial where it cannot | | `codex` on `PATH`, logged in | the judge | each bundle's judging fails | `score.py --no-web` skips both browser-side probes and installs no browser. @@ -48,16 +45,16 @@ The suite then skips its browser flows, and the trial is flagged as partial. |---|---| | `none` | `SPEC.md` alone: the bare agent | | `full` | the templates' file, generated through the interview, inline | -| `short` | the templates' file, generated with a 40-line budget | -| `hybrid` | the templates' file in the hybrid model, the templates vendored | +| `short` | the templates' file, generated under a 40-line budget | +| `hybrid` | the templates' file in the hybrid model, with the templates vendored | | `hand` | the hand-written file | A trial is `-`: `short-2` is arm `short`'s trial in block 2, paired with `none-2`. Every workspace, tarball, score, judging and `--from` -or `--trial` argument uses that name. Round 1 named its arms by letter and -its trials `A1`, `B2`, `C3`; its run root and scoring area keep those -spellings, and every reader turns them into `none`, `full` and `hand`, so -the round is never rewritten on disk. +or `--trial` argument uses that name. + +Round 1 named its trials by letter: `A1`, `B2`, `C3`. Its files keep those +names on disk, and every reader maps them to `none`, `full` and `hand`. ## Generating an arm @@ -67,192 +64,167 @@ py tests/efficacy/generate_arm.py --arm short --root - py tests/efficacy/generate_arm.py --arm short --root ``` -One non-interactive invocation per generated arm (`full`, `short`, -`hybrid`), per the design's sections 11 and 12. Nobody answers questions: -the brief is read from `brief.txt` and treated as the client's answers, -so the arm is reproducible and re-running it would produce a different one. -It refuses to overwrite an existing file without `--replace`. - -The instruction names the arm's output model, and for `short` its budget: -at most 40 lines, none over 88 characters. A result over either bound is -refused and kept beside the record as rejected, never trimmed, because a -trim by hand would put a person's judgement into the arm. - -The record of the generation behind each committed file sits beside it as -`generation.json`: the release and the chain it resolved, the model, the -CLI's result, both leak scans and the budget check. The generator writes it -there whenever it writes the file, and the report reads it from there. - -The self test proves the leak scan and the budget can fail before they are -trusted. The dry run builds and scans the prompt without calling a model, -which is the cheap way to check the wiring after any change to the brief, -the roots or an arm's clause. - -Two scans run, and they ask different questions. The prompt scan is broad, -because a generation is never handed `SPEC.md` and a hit means the -plumbing is wrong. The output scan is narrow, and the prompt scan earns it: -once the prompt is clean the model demonstrably never read the -specification, so only data nobody could derive, a seed sku or a rule id or -a figure from the worked example, is evidence of a leak. +**One non-interactive invocation per generated arm** (`full`, `short`, +`hybrid`), per the design's §3.3. + +- The brief is read from `brief.txt` and treated as the client's answers. + Nobody answers questions. +- The instruction names the arm's output model and, for `short`, its budget: + at most 40 lines, none over 88 characters. +- A result over either bound is refused and kept beside the record as + rejected, never trimmed. +- An existing file is never overwritten without `--replace`. + +Why: a person answering or trimming would put their judgement into the arm. + +**The record.** Each generated file has `generation.json` beside it: the +release and the chain it resolved, the model, the CLI's result, both leak +scans and the budget check. The report reads it from there. + +**Two leak scans ask different questions.** + +- The prompt scan is broad. A generation is never handed `SPEC.md`, so any + spec-only token in the prompt means the wiring is wrong. +- The output scan is narrow. Once the prompt is clean, only data nobody + could derive, such as a seed SKU, a rule id or a worked figure, is a leak. + +**Checking it.** The self test proves the leak scan and the budget can fail. +The dry run builds and scans the prompt without calling a model: the cheap +check after any change to the brief, the roots or an arm's clause. ## Running it ```bash py tests/efficacy/harness.py --root \ - --arms none,short,hybrid --k 3 --model --dry-run + --arms none,full,short,hybrid,hand --k 3 --model --dry-run ``` -A dry run prepares every workspace and records the command without calling -a model. It works in `-dry-run` beside the root, cleared first, and -leaves the root as it found it, so the run that follows uses the same root: -a workspace is never reused, and one a dry run left there would refuse the -trial. Drop `--dry-run` to run the trials. `--arms` is required, because -two of the arms are reused from an earlier round and never re-run. Trials -are interleaved — none-1, short-1, hybrid-1, none-2 and so on — so a -model-side change part-way through lands across the arms rather than on +**A dry run** prepares every workspace and records the command without +calling a model. + +- It works in `-dry-run` beside the root, cleared first. +- It leaves the root untouched, so the real run can use the same root. +- Drop `--dry-run` to run the trials. + +**`--arms` is required.** A run names the arms it holds. Trials interleave: +`none-1`, `full-1`, `short-1`, `hybrid-1`, `hand-1`, `none-2`, and so on. + +Why: a model-side change part-way through then lands across the arms, not on one of them. -The root must lie outside this repository, and the harness refuses one -that does not: an arm working inside the templates repository can read the -templates that arm `none` is defined not to have. It also refuses a run -that a context file above the workspace would reach, and refuses to reuse -a workspace. - -A generated arm refuses until its context file exists. That file is -generated once, through the interview at the recorded release, and is -never hand-written. - -Arm `hybrid`'s file refers to the templates rather than inlining them, so -its workspace carries this repository's `templates/` directory at the -recorded release under `docs/solid-ai-templates/templates/`, where a -submodule would put it. Only that directory: the release's tree also holds -this benchmark's own design, and the arm would be reading its brief. The -copy is the arm's reading matter, not its output, so it is kept out of the -workspace's index through `.git/info/exclude` and counts in no size or -scope metric; the trial record names the release and the tree it was taken -from. For that arm the reach scan counts the repository as reached only -under an owner's name, `solid-ai-dev/solid-ai-templates` or the redirected -`braboj/solid-ai-templates`, which every clone or fetch names and no path -into the vendored copy does; the judge strips the -copy from the bundle with the context file. - -One preflight runs before the first trial: a trivial prompt through the -isolated home. The CLI answers an unauthenticated run with a result object -rather than a crash, so without it a whole run can complete having never -reached a model. - -The isolated home's credential file is a hard link to the account's live -one, made again before every trial and every probe, never a copy taken -once: another client of the account — a parallel session starting on this -machine — rotates the tokens, and a snapshot taken before that is a -revoked token, which ended a trial eighteen minutes in. Where no link can -be made the file is copied, fresh each time. - -A trial inherits this machine's environment, not the launching process's. -A shell inside an agent session or an editor carries that session's -variables, an activated virtual environment on `PATH` and credentials, and -a trial started there would join the session, share one interpreter with -every other trial, and hold keys it has no use for. All of them are removed -before the CLI starts, and each trial record lists the names removed. - -Three more routes reach past the environment, and each is closed or -caught: - -- **Git's stored credentials.** A helper the machine configures answers - any Git that asks, and the hidden suite is one authenticated clone away. - A trial's Git runs with every helper cleared. +**The harness refuses:** + +- a root inside this repository, where an arm could read the templates + `none` is defined not to have +- a run that a context file above the workspace would reach +- a workspace that already exists +- a generated arm whose context file does not exist yet + +**`hybrid` gets the templates in its workspace.** The repository's +`templates/` directory at the recorded release is copied to +`docs/solid-ai-templates/templates/`, where a submodule would put it. + +- Only that directory is copied. The release also holds the benchmark's + design, and the arm would read its brief. +- The copy is kept out of the workspace's index through `.git/info/exclude` + and counts in no size or scope metric. +- For this arm the reach scan counts the repository as reached only when a + call names its owner (`solid-ai-dev/solid-ai-templates` or the redirected + `braboj/solid-ai-templates`). No path into the vendored copy does. +- The judge strips the copy from the bundle along with the context file. + +**A preflight runs before the first trial:** a trivial prompt through the +isolated home. + +Why: the CLI answers an unauthenticated run with a result object, not a +crash, so a whole run could otherwise finish without reaching a model. + +**The isolated home's credential file is a hard link to the account's live +one**, remade before every trial and probe. Where no link can be made, the +file is copied fresh each time. + +Why: another client of the account, such as a parallel session, rotates the +tokens, and a copy taken earlier is a revoked token. + +**A trial inherits this machine's environment, not the launching +process's.** The launching session's variables, active virtual environment +and credentials are removed before the CLI starts, and each trial record +lists the names removed. + +Why: otherwise a trial would join that session, share its interpreter with +every other trial, and hold keys it has no use for. + +**Three more routes past the environment are closed or caught:** + +- **Git's stored credentials.** The hidden suite is one authenticated clone + away, so a trial's Git runs with every credential helper cleared. - **The temporary directory.** Each trial gets its own under `tmp/` in the run root. Git Bash maps `/tmp` to the machine's temporary directory - whatever `TEMP` says, so an entry that appeared there during the trial, - and that one of its tool calls names, is moved into the trial's own when - it ends. -- **Processes.** A server started in the background outlives the CLI. When - a trial ends, every process running from its workspace, its temporary - directory, a moved entry or the scratch home is stopped before the - workspace is frozen, and the record lists them. - -The shell keeps its network, because every trial installs packages; only -the two web tools are disallowed. Arm `full`'s file names this repository in its -footer, and a trial's workspace sits in the run root beside earlier trials, -so each transcript is scanned for a tool call naming this repository, the -hidden suite, the scoring area, the run root, or any entry of the root other -than the trial's own workspace and temporary directory, whether spelled from -the root or climbed to with `..`. The trial record carries the hits. The -report scans again under the current rule wherever the transcripts remain, -and names every trial with a hit. A trial with no transcript is recorded as -not scanned, never as clean. + whatever `TEMP` says, so an entry that appeared there during the trial + and that a tool call names is moved into the trial's own. +- **Processes.** A background server outlives the CLI. When a trial ends, + every process running from its workspace, temporary directory or scratch + home is stopped before the freeze, and the record lists them. + +**The shell keeps its network**, because every trial installs packages. Only +the two web tools are disallowed. So each transcript is scanned for a tool +call naming: + +- this repository or the hidden suite +- the scoring area or the run root +- any entry of the root other than the trial's own workspace and temporary + directory, whether spelled from the root or reached with `..` + +The trial record carries the hits, and the report scans again under the +current rule. A trial with no transcript is recorded as not scanned, never +as clean. ```bash py tests/efficacy/harness.py --self-test ``` -The self test plants one variable of each kind in a copy of the environment -and fails if any survives, or if `PATH` loses an entry it should keep. It -also plants a Git credential helper, a process running from a trial's -directory beside one whose name only extends it, temporary entries made -before, during and without being named, and a transcript that fetches this -repository, and fails if any of them is handled otherwise. +The self test plants one of each thing the harness removes or catches: an +environment variable of each kind, a Git credential helper, a process +running from a trial's directory, temporary entries, and a transcript that +fetches this repository. It fails if any is handled otherwise, or if `PATH` +loses an entry it should keep. -### How a trial ends +## How a trial ends | Outcome | What ended it | Scored | |---|---|---| | `completed` | the agent finished | yes | | `budget` | `--budget`, $100 by default, read against the CLI's own cost figure | yes | | `timeout` | `--timeout`, two hours by default | yes | -| `blocked` | any other error: a usage limit, a rate limit, a crash, a run that returned no result | no | -| `refused` | the harness would not start it, such as a workspace that already exists | no | - -The CLI's output is watched for its result line. A CLI that prints one and -then does not exit — a shell it left running keeps its pipes open, and on -Windows a plain timeout would kill only the `.CMD` shim it runs behind, -leaving it an orphan — is ended with everything under it two minutes -after the result, and the trial is read from that result as if it had -exited. At the timeout the whole tree is ended the same way, and whatever -was printed is kept. The record says how the CLI ended. - -A blocked trial is the provider's cut, not the agent's work. Its workspace -and tarball move under `void/`, and the run stops unless it was started with -`--resume-after-block`. That flag probes the generator every `--probe-every` -minutes and re-runs the trial in its own place once it answers. A second -block of the same trial stops the run, and so do `--give-up-after` hours -without an answer. - -The run record is rewritten after every trial, so a run stopped part-way -keeps the record of everything it finished. `--from` starts a new run at a -given trial in the interleaved order: +| `blocked` | any other error: a usage limit, a rate limit, a crash, no result | no | +| `refused` | the harness would not start it, such as a workspace that exists | no | -```bash -py tests/efficacy/harness.py --root --arms none,short,hybrid \ - --from short-2 --resume-after-block -``` +**A CLI that prints its result but does not exit** is ended, with everything +under it, two minutes after the result. The trial is read from that result. +At the timeout the whole process tree is ended the same way, and whatever +was printed is kept. + +Why: a shell the agent left running keeps the CLI's pipes open. On Windows a +plain timeout kills only the `.CMD` shim and leaves the CLI an orphan. + +**A blocked trial is the provider's cut, not the agent's work.** -## Reusing an earlier round's trials +- Its workspace and tarball move under `void/`. +- The run stops, unless started with `--resume-after-block`. +- With that flag the harness probes the generator every `--probe-every` + minutes and re-runs the trial in its own place once it answers. +- A second block of the same trial stops the run, and so do + `--give-up-after` hours without an answer. + +**A stopped run keeps what it finished.** The run record is rewritten after +every trial. `--from` starts a new run at a given trial in the interleaved +order: ```bash -py tests/efficacy/reuse.py --self-test -py tests/efficacy/reuse.py --from --into --arms full,hand +py tests/efficacy/harness.py --root --arms none,full,short,hybrid,hand \ + --from short-2 --resume-after-block ``` -Round 2 reuses round 1's `full` and `hand` trials rather than running them -again. The reuse copies what the report reads for each: its run record, -frozen tarball and transcripts into the new root, and its score, judging -and security reading into the new scoring area, each filed under the arm's -word and marked with where it came from. The judgings keep their blind -labels and the map carries them. The earlier round's tool lock comes along -too, so both rounds are measured with one ruler, and a target already -holding a different lock is refused as another run. The earlier root is -never touched. - -Nothing copied is measured again. `score.py` skips a trial whose score is -already written, `judge.py` one already judged as often as asked, and -`security.py` one already read, each saying so and each with a flag -(`--rescore`, `--rejudge`, `--reread`) to do it anyway; the judge numbers new blind -labels past any the area holds. `score.py` also refuses a root whose -existing scores were graded at another hidden-suite revision than the one -it cloned: a corrected grader is never run on the same trials, so a run at -another revision is another root. - ## The change task ```bash @@ -262,20 +234,20 @@ py tests/efficacy/score.py --root --task change ``` The change task measures how far each design has to be disturbed to take a -change it was not built for. Its prompt is read from `change-prompt.txt`, -never restated here. Each change task starts from a copy of its build trial's -frozen workspace and runs under the same isolation, bounds and outcome -rules, in the same order. The copy's state is committed first, so churn -counts only what the agent changed. Scoring re-runs the build suite, runs -the change suite and counts the files and lines changed against that commit, -and writes to `scores-change/` in the scoring area, apart from the build -scores. - -Both tasks hand the CLI its prompt on standard input rather than as an -argument. On Windows the CLI is a `.CMD` shim, and `cmd.exe` expands a -`%NAME%` pair inside an argument: sent through one, `%PATH%` arrives as the -value of `PATH`. A prompt passed as an argument is safe only while it -happens to hold no such pair. +change it was not built for. Its prompt is read from `change-prompt.txt`. + +- Each change task starts from a copy of its build trial's frozen workspace. +- The copy's state is committed first, so churn counts only what the agent + changed. +- It runs under the same isolation, bounds and outcome rules, in the same + order. +- Scoring re-runs the build suite, runs the change suite, and counts files + and lines changed. It writes to `scores-change/` in the scoring area. + +**Both tasks hand the CLI its prompt on standard input.** + +Why: on Windows the CLI is a `.CMD` shim, and `cmd.exe` expands a `%NAME%` +pair inside an argument, so `%PATH%` would arrive as the value of `PATH`. ## Scoring, judging and reporting @@ -285,201 +257,210 @@ py tests/efficacy/score.py --root py tests/efficacy/judge.py --root --repeat 3 --dry-run py tests/efficacy/judge.py --root --repeat 3 py tests/efficacy/judge.py --self-test -py tests/efficacy/report.py --self-test -py tests/efficacy/report.py --root py tests/efficacy/security.py --self-test py tests/efficacy/security.py --root +py tests/efficacy/report.py --self-test +py tests/efficacy/report.py --root +``` + +### Scoring + +**Scoring writes nothing into the run root.** It writes to a scoring area +beside it, `-scoring`: the hidden suite's clone, each trial's +extracted tree and environment, the tool environment and its lock, the +scores and the judge's bundles. + +Why: inside the root they would sit one `..` away from the next trial's +workspace. + +**It reads what the harness froze.** + +- It reads every run record in the root, so a run resumed with `--from` + scores whole. +- A trial offered by two records is refused rather than picked. +- It reads the frozen tarball, never the directory the agent worked in. + +**Each trial is scored in a clean virtual environment.** The trial's package +is installed, the hidden suite runs against that interpreter, and the static +tools run at one resolved set of versions. + +- The first trial scored resolves `scoring-requirements.txt` in an + environment of its own and freezes it to `tool-lock.txt`. +- Every trial installs the tools from that lock, so the ruler is identical + across arms. +- The lock leaves out anything installed from a local path, in editable + mode, or under the package's own name. +- After the tools install, scoring checks the package under score still + comes from the trial's own tree. If not, every metric that imports it is + missing, with that reason. + +Why: a lock carrying the scored trial's package would install the first +trial's code over every later trial's. + +**A check the grader skipped is not one the trial passed.** The pass rate is +taken over the checks that ran, the skipped count is recorded beside it, and +the trial is flagged. Scoring installs the browser Playwright needs, and +refuses the trial where it cannot. + +**HTML validity sets aside errors on HTMX's `hx-*` attributes.** The spec +requires HTMX and the HTML standard lacks them, so without the filter the +metric would count how much HTMX a trial uses. The score records the +pattern and how many errors it set aside. + +**One hidden-suite revision per root.** `score.py` refuses a root whose +scores were graded at another revision than the one it cloned. + +Why: a corrected grader never re-grades the same trials, so a run at +another revision is another root. + +**The self test is a control, not smoke.** + +- It plants a real module in an environment with no tools and requires + every metric to come back missing. A tool that scanned nothing must never + record zero. +- It plants a freeze carrying a local, an editable and a same-named package, + and fails if any reaches the lock or a replaced package goes unnoticed. + +### Judging + +```mermaid +flowchart LR + T[Trial tree] --> B[Blind bundle] + B --> J1[Judging 1] + B --> J2[Judging 2] + B --> J3[Judging 3] + J1 & J2 & J3 --> M[Mean per row] + J1 & J2 & J3 --> E[Evidence check] +``` + +**The judge reads each trial three times, and the report takes the mean.** +One judging of a tree is not reproducible (the design's §5.6 and §11). + +- `--repeat` tops a trial up to that count rather than adding to it. +- A run cut short by the judge's usage limit resumes by being started + again. +- A trial already holding three judgings is left alone. +- Rounds 1 and 2 were judged once each and read as they were. + +**The bundle is blind.** The context file is removed, condition markers are +masked, the order is shuffled at a recorded seed, and the unblinding map is +written where the judge cannot reach it. A bundle that still names its arm +is not judged. + +**The bundle leaves out scoring's own output:** tool caches, the complexity +report, coverage data and the install's egg-info. Their absolute paths name +the trial. The leak scan reads every file in the bundle, and counts the +trial's name between path separators as a leak. + +**The rubric is passed as a JSON Schema**, so a score is machine-read, not +parsed from prose. Every evidence line is looked up in the bundle it was +quoted from. + +Why: a judge that never opened the code still returns plausible numbers, +and the evidence check tells the two apart. + +**The judge's CLI gets its prompt on standard input and is launched by its +resolved path.** + +- It must report its version before any bundle is built, or the run is + refused. +- A `.CMD` shim ends an argument at its first newline, so a prompt passed as + one would arrive cut short. +- A CLI refused by the model leaves its final message empty. A failed + judging records the tail of the CLI's output beside it. + +The self test installs a stand-in CLI behind a shim, checks the prompt and +every argument arrive whole, and has the stand-in refuse. + +**The control fixture checks the judge itself.** Six trees built from one +application, some damaged, some improved, all passing the same tests. A +primary row counts only if it falls on the damaged trees and rises on the +improved ones. See `control/README.md`. + +A second judge reads the control fixture only, never a round, through +`judge.py --cli claude --model `, in a root of its own. Its scores +are never averaged with the rounds' judge. + +### Security + +`security.py` reads each scored trial for: + +- hard-coded secret keys and debug left on +- SQL built from strings +- known vulnerabilities in the installed dependencies +- session cookie flags and security headers +- stack traces in answers to malformed requests + +Each trial is installed into a fresh environment under `security/` in the +scoring area. The results go to `security-scores/`. + +These checks were declared after round 1, so the report prints them apart, +with means and intervals and no verdict. From round 3, security and data +protection become primary dimensions (the design's §5.8). + +### Reporting + +The report aggregates the scores, judgings and security readings into +`docs/audits/YYYY-MM-DD-efficacy.md`. The design's §6 says what it prints. + +**It computes the one escalation to K = 5**, so nobody decides it by eye. + +- It is owed where a primary dimension's interval contains zero while its + mean difference exceeds 0.5 points, in either direction, on any contrast. +- At K = 3 the report names the rows that owe it, and the + `--k 5 --from none-4` run that settles it. +- At K = 5 it judges the first three blocks alone, prints their verdicts + beside the K = 5 ones, and flags an escalation no row owed. +- The harness refuses `--k` above 5, and the report refuses a run past it. + +**A withdrawn metric** is listed in `withdrawn.json`, keyed by the grader's +revision. The report blanks its values, intervals and verdicts, and lists it +under "Withdrawn measurements" with the reason. A run graded by any other +revision keeps the metric. + +The self test runs the design's worked cases through the verdict rule, +plants a row on each side of the escalation rule, and plants a run on each +side of a withdrawal. + +### One run at a time + +`score.py`, `judge.py` and `security.py` each clear a trial's directory +before rebuilding it. So only one run of each tool may work on a scoring +area at a time. + +- A run writes `.running` in the scoring area, naming its process, and + removes it on exit. +- A run that finds one naming a live run of the same tool is refused. +- One naming a process that has ended, or a pid now held by something else, + is taken over, and the run says so. + +Why: a second run would delete what a live one is reading, and two judge +runs would take the same next label. + +### Reusing an earlier round's trials + +```bash +py tests/efficacy/reuse.py --self-test +py tests/efficacy/reuse.py --from --into --arms full,hand ``` -The judge reads each trial three times, and the report means the three. One -reading of a tree is not reproducible — six calls on one unchanged bundle -returned readability 4, 4, 4, 3, 4, 4 — and section 12 of the design fixes -three before the round. `--repeat` tops a trial up to that count rather than -adding to it, so a run cut short by the judge's usage limit resumes by being -started again, and a trial already holding three is left alone. Rounds 1 and -2 were judged once each and read as they did. - -One run of each tool at a time works on a scoring area. `score.py`, -`judge.py` and `security.py` each clear a trial's directory before rebuilding -it, so a second run would delete what a live one is reading: two judge runs -also take the same next label. A run writes `.running` in the scoring -area, naming its process, and removes it on exit. A run that finds one naming -a live run of the same tool is refused. One naming a process that has ended, -or a pid now held by something else, is taken over and said so, so a run that -crashed does not hold the area. - -Scoring reads every run record in the root, so a run stopped and resumed -`--from` a later trial scores whole; a trial two records both offer is -refused rather than picked. It reads the tarball the harness froze, never -the directory the agent worked in. - -Scoring, judging and the aggregate write nothing into the run root. They -write to a scoring area beside it, `-scoring`: the hidden suite's -clone, each trial's extracted tree and environment, the tool environment and -its lock, the scores and the judge's bundles. Inside the root they would sit -one `..` away from the next trial's workspace. - -A check the grader skipped is not one the trial passed. The pass rate is -taken over the checks that ran, so the skipped count is recorded beside it -and the trial is flagged. Scoring installs the browser for the Playwright it -resolves — once before the grader and again after the battery installs the -lock's own version — and refuses the trial rather than score it on the -checks a missing browser leaves behind. - -Each trial gets a clean virtual environment, the trial's package -installed into it, the hidden suite run against that interpreter, and the -static battery at one resolved set of tool versions. The first trial scored -resolves `scoring-requirements.txt` in an environment of its own that holds -nothing else and freezes it to `tool-lock.txt`; every trial installs from -there, so the ruler is identical across the arms and carries no trial's -dependencies. - -HTML validity counts the validator's errors less those on HTMX's `hx-*` -attributes. The specification requires HTMX and the HTML standard has no such -attributes, so without the filter the metric would count how much HTMX a trial -uses. The score records the pattern and how many errors it set aside. - -The lock leaves out anything installed from a local path, in editable mode -or under the package's own name. A freeze lists the scored trial's package -too, and a lock carrying it installs the first trial's code over every later -trial's. After the battery installs, scoring also checks that the package -under score still comes from the trial's own tree. If it does not, every -metric that imports the package is recorded missing with that reason, never -measured on another trial's code. - -Both self tests are controls rather than smoke. `score.py --self-test` plants -a real module in an environment with no tools and requires every metric to -come back *missing*: a tool that scanned nothing must never record a zero, -because zero findings and nothing scanned are the same number and opposite -facts. It also plants a freeze carrying a package installed from a local -path, one in editable mode and one under the package's own name, and fails -if any of them reaches the lock or if a replaced package goes unnoticed. -`report.py --self-test` runs the design's own worked cases through the -verdict rule, including the row an earlier draft of the design read wrongly. - -The report also computes the design's one escalation to K = 5, so nobody -decides it by eye. It is owed where a primary dimension's interval contains -zero while its mean paired difference exceeds 0.5 points, in either direction, -on any contrast. At K = 3 the report names the rows that owe it and the -`--k 5 --from none-4` run that settles it. At K = 5 it judges the first three -blocks alone, prints their verdict vector beside the K = 5 one, and flags an -escalation no row owed. The harness refuses `--k` above 5, and the report -refuses a run past it. The self test plants a row on each side of the rule. - -Judging builds a blind bundle per trial: the context file removed, condition -markers masked, the order shuffled at a recorded seed, the unblinding map -written where the judge cannot reach it. A bundle that still names its -condition is not judged at all. The rubric is passed as a JSON Schema, so a -score is machine-read rather than parsed out of prose, and every evidence line -is checked against the bundle it was quoted from — a judge that never opened -the code returns plausible numbers, and that check is what tells the two -apart. - -Scoring's own output stays out of the bundle: the tool caches, the complexity -report, the coverage data and the install's egg-info. The caches hold absolute -paths through the scoring area, and a path names the trial. The leak scan reads -every file in the bundle and counts the trial's name between path separators -as a leak. `judge.py --self-test` plants both cases. - -The judge's CLI is launched by its resolved path, with its prompt on standard -input, for the reason the harness does both. It must report its version before -any bundle is built, the dry run included, or the run is refused. A `.CMD` shim -also ends an argument at its first newline, so a prompt passed as one arrives -cut short. The self test installs a stand-in CLI behind a shim and checks that -the prompt and every argument arrive whole. - -A CLI refused by the model creates its final-message file and leaves it empty, -with the cause only in what it printed. A failed judging records the tail of -that output beside the empty or unreadable message, and the self test has the -stand-in refuse to prove it. - -`security.py` reads scored trials with security checks declared after a run: -hard-coded secret keys, debug left on, SQL built from strings, known -vulnerabilities in the installed dependencies, session cookie flags, security -headers, and stack traces in answers to malformed requests. Each trial is -installed into a fresh environment under `security/` in the scoring area, and -the results go to `security-scores/`. The report prints them in a section of -their own, with means and intervals and no verdict, and keeps them out of the -verdict vector and the escalation. A check chosen after the results were seen -can describe a run but not decide it. - -A metric whose grader turns out to measure something other than the metric is -withdrawn in `withdrawn.json`, keyed by the grader's revision. The report -blanks its values, intervals and verdicts, and lists it under "Withdrawn -measurements" with the reason. A run graded by any other revision keeps the -metric. The self test plants a run on each side. +Round 2 reused round 1's `full` and `hand` trials. Round 3 reuses nothing, +because the spec and the judge changed. + +`reuse.py` copies what the report reads for each trial: its run record, +tarball, transcripts, score, judgings and security reading, marked with where +they came from. The earlier round's tool lock comes along too, so both +rounds are measured with one ruler. Nothing copied is measured again: each +tool skips what is already there, unless given `--rescore`, `--rejudge` or +`--reread`. ## What is not here -The hidden acceptance suite lives in the private repository -`braboj/tariff-hidden-suite`. This repository is public, so a suite kept -here would be reachable by any trial with web access, and the design's web -ban protects only the arms of one run. The harness clones it at scoring -time and never into a workspace under test. - -It is written and validated, ahead of any trial as the design requires. - - -It holds 377 checks across 16 modules, covering the public API, every -refusal, the pricing arithmetic, every route and form, both export formats -byte by byte, the seed, the JavaScript-disabled path and four browser -flows. No generator here can recount them, because the suite is in another -repository; its own README carries the live figure. - - -Two things establish that it grades rather than merely runs, and both live -beside it in that repository. - -- A reference implementation of `SPEC.md`, written from the specification - alone by an agent that never saw the tests. Every check passes against - it, so a correct implementation is not marked wrong. It stays out of - every workspace, being a worked answer to the task the arms are set. -- A mutation control that plants one specification violation at a time and - expects a failure: half-up rounding, a tier boundary excluding its own - threshold, the line rules in the wrong kind order, bulk charging every - unit, and the allocation's spare cents going to the wrong lines. Five of - five caught, with a green baseline either side. - -Writing the two against the same specification, independently, is what -found the specification's own gaps. It named no form field, so no POST -could be issued at all; it left the fragment response and the -JavaScript-disabled response contradicting each other; and it admitted two -readings of whether a rule that changed nothing counts as applied. Those -are fixed in `SPEC.md`, before any arm was asked to build against it. - -A fourth gap was not found that way, because the suite and the reference -implementation read it alike. Section 6 put the application factory "in -the package": both agents in the first two trials read that as anywhere -inside it, while the suite reads the package root, so every check that -needs the application errored in both. Section 6 now names -`tariff.create_app`. - -## What the arms receive - -Each arm starts from an empty git repository holding `SPEC.md` and, for -the arms that have one, a `CLAUDE.md`. The generated context file is -produced once at `v2.90.0`, the last release before the v3.0 split, and -recorded with the report. Nothing else about the arms differs — the same -prompt, the same model, the same turn limit, the same isolated agent -configuration. - -`SPEC.md` fixes the domain semantics, the public Python API, the HTTP -routes and the export formats, because the hidden suite drives them. It -deliberately fixes nothing about layout, tooling, error hierarchy shape, -logging, test convention or template organisation: those are what a -context file adds, so a spec that named them would answer the question -being asked. - -## Reports - -Each run writes a dated report to `docs/audits/`. It opens with a finding -table generated from its own verdicts: one column per context file against -none, with whether it improves the code, what moved, hidden tests passed, -size, cost and a one-phrase reading, then a line per file on why it scored -so and a line reading the columns together. Under it, one line scores the -contrasts between two files, 1 to 10 with the metrics won and failed, and -one line of caveats follows. The score is 1 + 9 × wins ÷ (wins + fails) and -decides nothing, and neither does the finding table. The report goes on to -name the model ids, the template revision, K, every trial's raw numbers, -the judge's evidence check and the verdict vector. No person scores the -judge: the evidence check is the only check on it. +**The hidden acceptance suite** lives in the private repository +`braboj/tariff-hidden-suite`. The harness clones it at scoring time and never +into a workspace. It holds a reference implementation that passes every +check, and a mutation control that the suite catches in full. Its own README +carries the live check count. The design's §11 records how it was validated. + +Why private: this repository is public, so a suite kept here would be +reachable by any trial with web access. diff --git a/tests/efficacy/harness.py b/tests/efficacy/harness.py index 8a475d62..af5706f2 100644 --- a/tests/efficacy/harness.py +++ b/tests/efficacy/harness.py @@ -1,6 +1,6 @@ """Set up and run one efficacy-benchmark trial. -The design, the pre-registered question and the verdict rule live in +The design, its question and its verdict rule live in `docs/design/efficacy-benchmark.md`. This module owns section 4 of it: the workspace an arm starts from, the isolated configuration it runs under, the one prompt every arm receives, and the freeze that ends a trial. @@ -55,7 +55,7 @@ BUDGET_USD = 100.0 TIMEOUT_S = 7200 -# The pre-registered K, and the ceiling of the design's one escalation. No run +# K as the design fixes it, and the ceiling of the design's one escalation. No run # goes past the ceiling, because sampling until an interval clears zero is the # failure the bound exists to prevent. K_PRIMARY = 3 @@ -2245,12 +2245,12 @@ def parse_args(argv): "each scorable build trial") parser.add_argument("--arms", help="which arms to run, as words separated by " - "commas: none,short,hybrid. Required for a run, " - "because two of the arms are reused from an " - "earlier round and never re-run") + "commas: none,full,short,hybrid,hand. Required " + "for a run, so a run names the arms it " + "holds") parser.add_argument("--k", type=int, default=K_PRIMARY, - help="trials per arm (default %d, the pre-registered " - "K; at most %d, the escalation's ceiling)" + help="trials per arm (default %d, the K the design " + "fixes; at most %d, the escalation's ceiling)" % (K_PRIMARY, K_CEILING)) parser.add_argument("--model", default="claude-sonnet-5", help="exact generator model id, recorded in the report") diff --git a/tests/efficacy/security.py b/tests/efficacy/security.py index 739578f3..8ebbf7e9 100644 --- a/tests/efficacy/security.py +++ b/tests/efficacy/security.py @@ -1,7 +1,7 @@ """Read frozen efficacy trials for security, with checks declared after a run. -A run's pre-registered security checks can pass on every trial and separate -nothing. The checks here were declared in writing after such a run and before +A run's security checks, fixed before it, can pass on every trial and +separate nothing. The checks here were declared in writing after such a run and before any trial was read for them, so they can describe a run but never decide it: the report prints them apart from the verdict vector.