Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions docs/design/efficacy-benchmark.md
Original file line number Diff line number Diff line change
Expand Up @@ -589,11 +589,11 @@ No person scores the judge. Two automated checks do:
bundle it was quoted from, and the report prints the share found per
trial. A judge that never opened the code still returns plausible
numbers; this is what tells the two apart.
- **The control fixture:** six trees built from one pinned application
(`tests/efficacy/control/`). Every tree passes the same tests, so only
structure separates them. Some are damaged, some improved. A primary
dimension counts only if it falls on the damaged trees and rises on the
improved ones.
- **The control fixture:** eleven trees built from two pinned applications
(`tests/efficacy/control/`). Every tree passes the same tests as its
application, so only what was changed separates them. Some are damaged,
some improved. A primary dimension counts only if it falls on the damaged
trees and rises on the improved ones.

**Where the control's application comes from.** A row can only fall or
rise on code that has what it measures. The first pinned application is a
Expand Down
52 changes: 42 additions & 10 deletions tests/efficacy/control/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,16 @@ not change when the code changes measures nothing, and nothing in a normal run
would show that: an arm that moves no row looks like an arm that changed
nothing.

This control settles it. Six trees are built from one pinned application and
the judge scores each. Every tree passes the application's whole suite, so what
separates them is structure alone, and each damages or improves one thing, so a
row that moves says which.
This control settles it. Trees are built from two pinned applications and the
judge scores each. Every tree passes its application's whole suite, so what
separates it from its base is the one thing it damages or improves, and a row
that moves says which.

The first application, from round 1, carries the six trees for `design`,
`readability` and `maintainability`. It predates sign-in and customers, so the
`security` and `data_protection` rows cannot move on it; their five trees are
built from the second, one uncounted calibration trial on the extended
specification (design §5.7).

## Running it

Expand Down Expand Up @@ -59,6 +65,30 @@ reading. A root left claimed by a run that crashed is taken over by the next.
| `improved-2` | `improved-1`, and the store maps a rule's own terms onto its columns with a codec per column, so no module dispatches on a kind |
| `improved-3` | `improved-2`, and each kind declares the fields it asks a form for, so the form-building code and the template are derived from the registry and no layer names a kind |

The second application's trees:

| Tree | What it is |
|---|---|
| `base-2` | the calibration trial's application, unaltered: one guard before every route but sign-in, a CSRF token on every form, a salted password hash, `next` followed only to a local path, a secret key generated into a file; erasure clears the customer's fields, and nothing is logged |
| `insecure-1` | the secret key a literal, the password stored and compared as typed, the user lookup built from a string, and `next` followed wherever it points |
| `secured-1` | the secret key read from the environment first, the session cookie marked `HttpOnly` and `SameSite`, three security headers on every answer, and the check on `next` a named function that also refuses a backslash |
| `leaky-1` | the customer's name, email and address written to the log on creation and erasure, and erasure only raising the flag, leaving every field in the database |
| `protected-1` | one tuple names the personal fields, which erasure clears and the export returns; one lookup answers None for an erased customer, so no view or template reads the flag for itself; SQLite overwrites what it deletes |

The damage in `insecure-1` and `leaky-1` sits where the hidden suite does not
look: it states that password storage, cookie flags and an off-site `next` are
not graded, and it reads erasure through the pages.

<!-- measured: 2026-09-25 -->
Each tree's hidden-suite result is identical, test by test, to `base-2`'s,
at suite revision `65bfb9a`: 293 of 534 pass. The
calibration trial seeds only through its command, not when the application
opens an empty database as the specification asks, so the suite's
administrator does not exist and 240 checks behind sign-in fail on every tree
alike. That comparison therefore covers the 293 that pass, the pricing and API
checks among them.
<!-- /measured -->

## Reading a result

A row is doing its job when it falls on the tree that damages what it
Expand Down Expand Up @@ -198,14 +228,16 @@ rises on a tree built to improve it. `design` and `maintainability` fall on
`improved-3`. The one-point flips of single judgings remain inside each
triple, which is what the mean is there to absorb.

## Why the base is a pinned archive
## Why the bases are pinned archives

`base.zip` holds one trial's output: the application every tree is built
from. It is an input to this control, not an artifact of it:
regenerating it would compare a later run against a different application, and
every number above would silently stop meaning what it says.
`base.zip` and `base-2.zip` each hold one trial's output: the application its
trees are built from. `base-2.zip` is `git archive` of the calibration trial's
commit `f74c0cc`, which reproduces it byte for byte. Each is an input to this
control, not an artifact of it: regenerating one would compare a later run
against a different application, and every number above would silently stop
meaning what it says.

Two things hold it still. The builder refuses any archive whose sha256 is not
Two things hold them still. The builder refuses any archive whose sha256 is not
the one recorded in `control.py`, so a changed base stops the control instead
of quietly shifting what it measures. And every edit the builder makes is an
exact-text swap that refuses when its target is absent, so a base that drifts
Expand Down
Binary file added tests/efficacy/control/base-2.zip
Binary file not shown.
Loading
Loading