From 49b224a0a0245375a519f5164f487378df30f93b Mon Sep 17 00:00:00 2001 From: Branimir Georgiev Date: Thu, 24 Sep 2026 12:55:00 +0300 Subject: [PATCH 1/2] docs(design): rewrite the efficacy benchmark design for its reader MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The owner read the design section by section and found it dense, full of jargon, and out of date after round 3 was set to start anew. This rewrites it in plain language: - plain words, with jargon defined at first use - tables for settings, bullets for steps, a one-line reason after a rule - Mermaid diagrams for the verdict rule, the trial and the scoring pipeline - a worked example and an overview of `tariff` up front - every subsection numbered, and no section with a lone subsection - round history, dates and rejected alternatives moved into one decision log (§10), and the derivations into an appendix (§11) Content now matches the code and the decisions: - five arms and eight contrasts - the anchored judge, read three times, and the control fixture that checks it - only the tools and probes that actually run - the confounds the code already controls - a budget from measured round 1–2 figures - "Arm C runs", "18 runs" and "the evidence check is the only check" are gone Section 12 dissolves into §3, §5, §6 and §10. Code comments that cited §1.1, §11 or §12 now cite §1.3, §3.3 and §6.1. Part of #1767. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/design/efficacy-benchmark.md | 1572 +++++++++++++++-------------- tests/efficacy/generate_arm.py | 12 +- tests/efficacy/report.py | 8 +- tests/efficacy/reuse.py | 2 +- 4 files changed, 803 insertions(+), 791 deletions(-) diff --git a/docs/design/efficacy-benchmark.md b/docs/design/efficacy-benchmark.md index ee536f78..aa339a29 100644 --- a/docs/design/efficacy-benchmark.md +++ b/docs/design/efficacy-benchmark.md @@ -1,823 +1,835 @@ # Efficacy benchmark — does a generated context file improve the result? -**Status:** round 1 run and reported, 2026-09-17 -(`docs/audits/2026-09-17-efficacy.md`); round 2 pre-registered in §12, and -no round 2 trial has run. -**Date:** 2026-09-12; §12 added 2026-09-17 **Owning issue:** #1184, applying the generic method in `testing-ai-assets.md`. -**Artifacts to follow:** the spec and runner under `tests/efficacy/`; the -hidden suite in a private repository (§10); each report as a dated file -under `docs/audits/`. - -## 1. Question, pre-registered - -> Given the same spec, model and tools, does an agent with a -> `solid-ai-templates`-generated `CLAUDE.md` produce a better application -> than the same agent with no context file — and at what cost? - -### 1.1 The verdict rule, fixed before any run - -**Unit of analysis.** The paired difference at trial index *k*. Trials run -interleaved as blocks — A₁ B₁ C₁, then A₂ B₂ C₂ — so the arms within a -block meet the same model on the same day. That block is what makes the -pairing real; without it these would be independent runs and the pairing -would be a convenient fiction. - -**Statistic.** The mean of the K paired differences, per metric, per -contrast. - -**Interval.** A bias-corrected and accelerated bootstrap over those K -differences, 10,000 resamples, 95 %, seed recorded. - -**Direction.** Declared per metric in §6 before the run, because half of -them improve downward. An interval is read against its metric's declared -direction, never against "bigger is better". - -**Verdicts.** **Better** where the interval lies wholly on the improving -side of zero; **worse** where it lies wholly on the other; **no -improvement shown** where it contains zero. "No improvement shown" is not -a claim that the arms are equal — for that, see the non-inferiority rule -in §1.2. - -**What K = 3 can and cannot support.** Three paired differences give a -bootstrap 27 distinct resamples and an exact sign test a minimum two-sided -p of 0.25. No arrangement of three trials reaches conventional -significance, so every interval in the primary run is **descriptive**: it -reports where the effect sits and how unstable it is, and the report says -so beside every row rather than in a footnote. A benchmark that presented -an n = 3 interval as an inferential result would be doing the thing this -document exists to stop. - -**Escalation, pre-set and bounded.** One escalation is permitted, once, -and only where a *primary* dimension's interval contains zero while its -observed effect exceeds that metric's practical threshold in §1.2: run one -further block to K = 5 and report both the K = 3 and the K = 5 vectors. -K never exceeds 5, no metric outside the primary dimensions triggers it, -and a second inconclusive result is reported as inconclusive. Sampling -until an interval clears zero is the failure this bound exists to prevent. - -Any of the three contrasts in §6 can trigger it, and the observed effect — -the mean paired difference — is read by its size in either direction. An -escalation owed only to a favourable effect would be a stopping rule -leaning toward "better". A row whose interval was not computed triggers -nothing. Both readings were settled by the owner on 2026-09-15, before the -first counted trial. The report computes the rule and states whether the -escalation is owed, so nobody reads it off the K = 3 table by eye. - -**Failure handling.** A trial whose agent produced no installable workspace -scores zero on task success, records its quality metrics as missing rather -than imputed, and is **not** re-run: re-running the failures of one arm is -how a control arm is quietly improved. A trial lost to harness or provider -error is re-run once, in its own place in the order, and the substitution -recorded in the report. - -A trial is lost when the CLI ends it with any error other than the budget -cap: a usage limit, a rate limit, a crash, or no result at all. That holds -even where the agent left partial work, because the cut was the provider's -and not the agent's. The partial workspace is kept under the run's `void/` -directory and never scored, and a second loss of the same trial stops the -run. A trial ended by the budget cap or the timeout is not lost: both are -bounds §4 sets, and the trial is scored as it stands. - -### 1.2 Non-inferiority, where the claim is that nothing got worse - -An interval containing zero is absence of evidence. To claim a metric was -*preserved* — which §9 needs for every template trim — the whole -degradation interval must sit inside a margin fixed here: - -| Metric | Margin | Practical threshold for §1.1 escalation | +**Results:** one dated report per round under `docs/audits/`. +**What runs:** the harness, scorer, judge and report under `tests/efficacy/`, +described in its README. The hidden test suite lives in a private +repository (§10). + +## 1. Goal + +Find out whether an agent given a `CLAUDE.md` generated by +`solid-ai-templates` builds a better application than the same agent with no +context file, given the same spec, model and tools, and at what cost. + +**The idea in one example.** The agent builds the same app three times +without a context file and three times with one. Every build is scored on +each metric, so one metric gives three numbers per side. + +Runs are compared with their partner, not side against side. Run 1 without +a file is compared with run 1 with a file, because the two ran on the same +day against the same model. The same goes for runs 2 and 3. The three +differences are what gets averaged. + +| | run 1 | run 2 | run 3 | differences | verdict | +|---|---|---|---|---|---| +| without a file | 0.42 | 0.50 | 0.58 | | | +| with a file | 0.75 | 0.83 | 0.92 | +0.33, +0.33, +0.34 | **better**: every pair improved, by nearly the same amount | +| with a file, a metric that moved less | 0.45 | 0.60 | 0.70 | +0.03, +0.10, +0.12 | **better**: every pair improved, by an unsteady amount | +| with a file, a metric that did not move | 0.55 | 0.44 | 0.62 | +0.13, −0.06, +0.04 | **not shown**: two pairs improved and one got worse | + +Overlapping columns say nothing on their own. The middle row's raw numbers +overlap, yet every pair moved the same way, so it reads **better**. Only the +bottom row, where the differences change sign, is undecided. + +The report prints one such row per metric, the primary dimensions first. The +harness computes each verdict, so nobody decides one by eye. + +### 1.1 Rules, fixed before the first run + +The question, the metrics and the rule for reading a result are written down +before any trial runs. Nobody can then pick the metric or the threshold that +makes the templates look good after seeing the numbers. + +| Setting | Value | Why | |---|---|---| -| Task success, hidden suite pass rate | 2 percentage points | 5 pp | -| Adherence checklist fraction | 5 percentage points | 10 pp | -| Primary subjective dimensions, 1–5 | 0.3 points | 0.5 points | -| Static-analysis counts per KLOC | 10 % relative | 25 % relative | -| Change-task churn, files and lines | 15 % relative | 30 % relative | +| Unit compared | The difference between two arms' trials in the same block: `full-1` minus `none-1` | The arms in a block meet the same model on the same day | +| Result | The mean of the K differences, per metric, per contrast | — | +| Interval | Bias-corrected and accelerated bootstrap over the K differences: 10,000 resamples, 95 %, seed recorded | Shows how steady the effect is | +| Direction | Declared per metric (§6.2) | Half the metrics improve downward | +| Verdict | **Better**: interval wholly on the improving side of zero. **Worse**: wholly on the other side. **No improvement shown**: contains zero | "No improvement shown" does not mean "equal"; §1.2 covers that | + +```mermaid +flowchart LR + D[K paired differences] --> I[bootstrap interval] + I -->|wholly on the improving side| B[Better] + I -->|wholly on the other side| W[Worse] + I -->|contains zero| N[No improvement shown] + N -->|whole interval inside the margin, §1.2| H[Held: nothing got worse] +``` + +### 1.2 Rules for claiming "nothing got worse" + +A trim is a change that shortens the templates. A trim may claim a metric +held only when the whole interval of its change lies inside the margin +below. "No improvement shown" does not mean "no worse". + +| Metric | Unit | Margin: largest tolerated loss | Smallest change that matters | +|---|---|---|---| +| Hidden-test pass rate | percentage points | 2 | 5 | +| Adherence checklist | percentage points | 5 | 10 | +| Primary judge scores, 1–5 | scale points | 0.3 | 0.5 | +| Static-analysis counts per 1,000 lines | relative % | 10 | 25 | +| Change-task churn, files and lines | relative % | 15 | 30 | + +- An adherence gain counts only if pass rate held. Following rules while + breaking the app is not an improvement. +- A trim is free only if every metric held. +- The report shows every metric. Its summary score decides nothing. + +This is a non-inferiority test, as used in drug trials to show a new +treatment is no worse than the standard one by more than a set margin. With +K = 3 it describes the result; it does not prove it. + +### 1.3 Few trials, and the one escalation + +**Three trials are a description, not proof.** + +- Three differences give the bootstrap only 27 distinct resamples. +- No arrangement of three trials reaches conventional significance. +- The report says so beside every interval, not in a footnote. + +**One escalation to K = 5 is allowed, once, under fixed conditions.** + +- It is owed where a primary dimension's interval contains zero while its + mean difference exceeds the smallest change that matters (§1.2). +- Any contrast can trigger it, and the effect counts in either direction. +- One further block runs. The report prints the K = 3 and K = 5 results side + by side. +- K never exceeds 5. A second inconclusive result is reported as + inconclusive. +- The report computes whether the escalation is owed. + +Why: sampling until an interval clears zero is the failure this bound +prevents. + +### 1.4 Failed and lost trials + +**A failed trial is scored as it stands and never re-run.** -A "better" claim on adherence counts only where task success is -non-inferior by this table, and a trim is a free win only where every -quality metric is non-inferior by it — not merely where each interval -happens to contain zero. +- A trial that produced no installable workspace scores zero on task + success. +- Its quality metrics are recorded as missing, never filled in. -The whole vector is reported. The score in §6's summary digests it and -decides nothing. +Why: re-running one arm's failures quietly improves that arm. + +**A lost trial is re-run once, in its own place in the order.** + +- A trial is lost when the CLI ends it with any error other than the budget + cap: a usage limit, a rate limit, a crash, or no result at all. +- Its partial workspace is kept under the run's `void/` directory and never + scored. +- A second loss of the same trial stops the run. +- A trial ended by the budget cap or the timeout is not lost. Both are + bounds §4.1 sets, and the trial is scored as it stands. + +Why: the provider cut it, not the agent. ## 2. The app — `tariff` -A pricing and invoicing engine with a server-rendered web UI. Decided -2026-09-12 over `logsift`, `flowkit`, `ledger` and `convert`: a domain -model where OOP is the natural shape, two orthogonal extension axes so a -missed Strategy is visible, Decimal arithmetic for sharp hidden tests. -Offline, deterministic, ~1,500–2,000 LOC. +### 2.1 What it does -| Concern | What the spec fixes | -|---|---| -| Domain | products with unit prices; discount rules — percentage, bulk (n-for-m), tiered by quantity, coupon code — with declared precedence and stacking; tax jurisdictions with rate tables and rounding mode; an invoice is lines + applied rules + tax + totals | -| Money | `Decimal` throughout, currency fixed to one per invoice, rounding half-even at line level, totals reproducible to the cent — the hidden suite's sharpest edge | -| Public API (Python) | `tariff.Catalog`, `tariff.Invoice`, `tariff.price(invoice, rules, jurisdiction) -> PricedInvoice`; error base `TariffError`; importable without the web app | -| Web UI | Flask + Jinja + HTMX: pages for products, rules, jurisdictions; an invoice builder that re-prices on every change via HTMX partials; invoice preview and export (JSON, CSV, printable HTML) | -| HTTP | routes named in the spec so the hidden suite can drive them with the Flask test client; JSON export endpoint; form posts with server-side validation and CSRF | -| Persistence | SQLite, single file, schema owned by the app; seed fixture in the spec | -| Constraints | no network, Python 3.12, Flask + Jinja + one HTMX script tag; HTML must validate; forms usable without JavaScript | -| Sign-in and customers, from round 3 | one seeded administrator whose password the environment supplies, and every route behind sign-in; customers with a name, email and address, billed on invoices; erasure on request; a JSON export of one customer's data | - -The spec names the API, the routes and the rule semantics so the hidden -suite can drive them. It does not name a layout, linter, type checker, -error hierarchy shape, logging policy, test convention, template -organisation or accessibility bar — those are what the templates add, so -they stay out of the spec. Round 3's additions keep the same line: the spec -names what sign-in, erasure and export do, and never how a password is -stored, what a failed sign-in says, where a redirect may lead, or whether an -erased customer's data leaves the database file. +"Tariff" here means a price list. `tariff` is a small invoicing engine: it +takes a basket of products and computes the final bill, with discounts and +tax. The tax is VAT or sales tax, not customs duty. + +| Piece | What it is | Example | +|---|---|---| +| Product | A SKU, a name, a unit price and a tax category | `COF-250G`, "Coffee", €10.00, category `food` | +| Discount rules | Percentage, bulk ("buy 3, pay 2"), tiered (cheaper unit price from a quantity), coupon code | 10 % off coffee; coupon `WELCOME` takes €5 off | +| Jurisdiction | A country or region with a tax rate per category and a default rate | DE: `food` 7 %, everything else 19 % | +| Invoice | A currency, lines of product × quantity, and coupon codes | 3 × coffee, EUR, `WELCOME` | + +A SKU (Stock Keeping Unit) is the unique code a shop gives each product it +sells. A 1 kg bag of the same coffee would be another SKU. + +`price()` works in a fixed order: line discounts first, then invoice-wide +discounts, then those discounts spread back over the lines, then each line +taxed at its category's rate. Every amount is exact to the cent. + +It suits the benchmark for three reasons: + +- Its domain is naturally modelled with objects and classes. +- It has two independent extension axes, discount kinds and jurisdictions, + so a missed Strategy pattern shows. +- Exact decimal arithmetic makes the hidden tests unforgiving. + +The app is offline and deterministic, about 1,500–2,000 lines of code. + +### 2.2 What the spec fixes + +`tests/efficacy/SPEC.md` names what other software depends on, so the +hidden suite can drive it: + +- **Domain:** products, the four discount kinds with their precedence and + stacking, jurisdictions with rate tables, invoices. +- **Money:** `Decimal` throughout, one currency per invoice, half-even + rounding at named points, totals reproducible to the cent. +- **Python API:** `tariff.Catalog`, `tariff.Invoice`, + `tariff.price(invoice, rules, jurisdiction)`, the error base + `TariffError`. The package imports without the web app. +- **Web UI:** Flask, Jinja and HTMX pages for products, rules and + jurisdictions, and an invoice builder that re-prices on every change. + Pages work with JavaScript disabled. +- **HTTP:** named routes, a JSON export, form posts with server-side + validation and CSRF. +- **Persistence:** one SQLite file, schema owned by the app, a seed fixture. +- **Sign-in and customers:** one seeded administrator and every route + behind sign-in; customers with a name, email and address, billed on + invoices; erasure on request; a JSON export of one customer's data. +- **Constraints:** no network, Python 3.12, valid HTML. + +### 2.3 What the spec leaves out + +**The spec says what the app does, never how it is engineered.** It does not +name: + +- a layout, linter, type checker, logging policy or test convention +- an error hierarchy's shape or a template organisation +- an accessibility bar +- how a password is stored, what a failed sign-in says, or where a redirect + may lead +- whether an erased customer's data leaves the database file + +Why: the "how" is what a context file adds. If the spec said "hash the +password", every arm would, and the benchmark would measure nothing. ## 3. Arms -| Arm | Workspace at start | Purpose | -|---|---|---| -| A control | `SPEC.md`, empty git repo | the bare agent | -| B candidate | A + `CLAUDE.md` generated once via the interview at `v2.90.0` from `Stack: python-flask`, `Extras: stack-htmx, frontend-ux, frontend-quality` (~450 lines, ~21 KB — the same shape as `examples/`) | the product as shipped; no Python web chain carries the frontend rules on its own, so the extras are how a real project would get them | -| C reference | A + a hand-written `CLAUDE.md` of ≤40 lines: layout, `ruff`+`mypy --strict`, tests, one-hierarchy errors, domain free of Flask, escape everything, CSRF on every form | whether the effect is the templates or merely *having* a file | +### 3.1 The five arms -Arm C is the one that can hurt and the one an adopter will ask about; -running it is recommended. +Every arm starts from an empty git repository holding `SPEC.md`. All but +`none` add a `CLAUDE.md`. -These are round 1's arms. Round 2 names every arm by a word, reuses B and C -as `full` and `hand`, and adds two; see §12. +| Arm | Adds | Budget | Asks | +|---|---|---|---| +| `none` | nothing: the bare agent | — | the baseline every file is compared with | +| `full` | the file the interview generates, inline | none | do the templates help as shipped | +| `short` | the file the interview generates, inline, under a budget | ≤ 40 lines, ≤ 88 characters wide | does a short generated file help | +| `hybrid` | the file the interview generates in its hybrid model, with the templates vendored into the workspace | none | does the model the owner's projects use help | +| `hand` | a hand-written file | ≤ 40 lines | is the effect the templates, or merely having a file | -## 4. Protocol +The report prints each file's measured line count. -Per trial (arm × k, K = 3 → 9 trials): - -1. Fresh directory, `git init`, copy the arm's starting files. -2. Isolated agent config: scratch `HOME`/`USERPROFILE` and `--settings` - pointing at a minimal file — no global `CLAUDE.md`, no hooks, no - auto-memory, no MCP. The environment is the machine's, not the - launching process's: the variables of a calling agent session, its - editor, an activated interpreter and any credential are removed, and - the names removed are recorded. Model pinned by exact ID and recorded. - `--effort` fixed. Web tools disallowed; the shell keeps its network, - because every trial installs packages. The templates repository is not - mounted, Git's stored credentials are cleared, and each trial has its - own temporary directory. When a trial ends, whatever it left running is - stopped, and whatever it left in the machine's temporary directory is - moved into its own. -3. One prompt, identical for every arm: - > Implement the application described in SPEC.md. Done means: - > `pip install .` succeeds in a clean virtualenv, your own tests pass, - > and every page and route in the spec works end to end against the - > seed fixture. Commit when done. -4. `claude -p … --output-format json`, bounded per trial by - `--max-budget-usd` and a wall-clock timeout, both fixed: $100 and two - hours. Transcript, token usage, turns and wall time captured, and the - record states which bound ended a trial. This replaces the - `--max-turns N` named when the design was drafted: the installed CLI, - 2.1.153, carries no such flag. Corrected before the first trial, which - is when §7's rule allows the protocol to move at all. - - The budget is read against the CLI's own cost figure, which prices - `claude-sonnet-5` at $5 per million input tokens and $25 per million - output (arm B's generation record reproduces to the cent at those - rates). The CLI checks it between turns, so a trial can pass it by one - turn's cost. The agent that built the hidden suite's reference - implementation, on a stronger model, spent about $13 at that rate in 26 - minutes by its transcript's token counts. At that pace a two-hour trial - stays under the cap, so the timeout is the bound that shapes a trial - and the budget stops a runaway. -5. The workspace is frozen (tarball + commit hash) before scoring. The run - record is rewritten after every trial, so a run stopped part-way keeps - the record of every trial it finished. - -Trials run interleaved (A1, B1, C1, A2, …) so a model-side change mid-run -does not land on one arm. - -### 4.1 Every step is automated, and that is a requirement - -The run is one command. Arm B's generation, the nine build trials, the -nine change tasks, the acceptance suites, the static and structural tools, -the judge and the aggregation are all scripted, and each records its -inputs, its exact invocation and its outputs. No step waits for a person -to answer, choose or paste. - -This is not convenience. A benchmark with a human in its loop cannot be -re-run to check a surprising result, cannot be re-run against a later -template revision to produce the comparison §9 exists for, and quietly -admits the operator's judgement into whichever arm they touched. An -unreproducible measurement of reproducibility would be a poor joke. - -Nothing sits outside the loop, the check on the judge included: it is the -evidence check of §5, run by the judging script, and no person scores -anything. - -## 5. Scoring — the layered judge - -### Backbone (deterministic, per trial, run by the harness) - -| Metric | Measure | Source | -|---|---|---| -| Task success | hidden acceptance suite pass rate, 377 checks across 16 modules: Python API — rule precedence, stacking, tier boundaries, rounding to the cent, jurisdiction tables, invalid rules refused; HTTP — every spec route via the Flask test client, HTMX fragments re-price correctly, JSON/CSV export byte-exact against the seed; browser — 4 Playwright flows; forms work with JavaScript disabled | the private hidden-suite repository (§10), cloned by the harness at scoring time and never into the workspace | -| Install | `pip install .` in a clean venv, `python -c "import tariff"`, app boots and serves `/` | harness | -| Web quality | axe-core WCAG 2.1 AA violations; HTML validity (`html5validator`), less its errors on HTMX's `hx-*` attributes, which `SPEC.md` requires and the HTML standard lacks — owner decision 2026-09-15, taken after reading A1's validator output and before any contrast was computed; XSS probe — a payload in a product name renders inert on every page; CSRF — a form post without a token is refused; response size and request count of the invoice builder | harness, per arm | -| Adherence | fraction of a fixed checklist: `ruff` clean, `mypy --strict` clean, coverage ≥ 80 %, cognitive complexity ≤ 15, `src/` layout, error-contract AST test, NullHandler check, no `print` in library code, citation ban, pyproject metadata complete, README with install+usage, tests discoverable | the python-lib chain's own fenced checks where one exists, else a standard tool | -| Cost | input/output tokens, turns, wall time, API-equivalent $ | harness | -| Scope | files and LOC; count of artifacts nobody asked for (ADR, journal, CHANGELOG, PLAYBOOK) | harness | -| Consistency | spread of every metric above across the three trials of an arm | aggregate | - -Adherence runs on **every arm**: it is a quality checklist, not a -template checklist, so the control can score on it. - -### Code quality (static, per trial) - -Run with the harness's own pinned tool versions and configuration -(`--isolated` / `--config` pointing at the harness file), never the -configuration the agent wrote, so every arm is measured by the same ruler. -Reported as absolute counts and per KLOC. - -**Source discovery is layout-independent, and it has to be.** The arms are -free to choose their layout, and `src/` is one of the things a context -file might introduce, so a tool pointed at a hard-coded `src` reads an -empty directory for a valid flat-layout control and reports no findings. -Zero findings and nothing scanned are the same number and opposite facts. -The harness therefore resolves the roots by importing the installed -package and taking its directory, plus every top-level package directory -in the workspace, and passes those paths explicitly. - -**A tool that scanned nothing, errored, or timed out records the metric as -missing and flags the trial.** It never records zero. Every table below -also reports the file and line count each tool actually saw, so a -suspiciously clean arm can be told from an unmeasured one by reading the -report rather than by rerunning it. - -| Tool | Reports | Delta metric | -|---|---|---| -| `ruff check --select ALL` (fixed ignore list) | violations by category (E/W, F, B, S, D, N, C90, PL, RUF…) | total and per-category count | -| `ruff format --check` | files needing reformat | count | -| `mypy --strict` | errors | count | -| `bandit -r -f json` | findings by severity and confidence | high+medium count | -| `complexipy` | cognitive complexity per function | max, mean, functions > 15 | -| `radon cc` / `radon mi` | cyclomatic complexity, maintainability index per module | mean CC, min MI | -| `interrogate` | docstring coverage % (modules, classes, functions) | coverage | -| `ruff --select D` (pydocstyle) | docstring convention violations | count | -| `vulture` | unused code | count | -| `pytest --cov` | line and branch coverage | % | -| `mutmut` (optional, expensive) | mutation score | % killed | - -The adherence checklist above takes its pass/fail from these where the -same tool answers both; the counts here are the graded view. - -### Design (structural, per trial) - -| Measure | Tool | Delta metric | -|---|---|---| -| Layering: domain imports neither `flask` nor the persistence module; routes hold no pricing logic; templates receive view models, not ORM rows | `import-linter` contracts written by the harness against the spec's roles (domain, persistence, web), mapped to the trial's module names; an AST probe for `Decimal` arithmetic inside route functions | violations | -| Import cycles, fan-in/fan-out, instability and abstractness per module (Martin) | `grimp` | cycles; mean instability of domain modules | -| Size and shape smells: too many arguments/branches/attributes, god class, boolean-flag parameters, inheritance depth | `pylint --disable=all --enable=R` plus a harness AST walk for `bool` parameters | count | -| Class cohesion | `cohesion` | mean | -| Extension points present: a new discount rule kind and a new jurisdiction reachable without editing existing code (Protocol/ABC or registry), and a new export format likewise | harness AST probe | present / absent, per axis | -| Public surface: names exported vs names the spec requires | harness diff of `__all__` and module-level names | extra + missing | -| IO isolation: pricing accepts catalog, rule and jurisdiction values, not database rows or a live connection; export accepts a `PricedInvoice`, not a request | harness signature probe | fraction | - -### Design (change task) — the OCP measure - -The task must lie **outside** the semantics `SPEC.md` already fixes, or it -measures data entry rather than design. The first draft asked for a -buy-one-get-one rule and a jurisdiction with a reduced category rate, and -both are already expressible: buy-one-get-one is the existing bulk rule -with `buy` 2 and `pay` 1, and a reduced category rate is a key in a -jurisdiction's existing `rates` table. Every arm would have scored zero -churn by typing two rows into a form, and the measure would have reported -that all three designs were equally extensible. - -The task adds a **spend-threshold** discount and a **capped reduced -rate**. Its prompt is fixed in -[`tests/efficacy/change-prompt.txt`](../../tests/efficacy/change-prompt.txt), -the only copy, which the harness hands every change trial as it stands. - -Neither is reachable by configuration. The threshold rule is the first -invoice-scoped rule whose application depends on a *predicate over the -invoice*, where every existing kind is either unconditional or gated only -by a coupon code being present. The capped rate is the first tax that is -not one rate times one taxable amount, so it changes the shape of §4.3 -step 12 rather than its inputs. - -Measured: files touched, lines changed, whether the build suite is still -green, whether the change-task acceptance module passes, and whether the -new rule and rate are reachable through the same API and UI. A design that -adds one rule class plus one registry entry, and one rate strategy plus one -entry, scores low churn; one that edits the pricing function, the tax step, -three routes and two templates scores high. - -**Its acceptance module is owed before any change-task run.** Churn is only -interpretable beside a pass: an arm that touched four lines and broke the -extension has not scored well. The module lives with the hidden suite, -tests the two new behaviours against worked figures fixed the same way §9 -fixes the build example, and is written before the first change-task run -for the reason §10 item 8 gives. - -Nine extra runs of ~15 minutes. Each starts from a copy of its own build -trial's frozen workspace, so arm B's change task still carries arm B's -context file, and runs under the same isolation, bounds and outcome rules, -interleaved in the same order. The copy's state is committed before the -agent starts and churn is measured against that commit, so work the build -trial left uncommitted is not counted as the change. A build trial with no -scorable outcome has no change task. - -### Pattern use (judge with evidence, per trial) - -For each GoF pattern found, the judge records: where, what it removes (a -duplication, a conditional ladder) or opens (an extension point), and -whether it has more than one implementor. The patterns the app's two -extension axes invite are Strategy for the discount rule kinds, -registry/Factory for resolving a rule or a jurisdiction by name, -Composite or a precedence chain for stacking rules, Adapter for the JSON, -CSV and printable exports, and Builder for assembling an invoice. The list -is what the judge looks for, not what it must find: a design that reaches -the same extension points another way is scored on the property, not the -name. - -A pattern with one implementor and nothing removed is scored as -over-engineering, per `oop.md`'s own rule; a conditional ladder over rule -kinds where a Strategy was warranted is scored as a missed pattern. -Reported as warranted / missed / over-engineered counts. - -### Subjective (calibrated model judge) - -Rubric, 1–5 each with a quoted evidence line per score: SRP (one reason -to change per module), OCP (rule kinds, jurisdictions and export formats -are extension points), LSP (every rule kind honours the rule contract, and -every export the renderer contract), ISP (public API no larger than the -spec), DIP (domain independent of Flask, SQLite and the request cycle), -naming and abstraction level, error design, readability (function length, -nesting, names that carry the intent), maintainability (how a reader finds -where a change goes), test quality. - -**Primary dimensions, owner-declared:** design (SOLID and pattern use), -readability, maintainability. The report leads with these; task success -and cost follow. Judged by a different vendor from the generator, blind to -arm, order shuffled, condition markers stripped. Judge model ID recorded. - -### How the judge is checked - -Every evidence line the judge quotes is looked up in the bundle it was -quoted from, and the report prints the share found for each trial. A judge -that never opened the code still returns plausible numbers, and the share -of its evidence actually present in the tree is what tells the two apart. - -No person checks the judge. The primary dimensions rest on the model judge -and its evidence check alone, and the report says so. §10 item 9 records -why the human holdout was removed. - -## 6. Aggregation and report - -Per metric: mean per arm, and three paired contrasts with the bootstrap -interval and verdict §1.1 fixes. - -| Contrast | The question it answers | -|---|---| -| B − A | do the templates beat no context file at all | -| C − A | does *any* context file beat none, which is how much of B − A is not the templates | -| **B − C** | do the generated templates beat forty hand-written lines | - -B − C is the one an adopter actually asks and the first draft omitted it, -reporting only each arm against the control. A large B − A beside an -equally large C − A is not a result for the templates, and only B − C -separates them. Round 2's contrasts are in §12. - -**Metric direction, declared before the run.** Improving upward: task -success, adherence, coverage, docstring coverage, mutation score, -extension points present, every subjective rubric score. Improving -downward: every static-analysis count, cognitive and cyclomatic -complexity, tokens, turns, wall time, cost, files, lines, artifacts nobody -asked for, change-task churn, axe violations, HTML invalidity. Neutral, -reported without a verdict: maintainability index, instability. - -The report opened, from 2026-09-17, with a summary table generated from its -own verdicts: per contrast, a score from 1 to 10 and the number of metrics -won and failed. A win or a fail is a metric read better or worse; one -showing no improvement counts neither way. The score is 1 + 9 × wins ÷ -(wins + fails), each metric counting once, and a pair no metric separated -has none. One line of caveats follows: the escalation, withdrawn metrics, -lost trials, reaches past a workspace and the judge's evidence check. - -The owner asked for the score on 2026-09-17, after round one's results were -seen (#1786). It replaces §1.2's earlier "no single headline number". Like a -check declared after a run, it describes the vector and decides nothing: no -verdict, escalation or non-inferiority claim reads it. On 2026-09-18, on -reading round two's report, the owner found the table redundant beside the -finding table (#1820): the score stays only for the contrasts between two -files, which the table has no column for, in one line under it, and the -caveats line follows. - -The finding table opens the report, asked for the same day after the score -table was read (#1789, distilled in #1791). One column per context file -against no file, headed by the file's line count. Rows: whether it improves the code — -Yes where a primary dimension or task success is better and none worse, -Worse where any is worse, No where none is better — the primary dimensions -that moved, the hidden-suite pass rate against the bare arm naming any run -under it by more than §1.2's practical threshold, code size (source lines -and files) and cost as signed shares of the bare arm's where the interval -separated them, and one phrase from the answer and the size and cost -verdicts. One line per file follows, on why it scored so, in words with the -numbers left in the table (#1796, #1798): a clean install that could not -boot, structure built beyond the task and the work it took, the follow-up -change moving through that structure, what the judge read its code as or -that it saw none of it as better, wins that are all tool counts, and -patterns missed — each sentence said only where its record or verdict -holds. One line then reads the columns together: length is not quality -where the shorter file's answer is Yes and the longer's is not, and the -other three pairings have their own words. Every cell and sentence stands -on a verdict or a record below it, and the table decides nothing either. - -The owner's reading of round one, 2026-09-17: the generated file is too -long to add quality, and a short, focused one can. It agrees with -Anthropic's guidance on context engineering, which asks for the smallest -set of high-signal tokens: -[Effective context engineering for AI agents](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents). - -Report file: `docs/audits/YYYY-MM-DD-efficacy.md` with the model IDs, the -CLI versions, the template revision, the arm B brief and its token scan, -K, the bootstrap seed, every trial's raw numbers, every trial that failed -or was re-run and why, the judge's evidence check, and the verdict vector. A -crossing interval is written as "no improvement shown", and every interval -carries the reminder from §1.1 that at K = 3 it is descriptive. - -### Checks declared after a run - -A check chosen after the results were seen can describe a run but never -decide it. Such a check is declared in writing, with its direction, before -any trial is read for it. The report prints it in a section of its own, with -the arm means and the interval and no verdict. It never enters the verdict -vector, a non-inferiority claim or the escalation. A check worth deciding on -is pre-registered for the next run instead. - -The first such checks read round one for security, declared by the owner on -2026-09-16 after that round's pre-registered security checks passed on every -trial. - -### Measurements withdrawn after grading - -A metric whose grader is found, after grading, to measure something other -than what the metric names is withdrawn rather than reported. The withdrawal -is recorded in `tests/efficacy/withdrawn.json` against the grader's revision, -so every run that revision graded loses the metric and a run graded by a -corrected revision keeps it. The report prints the metric as withdrawn with -its reason, and no number, interval or verdict. A grader corrected after the -results were seen is never run on the same trials: the correction is -pre-registered for the next run. - -The first withdrawal is round one's change-task acceptance pass rate, decided -by the owner on 2026-09-17 (#1772). Its modules construct the threshold rule -and the capped rate through names the change prompt never gave, so the rate -recorded which trials guessed them. Churn and the build suite after the change -construct nothing by name, and they stand. - -### In plain terms - -The agent is run three times without templates and three times with. -Every run is scored on each metric. For one metric, that gives three -numbers per side. - -The comparison is **run against its own partner**, not side against side. -The first run of the bare arm is compared with the first run of the -templated one, because the two went out the same day against the same -model; then the second against the second, and the third against the -third. What gets averaged is those differences. - -| | run 1 | run 2 | run 3 | difference | verdict | -|---|---|---|---|---|---| -| without templates | 0.42 | 0.50 | 0.58 | | | -| with templates | 0.75 | 0.83 | 0.92 | +0.33, +0.33, +0.34 | **better** — every pairing improved, and by nearly the same amount | -| with templates, a metric that moved less | 0.45 | 0.60 | 0.70 | +0.03, +0.10, +0.12 | **better** — every pairing improved, but the size is unstable | -| with templates, a metric that did not move | 0.55 | 0.44 | 0.62 | +0.13, −0.06, +0.04 | **not shown** — two runs improved and one got worse | - -An earlier draft of this table read the middle row as "not shown" because -the two rows of raw numbers overlap. That is the wrong test, and it is -worth naming: those three pairings improved by 0.03, 0.10 and 0.12, so -every resampling of them averages to something positive and the interval -cannot contain zero. Overlapping columns say nothing when each pairing -moved the same way. Only the bottom row, where the differences change -sign, is genuinely undecided. - -The "interval" is the mathematical form of "how consistent are the three -differences"; the harness computes it and prints the verdict, so nobody -decides by eye. With only three of them it is a coarse instrument, which -is why §1.1 has the report call it descriptive. - -The report has one such row per metric. Some rows will say better, some -not shown, some worse (tokens, files nobody asked for); all of them are -printed, the primary dimensions first. +### 3.2 Trial names -## 7. Confounds and their controls +A trial is `-`: `short-2` is arm `short`'s trial in block 2, +paired with `none-2`. Blocks run in order and arms interleave within each: +`none-1`, `full-1`, `short-1`, `hybrid-1`, `hand-1`, `none-2`, and so on. -| Confound | Control | -|---|---| -| Your global `CLAUDE.md` and hooks reach the "bare" arm | scratch profile (§4.2); asserted by a dry run that prints the loaded context | -| Agent reads the templates, the hidden suite or the web | no mount; web tools disallowed; Git's stored credentials cleared; every transcript scanned for a tool call naming the templates repository or the hidden suite, and the report names each trial with one. The shell's network stays open, because every trial installs packages, so the scan is the control | -| Model changes between trials | exact ID pinned, trials interleaved | -| Agent sees the acceptance tests | hidden suite lives outside the workspace; harness copies it in only for scoring | -| Judge prefers its own family or the longer output | different vendor, blind, shuffled, length reported | -| The judge reads the workspace and sees which arm it is | the judge runs against a copy with every context file removed — arm B's `CLAUDE.md` names the arm outright, and arm C's does too. The harness asserts the copy carries no `CLAUDE.md` before the judge is called, and the assertion is a refusal, not a warning | -| Bar moved after seeing results | §1.1's verdict rule, §1.2's margins and this document are committed before the first trial, and the escalation to K = 5 is bounded there rather than decided on seeing a result | -| One task measures one task | stated limitation; a second app is the follow-up, not this run | +Round 1 named its arms by letter. Reading its report, A is `none`, B is +`full` and C is `hand`. -## 8. Budget +### 3.3 How the generated files are made -9 build trials at roughly 60–90 minutes each plus 9 change-task runs of -~15 minutes, and 6 more of each if §1.1's single escalation to K = 5 -fires; on the Max plan this is quota, not invoice. 9 judge calls and 3 -human reviews. Hidden-suite and harness authoring: one session, plus -a reference implementation of the specification, which is what proves the -suite can grade anything at all — a grader that has never graded is a -control that has never been exercised. The change task's acceptance module -is a further part-session, owed before the first change-task run. -Total: about three sessions plus wall time. +**One non-interactive invocation per generated arm, and no person answers +questions.** `tests/efficacy/generate_arm.py` hands the generator: -## 9. Reuse as the template benchmark +- `INTERVIEW.md` and the chain resolved at `v2.90.0` +- the project brief, [`tests/efficacy/brief.txt`](../../tests/efficacy/brief.txt) +- the arm's output model (inline or hybrid) and, for `short`, its budget -The same harness measures a template change: control = current templates, -candidate = the trimmed or rewritten version, same paired design, same -verdict rule. That is how "downgrade unjustified MUSTs" (design-notes §19) -stops being an opinion. +A record of each generation is committed beside its file as +`generation.json`: the release and the chain it resolved, the model, the +CLI's result, both leak scans and the budget check. -A trim is a **free win** only where every quality metric is non-inferior -by §1.2's margins and tokens fall. It is not enough that each quality row -reads "no improvement shown": three trials are far too few to detect a -real regression, so a crossing interval is the expected result of a trim -that genuinely broke something, and accepting on that basis would let the -benchmark launder damage as evidence of safety. The test is that the -degradation interval sits inside the margin, which is a claim the data can -fail to support — and where K = 3 cannot support it for a given metric, -the honest reading is that the trim is unproven, not that it is safe. +Why: a conversation would put the answerer's judgement into the file, and a +re-run would produce a different arm. -A trim that turns any design row to "worse" is rejected by the run. +**The length rule.** A file's length is its count of newline-terminated +lines. -Two limits. Each iteration costs a full set of trials, so trims are -batched per file, not per rule. And tuning against one app overfits to -it: `tariff` is the dev set, and a second app (`ledger`, §2 runner-up) -stays sealed for the final v3.0 claim, per the method's dev/test split. +- `short`'s file must be at most 40 lines, none longer than 88 characters. +- The generator refuses a result over either bound rather than trimming it. +- `hybrid` has no budget. The interview's hybrid model fixes what it inlines + and what it refers to. +- `short` and `hand` sit within a line of each other, so `short − hand` + compares content at a fixed length. -## 10. Decisions +Why: trimming by hand would put a person's judgement into the arm. -Every question the harness depends on is decided, 2026-09-12. §11 pins -arm B's project brief, which is the last of them and long enough to own a -section. - -1. The app: `tariff` with a Flask + HTMX UI (§2). -2. Arm C runs. It is the only arm that separates the templates' effect - from the effect of having any file, and the first question an adopter - asks. -3. K = 3, escalating once to 5 under §1.1's bounded rule and never - further. Three is the smallest set an interval can be computed on at - all, and §1.1 states plainly what it cannot support: no arrangement of - three trials reaches conventional significance, so the primary run's - intervals are descriptive and the report says so on every row. The - alternative, sampling until an interval clears zero, is the failure the - bound exists to prevent; the honest cost of K = 3 is that a real effect - can go unshown, and that is reported rather than sampled away. -4. Generator: `claude-sonnet-5` through the `claude` CLI at effort `high`. - Judge: `gpt-6-astra` through `codex exec`, a different vendor and not - merely a different family, which is the stronger form of §7's control. - Both strings are the ones their tools accept, taken from the CLI and - from the Codex model cache rather than from memory, and both go in the - report with the CLI versions. - - Sonnet rather than the strongest available generator: 18 runs is a lot - of quota, it is what a typical adopter runs, and a ceiling effect on a - stronger model would hide the very contribution being measured. - - Both tools bill against a subscription rather than an API account, so - §8's "quota, not invoice" covers the judging too. The judge does not - inherit this machine's Codex configuration: `model_reasoning_effort` is - set explicitly on the command and recorded, because the local default - is `low` and the rubric is a reasoning task over long code. -5. The hidden suite lives in a private repository. This repository is - public, so a suite under `tests/efficacy/` is one search away from any - future trial with web access, and §7's web ban protects only this - run's arms. The harness clones it at scoring time; the spec, runner - and reports stay here. -6. Flask stays. Server-rendered Jinja pages with HTMX partials are its - home ground, and `python-flask` is the chain arm B names. -7. Timing: the baseline runs against `v2.90.0` before the v3.0 split - moves any template, so §9's comparison has a control that predates - the move. The v3.0 plan carries the ordering. -8. The hidden suite is written before any trial runs, against `SPEC.md` - alone, and committed to its repository before the first arm starts. - Written afterwards it would be shaped, consciously or not, by what the - first outputs happened to do, and the bar §1.1 fixes would move with the - results it grades. The cost is a session that produces no result. - - - Done: 377 checks in `braboj/tariff-hidden-suite`, validated - two ways before any arm exists. - A reference implementation written from - the specification alone passes all of them, so a correct implementation - is not marked wrong; a mutation control plants five specification - violations one at a time and the suite catches five. Both live in that - repository and neither reaches a workspace. -9. No human holdout. On 2026-09-12 the owner kept one: three outputs - scored blind on the three primary dimensions, with agreement reported - as an exact-or-adjacent share. On 2026-09-17 the owner removed it - (#1774). Scoring many implementations by hand cannot be sustained, and - a step that waits on a person breaks §4.1. The cost is stated rather - than hidden: the report leads with design, readability and - maintainability, those rows come from the model judge, and §9 gives the - same judge a say in which template trims ship. The judge is checked - only by its evidence lines (§5). - -## 11. Arm B's project brief - -Decided by the owner, 2026-09-12. The brief names the domain's two -extension axes. It is what an adopter would actually type: someone -building this application tells the interview that there are discount -rules and tax jurisdictions, because that is what the application is. -A brief withholding them would measure a version of the product nobody -uses. - -Pinned in advance rather than written on the day, because a brief composed -after the spec is fresh in mind drifts toward it. The brief is -[`tests/efficacy/brief.txt`](../../tests/efficacy/brief.txt), the only -copy, which the generator reads as it stands. It gives the domain in one -line, the stack, and a paragraph on the system's boundaries and actors. - -The stack is named because the spec fixes it for every arm; an interview -left to choose might pick FastAPI, and arm B would then build a different -application from arms A and C. - -The boundary paragraph is there because the interview's job, per the -owner, is to establish a system's boundaries and its interfaces to other -systems and actors. It states the *shape* of those boundaries and no -requirement across them: which actors exist, that the engine is importable -without the web layer, that the store is local and nothing is remote. It -names no route, no API name, no export format and no rule semantics, all -of which arms A and C read in `SPEC.md` and arm B's interview does not. - -### The generation is scripted, not conducted - -Arm B's file is produced by one non-interactive invocation that supplies -`INTERVIEW.md`, the resolved chain at `v2.90.0` and the brief, and -by nothing else. Nobody answers questions as they arrive. - -That is what keeps the arm reproducible: a conversation would put the -answerer's judgement into arm B's file, and re-running it later would -produce a different file and a different arm. The transcript, the exact -prompt and the resulting file are committed with the report, and the -token scan below runs against the result before any trial starts. - -### What the disclosure costs, and where it does not reach - -Naming the axes tells arm B's interview the shape the open-closed measure -scores. The report states this beside the affected rows rather than -leaving a reader to find it: +**`hybrid` reads the templates from its own workspace.** Its workspace +carries this repository's `templates/` directory at `v2.90.0` under +`docs/solid-ai-templates/templates/`. + +- Only that directory is copied. The release also holds this design, and + the arm would read its brief. +- The copy is kept out of the workspace's index and counts in no size or + scope metric. + +### 3.4 What the brief tells the interview + +The brief is what an adopter would type: + +- the domain in one line, naming discount rules and tax jurisdictions +- the stack: Python 3.12, Flask 3, Jinja, HTMX, SQLite, uv, ruff, mypy + strict, pytest +- the system's boundaries and actors: a person in a browser and another + program; the pricing engine importable without the web app; a local + store; no network + +Why each part is there: + +- The two extension axes are named because an adopter building this app + would name them. A brief withholding them measures a product nobody uses. +- The stack is named because the spec fixes it for every arm. An interview + left to choose might pick FastAPI and build a different app. +- The boundaries are there because establishing a system's boundaries and + its interfaces is the interview's job. + +Before round 3 the brief also states that customers carry personal data +(#1767). + +**What naming the axes affects.** The report states this beside the rows it +touches: | Measure | Affected | |---|---| -| Extension points for rule kinds and jurisdictions; the OCP rubric row for those two axes; the change task, whose fixed prompt adds one of each | Yes. Arm B was told these axes exist. Arms A and C read the same axes in `SPEC.md`, so no arm is ignorant of them, but only arm B had them in front of the interview | -| Extension point for a new export format | **No.** Exports are named nowhere in the brief, so this is the one extension axis every arm meets only through the spec | -| Task success, cost, scope, code quality counts, layering, cohesion, complexity, SRP, LSP, ISP, DIP, error design, naming, readability, maintainability, test quality | No. None of them turns on knowing the axes | +| Extension points for rule kinds and jurisdictions; the OCP rubric row for them; the change task | Yes. The generated arms' interview was told the axes exist. The other arms read the same axes in `SPEC.md` | +| Extension point for a new export format | **No.** The brief names no export, so every arm meets this axis only through the spec | +| Everything else: task success, cost, quality counts, layering, complexity, the other rubric rows | No | -The export axis is therefore the uncontaminated open-closed probe, and the -report leads the OCP finding with it. +The export axis is therefore the untouched open-closed probe, and the report +leads its OCP finding with it. -### Still out of bounds +**What the brief never states.** No rule semantics, rounding mode, +precedence or stacking order, route path, API name, error hierarchy or +fixture. -The brief stops at the domain's shape. It states no rule semantics, no -rounding mode, no precedence or stacking order, no route paths, no API -names, no error hierarchy and no fixture. Control, run before the first -trial: the generated context file is scanned for the spec's distinctive -requirement tokens, and a hit refuses the run rather than warning. The -token list is drawn when the brief is pinned, which is now, because -drawing it after seeing the generated file is the drift §10 item 8 exists -to prevent. +**Checked before the first trial:** two scans guard the generated files. -## 12. Round 2 — file length and the context model +- The prompt scan is broad. A generation is never handed `SPEC.md`, so any + spec-only token in the prompt means the wiring is wrong. +- The output scan is narrow. Once the prompt is clean, only data nobody + could derive, such as a seed SKU, a rule id or a worked figure, counts as + a leak. +- A hit refuses the generation rather than warning. -Pre-registered 2026-09-17, after round 1's report and before any round 2 -trial; decided with the owner under #1795. +## 4. Protocol -Round 1 compared a 406-line generated inline file with a 39-line -hand-written one, so it could not tell *generated* from *long*, and it ran -no hybrid file, which is the model the owner's own projects use. Round 2 -holds length fixed and adds the hybrid model. The question, the verdict -rule, the margins, the metrics and their directions are §1 and §6's, -unchanged. +### 4.1 The build trial -### Arms +```mermaid +sequenceDiagram + participant H as Harness + participant W as Workspace + participant A as Agent (claude -p) + H->>W: git init, copy SPEC.md and the arm's CLAUDE.md + H->>A: one prompt, isolated home, pinned model + A->>W: build the app, commit + A-->>H: transcript, tokens, turns, cost + Note over H,A: ends at $100 or two hours + H->>W: freeze: tarball and commit hash +``` -Arms are named by word, and a trial is `-`: `full-1` is arm -`full`'s trial in block 1, paired with `none-1` under §1.1. Round 1's -letters map A → `none`, B → `full`, C → `hand`. Its run root and scoring -area keep the letters on disk, and every reader turns them into the words, -so nothing round 1 froze is rewritten. +Every arm gets one prompt, identical for all: -| Arm | Workspace at start | Lines | Trials | -|---|---|---|---| -| `none` | `SPEC.md`, empty git repo — the bare agent | — | 3, run again in round 2's blocks | -| `full` | `none` + the file generated through the interview at `v2.90.0`, inline, as round 1's arm B | 406 | none: round 1's B₁ B₂ B₃ are reused | -| `short` | `none` + a file generated through the same interview, brief and release, under a 40-line budget | ≤ 40 | 3 | -| `hybrid` | `none` + a file generated through the interview's hybrid model, with the templates vendored into the workspace under `docs/solid-ai-templates/` at `v2.90.0` | about 100, plus the list of template files to read | 3 | -| `hand` | `none` + round 1's hand-written file, as arm C | 39 | none: round 1's C₁ C₂ C₃ are reused | +> Implement the application described in SPEC.md. Done means: +> `pip install .` succeeds in a clean virtualenv, your own tests pass, +> and every page and route in the spec works end to end against the +> seed fixture. Commit when done. -**The length rule.** A file's length is its count of newline-terminated -lines, the figure the finding table prints beside each column. `short`'s -file MUST be at most 40 lines, none longer than 88 characters — the width -the templates keep for their own documents, so that a line cannot carry a -paragraph. It is generated as §11 generates `full`'s — one non-interactive -invocation, the same brief, the same release, its record committed beside -it — with the budget stated in the instruction, and the generator refuses a -result over either bound rather than trimming it, because a trim by hand -would put a person's judgement into the arm. `hybrid`'s inline part carries -no budget; the interview's hybrid model fixes what it inlines and what it -refers to, and its length is reported. `short` and `hand` then sit within a -line of each other, which is what makes short − hand a comparison of -content at a fixed length. - -Same spec, generator (`claude-sonnet-5` at effort `high`), judge -(`gpt-6-astra` at effort `high`), hidden-suite revision (`48f4785`), bounds -and isolation as round 1. K = 3, escalating once under §1.1's rule. Nine -build trials, about $90 at round 1's cost per trial. - -**What the reach scan reads for `hybrid`.** §7's control flags a tool call -naming the templates repository. `hybrid`'s workspace carries the templates -by design, under its own `docs/solid-ai-templates/`, so a call naming that -tree is the arm reading its own file and is not a reach; the scan reads the -vendored tree as part of the workspace. A call naming the repository's URL, -the hidden suite, the scoring area or any other entry of the run root is -flagged as before. The leak scan that guards the judge reads the vendored -tree too, since a bundle carrying it would name the arm; the tree is -stripped from the bundle with the context file. - -### Contrasts - -| Contrast | The question it answers | +**Isolation.** Each trial runs apart from the machine and from the others: + +| What | How | +|---|---| +| Your global `CLAUDE.md`, hooks, memory, MCP | A scratch home and a minimal settings file | +| The launching session's environment | Its variables, editor, active interpreter and credentials are removed; the names removed are recorded | +| The model | Pinned by exact id; effort fixed | +| The web | Web tools disallowed. The shell keeps its network, because every trial installs packages | +| The templates repository | Not mounted | +| Git's stored credentials | Cleared | +| Temporary files | Each trial gets its own directory; entries it left in the machine's are moved into it | +| Leftover processes | Stopped when the trial ends | + +**Bounds.** A trial ends at $100 of the CLI's own cost figure or at two +hours. The timeout is the bound that shapes a trial; the budget stops a +runaway. §11 gives the figures behind them. + +**Records.** Transcript, token usage, turns, wall time and cost are kept, +and the record says which bound ended the trial. The workspace is frozen as +a tarball plus commit hash before scoring. The run record is rewritten after +every trial, so a stopped run keeps every trial it finished. + +### 4.2 The change task + +The change task measures how far each design has to be disturbed to take a +change it was not built for. + +- It starts from a copy of its own build trial's frozen workspace, so each + arm's change task still carries that arm's context file. +- The copy's state is committed first. Churn counts only what the agent + changed after that commit. +- It runs under the same isolation, bounds and outcome rules, interleaved in + the same order. +- There is one per build trial. A build trial with nothing scorable has + none. +- Its prompt is fixed in + [`tests/efficacy/change-prompt.txt`](../../tests/efficacy/change-prompt.txt). + +§5.4 describes the task and how it is scored. + +### 4.3 Everything is automated + +A run is one command. Generating the arms, the build trials, the change +tasks, the hidden suites, the static and structural tools, the judge and the +report are all scripted. Each records its inputs, exact invocation and +outputs. No step waits for a person. + +Why: + +- A result can be re-run to check it. +- A later template revision can be measured against the same run (§9). +- An operator's judgement cannot leak into the arm they touched. + +## 5. Scoring + +Each frozen trial goes through the same pipeline: + +```mermaid +flowchart LR + T[Frozen tree] --> S[Hidden suite] + T --> Q[Static tools] + T --> P[Structural probes] + T --> J[Model judge, blind] + J --> E[Evidence check] + S & Q & P & E --> R[Report] +``` + +### 5.1 Task success and the backbone + +These run on every trial, deterministically. + +**Task success** is the hidden suite's pass rate. The suite lives in a +private repository and is cloned only at scoring time, never into a +workspace. It covers: + +- the Python API: rule precedence, stacking, tier boundaries, rounding to + the cent, jurisdiction tables, invalid rules refused +- every route through the Flask test client, including HTMX fragments that + re-price correctly +- the JSON and CSV exports, byte-exact against the seed +- forms with JavaScript disabled, and browser flows through Playwright + +The rest of the backbone: + +- **Install:** `pip install .` in a clean virtualenv, `import tariff`, and + the app boots and serves `/`. +- **Web quality:** axe-core WCAG 2.1 AA violations; HTML validity less + errors on HTMX's `hx-*` attributes, which the spec requires and the HTML + standard lacks; an XSS probe; a CSRF probe; the invoice builder's + response size and request count. +- **Adherence:** the fraction of a fixed checklist passed — `ruff` clean, + `mypy --strict` clean, coverage ≥ 80 %, cognitive complexity ≤ 15, a + `src/` layout, an error-contract test, a `NullHandler`, no `print` in + library code, no citations, complete package metadata, a README with + install and usage, discoverable tests. +- **Cost:** tokens, turns, wall time, API-equivalent dollars. +- **Scope:** files and lines, and artifacts nobody asked for (ADR, journal, + CHANGELOG, PLAYBOOK). +- **Consistency:** the spread of every metric across an arm's trials. + +Adherence runs on every arm. It is a quality checklist, not a template +checklist, so the bare arm can score on it. + +### 5.2 Code quality + +**One ruler for every arm.** The tools run at the harness's pinned versions +and configuration, never the configuration the agent wrote. Counts are +reported absolute and per 1,000 lines. + +**Finding the code.** Each arm picks its own layout, so the tools never +assume `src/`. + +- The harness imports the installed package and takes its directory. +- It adds every top-level package directory in the workspace. +- It passes those paths to every tool explicitly. + +Why: a tool pointed at an empty `src/` reports zero findings, and "zero +findings" must never mean "nothing scanned". + +**A tool that scanned nothing, failed or timed out records the metric as +missing, never zero.** The report shows how many files and lines each tool +saw, so a clean arm can be told from an unmeasured one. + +| Tool | Measures | +|---|---| +| `ruff check --select ALL` | violations, by category | +| `ruff format --check` | files needing reformatting | +| `mypy --strict` | type errors | +| `bandit` | security findings, high and medium | +| `complexipy` | cognitive complexity: max, mean, functions over 15 | +| `radon` | cyclomatic complexity and maintainability index | +| `interrogate` | docstring coverage | +| `vulture` | unused code | +| `pytest --cov` | line and branch coverage of the agent's own tests | + +Not measured: mutation score, class cohesion, and pylint's refactoring +checks. + +### 5.3 Design structure + +The harness probes each tree's structure directly: + +- **Layering:** the domain imports neither Flask nor the persistence + module, and routes do no `Decimal` arithmetic. +- **Import graph:** mutual imports, and the mean instability of domain + modules (Martin), through `grimp`. +- **Extension points:** can a new discount kind, jurisdiction or export + format be added without editing existing code? Present or absent, per + axis. +- **Kind ladders:** conditionals that dispatch on a rule's kind. +- **Boolean parameters:** a branch the caller cannot name. +- **Public surface:** names exported against names the spec requires. +- **IO isolation:** pricing takes values, not database rows or a + connection. +- **Hygiene:** `print` calls, exception bases, citations, logging set up + with a `NullHandler`. + +### 5.4 The change task: the open-closed measure + +**The task must lie outside what the spec already allows.** Otherwise it +measures data entry, not design. It adds two things: + +- a **spend-threshold discount**: 5 % off once the subtotal reaches 200.00, + applied before coupons +- a **capped reduced rate**: one tax category taxed at the reduced rate on + the first 50.00 of a line and at the standard rate above it + +Neither is reachable by configuration: + +- The threshold rule is the first invoice rule that depends on a condition + over the invoice. Every existing kind is unconditional or gated only by a + coupon code. +- The capped rate is the first tax that is not one rate times one amount, + so it changes the shape of the tax step, not its inputs. + +**Measured:** files touched, lines changed, whether the build suite is +still green, whether the change-task acceptance module passes, and whether +the new rule and rate show in the same API and UI. A new rule class plus a +registry entry scores low churn. Editing the pricing function, the tax +step, three routes and two templates scores high. + +**The acceptance module is written before any change-task run.** It tests +the two new behaviours against worked figures. + +Why: churn means something only beside a pass. An arm that touched four +lines and broke the extension has not scored well. + +### 5.5 Pattern use + +For each design pattern found, the judge records where it is, what it +removes or opens, and whether it has more than one implementation. + +- **Invited by the app:** Strategy for the discount kinds, a registry or + Factory to resolve a rule or jurisdiction by name, Composite or a + precedence chain for stacking, Adapter for the exports, Builder for an + invoice. +- **Scored on the property, not the name:** a design that reaches the same + extension points another way scores the same. +- **Over-engineering:** a pattern with one implementation that removes + nothing, per `oop.md`. +- **Missed:** a conditional ladder over rule kinds where a Strategy was + warranted. + +Reported as warranted, missed and over-engineered counts. + +### 5.6 The model judge + +**The rubric.** Each row is scored 1–5 with a quoted evidence line: SRP, +OCP, LSP, ISP, DIP, naming and abstraction level, error design, +readability, maintainability, and test quality. + +**Primary dimensions:** design (SOLID and pattern use), readability and +maintainability. The report leads with them; task success and cost follow. +Security and data protection join them from round 3 (§5.8). + +**How it judges:** + +- **Anchored:** each primary describes its 1, 3 and 5 in terms of this + domain. A scale saying only "1 is poor, 5 is excellent" put 3 on ordinary + work every time. +- **Three times:** each tree is judged three times and the scores are + averaged. One judging is not reproducible. +- **Blind:** the context file and any condition marker are stripped, the + order is shuffled at a recorded seed, and a bundle that still names its + arm is not judged. +- **A different vendor:** `gpt-6-astra` through `codex exec`, while the + generator is Claude. The model id and reasoning effort are recorded. + +### 5.7 How the judge is checked + +No person scores the judge. Two automated checks do: + +- **The evidence check:** every line the judge quotes is looked up in the + bundle it was quoted from, and the report prints the share found per + trial. A judge that never opened the code still returns plausible + numbers; this is what tells the two apart. +- **The control fixture:** six trees built from one pinned application + (`tests/efficacy/control/`). Every tree passes the same tests, so only + structure separates them. Some are damaged, some improved. A primary + dimension counts only if it falls on the damaged trees and rises on the + improved ones. + +A second judge, Claude through the `claude` CLI, reads the control fixture +only, never a round, and its scores are never averaged with the first +judge's. + +### 5.8 Security and data protection + +**Now:** `tests/efficacy/security.py` reads each trial for hard-coded secret +keys, debug left on, SQL built from strings, vulnerable dependencies, +session cookie flags, security headers, and stack traces in error answers. +These checks were declared after a run, so they describe and never decide +(§6.4). + +**From round 3 they become primary dimensions** (#1767). The scoring adds +probes for: + +- a plaintext password at rest +- a sentinel email absent from the logs +- erasure removing the sentinel from the database file +- sign-in failures that do not reveal which accounts exist +- no open redirect after sign-in +- CSRF on sign-in + +Their metrics, directions, margins and escalation thresholds are fixed here +before round 3's first trial. Each needs an anchored judge row that passes +the control fixture. + +## 6. Report + +### 6.1 Contrasts + +Each metric is reported per arm and for eight paired contrasts: + +| Contrast | Asks | +|---|---| +| `full − none` | do the templates help as shipped | +| `short − none` | does a short generated file help | +| `hybrid − none` | does the hybrid model help | +| `hand − none` | does any file help | +| `full − hand` | do the templates beat forty hand-written lines | +| `short − hand` | at equal length, can the templates pick the right forty lines | +| `hybrid − full` | does the hybrid model beat the same content inline | +| `short − full` | does length matter | + +`full − hand` is the question an adopter asks. A large `full − none` beside +an equally large `hand − none` is not a result for the templates. + +**Reading `short − hand`:** + +- `short` ≈ `hand` (no improvement shown, every quality metric held): the + templates' content is sound, and the interview emits too much. +- `short` worse than `hand` on a primary dimension: the content itself is + the defect. + +**A contrast pairing trials from different rounds gets no verdict.** It keeps +its mean and interval and counts as neither a win nor a fail. + +### 6.2 Metric directions + +- **Better upward:** task success, adherence, coverage, docstring coverage, + extension points present, every judge score. +- **Better downward:** every static-analysis count, cognitive and cyclomatic + complexity, tokens, turns, wall time, cost, files, lines, artifacts nobody + asked for, change-task churn, axe violations, HTML errors. +- **Neutral, reported without a verdict:** maintainability index, + instability. + +### 6.3 What the report prints + +The report is `docs/audits/YYYY-MM-DD-efficacy.md`. + +**It opens with the finding table.** One column per context file against +`none`, headed by the file's line count: + +- **Improves the code?** Yes where a primary dimension or task success is + better and none worse; Worse where any is worse; No where none is better. +- the primary dimensions that moved +- the hidden-suite pass rate against `none`, naming any run below it by more + than the smallest change that matters +- code size and cost, as signed shares of `none`'s, where the interval + separated them +- a one-phrase reading + +Under the table: + +- one line per file on why it scored so, in words +- one line reading the columns together +- one line scoring the contrasts between two files, 1 to 10: + 1 + 9 × wins ÷ (wins + fails). A win or fail is a metric better or worse; + "no improvement shown" counts neither way. +- one line of caveats: the escalation, withdrawn metrics, lost trials, + reaches past a workspace, and the judge's evidence check + +The table and the score describe the verdicts and decide nothing. Every cell +rests on a verdict or record below it. + +**Then the record:** model ids, CLI versions, the template revision, each +generated arm's brief and token scan, K, the bootstrap seed, every trial's +raw numbers, every trial that failed or was re-run and why, the judge's +evidence check, and the verdict vector. + +### 6.4 Checks declared after a run + +**A check chosen after the results were seen can describe a run but never +decide it.** + +- It is declared in writing, with its direction, before any trial is read + for it. +- The report prints it in its own section, with means and intervals and no + verdict. +- It never enters the verdict vector, a "nothing got worse" claim or the + escalation. +- A check worth deciding on is fixed before the next run instead. + +Why: otherwise the bar moves with the results. + +### 6.5 Withdrawn measurements + +**A metric whose grader turns out to measure something else is withdrawn, +not reported.** + +- The withdrawal is recorded in `tests/efficacy/withdrawn.json` against the + grader's revision. +- Every run that revision graded loses the metric; a run graded by a + corrected revision keeps it. +- The report names the metric and the reason, with no number, interval or + verdict. +- A corrected grader never re-grades the same trials. The correction is + fixed before the next run. + +## 7. Confounds and their controls + +A confound is a second cause mixed up with the one being tested, so the +result cannot say which produced it. A control is what rules it out. + +| Confound | Control | |---|---| -| short − none | does a short generated file help | -| hybrid − none | does the model the owner's projects run help | -| short − hand | can the templates pick the right forty lines, length held fixed | -| hybrid − full | does the hybrid model beat the same content inline | - -**Reading rule, fixed now.** short ≈ hand — short − hand shows no -improvement and every quality metric is non-inferior by §1.2 — means the -templates' content is sound and the interview over-emits. short worse than -hand on a primary dimension means the content is the defect. The finding -table (§6) takes one column per file arm against `none`, so full − none -and hand − none are printed again, each pairing a round 1 trial with round -2's `none`. - -**What crosses rounds, stated.** §1.1's pairing rests on the arms within a -block meeting the same model on the same day. short − none and -hybrid − none hold that: both arms run in round 2's blocks. short − hand -and hybrid − full pair a round 2 trial with one round 1 ran on 2026-09-15 -or 16, and so do the finding table's `full` and `hand` columns. The model -is pinned by exact ID, so the day is the residual confound; the report -states it beside those rows rather than in a footnote. - -**A crossing contrast carries no verdict.** Adopted 2026-09-18, after round -2's report was first rendered, committed and posted on #1184. A contrast -pairing an arm judged in one round with an arm judged in another keeps its -mean and its interval and is given no verdict, and counts as neither a win -nor a fail. Stating the day beside a verdict does not stop the verdict from -being read; `full` did not change between the rounds — its judgings are -round 1's records, reused — so the −0.667 that read as Worse was `none` -moving, not the file. The amendment withholds a reading, changes no metric, -no direction and no interval, and is why round 2's `full` and `hand` columns -read not measured where they read Worse and No. - -**Each trial is judged three times and the scores are meaned.** Adopted -2026-09-19, before round 3. One judging of a tree is not reproducible: six -calls on one byte-identical bundle returned readability 4, 4, 4, 3, 4, 4 — -mean 3.83, sample SD 0.41, roughly 0.33 standard error on a difference of -two arm means at K = 3. Readability is the only primary that moved in -either round, so every verdict both reports carry rests on the row that is -not reproducible, and round 2's −0.667 is about two standard errors of the -judge resampling itself. Three judgings cut that standard error by about -40 % for three times the judge quota and no extra trials. Eight of the -eleven rows returned the same integer on all six calls, so this buys -precision on the three that do not: `readability`, `lsp` and -`test_quality`. N is fixed at three here, before the round, and the report -states the count it actually had per trial — a round that judged unevenly -says so rather than averaging the fact away. - -**The three primary dimensions are anchored.** Adopted 2026-09-19, before -round 3, and validated by the control rather than argued. Each of `design`, -`readability` and `maintainability` describes its 1, 3 and 5 in terms of -the domain the submissions implement, because a scale saying only that 1 is -poor and 5 is excellent put 3 wherever ordinary work landed: `design` and -`maintainability` returned 3 on all 24 trials of rounds 1 and 2, fell a -point when the control damaged the code, and did not move when it improved -it. A prompt change means round 3's scores are not comparable to the -rounds before it on these rows; what carries forward is the form, so an -earlier round's trees can be re-judged into the anchored scale if that -comparison is wanted. - -**What is not run.** The change task is not part of round 2. Its -acceptance grader was withdrawn (§6) and its corrected prompt is -pre-registered for round 3 (#1767), so every change-task row reads not -computed for this round's contrasts; round 1's change-task churn stands in -round 1's report. Security is read after the run as in round 1 (§6, -"Checks declared after a run"), and decides nothing. - -Round 3 (#1767) extends the specification with security and data -protection as primary dimensions and runs on this round's winning arm -against `none`. Which arm that is gets read off this round's finding table -and fixed in #1767 before round 3's first trial. - -The brief and the scan's result are committed beside the report. +| Your global `CLAUDE.md` or hooks reach the bare arm | A scratch home per trial; the harness refuses a run when a `CLAUDE.md` exists in that home or any parent directory | +| The agent reads the templates, the hidden suite or the web | Nothing mounted; web tools off; Git credentials cleared; every transcript scanned for a tool call naming them, and the report names each trial with one | +| `hybrid` legitimately carries the templates | Its reads of its own vendored copy are not a reach; naming the repository by URL or owner still is | +| The model changes between trials | Exact id pinned; trials interleaved | +| The agent sees the acceptance tests | The hidden suite stays outside the workspace until scoring | +| The judge favours its own family or longer output | A different vendor from the generator; blind; shuffled; length reported | +| The judge sees which arm it reads | Context files, the vendored templates and condition markers are stripped; a bundle still naming its arm is refused | +| The judge's score is noise | Three judgings per tree, averaged | +| The judge's scale cannot move | Anchored primaries, validated by the control fixture (§5.7) | +| A grader measures something else | Withdrawn measurements (§6.5) | +| A trial is lost to the provider | One re-run in place (§1.4) | +| The spec changes between rounds | No trial is reused across spec versions | +| The bar moves after seeing results | The rules in §1 and this document are committed before the first trial | +| One task measures one task | A stated limitation; `ledger`, a second app, stays sealed for the final claim | + +## 8. Budget + +Round 3 runs five arms at K = 3. Figures per trial are measured in rounds 1 +and 2. + +| Item | Count | Per item | Round total | +|---|---|---|---| +| Build trials | 15 | 16–28 min, $8.7–$16.7 | about 4–7 hours, about $195 | +| Change tasks | 15 | $5–$17 (round 1) | about $130 | +| Judgings | 45 (15 trees × 3) | — | ChatGPT plan quota | +| Escalation to K = 5, if owed | +10 build, +10 change, +30 judgings | as above | about $215 more | + +Both CLIs bill against a subscription, so this is quota, not invoice. + +## 9. Reuse as the template benchmark + +The same harness measures a template change. The **baseline** is the current +templates; the **candidate** is the trimmed or rewritten version. Same +paired design, same rules. + +Round 3 is that baseline for v3.0: v3.0's evidence phase re-runs this +harness with the fork's chain and compares against it. + +**A trim is a free win only where every quality metric held (§1.2) and +tokens fell.** + +- "No improvement shown" on each row is not enough. Three trials are too + few to detect a real regression. +- Where K = 3 cannot support "held" for a metric, the trim is unproven, not + safe. +- A trim that turns any primary dimension worse is rejected. + +Why: accepting on "no improvement shown" would pass a trim that broke +something. + +Two limits: + +- Each iteration costs a full set of trials, so trims are batched per file, + not per rule. +- Tuning against one app overfits to it. `tariff` is the development app; + `ledger` stays sealed for the final v3.0 claim. + +## 10. Decisions + +| Date | Decision | Why | Issue | +|---|---|---|---| +| 2026-09-12 | The app is `tariff`, chosen over `logsift`, `flowkit`, `ledger` and `convert` | Natural object model, two extension axes, sharp decimal tests | #1712 | +| 2026-09-12 | The hand-written arm runs | Only it separates the templates from having any file | #1705 | +| 2026-09-12 | K = 3, one bounded escalation to 5 | The smallest set an interval can be computed on; the bound stops sampling to significance | #1705 | +| 2026-09-12 | Generator `claude-sonnet-5` at effort `high`; judge `gpt-6-astra` through `codex exec`, effort set explicitly | What a typical adopter runs; a stronger model's ceiling would hide the effect; a different vendor judges | #1711 | +| 2026-09-12 | The hidden suite lives in the private `braboj/tariff-hidden-suite` | This repository is public, so a suite here is one search away from a future trial | #1705 | +| 2026-09-12 | Flask stays | Server-rendered Jinja with HTMX is its home ground | #1705 | +| 2026-09-12 | The baseline runs against `v2.90.0`, before the v3.0 split moves any template | §9's comparison needs a baseline older than the move | #1705 | +| 2026-09-12 | The hidden suite is written and validated before any trial | Written afterwards, it would be shaped by the first outputs (§11) | #1714, #1715 | +| 2026-09-12 | The brief names the two extension axes | An adopter would; §3.4 states what that costs | #1713 | +| 2026-09-12 | The change task adds a spend threshold and a capped rate, replacing a first draft of buy-one-get-one and a reduced rate | Both first-draft items were already expressible by configuration, so every arm would score zero churn | #1717 | +| 2026-09-12 | Trials are bounded by `--max-budget-usd` and a timeout | The installed CLI (2.1.153) has no `--max-turns` | #1710 | +| 2026-09-15 | HTML validity sets aside errors on `hx-*` attributes | The spec requires HTMX; otherwise the metric counts how much HTMX a trial uses | — | +| 2026-09-15 | The escalation reads the effect in either direction, on any contrast; a row with no interval triggers nothing | An escalation owed only to a favourable effect would lean toward "better" | — | +| 2026-09-16 | Security and data protection become primary dimensions from round 3 | Owner decision after reading round 1's report | #1767 | +| 2026-09-16 | Round 1 is read for security with checks declared after the run | Its fixed security checks passed on every trial and separated nothing | #1766 | +| 2026-09-17 | The report opens with a 1–10 score per contrast, replacing "no single headline number" | Asked for after reading round 1 | #1786 | +| 2026-09-17 | The finding table opens the report | Asked for after reading the score table | #1789, #1791, #1796, #1798 | +| 2026-09-17 | Round 1's change-task pass rate is withdrawn | Its grader built the new rule and rate through names the prompt never gave | #1772 | +| 2026-09-17 | No human holdout | Scoring many implementations by hand cannot be sustained, and a person in the loop breaks §4.3 | #1774 | +| 2026-09-17 | Round 2 holds length fixed and adds the hybrid model | The owner read round 1 as: the generated file is too long to add quality, and a short one can. This agrees with Anthropic's [context engineering guidance](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) | #1795 | +| 2026-09-18 | The score table stays only for contrasts between two files | Beside the finding table it repeated itself | #1820 | +| 2026-09-18 | A contrast crossing rounds gets no verdict | Round 2's `full` read Worse because `none` moved between rounds, not the file | #1826 | +| 2026-09-19 | The primaries are anchored and checked by the control fixture | Unanchored, `design` and `maintainability` scored 3 on all 24 trials of rounds 1–2 | #1827, #1829, #1830 | +| 2026-09-19 | Each tree is judged three times | Six judgings of one unchanged tree disagreed (§11) | #1830 | +| 2026-09-19 | A second judge, Claude, reads the control fixture only | A second vendor's reading of the same trees; never averaged with the rounds' judge | #1831 | +| 2026-09-24 | Round 3 starts anew with all five arms and reuses no trial | The spec gained sign-in and customers, and the judge changed | #1767 | +| 2026-09-24 | Eight contrasts, adding `short − full` | Does length matter | #1767 | +| 2026-09-24 | The brief and the change prompt live in files of their own | The design's restructure would have broken the code that read them by heading | #1843 | + +## 11. Appendix — the numbers behind the rules + +**Why three trials cannot reach significance.** Three paired differences +give a bootstrap 27 distinct resamples and an exact sign test a minimum +two-sided p of 0.25. + +**Why the judge reads each tree three times.** Six judgings of one unchanged +bundle returned readability 4, 4, 4, 3, 4, 4: mean 3.83, sample SD 0.41, +about 0.33 standard error on a difference of two arm means at K = 3. +Readability was the only primary that moved in rounds 1 and 2, so every +verdict rested on the row that was not reproducible. Three judgings cut +that standard error by about 40 %. Eight of the eleven rows returned the +same score on all six calls. + +**Why the budget is $100 and the timeout two hours.** The CLI prices +`claude-sonnet-5` at $5 per million input tokens and $25 per million output. +It checks the budget between turns, so a trial can pass it by one turn's +cost. The agent that built the hidden suite's reference implementation, on +a stronger model, spent about $13 at that rate in 26 minutes. At that pace a +two-hour trial stays under the cap. + +**How the hidden suite was validated.** + + +377 checks across 16 modules in `braboj/tariff-hidden-suite`, validated two +ways before any arm existed. + + +- A reference implementation written from the spec alone passes every + check, so a correct implementation is not marked wrong. +- A mutation control plants five spec violations one at a time, and the + suite catches all five. + +Round 3 extends the suite to sign-in and customers, and both validations +run again before its first trial (#1767). diff --git a/tests/efficacy/generate_arm.py b/tests/efficacy/generate_arm.py index 5ba80c61..893de393 100644 --- a/tests/efficacy/generate_arm.py +++ b/tests/efficacy/generate_arm.py @@ -1,9 +1,9 @@ """Generate one arm's context file, once, without anybody answering anything. -The design's section 11 fixes what the interview is told and requires the -generation to be one non-interactive invocation; section 12 adds the -40-line budget arm `short` is generated under and the hybrid model arm -`hybrid` asks for. This module is that invocation. It resolves the chain at +The design's section 3.3 fixes what the interview is told, requires the +generation to be one non-interactive invocation, and sets the 40-line +budget arm `short` is generated under and the hybrid model arm `hybrid` +asks for. This module is that invocation. It resolves the chain at the recorded release, builds the prompt from `INTERVIEW.md`, that chain, the pinned brief and the arm's model and budget, calls the generator, writes `arms//CLAUDE.md`, and refuses to keep a result the @@ -43,7 +43,7 @@ # The arms the interview generates, each with the directory under `arms/` # its file and record live in, the interview's output model it is asked for, -# and its budget as (lines, width) where the design's section 12 sets one. +# and its budget as (lines, width) where the design's section 3.3 sets one. # Each directory is named after its arm. A record written before the rename # still gives the path the file was generated to, as a record should. GENERATED = { @@ -160,7 +160,7 @@ def record_path(arm): % (VENDORED["into"], VENDORED["path"], VENDORED["path"])), } -# The budget clause, stated in the instruction as the design's section 12 +# The budget clause, stated in the instruction as the design's section 3.3 # requires. The width keeps a line from carrying a paragraph. BUDGET = """\ The finished `CLAUDE.md` MUST be at most %d lines, blank lines counted, and diff --git a/tests/efficacy/report.py b/tests/efficacy/report.py index 560a5bf5..e4dfca6f 100644 --- a/tests/efficacy/report.py +++ b/tests/efficacy/report.py @@ -47,8 +47,8 @@ RESAMPLES = 10000 CONFIDENCE = 0.95 -# Every contrast the design declares, each a treatment arm against a -# baseline: round 1's three, then round 2's four (design, section 12). A run +# Every contrast the design declares in section 6.1, each a treatment arm +# against a baseline. A run # prints the contrasts whose arms it holds; the rest are computed as not # computed and left out of every table. full - hand is the one an adopter # asks: a large full - none beside an equally large hand - none is not a @@ -691,7 +691,7 @@ def escalation_triggers(results): if low is None or high is None or not contrast["pairs"]: continue - # Read by size in either direction, as the design's section 1.1 + # Read by size in either direction, as the design's section 1.3 # fixes: an escalation owed only to a favourable effect would lean # the stopping rule toward "better". effect = statistics.fmean(contrast["pairs"]) @@ -1530,7 +1530,7 @@ def finding_lines(trials, table, results, active): lines.append("") # A column or a contrast pairing a reused trial with one run in this - # round crosses rounds, which the design's section 12 has the report say + # round crosses rounds, which the design's section 6.1 has the report say # here rather than in a footnote. reused = reused_arms(trials) if reused: diff --git a/tests/efficacy/reuse.py b/tests/efficacy/reuse.py index 6896696e..620ec43c 100644 --- a/tests/efficacy/reuse.py +++ b/tests/efficacy/reuse.py @@ -2,7 +2,7 @@ they were. Round 2 reuses round 1's `full` and `hand` trials rather than running them -again, as the design's section 12 fixes. This module copies what the report +again, as the design's decision log records. This module copies what the report reads for a trial — its run record, frozen tarball, transcripts, score, judging and security reading — from the earlier root and scoring area into the new ones, under the arm's word, and marks each copy with where it came From 100131cfe3308bf7079c5c29d4bfa83571f4fcf9 Mon Sep 17 00:00:00 2001 From: Branimir Georgiev Date: Thu, 24 Sep 2026 12:59:32 +0300 Subject: [PATCH 2/2] docs(design): split the design's six densest passages The adherence and web-quality bullets, the report's record, the judge-noise paragraph, one long sentence and two long table cells become sub-bullets, lists and shorter cells. The context-engineering link moves out of the decision log to a note under it. Part of #1767. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/design/efficacy-benchmark.md | 69 ++++++++++++++++++++----------- 1 file changed, 44 insertions(+), 25 deletions(-) diff --git a/docs/design/efficacy-benchmark.md b/docs/design/efficacy-benchmark.md index aa339a29..530d4472 100644 --- a/docs/design/efficacy-benchmark.md +++ b/docs/design/efficacy-benchmark.md @@ -416,15 +416,19 @@ The rest of the backbone: - **Install:** `pip install .` in a clean virtualenv, `import tariff`, and the app boots and serves `/`. -- **Web quality:** axe-core WCAG 2.1 AA violations; HTML validity less - errors on HTMX's `hx-*` attributes, which the spec requires and the HTML - standard lacks; an XSS probe; a CSRF probe; the invoice builder's - response size and request count. -- **Adherence:** the fraction of a fixed checklist passed — `ruff` clean, - `mypy --strict` clean, coverage ≥ 80 %, cognitive complexity ≤ 15, a - `src/` layout, an error-contract test, a `NullHandler`, no `print` in - library code, no citations, complete package metadata, a README with - install and usage, discoverable tests. +- **Web quality:** + - accessibility: axe-core WCAG 2.1 AA violations + - HTML validity, less errors on HTMX's `hx-*` attributes, which the spec + requires and the HTML standard lacks + - an XSS probe and a CSRF probe + - the invoice builder's response size and request count +- **Adherence:** the share of a fixed checklist passed. + - Tools: `ruff` clean, `mypy --strict` clean, coverage ≥ 80 %, cognitive + complexity ≤ 15. + - Structure: a `src/` layout, an error-contract test, a `NullHandler`. + - Hygiene: no `print` in library code, no citations. + - Packaging: complete metadata, a README with install and usage, + discoverable tests. - **Cost:** tokens, turns, wall time, API-equivalent dollars. - **Scope:** files and lines, and artifacts nobody asked for (ADR, journal, CHANGELOG, PLAYBOOK). @@ -449,9 +453,10 @@ assume `src/`. Why: a tool pointed at an empty `src/` reports zero findings, and "zero findings" must never mean "nothing scanned". -**A tool that scanned nothing, failed or timed out records the metric as -missing, never zero.** The report shows how many files and lines each tool -saw, so a clean arm can be told from an unmeasured one. +**A tool that scanned nothing records the metric as missing, never zero.** +The same holds for a tool that failed or timed out. The report shows how +many files and lines each tool saw. A clean arm can then be told from an +unmeasured one. | Tool | Measures | |---|---| @@ -667,10 +672,15 @@ Under the table: The table and the score describe the verdicts and decide nothing. Every cell rests on a verdict or record below it. -**Then the record:** model ids, CLI versions, the template revision, each -generated arm's brief and token scan, K, the bootstrap seed, every trial's -raw numbers, every trial that failed or was re-run and why, the judge's -evidence check, and the verdict vector. +**Then the record:** + +- model ids, CLI versions and the template revision +- each generated arm's brief and token scan +- K and the bootstrap seed +- every trial's raw numbers +- every trial that failed or was re-run, and why +- the judge's evidence check +- the verdict vector ### 6.4 Checks declared after a run @@ -709,7 +719,7 @@ result cannot say which produced it. A control is what rules it out. | Confound | Control | |---|---| | Your global `CLAUDE.md` or hooks reach the bare arm | A scratch home per trial; the harness refuses a run when a `CLAUDE.md` exists in that home or any parent directory | -| The agent reads the templates, the hidden suite or the web | Nothing mounted; web tools off; Git credentials cleared; every transcript scanned for a tool call naming them, and the report names each trial with one | +| The agent reads the templates, the hidden suite or the web | Nothing mounted; web tools off; Git credentials cleared; transcripts scanned, and each hit named in the report | | `hybrid` legitimately carries the templates | Its reads of its own vendored copy are not a reach; naming the repository by URL or owner still is | | The model changes between trials | Exact id pinned; trials interleaved | | The agent sees the acceptance tests | The hidden suite stays outside the workspace until scoring | @@ -788,7 +798,7 @@ Two limits: | 2026-09-17 | The finding table opens the report | Asked for after reading the score table | #1789, #1791, #1796, #1798 | | 2026-09-17 | Round 1's change-task pass rate is withdrawn | Its grader built the new rule and rate through names the prompt never gave | #1772 | | 2026-09-17 | No human holdout | Scoring many implementations by hand cannot be sustained, and a person in the loop breaks §4.3 | #1774 | -| 2026-09-17 | Round 2 holds length fixed and adds the hybrid model | The owner read round 1 as: the generated file is too long to add quality, and a short one can. This agrees with Anthropic's [context engineering guidance](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) | #1795 | +| 2026-09-17 | Round 2 holds length fixed and adds the hybrid model | Owner's reading of round 1: the generated file is too long to add quality, and a short one can¹ | #1795 | | 2026-09-18 | The score table stays only for contrasts between two files | Beside the finding table it repeated itself | #1820 | | 2026-09-18 | A contrast crossing rounds gets no verdict | Round 2's `full` read Worse because `none` moved between rounds, not the file | #1826 | | 2026-09-19 | The primaries are anchored and checked by the control fixture | Unanchored, `design` and `maintainability` scored 3 on all 24 trials of rounds 1–2 | #1827, #1829, #1830 | @@ -798,19 +808,28 @@ Two limits: | 2026-09-24 | Eight contrasts, adding `short − full` | Does length matter | #1767 | | 2026-09-24 | The brief and the change prompt live in files of their own | The design's restructure would have broken the code that read them by heading | #1843 | +¹ This agrees with Anthropic's guidance on context engineering, which asks +for the smallest set of high-signal tokens: +[Effective context engineering for AI agents](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents). + ## 11. Appendix — the numbers behind the rules **Why three trials cannot reach significance.** Three paired differences give a bootstrap 27 distinct resamples and an exact sign test a minimum two-sided p of 0.25. -**Why the judge reads each tree three times.** Six judgings of one unchanged -bundle returned readability 4, 4, 4, 3, 4, 4: mean 3.83, sample SD 0.41, -about 0.33 standard error on a difference of two arm means at K = 3. -Readability was the only primary that moved in rounds 1 and 2, so every -verdict rested on the row that was not reproducible. Three judgings cut -that standard error by about 40 %. Eight of the eleven rows returned the -same score on all six calls. +**Why the judge reads each tree three times.** + +- **Measured:** six judgings of one unchanged bundle returned readability + 4, 4, 4, 3, 4, 4. Mean 3.83, sample SD 0.41. +- **Effect:** about 0.33 standard error on a difference of two arm means at + K = 3. +- **Why it mattered:** readability was the only primary that moved in + rounds 1 and 2, so every verdict rested on the row that was not + reproducible. +- **The fix:** three judgings cut that standard error by about 40 %. +- **Rows that did not need it:** eight of the eleven returned the same + score on all six calls. **Why the budget is $100 and the timeout two hours.** The CLI prices `claude-sonnet-5` at $5 per million input tokens and $25 per million output.