Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 21 additions & 7 deletions docs/design/efficacy-benchmark.md
Original file line number Diff line number Diff line change
Expand Up @@ -512,14 +512,27 @@ Neither is reachable by configuration:
- The capped rate is the first tax that is not one rate times one amount,
so it changes the shape of the tax step, not its inputs.

**Measured:** files touched, lines changed, whether the build suite is
still green, whether the change-task acceptance module passes, and whether
the new rule and rate show in the same API and UI. A new rule class plus a
registry entry scores low churn. Editing the pricing function, the tax
step, three routes and two templates scores high.
**Measured:**

**The acceptance module is written before any change-task run.** It tests
the two new behaviours against worked figures.
- files touched and lines changed
- whether the build suite is still green
- whether the change-task acceptance modules pass, through the API
- from round 3, whether the new rule and cap work through the forms and
the invoice preview

A new rule class plus a registry entry scores low churn. Editing the
pricing function, the tax step, three routes and two templates scores high.

**The prompt states everything the grader drives.** It names the new
class, the cap's keyword, the form fields and the rules the acceptance
modules check.

Why: round 1's grader constructed the new behaviours through names the
prompt never gave, so it graded which trials guessed them, and its pass
rate was withdrawn (§6.5).

**The acceptance modules are written before any change-task run.** They
test the two new behaviours against worked figures.

Why: churn means something only beside a pass. An arm that touched four
lines and broke the extension has not scored well.
Expand Down Expand Up @@ -839,6 +852,7 @@ Two limits:
| 2026-09-24 | Round 3 starts anew with all five arms and reuses no trial | The spec gained sign-in and customers, and the judge changed | #1767 |
| 2026-09-24 | Eight contrasts, adding `short − full` | Does length matter | #1767 |
| 2026-09-24 | The brief and the change prompt live in files of their own | The design's restructure would have broken the code that read them by heading | #1843 |
| 2026-09-24 | The change prompt states the API, the form fields and the rules its acceptance modules check; the UI is graded by HTTP cases from round 3 | A grader driving names the prompt never gave grades guessing; the UI was asked for but never graded | #1767 |
| 2026-09-24 | Security and data protection: an anchored judge row and a probe pass rate each, all primary; probe margins mean no probe lost | A judge row alone is an opinion; the probes are deterministic. Security regressions get no tolerance | #1767 |

¹ This agrees with Anthropic's guidance on context engineering, which asks
Expand Down
24 changes: 24 additions & 0 deletions tests/efficacy/change-prompt.txt
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,27 @@ at the reduced rate on the first 50.00 of each line's taxable amount and
at the standard rate on the excess. Both must appear on the rules and
jurisdictions pages and take effect in the invoice builder. Keep the
existing behaviour and your tests passing.

The Python API other software will call:

- `tariff.ThresholdRule(rule_id, percent=..., minimum=...)`, an
invoice-scoped rule. It is eligible when the subtotal is at least
`minimum`; a rule that is not eligible does not apply and is not
reported in `applied`. It and the invoice percentage rule share one
slot: at most one of them applies, chosen as rules of one kind are.
Constructing it raises when `percent` is outside the range a
percentage rule accepts, or when `minimum` is negative.
- `tariff.Jurisdiction(..., caps={category: amount})`. The cap applies to
each line separately: `tax = round(min(taxable, cap) * reduced +
max(0, taxable - cap) * standard)`. An amount exactly at the cap is
wholly at the reduced rate. A category without a cap is taxed as
before.

Through the web application:

- POST `/rules` accepts `kind` `threshold`, with `percent` and `minimum`,
and the rule is listed on `/rules`.
- POST `/jurisdictions` accepts `caps`, written like `rates`:
`category=amount` pairs separated by commas. The cap is shown on
`/jurisdictions`.
- POST `/invoices/preview` prices with both.
20 changes: 16 additions & 4 deletions tests/efficacy/harness.py
Original file line number Diff line number Diff line change
Expand Up @@ -1581,11 +1581,23 @@ def writable(function, target, _):


def change_checks():
"""The change prompt is the design's, and its workspace starts committed."""
"""The change prompt is its file's, whole, and names what the grader
drives; its workspace starts committed."""
prompt = read_change_prompt()
checks = [("the change prompt is read from the design",
prompt.startswith("Add a **spend-threshold** discount")
and prompt.endswith("your tests passing."))]
with io.open(CHANGE_PROMPT, encoding="utf-8") as handle:
whole = handle.read().strip()
checks = [
("the change prompt is read whole from its file",
prompt == whole
and prompt.startswith("Add a **spend-threshold** discount")),

# Round 1's grader drove names the prompt never gave, and its pass
# rate measured which trials guessed them.
("the change prompt names the API and fields the grader drives",
all(name in prompt for name in (
"tariff.ThresholdRule", "caps=", "`threshold`", "`minimum`",
"`caps`", "/invoices/preview"))),
]

scratch = os.path.join(os.environ.get("TEMP", "."),
"efficacy-change-self-test")
Expand Down
Loading