diff --git a/docs/design/efficacy-benchmark.md b/docs/design/efficacy-benchmark.md index a2b984c3..fc0b048a 100644 --- a/docs/design/efficacy-benchmark.md +++ b/docs/design/efficacy-benchmark.md @@ -512,14 +512,27 @@ Neither is reachable by configuration: - The capped rate is the first tax that is not one rate times one amount, so it changes the shape of the tax step, not its inputs. -**Measured:** files touched, lines changed, whether the build suite is -still green, whether the change-task acceptance module passes, and whether -the new rule and rate show in the same API and UI. A new rule class plus a -registry entry scores low churn. Editing the pricing function, the tax -step, three routes and two templates scores high. +**Measured:** -**The acceptance module is written before any change-task run.** It tests -the two new behaviours against worked figures. +- files touched and lines changed +- whether the build suite is still green +- whether the change-task acceptance modules pass, through the API +- from round 3, whether the new rule and cap work through the forms and + the invoice preview + +A new rule class plus a registry entry scores low churn. Editing the +pricing function, the tax step, three routes and two templates scores high. + +**The prompt states everything the grader drives.** It names the new +class, the cap's keyword, the form fields and the rules the acceptance +modules check. + +Why: round 1's grader constructed the new behaviours through names the +prompt never gave, so it graded which trials guessed them, and its pass +rate was withdrawn (§6.5). + +**The acceptance modules are written before any change-task run.** They +test the two new behaviours against worked figures. Why: churn means something only beside a pass. An arm that touched four lines and broke the extension has not scored well. @@ -839,6 +852,7 @@ Two limits: | 2026-09-24 | Round 3 starts anew with all five arms and reuses no trial | The spec gained sign-in and customers, and the judge changed | #1767 | | 2026-09-24 | Eight contrasts, adding `short − full` | Does length matter | #1767 | | 2026-09-24 | The brief and the change prompt live in files of their own | The design's restructure would have broken the code that read them by heading | #1843 | +| 2026-09-24 | The change prompt states the API, the form fields and the rules its acceptance modules check; the UI is graded by HTTP cases from round 3 | A grader driving names the prompt never gave grades guessing; the UI was asked for but never graded | #1767 | | 2026-09-24 | Security and data protection: an anchored judge row and a probe pass rate each, all primary; probe margins mean no probe lost | A judge row alone is an opinion; the probes are deterministic. Security regressions get no tolerance | #1767 | ¹ This agrees with Anthropic's guidance on context engineering, which asks diff --git a/tests/efficacy/change-prompt.txt b/tests/efficacy/change-prompt.txt index 685bfa90..2805f0bd 100644 --- a/tests/efficacy/change-prompt.txt +++ b/tests/efficacy/change-prompt.txt @@ -6,3 +6,27 @@ at the reduced rate on the first 50.00 of each line's taxable amount and at the standard rate on the excess. Both must appear on the rules and jurisdictions pages and take effect in the invoice builder. Keep the existing behaviour and your tests passing. + +The Python API other software will call: + +- `tariff.ThresholdRule(rule_id, percent=..., minimum=...)`, an + invoice-scoped rule. It is eligible when the subtotal is at least + `minimum`; a rule that is not eligible does not apply and is not + reported in `applied`. It and the invoice percentage rule share one + slot: at most one of them applies, chosen as rules of one kind are. + Constructing it raises when `percent` is outside the range a + percentage rule accepts, or when `minimum` is negative. +- `tariff.Jurisdiction(..., caps={category: amount})`. The cap applies to + each line separately: `tax = round(min(taxable, cap) * reduced + + max(0, taxable - cap) * standard)`. An amount exactly at the cap is + wholly at the reduced rate. A category without a cap is taxed as + before. + +Through the web application: + +- POST `/rules` accepts `kind` `threshold`, with `percent` and `minimum`, + and the rule is listed on `/rules`. +- POST `/jurisdictions` accepts `caps`, written like `rates`: + `category=amount` pairs separated by commas. The cap is shown on + `/jurisdictions`. +- POST `/invoices/preview` prices with both. diff --git a/tests/efficacy/harness.py b/tests/efficacy/harness.py index af5706f2..61116f85 100644 --- a/tests/efficacy/harness.py +++ b/tests/efficacy/harness.py @@ -1581,11 +1581,23 @@ def writable(function, target, _): def change_checks(): - """The change prompt is the design's, and its workspace starts committed.""" + """The change prompt is its file's, whole, and names what the grader + drives; its workspace starts committed.""" prompt = read_change_prompt() - checks = [("the change prompt is read from the design", - prompt.startswith("Add a **spend-threshold** discount") - and prompt.endswith("your tests passing."))] + with io.open(CHANGE_PROMPT, encoding="utf-8") as handle: + whole = handle.read().strip() + checks = [ + ("the change prompt is read whole from its file", + prompt == whole + and prompt.startswith("Add a **spend-threshold** discount")), + + # Round 1's grader drove names the prompt never gave, and its pass + # rate measured which trials guessed them. + ("the change prompt names the API and fields the grader drives", + all(name in prompt for name in ( + "tariff.ThresholdRule", "caps=", "`threshold`", "`minimum`", + "`caps`", "/invoices/preview"))), + ] scratch = os.path.join(os.environ.get("TEMP", "."), "efficacy-change-self-test")