From 30cfa762a161ca62fc5ce22e4aabded258f758fd Mon Sep 17 00:00:00 2001 From: Branimir Georgiev Date: Thu, 24 Sep 2026 17:37:55 +0300 Subject: [PATCH 1/2] feat(efficacy): the change prompt states what its grader drives MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round 1's change-task pass rate was withdrawn. The acceptance modules built the new rule and the cap through names the prompt never gave, so they graded which trials guessed them. They also checked four rules that only the private suite wrote down: an ineligible threshold rule is not reported, construction refuses bad values, the cap is per line, and an amount at the cap is wholly reduced. The prompt now names all of it: - `tariff.ThresholdRule(rule_id, percent=..., minimum=...)` and its eligibility and slot rules - `Jurisdiction(..., caps={...})` and the per-line tax formula - the form fields and pages the web application must accept The UI half was asked for but never graded. The suite's HTTP cases for it come with the round-3 suite extension, and design §5.4 now says so. The harness self test checks that the prompt is read whole from its file and names every API and field the grader drives. Removing the class name makes that check fail. Part of #1767. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/design/efficacy-benchmark.md | 28 +++++++++++++++++++++------- tests/efficacy/change-prompt.txt | 24 ++++++++++++++++++++++++ tests/efficacy/harness.py | 19 +++++++++++++++---- 3 files changed, 60 insertions(+), 11 deletions(-) diff --git a/docs/design/efficacy-benchmark.md b/docs/design/efficacy-benchmark.md index a2b984c3..fc0b048a 100644 --- a/docs/design/efficacy-benchmark.md +++ b/docs/design/efficacy-benchmark.md @@ -512,14 +512,27 @@ Neither is reachable by configuration: - The capped rate is the first tax that is not one rate times one amount, so it changes the shape of the tax step, not its inputs. -**Measured:** files touched, lines changed, whether the build suite is -still green, whether the change-task acceptance module passes, and whether -the new rule and rate show in the same API and UI. A new rule class plus a -registry entry scores low churn. Editing the pricing function, the tax -step, three routes and two templates scores high. +**Measured:** -**The acceptance module is written before any change-task run.** It tests -the two new behaviours against worked figures. +- files touched and lines changed +- whether the build suite is still green +- whether the change-task acceptance modules pass, through the API +- from round 3, whether the new rule and cap work through the forms and + the invoice preview + +A new rule class plus a registry entry scores low churn. Editing the +pricing function, the tax step, three routes and two templates scores high. + +**The prompt states everything the grader drives.** It names the new +class, the cap's keyword, the form fields and the rules the acceptance +modules check. + +Why: round 1's grader constructed the new behaviours through names the +prompt never gave, so it graded which trials guessed them, and its pass +rate was withdrawn (§6.5). + +**The acceptance modules are written before any change-task run.** They +test the two new behaviours against worked figures. Why: churn means something only beside a pass. An arm that touched four lines and broke the extension has not scored well. @@ -839,6 +852,7 @@ Two limits: | 2026-09-24 | Round 3 starts anew with all five arms and reuses no trial | The spec gained sign-in and customers, and the judge changed | #1767 | | 2026-09-24 | Eight contrasts, adding `short − full` | Does length matter | #1767 | | 2026-09-24 | The brief and the change prompt live in files of their own | The design's restructure would have broken the code that read them by heading | #1843 | +| 2026-09-24 | The change prompt states the API, the form fields and the rules its acceptance modules check; the UI is graded by HTTP cases from round 3 | A grader driving names the prompt never gave grades guessing; the UI was asked for but never graded | #1767 | | 2026-09-24 | Security and data protection: an anchored judge row and a probe pass rate each, all primary; probe margins mean no probe lost | A judge row alone is an opinion; the probes are deterministic. Security regressions get no tolerance | #1767 | ¹ This agrees with Anthropic's guidance on context engineering, which asks diff --git a/tests/efficacy/change-prompt.txt b/tests/efficacy/change-prompt.txt index 685bfa90..2805f0bd 100644 --- a/tests/efficacy/change-prompt.txt +++ b/tests/efficacy/change-prompt.txt @@ -6,3 +6,27 @@ at the reduced rate on the first 50.00 of each line's taxable amount and at the standard rate on the excess. Both must appear on the rules and jurisdictions pages and take effect in the invoice builder. Keep the existing behaviour and your tests passing. + +The Python API other software will call: + +- `tariff.ThresholdRule(rule_id, percent=..., minimum=...)`, an + invoice-scoped rule. It is eligible when the subtotal is at least + `minimum`; a rule that is not eligible does not apply and is not + reported in `applied`. It and the invoice percentage rule share one + slot: at most one of them applies, chosen as rules of one kind are. + Constructing it raises when `percent` is outside the range a + percentage rule accepts, or when `minimum` is negative. +- `tariff.Jurisdiction(..., caps={category: amount})`. The cap applies to + each line separately: `tax = round(min(taxable, cap) * reduced + + max(0, taxable - cap) * standard)`. An amount exactly at the cap is + wholly at the reduced rate. A category without a cap is taxed as + before. + +Through the web application: + +- POST `/rules` accepts `kind` `threshold`, with `percent` and `minimum`, + and the rule is listed on `/rules`. +- POST `/jurisdictions` accepts `caps`, written like `rates`: + `category=amount` pairs separated by commas. The cap is shown on + `/jurisdictions`. +- POST `/invoices/preview` prices with both. diff --git a/tests/efficacy/harness.py b/tests/efficacy/harness.py index af5706f2..cfc30d69 100644 --- a/tests/efficacy/harness.py +++ b/tests/efficacy/harness.py @@ -1581,11 +1581,22 @@ def writable(function, target, _): def change_checks(): - """The change prompt is the design's, and its workspace starts committed.""" + """The change prompt is its file's, whole, and names what the grader + drives; its workspace starts committed.""" prompt = read_change_prompt() - checks = [("the change prompt is read from the design", - prompt.startswith("Add a **spend-threshold** discount") - and prompt.endswith("your tests passing."))] + with io.open(CHANGE_PROMPT, encoding="utf-8") as handle: + whole = handle.read().strip() + checks = [ + ("the change prompt is read whole from its file", + prompt == whole + and prompt.startswith("Add a **spend-threshold** discount")), + # Round 1's grader drove names the prompt never gave, and its pass + # rate measured which trials guessed them. + ("the change prompt names the API and fields the grader drives", + all(name in prompt for name in ( + "tariff.ThresholdRule", "caps=", "`threshold`", "`minimum`", + "`caps`", "/invoices/preview"))), + ] scratch = os.path.join(os.environ.get("TEMP", "."), "efficacy-change-self-test") From edc74c3a5fd2a4cfdb0b1b28c27cd3a5808e8d2f Mon Sep 17 00:00:00 2001 From: Branimir Georgiev Date: Thu, 24 Sep 2026 17:39:33 +0300 Subject: [PATCH 2/2] fix(efficacy): separate the self test's comment from the check above it Conformance's comment-layout check wants a blank line between a comment block and the code above it. Part of #1767. Co-Authored-By: Claude Opus 5.5 (1M context) --- tests/efficacy/harness.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/efficacy/harness.py b/tests/efficacy/harness.py index cfc30d69..61116f85 100644 --- a/tests/efficacy/harness.py +++ b/tests/efficacy/harness.py @@ -1590,6 +1590,7 @@ def change_checks(): ("the change prompt is read whole from its file", prompt == whole and prompt.startswith("Add a **spend-threshold** discount")), + # Round 1's grader drove names the prompt never gave, and its pass # rate measured which trials guessed them. ("the change prompt names the API and fields the grader drives",