From ac5ce768f67f6d6a47368d6040c06d5a252b4b78 Mon Sep 17 00:00:00 2001 From: Branimir Georgiev Date: Thu, 24 Sep 2026 18:03:56 +0300 Subject: [PATCH] feat(efficacy): round 3's brief names personal data, and hand gains two lines The spec gained sign-in and customers for round 3. Now the brief and the hand-written arm catch up. - The brief names customers as people, with a name, email and address. It says the GDPR applies, so a customer may ask for erasure or a copy of their data, and that one administrator signs in. It still states no route, field or semantics. - `hand` gains a "Security and data" section of two generic lines: OWASP ASVS level 1, and customer data as personal data under the GDPR. Its other sections are condensed so the file stays at 39 lines. Lines mirroring the round-3 probes were drafted and rejected: they would hand the arm the answer key. - The leak scans gain the extended spec's tokens: `TARIFF_ADMIN_PASSWORD`, `/sign-in`, `/sign-out`, and the export example's `Ada Example` and `ada@example.com`. `/customers/` was tried and dropped, because the templates' REST rule uses it in an example. Part of #1767. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/design/efficacy-benchmark.md | 18 ++++++++------ tests/efficacy/arms/hand/CLAUDE.md | 38 +++++++++++++++--------------- tests/efficacy/brief.txt | 14 +++++++---- tests/efficacy/generate_arm.py | 6 ++++- 4 files changed, 45 insertions(+), 31 deletions(-) diff --git a/docs/design/efficacy-benchmark.md b/docs/design/efficacy-benchmark.md index fc0b048a..0f7c6b4b 100644 --- a/docs/design/efficacy-benchmark.md +++ b/docs/design/efficacy-benchmark.md @@ -264,12 +264,16 @@ carries this repository's `templates/` directory at `v2.90.0` under The brief is what an adopter would type: -- the domain in one line, naming discount rules and tax jurisdictions +- the domain in one line, naming discount rules, tax jurisdictions and + customers - the stack: Python 3.12, Flask 3, Jinja, HTMX, SQLite, uv, ruff, mypy strict, pytest -- the system's boundaries and actors: a person in a browser and another - program; the pricing engine importable without the web app; a local - store; no network +- personal data: customers are people with a name, email and address; the + GDPR applies, so a customer may ask for erasure or a copy of their data; + one administrator signs in +- the system's boundaries and actors: the administrator in a browser and + another program; the pricing engine importable without the web app; a + local store; no network Why each part is there: @@ -277,12 +281,11 @@ Why each part is there: would name them. A brief withholding them measures a product nobody uses. - The stack is named because the spec fixes it for every arm. An interview left to choose might pick FastAPI and build a different app. +- Personal data is named because an adopter storing customers would say + so, and it is what lets the interview reach for data-protection rules. - The boundaries are there because establishing a system's boundaries and its interfaces is the interview's job. -Before round 3 the brief also states that customers carry personal data -(#1767). - **What naming the axes affects.** The report states this beside the rows it touches: @@ -853,6 +856,7 @@ Two limits: | 2026-09-24 | Eight contrasts, adding `short − full` | Does length matter | #1767 | | 2026-09-24 | The brief and the change prompt live in files of their own | The design's restructure would have broken the code that read them by heading | #1843 | | 2026-09-24 | The change prompt states the API, the form fields and the rules its acceptance modules check; the UI is graded by HTTP cases from round 3 | A grader driving names the prompt never gave grades guessing; the UI was asked for but never graded | #1767 | +| 2026-09-24 | The brief names customers as personal data under the GDPR and one signing-in administrator; `hand` gains two generic lines (OWASP ASVS level 1, GDPR) | An adopter storing customers would say so. `hand`'s lines stay generic: lines mirroring the probes would hand it the answer key | #1767 | | 2026-09-24 | Security and data protection: an anchored judge row and a probe pass rate each, all primary; probe margins mean no probe lost | A judge row alone is an opinion; the probes are deterministic. Security regressions get no tolerance | #1767 | ¹ This agrees with Anthropic's guidance on context engineering, which asks diff --git a/tests/efficacy/arms/hand/CLAUDE.md b/tests/efficacy/arms/hand/CLAUDE.md index 44a36673..3535effb 100644 --- a/tests/efficacy/arms/hand/CLAUDE.md +++ b/tests/efficacy/arms/hand/CLAUDE.md @@ -2,38 +2,38 @@ ## Layout -- `src/tariff/` for the package, `tests/` for the tests, `pyproject.toml` - for the metadata and tool configuration. -- The pricing engine is pure Python. It imports neither Flask nor the +- `src/tariff/` for the package, `tests/` for tests, `pyproject.toml` for + metadata and tool configuration. +- The pricing engine is pure Python and imports neither Flask nor the persistence layer. The web application imports the engine, never the - reverse. -- Templates receive plain values, not database rows. + reverse. Templates receive plain values, not database rows. ## Quality -- `ruff check` and `ruff format --check` are clean. -- `mypy --strict` is clean. Every public function is annotated. -- Tests run under `pytest` and cover the pricing rules, the routes and - the exports. Aim for 80% line coverage and cover every boundary the - spec names. -- Keep functions short and nesting shallow. Name things after what they - mean in the domain. +- `ruff check`, `ruff format --check` and `mypy --strict` are clean. +- Tests under `pytest` cover the pricing rules, the routes and the exports, + every boundary the spec names, and 80% of lines. +- Keep functions short and nesting shallow. Name things after the domain. ## Errors -- One exception hierarchy, rooted at `TariffError`. Every error the - package raises on purpose derives from it. -- Validation failures carry the field they are about. +- One exception hierarchy rooted at `TariffError`; every error the package + raises on purpose derives from it, and validation errors name the field. - Never swallow an exception silently. ## Web - Escape everything rendered. Never mark untrusted text safe. - Every POST form carries a CSRF token and is refused without one. -- Validate on the server. The client is not a validator. -- Return 404 for an unknown identifier, not 500. +- Validate on the server. Return 404 for an unknown identifier, not 500. + +## Security and data + +- Follow OWASP ASVS level 1 for sign-in, sessions and input handling. +- Customer data is personal data under the GDPR: store only what the spec + asks for, and honour a customer's erasure and access requests. ## Money -- `Decimal` everywhere in the pricing path. No floats. -- Round only where the spec says to round. +- `Decimal` everywhere in the pricing path. No floats. Round only where + the spec says to round. diff --git a/tests/efficacy/brief.txt b/tests/efficacy/brief.txt index 895b3f5d..ebbc8c1f 100644 --- a/tests/efficacy/brief.txt +++ b/tests/efficacy/brief.txt @@ -1,11 +1,17 @@ `tariff` — Imbra Ltd — a pricing and invoicing web application: a product catalog, discount rules of several kinds, tax jurisdictions, -and invoices assembled from them — Python 3.12 / Flask 3 / Jinja / -HTMX / SQLite — uv, ruff, mypy strict, pytest — runs locally, no +customers, and invoices assembled from them — Python 3.12 / Flask 3 / +Jinja / HTMX / SQLite — uv, ruff, mypy strict, pytest — runs locally, no deployment target -Boundaries and actors: two actors, a person working in a browser and -another program; the pricing engine is a library the program imports +Personal data: customers are people, stored with their name, email and +postal address and named on the invoices billed to them. The business is +in the EU, so the GDPR applies: a customer may ask for their data to be +erased or for a copy of it. One administrator signs in to use the +application; nobody else has an account. + +Boundaries and actors: two actors, the administrator working in a browser +and another program; the pricing engine is a library the program imports without the web application present, and the web application is one caller of it. The store is a local file. Nothing reaches the network. The program's interfaces are the library's own API and the invoice diff --git a/tests/efficacy/generate_arm.py b/tests/efficacy/generate_arm.py index 893de393..b7617dbd 100644 --- a/tests/efficacy/generate_arm.py +++ b/tests/efficacy/generate_arm.py @@ -85,6 +85,8 @@ def record_path(arm): "PricedInvoice", "TariffError", "/invoices/preview", "/invoices/new", "export.csv", "export.json", "228.35", "227.07", "195.24", "0.135", + "TARIFF_ADMIN_PASSWORD", "/sign-in", "/sign-out", + "Ada Example", "ada@example.com", ] # The OUTPUT scan asks something narrower, and the prompt scan is what @@ -97,13 +99,15 @@ def record_path(arm): "r-tier-wid", "r-bulk-see", "r-pct-bok", "r-pct-all", "c-welcome", "c-tenoff", "228.35", "227.07", "195.24", + "Ada Example", "ada@example.com", ] # Two tokens taught this the hard way. # # `HX-Request` came off the prompt list: the templates document the header # themselves, as any HTMX guidance would, so the scan fired on a clean -# prompt. Every other token was then checked the same way against the chain +# prompt. `/customers/` came off it the same way, from the templates' REST +# rule, whose example nests `/customers/{id}/orders`. Every other token was then checked the same way against the chain # at the release and appears nowhere in it. # # `TariffError` came off the *output* list on the first real generation. The