Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 11 additions & 7 deletions docs/design/efficacy-benchmark.md
Original file line number Diff line number Diff line change
Expand Up @@ -264,25 +264,28 @@ carries this repository's `templates/` directory at `v2.90.0` under

The brief is what an adopter would type:

- the domain in one line, naming discount rules and tax jurisdictions
- the domain in one line, naming discount rules, tax jurisdictions and
customers
- the stack: Python 3.12, Flask 3, Jinja, HTMX, SQLite, uv, ruff, mypy
strict, pytest
- the system's boundaries and actors: a person in a browser and another
program; the pricing engine importable without the web app; a local
store; no network
- personal data: customers are people with a name, email and address; the
GDPR applies, so a customer may ask for erasure or a copy of their data;
one administrator signs in
- the system's boundaries and actors: the administrator in a browser and
another program; the pricing engine importable without the web app; a
local store; no network

Why each part is there:

- The two extension axes are named because an adopter building this app
would name them. A brief withholding them measures a product nobody uses.
- The stack is named because the spec fixes it for every arm. An interview
left to choose might pick FastAPI and build a different app.
- Personal data is named because an adopter storing customers would say
so, and it is what lets the interview reach for data-protection rules.
- The boundaries are there because establishing a system's boundaries and
its interfaces is the interview's job.

Before round 3 the brief also states that customers carry personal data
(#1767).

**What naming the axes affects.** The report states this beside the rows it
touches:

Expand Down Expand Up @@ -853,6 +856,7 @@ Two limits:
| 2026-09-24 | Eight contrasts, adding `short − full` | Does length matter | #1767 |
| 2026-09-24 | The brief and the change prompt live in files of their own | The design's restructure would have broken the code that read them by heading | #1843 |
| 2026-09-24 | The change prompt states the API, the form fields and the rules its acceptance modules check; the UI is graded by HTTP cases from round 3 | A grader driving names the prompt never gave grades guessing; the UI was asked for but never graded | #1767 |
| 2026-09-24 | The brief names customers as personal data under the GDPR and one signing-in administrator; `hand` gains two generic lines (OWASP ASVS level 1, GDPR) | An adopter storing customers would say so. `hand`'s lines stay generic: lines mirroring the probes would hand it the answer key | #1767 |
| 2026-09-24 | Security and data protection: an anchored judge row and a probe pass rate each, all primary; probe margins mean no probe lost | A judge row alone is an opinion; the probes are deterministic. Security regressions get no tolerance | #1767 |

¹ This agrees with Anthropic's guidance on context engineering, which asks
Expand Down
38 changes: 19 additions & 19 deletions tests/efficacy/arms/hand/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,38 +2,38 @@

## Layout

- `src/tariff/` for the package, `tests/` for the tests, `pyproject.toml`
for the metadata and tool configuration.
- The pricing engine is pure Python. It imports neither Flask nor the
- `src/tariff/` for the package, `tests/` for tests, `pyproject.toml` for
metadata and tool configuration.
- The pricing engine is pure Python and imports neither Flask nor the
persistence layer. The web application imports the engine, never the
reverse.
- Templates receive plain values, not database rows.
reverse. Templates receive plain values, not database rows.

## Quality

- `ruff check` and `ruff format --check` are clean.
- `mypy --strict` is clean. Every public function is annotated.
- Tests run under `pytest` and cover the pricing rules, the routes and
the exports. Aim for 80% line coverage and cover every boundary the
spec names.
- Keep functions short and nesting shallow. Name things after what they
mean in the domain.
- `ruff check`, `ruff format --check` and `mypy --strict` are clean.
- Tests under `pytest` cover the pricing rules, the routes and the exports,
every boundary the spec names, and 80% of lines.
- Keep functions short and nesting shallow. Name things after the domain.

## Errors

- One exception hierarchy, rooted at `TariffError`. Every error the
package raises on purpose derives from it.
- Validation failures carry the field they are about.
- One exception hierarchy rooted at `TariffError`; every error the package
raises on purpose derives from it, and validation errors name the field.
- Never swallow an exception silently.

## Web

- Escape everything rendered. Never mark untrusted text safe.
- Every POST form carries a CSRF token and is refused without one.
- Validate on the server. The client is not a validator.
- Return 404 for an unknown identifier, not 500.
- Validate on the server. Return 404 for an unknown identifier, not 500.

## Security and data

- Follow OWASP ASVS level 1 for sign-in, sessions and input handling.
- Customer data is personal data under the GDPR: store only what the spec
asks for, and honour a customer's erasure and access requests.

## Money

- `Decimal` everywhere in the pricing path. No floats.
- Round only where the spec says to round.
- `Decimal` everywhere in the pricing path. No floats. Round only where
the spec says to round.
14 changes: 10 additions & 4 deletions tests/efficacy/brief.txt
Original file line number Diff line number Diff line change
@@ -1,11 +1,17 @@
`tariff` — Imbra Ltd — a pricing and invoicing web application:
a product catalog, discount rules of several kinds, tax jurisdictions,
and invoices assembled from them — Python 3.12 / Flask 3 / Jinja /
HTMX / SQLite — uv, ruff, mypy strict, pytest — runs locally, no
customers, and invoices assembled from them — Python 3.12 / Flask 3 /
Jinja / HTMX / SQLite — uv, ruff, mypy strict, pytest — runs locally, no
deployment target

Boundaries and actors: two actors, a person working in a browser and
another program; the pricing engine is a library the program imports
Personal data: customers are people, stored with their name, email and
postal address and named on the invoices billed to them. The business is
in the EU, so the GDPR applies: a customer may ask for their data to be
erased or for a copy of it. One administrator signs in to use the
application; nobody else has an account.

Boundaries and actors: two actors, the administrator working in a browser
and another program; the pricing engine is a library the program imports
without the web application present, and the web application is one
caller of it. The store is a local file. Nothing reaches the network.
The program's interfaces are the library's own API and the invoice
Expand Down
6 changes: 5 additions & 1 deletion tests/efficacy/generate_arm.py
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,8 @@ def record_path(arm):
"PricedInvoice", "TariffError",
"/invoices/preview", "/invoices/new", "export.csv", "export.json",
"228.35", "227.07", "195.24", "0.135",
"TARIFF_ADMIN_PASSWORD", "/sign-in", "/sign-out",
"Ada Example", "ada@example.com",
]

# The OUTPUT scan asks something narrower, and the prompt scan is what
Expand All @@ -97,13 +99,15 @@ def record_path(arm):
"r-tier-wid", "r-bulk-see", "r-pct-bok", "r-pct-all",
"c-welcome", "c-tenoff",
"228.35", "227.07", "195.24",
"Ada Example", "ada@example.com",
]

# Two tokens taught this the hard way.
#
# `HX-Request` came off the prompt list: the templates document the header
# themselves, as any HTMX guidance would, so the scan fired on a clean
# prompt. Every other token was then checked the same way against the chain
# prompt. `/customers/` came off it the same way, from the templates' REST
# rule, whose example nests `/customers/{id}/orders`. Every other token was then checked the same way against the chain
# at the release and appears nowhere in it.
#
# `TariffError` came off the *output* list on the first real generation. The
Expand Down
Loading