Skip to content

feat(efficacy): round 3's brief names personal data, and hand gains two lines - #1849

Merged
braboj merged 1 commit into
mainfrom
feat/efficacy-round3-brief
Sep 24, 2026
Merged

braboj merged 1 commit into
mainfrom
feat/efficacy-round3-brief

Conversation

@braboj

@braboj braboj commented Sep 24, 2026

Copy link
Copy Markdown
Collaborator

Part of #1767, the checklist item "Regenerate arm B from a brief naming personal data, and extend arm C within 40 lines". This PR covers the brief and hand. The generated arms are regenerated from this brief in the last step, once the rest of round 3's inputs are in.

What changes

  • brief.txt names customers in the domain line, and adds a paragraph on personal data:

    • customers are people with a name, email and address
    • the GDPR applies, so a customer may ask for erasure or a copy of their data
    • one administrator signs in

    The boundaries paragraph names the administrator as the browser actor. As before, the brief states no route, field or semantics.

  • arms/hand/CLAUDE.md gains a "Security and data" section with two generic lines: follow OWASP ASVS level 1, and treat customer data as personal data under the GDPR. The other sections are condensed, and the file is 39 lines. The owner chose generic lines over lines that mirror each round-3 probe, which would have handed the arm the answer key.

  • The leak scans gain the extended spec's tokens:

    • the prompt scan: TARIFF_ADMIN_PASSWORD, /sign-in, /sign-out, Ada Example, ada@example.com
    • the output scan: the two export-example values, which nobody could derive
    • /customers/ fired on a clean prompt, from the templates' REST rule example /customers/{id}/orders. It comes off the list, noted beside the HX-Request precedent.
  • Design §3.4 describes the new brief and why it names personal data. The decision log gains a row.

Evidence

  • Dry runs: generate_arm.py --dry-run for full, short and hybrid builds each prompt, carrying the new brief, and each scans clean.
  • Control: the change was staged. Appending a sentence naming TARIFF_ADMIN_PASSWORD to the brief (its SHA changed) makes the dry run refuse, with that token as the hit. The brief was then restored to its staged SHA.
  • Self tests: generate_arm passes; report 83/83; harness 94/94.
  • Gates: smoke 33, 0 failed. Conformance: 0 failed. hand is 39 lines, none over 88 characters.

Changelog

No entry. This changes the benchmark's own trial inputs.

🤖 Generated with Claude Code

…wo lines

The spec gained sign-in and customers for round 3. Now the brief and the
hand-written arm catch up.

- The brief names customers as people, with a name, email and address.
  It says the GDPR applies, so a customer may ask for erasure or a copy
  of their data, and that one administrator signs in. It still states no
  route, field or semantics.
- `hand` gains a "Security and data" section of two generic lines:
  OWASP ASVS level 1, and customer data as personal data under the GDPR.
  Its other sections are condensed so the file stays at 39 lines. Lines
  mirroring the round-3 probes were drafted and rejected: they would hand
  the arm the answer key.
- The leak scans gain the extended spec's tokens:
  `TARIFF_ADMIN_PASSWORD`, `/sign-in`, `/sign-out`, and the export
  example's `Ada Example` and `ada@example.com`. `/customers/` was tried
  and dropped, because the templates' REST rule uses it in an example.

Part of #1767.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@braboj
braboj merged commit 30889d5 into main Sep 24, 2026
1 check passed
@braboj
braboj deleted the feat/efficacy-round3-brief branch September 24, 2026 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant