feat(efficacy): round 3's brief names personal data, and hand gains two lines - #1849
Merged
Merged
Conversation
…wo lines The spec gained sign-in and customers for round 3. Now the brief and the hand-written arm catch up. - The brief names customers as people, with a name, email and address. It says the GDPR applies, so a customer may ask for erasure or a copy of their data, and that one administrator signs in. It still states no route, field or semantics. - `hand` gains a "Security and data" section of two generic lines: OWASP ASVS level 1, and customer data as personal data under the GDPR. Its other sections are condensed so the file stays at 39 lines. Lines mirroring the round-3 probes were drafted and rejected: they would hand the arm the answer key. - The leak scans gain the extended spec's tokens: `TARIFF_ADMIN_PASSWORD`, `/sign-in`, `/sign-out`, and the export example's `Ada Example` and `ada@example.com`. `/customers/` was tried and dropped, because the templates' REST rule uses it in an example. Part of #1767. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1767, the checklist item "Regenerate arm B from a brief naming personal data, and extend arm C within 40 lines". This PR covers the brief and
hand. The generated arms are regenerated from this brief in the last step, once the rest of round 3's inputs are in.What changes
brief.txtnames customers in the domain line, and adds a paragraph on personal data:The boundaries paragraph names the administrator as the browser actor. As before, the brief states no route, field or semantics.
arms/hand/CLAUDE.mdgains a "Security and data" section with two generic lines: follow OWASP ASVS level 1, and treat customer data as personal data under the GDPR. The other sections are condensed, and the file is 39 lines. The owner chose generic lines over lines that mirror each round-3 probe, which would have handed the arm the answer key.The leak scans gain the extended spec's tokens:
TARIFF_ADMIN_PASSWORD,/sign-in,/sign-out,Ada Example,ada@example.com/customers/fired on a clean prompt, from the templates' REST rule example/customers/{id}/orders. It comes off the list, noted beside theHX-Requestprecedent.Design §3.4 describes the new brief and why it names personal data. The decision log gains a row.
Evidence
generate_arm.py --dry-runforfull,shortandhybridbuilds each prompt, carrying the new brief, and each scans clean.TARIFF_ADMIN_PASSWORDto the brief (its SHA changed) makes the dry run refuse, with that token as the hit. The brief was then restored to its staged SHA.generate_armpasses; report 83/83; harness 94/94.handis 39 lines, none over 88 characters.Changelog
No entry. This changes the benchmark's own trial inputs.
🤖 Generated with Claude Code