feat(efficacy): the judge scores security and data protection, anchored - #1854
Merged
Merged
Conversation
The rubric gains two primary rows, each anchored at 1, 3 and 5 in terms of the application's sign-in, secrets, queries and customer data. The report leads with them under the design's 0.3-point margin and 0.5-point threshold, and its self-test fails when a primary is added to the judge or the report without the other, or without its margin, threshold and words. Part of #1767. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
braboj
added
P1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1767. This covers two of its tasks: the two rows added to the judge's rubric, and
PRIMARYinjudge.pyandreport.py. The probe pass rates and their margins follow with the probes.Judge (
judge.py).securityanddata_protectionjoin the rubric andPRIMARY. Each row is described at 1, 3 and 5 in terms of the application: sign-in, secrets, queries andnextfor security, and logs, erasure and export of customer data for data protection. The 5 describes where the protection lives in the code, one guard and one place for the personal data, so the row reads structure. The probes already read behaviour.Report (
report.py). Two metric rows, lead primaries, margin 0.3 and practical threshold 0.5 per design §5.8. The finding table's words gain "more secure" and "more careful with personal data". The why-line's "saw none of it as better …" sentence names all five primaries.Existing test changed. One self-test expectation changed with that sentence. A new self-test check fails when a primary is in the judge and not the report, or the reverse, or lacks its margin, threshold, metric row or words. Control: deleting the security entry from the finding table's words made it fail, naming
judge_securityas lacking. Restoring the entry made it pass again (84/84).The control is owed a re-run. A rubric prompt change invalidates every control reading, so the six current trees are re-judged under this prompt together with the four new trees built from the calibration trial (design §5.7).
No CHANGELOG entry: the benchmark is this repository's own.
Checks run: smoke 33/33;
report.py --self-test84/84; conformance 0 failed, its readings read (two files undertests/, both named above).🤖 Generated with Claude Code