Apache-2.0, the Agent Governance Leaderboard, and an honest competitor score - #35
Merged
Conversation
Signetry moves to an open-core model. This repository is part of the integration surface, so it is now Apache-2.0: use it, fork it, ship it commercially, no permission needed. The engine (Signetry/core) is source-available under BUSL-1.1 and converts to Apache-2.0 on 2030-08-31. Until now every repo in the org was "All Rights Reserved" with no LICENSE file, which GitHub reports as no license at all. That is the strictest possible default: it blocks corporate evaluation outright and makes outside contribution legally murky. This fixes that. What changed: - LICENSE (Apache-2.0) added, so GitHub detects and displays it. - README, CONTRIBUTING, CLA, CONTRIBUTORS and the CLA bot comment no longer claim the project is "not open source" or withhold usage rights. - The CLA is kept, because open core means code moves across the licence line and relicensing needs those rights. Its fallback licence grant is now non-exclusive: the previous *exclusive* grant would have stripped contributors of the right to use their own contribution, contradicting the rights the LICENSE grants everyone. The CLA text is now identical across all Signetry repos so the legal terms cannot drift per-repo again. - Community health files (SECURITY.md, CODE_OF_CONDUCT.md, issue templates) where they were missing. No functional or API change.
Signetry Reviewer — 🟣 Escalate to a designated reviewerEscalate to a designated reviewer — this PR touches security-sensitive surface (.github/workflows/benchmark.yml, .github/workflows/cla.yml, .github/workflows/eval.yml, .github/workflows/leaderboard.yml). No blocking issue was found automatically, but a human owner should sign off. Deterministic gates (the authority)
Findings (6, 0 blocking)
Sensitive surfaceThis PR changes security-sensitive paths that warrant a designated reviewer:
MergeA designated reviewer / code owner should sign off before merge (sensitive surface).
|
The published head-to-head understated a named competitor by ~9 recall points, and it did so on evidence that does not exist. `captured_corpus_adapter` returned an empty finding set for any case absent from a scanner's capture, and the scorer counted that as a miss. Captures are point-in-time; this corpus grows. The 8 cases added in the 52 -> 60 expansion were therefore charged to claude-code-security-review as failures — 5 of them vulnerable — so its published recall read 81% (38/47) when the honest figure over the cases it was actually given is 90% (38/42). The old module docstring stated the bug outright: "recorded as a miss, never fabricated". That is precisely the defect this suite calls out everywhere else, an unearned number, except pointed outward at someone else's tool and in our favour. - An absent case now returns `ran=False`, scores as `covered=False`, and is excluded from that scanner's recall, false positives, and the per-family / per-language breakdowns. A case that IS present but lists no findings is still a genuine miss; only "never run" is excused, and a test pins that distinction. - Every table prints "Cases scored" per scanner and lists the excluded ids under Notes. Silently excluding them would be its own dishonesty now that the two scanners are no longer scored over the same set. - `CorpusScore.recall` is `float | None` and returns None rather than 0.0 when nothing was measured. The `--min-recall` gate now fails on an unmeasured recall instead of passing it by default. The fix also corrects the by-language rows: go and java each read 4/6 because two of the missing cases were the new ones. 8 tests added.
Detection is table stakes — every scanner publishes a recall number. The axis nobody publishes is governance: when the repository itself is hostile, does the agent's change still get admitted, and what does the defense cost in benign work? docs/LEADERBOARD.md publishes both axes on one page and takes third-party submissions, so the governance axis can become a real comparison instead of a self-report. A leaderboard whose author is also its referee is only worth something if the rules are mechanical, so the renderer enforces three in code rather than by good intentions: - an unmeasured number renders as an em dash and never as 0%. A rate submitted without its denominator is refused and shown as unmeasured, with the reason printed on the page. - reproduced and self-reported rows never share a table. We ran it here, or we did not, and the page says which. - every rate is printed beside the sample size it was computed over. The corpus is 5 scenarios and the page says so in prose. Validation is where the honesty actually lives, so it is pinned by 32 tests, one per rule — including that bool is a subclass of int (True must not be accepted as an ASR of 100%) and that NaN passes isinstance and fails every comparison, so a naive 0 <= x <= 1 guard lets it render as "nan%". - `signetry-eval leaderboard` (--with-detection, --json, --entries). Exits non-zero if any governed defense failed. If the detection axis cannot run, it is omitted with a warning rather than published empty. - leaderboard/entries/ takes one JSON file per system. The ungoverned baseline ships as its own visible row rather than being implied by a column heading — a control condition you cannot see is not a control. - docs/SUBMITTING.md covers both contribution paths, including submitting a system that beats Signetry. A leaderboard only its author can win is marketing, and everyone can tell. Also fixed: Report.by_category() reported 0 for metrics with nothing behind them, so an attack-only category showed "Utility 0%" — reading as "Signetry destroyed utility here" — and the utility category showed an ASR of 0% over zero attacks. Both are None now, render as an em dash, and serialize as null. A measured zero is still 0%: the rule is no unearned numbers, not no zeros, and a test pins that too. CI: leaderboard.yml regenerates the page weekly, on release, and when an entry or scenario changes. benchmark.yml moves onto the same schedule — both pages were release-only, so published numbers could be months stale while reading as current. Each workflow owns exactly one page so every "Generated by ..." header names the workflow that actually wrote it, and both share a concurrency group because they both push to main. benchmark.yml's release-time push could never have worked: on a release event the checkout is detached at the tag, so `git push HEAD:$REF_NAME` targeted refs/tags/vX.Y.Z and the failure was swallowed by `|| echo "push skipped"`. Both workflows now check out and push the default branch explicitly. The regression gate runs last in both, so a broken defense is published and then fails the run, rather than failing first and leaving the stale page up.
bkd-dotcom
changed the title
The page says the signetry-core row is "measured live by `signetry-eval run` in this repository's CI", but recorded `version: null` for it. A reader cannot reproduce a result without knowing which build produced it, and reproducibility is the one thing this leaderboard exists to support — it is what separates "reproduced here" from "self-reported" in the Evidence column. The `--version` flag already existed and was plumbed through to both renderers. CI simply never passed it, so the field was silently unmeasured on every run. CI now reads the label from the installed package rather than hardcoding it, because a stale label is worse than the null it replaces: it would assert a specific build was tested when it wasn't. If the version cannot be read, the argument is omitted and the row degrades to "not measured" — the same rule this leaderboard applies to every rate it prints. Two tests: the label reaches both the markdown row and the JSON envelope, and an unknown version stays null rather than rendering an empty backtick label. Regenerated the committed page and envelope. Verified first that a no-version regen reproduces the committed files byte-for-byte, so the diff here is exactly the one line each that the version adds and nothing else drifted.
Member
Author
|
Pushed The page says the signetry-core row is "measured live by The CI now reads the label from the installed package rather than hardcoding it — a stale label would be worse than the null it replaces, since it asserts a specific build was tested when it was not. If the version cannot be read the argument is omitted and the row degrades to "not measured", the same rule the leaderboard applies to every rate it prints. Before regenerating the committed page I confirmed a no-version regen reproduces both committed files byte-for-byte, so the diff is exactly the one line each the version adds and nothing else drifted. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three changes, in increasing order of how much they matter.
1. Apache-2.0 — this repo has no strings, deliberately
Signetry goes open core:
the engine (
core) is source-available underBUSL-1.1 and converts to Apache-2.0 on 2030-08-31; everything you integrate with,
including this repo, is Apache-2.0.
A benchmark nobody can freely run, audit, and reproduce is worthless as evidence, so
the eval suite carries the most permissive licence of anything in the platform. Fork
it, re-run it, publish results that disagree with ours.
The CLA is kept — open core means code sometimes has to move across the line — but its
fallback grant is now non-exclusive, so a contributor never loses the right to use
their own contribution.
2. A competitor was scored on cases it was never run on
This one is a correction against a named third party, so it is called out on its own.
captured_corpus_adapterreturned an empty finding set for any case absent from ascanner's capture, and the scorer counted that as a miss. Captures are point-in-time;
this corpus grows. The 8 cases added in the 52 → 60 expansion were charged to
claude-code-security-reviewas failures — 5 of them vulnerable — so its publishedrecall read 81% (38/47) when the honest figure over the cases it was actually given
is 90% (38/42).
That is exactly the defect this suite calls out everywhere else — an unearned number —
except pointed outward at someone else's tool, and in our favour by about nine points.
An absent case is now excluded from that scanner's denominator. A case that is present
but lists no findings is still a genuine miss; only "never run" is excused, and a test
pins that distinction. Every table prints Cases scored per scanner and lists the
excluded ids, because silently excluding them would be its own dishonesty once the two
scanners are no longer scored over the same set.
3. The Agent Governance Leaderboard
Detection is table stakes — every scanner publishes a recall number. The axis nobody
publishes is governance: when the repository itself is hostile, does the agent's
change still get admitted, and what does the defense cost in benign work?
docs/LEADERBOARD.mdpublishes both axes on one page and takesthird-party submissions, so the governance axis can become a real comparison instead
of a self-report.
docs/SUBMITTING.mdcovers submitting an attackthat beats the governed pipeline, and submitting a system — including one that beats
Signetry.
A leaderboard whose author is also its referee is only worth something if the rules are
mechanical, so the renderer enforces three in code:
—, never0%; a rate submitted without itsdenominator is refused and the reason is printed on the page
Also fixed here:
Report.by_category()reported0for metrics with nothing behindthem, so an attack-only category showed
Utility 0%— reading as "Signetry destroyedutility here". A measured zero is still
0%; the rule is no unearned numbers, not nozeros.
Verification
58of them at commit 2 in isolation, so each commit stands alone);ruff checkcleanboolis asubclass of
int(Truemust not be accepted as an ASR of 100%) and thatNaNpasses
isinstanceand fails every comparison, so a naive0 <= x <= 1guard wouldrender it as
nan%CI
leaderboard.ymlregenerates the page weekly, on release, and when an entry or scenariochanges;
benchmark.ymlmoves onto the same schedule. Both pages were previouslyrelease-only, so published numbers could be months stale while reading as current.
benchmark.yml's release-time push could never have worked: on a release event thecheckout is detached at the tag, so
git push HEAD:$GITHUB_REF_NAMEtargetedrefs/tags/vX.Y.Zand the failure was swallowed by|| echo "push skipped". Bothworkflows now push the default branch explicitly, share a concurrency group, and run
their regression gate last — so a broken defense is published and then fails the
run, rather than failing first and leaving a stale page up.