signetry-eval

The public adversarial evaluation suite for Signetry.

Measures attack success rate (ASR) and utility-under-defense for coding-agent threats — governed by the same signetry-core pipeline production uses.

---

Signetry's honest claim is never "prompt injection solved." It is bounded + quarantined + dual-verified + receipted: the governed run keeps the attacker's objective out of the admitted change and caps authority on the evidence, while the legitimate task still ships with a signed receipt. This suite measures exactly that — with the ungoverned baseline right beside it, so the number is honest.

Every scenario is deterministic and offline (a scripted adversary that models a non-compliant agent — one that obeys instructions it can read). No network, no API keys, reproducible in CI.

What it measures

For each scenario, two conditions:

• Ungoverned — the agent/tooling runs with no Signetry checkpoint.
• Governed — the same run passes through signetry-core (run_admission / admit_extension), exactly as production uses it.

…and two questions: did the attack succeed? (ASR) and did the benign task still complete? (utility). A defense that blocks everything scores ASR 0 with zero utility — useless. The number that matters is ASR under defense at preserved utility.

Detection benchmark (head-to-head)

Beyond the adversarial suite, signetry-eval runs a public detection benchmark that scores Signetry's SAST engine against LLM security scanners on a shared, provenance- cited corpus — 60 cases across 7 languages (Python, JavaScript, Go, Java, Ruby, PHP, C#) in six families (public/OWASP, academic/CWE, crafted, hard cross-file taint, multilang, cross-file-lang), with safe decoys for false-positive measurement.

Scanner	Cases scored	Recall	False positives	Cost
signetry-core (deterministic)	60 / 60	100% (47/47)	0	free · offline · reproducible
claude-code-security-review (Claude Opus 4.8)	52 / 60¹	90% (38/42)	0	paid per scan · non-deterministic
@openai/codex-security	—	not run²	—	paid per scan

¹ Scored only over the cases its capture covers. Captures are taken at a point in time and this corpus grows; the 8 cases added since this one was recorded are excluded from its recall rather than counted as misses, because the scanner was never run on them. The excluded ids are listed in docs/BENCHMARK.md. Reporting them as failures would inflate our lead by ~9 points on evidence that does not exist.

² Competitor scores replay a committed capture; a tool that wasn't run is shown as not run, never scored as zero.

The live numbers are regenerated by CI, not typed here by hand — see docs/BENCHMARK.md.

Signetry reaches this on the deterministic, offline, free layer — same result every run. The optional Semgrep layer (--semgrep) broadens coverage but can add false positives (its generic rules don't model every sanitizer); Signetry's deterministic engine is 0-FP on the corpus, including sanitizer-aware cases where Semgrep is not — so the Semgrep layer is opt-in and non-gating. Detection parity is table stakes; the governance Signetry adds on top (earned authority, injection quarantine, independent verifier, signed receipts) is what the scanners don't attempt, and is measured by the adversarial suite below.

The Agent Governance Leaderboard

Detection is table stakes — several tools do it well, and the numbers above are public. The axis nobody publishes is governance: when the repository itself is hostile, does the agent's change still get admitted, and what does the defense cost in benign work?

docs/LEADERBOARD.md publishes both axes on one page, and it takes submissions — so the governance axis can become a real comparison instead of a self-report. Three rules are enforced by the renderer rather than by good intentions:

1. A number nobody measured renders —, never 0% — that applies to us too.
2. Reproduced and self-reported never share a table.
3. Every rate is printed next to its sample size. 0% ASR over 5 scenarios is a different claim from 0% over 500, and the reader gets to see which one this is.

Two ways on:

• Submit an attack that beats the governed pipeline. If it lands and the defense fails, that is a published gap with your name on it. This is the contribution we want most — see docs/SUBMITTING.md.
• Submit a system — any agent-governance, guardrail, or admission tool, including one that beats Signetry. A leaderboard only its author can win is marketing, and everyone can tell.

signetry-eval leaderboard                    # governance axis (offline, seconds)
signetry-eval leaderboard --with-detection   # both axes

Threat categories (mapped to the research)

Category	Threat	Basis
ipi	Indirect prompt injection via repository text (README / CLAUDE.md)	AgentDojo; OWASP LLM01
skill_poison	Poisoned skill docs / hijacking MCP tool descriptions	SkillJect; ToolHijacker / MCPTox
minja	Memory / session-freshness injection (replay a stale authorization)	MINJA (NeurIPS 2025)
utility	A benign task / clean extension must not be blocked	utility-under-defense

Run it

pip install -e .                 # pulls signetry-core from its source repo (not on PyPI)

signetry-eval run                   # human summary (ASR / utility)
signetry-eval run --markdown        # publishable report
signetry-eval run --json            # machine-readable
signetry-eval run --category ipi    # one threat category
signetry-eval list                  # list scenarios

signetry-eval leaderboard           # the public two-axis leaderboard
signetry-eval corpus                # detection head-to-head (recall, FP, by-language)
signetry-eval corpus --markdown     # publishable comparison table
signetry-eval corpus --semgrep      # add the optional Semgrep layer (report-only)
signetry-eval realrepo              # live scan of real vulnerable repos

signetry-eval run exits non-zero if the defense did not hold on every adversarial scenario, so it doubles as a CI regression guard.

from signetry_eval import run_all
report = run_all()
print(report.overall())          # asr_ungoverned, asr_governed, utility_governed, …

Interpreting the results

• ASR (governed) → low means Signetry kept the tested attacks out of the admitted change. It is not proof of coverage against unseen phrasings — a fixed pattern detector can be paraphrased around. New adversarial phrasings belong in this repo as new scenarios (adaptive red-teaming), and the curve is published honestly.
• Utility (governed) → high means real work still ships under the same defense.
• The MINJA scenario shows why a receipt can't be replayed: it is bound to the base commit, the diff hash, and the plan capability set's hash, so a prior authorization cannot bless a different change.

Design

• No governance logic lives here — it is imported from signetry-core. This repo only poses attacks and scores outcomes.
• Scenarios implement a tiny Scenario protocol (run() -> ScenarioResult) and are registered in signetry_eval/scenarios.
• Add a scenario: model the attack, run it ungoverned + governed via the harness, and return a ScenarioResult.

Contributing

Apache-2.0, PRs welcome. Contribute under the CLA — it lets a well-built case or adapter move across the open-core line later — and you're credited in CONTRIBUTORS.md. Start at the good-first-issues board. The best contribution is a new test case. Add a detection corpus case (signetry_eval/detection/corpus/, with a cited provenance; SAFE decoys must stay 0 false positives) or an adversarial scenario (signetry_eval/scenarios/). See docs/SUBMITTING.md for the leaderboard paths, plus CONTRIBUTING.md and CODE_OF_CONDUCT.md. Every case ships with a test; CI gates the benchmark on 100% recall / 0 FP.

Part of the Signetry platform — see the umbrella overview.

License

Apache-2.0. Use it, fork it, ship it commercially — no strings.

This repository is part of Signetry's open-core model: the integration surface is Apache-2.0 so anyone can add an agent, an editor, or a CI adapter, while the engine (Signetry/core) is source-available under BUSL-1.1 and converts to Apache-2.0 on 2030-08-31.

Contributions are accepted under the CLA — it lets us move a well-built adapter into the engine later without asking every contributor for permission again.