Apache-2.0, the Agent Governance Leaderboard, and an honest competitor score

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,71 @@
+name: Bug report
+description: Something in the Codex integration behaves incorrectly
+labels: ["bug"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for the report. This repository is the **Codex integration surface**
+        (MCP server wiring, the `signetry guard` lifecycle hook, `config.toml`).
+        Governance logic — contract evaluation, guard decisions, the verifier, receipts —
+        lives in [signetry-core](https://github.com/Signetry/core); please file those
+        there.
+
+        For a **security vulnerability**, do not open an issue: use
+        [private vulnerability reporting](https://github.com/Signetry/eval/security/advisories/new)
+        instead. See [SECURITY.md](https://github.com/Signetry/eval/blob/main/SECURITY.md).
+
+  - type: textarea
+    id: what-happened
+    attributes:
+      label: What happened
+      description: What you saw, and what you expected instead.
+    validations:
+      required: true
+
+  - type: textarea
+    id: repro
+    attributes:
+      label: Steps to reproduce
+      description: The smallest sequence that shows the problem.
+      placeholder: |
+        1. Wired the MCP server into ~/.codex/config.toml as documented
+        2. Asked Codex to ...
+        3. Observed ...
+    validations:
+      required: true
+
+  - type: textarea
+    id: config
+    attributes:
+      label: Relevant config
+      description: >
+        Your `~/.codex/config.toml` Signetry block and/or `.signetry/admission.yaml`.
+        Redact anything private — never paste an API key.
+      render: toml
+
+  - type: textarea
+    id: output
+    attributes:
+      label: Command output
+      description: >
+        Output from the failing command (e.g. `signetry guard ...`), or the Codex
+        transcript around the failure.
+      render: shell
+
+  - type: input
+    id: core-version
+    attributes:
+      label: signetry-core version
+      description: The pin you installed (e.g. `v0.7.0`), or the output of `signetry --version`.
+    validations:
+      required: true
+
+  - type: input
+    id: env
+    attributes:
+      label: Environment
+      description: OS, Python version, Codex version.
+      placeholder: macOS 15.3, Python 3.12.4, Codex CLI 0.x
+    validations:
+      required: true

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,13 @@
+blank_issues_enabled: true
+contact_links:
+  - name: Governance engine (signetry-core)
+    url: https://github.com/Signetry/core/issues
+    about: >
+      Contract evaluation, guard decisions, the verifier, receipts, and scanning live
+      in the engine. File engine behaviour there, not here.
+  - name: Signetry platform overview
+    url: https://github.com/Signetry/signetry
+    about: How the integrations, the engine, and the CI action fit together.
+  - name: Report a security vulnerability (private)
+    url: https://github.com/Signetry/eval/security/advisories/new
+    about: Signetry is a security tool. Please report privately, never in a public issue.

.github/ISSUE_TEMPLATE/feature_request.yml

@@ -0,0 +1,43 @@
+name: Feature request
+description: Suggest an improvement to the Codex integration
+labels: ["enhancement"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        This repository is Apache-2.0 and contributions are welcome — see
+        [CONTRIBUTING.md](https://github.com/Signetry/eval/blob/main/CONTRIBUTING.md).
+
+        Scope check: this repo wires Codex to the engine. It **never reimplements
+        policy** and never auto-merges. Requests to change how contracts are
+        evaluated, how receipts are signed, or what the guard decides belong in
+        [signetry-core](https://github.com/Signetry/core).
+
+  - type: textarea
+    id: problem
+    attributes:
+      label: The problem
+      description: What are you trying to do that this integration makes hard or impossible today?
+    validations:
+      required: true
+
+  - type: textarea
+    id: proposal
+    attributes:
+      label: Proposed change
+      description: What should the integration do instead? Config snippets welcome.
+    validations:
+      required: true
+
+  - type: textarea
+    id: alternatives
+    attributes:
+      label: Alternatives considered
+      description: Workarounds you tried, and why they were not enough.
+
+  - type: checkboxes
+    id: contribute
+    attributes:
+      label: Would you like to implement this?
+      options:
+        - label: I'm willing to open a pull request (I'll sign the CLA).

.github/workflows/benchmark.yml

@@ -1,30 +1,55 @@
 name: benchmark
 
-# Regenerate the detection head-to-head table on every release (and on demand),
-# publish it as a release artifact, and commit the refreshed markdown into
-# docs/BENCHMARK.md so the published numbers always match the shipped engine.
+# Regenerates docs/BENCHMARK.md — the detection head-to-head against the other
+# scanners — and publishes it as a release artifact, so the published numbers always
+# match the shipped engine.
+#
+# This workflow owns docs/BENCHMARK.{md,json} and nothing else; the governance
+# leaderboard is owned by leaderboard.yml. Both push to main, so both sit in the
+# `published-numbers` concurrency group.
+#
+# It runs on a schedule as well as on releases: the detection corpus and the competing
+# scanners both change between tags, and a table only refreshed at release time can be
+# months out of date while still reading as current.
 on:
+  schedule:
+    # Mondays 06:47 UTC. Off the hour on purpose (scheduled jobs queued at :00 are
+    # delayed under load) and clear of leaderboard.yml's 07:23 slot.
+    - cron: "47 6 * * 1"
   release:
     types: [published]
+  push:
+    branches: [main]
+    paths:
+      - "signetry_eval/corpus/**"
+      - "signetry_eval/detection.py"
   workflow_dispatch:
 
 permissions:
   contents: write
 
+concurrency:
+  group: published-numbers
+  cancel-in-progress: false
+
 jobs:
   benchmark:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v7
+        with:
+          # Never the incoming ref: on a release event that is the tag, which
+          # leaves us in detached HEAD and turns the push below into a tag write.
+          ref: ${{ github.event.repository.default_branch || 'main' }}
       - uses: actions/setup-python@v7
         with:
           python-version: "3.12"
       - name: Install
         run: pip install -e ".[dev]"
       - name: Require the signetry-core detection engine
         run: |
-          # The release benchmark must run against a signetry-core that has the SAST
-          # engine. Fail loudly if it is missing (a real release misconfiguration).
+          # This benchmark has no meaning without the SAST engine, and an absent scanner
+          # must never be published as a zero. Fail rather than emit a table.
           python -c "from signetry_core import scan_repository" || {
             echo "::error::Installed signetry-core lacks scan_repository; pin a signetry-core with the detection engine."
             exit 1
@@ -38,19 +63,26 @@ jobs:
             signetry-eval corpus --markdown
           } > docs/BENCHMARK.md
           signetry-eval corpus --json > docs/BENCHMARK.json
-      - name: Gate on parity (Signetry must keep 100% recall, 0 false positives)
-        run: signetry-eval corpus --min-recall 1.0 --max-fp 0
       - uses: actions/upload-artifact@v7
         with:
           name: detection-benchmark
           path: |
             docs/BENCHMARK.md
             docs/BENCHMARK.json
-      - name: Commit refreshed benchmark (on release)
-        if: github.event_name == 'release'
+          if-no-files-found: error
+      - name: Commit the refreshed benchmark
         run: |
           git config user.name "github-actions[bot]"
           git config user.email "github-actions[bot]@users.noreply.github.com"
           git add docs/BENCHMARK.md docs/BENCHMARK.json
-          git commit -m "docs: refresh detection head-to-head benchmark for ${{ github.event.release.tag_name }}" || echo "no changes"
-          git push || echo "push skipped"
+          git diff --cached --quiet && { echo "no change to publish"; exit 0; }
+          git commit -m "docs: refresh the detection head-to-head benchmark"
+          # leaderboard.yml pushes to the same branch. The concurrency group serializes
+          # us; rebase anyway rather than force or swallow the failure.
+          git pull --rebase --autostash origin "${{ github.event.repository.default_branch || 'main' }}"
+          git push origin "HEAD:${{ github.event.repository.default_branch || 'main' }}"
+      - name: Gate on parity (Signetry must keep 100% recall, 0 false positives)
+        run: |
+          # Last, deliberately: the regression is published above whatever this says,
+          # then this step fails the run so it cannot be missed.
+          signetry-eval corpus --min-recall 1.0 --max-fp 0

.github/workflows/cla.yml

@@ -3,8 +3,9 @@ name: CLA
 # Requires every contributor to sign the Contributor License Agreement (CLA.md)
 # before their pull request can be merged. Runs entirely in this repo's Actions —
 # no third-party OAuth app. Signatures are recorded in signatures/cla.json on this
-# repo, so there is a durable, auditable record that each contributor assigned
-# copyright/ownership of their contribution to the Owner.
+# repo, so there is a durable, auditable record that each contributor granted the
+# rights the CLA describes — which is what keeps relicensing across the open-core
+# line (Apache-2.0 integrations <-> the BUSL-1.1 engine) clean.
 #
 # A contributor signs by commenting the exact statement below on their PR:
 #   I have read the CLA Document and I hereby sign the CLA
@@ -48,7 +49,7 @@ jobs:
           path-to-document: "https://github.com/Signetry/eval/blob/main/CLA.md"
           branch: "cla-signatures"
           allowlist: "dependabot[bot],bkd-dotcom"
-          custom-notsigned-prcomment: "Thanks for your contribution! Before it can be merged, please read the **[Contributor License Agreement](https://github.com/Signetry/eval/blob/main/CLA.md)** — Signetry is **All Rights Reserved**, and by signing you assign copyright/ownership of your contribution to the Owner (you may not use, sell, or commercialize it yourself). To agree, reply with exactly:"
+          custom-notsigned-prcomment: "Thanks for your contribution! Before it can be merged, please read the **[Contributor License Agreement](https://github.com/Signetry/eval/blob/main/CLA.md)** — this repository is **Apache-2.0**, and the CLA is what lets us relicense a contribution across the open-core line (for example, moving a well-built adapter into the BUSL-1.1 engine) without asking every contributor again. You keep every right the Apache-2.0 licence gives you. To agree, reply with exactly:"
           custom-pr-sign-comment: "I have read the CLA Document and I hereby sign the CLA"
           custom-allsigned-prcomment: "All contributors have signed the CLA. ✅"
           lock-pullrequest-aftermerge: false

.github/workflows/eval.yml

@@ -19,7 +19,8 @@ jobs:
       - name: Install
         run: |
           # signetry-core is installed from its source repo (a git+https dependency in
-          # pyproject.toml); it is All Rights Reserved and not on PyPI.
+          # pyproject.toml); it is source-available under BUSL-1.1 and not on PyPI.
+          # This repo (signetry-eval) is Apache-2.0.
           pip install -e ".[dev]"
       - name: Ruff
         run: ruff check signetry_eval/ tests/

.github/workflows/leaderboard.yml

@@ -0,0 +1,120 @@
+name: leaderboard
+
+# Regenerates docs/LEADERBOARD.md — the two-axis governance leaderboard, including
+# third-party submissions from leaderboard/entries/.
+#
+# Why a schedule and not just releases: the governance axis takes submissions, so the
+# page changes when someone else's PR lands, not only when we cut a tag. A leaderboard
+# whose numbers are as old as the last release is a changelog.
+#
+# This workflow owns docs/LEADERBOARD.{md,json} and nothing else. The detection
+# head-to-head (docs/BENCHMARK.md) is owned by benchmark.yml — one generator per file,
+# so every "Generated by ..." header names the workflow that actually wrote it. Both
+# share the `published-numbers` concurrency group because both push to main.
+on:
+  schedule:
+    # Mondays 07:23 UTC. Off the hour on purpose: scheduled jobs queued at :00 are
+    # delayed under load, and 07:23 leaves benchmark.yml's 06:47 slot clear.
+    - cron: "23 7 * * 1"
+  release:
+    types: [published]
+  push:
+    branches: [main]
+    paths:
+      - "leaderboard/entries/**"
+      - "signetry_eval/leaderboard.py"
+      - "signetry_eval/report.py"
+      - "signetry_eval/scenarios/**"
+  workflow_dispatch:
+
+permissions:
+  contents: write
+
+concurrency:
+  group: published-numbers
+  cancel-in-progress: false
+
+jobs:
+  leaderboard:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          # Never the incoming ref: on a release event that is the tag, which
+          # leaves us in detached HEAD and turns the push below into a tag write.
+          ref: ${{ github.event.repository.default_branch || 'main' }}
+      - uses: actions/setup-python@v7
+        with:
+          python-version: "3.12"
+      - name: Install
+        run: pip install -e ".[dev]"
+
+      - name: Does this build have the detection engine?
+        id: engine
+        run: |
+          # Axis 2 needs signetry-core's SAST engine. If it is absent we publish axis 1
+          # alone and say so on the page. We do not print an empty table, and we do not
+          # score a scanner that did not run as zero.
+          if python -c "from signetry_core import scan_repository" 2>/dev/null; then
+            echo "available=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "available=false" >> "$GITHUB_OUTPUT"
+            echo "::warning::signetry-core lacks scan_repository; publishing the governance axis only."
+          fi
+
+      - name: Regenerate the leaderboard
+        run: |
+          mkdir -p docs
+          # Record WHICH signetry-core produced the live row. A leaderboard that says
+          # "measured live in CI" without naming the version measured cannot be
+          # reproduced by a reader, which is the one thing it exists to support.
+          # Read from the installed package, never hardcoded — a stale label would be
+          # worse than the null it replaces. Left unset if it cannot be read, so the
+          # row degrades to "not measured" rather than to a guess.
+          CORE_VERSION="$(python -c 'import signetry_core; print(signetry_core.__version__)' 2>/dev/null || true)"
+          if [ -n "$CORE_VERSION" ]; then
+            VERSION_ARG="--version $CORE_VERSION"
+            echo "live row labelled signetry-core $CORE_VERSION"
+          else
+            VERSION_ARG=""
+            echo "::warning::could not read signetry_core.__version__; the live row will be unlabelled."
+          fi
+
+          # Runs the adversarial suite for the live signetry-core row. A failing defense
+          # still publishes — the row shows the failure. The gate below is what fails.
+          if [ "${{ steps.engine.outputs.available }}" = "true" ]; then
+            signetry-eval leaderboard --with-detection $VERSION_ARG > docs/LEADERBOARD.md
+            signetry-eval leaderboard --with-detection $VERSION_ARG --json > docs/LEADERBOARD.json
+          else
+            signetry-eval leaderboard $VERSION_ARG > docs/LEADERBOARD.md
+            signetry-eval leaderboard $VERSION_ARG --json > docs/LEADERBOARD.json
+          fi
+
+      - uses: actions/upload-artifact@v7
+        with:
+          name: governance-leaderboard
+          path: |
+            docs/LEADERBOARD.md
+            docs/LEADERBOARD.json
+          if-no-files-found: error
+
+      - name: Commit the refreshed page
+        run: |
+          git config user.name "github-actions[bot]"
+          git config user.email "github-actions[bot]@users.noreply.github.com"
+          git add docs/LEADERBOARD.md docs/LEADERBOARD.json
+          git diff --cached --quiet && { echo "no change to publish"; exit 0; }
+          git commit -m "docs: refresh the governance leaderboard"
+          # benchmark.yml pushes to the same branch on release. The concurrency group
+          # serializes us, but rebase anyway rather than force or fail.
+          git pull --rebase --autostash origin "${{ github.event.repository.default_branch || 'main' }}"
+          git push origin "HEAD:${{ github.event.repository.default_branch || 'main' }}"
+
+      - name: Gate — every governed defense must hold
+        run: |
+          # Last, deliberately: the page is published above whatever this says, then this
+          # step fails the run so a broken defense is impossible to miss.
+          signetry-eval run > /dev/null || {
+            echo "::error::An adversarial scenario's defense did not hold. The leaderboard was published with the failing row."
+            exit 1
+          }

.github/workflows/reviewer.yml

@@ -34,7 +34,7 @@ jobs:
         with:
           python-version: "3.12"
       - name: Install signetry-reviewer
-        # source-available (All Rights Reserved); install from source, not PyPI.
+        # signetry-reviewer is not published on PyPI; install it from its source repo.
         run: pip install "signetry-reviewer @ git+https://github.com/Signetry/reviewer@v0.2.0"
       - name: Compute the PR diff
         env: