Skip to content

ci(security): fail the audit only on advisories a PR introduces, and gate on it - #275

Merged
Nic-dorman merged 2 commits into
mainfrom
nic/v2-1800-audit-pr-regressions
Nov 5, 2026
Merged

Nic-dorman merged 2 commits into
mainfrom
nic/v2-1800-audit-pr-regressions

Conversation

@Nic-dorman

@Nic-dorman Nic-dorman commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Round 2 (5cae049): fixes Hermes's finding that a PR could hide the advisory it introduces by ignoring it in .cargo/audit.toml. Both audits now apply the base branch's ignores, never the PR's. Review range 167bb22..5cae049.

Linear issue

Closes V2-1800

Security Audit ran a plain cargo audit on every PR and failed on 43 of the last 50. A new advisory comes from the RustSec database, not from a PR, so the day one is published against a dependency main already has, every open PR goes red. The current one is RUSTSEC-2026-0387 (crypto-bigint 0.5.5, V2-1743), and it has no upstream fix. Red became the normal state, so the job was kept out of CI Gate (#269).

The job now fails only on vulnerabilities the PR introduces, and it joins CI Gate:

  • .github/scripts/audit_regressions.sh [--config AUDIT_TOML] HEAD_LOCK [BASE_LOCK]:
    • Only the ignores in AUDIT_TOML apply, to both audits; with no --config, none do. cargo audit runs from a scratch directory that holds only that file, so the checkout's .cargo/audit.toml is never read (round 2). cargo-audit 0.22 reads only ./.cargo/audit.toml and $CARGO_HOME/audit.toml, which I checked on dev2; the second belongs to the runner. The log lists the ignores that were applied.
    • Audits both lockfiles against the same advisory database. The first run fetches it; the second uses --no-fetch.
    • Fails on any (advisory, crate) pair that is in the PR's lockfile and not in the base's. Moving an already-vulnerable crate to another vulnerable version is not new.
    • Vulnerabilities both lockfiles share are printed as ::warning::, and new informational advisories (unmaintained / unsound / yanked) as warnings, so they stay visible.
    • A cargo audit run that produces no parseable report fails the check. Its exit status can't tell a finding from a crash.
  • The job:
    • fetch-depth: 2, so the base lockfile is HEAD^1:Cargo.lock of the PR's merge commit.
    • It audits only when the PR changes Cargo.lock, .cargo/audit.toml, the script or ci.yml. Otherwise it passes with a notice, since nothing new can come in. It still runs and succeeds, because CI Gate counts a skipped job as a failure.
    • Fail safe: if HEAD isn't a PR merge commit, it audits the whole lockfile with nothing counted as existing.
    • It passes the base branch's .cargo/audit.toml (HEAD^1), or an empty file when the base has none, as --config. If the PR changes .cargo/audit.toml, the job prints a warning with the diff; the PR's ignores apply once it merges (round 2).
  • cargo-audit is pinned: cargo install cargo-audit --locked --version 0.22.2, which is the latest. It used to be unpinned, without --locked, and built on every PR. Now it's built only when the audit runs.
  • security is added to CI Gate's needs.

Decisions (Nic, 2026-11-04):

  • No ignore for RUSTSEC-2026-0387. V2-1743 stays the tracker. alloy-signer 2.5.0 and dtls 0.17.2 are still on RustCrypto 0.13. So the ticket's first "done when", Security Audit green on main's dependency set, isn't met by this PR. It waits on V2-1743, and this job reports 0387 as an existing-advisory warning until then.
  • No per-repo daily cron. Finding advisories newly published against main is V2-1767, the org-wide daily sweep that files one Linear triage issue per advisory. This job only answers "did this PR bring one in".

Risk tier

  • T0 — docs / tooling / CI / pure UX-output. Repo CI only.
  • T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
  • T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
  • T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Compatibility

  • Wire: none
  • Storage: none
  • API: none

Semver impact

  • breaking
  • feature
  • fix

Test evidence

actionlint .github/workflows/ci.yml: clean.

I ran the script on dev2 (cargo-audit 0.22.1, advisory DB of 2026-11-03, 1403 advisories) with real lockfiles:

PR lockfile Base Result
main pre260 pass. 0387 is a warning ("already in the base"); 0 new
webpki-vuln main fail: introduced by this change: RUSTSEC-2026-0382 rustls-webpki 0.103.15
bigint-moved main pass. 0387 on 0.5.4 counts as existing
pre260 main pass, with a new informational advisory: unsound RUSTSEC-2026-0386 tokio 1.53.1 warning
main (none) fail: 0387 counted as new (fail-safe path)
not a lockfile main fail: cargo audit produced no report
main not a lockfile fail: same

One bug turned up in testing and is fixed: with an empty base list, awk's NR == FNR also matched the PR's list, so the no-base path passed. The script now uses FILENAME == ARGV[1].

Round 2 (5cae049), on Windows with cargo-audit 0.22.1. The script was run from a PR checkout that adds rustls-webpki 0.103.15 (RUSTSEC-2026-0382), with base = main's Cargo.lock:

Case 167bb22 5cae049
PR's .cargo/audit.toml ignores 0382 (Hermes's case) pass, "0 new" fail: introduced by this change: RUSTSEC-2026-0382
same, no --config n/a fail, same
no ignore anywhere fail fail
lockfile unchanged pass, 0387 existing pass, 0387 existing
base config ignores 0387 n/a pass, ignores applied: RUSTSEC-2026-0387
PR ignores the inherited 0387 pass pass, ignore not applied, 0387 existing
--config file missing n/a fail: no audit config
no base lockfile fail, 0387 new fail, 0387 new

The job's audit step, extracted from the YAML, run on simulated PR merge commits:

  • base with no config, PR adds 0382 and an ignore: fail, plus the .cargo/audit.toml warning;
  • base ignores 0387, PR leaves it alone: pass, ignores applied: RUSTSEC-2026-0387;
  • base ignores 0387, PR swaps that ignore for one on 0382 and adds 0382: fail.

actionlint is clean.

This PR changes ci.yml, so its own Security Audit runs the full audit path, with an unchanged lockfile. It should pass, with 0387 as a warning.

New dependency

none. cargo-audit was already used; it's now pinned.

ADR

n/a

Mitigation / rollback

Revert the PR. The job goes back to a plain cargo audit outside CI Gate.

🤖 Generated with Claude Code

…gate on it

A plain cargo audit fails every PR the day an advisory is published
against a dependency main already has; that made Security Audit red on
43 of 50 PRs (RUSTSEC-2026-0387) and kept it out of CI Gate. Audit the
base and PR lockfiles against the same advisory database and fail only
on (advisory, crate) pairs the PR adds; existing ones are warnings. A PR
that doesn't change Cargo.lock passes without auditing. cargo-audit is
pinned to 0.22.2 with --locked, and the job joins CI Gate.

New advisories against main are the org sweep's job (V2-1767).

Closes V2-1800

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@MaidSafe-Hermes MaidSafe-Hermes left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — REQUEST_CHANGES

Finding: a PR can suppress the vulnerability this gate is meant to detect.

.github/scripts/audit_regressions.sh runs both cargo audit commands from the PR checkout, where cargo-audit reads .cargo/audit.toml. If a PR adds an advisory to that file’s ignore list, the advisory is omitted from both reports and the comparison passes. I reproduced this on head 167bb22ccddea2cb5fe2f5fe9572d263d0652703: adding an ignore for RUSTSEC-2026-0387 made the script report zero vulnerabilities and exit successfully, although the lockfile contains that advisory. The reproduction used local cargo-audit 0.22.1; CI on this PR uses 0.22.2.

Please run the head/base comparison with the base branch’s audit configuration, or explicitly reject new ignore entries that suppress findings. The workflow already triggers the audit when .cargo/audit.toml changes, but that does not prevent the bypass.

Other verification: the PR’s Security Audit and CI Gate pass on this head. Local bash -n passed and the audit against the base lockfile reported the existing RUSTSEC-2026-0387 as inherited. I could not run actionlint locally because it is not installed. Review coverage: 2/2 seats (sol-1 gpt-6.1-sol xhigh and glm-1 z-ai/glm-5.3); their verdicts differed, and the finding above was confirmed by the local reproduction.

This review was submitted by the authenticated GitHub account MaidSafe-Hermes.

Hermes round 1: cargo audit reads .cargo/audit.toml from its working
directory, so both audits ran with the PR's ignores. A PR could add an
ignore for the advisory it introduces and hide it from both reports,
and the gate passed. Reproduced: a lockfile with rustls-webpki 0.103.15
(RUSTSEC-2026-0382) plus an ignore for it passed with "0 new".

audit_regressions.sh now takes --config AUDIT_TOML and runs cargo audit
from a scratch directory holding only that file, so the checkout's
config is never read. (cargo audit 0.22 reads only ./.cargo/audit.toml
and $CARGO_HOME/audit.toml, checked on dev2.) CI passes the base
branch's copy, or an empty file when the base has none, and warns when
the PR changes .cargo/audit.toml: its ignores apply once it merges. The
log lists the ignores that were applied.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@MaidSafe-Hermes MaidSafe-Hermes left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — APPROVE

Focused re-review of the prior audit-config bypass on head 5cae04903605e76a3f5b8547c196bdfee98f9e91; this is not a fresh full-PR review.

Prior finding resolved. The script now runs cargo-audit from a scratch directory containing only the explicitly supplied config. For PR merge commits, the workflow supplies the base branch’s .cargo/audit.toml (or an empty config if none exists), not the PR’s version. It warns when the PR changes that file. This prevents a PR-added ignore from hiding an introduced advisory from both reports.

I verified the exact-head fix with an adversarial local run: the test PR config ignored RUSTSEC-2026-0382, but the script used the empty base config, reported the introduced rustls-webpki 0.103.15 advisory, and exited 1; RUSTSEC-2026-0387 remained classified as inherited. The actual PR lockfile audit also passed with 0387 reported as inherited. These local runs used cargo-audit 0.22.1; the PR pins 0.22.2.

Review coverage: 2/2 seats (sol-1 gpt-6.1-sol xhigh; glm-1 z-ai/glm-5.3). glm-1 approved. sol-1 returned UNCERTAIN because its supplied evidence did not establish the workflow checkout/config-selection path; I checked the full workflow and the exact-head Security Audit result, which confirm the base-config path for this pull_request run. No remaining blocker from my side.

Checks at review time: Security Audit, Quick checks, MSRV, build, storage jobs and PR metadata checks passed. Linux, macOS and Windows test jobs were still running; the CI Gate had not yet completed. Approval is not a merge signal. actionlint was unavailable locally; bash -n and git diff --check passed.

Submitted by authenticated GitHub account MaidSafe-Hermes.

@Nic-dorman
Nic-dorman merged commit dbd4f1b into main Nov 5, 2026
18 checks passed
@Nic-dorman
Nic-dorman deleted the nic/v2-1800-audit-pr-regressions branch November 5, 2026 13:56
grumbach added a commit to grumbach/ant-node that referenced this pull request Nov 6, 2026
Brings in main up to 360d472: OS-assigned ports for e2e test nodes (WithAutonomi#281)
and devnet nodes (WithAutonomi#282), release-key and signature checks (WithAutonomi#276), the
third-party notices and audit-regression CI gates (WithAutonomi#274, WithAutonomi#275), the 10 s
DHT request timeout for test nodes (WithAutonomi#277), and the 0.24.0 release (WithAutonomi#283).

One conflict, in tests/e2e/harness.rs: main removed the network bring-up
retry on a fresh port range together with NETWORK_START_ATTEMPTS and its
`warn` import, and this branch had added two helpers next to them
(POSSESSION_PENALTY_TRUST_RATIO and parseable_single_node_proof). The
resolution keeps both helpers and takes main's removal.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants