Repository navigation
ci: add a weekly mutation check for the security checks' tests - #811
Merged
Merged
Conversation
Add mutation.yml, which re-applies a curated list of mutations to the library's security checks and fails if any mutant survives, so a refactor can't quietly remove the test protection a check has. - .github/mutation/mutants.json lists each mutant: the check's file, a literal snippet that must occur exactly once there (never a line number), its replacement, the packages and tests that must fail, and what the check guards. - .github/scripts/mutation-check.go (go run, build-tagged ignore so it stays out of the module's packages) runs a baseline of every listed test unmutated, checks each run filter names a real test, then applies each mutant, runs its tests and restores the file from the copy read before mutating it. A stale anchor, a mutant that doesn't compile or one that survives fails the run; the job summary gets a table of every result. - The list covers the checks given tests in #808, #809 and #810 (with boundary mutants where those tests pin a boundary), the security checks already protected (signature checks, caller key bounds, sealed-cookie key ID and expiry, code-exchange grant revocation, opaque-token issuer, notifier hardening, vouched keys at later chain hops) and sqlstore's PostgreSQL concurrency checks, against a PostgreSQL service container. - Runs weekly, on demand, and on pull requests that change the list or the runner. It isn't a required check. - check-postgres-pin.sh now checks every workflow's Postgres service image against .github/postgres/Dockerfile, mutation.yml's included. - CONTRIBUTING.md says a change moving an anchored check updates its entry, and how to run the list locally. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Split load and baseline into smaller functions to bring their cognitive complexity under Sonar's limit, run the go command from runtime.GOROOT() rather than a PATH lookup, and exclude .github/scripts from Sonar's coverage measure: it's CI tooling that mutation.yml exercises, never imported by the library. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a scheduled mutation check: a curated list of mutations to the library's security checks (a check deleted, inverted or moved by one), each run against the tests that must catch it. If a mutant survives, a refactor has removed the test protection that check had, even though
go test ./...stays green..github/mutation/mutants.json: one entry per mutant, giving the file, a literal snippet that must occur exactly once in it (anchored by text, never by line number), the replacement, the packages and-runfilter whose tests must fail, and a line saying what the check guards..github/scripts/mutation-check.go(go run,//go:build ignore):go run .github/scripts/mutation-check.go [ids...]..github/workflows/mutation.yml: runs weekly (Sunday 09:00 UTC), on demand, and on pull requests that change the list or the runner. It uses a PostgreSQL service with the same pin as ci.yml's sqlstore job. It's informational and isn't a required check.ValidateCallerCertificateKey, sealed-cookie key ID and expiry, code-exchange grant revocation, the opaque-token issuer, notifier outbound hardening, and vouched keys at later chain hops;check-postgres-pin.shnow checks every workflow's Postgres service image against.github/postgres/Dockerfile, which includes mutation.yml's.All 54 mutants are killed today. A full run takes about 2 minutes locally.
🤖 Generated with Claude Code