ci: add the fail-open start-sccache.sh probe for the native Rust jobs - #342
Conversation
The dashboard provenance column was blank on every row: the observer resolved Absent for every image, including protector's own signed+attested images. The feature had been inert since it shipped. Root cause is two independent dead ends in the sigstore crate (0.14, newest): its trusted_signature_layers hardcodes the cosign sign/v1 predicate and drops every SLSA attestation, and its bundle::verify::Verifier recomputes the Rekor DSSE envelopeHash from a proto round-trip that never matches what Rekor stored (offline always fails; online is unimplemented). Both confirmed against the live agent image; the primitives to hand-roll around them are pub(crate). Add the sigstore-verify crate (prefix-dev) for the provenance verify step only — it handles GitHub artifact attestation correctly and ships a built-in trust root. A new provenance_observer module fetches the image's OCI referrers directly, selects the slsa.dev/provenance bundle, verifies it offline against the same registry (no new egress), and feeds the unchanged classify_provenance pipeline. A registry without the referrers API resolves to Absent, not perpetual Checking. Validated live: protector-agent -> Verified (correct source + builder); redis / ollama / argocd -> Absent. Adds unit tests plus an ignored live integration test guarding the exact green-on-fixtures / dead-in-prod failure mode. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ries The audit gate flagged two newly-published advisories in pre-existing transitive deps: h2 (RUSTSEC-2026-0258, unbounded empty DATA frames) and webbrowser (RUSTSEC-2026-0257, Unix BROWSER argument injection). Both are lock-only, semver-compatible bumps. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sccache's S3/R2 backend is EAGER: when the bucket is unreachable, `sccache --start-server` fails outright, and the first rustc-through-sccache call in the `lint`/`test` jobs then dies with "sccache: Timed out waiting for server startup" — an R2 outage becomes a hard CI outage, not a slow build. The Docker-build side of this was already fixed (scripts/start-sccache-docker.sh, wired into the buildkit `docker`/`image` jobs via BuildKit secrets), and its header comment even names a sibling ".github/scripts/start-sccache.sh" that never existed — the native `lint` and `test` jobs in rust.yml install sccache and set RUSTC_WRAPPER but never pre-flight the backend, so they were still exposed to the same failure mode this ticket is about. Port the probe (proven in the metrics/murmurify repos): try --start-server against R2 up to 3 times, and on failure fall back to a local disk cache by appending to GITHUB_ENV. The load-bearing detail: GITHUB_ENV can't *unset* a variable, and a non-empty SCCACHE_BUCKET always wins over SCCACHE_DIR in sccache's backend selection — so the fallback exports an EMPTY SCCACHE_BUCKET alongside SCCACHE_DIR, not just SCCACHE_DIR alone. Both a missing bucket (Secret never reached the pod) and an unreachable one degrade to local disk with a `::warning::` log line instead of failing the job. Adds a --selftest mode (mirroring scripts/start-sccache-docker.sh's own fixture harness) that drives all three states against a stubbed sccache binary, and wires it into the `lint` job so a regression in the fallback logic — in particular the empty-SCCACHE_BUCKET export — fails loudly in CI instead of during a real R2 outage. The reachability check is sccache's own --start-server call (real HTTPS egress to R2, same path the real cache traffic uses), not a separate network probe — so it needs nothing beyond what protector-runners already have for reaching R2, meshed or not. Closes JEF-946. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
rust.yml's lint and test jobs run on protector-runners with no fork check on pull_request. The R2 cutover (this PR) means those pods now carry a bucket-scoped Cloudflare R2 credential with Object Read & Write on the shared sccache bucket, injected via envFrom from the cluster repo's sccache-r2 Secret. Under the old redis backend this wasn't worth attacking (the credential was an unforgeable in-cluster mTLS identity); under R2 it's a portable key, so a `run: env` step in a PR from any forked repo could walk off with write access to the shared cache. ADR-0020 keeps one flat shared keyspace across every repo's CI (a dep crate compiles once, every repo reuses the object), so a poisoned object written from here is consumed everywhere else too. The prescribed remedy for a suspected leak is a full bucket wipe + token rotation — worth guarding against up front. Adds the same job-level `if` guard agent.yml already carries on its self-hosted jobs (and whisperer's rust.yml), scoped to the job so a fork PR never gets the pod (env) in the first place, not just a step inside it. The `docker` job doesn't need it: its own `if: startsWith(github.ref, 'refs/tags/v')` already excludes every pull_request event structurally (no PR ref ever matches a tag ref). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Folded in JEF-957 while this PR was open: added the job-level fork-PR guard to `rust.yml`'s `lint` and `test` jobs (self-hosted `protector-runners`). Why here: this PR's own R2 cutover is what makes the missing guard exploitable — the runner pod now carries a bucket-scoped Cloudflare R2 credential with Object Read & Write on the shared `sccache` bucket (envFrom the cluster repo's `sccache-r2` Secret), vs. the old redis backend's unforgeable in-cluster mTLS identity. Without a guard, any fork-PR's `run: env` step walks off with write access to a bucket every repo's CI shares (ADR-0020's flat keyspace), and the fix for a suspected leak is a full bucket wipe + rotation. Copied the exact guard already on `agent.yml`'s self-hosted jobs (and whisperer's `rust.yml`) so the pattern stays grep-able across repos: ```yaml Left the `docker` job alone — its own `if: startsWith(github.ref, 'refs/tags/v')` already structurally excludes every `pull_request` event (no PR ref matches a tag ref), so it doesn't need the guard. |
…-open-start-sccachesh-probe-r2-backend
Problem
sccache's S3/R2 backend is EAGER: when the bucket is unreachable,
sccache --start-serverfails outright, and the first rustc-through-sccache call dieswith
sccache: Timed out waiting for server startup, killing the job. An R2outage is a hard CI outage, not a slow build — the same blast radius the old
redis backend had, now pointed at an off-cluster dependency.
What I found
Protector's Docker-build side of this cutover was already fixed:
scripts/start-sccache-docker.sh(fail-soft, wired into the buildkitdockerjob inrust.ymland theimagejob inagent.ymlvia BuildKitsecrets). Its own header comment even names a sibling
.github/scripts/start-sccache.shthat never existed.Enumerating what actually starts sccache natively (not inside a Docker build):
rust.yml'slintandtestjobs both install sccache and setRUSTC_WRAPPER: sccache, but neither had a pre-flight probe — sccache startslazily on the first wrapped
cargo/rustccall, so both were exposed to theexact failure mode above.
agent.yml'sebpfjob compiles with the toolchainbaked into the runner image and never touches sccache, so it's out of scope.
What changed
.github/scripts/start-sccache.sh(new): ported from the proven probein the metrics/murmurify repos. Tries
sccache --start-serveragainst R2 upto 3 times; on failure, falls back to a local disk cache by appending to
GITHUB_ENV. The load-bearing detail:GITHUB_ENVcan't unset avariable, and sccache picks a non-empty
SCCACHE_BUCKEToverSCCACHE_DIRregardless of order — so the fallback exports an empty
SCCACHE_BUCKETalongside
SCCACHE_DIR, notSCCACHE_DIRalone. A missing bucket (thesccache-r2Secret never reached the pod) and an unreachable one bothdegrade the same way, with a loud
::warning::line so a silently-coldcache doesn't read as fine.
.github/workflows/rust.yml: wired the probe into bothlintandtest, right afterInstall sccacheand before anything that would triggera lazy server start.
--selftestmode to the new script (mirroringscripts/start-sccache-docker.sh's own fixture harness) that drives allthree states — no config, R2 up, R2 down — against a stubbed
sccachebinary, and wired it into the
lintjob so a regression in the fallback(especially the empty-bucket export) fails loudly in CI instead of during a
real outage.
On the meshed-runner note
protector-runners are meshed (JEF-84), so I checked whether the probe's
reachability method would be blocked. It isn't a separate network check —
it's sccache's own
--start-servercall, i.e. the same HTTPS egress to R2the real cache traffic already uses. Nothing about meshing changes that path,
so the ported method carries over unchanged; I'm not aware of anything to
flag here.
Testing
CI-only change — no Rust code touched, so
cargo fmt/clippy/tests don'tapply.
bash .github/scripts/start-sccache.sh --selftest— all 9 assertions passlocally, covering: no bucket in pod env, R2 reachable, R2 unreachable
(3 retries then fallback with the empty-bucket export).
export line and reran
--selftest— 2 of 9 assertions failed as expected.shellcheckon the new script: one pre-existing-patternSC1007warningidentical to the one already present in the reference metrics script and
in
scripts/start-sccache-docker.sh(theVAR= cmdempty-assignmentidiom) — not a real issue, and this repo runs no shellcheck gate in CI.
python3 -c "import yaml; yaml.safe_load(...)"onrust.yml— parses.actionlinton both touched workflows — only pre-existing findings(
protector-runnersself-hosted label unknown to actionlint's defaultconfig, one unrelated
SC2086in an untouched step further down); nothingnew from this diff.
/soundcheck:pr-review— no Critical/High findings (no secrets, noinjectable input;
SCCACHE_BUCKETcomes from a cluster Secret, notattacker-controlled data).
Closes JEF-946.
🤖 Generated with Claude Code