Skip to content

ci: add the fail-open start-sccache.sh probe for the native Rust jobs - #342

Merged
thejefflarson merged 5 commits into
mainfrom
thejefflarson/jef-946-protector-add-the-fail-open-start-sccachesh-probe-r2-backend
Sep 6, 2026
Merged

thejefflarson merged 5 commits into
mainfrom
thejefflarson/jef-946-protector-add-the-fail-open-start-sccachesh-probe-r2-backend

Conversation

@thejefflarson

Copy link
Copy Markdown
Owner

Problem

sccache's S3/R2 backend is EAGER: when the bucket is unreachable, sccache --start-server fails outright, and the first rustc-through-sccache call dies
with sccache: Timed out waiting for server startup, killing the job. An R2
outage is a hard CI outage, not a slow build — the same blast radius the old
redis backend had, now pointed at an off-cluster dependency.

What I found

Protector's Docker-build side of this cutover was already fixed:
scripts/start-sccache-docker.sh (fail-soft, wired into the buildkit
docker job in rust.yml and the image job in agent.yml via BuildKit
secrets). Its own header comment even names a sibling
.github/scripts/start-sccache.sh that never existed.

Enumerating what actually starts sccache natively (not inside a Docker build):
rust.yml's lint and test jobs both install sccache and set
RUSTC_WRAPPER: sccache, but neither had a pre-flight probe — sccache starts
lazily on the first wrapped cargo/rustc call, so both were exposed to the
exact failure mode above. agent.yml's ebpf job compiles with the toolchain
baked into the runner image and never touches sccache, so it's out of scope.

What changed

  • .github/scripts/start-sccache.sh (new): ported from the proven probe
    in the metrics/murmurify repos. Tries sccache --start-server against R2 up
    to 3 times; on failure, falls back to a local disk cache by appending to
    GITHUB_ENV. The load-bearing detail: GITHUB_ENV can't unset a
    variable, and sccache picks a non-empty SCCACHE_BUCKET over SCCACHE_DIR
    regardless of order — so the fallback exports an empty SCCACHE_BUCKET
    alongside SCCACHE_DIR, not SCCACHE_DIR alone. A missing bucket (the
    sccache-r2 Secret never reached the pod) and an unreachable one both
    degrade the same way, with a loud ::warning:: line so a silently-cold
    cache doesn't read as fine.
  • .github/workflows/rust.yml: wired the probe into both lint and
    test, right after Install sccache and before anything that would trigger
    a lazy server start.
  • Added a --selftest mode to the new script (mirroring
    scripts/start-sccache-docker.sh's own fixture harness) that drives all
    three states — no config, R2 up, R2 down — against a stubbed sccache
    binary, and wired it into the lint job so a regression in the fallback
    (especially the empty-bucket export) fails loudly in CI instead of during a
    real outage.

On the meshed-runner note

protector-runners are meshed (JEF-84), so I checked whether the probe's
reachability method would be blocked. It isn't a separate network check —
it's sccache's own --start-server call, i.e. the same HTTPS egress to R2
the real cache traffic already uses. Nothing about meshing changes that path,
so the ported method carries over unchanged; I'm not aware of anything to
flag here.

Testing

CI-only change — no Rust code touched, so cargo fmt/clippy/tests don't
apply.

  • bash .github/scripts/start-sccache.sh --selftest — all 9 assertions pass
    locally, covering: no bucket in pod env, R2 reachable, R2 unreachable
    (3 retries then fallback with the empty-bucket export).
  • Verified the test actually catches a regression: broke the empty-bucket
    export line and reran --selftest — 2 of 9 assertions failed as expected.
  • shellcheck on the new script: one pre-existing-pattern SC1007 warning
    identical to the one already present in the reference metrics script and
    in scripts/start-sccache-docker.sh (the VAR= cmd empty-assignment
    idiom) — not a real issue, and this repo runs no shellcheck gate in CI.
  • python3 -c "import yaml; yaml.safe_load(...)" on rust.yml — parses.
  • actionlint on both touched workflows — only pre-existing findings
    (protector-runners self-hosted label unknown to actionlint's default
    config, one unrelated SC2086 in an untouched step further down); nothing
    new from this diff.
  • Ran /soundcheck:pr-review — no Critical/High findings (no secrets, no
    injectable input; SCCACHE_BUCKET comes from a cluster Secret, not
    attacker-controlled data).

Closes JEF-946.

🤖 Generated with Claude Code

thejefflarson and others added 4 commits August 25, 2026 00:39
The dashboard provenance column was blank on every row: the observer resolved
Absent for every image, including protector's own signed+attested images. The
feature had been inert since it shipped.

Root cause is two independent dead ends in the sigstore crate (0.14, newest):
its trusted_signature_layers hardcodes the cosign sign/v1 predicate and drops
every SLSA attestation, and its bundle::verify::Verifier recomputes the Rekor
DSSE envelopeHash from a proto round-trip that never matches what Rekor stored
(offline always fails; online is unimplemented). Both confirmed against the live
agent image; the primitives to hand-roll around them are pub(crate).

Add the sigstore-verify crate (prefix-dev) for the provenance verify step only —
it handles GitHub artifact attestation correctly and ships a built-in trust root.
A new provenance_observer module fetches the image's OCI referrers directly,
selects the slsa.dev/provenance bundle, verifies it offline against the same
registry (no new egress), and feeds the unchanged classify_provenance pipeline.
A registry without the referrers API resolves to Absent, not perpetual Checking.

Validated live: protector-agent -> Verified (correct source + builder); redis /
ollama / argocd -> Absent. Adds unit tests plus an ignored live integration test
guarding the exact green-on-fixtures / dead-in-prod failure mode.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ries

The audit gate flagged two newly-published advisories in pre-existing transitive
deps: h2 (RUSTSEC-2026-0258, unbounded empty DATA frames) and webbrowser
(RUSTSEC-2026-0257, Unix BROWSER argument injection). Both are lock-only,
semver-compatible bumps.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sccache's S3/R2 backend is EAGER: when the bucket is unreachable,
`sccache --start-server` fails outright, and the first rustc-through-sccache
call in the `lint`/`test` jobs then dies with "sccache: Timed out waiting for
server startup" — an R2 outage becomes a hard CI outage, not a slow build.

The Docker-build side of this was already fixed (scripts/start-sccache-docker.sh,
wired into the buildkit `docker`/`image` jobs via BuildKit secrets), and its
header comment even names a sibling ".github/scripts/start-sccache.sh" that
never existed — the native `lint` and `test` jobs in rust.yml install sccache
and set RUSTC_WRAPPER but never pre-flight the backend, so they were still
exposed to the same failure mode this ticket is about.

Port the probe (proven in the metrics/murmurify repos): try --start-server
against R2 up to 3 times, and on failure fall back to a local disk cache by
appending to GITHUB_ENV. The load-bearing detail: GITHUB_ENV can't *unset* a
variable, and a non-empty SCCACHE_BUCKET always wins over SCCACHE_DIR in
sccache's backend selection — so the fallback exports an EMPTY SCCACHE_BUCKET
alongside SCCACHE_DIR, not just SCCACHE_DIR alone. Both a missing bucket
(Secret never reached the pod) and an unreachable one degrade to local disk
with a `::warning::` log line instead of failing the job.

Adds a --selftest mode (mirroring scripts/start-sccache-docker.sh's own
fixture harness) that drives all three states against a stubbed sccache
binary, and wires it into the `lint` job so a regression in the fallback
logic — in particular the empty-SCCACHE_BUCKET export — fails loudly in CI
instead of during a real R2 outage.

The reachability check is sccache's own --start-server call (real HTTPS
egress to R2, same path the real cache traffic uses), not a separate
network probe — so it needs nothing beyond what protector-runners already
have for reaching R2, meshed or not.

Closes JEF-946.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
rust.yml's lint and test jobs run on protector-runners with no fork
check on pull_request. The R2 cutover (this PR) means those pods now
carry a bucket-scoped Cloudflare R2 credential with Object Read &
Write on the shared sccache bucket, injected via envFrom from the
cluster repo's sccache-r2 Secret. Under the old redis backend this
wasn't worth attacking (the credential was an unforgeable in-cluster
mTLS identity); under R2 it's a portable key, so a `run: env` step in
a PR from any forked repo could walk off with write access to the
shared cache.

ADR-0020 keeps one flat shared keyspace across every repo's CI (a dep
crate compiles once, every repo reuses the object), so a poisoned
object written from here is consumed everywhere else too. The
prescribed remedy for a suspected leak is a full bucket wipe + token
rotation — worth guarding against up front.

Adds the same job-level `if` guard agent.yml already carries on its
self-hosted jobs (and whisperer's rust.yml), scoped to the job so a
fork PR never gets the pod (env) in the first place, not just a step
inside it. The `docker` job doesn't need it: its own
`if: startsWith(github.ref, 'refs/tags/v')` already excludes every
pull_request event structurally (no PR ref ever matches a tag ref).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@thejefflarson

Copy link
Copy Markdown
Owner Author

Folded in JEF-957 while this PR was open: added the job-level fork-PR guard to `rust.yml`'s `lint` and `test` jobs (self-hosted `protector-runners`).

Why here: this PR's own R2 cutover is what makes the missing guard exploitable — the runner pod now carries a bucket-scoped Cloudflare R2 credential with Object Read & Write on the shared `sccache` bucket (envFrom the cluster repo's `sccache-r2` Secret), vs. the old redis backend's unforgeable in-cluster mTLS identity. Without a guard, any fork-PR's `run: env` step walks off with write access to a bucket every repo's CI shares (ADR-0020's flat keyspace), and the fix for a suspected leak is a full bucket wipe + rotation.

Copied the exact guard already on `agent.yml`'s self-hosted jobs (and whisperer's `rust.yml`) so the pattern stays grep-able across repos:

```yaml
if: >-
github.event_name != 'pull_request' ||
github.event.pull_request.head.repo.full_name == github.repository
```

Left the `docker` job alone — its own `if: startsWith(github.ref, 'refs/tags/v')` already structurally excludes every `pull_request` event (no PR ref matches a tag ref), so it doesn't need the guard.

@thejefflarson
thejefflarson merged commit 0c2dd1b into main Sep 6, 2026
5 checks passed
@thejefflarson
thejefflarson deleted the thejefflarson/jef-946-protector-add-the-fail-open-start-sccachesh-probe-r2-backend branch September 6, 2026 06:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant