Warden flagged this in PR #1254 review (#1254 (comment), severity: high). Pre-existing risk carried since d4204f3 (PR #1306); out of scope for #1254's monorepo pre-shape, but worth a follow-up.
Background
eval-skill-fork.yml is the workflow that runs the SKILL.md doc eval on fork PRs. Today it:
- triggers on
pull_request_target: [labeled, synchronize] → has access to base-repo secrets
- checks out the fork's
head.sha (line 46) → runs attacker-controlled code
- runs
pnpm install --frozen-lockfile → invokes fork package.json preinstall / install hooks
- runs
pnpm run generate:schema && pnpm run generate:docs → fork-declared scripts
- runs
pnpm run eval:skill with OPENROUTER_API_KEY / ANTHROPIC_API_KEY in env
- later mints a
SENTRY_RELEASE_BOT app token from secrets.SENTRY_RELEASE_BOT_PRIVATE_KEY
The current gating control is that a maintainer must add the eval-skill label to start the job. That's a meaningful but soft mitigation — any forked repo with a label-arranged maintainer can still exfiltrate the secrets during the install/build phase before the LLM steps even run.
Suggested directions (pick the one that fits our threat model best — happy to discuss before implementing)
- Two-job pattern: split into a
pull_request job that runs on the fork checkout (no secrets, only does the build + skill validation), and a separate workflow_run job that re-runs validation from a trusted context if pull_request succeeded. Pass eval inputs via artifacts, not env.
- Pre-build + reuse: build the eval harness in a separate
push-main_only job, upload it as an artifact, and download it in the pull_request_target job instead of running pnpm install against the fork's lockfile. The fork eval only ever sees the artifact + SKILL.md.
- Network allowlist + drop-on-postinstall: still run
pnpm install but set up an egress allowlist + --ignore-scripts for the install phase. Reduces risk; doesn't eliminate.
- Lock fork checkout to base: check out the base branch and only git archive the changed files from the fork head SHA via the API, into a tarball. Eval harness runs against the tarball in isolation, no pnpm install against a fork-modified pnpm-lock.yaml.
Acceptable risk for now (PR #1254)
Marking Warden finding as /ar for the immediate PR — the workflow only runs when a maintainer eval-skill label is present, plus the LLM secrets are scoped to a single step. We'll come back to this in a follow-up PR with security review.
Warden flagged this in PR #1254 review (#1254 (comment), severity: high). Pre-existing risk carried since d4204f3 (PR #1306); out of scope for #1254's monorepo pre-shape, but worth a follow-up.
Background
eval-skill-fork.ymlis the workflow that runs the SKILL.md doc eval on fork PRs. Today it:pull_request_target: [labeled, synchronize]→ has access to base-repo secretshead.sha(line 46) → runs attacker-controlled codepnpm install --frozen-lockfile→ invokes forkpackage.jsonpreinstall/installhookspnpm run generate:schema && pnpm run generate:docs→ fork-declared scriptspnpm run eval:skillwithOPENROUTER_API_KEY/ANTHROPIC_API_KEYin envSENTRY_RELEASE_BOTapp token fromsecrets.SENTRY_RELEASE_BOT_PRIVATE_KEYThe current gating control is that a maintainer must add the
eval-skilllabel to start the job. That's a meaningful but soft mitigation — any forked repo with a label-arranged maintainer can still exfiltrate the secrets during the install/build phase before the LLM steps even run.Suggested directions (pick the one that fits our threat model best — happy to discuss before implementing)
pull_requestjob that runs on the fork checkout (no secrets, only does the build + skill validation), and a separateworkflow_runjob that re-runs validation from a trusted context ifpull_requestsucceeded. Pass eval inputs via artifacts, not env.push-main_onlyjob, upload it as an artifact, and download it in thepull_request_targetjob instead of runningpnpm installagainst the fork's lockfile. The fork eval only ever sees the artifact + SKILL.md.pnpm installbut set up an egress allowlist +--ignore-scriptsfor the install phase. Reduces risk; doesn't eliminate.Acceptable risk for now (PR #1254)
Marking Warden finding as /ar for the immediate PR — the workflow only runs when a maintainer eval-skill label is present, plus the LLM secrets are scoped to a single step. We'll come back to this in a follow-up PR with security review.