Skip to content

ci: isolate pull_request_target secrets in eval-skill-fork.yml (follow-up from #1254 Warden finding) #1321

Description

@BYK

Warden flagged this in PR #1254 review (#1254 (comment), severity: high). Pre-existing risk carried since d4204f3 (PR #1306); out of scope for #1254's monorepo pre-shape, but worth a follow-up.

Background

eval-skill-fork.yml is the workflow that runs the SKILL.md doc eval on fork PRs. Today it:

  • triggers on pull_request_target: [labeled, synchronize] → has access to base-repo secrets
  • checks out the fork's head.sha (line 46) → runs attacker-controlled code
  • runs pnpm install --frozen-lockfile → invokes fork package.json preinstall / install hooks
  • runs pnpm run generate:schema && pnpm run generate:docs → fork-declared scripts
  • runs pnpm run eval:skill with OPENROUTER_API_KEY / ANTHROPIC_API_KEY in env
  • later mints a SENTRY_RELEASE_BOT app token from secrets.SENTRY_RELEASE_BOT_PRIVATE_KEY

The current gating control is that a maintainer must add the eval-skill label to start the job. That's a meaningful but soft mitigation — any forked repo with a label-arranged maintainer can still exfiltrate the secrets during the install/build phase before the LLM steps even run.

Suggested directions (pick the one that fits our threat model best — happy to discuss before implementing)

  1. Two-job pattern: split into a pull_request job that runs on the fork checkout (no secrets, only does the build + skill validation), and a separate workflow_run job that re-runs validation from a trusted context if pull_request succeeded. Pass eval inputs via artifacts, not env.
  2. Pre-build + reuse: build the eval harness in a separate push-main_only job, upload it as an artifact, and download it in the pull_request_target job instead of running pnpm install against the fork's lockfile. The fork eval only ever sees the artifact + SKILL.md.
  3. Network allowlist + drop-on-postinstall: still run pnpm install but set up an egress allowlist + --ignore-scripts for the install phase. Reduces risk; doesn't eliminate.
  4. Lock fork checkout to base: check out the base branch and only git archive the changed files from the fork head SHA via the API, into a tarball. Eval harness runs against the tarball in isolation, no pnpm install against a fork-modified pnpm-lock.yaml.

Acceptable risk for now (PR #1254)

Marking Warden finding as /ar for the immediate PR — the workflow only runs when a maintainer eval-skill label is present, plus the LLM secrets are scoped to a single step. We'll come back to this in a follow-up PR with security review.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions