Skip to content

Add CI to validate contributor runner archives#13

Merged
Kyome22 merged 4 commits into
mainfrom
add-runner-validation-ci
Jul 15, 2026
Merged

Add CI to validate contributor runner archives#13
Kyome22 merged 4 commits into
mainfrom
add-runner-validation-ci

Conversation

@Kyome22

@Kyome22 Kyome22 commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Type of Change

  • Add a new runner
  • Fix or improve an existing runner
  • Other (site, docs, tooling)

Summary

Runner zips are committed by contributors and handed to RunCat Neo users as-is from the gallery download link. Today the only thing between the two is a maintainer reading a checklist. This adds a check that runs automatically on every pull request and reports a pass/fail verdict.

Adds .github/workflows/validate-runners.yml and .github/scripts/validate_runners.py. Only runners touched by the PR are validated; manifest/directory correspondence is always checked in full, since a manifest edit can drop an entry for a runner the PR never touched.

What it actually guarantees

The core property is that a frame archive may contain nothing but PNG files, and that every PNG consists entirely of CRC-valid chunks ending at IEND with no trailing bytes. Together those leave nowhere in the archive for a payload to hide — a stronger guarantee than scanning for known signatures, which is why there is no ClamAV step here. On top of that:

  • Rejects symlinks, executables, setuid bits, encrypted entries, zip-slip paths (.., absolute paths, drive letters), zip bombs, archive comments, and data appended before or after the archive.
  • Fully decodes every frame through zlib/libpng, then checks height is exactly 36px, width is 10–100px, and all frames share one size, numbered 0..N-1.
  • Checks metadata.json shape, that preview.png is a real APNG, and that manifest.json and the runner directories agree in both directions — a name listed with a broken directory is silently dropped from the gallery by script.js, which is the failure this is most worth catching.

Interlaced PNGs are rejected. Adam7 is pointless at 36px tall, and Pillow cannot decode the frames of an interlaced APNG at all (load_prepare appends to decoderconfig on every seek, so frame 1 raises TypeError), so accepting one would mean giving up the full-decode check for exactly the files that are unusual.

Security posture

Runs on pull_request, not pull_request_target. This job feeds untrusted binaries to zlib, libpng, and Pillow's C decoders, so it is the last thing in this repository that should hold a writable token or secrets — the read-only fork sandbox is the actual control. That costs the ability to post a PR comment, so results land as inline annotations on the offending file plus a job summary.

There is deliberately no paths: filter: once this is a required check, a filtered-out PR would never report it and could never merge. The job takes well under a second.

Verification

Passes on all 18 existing runners in 0.13s. Verified against deliberately hostile inputs — payload appended to the zip, an MZ executable appended after a frame's IEND, a symlink entry, zip-slip, an ELF disguised as .png, a 9KB payload hidden in a tEXt chunk, Finder junk, frame gaps — each is caught with an actionable message.

Calibrated against the currently open runner PRs, which is where the real value shows:

PR Result
#4 takibi-guruguru passes
#8 huang-dachui passes
#9 penguin passes
#5 chihuahua preview.gif instead of preview.png, 8 AppleDouble files in the zip, escapement/linkage listed twice in the manifest
#10 spamton frames at the archive root with no spamton-frames/ directory, interlaced preview, preview is 64x76

Notes

  • record-player-frames.zip is rebuilt in its own commit. It was the one runner that named frames record-player-<N>.png instead of record-player-frame-<N>.png, so it was the only existing violation. The frame images are byte-for-byte unchanged; only the zip entry names differ.
  • Once merged, the check to require on main is named Validate runner assets.
  • The open PRs need a close/reopen (or "Update branch") to pick the workflow up.

🤖 Generated with Claude Code

Kyome22 and others added 4 commits July 15, 2026 23:12
Runner zips are committed by contributors and handed to RunCat Neo users
as-is from the gallery download link, with nothing between the two but a
maintainer reading a checklist. This checks every pull request instead.

The core property is that a frame archive may contain nothing but PNG
files, and that every PNG consists entirely of CRC-valid chunks ending at
IEND with no trailing bytes. Together those leave no room in the archive
for a payload to hide, which is a stronger guarantee than scanning for
known signatures would give. On top of that the validator rejects
symlinks, executables, encrypted entries, zip-slip paths, zip bombs, and
data appended to the archive, then fully decodes every frame and checks
it against the sizes CONTRIBUTING.md documents.

Interlaced PNGs are rejected: Adam7 is pointless at 36px tall, and Pillow
cannot decode the frames of an interlaced APNG at all, so accepting one
would mean giving up the full-decode check for exactly the files that are
unusual.

Runs on pull_request rather than pull_request_target. The job feeds
untrusted binaries to zlib, libpng, and Pillow's C decoders, so it is the
last thing here that should hold a writable token or secrets; the
read-only sandbox is the actual control. That costs the ability to post a
comment, so results land as inline annotations on the offending file plus
a job summary.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every other runner names its frames <runner-name>-frame-<N>.png, but this
archive used record-player-<N>.png, so it was the one runner the new
validation would reject. The frame images themselves are byte-for-byte
unchanged; only the zip entry names differ.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The frame file name was only ever shown as an example, so state it as a
requirement now that CI enforces it, and note that PNGs must not be
interlaced.

Also point contributors at the validator itself. Someone who runs it
before pushing never has to see a red check at all, which matters more
than any wording in the failure messages.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The first run produced a completely empty step log: on success the
validator emits no annotations, and the summary goes to a file. That
leaves no way to tell from the log whether it validated a runner or
silently validated nothing, which matters because the scope comes from a
diff and an empty scope still exits 0.

Print the result to stdout in both modes, and say so explicitly when
nothing was in scope.

Also move off Node 20, which the runner now warns about.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Kyome22
Kyome22 merged commit c083934 into main Jul 15, 2026
1 check passed
@Kyome22
Kyome22 deleted the add-runner-validation-ci branch July 15, 2026 14:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant