Skip to content

feat(docker): versioned non-root WASM builder image (#529) - #530

Closed
apotema wants to merge 8 commits into
mainfrom
feat/wasm-builder-image
Closed

apotema wants to merge 8 commits into
mainfrom
feat/wasm-builder-image

Conversation

@apotema

@apotema apotema commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Implements #529 (epic #528, work package 1). Draft: the image has not been built or run yet. See "Validation" below.

What this adds

  • docker/wasm-builder/Dockerfile: a non-root (UID 10001) CI image in which the normal labelle pipeline builds, bundles and smoke-tests --platform=wasm. No credentials and no game source go into it. Toolchains sit read-only under /opt.
  • docker/wasm-builder/versions.env: every pin and checksum in one file. The image also carries a measured inventory at /opt/labelle/toolchain-inventory.json.
  • bin/labelle-builder-check: job preflight. It checks that the caches are writable, compares the project's pins with the image (labelle.lock CLI, project.labelle Zig, web provider emsdk) and fails on a mismatch, and seeds the assembler. Also supports --self-test and --inventory. bin/labelle-builder-smoke-python uses the image's Playwright venv when the project's requirements match it, and otherwise builds a venv keyed on the requirements in the cache. bin/labelle-builder-inventory writes the inventory.
  • .github/workflows/wasm-builder-image.yml: runs only on a wasm-builder-v* tag or a manual dispatch. Steps: hadolint/shellcheck → build → non-root self-test → fresh named volume plus a second fresh container on it → push the tested image under an immutable tag (refuses to overwrite an existing tag) → digest in the summary → GHCR provenance. The registry is a parameter (dispatch input, then the WASM_BUILDER_REGISTRY/WASM_BUILDER_IMAGE repo variables, then ghcr.io/labelle-toolkit/labelle-wasm-builder).
  • docker/wasm-builder/gitlab-ci.example.yml: FP's test-game-wasm job on a Docker executor, with the threaded (COOP/COEP) smoke, the fallback smoke, and artifacts.
  • docs/wasm-builder-image.md covers:
    • image/toolkit compatibility and how pins are handled
    • cache layout, mounts, ownership/UID, keys and invalidation, concurrency, pruning
    • the runner config.toml (volume, shm_size, memory)
    • resource guidance: 14 GiB per job now (10.41 GiB peak), 8 GiB after labelle-engine#921 (5.34 GiB)
    • publishing, digest pinning, upgrade and rollback
  • Small pointers from docs/migrating-to-3.0.md and docs/provider-targets.md.

Pins (image 4.1.2-1)

Version Source
base ubuntu:24.04@sha256:fe815609…7ef852 Docker Hub index digest, resolved 2026-10-30
labelle CLI 4.1.2 (sha256 43d84d3b…) FP labelle.lock / ensure-tool.sh; I recomputed the checksum from the release file
Zig 0.16.0 (sha256 70e49664…) CLI default, FP ensure-tool.sh; checksum recomputed
assembler (seed) 0.124.0 (sha256 4d169a6c…) FP project.labelle; release SHA256SUMS, recomputed
emsdk 4.0.9 @ 3bcf1dcd labelle-web 0.4.0 default_version/default_commit
Node 20.18.0 bundled by emsdk 4.0.9 (manifest)
Python 3.12 (noble) emsdk/emcc and FP smoke
Playwright / Pillow 1.64.1 / 11.3.0 FP tools/web_smoke/requirements.txt

Caches

Everything lives under /cache/<key>, where <key> is v1-x86_64-linux-zig0.16.0-emsdk4.0.9:

  • LABELLE_HOME. The CLI derives ZIG_GLOBAL_CACHE_DIR and ZIG_LOCAL_CACHE_DIR from it, and it also holds packages, providers and the assembler.
  • EM_CACHE
  • XDG_CACHE_HOME

The web provider's default managed source passes the image's activated EMSDK through, so no ~1 GB emsdk lands in the cache. The provider does not compare versions when it does this, which is why labelle-builder-check checks the emsdk version itself.

Relationship to --docker

--docker is unchanged and still the only container path inside the CLI: desktop only, ubuntu:24.04, root, Zig downloaded at run time. Nothing in the CLI starts this image. It is a CI image: in which the whole native pipeline runs, provider hooks included. Proposed follow-up, kept out of this PR because it changes --docker behaviour: have --docker reuse this image and the host UID and mount LABELLE_HOME, and later run the CLI itself in the container, which is what lifting the provider-target refusal would need.

Validation

  • Ran: hadolint 2.12.0 (ignored rules are documented in .hadolint.yaml), shellcheck 0.10.0 and actionlint 1.7.12. All clean.
  • Ran outside a container: labelle-builder-check against an FP checkout. It passed, seeded the assembler once and reused it on the second run. Against a copy with mismatched CLI, emsdk and assembler pins it failed as expected, and the mismatches became warnings with the override set. labelle-builder-smoke-python matched FP's CRLF checkout of the requirements.
  • Not built or run: Docker isn't available on the machine I worked on. So the image build, the self-test, the acceptance runs (FP WASM bundle as non-root in a fresh container, a second container reusing the caches) and Linux memory peaks are all still open. The first dispatch of the workflow with push unticked does the image-level checks. The FP runs belong to the FP adoption ticket.
  • The resource figures come from the epic's local Windows evidence (findings.json): windows-threaded-cold peak private 11 173 097 472 B = 10.41 GiB, and windows-game-outline 5.34 GiB. They are not Linux numbers.

Open decisions for the owner

  1. Registry: GHCR (public package, internet pull) or FP's GitLab registry (LAN). The workflow supports both. A GHCR package starts private after its first push.
  2. Runner: a GitLab runner with the Docker executor on the FP box (fp-docker tag in the example). Suggested settings: limit = 1 for WASM jobs until engine#921 ships, memory = 14g, shm_size = 2g. Also decide between one shared cache volume and per-trust volumes (a separate main runner).
  3. Whether to pin apt packages through snapshot.ubuntu.com. Right now apt versions are recorded in the inventory, not pinned.
  4. Whether an arm64 variant is wanted.

https://claude.ai/code/session_01Hpt1haxcboZPznpG1KqDRS

Adds docker/wasm-builder: a pinned, non-root CI image in which the
ordinary labelle pipeline builds, bundles and smoke-tests a game for
--platform=wasm with no host toolchain (epic #528 WP1).

- Dockerfile on ubuntu:24.04 pinned by digest; Zig 0.16.0, labelle CLI
  4.1.2 and an assembler 0.124.0 seed, each sha256-verified; emsdk 4.0.9
  (commit-verified, activated, passed through by labelle-web's managed
  source via EMSDK) with its Node 20.18.0; Playwright 1.64.1 + Pillow
  11.3.0 venv with Chrome for Testing and its system libraries;
  libatomic1. versions.env is the single source of every pin; the image
  bakes a measured toolchain inventory.
- All caches under /cache/<key> (LABELLE_HOME, hence the Zig global and
  local caches, packages and providers; EM_CACHE; XDG_CACHE_HOME), keyed
  by platform + Zig + emsdk. UID 10001; a fresh named volume inherits it.
- labelle-builder-check: cache writability, labelle.lock / project.labelle
  / web provider emsdk pins vs the image (fails on mismatch), assembler
  seeding, --self-test, --inventory. labelle-builder-smoke-python picks
  the baked venv or a keyed cache venv.
- .github/workflows/wasm-builder-image.yml: tag (wasm-builder-v*) or
  manual dispatch only; lint, build, non-root self-test, two fresh
  containers on one volume, push of the tested image under an immutable
  tag, digest in the summary, GHCR provenance. Registry parameterised.
- docs/wasm-builder-image.md: compatibility and pin handling, cache
  layout/ownership/keys/invalidation, GitLab Docker executor example
  (FP test-game-wasm with threaded + fallback smoke and artifacts),
  resource guidance (10.4 GiB peak today, 5.3 GiB after engine#921),
  publishing, upgrade and rollback, and the relationship to --docker,
  which is unchanged.

Claude-Session: https://claude.ai/code/session_01Hpt1haxcboZPznpG1KqDRS
@coderabbitai

coderabbitai Bot commented Oct 30, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

…lder (#529)

Owner decisions on #530:
- GHCR, public package: default kept; docs say to set the package public
  after the first push, after which consumers pull without a token.
- Ubuntu packages from snapshot.ubuntu.com at a pinned UBUNTU_SNAPSHOT
  (http, archive signatures checked), top-level packages pinned to their
  snapshot versions; docs explain how to bump it.
- linux/arm64 as a first-class variant: per-arch pins and recomputed
  checksums for the CLI, Zig, assembler, Emscripten release archive, Node
  and Playwright's Chrome for Testing headless shell + ffmpeg. emsdk's two
  downloads and Playwright's browsers are now fetched and verified by the
  Dockerfile (emsdk must download nothing else; Playwright installs from a
  local mirror of the verified zips); pip installs from a hash lock. The
  cache key carries TARGETARCH (v1-<amd64|arm64>-linux-...).

Workflow: also runs on pull_request (never pushes); builds natively on
ubuntu-24.04 and ubuntu-24.04-arm; self-test, two fresh containers on one
volume, and an end-to-end bundle of e2e/game (labelle init + labelle-web
0.4.0, threads on, FP's toolkit pins) with threaded and fallback Chromium
smokes, cold then warm, recording phase timings and peak memory. Publishing
pushes the tested per-arch images and a multi-arch index.

Claude-Session: https://claude.ai/code/session_01Hpt1haxcboZPznpG1KqDRS
…xists

snapshot.ubuntu.com redirects http to https and the base image has no CA
bundle, so the first apt run skips TLS peer verification; apt still checks
the signed InRelease and every index and package hash.

Claude-Session: https://claude.ai/code/session_01Hpt1haxcboZPznpG1KqDRS
Some bundled test scripts do not compile under Python 3.12; compileall's
non-zero exit must not fail the image.

Claude-Session: https://claude.ai/code/session_01Hpt1haxcboZPznpG1KqDRS
A fresh project has no labelle.lock until its first build; read the web
provider pin from project.labelle then instead of failing on the missing
file.

Claude-Session: https://claude.ai/code/session_01Hpt1haxcboZPznpG1KqDRS
@apotema

apotema commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

CI on 8243052 is green: https://github.com/labelle-toolkit/labelle-cli/actions/runs/41086043350. Both architectures build natively (amd64 on ubuntu-24.04, arm64 on ubuntu-24.04-arm). On each one these all pass as UID 10001: the self-test, two fresh containers sharing one volume, and the e2e fixture (labelle init + labelle-web 0.4.0 with threads on, using FP's toolkit pins). The e2e runs a threaded smoke (cross-origin isolated, loads threaded/game.js) and a fallback smoke (loads game.js), first with a cold cache and then warm in a second fresh container.

phase (s) amd64 arm64
image build 169 151
bundle, cold 177 306
bundle, warm (2nd container) 23 38
each smoke 11 11
container peak memory, cold / warm 3.0 / 2.2 GiB 3.1 / 2.2 GiB

The pushes since the last update fixed these:

  • snapshot.ubuntu.com only answers over https and the base image has no CA bundle, so the first apt run now skips TLS verification. apt still checks the signed InRelease and every hash.
  • compileall is now best effort.
  • The emcc version check now parses the right line.
  • The preflight no longer fails on a project that has no labelle.lock yet.

https://claude.ai/code/session_01Hpt1haxcboZPznpG1KqDRS

…'s cache

Local Docker Desktop runs of FP's test-game-wasm in the image (cold, warm,
changed source): timings and cgroup memory peaks. A /cache bind mount of a
Windows drive fails Zig's cache rename with AccessDenied; document a
Linux filesystem (named volume, or an ext4 image loop-mounted) instead.
Note the second emsdk FP's generated build installs into its checkout.

Claude-Session: https://claude.ai/code/session_01Hpt1haxcboZPznpG1KqDRS
@apotema

apotema commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

FP test-game-wasm in the WASM builder image: local Docker results

GitHub Actions billing for FP is blocked and GitLab is down, so I ran the same steps as the FP#1095 workflow in local Docker:

  • labelle providers fetch
  • labelle bundle --platform=wasm --optimize=ReleaseSafe
  • tools/ci/prepare-web-dist.sh
  • the smoke unit tests
  • the threaded smoke (--isolated --expect-threads yes) and the fallback smoke (--expect-threads no)

Setup

  • Image labelle-wasm-builder:local (amd64), built from labelle-cli feat/wasm-builder-image @ 8243052. Self-test passes. 3.4 GB on disk, 852 MB compressed.
  • Docker Desktop 29.8.1 on Windows: WSL2 VM with 16 GiB / 24 CPUs.
  • Each run used a fresh container with --memory=14g --memory-swap=14g, with all pipeline steps as UID 10001.
  • FP at efc8b1a (main + the trial workflow), engine 4.2.0-rc.2, so before engine#921.
  • Each run started from a clean checkout (git clean -xfd), so only /cache carried over, as in CI.
run bundle whole job peak memory: cgroup / docker stats exit / OOMKilled smokes
cold cache 1144 s 1215 s 11.5 GiB / 10.6 GiB 0 / false threaded pass, fallback pass
warm cache, second fresh container 297 s 368 s 1.8 GiB / 1.2 GiB 0 / false pass, pass
warm cache + one changed script (scripts/playing/00_game_time.zig, a comment line appended) 979 s 1052 s 10.8 GiB / 10.3 GiB 0 / false pass, pass

The other phases are small in every run: preflight 0 s, providers fetch 0–1 s, prepare-web-dist 3 s, smoke unit tests 26–27 s, each smoke 17–20 s. /cache was 2.1 GiB after the three runs (one cache key).

Findings

  1. 14 GiB is enough; GitHub's 8 GB private-repo runners are not. The cold compile peaked at 11.5 GiB in a Linux container, matching the 10.4 GiB measured on Windows plus the CLI and link. Until engine#921, FP#1095 on a standard ubuntu-24.04 runner is expected to be OOM-killed.
  2. One changed game script costs almost a full compile: 979 s and 10.8 GiB. The game is one Zig compilation unit, so the warm cache only saves the libraries and toolchain work. Size runners for the cold peak, not the warm one.
  3. The cache must be on a Linux filesystem. With /cache bind-mounted from a Windows drive (-v D:/…:/cache), Zig fails in the web provider's toolchain hook: failed to rename compilation results … into local cache …: AccessDenied. A named volume works. To keep the bytes on D: here, I loop-mounted an ext4 image file stored on D: at /cache. The container ran --privileged as root only for the mount, then setpriv dropped to UID 10001 for every step. The FP checkout itself is a plain Windows bind mount and works. This is now in labelle-cli docs/wasm-builder-image.md (21b4167).
  4. FP's build installs a second emsdk inside the checkout. The generated .labelle/bgfx_wasm/build.zig.zon declares git+https://github.com/emscripten-core/emsdk#4.0.9 as a Zig package (labelle_audio and sokol read headers from it). The build installs it under .labelle/bgfx_wasm/zig-pkg/, about 1.4 GB, including its own Emscripten release and Node downloads. That is not in /cache and doesn't use the image's /opt/emsdk, so every fresh checkout downloads and installs it again, inside the 297 s warm bundle. This is a toolkit issue, not an image one: the generated build should use the provider's EMSDK. I haven't filed it yet.

No image bugs turned up in these runs, so nothing changed in the image itself. The labelle-cli PR CI is still green on 8243052; 21b4167 changes only docs.

https://claude.ai/code/session_01Hpt1haxcboZPznpG1KqDRS

@apotema

apotema commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

Closing: the owner decided the WASM builder image is Flying Platform-specific, so it moved to FP as a private image: moca-tecnologia/flying-platform-labelle#1101 (tools/ci/wasm-builder/, published to FP's GitLab registry). The Dockerfile, pins, helpers, e2e fixture and docs all moved, along with the measurements from this PR's CI and the local FP runs. The branch feat/wasm-builder-image is kept for reference.

https://claude.ai/code/session_01Hpt1haxcboZPznpG1KqDRS

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant