feat: `fbuild install -e <env>` as a separate CI step, and split the CI cache into packages vs per-board build payloads

[zackees]

Summary

Add a dependency-only command (fbuild install -e <env>) so toolchain / platform / framework / tool provisioning is its own observable, cacheable CI step, and split the setup action's single per-board actions/cache blob into (a) a packages cache shared across boards and (b) a per-board cache of build payloads (core/, framework-libs/, library-selection/, optionally the zccache store) that can be restored instead of recompiled.

This came out of a FastLED build-speed investigation. The original hypothesis was that a ~9-minute "pre-daemon stall" was fbuild downloading and installing the platform on the first sketch. The evidence does not support that. Nothing was downloaded, locally or in CI; the "stall" was the run's own duration. What the evidence does show is (1) every board job restores and re-saves a 1–1.9 GB blob, (2) the repository's cache is trimmed to ~10 GB by LRU eviction and the master board matrix alone overflows it, (3) the first example of every job pays 60–90 s that a cache hit does not remove, and (4) the zccache store is never saved. Details and corrections below; anything not checked against a log is marked unverified.

Evidence

1. Local run (ESP32-S3 + Teensy 4.1, fbuild 2.5.23, 2026-09-13, UTC)

Sources: ~/.fbuild/prod/daemon/daemon.log (ANSI stripped) and the transcript of the session that launched the builds.

• The two bash compile commands were launched at 09:50:16 (esp32s3, 7 examples: Blink Audio AudioFftParity ColorBoost LuminescentGrand MoodRing AutoResearch) and 09:50:22 (teensy41, 5 examples). The build-speed report's 09:41 start is wrong; nothing ran between the previous daemon's self-eviction at 09:04:29 ("daemon empty for 30.0s") and 09:50:18.
• Daemon lifecycle: fbuild daemon starting on port 60754 at 09:50:18; a second daemon spawned at 09:50:31 and yielded ("another fbuild-daemon already owns 0.0.0.0:60754; yielding (single-flight)"); zccache backend ready (embedded, cache_root=~/.fbuild/prod/zccache) at 09:50:40.9; both project locks acquired at 09:50:40.96. So daemon start → first build request was 22.8 s, almost all of it the embedded zccache backend loading its index (14 026 artifacts).
• Every package resolved from the existing cache. The only package lines are resolved toolchain-xtensa-esp-elf toolchain URL from metadata, resolved framework URL from platform.json, provisioned esptool at .../tasmota-esptool/.../5.3.0/..., ESP32 Xtensa toolchain at .../toolchains/toolchain-xtensa-esp-elf/164cda8d219cc6bd/..., ESP32 framework at .../platforms/framework-arduinoespressif32/9ef436ac06b7bf7f/3.3.11/..., Teensy ARM GCC toolchain at .../toolchain-gccarmnoneeabi-teensy/.... There are no download / extract / verify lines. find over ~/.fbuild/prod/cache/{toolchains,platforms,archives,installed} for files modified 08:00–10:30 UTC returns nothing (the toolchain and platform directories carry mtimes from 2026-08-31 … 2026-09-13T05:53); the only cache entries touched in the window are four core/ directories (framework core cache store … to ~/.fbuild/prod/cache/core/<key>), which are build output, not packages. (Correction: the earlier claim that "no file under ~/.fbuild/prod/cache was written" was too broad — core/ was written.)
• First ESP32-S3 example (build completed in 341.5s, 09:50:41 → 09:56:22), phase by phase from the daemon log:
  • 09:50:44 → 09:54:40: framework libraries — compiled 42 framework built-in libraries (231 TUs in the first batch, then per-library batches: matter, network, ble, zigbee, usb, …) into <project>/.fbuild/build/release/fw_libs/lib*.a. (Correction: 42, not ~30.) This ran concurrently with the Teensy builds, under the embedded zccache's compile concurrency capped at 15 and priority defaults to Low on an interactive host.
  • 09:54:40 → 09:54:55: core, 59 TUs at 32 jobs; framework core cache store key=081c81… copied=177 (first store of that key — no hydrate line, so the global core cache missed for this fbuild version / env).
  • 09:54:55 → 09:55:01: the sketch (1 TU).
  • 09:55:01 → 09:56:12: compiling local library 'FastLED': 30 source files — 71 s.
  • 09:56:12 → 09:56:22: link, elf2image, bootloader, partitions.
• Teensy 4.1: four build completed lines (49.9 s, 6.6 s, 47.6 s, 15.7 s), last at 09:53:02. (Correction: four visible, not five; the fifth requested example has no separate completion line in the window — unverified why.)
• Remaining ESP32-S3 examples: five further build completed lines — 17.3, 16.5, 19.4, 21.0 and 63.1 s (AutoResearch, 23 TUs) — the last at 09:59:09. Each shows framework core cache hydrate … copied=0 skipped=118 and compiled 42 framework built-in libraries in ~0.4 s, i.e. the per-project fast path works. Total 09:50:16 → 09:59:09 ≈ 9 minutes: the "stall" was the run.
• Side finding (separate issue material, cause unverified): at 09:53:02, right after the last Teensy build, the daemon logged Daemon is idle; self-eviction in 30s unless new work arrives although the ESP32-S3 build had been running since 09:50:41 (its zccache compiles continue in the log). Self-eviction triggered fired at 09:53:32; daemon exiting followed the S3 build completed at 09:56:22; the next example had to spawn a new daemon (09:56:26) and wait for zccache backend ready (09:56:35). That is ~13 s lost mid-run and an idle detector that did not count an in-flight ESP32 build.

2. CI: FastLED ESP32-S3 examples on PR #4418 (esp32s3 smoke, 8 examples), three consecutive runs

Job step timings from gh run view <id> --json jobs; phase timings from the job logs.

Phase	34878689217 (18:05)	34879677731 (18:15)	34881770380 (18:36)
fbuild cache restore: key matched	restore-key fallback → esp32dev_idf5_component (1.76 GB)	exact esp32s3 key (1.76 GB)	restore-key fallback → teensy41 (1.88 GB)
Set up fbuild step (install cache miss/hit, wheel install, restore)	26 s (download 6.5 s, extract 17 s)	52 s (download 13 s, extract 36 s)	55 s (download 12 s, extract 37 s)
./install (because .venv is absent)	30 s	29 s	31 s
Blink (first example)	88 s	63 s	68 s
Apa102HD	14 s	12 s	10 s
AutoResearch (23 TUs)	98 s	77 s	71 s
5 remaining examples	~15 s each	~12 s each	~11 s each
Build step total	5 m 04 s	4 m 02 s	3 m 54 s
Build Blink.ino for library info (a 9th compile)	15 s	13 s	11 s
Post: fbuild cache save	28 s (saved as esp32s3, 1.76 GB)	1 s (exact hit → no save)	31 s (saved as esp32s3, 1.88 GB)
Job total	6 m 28 s	5 m 26 s	5 m 46 s

Checks on the numbers: for 34881770380 the build step is 18:37:01 → 18:40:55 = 234 s = 31 (./install) + 68 + 10 + 71 + 5×~11; the job is 18:35:58 → 18:41:44 = 346 s = 55 + 234 + 11 + 31 + ~15 s of checkout / uv / artifact steps. They add up.

What the logs show:

• No package download in any of the three build steps. The only downloads are the fbuild wheel (32 MB, when the install cache misses), uv sync wheels, and Node.js for the JS linter. fbuild's build output prints Toolchain: xtensa-esp-elf-gcc 14.2.0 0.3 s into every build. (Caveat: fbuild build stdout does not print download/extract lines even when they happen — the daemon log does, and CI does not capture it. The stronger evidence is that the exact-hit run and the two fallback runs have the same first-example cost.)
• The first example costs 63–88 s regardless of whether the exact per-board cache hit. In 34881770380 the sketch compiled in 6.0 s; the remaining 61.5 s (core, framework libs, the 30-TU FastLED library, link) is not broken down in stdout. The restored blob contains fbuild's global core/ and framework-libs/ caches (they live under FBUILD_CACHE_DIR, see below), but the hydrate/miss lines that would tell us whether they were used are daemon-log only. Unverified which of the four sub-phases dominates in CI.
• The zccache store is empty in every job. The action exports ZCCACHE_DIR=$RUNNER_TEMP/zccache and outputs zccache-store-path, but no actions/cache step in the action or in FastLED's build_template.yml saves it (fbuild's own docs/CI_CACHE.md tells consumers to add that step; FastLED has not). Every TU in every job is therefore a cold compile.
• The accumulated-blob mechanism is confirmed twice. restore-keys includes the prefix fbuild-v1-Linux-X64-<fbuild-hash>-, which matches any board's entry; actions/cache picks the newest match in scope. Run 1 restored the esp32dev_idf5_component entry (1.76 GB) and saved the result as esp32s3 (1.76 GB); run 3 restored teensy41 (1.88 GB) and saved it as esp32s3 (1.88 GB). For comparison the esp32s3 entry saved by the master sweep on 2026-09-14 is 1.017 GB and teensy40 is 1.03 GB. Each fallback restore-then-save adds the current board's packages to whatever it inherited. (The earlier "PR teensy41 1.88 GB vs master teensy41 161 MB" comparison can no longer be re-checked: there is no teensy41 entry on any ref at the time of writing.)
• The exact esp32s3 key vanished within 21 minutes of its last use: saved 18:12:01, restored 18:15:39, Cache not found at 18:36:12 — all on the same PR merge ref. Consistent with size-based eviction (see below); the eviction order is not observable, so the specific cause is unverified.
• ./install is 30 s, of which ~21 s is workstation setup: uv venv + uv sync --refresh ×2 takes ~10 s (needed), then uv run test.py --setup-only --quick (meson setup + native tool build) 17 s, JS-linter setup (fails: no .cache/js-tools) + npm install -g typescript 2.6 s, VSCode extension guides + submodule update 1.5 s. FastLED's ./compile runs ./install whenever .venv is missing, which in CI is always.

3. Cache limit and eviction (FastLED/FastLED)

• gh api repos/FastLED/FastLED/actions/cache/usage returned 10.44 GB / 25 entries on 2026-09-13, then 19.2 GB / 35 and 12.5 GB / 17 within an hour on 2026-09-14 while the master sweep for #4418 was saving entries. The paginated actions/caches list taken at the same times showed 17, 20 and 18 entries totalling 11.3 GB → 11.3 GB → 10.09 GB. The usage counter and the list disagree by a few entries at any instant; the list is the one that shows what is restorable.
• Between two of those listings, the entries with the oldest last_accessed_at (apollo3_red, apollo3_thing_explorable, esp32dev_idf5_component, esp32c3, created 18:49–18:51) disappeared as esp32c2 and xiaoblesense_adafruit (19:01–19:02) were added, and the total was trimmed from 11.3 GB to 10.09 GB. That is GitHub's documented behaviour: the new cache is saved, then least-recently-accessed entries are evicted until the repository is under its limit. The effective limit behaves as 10 GB. Whether a higher per-repository limit is configured is unverified — actions/cache/usage-policy returns 404 with the token used (it needs repository admin), but the observed trimming to ~10.09 GB says no.
• Master alone overflows the limit. 71 workflows (87 jobs, 67 distinct board names) call build_template.yml, each with cache-key-extra = <board>-<hashFiles(...)>, and the per-board blobs are 0.2–1.3 GB on master. A single master sweep saves far more than 10 GB, so master entries evict each other on every sweep even with no PR traffic. (Correction: the earlier framing "PR saves and master saves evict each other" understated this; PR saves make it worse but are not the root cause.)
• PR-ref caches cannot shadow master caches. GitHub scopes cache restores: a pull_request run can read entries from its own merge ref and from the base branch; a push run on master reads only master entries. So a PR job restoring the newest prefix match can pick up a PR-ref blob (run 3 above restored a teensy41 entry from the same PR), but master jobs never restore PR blobs. PR-ref entries do count toward the 10 GB and therefore drive eviction of master entries by size. There is no cache-cleanup workflow in FastLED; the PR #4418 merge-ref entries were gone by the time of the last listing (cause unverified).

What fbuild already has (from the source at 2.5.23, crates/)

Naming the real pieces so the design reuses them rather than adding parallel code.

• No install-only command today. fbuild-cli/src/cli/args.rs enum Commands has Build, Clean, Sync, Daemon, Ide, CompileMany, Cache, … but no install / provision / prefetch, and no --deps-only. Build --dry-run only parses platformio.ini and returns (cli/build.rs); it resolves no packages.
• fbuild ide is not the model. (Correction to the draft.) Its install_declared_deps calls the daemon route POST /api/install-deps → fbuild_build::install_platform_deps → PlatformSupport::install_deps (fbuild-build-engine/src/lib.rs). The ESP32 implementation (fbuild-build-esp/src/esp32/mod.rs) fetches only the legacy Xtensa toolchain — no platform.json, no framework, no SDK libs, no esptool, and Xtensa even for RISC-V parts. The other platforms fetch a toolchain only; none fetch lib_deps. The complete ESP32 provisioning that ide benefits from actually comes from its later compiledb build.
• The full provisioning lives in the orchestrators. ESP32: resolve_pioarduino_packages(project_dir, mcu, mcu_config, env_config) in fbuild-build-esp/src/esp32/orchestrator/packages.rs — Esp32Platform::new/with_override + ensure_installed, resolve_and_create_toolchain (via esp32_metadata::resolve_toolchain_url_sync), provision_helper_toolchains (fbuild ESP toolchain resolution: surface better diagnostic + provision all platform.json packages (FastLED#2700) #401), Esp32Framework::from_url + ensure_libs/ensure_mcu_libs, and resolve_esptool (feat(deploy): manage esptool as a provisioned package instead of requiring pip install #954, Esptool::ensure_installed), fetched concurrently with tokio::join! (perf(packages): overlap download/extract/install in cold pioarduino-resolve (~185-237s serialized) #953). Called from orchestrator/build.rs before the compiledb_only early return.
• The install primitive is PackageBase::staged_install in fbuild-packages-fetch/src/lib.rs (install lock → downloader::download_file_with_progress → verify_checksum → extractor::extract → validate → rename → .install_complete sentinel → DiskCache::record_install), with PackageBase::is_cached as the presence check. It already publishes fbuild_core::install_status::InstallPhase {WaitingForLock, Downloading, Verifying, Extracting, Installed, Failed} events and records installed bytes in index.sqlite; nothing aggregates those into a per-command report yet.
• fbuild sync (feat: add fbuild sync and JSON platformio.lock #618) classifies lib_deps and writes platformio.lock; it downloads nothing and does not touch toolchains/platforms/frameworks (Phase 2 deferred). fbuild install is the natural Phase 2 executor for it.
• Cache layout (fbuild-packages-fetch/src/cache.rs, fbuild-paths/src/lib.rs): get_cache_root() is FBUILD_CACHE_DIR if set, else ~/.fbuild/{dev|prod}/cache. Under it: toolchains/, platforms/, packages/, libraries/, archives/, installed/, index.sqlite (packages) and core/<sha256>/ (FrameworkCoreCache, fbuild-build-engine/src/framework_core_cache.rs, key = fbuild version + platform + env + profile + per-TU signature), framework-libs/<sha256>/ (FrameworkLibraryCache, ESP32 only, key = fbuild version + profile + compile signature + framework libraries/ tree hash + project headers), and library-selection/ (FileKvStore). The zccache store is a sibling of the cache dir, not inside it. So today's single actions/cache entry (path FBUILD_CACHE_DIR) mixes shared packages with per-board build payloads, and excludes zccache.
• fbuild cache save|restore|list|verify (Proposal: first-class fbuild cache save / restore for toolchains + platforms + framework + zccache sidecar #527) in fbuild-cli/src/cli/cache.rs + fbuild-packages-fetch/src/cache_archive.rs: slices toolchains, platforms, packages, libraries, archives, installed, index (default) and zccache (opt-in), zstd archive with a prost FBUILD_CACHE_MANIFEST.pb (per-slice file count, byte count, content hash). Not covered by any slice: core/, framework-libs/, library-selection/. No workflow uses it yet.
• Setup action (.github/actions/setup/action.yml): two plain actions/cache@v5 steps (install dir; FBUILD_CACHE_DIR), keys fbuild-install-<ver>-<os>-<arch>-py<py>-<pip-spec>-<extra> and fbuild-<ver>-<os>-<arch>-<fbuild-hash>-<cache-key-extra> with restore-keys down to fbuild-<ver>-<os>-<arch>-. No save/restore-only/lookup-only input; it does not use the actions/cache/restore + actions/cache/save split. docs/CI_CACHING.md still says fbuild cache export/import is "not implemented".

Proposal

A. fbuild install (fbuild CLI)

fbuild install [<project_dir>] [-e <env> | --all-envs] [--check] [--dry-run] [--json] [--jobs N]

• Resolves and provisions everything the named env's build would provision — platform package, toolchain(s) incl. helper toolchains, framework and its per-MCU libs, tools (esptool), and lib_deps — then exits. No compile, no daemon build lock beyond the existing install locks.
• Implementation: promote the orchestrator provisioning into a PlatformSupport::provision(env_config, project_dir) -> ProvisionReport (or widen install_deps to take env_config and return a report), backed for ESP32 by resolve_pioarduino_packages and for the others by their orchestrators' package resolution; keep /api/install-deps (InstallDepsRequest gains environment) as the daemon route and add an in-process path like sync/cache for CI use. Add install to KNOWN_SUBCOMMANDS so fbuild <dir> install -e … reorders like the others.
• Output: one line per package — kind name version status(present|fetched|failed) bytes duration url sha256 — assembled from install_status events and DiskCache::record_install, plus totals; --json emits the same as a manifest and a packages_hash (sha256 over the sorted (kind, name, version, url, sha256) tuples). --dry-run prints the resolved set with would-fetch statuses.
• Exit codes: 0 everything present or fetched; 2 with --check when at least one package would need fetching (nothing fetched); 1 on resolution or install failure.
• Idempotent: a second run is present for every line and takes well under a second per package (it is PackageBase::is_cached).
• Relationship to fbuild sync: install executes what sync Phase 2 would lock; once platformio.lock carries toolchain/platform/framework entries, install --locked should install exactly that set.

B. Split the CI cache in the setup action

1. Packages cache — slices toolchains, platforms, packages, libraries, archives, installed, index (exactly fbuild cache save's default set). Restore by prefix, save by content:
   • restore-keys: fbuild-pkgs-<cache-version>-<os>-<arch>-<platform-family>- (family = esp32, teensy, stm32, …, from the env's platform; a per-board split is unnecessary and a repo-wide blob is too big — locally the xtensa toolchain is 1.1 GB, riscv32 2.1 GB, the ESP32 framework 3.3.11 2.2 GB, the Teensy toolchain 275 MB, uncompressed);
   • save key: the same prefix + packages_hash from fbuild install --json, so an unchanged package set produces an existing key and actions/cache skips the save. No board name, no fbuild-hash (packages do not depend on the fbuild build).
2. Build-payload cache per board — core/, framework-libs/, library-selection/ (add them as fbuild cache slices) and the zccache store (zccache slice). Key: fbuild-build-<cache-version>-<os>-<arch>-<fbuild-hash>-<board>-<cache-key-extra> with a prefix restore-key that drops only cache-key-extra. Never fall back across boards. Whether restoring these actually removes the 60–90 s first-example cost must be measured (see acceptance); the hydrate lines need to be visible in CI stdout to know.
3. Drop the fbuild-<ver>-<os>-<arch>- and fbuild-<ver>-<os>-<arch>-<fbuild-hash>- restore-keys from the current single blob (they are what turns per-board entries into all-toolchain blobs).
4. Add save: true|false (and/or save-on-default-branch-only) implemented with actions/cache/restore + actions/cache/save, so PR runs restore master's entries and do not write their own. This removes PR-ref entries from the 10 GB budget; it does not by itself fix master overflowing the budget (see C).
5. Use fbuild cache save/restore for the archive step instead of actions/cache's tar: the Proposal: first-class fbuild cache save / restore for toolchains + platforms + framework + zccache sidecar #527 archive is zstd and parallel, and Windows extract is where the current 36 s becomes minutes.

C. FastLED-side follow-ups (for context; land in FastLED/FastLED)

• Board CI jobs should run uv sync only, not ./install (saves ~21 s/job: meson setup + native tools, JS/TS tooling, VSCode guides).
• Cache ZCCACHE_DIR (the action already outputs zccache-store-path) until B.2 lands.
• The master board matrix (87 template jobs over 67 boards) cannot fit that many 0.2–1.3 GB blobs in 10 GB, so even after A/B the per-board payload entries need to be small (payload only, no packages) and the packages cache must be shared per platform family; otherwise the sweep keeps evicting itself. Consider pruning boards that do not need their own entry (the --examples Blink,Apa102 jobs).
• Correct the build-speed report's "9-minute pre-daemon stall" section (the 09:41 start and the download hypothesis).
• File the mid-build daemon self-eviction (section 1, side finding) as its own fbuild bug.

Phased plan

1. Observability first (fbuild): print core cache hydrate/store, framework-libs hydrate/store and the zccache backend cache_root in fbuild build stdout (or under an env flag CI can set), so the CI logs answer "did the restored payload hydrate?".
2. fbuild install (A) with --json/--check; ESP32 first (reuse resolve_pioarduino_packages), then the other PlatformSupport impls; tests: cold cache → every line fetched; second run → every line present, exit 0; --check on cold → exit 2 and no network.
3. fbuild cache slices for core, framework-libs, library-selection; update docs/CI_CACHING.md (it predates Proposal: first-class fbuild cache save / restore for toolchains + platforms + framework + zccache sidecar #527).
4. Setup action v2 (B.1–B.5) behind a new input so existing consumers are unaffected; FastLED opts in on one board job first (esp32s3), measures, then the matrix.
5. FastLED (C).

Acceptance

• fbuild install -e esp32s3 --json on a cold cache reports every package fetched with bytes and durations; a following fbuild build logs zero package fetches; a second install reports all present; install --check exits 2 on cold and 0 on warm.
• In FastLED's esp32s3 PR job the install phase is its own step; with a warm packages cache it completes in < 5 s (unverified target; to be measured).
• After B.2, the first-example time on a payload-cache hit is measured and reported against the current 63–88 s; if hydration does not remove most of it, the daemon-log phase breakdown says why.
• gh api …/actions/cache/usage for FastLED stays under 10 GB after a full master sweep, and PR runs create no cache entries.

Related

• Proposal: first-class fbuild cache save / restore for toolchains + platforms + framework + zccache sidecar #527 fbuild cache save/restore (slices, manifest) — the archive format for B.
• feat: add fbuild sync and JSON platformio.lock #618 fbuild sync + platformio.lock — install is its Phase 2 executor.
• feat(build): retain verified framework caches for clean builds #1089 retained framework caches under get_cache_root() (core/, framework-libs/) — the payloads B.2 restores.
• perf(build): framework libs (~150s) recompiled per project; core cache never hydrates #952, perf(build): fw-lib (framework library) object compiles still miss zccache cross-project #986, perf(build): decide fate of core-cache-hydrate after fw-libs zccache fix #995 fw-libs / core cross-project reuse via zccache — why the zccache store matters in CI.
• [META] fbuild build cache must survive CI tar-extract / cross-runner restore #147, daemon/setup: expose zccache store location so consumer workflows can actions/cache it #149 zccache store must be cached across CI runs; the action's zccache-store-path output came from daemon/setup: expose zccache store location so consumer workflows can actions/cache it #149 but FastLED never wired it.
• Cross-run framework artifact cache (setup-fbuild + content-addressed core/) #362 original cross-run framework artifact cache proposal for the setup action (closed 2026-06-02).
• feat(ci): ship reusable GitHub Action for fbuild install + cache setup (external consumers) #101 the setup action itself.
• compile-many stage 2 re-compiles the framework per sketch — defeats stage-1 warmup, 2× slowdown vs single build on warm machine, ~3× on slow CI #335 (compile-many stage-2 framework rebuild) is not the right cross-reference: FastLED's board jobs run fbuild build per example through ci-compile.py, not compile-many.