[Pedro Henrique Penna (ppenna)]

Closes #703.
Part of #430.

Summary

The shared artifacts job restored or built the kernel, Alpine, Ubuntu, Ubuntu distro layer, and Azure Linux images one after another, and every job that boots a guest downloaded all of them and waited for all of them. This PR:

1. lets build-guest-artifacts build any subset of the guest components (a components input replaces guest-images and debug-kernel);
2. hands each component over as its own one-day workflow artifact (guest-kernel, guest-alpine, guest-ubuntu, guest-ubuntu-distro, guest-ubuntu-scratch, guest-azurelinux, beside guest-debug-kernel), and makes each job download only the components that it reads;
3. splits the producer in two: artifacts now builds the kernel and the Alpine initramfs, and a new distro-artifacts job builds the Ubuntu and Azure Linux images beside it, so the OpenVMM VMM tests and the debug-kernel jobs no longer wait for the Ubuntu and Azure Linux builds.

The debug-kernel lane, the Ubuntu determinism check, the provenance, and the cache keys, which #710 (#699) narrowed to the guest binaries' inputs, are unchanged.

Why

The measurements are in this comment on #703. In short, over the 200 artifacts jobs from #561 (2026-10-19 10:52 UTC, when the guest keys started to cover the Rust guest sources) to 2026-10-21 13:55 UTC, before #710 merged:

• 159 of 200 jobs missed the cache and took a median 5.3 min instead of 0.3, and 10.3 with a cold kernel (13 runs). With a warm kernel, the Alpine initramfs took 0.9 min of that, while the Ubuntu initramfs (0.6), its determinism check (2.9), its distro layer (0.3), and the Azure Linux initramfs (0.4) took 4.1.
• The OpenVMM VMM tests boot only the kernel and the Alpine initramfs, but their WHP job started a median 6.0 min after the run started on cold runs, against 0.8 on warm ones, and it finished last in 74 of the 119 cold runs that ran it. Examples: cold push 39603556404 (artifacts done at 5.9 min, WHP VMM tests 5.9–26.8 min), warm push 39596666239 (0.7–21.9 min).
• ci: key the guest images and OpenVMM VMM tests on the guest binaries' sources, not all of nvx-rs #699 and ci: reuse a pull request's guest images on its dev push #702 remove only about a third of these misses. Replaying the 172 jobs' keys from the trees that they tested, with GitHub's cache scopes (calibrated: 166 of 172 Alpine predictions match), ci: key the guest images and OpenVMM VMM tests on the guest binaries' sources, not all of nvx-rs #699's narrower keys and ci: reuse a pull request's guest images on its dev push #702's reuse together leave 91 of 172 runs cold instead of 137. Most of the rest are pull requests that change the guest binaries, which still run the VMM tests now that ci: key the guest images and VMM tests on the guest binaries' sources #710 has merged.
• The split still pays off after ci: key the guest images and OpenVMM VMM tests on the guest binaries' sources, not all of nvx-rs #699 and ci: reuse a pull request's guest images on its dev push #702. Replaying the jobs of the 27 runs in which no self-hosted job queued 2 minutes or more, with their measured queue times and durations, gives p90 27.3 → 23.1 min and p99 28.2 → 24.0 min. Over all runs, the p90 moves 48.2 → 47.4 and the p99 doesn't move, because it is set by self-hosted queueing. Four producers, one per component, gain nothing more than these two, because the Ubuntu chain bounds the jobs that read every image either way.
• Every job downloaded the whole 140 MB guest-artifacts artifact; the VMM tests and the debug-kernel jobs need only the kernel and the Alpine initramfs.

Cold-path evidence

This PR's CI, every image cold. #652 changed the guest binaries' sources, so in run 39690060426 of the head 46da18e49, every guest image missed the cache, while the kernel and the debug kernel hit. That is the common cold case:

Job, images cold	Kernel	Alpine	Ubuntu initramfs	Determinism	Distro layer	Azure Linux	Job wall time
artifacts	hit	0.9 min					1.0 min
distro-artifacts			1.4¹	3.0	0.3	0.4	5.3 min

¹ Including the base and guest-binary Docker stages, which artifacts builds too.

Both producers started 0.4 min after the run was created.

• The three OpenVMM VMM test jobs started at 1.6 min, as soon as artifacts finished, and the KVM debug-kernel job finished at 5.2 min, before distro-artifacts did (5.7 min).
• The microVM test and platform jobs started when distro-artifacts finished.
• With the single job, which built the same components one after another, every one of these jobs would have waited for all of them. That single job took a median 5.3 min when it missed.

Experiment, every component cold. My previous head's run hit the cache, because #710's dev push had just built and saved its new keys. To also measure a cold kernel, I ran ci.yml's two producers once on a temporary branch on GitHub-hosted runners: unchanged, but without their change-detection dependency, with a comment line added to .dockerignore (which every guest key hashes) so that every component missed, and with save-cache: false. The experiment is run 39665813966. The baseline is the single job of #710's dev push 39661815501, which also built every component cold under the same keys an hour earlier:

Job, all components cold	Kernel	Alpine	Ubuntu initramfs	Determinism	Distro layer	Azure Linux	Job wall time
Before: one artifacts job (39661815501)	5.6 min	0.7	0.6	2.9	0.3	0.4	10.9 min
After: artifacts (39665813966)	6.1	0.8					7.0 min
After: distro-artifacts (39665813966)			1.3²	3.0	0.3	0.4	5.2 min

² Including the base and guest-binary Docker stages, which the other job builds again (0.6 min more).

Both jobs started 5 s after the run was created and uploaded every artifact (if-no-files-found: error) in 3–8 s. With a cold kernel, every guest consumer can start after about 7.0 min instead of 10.9. With a warm kernel, artifacts takes about 1.0 min, as in this head's run above. So the VMM tests and the debug-kernel jobs start about 4 min earlier, and the microVM test and platform jobs wait for distro-artifacts about as long as for the single job. The experiment saved no cache, and its branch is deleted.

Changes

• build-guest-artifacts: a components input (kernel, alpine, ubuntu, azurelinux, debug-kernel; default all but debug-kernel) replaces guest-images and debug-kernel. A first step validates it and fails on an unknown or empty selection, and every restore, build, determinism, and save step requires its component. The keys, which end with the guest binaries' source identity, are untouched. The Copilot setup keeps restoring every guest image through the default.

• ci.yml:

  • artifacts (now Build kernel and Alpine artifacts) builds kernel alpine; the new distro-artifacts (Build Ubuntu and Azure Linux artifacts) builds ubuntu azurelinux beside it, under the same condition (ci: classify microVM tests, benchmarks, and artifact builds separately #712's run-guest-artifacts); debug-kernel builds debug-kernel.
  • Each uploads one artifact per component, under its build filenames, with if-no-files-found: error.
  • The microVM test and platform jobs wait for artifacts and distro-artifacts; the VMM tests and the debug-kernel jobs still wait only for artifacts (and debug-kernel).
  • Required status check checks distro-artifacts; the release, the performance gate, and performance persistence already require the platform jobs, which need it.

• Consumers download by name, into build/, only what they read:

  Job	Downloads
  OpenVMM VMM tests	guest-kernel, guest-alpine
  microVM tests	every component but guest-debug-kernel
  microVM debug-kernel jobs	guest-kernel (runner qualification), guest-alpine, guest-debug-kernel
  platform jobs	guest-kernel, guest-alpine, and, when they package a release, guest-ubuntu, guest-ubuntu-distro, guest-azurelinux

• check-required-ci (ci.py): expects distro-artifacts to succeed whenever run-guest-artifacts is set, and to be skipped otherwise, like artifacts.

• Docs: doc/ci.md gains a Guest artifact handoff section with the producers, artifacts, files, and consumers, and updates the job graph and the debug-kernel lane; doc/build.md names the new input.

Interaction with the sibling pull requests

The artifact names, one per cache entry plus the scratch template, are the handoff contract: a job that boots the kernel and the Alpine initramfs downloads guest-kernel and guest-alpine and waits for artifacts, and a job that reads an Ubuntu or Azure Linux image also downloads it and waits for distro-artifacts. The structural test below checks every consumer, so a sibling that adds one learns at once what it must download and wait for.

• ci: key the guest images and VMM tests on the guest binaries' sources #710 (ci: key the guest images and OpenVMM VMM tests on the guest binaries' sources, not all of nvx-rs #699), merged: this branch is rebased onto it. The image keys keep its guest-binary identity at their end and hash no nvx-rs/**; the split only selects which job restores and builds each key.
• ci: run the Linux OpenVMM unit tests on a GitHub-hosted runner #717, merged: the job graph in doc/ci.md keeps its GitHub-hosted Linux unit-test leg.
• ci: classify microVM tests, benchmarks, and artifact builds separately #712 (ci: classify the microVM tests, benchmarks, guest artifacts, and OpenVMM binaries separately #698), merged: this branch is rebased onto it and merges second. distro-artifacts is gated on run-guest-artifacts like artifacts and debug-kernel, and the change-classification test lists it with them. The drift guard counts it among the jobs that build the microVM test and platform jobs' artifacts. The required-results policy has it too (REQUIRED_CI_DISTRO_ARTIFACT_JOB, DISTRO_ARTIFACTS_RESULT). A new assertion checks that every VMM-test input is also a microVM-test input, so whenever the VMM tests select the guest artifacts, distro-artifacts has a consumer. The microVM test and platform jobs check needs.distro-artifacts.result == 'success', so a skipped or failed distro-artifacts fails the required check and blocks the release and performance persistence, which require the platform jobs.
• ci: queue the dev push runs instead of cancelling the waiting one #734, setup: install the pinned Rust release without updating rustup #732, Reject unrepresentable sandbox resource limits #735, nvx-rs: add a guest mapping API with explicit export and target roots #652, nvx-rs: separate distro verification from launch planning #653, and nvx-rs: parse the mount mode, owner, and kind names with one function #685, merged: in the base, c61080e23. ci: queue the dev push runs instead of cancelling the waiting one #734's queue: concurrency key is untouched, and nvx-rs: add a guest mapping API with explicit export and target roots #652's guest changes are why this head's run built every image cold.
• ci: reuse a pull request's guest images on its dev push #713 (ci: reuse a pull request's guest images on its dev push #702), open, rebases onto this: its reuse steps must follow steps.components.outputs.<component> instead of guest-images, run in both producers (each keeps id: guest-artifacts, and needs actions: read and its own key outputs), and its rehearsal job must read both producers' keys. Its key record must not ride in a guest-* component artifact, or the structural test's component table must name it.
• ci: build the Linux OpenVMM VMM tests once for the KVM and MSHV jobs #708 (ci: build the OpenVMM VMM tests once per target and hand them to the backend jobs #690), open, rebases onto this: its VMM-test consumers download guest-kernel and guest-alpine instead of guest-artifacts, and wait for artifacts only; its build-only producer needs no guest artifact. Any new nvx.py command that a consumer runs goes into the structural test's command table.
• ci: run the guest and sandbox steps beside the default microVM suite #706 (ci: run the guest smoke and sandbox steps in a job beside the default microVM suite #695), open, rebases onto this: a job that runs the Ubuntu, Azure Linux, or sandbox steps downloads those components and waits for distro-artifacts; a default-suite job downloads only guest-kernel and guest-alpine and waits only for artifacts.
• microvm tests: record each scenario's duration in a JSON Lines report #707 (ci: record each microVM scenario's duration in a machine-readable report #694): merges cleanly; its new upload step downloads no guest artifact, so the structural test ignores it.

Tests

• Structural test (test_ci_jobs_download_exactly_the_guest_artifacts_that_they_read): maps every step of each job that downloads guest artifacts to the guest files that it reads, through the nvx.py command or local action that it runs, the guest and kernel that test-microvm selects, and the build/ paths that its script names, evaluating each step's condition per caller and matrix leg. It fails when a job lacks a component that a step reads, downloads one that none reads, waits for a guest artifact job that uploads none of its components, or runs a command or local action that it cannot classify. It also checks that the uploads cover every file that the guest builds produce, that each producer builds exactly the components that it uploads and none twice, and, by mocking, that test-openvmm reads the kernel and the Alpine initramfs. Eight seeded regressions, such as a missing Azure Linux download or an Ubuntu sandbox step in the debug-kernel jobs, each fail it, before and after the rebase.
• The action: every component step follows exactly its own selection; the default leaves out the debug kernel; with bash, the selection rejects unknown, empty, glob, and comma-joined lists.
• Updated: required-status policy, the producers' shared condition and dependencies, the microVM and platform jobs' needs, the cache-write drift guard (four save-cache callers), the classification drift guard, and the download counts and order.
• ci: classify microVM tests, benchmarks, and artifact builds separately #712's change classification: distro-artifacts follows run-guest-artifacts, the drift guard counts it among the producers of the microVM test and platform jobs' inputs, the nvx-rs host-change policy expects it to succeed, and every VMM-test input matches a microVM-test input pattern.

Validation

• Windows, after rebasing onto dev at c61080e23 (ci: classify microVM tests, benchmarks, and artifact builds separately #712, ci: queue the dev push runs instead of cancelling the waiting one #734, setup: install the pinned Rust release without updating rustup #732, Reject unrepresentable sandbox resource limits #735, nvx-rs: add a guest mapping API with explicit export and target roots #652, nvx-rs: separate distro verification from launch planning #653, and nvx-rs: parse the mount mode, owner, and kind names with one function #685) for the head 46da18e49:
  • the full test_nvx_tools, test_markdown_links, and test_guest_binaries modules: 609 tests pass, 29 skipped (Linux- or POSIX-only);
  • ruff check, ruff format --check, pyright for Linux and Windows, check_markdown_links.py, and git diff --check: pass;
  • actionlint 1.7.12 (official release binary) on ci.yml, the two reusable workflows, and the Copilot setup: only unexpected key "queue" for "concurrency", a false positive of that release on ci: queue the dev push runs instead of cancelling the waiting one #734's key, which GitHub accepts;
  • the eight seeded regressions: each fails the structural test.
  • CI run 39690060426: passed in full, with every guest image built cold (above).
• Earlier rounds, for the head 012f72d4b: the same checks passed on 6f0facefd (ci: key the guest images and VMM tests on the guest binaries' sources #710, ci: link the Windows OpenVMM test builds with /Brepro for sccache #714, ci: run the Linux OpenVMM unit tests on a GitHub-hosted runner #717, openvmm: refuse kernel command lines that the guest kernel would truncate #661, and nvx-rs: add the VMM supervisor with a spawn allowlist, stop, and reattach #656; 606 tests), and the CI test classes again on f2b4a11ef (nvx-rs: parse legacy egress rules with the MXC form's helpers #682). CI run 39665274328 passed in full.
• Before the first rebase, on 3f6ad206f: every validate-nvx module and test_adversarial.py, test_hosts.py, and the nvx.py --help commands passed, except 18 test_setup_azure shell-script failures that dev shows on this host too (no scripts/setup change here). CI run 39624800851 passed in full.
• Bash (Git for Windows): the component selection script behaves as the POSIX-only test expects.
• Not run locally: ShellCheck and shfmt (no Docker daemon; no shell script changed), the Specula wiring tests (need Linux), and every runner and hypervisor job. This PR changes ci.yml, so its CI runs the full matrix.

Expected effect

• When the guest binaries change, every image misses and the VMM tests run: the VMM tests and the debug-kernel jobs start after about 1.6 min instead of about 6, when the single job would have built every image (run 39690060426, above). When the kernel misses too, every guest consumer starts after about 7.0 min instead of 10.9 (the experiment above).
• The microVM test and platform jobs wait for distro-artifacts about as long as for the single job, which distro-artifacts pays by building the base and guest-binary Docker stages again (about 0.6 min of GitHub-hosted time).
• Consumers download only what they read. In run 39624800851, the VMM tests downloaded 15.7 MB in two steps (2–6 s in all) instead of 140 MB (5–7 s), and the debug-kernel job 23 MB. The microVM test and platform jobs, which read every image, now take six and five download steps, which cost about 5–10 s more than the single download did.
• Once ci: build the OpenVMM VMM tests once per target and hand them to the backend jobs #690 lets the VMM tests build without waiting for guest artifacts, the gain on the VMM tests shrinks; the split then still serves cold-kernel runs and a default-suite job that reads only the kernel and the Alpine initramfs (ci: run the guest smoke and sandbox steps in a job beside the default microVM suite #695).

Post-merge verification

• Timing, with queue time, of at least five comparable runs with cold guest images and the VMM tests, before and after: the artifacts and distro-artifacts durations and the WHP VMM tests' start and wall time.
• Re-measure the per-component misses with ci: key the guest images and VMM tests on the guest binaries' sources #710's keys, and after ci: reuse a pull request's guest images on its dev push #702 merges, and record them on ci: build the guest-artifact components in parallel and download only what each job uses #703.

[Copilot]

🔵 Needs a closer look

Cross-platform artifact handoffs and release gates need full-matrix CI confirmation and final human review.

No open issues 🎉

What changed in this PR

Reduces CI wait time by separating kernel/Alpine builds from distro-image builds and downloading only the guest components each consumer uses.

Changes:

• Adds validated component selection and separate artifact uploads.
• Updates consumer dependencies, downloads, and required-status checks.
• Adds handoff tests and documents the new job structure.

File	Description
scripts/​test_nvx_tools.py	Tests component selection, handoffs, and gates.
scripts/​nvx_tools/​ci.py	Requires the distro-artifact job when workloads run.
doc/​ci.md	Documents producers, consumers, and dependencies.
doc/​build.md	Documents debug-kernel component selection.
.github/​workflows/​run-platform.yml	Downloads benchmark and release components selectively.
.github/​workflows/​run-nvx-microvm-tests.yml	Selects downloads for normal and debug tests.
.github/​workflows/​ci.yml	Splits producers and updates handoffs and gates.
.github/​actions/​build-guest-artifacts/​action.yml	Validates and builds selected components.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-platform artifact handoffs and required-gate changes warrant human review with exact-head CI confirmation.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-platform artifact handoffs and required-gate changes need final human confirmation against current-head CI results.

No open issues 🎉

_{Review effort: Balanced}