[Pedro Henrique Penna (ppenna)]

Part of #690.
Part of #430.

Summary

The KVM and MSHV legs of openvmm-vmm-tests each built the same Linux OpenVMM VMM tests before running their backend's filter, and every leg waited for the guest artifacts before it started building. This PR builds the Linux tests once, in a build-only producer that starts beside the guest-artifact build, and hands the test content to two backend consumers that only run their existing filters with OpenVMM's vmm-tests-run-target.

The WHP tests can't move yet: at the pinned OpenVMM revision, vmm-tests-run-target requires Hyper-V on Windows whatever tests it runs, which the WHP runners can't provide (see WHP below). The WHP job therefore still builds and runs its tests in one job, unchanged, under the new job ID openvmm-vmm-tests-windows.

Why

• In reference run 39585909588, the KVM and MSHV legs each spent about 4.6 and 4.9 minutes on flowey, the test-binary build, and OpenVMM before their first test, and both logs resolve the identical artifact selection (the same Resolved selections line: the x86_64-unknown-linux-gnu nextest archive, OpenVMM, openvmm_vhost, the musl pipette, guest_test_uefi, the test kernel, initrd, bzImage, and UEFI firmware) and lay out the same test content.
• In the 14 successful first-attempt full-matrix dev pushes from 2026-10-20 18:13 to 2026-10-21 13:32 UTC (runs and data on #690), the Linux legs took a median 15.6 (KVM) and 9.9 (MSHV) minutes, 25.6 Linux runner-minutes per run, and all three legs started only after the guest artifacts, which ran cold in 9 of the 14 runs for a median 5.3 minutes.
• One Linux build serves both backends: the target triple is the same, the backend is detected at run time, OpenVMM's Cargo configuration sets no host-CPU code generation, both pools run Azure Linux 3, and the GNU OpenVMM that a KVM runner builds already runs the MSHV microVM tests.

Changes

nvx.py build-openvmm-tests and test-openvmm --content (first commit)

• build-openvmm-tests --backend B [--backend B ...] --output DIR runs cargo xflowey vmm-tests-run --build-only on the union of the backends' filters ((<kvm>) | (<mshv>)), which writes the nextest archive, OpenVMM, the guest agents, the test firmware and kernels, and a flowey_hvlite that runs vmm-tests-run-target. It parses the run script that the build writes strictly (runner inside the directory, --dir ., the built filter, and only the options that the pinned flowey writes), and writes nvx-openvmm-tests.json: the format, the OpenVMM revision and whether its checkout was clean, the Rust release, each backend's filter, the build filter, the runner and its options, and every file's size and executable bit.
• test-openvmm --backend B --content DIR runs backend B's own filter from such a directory with its vmm-tests-run-target, without the OpenVMM checkout or a build. It refuses content of another OpenVMM revision than the gitlink, of another filter, or with a missing or truncated file, and it restores the executable bits that artifact downloads drop. It sets the same OPENVMM_MICROVM_TEST_KERNEL/INITRD, RUSTUP_TOOLCHAIN, and XDG_CACHE_HOME as before. Without --content, test-openvmm builds and runs the tests as before. On Windows, the build links with /Brepro, as test-openvmm does since ci: link the Windows OpenVMM test builds with /Brepro for sccache #714.

CI (second commit)

Job	Runner	Waits for	Does
build-openvmm-vmm-tests-linux (new)	general KVM, whose sccache holds the Linux test build	openvmm-changes only	build-openvmm-tests --backend kvm --backend mshv; uploads the one-day openvmm-vmm-tests-linux artifact (include-hidden-files for .config/nextest.toml, overwrite: true); restores and, on dev pushes, saves the vmm-tests input and Cargo caches
openvmm-vmm-tests-linux (KVM, MSHV legs)	general KVM / MSHV, as before	artifacts and the build	downloads the content and the guest artifacts, runs test-openvmm --content; no OpenVMM checkout, no caches, no sccache
openvmm-vmm-tests-windows	general WHP, as before	artifacts	unchanged: builds and runs the WHP tests

• Diagnostics keep the openvmm-vmm-tests-<os>-<backend>-<run-id>-<run-attempt> names; Linux now collects <runner-temp>/openvmm-vmm-tests/test_results.
• Gating: the build runs whenever run-openvmm-vmm-tests is set, which the classifier of ci: classify microVM tests, benchmarks, and artifact builds separately #712 sets only together with run-guest-artifacts, or when the change detector fails, so it runs exactly when its consumers do, and run-openvmm-vmm-tests=false skips producer and consumers together. It reads that output alone, as ci: classify microVM tests, benchmarks, and artifact builds separately #712's one-output-per-job test requires. check-required-ci expects it with them. Publish development release and Persist performance baseline, which accepted any skipped VMM-test job, now need openvmm-changes and require the three VMM jobs to succeed, or all three to be skipped because the change selected no VMM tests, so a failed or unexpectedly skipped producer or consumer blocks them. Fork, draft, and path-filter gating, cancellation, and the eight cache saves' keys and guards are unchanged.
• Reruns: a rerun of a failed consumer reuses the producer's artifact from the earlier attempt instead of rebuilding; a rerun of the producer replaces its artifact.
• Docs: doc/ci.md gains an OpenVMM VMM tests section on the producer, consumers, handoff, manifest, and WHP; doc/usage.md documents both commands.

WHP

At the pinned OpenVMM revision, 9f04d05c5 since #661, whose flowey/ is identical to that of d0bdbed8f, vmm-tests-run-target hard-codes hyperv: true for Windows targets (flowey/flowey_hvlite/src/pipelines/vmm_tests_run_target.rs), so install_vmm_tests_external_deps requires Get-VM to succeed and the AllowFirmwareLoadFromFile and EnableAdditionalComPorts values under HKLM\Software\Microsoft\Windows NT\CurrentVersion\Virtualization, or, with --install-missing-deps, runs DISM /Online, which needs elevation. The WHP runners run jobs as Network Service (validate-runner enforces S-1-5-20), which the runner setup doesn't add to Hyper-V Administrators, and the setup sets neither value. The combined vmm-tests-run computes hyperv from the selected tests, which is why the WHP job works today. Splitting WHP needs vmm-tests-run-target to take the dependencies that the build resolved, for example the --needs-hyperv and --needs-whp options of commit 79eb62981 on the copilot/vmm-tests-run-target-deps branch of nanvix/openvmm, for which I found no microsoft/openvmm pull request, and then a pin promotion. The content format and the consumer here already cover a Windows producer.

Tests

• The union build filter; both run-script formats and each rejection (filter, cd, pipeline, runner outside the directory, unknown option, missing value, other directory, missing --target).
• The build's commands, environment, and manifest; the refusal of another OS's backends, unknown backends, and used directories.
• The consumer's command, working directory, environment, and restored executable bits; the refusal of another format, revision, backend, or filter, a malformed runner or options, a file outside the directory, and a missing or truncated file.
• The CLI parsing and doc/usage.md sections.
• Workflow structure: the producer's runner, needs, and condition; the change detector's outputs that the VMM-test jobs and the two gates read; the consumers' needs and conditions; the upload's name, path, hidden files, overwrite, and retention; the download and test order; diagnostics on both OSes; the WHP job's independence of the Linux build; the release and persistence gate; the required-check policy for every event and flag combination; the cache keys and saves; and the VMM suite's inputs derived from every job that builds or runs it.

Validation

• Windows: the full scripts/test_nvx_tools.py module (500 tests, OK, after the rebase onto 4f9e7cbd1); ruff check, ruff format --check; pyright for the Linux and Windows platforms; check_markdown_links.py; git diff --check.
• WSL Ubuntu (Python 3.14): CiTests, CliTests, CiConfigurationTests, and the guest-artifact download test (132 tests, OK), which cover the Linux paths of the build and the consumer.
• actionlint v1.7.7 (release binary, checksum verified) with .github/actionlint.yaml: ci.yml reports only unexpected key "queue" for "concurrency", which ci: queue the dev push runs instead of cancelling the waiting one #734 added and GitHub accepts; actionlint 1.7.7 predates the key.
• CI on the rebased head 2f85af825, run 39691097674: every job passed, including Required status check, apart from the expected skips (the MSHV and WHP debug-kernel jobs on a pull request, the release, and the persistence). The producer built the same 214.7 MiB of content in 6.7 minutes, and the MSHV and KVM consumers ran the same 57 and 58 tests, all passing, in 4.8 and 10.3 minutes. Before the rebase, run 39625831399: the producer built the content (214.7 MiB, a 112.3 MiB artifact, uploaded in 6 seconds) in 7.0 minutes, and the MSHV and KVM consumers downloaded it in 8 and 6 seconds and ran 57 and 58 tests, all passing, with 50 and 49 skipped, exactly as before, in 4.6 and 10.5 minutes; every other job, including Required status check, passed. test-openvmm and test-openvmm-unit can't run locally.

Expected effect

From the PR runs (details on #690):

• Linux runner-minutes: 7.0 + 4.6 + 10.5 = 22.1 and 6.7 + 4.8 + 10.3 = 21.8 instead of a median 25.6 (−3.5 and −3.8, −14% and −15%). The producer's extra flowey_hvlite build for the target, about 1.5 minutes, missed sccache on each of the three KVM runners that have run the producer so far, each for the first time, so the saving may grow as their caches warm.
• Linux lanes: with cold guest artifacts (9 of the 14 runs above), the build overlaps them, so the Linux lanes finish about 3 minutes sooner: in run 39691097674, whose guest artifacts took 5.9 minutes, the MSHV and KVM lanes finished 11.8 and 18.3 minutes after the first job started, 3.5 and 2.8 minutes before the medians of the comparable runs above; with warm ones, the tests wait for the build and the handoff, so the KVM and MSHV lanes should finish about 1.5 and 1 minutes later, still well before the WHP lane.
• Wall time: unchanged, because the WHP lane, the last to finish in most full runs, is unchanged until the WHP split is unblocked.

Post-merge verification

• Compare at least five comparable full-matrix runs after merge with the 14 runs above: producer and consumer queue time and duration, the content's size, upload, and download times, the Linux lanes' finish times, and the Linux runner-minutes.

Interactions with other pull requests

This branch is rebased onto 4f9e7cbd1 and adapted to the pull requests that merged since its first version:

• ci: classify microVM tests, benchmarks, and artifact builds separately #712: the build job and the VMM-test clause of Publish development release and Persist performance baseline read run-openvmm-vmm-tests alone, check-required-ci expects the build whenever the VMM tests are selected (the test jobs also need the guest artifacts), and ci: classify microVM tests, benchmarks, and artifact builds separately #712's tests map the build job to that output and accept the two gates' reads of it.
• ci: key the guest images and VMM tests on the guest binaries' sources #710: its classification test checks the three VMM-test jobs instead of openvmm-vmm-tests; the VMM-test inputs derive from the actions of all three jobs and artifacts, and don't include the host side of nvx-rs.
• ci: link the Windows OpenVMM test builds with /Brepro for sccache #714: build-openvmm-tests links with /Brepro on Windows through _prepare_openvmm_test_environment, which a test now checks.
• ci: run the Linux OpenVMM unit tests on a GitHub-hosted runner #717, ci: queue the dev push runs instead of cancelling the waiting one #734, openvmm: refuse kernel command lines that the guest kernel would truncate #661: no change needed beyond merging the docs and tests; the flowey interfaces that this PR relies on are unchanged at OpenVMM 9f04d05c5.

Trial merges of this branch with the open siblings' heads as of 2026-10-21 23:10 UTC show what the pull request that merges second must change:

• ci: build the guest-artifact components in parallel and download only what each job uses #711 (46da18e49): conflicts in doc/ci.md and scripts/test_nvx_tools.py, and ci.yml merges cleanly but leaves openvmm-vmm-tests-linux downloading the guest-artifacts artifact that ci: build the guest-artifact components in parallel and download only what each job uses #711 removes. The Linux VMM-test jobs must download guest-kernel and guest-alpine (they already wait only for artifacts), ci: build the guest-artifact components in parallel and download only what each job uses #711's component test must list REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS instead of openvmm-vmm-tests among the consumers, and its command table must map build-openvmm-tests, which reads no guest file; the build job downloads no component.
• ci: run the guest and sandbox steps beside the default microVM suite #706 (289be9cb1): conflicts in doc/ci.md and the REQUIRED_CI_* constants; keep both sides.
• ci: reuse a pull request's guest images on its dev push #713 (7e3c899ac): conflicts in the job-graph paragraph of doc/ci.md; keep both sides.
• microvm tests: record each scenario's duration in a JSON Lines report #707 (a904264bc) and ci: report nightly full-run failures #709 (f4b9ae82f) merge cleanly.

[Copilot]

🟡 Changes recommended

Windows manifest paths can escape the content directory; cross-runner execution also still needs end-to-end CI validation.

1 open issue

• Windows path traversal bypasses content directory validation · New

What changed in this PR

Builds Linux OpenVMM VMM tests once for KVM and MSHV, reducing duplicate CI work while retaining the combined WHP job.

Changes:

• Adds build-only and prebuilt-content CLI commands.
• Splits Linux CI into a producer and backend consumers, updating required and publishing gates.
• Adds handoff validation tests and documentation.

File	Description
scripts/​test_nvx_tools.py	Tests CLI, manifests, handoffs, and workflow gates.
scripts/​nvx.py	Exposes build-only and content-consumer options.
scripts/​nvx_tools/​ci.py	Implements content production, validation, and execution.
doc/​usage.md	Documents new command options.
doc/​ci.md	Explains job dependencies and artifact handoffs.
.github/​workflows/​ci.yml	Shares Linux test builds and updates gating.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The cross-runner artifact handoff needs successful producer and KVM/MSHV consumer runs before human approval.

No open issues 🎉

1 resolved since last review

• Windows path traversal bypasses content directory validation

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The cross-runner executable handoff and release/performance gate changes warrant final human review with current-head CI results.

No open issues 🎉

_{Review effort: Balanced}