Repository navigation
ci: key the guest images and VMM tests on the guest binaries' sources - #710
Merged
Pedro Henrique Penna (ppenna) merged 3 commits intoOct 21, 2026
Merged
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The dependency guard accepts empty untracked directories, allowing later additions to leave guest-image cache keys unchanged.
1 open issue
What changed in this PR
Narrows guest-image cache keys and OpenVMM VMM-test selection to guest build inputs, reducing unnecessary CI work for host-only changes.
Changes:
- Shares one source definition between guest identities and VMM-test selection.
- Adds Cargo dependency checks and regression coverage.
- Documents the narrower inputs and cache behavior.
| File | Description |
|---|---|
| scripts/test_nvx_tools.py | Tests cache keys, classification, and release fixtures. |
| scripts/test_guest_binaries.py | Tests source identities and dependency validation. |
| scripts/nvx_tools/guest_binaries.py | Filters inputs and validates Cargo dependencies. |
| scripts/nvx_tools/ci.py | Narrows VMM-test triggers. |
| scripts/nvx_tools/build_constants.py | Defines guest inputs and bumps digest format. |
| nvx-rs/README.md | Documents guest build dependencies. |
| doc/usage.md | Explains identity and validation behavior. |
| doc/ci.md | Documents cache and test-selection changes. |
| .github/actions/build-guest-artifacts/action.yml | Removes whole-crate hashing from image keys. |
Review effort: Balanced
The guest image cache keys end with the source identity of the Rust guest binaries, a digest of the paths and contents of the files that GuestRustBuildConstants lists, which the package manifests also record. Nothing tied that list to what the build reads, though: a module that the build compiled from outside nvx-rs/src, an include_str! target there, or a file that the build script watches would change the executables without changing the keys, and CI would reuse stale guest images. Cargo writes a dep-info file beside each executable that lists every source file of the crate that rustc read, the targets of include_str! and include_bytes! included, and every path that the build script watches. build_guest_binaries now reads it for each of the nine executables after the build, and fails, naming the files, unless the source identity covers each of them. Each file of a directory that the build script watches must be covered. An empty directory never is, because a file added to it later could change the build without changing the identity, and neither is a path that does not exist or lies outside the checkout; a relative path is rejected. Files that the build generates in its fresh target directory, such as those in OUT_DIR, derive from the others and are exempt. The Docker guest stage, the nvx-rs job, native guest builds, and verify-guest-determinism all build through this routine, so each of them runs the check. The identity covers the build today: the nine dep-info files of a guest build of this tree list build.rs and 96 files under nvx-rs/src. Tests run the build against a stand-in Cargo whose dep-info lists covered files and a generated one. They check that a file, a directory, a missing path, a file outside the checkout, and a relative path that the identity omits each fail the build without installing an output, that a missing dep-info file fails it, that an empty watched directory fails the check until it holds a covered file, and that the parser reads escaped spaces, continuation lines, comments, and phony rules. doc/usage.md describes the check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The source identity of the Rust guest binaries, which the guest image cache keys end with and their package manifests and provenance record, covered every file under nvx-rs/src. The guest build compiles only part of it, though. Of the 168 tracked files there, the nine executables of a guest build of this tree read 96, as their Cargo dep-info shows. The host and testing modules, the nvx-rs and nvx-fake-openvmm binaries, the test modules of protocol, and the managed agent's loopback harness, which only host-featured test builds compile, make up the other 72. A change to any of them changed the identity without changing a guest binary. GuestRustBuildConstants.SOURCE_PATTERN now selects the inputs: the manifest, lockfile, and build script, the toolchain pin, and the two Python modules that define the build, as before; the crate's Cargo configuration under nvx-rs/.cargo, which a build in the crate reads and the identity didn't cover; and every file under nvx-rs/src but the host side and the test modules. The host side is the host and testing modules and the binaries other than the guest ones; the test modules are the tests.rs files, the tests directories, and the modules that TEST_MODULES lists, today the loopback harness. A new module elsewhere counts, so a new shared or guest module needs no change here, and the dep-info check that every guest build runs fails the build if an excluded file becomes an input. For this tree, the identity is the 97 files that the dep-info lists and the five that it can't: Cargo.toml, Cargo.lock, rust-toolchain.toml, and the two Python modules. A directory named target under nvx-rs/src no longer drops out of it; nothing builds there. SOURCE_DIGEST_FORMAT moves to 2, because the identity's definition changed, so every guest image cache key changes once. The initramfs provenance, the package manifests, the Ubuntu converter digest, and the Azure Linux input digest take the same files. The guest image keys still hash all of nvx-rs, so CI rebuilds the images as before; a following change narrows them. Tests check that the identity holds every source file but the host side and the test modules, against an independent statement of that rule over the tree; that changes to the host side, the tests, the README, or artifacts.json leave it as it is, while a change to any input, a new guest or top-level module, or Cargo configuration changes it; and that a guest build whose dep-info lists a host or test module fails. The release tests' fixture checkout now also copies the sources of the nvx-rs CLI, which packaging checks its provenance against and which the fixture took from the guest identity before. doc/usage.md describes the inputs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Alpine, Ubuntu, Ubuntu distro, and Azure Linux cache keys hashed all of nvx-rs but its target directory, and the toolchain pin, and the OpenVMM VMM tests took every nvx-rs path as an input, because they boot the Alpine initramfs. The images carry only the crate's Rust guest binaries, though, so a change to its host side or tests rebuilt every guest image and ran the VMM tests, the longest WHP jobs. Of the 42 dev merges through #659, the VMM tests ran on 33, 26 of them only because nvx-rs changed, and each of the 27 that changed nvx-rs changed every guest image key. A cold guest-artifact job takes a median 5.3 minutes instead of 0.4, and every job that boots a guest waits for it. The keys already end with the guest binaries' source identity, which now covers exactly the files that their build reads, and which every guest build checks against Cargo's dep-info. Drop nvx-rs and rust-toolchain.toml from the hashFiles of the image keys, so that the identity alone covers them, and select the VMM tests by the same pattern, GuestRustBuildConstants.SOURCE_PATTERN, instead of by every nvx-rs path. run-tests still covers all of nvx-rs, because the NVX microVM tests run the nvx-rs CLI. Replaying the classifier over those 42 merges, 11 of the 26 no longer run the VMM tests, and 12 of the 27 keep every guest image key: they changed only the host side, protocol tests, integration tests, or the loopback harness. The other 15 changed files that the guest binaries compile, such as guest and protocol modules, lib.rs, Cargo.toml, and Cargo.lock, and still run both. The drift guard that derives the VMM tests' inputs from the guest cache keys now also takes every file of the identity, and checks that the Alpine key ends with it. A new test checks every file under nvx-rs: those in the identity select the VMM tests, the others neither select them nor match a hashFiles pattern of an image key, and all of them select the microVM tests. New modules count unless they belong to the host side or the tests. For changes to the host side, the CLI and fake binaries, protocol tests, the loopback harness, integration tests, and artifacts.json, it checks that pushes and pull requests run the microVM tests but skip the VMM tests, and that the identity, and with it every image key, stays as it is. For changes to the manifest, lockfile, build script, lib.rs, protocol, a guest module, and a guest binary, it checks that they run both and change the identity. doc/ci.md, doc/usage.md, and the nvx-rs README describe the inputs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
force-pushed
the
agents/key-guest-images-on-guest-sources
branch
from
October 21, 2026 15:32
af768a7 to
6d40e1e
Compare
Pedro Henrique Penna (ppenna)
deleted the
agents/key-guest-images-on-guest-sources
branch
October 21, 2026 19:27
This was referenced Oct 21, 2026
Closed
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #699.
Part of #430.
Summary
The guest image cache keys and the OpenVMM VMM-test classification covered the whole
nvxcrate innvx-rs/, although the guest images carry only its nine Rust guest binaries. This PR keys both on the files that the guest binaries' build reads, so that a change to the crate's host side or tests neither rebuilds the guest images nor runs the VMM tests, while a change to any file that a guest binary compiles does both. The NVX microVM tests still run for everynvx-rschange, since they run thenvx-rsCLI.Every guest build now checks that set against Cargo's dep-info, which is authoritative, and fails when the build reads a file outside it, so the narrower keys stay fail-safe.
Why
The VMM tests ran for host-only changes. Replaying the classifier over the 42
devmerges through microvm tests: stream OpenVMM transcripts to disk as they are read #659 (as in ci: key the guest images and OpenVMM VMM tests on the guest binaries' sources, not all of nvx-rs #699), the VMM tests ran on 33, 26 only becausenvx-rs/changed. With this PR, 11 of those 26 skip them: nvx-rs: parse fake OpenVMM --mount-child values with one function #613, nvx-rs: check the guest agent's replies through shared methods #615, nvx-rs: convert OSErrors without a filename to CLI exits in one place #617, nvx-rs: derive the Artifacts of a directory from its ArtifactDir #632, nvx-rs: check guest-path containment with one function #634, nvx-rs: consume CLI options with one method for both parse paths #635, nvx-rs: encode the NVXS tests' records with one function #636, nvx-rs: run the guest-binary tests' nvx-time hooks with one helper #637, nvx-rs: run the control session on the shared NVXS session's host role #641, nvx-rs: build the NVXS unit tests' epoch identities with one function #678, and fix(sandbox): reserve /run, /run/nvx, and the /run/nvx/time tree as share targets #579 changed only the host side, protocol tests, integration tests, or the managed agent's test harness. The other 15 changed files that the guest binaries compile (guest and protocol modules,lib.rs,Cargo.toml,Cargo.lock) and still run them.Host-only changes rebuilt every guest image. All 27 merges that changed
nvx-rschanged every guest image key; 12 of them keep the keys with this PR (the 11 above and setup: integrate reusable runner setup and Azure VMSS bootstrap #522, which changed onlynvx-rs/tests/cli.rs).What that cost. 8 of those 12 merges have a completed
devpush run, all from before ci: run the OpenVMM suites on dev pushes only when their inputs change #660, and each ran the guest-artifact job cold and the three VMM-test jobs. With this PR, the guest images of all 8 would have come from the cache, and all but setup: integrate reusable runner setup and Azure VMSS bootstrap #522, which also changed the runner setup scripts, would have skipped the VMM tests:Medians: 5.3 minutes of cold guest builds (0.4 warm, per CI: maximize workflow parallelism and reduce turnaround time #430's baseline), which every job that boots a guest waits for, and, over the 7 that would skip them, about 47 runner-minutes of VMM tests (21.1 WHP, 15.7 KVM, 10.0 MSHV). The WHP VMM tests finished last in 11 of the 14 unqueued full-matrix runs of #430's baseline (reference run 39585909588).
What the guest build reads
A guest build of this tree (
build-guest-binarieson Linux, Rust 1.95.0) has nine Cargo dep-info files that list 97 files:build.rsand 96 of the 168 files undernvx-rs/src. The other 72 are thehost(62) andtesting(2) modules, thenvx-rsandnvx-fake-openvmmbinaries (2), the protocol test modules (5), andguest/managed_agent/loopback.rs, which only host-featured test builds compile (1). Cargo also reads the manifest, the lockfile, and any Cargo configuration in the crate, none of which dep-info lists.GuestRustBuildConstants.SOURCE_PATTERNselects exactly those inputs:nvx-rs/Cargo.toml,Cargo.lock,build.rs,rust-toolchain.toml, and thebuild_constants.pyandguest_binaries.pymodules that define the build, as before;nvx-rs/.cargo/, which the identity didn't cover;nvx-rs/srcexcept the host side (thehostandtestingmodules and the binaries other than the nine guest ones) and the test modules (tests.rsfiles,testsdirectories, andTEST_MODULES, today the loopback harness).A new module elsewhere counts by default, so new shared or guest modules need no change; the dep-info check catches the other direction.
Why not a separate guest crate? Moving the guest binaries into a crate of their own (#356, #619, #648) would still leave them depending on the shared
protocol,net,contract,error, andhypervisormodules, so it would need a three-way split ofnvx, and it would conflict with the ~20 opennvx-rsPRs (#652–#689) that edit those sources. The dep-info check gives the same exactness without moving a file.Changes
Three commits:
build: check the guest build's dep-info against its source identity.build_guest_binariesreads Cargo's dep-info for each of the nine executables after the build and fails, naming the files, unless the source identity covers every file that it lists: each source file that rustc read,include_str!/include_bytes!targets included, and each path that the build script watches. A watched directory must have every file covered, and an empty one never is, since a later file there could change the build without changing the identity; missing paths and paths outside the checkout never are either, and relative paths are rejected. Only files generated in the build's own fresh target directory, such asOUT_DIR, are exempt. The Docker guest stage, thenvx-rsjob, native builds, andverify-guest-determinismall build through this routine.build: limit the guest binaries' source identity to their build inputs. The identity, which the initramfs provenance, the package manifests, the Ubuntu converter digest, and the Azure Linux input digest all take, now usesSOURCE_PATTERN.SOURCE_DIGEST_FORMATmoves to 2, so every guest image key changes once.ci: key the guest images and VMM tests on the guest binaries' sources. The Alpine, Ubuntu, and Azure LinuxhashFilesdropnvx-rs/**,!nvx-rs/target/**, andrust-toolchain.toml, so that the identity at the end of each key alone covers them.OPENVMM_VMM_TEST_INPUT_PATHSreplacesnvx-rs/withSOURCE_PATTERN;CI_TEST_INPUT_PATHSkeeps all ofnvx-rs/.doc/ci.md,doc/usage.md, and thenvx-rsREADME describe the inputs.Tests
test_only_guest_binary_inputs_rebuild_guests_and_run_vmm_tests): every file undernvx-rs/(buttarget) selects the microVM tests, and selects the VMM tests if and only if it is in the identity; nohashFilespattern of an image key matches any of them, and each image key ends with the identity. For changes to the host side, the CLI and fake binaries, protocol tests, the loopback harness, integration tests, andartifacts.json, pushes and pull requests run the microVM tests but skip the VMM tests, and the identity, so every image key, is unchanged. Changes to the manifest, lockfile, build script,lib.rs, a protocol module, a guest module, and a guest binary run both and change the identity. New modules count unless they belong to the host side or the tests.nvx-rsCLI's sources explicitly instead of through the guest identity.Seeded regressions fail the new tests: selecting the VMM tests by all of
nvx-rs/, dropping the guest inputs or the anchor from the VMM pattern, hashingnvx-rs/**ornvx-rs/src/host/**in an image key, dropping the identity from the Azure Linux key, removing the host, test, or loopback exclusions, and disabling the dep-info check.Validation
validate-nvxunit-test modules,test_adversarial.py,test_hosts.py,compileall, and the CLI--helpsmoke checks pass, except 18test_setup_azuresubtests that also fail atorigin/devon this host (they run Linux shell scripts through WSL'sbashwith Windows paths); CI runs them.ruff check,ruff format --check,pyrightfor Linux and Windows, andcheck_markdown_links.pypass. After rebasing onto 5418be9, which changed only guest modules and performance data,test_guest_binaries.pyand the CI configuration, build, and release test classes pass again.nvx.py build-guest-binariesbuilds the nine executables with the check (identity623237a0…, 102 files: the 97 that dep-info lists and the 5 that it can't). Seedinginclude_str!("../host/mod.rs")intoguest/exit.rsfails the same build withGuestRustBuildConstants … must cover: nvx-rs/src/host/mod.rs.test_guest_binaries.pyand the Specula tests pass.6d40e1e99): every job passed. Because the identity's format changed,Build shared artifactsmissed every guest image cache, built the guest binaries in Docker with the dep-info check (identity8f5aa59e…, the suffix of the image keys), and ranverify-guest-determinism; the three platform jobs checked the guest provenance against their checkouts, the WHP one on Windows; and both OpenVMM suites ran, becauseci.pyandbuild_constants.pyare inputs of both.Expected effect
nvx-rsmerge, such as the 11 above, restores the guest images from the cache (about 0.4 instead of 5.3 minutes before the jobs that boot guests start) and skips the three VMM-test jobs (about 47 runner-minutes, 21 on WHP), so the WHP microVM job becomes its last job.devpush after this PR rebuilds the guest images, as this PR's run does.Post-merge verification
devpush or pull request after this merges that changes only host-sidenvx-rsfiles, confirm in the change detector's summary that the VMM tests are skipped and the microVM tests run, and thatBuild shared artifactsrestores every guest image from the cache.