From b09d275d1026c8c4131675ec5eadbb765a6423ac Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 21 Oct 2026 11:41:29 -0300 Subject: [PATCH 1/2] ci: build the OpenVMM VMM tests without running them test-openvmm built OpenVMM's VMM tests and ran them in one command, so every CI job that ran a backend's tests also built them: the KVM and MSHV jobs built the same Linux tests, and each job waited for the guest artifacts before it started its build, although only the tests boot the NVX kernel and initramfs. build-openvmm-tests builds the tests of one or more backends of the host's operating system into a test content directory without running them. It runs OpenVMM's vmm-tests-run in build-only mode on the union of the backends' filters, which writes the nextest archive, OpenVMM, the guest agents, the test firmware and kernels, and the flowey_hvlite that runs vmm-tests-run-target. It then reads that runner and its options from the run script that the build writes, and records them in nvx-openvmm-tests.json with the OpenVMM revision and whether its checkout was clean, the Rust release, each backend's filter, and every file's size and executable bit, so the content names what it holds and the inputs that built it. The run script is parsed strictly: it must run vmm-tests-run-target on the built filter in its own directory, with a runner inside it and only the options that the pinned flowey writes, so that a change of its format fails the build instead of running other tests. test-openvmm --content runs one backend's filter from such a directory with vmm-tests-run-target, which needs neither the OpenVMM checkout nor a build, only the guest artifacts: the microVM TTRPC test reads the NVX kernel and initramfs from its environment when it runs. It refuses content built from another OpenVMM revision than the gitlink or for another filter than the backend's, and content that lacks a file or holds a truncated one, and it restores the executable bits that an artifact download drops. Without --content, test-openvmm builds and runs the tests as before. Like it, the build links with /Brepro on Windows. doc/usage.md describes both commands, including that the pinned vmm-tests-run-target checks for Hyper-V on Windows whatever tests it runs. Tests cover the union filter, both run script formats and every rejection of a script, the build's commands, environment, and manifest, the refusal of another OS's backends and of used directories, the consumer's command, environment, and restored executable bits, and the refusal of foreign, malformed, and incomplete content. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/usage.md | 40 ++- scripts/nvx.py | 34 ++- scripts/nvx_tools/ci.py | 361 +++++++++++++++++++++++-- scripts/test_nvx_tools.py | 538 +++++++++++++++++++++++++++++++++++++- 4 files changed, 954 insertions(+), 19 deletions(-) diff --git a/doc/usage.md b/doc/usage.md index ec9a2cee9..ebb3f995f 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -84,6 +84,7 @@ stale build never ships or runs unnoticed. | `report-nightly-ci` | Open, update, or close the report of failed nightly CI runs. | | `check-static-elf` | Check that executables are static ELF files without an interpreter or an executable stack. | | `test-openvmm-unit` | Run the OpenVMM workspace unit and documentation tests. | +| `build-openvmm-tests` | Build OpenVMM Petri VMM tests without running them. | | `test-openvmm` | Run OpenVMM Petri VMM tests. | | `test-microvm` | Run NVX Linux and device correctness tests through OpenVMM. | | `doctor` | Qualify this host for the NVX time ABI. | @@ -457,10 +458,33 @@ Afterward, runs the workspace doctests with Cargo. On Windows, it appends can reuse the crates that use them; see [Compilation cache](ci.md#compilation-cache). +### `build-openvmm-tests` + +```text +python3 scripts/nvx.py build-openvmm-tests + --backend {kvm,mshv,whp} [--backend {kvm,mshv,whp} ...] + --output PATH +``` + +Builds the VMM tests that [`test-openvmm`](#test-openvmm) runs for each +`--backend`, without running them, into the new or empty directory that +`--output` names, so that another host can run them without building OpenVMM. +The backends must run on this host's operating system: Linux builds the KVM +and MSHV tests, and Windows the WHP tests. The command runs OpenVMM's +`vmm-tests-run` in build-only mode on the union of the backends' filters, +which puts the tests' nextest archive, OpenVMM, the guest agents, the test +firmware and kernels, and a `vmm-tests-run-target` runner into the directory. +It then reads the runner's options from the run script that the build writes, +and records them in the directory's `nvx-openvmm-tests.json` with the OpenVMM +revision and whether its checkout was clean, the Rust release, each backend's +filter, and every file's size and executable bit. The tests boot NVX's kernel +and initramfs only when they run, so the build needs no guest artifacts. On +Windows, it appends `/Brepro` to `_LINK_`, as `test-openvmm` does. + ### `test-openvmm` ```text -python3 scripts/nvx.py test-openvmm --backend {kvm,mshv,whp} +python3 scripts/nvx.py test-openvmm --backend {kvm,mshv,whp} [--content PATH] ``` Builds and runs OpenVMM's checkout-owned VMM tests. The Linux-direct microVM @@ -468,6 +492,20 @@ TTRPC test boots NVX's `build/vmlinux` and `build/initramfs.cpio.gz`, so build the guest first; the remaining test artifacts are produced by OpenVMM itself. On Windows, it appends `/Brepro` to `_LINK_`, as `test-openvmm-unit` does. +With `--content`, the command builds nothing: it runs the backend's tests from +a directory that [`build-openvmm-tests`](#build-openvmm-tests) built for that +backend, with the directory's `vmm-tests-run-target` runner and the options +that its manifest records, so the host needs neither the OpenVMM checkout nor +its Rust targets. It refuses a directory built from another OpenVMM revision +than the checkout pins or for another filter than the backend's, and one that +lacks a file of the build or holds a truncated one, and it restores the +executable bits that an artifact download drops. It still needs the guest +artifacts above, and Petri writes the test results to `test_results` in the +directory. On Windows, the pinned OpenVMM's runner checks for Hyper-V whatever +tests it runs: the account must be able to run `Get-VM`, and the +`AllowFirmwareLoadFromFile` and `EnableAdditionalComPorts` values must be set +under `HKLM\Software\Microsoft\Windows NT\CurrentVersion\Virtualization`. + ### `test-microvm` ```text diff --git a/scripts/nvx.py b/scripts/nvx.py index 00545bd8d..9c2ab73b8 100644 --- a/scripts/nvx.py +++ b/scripts/nvx.py @@ -52,6 +52,7 @@ OPENVMM_TEST_BACKENDS, REQUIRED_CI_RESULT_ENVIRONMENTS, CiChanges, + build_openvmm_tests, classify_ci_changes, describe_ci_changes, kernel_patch_identity, @@ -354,8 +355,12 @@ def command_check_static_elf(args: argparse.Namespace) -> None: check_static_executables(args.paths) +def command_build_openvmm_tests(args: argparse.Namespace) -> None: + build_openvmm_tests(args.backend, args.output) + + def command_test_openvmm(args: argparse.Namespace) -> None: - run_openvmm_tests(args.backend) + run_openvmm_tests(args.backend, args.content) def command_test_openvmm_unit(_: argparse.Namespace) -> None: @@ -941,6 +946,25 @@ def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: ) openvmm_unit_tests.set_defaults(handler=command_test_openvmm_unit) + openvmm_test_build = subparsers.add_parser( + "build-openvmm-tests", + help="build OpenVMM Petri VMM tests without running them", + ) + openvmm_test_build.add_argument( + "--backend", + choices=OPENVMM_TEST_BACKENDS, + action="append", + required=True, + help="build this backend's tests; repeat for each backend of this host's OS", + ) + openvmm_test_build.add_argument( + "--output", + type=Path, + required=True, + help="empty or new directory to build the test content into", + ) + openvmm_test_build.set_defaults(handler=command_build_openvmm_tests) + openvmm_tests = subparsers.add_parser( "test-openvmm", help="run OpenVMM Petri VMM tests", @@ -950,6 +974,14 @@ def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: choices=OPENVMM_TEST_BACKENDS, required=True, ) + openvmm_tests.add_argument( + "--content", + type=Path, + help=( + "run the tests from the directory that build-openvmm-tests built, " + "instead of building them" + ), + ) openvmm_tests.set_defaults(handler=command_test_openvmm) microvm_tests = subparsers.add_parser( diff --git a/scripts/nvx_tools/ci.py b/scripts/nvx_tools/ci.py index 52b19fad5..7185d3066 100644 --- a/scripts/nvx_tools/ci.py +++ b/scripts/nvx_tools/ci.py @@ -3,16 +3,20 @@ from __future__ import annotations import hashlib +import json import os import re import shutil +import stat import zipfile from collections.abc import Mapping, Sequence from dataclasses import dataclass -from pathlib import Path +from pathlib import Path, PurePosixPath +from typing import cast from .build_constants import ( AlpineBuildConstants, + BuildConstants, GuestRustBuildConstants, KernelBuildConstants, OpenVMMBuildConstants, @@ -21,15 +25,40 @@ from .common import ( ScriptError, artifact_path, + bytes_to_mib, download, + openvmm_git_state, + path_exists, require_file, require_success, require_tool, run_capture, run_checked, + strict_json_object, ) OPENVMM_TEST_BACKENDS = ("kvm", "mshv", "whp") +# The operating system that builds and runs each backend's VMM tests. +OPENVMM_TEST_OPERATING_SYSTEMS = {"kvm": "linux", "mshv": "linux", "whp": "windows"} +# `build-openvmm-tests` describes the test content that it builds in this file, +# which `test-openvmm --content` reads. +OPENVMM_TEST_CONTENT_MANIFEST = "nvx-openvmm-tests.json" +OPENVMM_TEST_CONTENT_FORMAT = 1 +# The script that `vmm-tests-run --build-only` writes to run the content. +OPENVMM_TEST_RUN_SCRIPTS = {"linux": "run.sh", "windows": "run.ps1"} +# The vmm-tests-run-target options that the pinned run scripts pass, and +# whether each takes a value. The consumer passes its own --dir and --filter. +OPENVMM_TEST_RUN_TARGET_OPTIONS = { + "--target": True, + "--repetitions": True, + "--artifacts": True, + "--prep-steps": True, + "--skip-vhd-prompt": False, + "--ci-profile": False, + "--no-reuse-prepped-vhds": False, + "--needs-hardware-isolation": False, + "--needs-igvm-agent": False, +} REQUIRED_CI_RESULT_ENVIRONMENTS = { "quality": "QUALITY_RESULT", "aci-edge-sandboxes": "ACI_EDGE_SANDBOXES_RESULT", @@ -664,11 +693,17 @@ def _link_reproducibly_with_msvc(environment: dict[str, str]) -> None: def _prepare_openvmm_test_environment( - backend: str, + backends: Sequence[str], rustup: str, ) -> dict[str, str]: toolchain = OpenVMMBuildConstants.RUST_TOOLCHAIN - targets = OpenVMMBuildConstants.TEST_RUST_TARGETS[backend] + targets = list( + dict.fromkeys( + target + for backend in backends + for target in OpenVMMBuildConstants.TEST_RUST_TARGETS[backend] + ) + ) installed = run_capture( [ rustup, @@ -720,13 +755,22 @@ def _prepare_openvmm_test_environment( env=environment, ) - if backend != "whp" and runner_temp_value: + _set_openvmm_test_cache(environment, backends) + return environment + + +def _set_openvmm_test_cache( + environment: dict[str, str], + backends: Sequence[str], +) -> None: + # Petri streams and layers guest disks in the XDG cache, and the Linux + # runner account has no home directory to hold it. + runner_temp_value = os.environ.get("RUNNER_TEMP") + if "whp" not in backends and runner_temp_value: environment["XDG_CACHE_HOME"] = os.fspath( Path(runner_temp_value) / OpenVMMBuildConstants.CACHE_DIRECTORY_NAME ) - return environment - def run_openvmm_unit_tests() -> None: require_file( @@ -783,14 +827,7 @@ def run_openvmm_unit_tests() -> None: ) -def run_openvmm_tests(backend: str) -> None: - validate_openvmm_test_backend(backend) - - require_file( - OpenVMMBuildConstants.DIRECTORY / "Cargo.toml", "initialized OpenVMM submodule" - ) - cargo = require_tool("cargo") - rustup = require_tool("rustup") +def _openvmm_microvm_test_guest() -> tuple[Path, Path]: kernel = require_file( artifact_path(KernelBuildConstants.BINARY_NAME), "microVM Linux direct kernel", @@ -799,8 +836,25 @@ def run_openvmm_tests(backend: str) -> None: artifact_path(AlpineBuildConstants.INITRAMFS_NAME), "microVM Alpine initramfs", ) + return kernel, initrd + + +def run_openvmm_tests(backend: str, content: Path | None = None) -> None: + """Run the backend's VMM tests, building them unless ``content`` names a + test content directory that ``build_openvmm_tests`` built.""" + validate_openvmm_test_backend(backend) + if content is not None: + _run_openvmm_test_content(backend, content) + return - rust_environment = _prepare_openvmm_test_environment(backend, rustup) + require_file( + OpenVMMBuildConstants.DIRECTORY / "Cargo.toml", "initialized OpenVMM submodule" + ) + cargo = require_tool("cargo") + rustup = require_tool("rustup") + kernel, initrd = _openvmm_microvm_test_guest() + + rust_environment = _prepare_openvmm_test_environment((backend,), rustup) rust_environment["OPENVMM_MICROVM_TEST_KERNEL"] = os.fspath(kernel.resolve()) rust_environment["OPENVMM_MICROVM_TEST_INITRD"] = os.fspath(initrd.resolve()) run_checked( @@ -830,6 +884,283 @@ def run_openvmm_tests(backend: str) -> None: run_checked(command, cwd=OpenVMMBuildConstants.DIRECTORY, env=rust_environment) +def openvmm_test_build_filter(backends: Sequence[str]) -> str: + """Return the filter that selects every test of the backends' filters.""" + filters = [ + OPENVMM_TEST_FILTERS[backend] + for backend in OPENVMM_TEST_BACKENDS + if backend in backends + ] + if len(filters) == 1: + return filters[0] + return " | ".join(f"({test_filter})" for test_filter in filters) + + +def _openvmm_test_content_backends(backends: Sequence[str]) -> tuple[str, ...]: + if not backends: + raise ScriptError("select at least one OpenVMM test backend") + for backend in backends: + if backend not in OPENVMM_TEST_BACKENDS: + choices = ", ".join(OPENVMM_TEST_BACKENDS) + raise ScriptError( + f"unsupported OpenVMM test backend {backend!r}; choose {choices}" + ) + host = "windows" if os.name == "nt" else "linux" + for backend in backends: + if OPENVMM_TEST_OPERATING_SYSTEMS[backend] != host: + raise ScriptError( + f"{backend.upper()} OpenVMM tests build on " + f"{OPENVMM_TEST_OPERATING_SYSTEMS[backend].capitalize()}" + ) + return tuple(backend for backend in OPENVMM_TEST_BACKENDS if backend in backends) + + +def _content_relative_path(value: object) -> PurePosixPath | None: + """Return a path inside a test content directory, or None if ``value`` + names none. + + Manifests and run scripts name files with forward slashes, so a backslash + or a drive, which Windows would resolve elsewhere, names no such path. + """ + if not isinstance(value, str) or "\\" in value or ":" in value: + return None + path = PurePosixPath(value) + if not path.parts or path.is_absolute() or ".." in path.parts: + return None + return path + + +def _openvmm_test_run_target(script: Path, build_filter: str) -> tuple[str, list[str]]: + """Return the runner and the options other than --dir and --filter of the + vmm-tests-run-target command that a build-only run script runs. + + The pinned flowey writes the command after a `cd` to the script's + directory, as its arguments joined with spaces, and quotes only the filter. + """ + text = require_file(script, "OpenVMM test run script").read_text(encoding="utf-8") + quoted_filter = f" --filter '{build_filter}'" + _, separator, command = text.strip().partition(" ; ") + if not separator or command.count(quoted_filter) != 1: + raise ScriptError(f"{script} does not run the filter that it was built for") + tokens = command.replace(quoted_filter, "", 1).split() + if tokens[1:4] != ["pipeline", "run", "vmm-tests-run-target"]: + raise ScriptError(f"{script} does not run vmm-tests-run-target") + runner = tokens[0] + runner_path = _content_relative_path(runner[2:].replace("\\", "/")) + if not runner.startswith(("./", ".\\")) or runner_path is None: + raise ScriptError(f"{script} runs {runner}, outside its directory") + arguments: list[str] = [] + directory: str | None = None + options = iter(tokens[4:]) + for option in options: + if option == "--dir": + directory = next(options, None) + continue + takes_value = OPENVMM_TEST_RUN_TARGET_OPTIONS.get(option) + if takes_value is None: + raise ScriptError(f"{script} passes unsupported option {option}") + arguments.append(option) + if takes_value: + value = next(options, None) + if value is None or value.startswith("--"): + raise ScriptError(f"{script} passes {option} without a value") + arguments.append(value) + if directory != "." or "--target" not in arguments: + raise ScriptError(f"{script} does not run the tests in its own directory") + return runner_path.as_posix(), arguments + + +def _openvmm_test_content_files(directory: Path) -> list[dict[str, object]]: + files: list[dict[str, object]] = [] + paths = { + path.relative_to(directory).as_posix(): path for path in directory.rglob("*") + } + for relative, path in sorted(paths.items()): + metadata = path.lstat() + if stat.S_ISDIR(metadata.st_mode) or relative == OPENVMM_TEST_CONTENT_MANIFEST: + continue + if not stat.S_ISREG(metadata.st_mode): + raise ScriptError(f"OpenVMM test content has a non-regular file {relative}") + files.append( + { + "path": relative, + "size": metadata.st_size, + # Artifact uploads drop file modes, so the consumer restores + # these. + "executable": os.name != "nt" and bool(metadata.st_mode & 0o111), + } + ) + return files + + +def build_openvmm_tests(backends: Sequence[str], output: Path) -> None: + """Build, without running them, the VMM tests of every named backend into + a test content directory that ``run_openvmm_tests`` runs on another host. + + The directory holds what OpenVMM's `vmm-tests-run-target` needs, and a + manifest that binds it to the OpenVMM revision, the Rust release, and the + backends' filters that built it. + """ + selected = _openvmm_test_content_backends(backends) + require_file( + OpenVMMBuildConstants.DIRECTORY / "Cargo.toml", "initialized OpenVMM submodule" + ) + if path_exists(output) and (not output.is_dir() or any(output.iterdir())): + raise ScriptError(f"{output} is not an empty directory") + cargo = require_tool("cargo") + rustup = require_tool("rustup") + revision, clean = openvmm_git_state(OpenVMMBuildConstants.DIRECTORY) + build_filter = openvmm_test_build_filter(selected) + if "'" in build_filter: + raise ScriptError("OpenVMM run scripts cannot quote a filter with a quote") + + environment = _prepare_openvmm_test_environment(selected, rustup) + run_checked( + [cargo, "xflowey", "restore-packages", "--no-compat-igvm"], + cwd=OpenVMMBuildConstants.DIRECTORY, + env=environment, + ) + output.mkdir(parents=True, exist_ok=True) + run_checked( + [ + cargo, + "xflowey", + "vmm-tests-run", + "--release", + "--ci-profile", + "--skip-vhd-prompt", + "--build-only", + "--filter", + build_filter, + "--dir", + os.fspath(output.resolve()), + ], + cwd=OpenVMMBuildConstants.DIRECTORY, + env=environment, + ) + + operating_system = OPENVMM_TEST_OPERATING_SYSTEMS[selected[0]] + runner, arguments = _openvmm_test_run_target( + output / OPENVMM_TEST_RUN_SCRIPTS[operating_system], build_filter + ) + files = _openvmm_test_content_files(output) + manifest = { + "format": OPENVMM_TEST_CONTENT_FORMAT, + "openvmm_revision": revision, + "openvmm_clean": clean, + "rust_toolchain": OpenVMMBuildConstants.RUST_TOOLCHAIN, + "operating_system": operating_system, + "filters": {backend: OPENVMM_TEST_FILTERS[backend] for backend in selected}, + "build_filter": build_filter, + "runner": runner, + "arguments": arguments, + "files": files, + } + (output / OPENVMM_TEST_CONTENT_MANIFEST).write_text( + json.dumps(manifest, indent=2) + "\n", encoding="utf-8" + ) + size = sum(cast(int, entry["size"]) for entry in files) + print( + f"Built the {', '.join(backend.upper() for backend in selected)} OpenVMM " + f"tests into {output}: {len(files)} files, {bytes_to_mib(size):.1f} MiB" + ) + + +def _openvmm_test_content(content: Path, backend: str) -> tuple[Path, list[str]]: + """Check that the content runs this checkout's tests of the backend, and + return its runner and its vmm-tests-run-target options.""" + path = require_file( + content / OPENVMM_TEST_CONTENT_MANIFEST, "OpenVMM test content manifest" + ) + try: + document = cast( + object, + json.loads( + path.read_text(encoding="utf-8"), + object_pairs_hook=strict_json_object, + ), + ) + except (UnicodeDecodeError, json.JSONDecodeError) as error: + raise ScriptError(f"{path} is not valid JSON: {error}") from error + if not isinstance(document, dict): + raise ScriptError(f"{path} is not a JSON object") + manifest = cast(dict[str, object], document) + if manifest.get("format") != OPENVMM_TEST_CONTENT_FORMAT: + raise ScriptError( + f"{path} has format {manifest.get('format')!r}, " + f"expected {OPENVMM_TEST_CONTENT_FORMAT}" + ) + gitlink = run_capture( + ["git", "-C", BuildConstants.REPO_ROOT, "rev-parse", ":openvmm"] + ) + require_success(gitlink, "OpenVMM gitlink query") + revision = gitlink.stdout.decode("ascii").strip() + if manifest.get("openvmm_revision") != revision: + raise ScriptError( + f"{path} describes tests of OpenVMM " + f"{manifest.get('openvmm_revision')}, but this checkout pins {revision}" + ) + filters = manifest.get("filters") + if ( + not isinstance(filters, dict) + or cast(dict[str, object], filters).get(backend) + != OPENVMM_TEST_FILTERS[backend] + ): + raise ScriptError( + f"{path} describes no tests of this checkout's {backend.upper()} filter" + ) + + runner = _content_relative_path(manifest.get("runner")) + arguments = manifest.get("arguments") + files = manifest.get("files") + if ( + runner is None + or not isinstance(arguments, list) + or not all(isinstance(item, str) for item in cast(list[object], arguments)) + or not isinstance(files, list) + ): + raise ScriptError(f"{path} is malformed") + for entry in cast(list[object], files): + details = cast(dict[str, object], entry) if isinstance(entry, dict) else {} + relative = _content_relative_path(details.get("path")) + if relative is None: + raise ScriptError(f"{path} lists a file outside its directory") + file = content.joinpath(*relative.parts) + if not file.is_file() or file.stat().st_size != details.get("size"): + raise ScriptError(f"OpenVMM test content in {content} lacks {relative}") + # Artifact downloads drop the file modes that the build recorded. + if details.get("executable") is True: + file.chmod(file.stat().st_mode | 0o111) + return content.joinpath(*runner.parts), cast(list[str], arguments) + + +def _run_openvmm_test_content(backend: str, content: Path) -> None: + kernel, initrd = _openvmm_microvm_test_guest() + root = content.resolve() + runner, arguments = _openvmm_test_content(root, backend) + + environment = os.environ.copy() + environment["RUSTUP_TOOLCHAIN"] = OpenVMMBuildConstants.RUST_TOOLCHAIN + _set_openvmm_test_cache(environment, (backend,)) + environment["OPENVMM_MICROVM_TEST_KERNEL"] = os.fspath(kernel.resolve()) + environment["OPENVMM_MICROVM_TEST_INITRD"] = os.fspath(initrd.resolve()) + run_checked( + [ + runner, + "pipeline", + "run", + "vmm-tests-run-target", + *arguments, + "--dir", + os.fspath(root), + "--filter", + OPENVMM_TEST_FILTERS[backend], + ], + cwd=root, + env=environment, + ) + + def setup_cross_os_cache() -> None: github_path_value = os.environ.get("GITHUB_PATH") runner_temp_value = os.environ.get("RUNNER_TEMP") diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index d0be35d60..da4e616b7 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -963,8 +963,34 @@ def test_release_commands_keep_their_cli_contract(self): openvmm_tests = nvx.parse_args(["test-openvmm", "--backend", "mshv"]) self.assertEqual(openvmm_tests.backend, "mshv") + self.assertIsNone(openvmm_tests.content) self.assertIs(openvmm_tests.handler, nvx.command_test_openvmm) + openvmm_content_tests = nvx.parse_args( + ["test-openvmm", "--backend", "kvm", "--content", "content"] + ) + self.assertEqual(openvmm_content_tests.content, Path("content")) + + openvmm_test_build = nvx.parse_args( + [ + "build-openvmm-tests", + "--backend", + "kvm", + "--backend", + "mshv", + "--output", + "content", + ] + ) + self.assertEqual(openvmm_test_build.backend, ["kvm", "mshv"]) + self.assertEqual(openvmm_test_build.output, Path("content")) + self.assertIs(openvmm_test_build.handler, nvx.command_build_openvmm_tests) + with ( + patch("sys.stderr", new_callable=io.StringIO), + self.assertRaises(SystemExit), + ): + nvx.parse_args(["build-openvmm-tests", "--output", "content"]) + def test_openvmm_help_describes_petri_vmm_tests(self): with ( patch("sys.stdout", new_callable=io.StringIO) as output, @@ -3410,7 +3436,7 @@ def test_openvmm_tests_pin_toolchain_without_runner_temp(self): patch.object(ci, "run_checked") as run_checked, patch.dict(os.environ, {}, clear=True), ): - environment = ci._prepare_openvmm_test_environment("whp", "rustup") + environment = ci._prepare_openvmm_test_environment(("whp",), "rustup") expected_environment = { "RUSTUP_TOOLCHAIN": OpenVMMBuildConstants.RUST_TOOLCHAIN, @@ -3536,7 +3562,9 @@ def test_openvmm_tests_install_missing_pinned_toolchain(self): clear=True, ), ): - environment = ci._prepare_openvmm_test_environment("kvm", "rustup") + environment = ci._prepare_openvmm_test_environment( + ("kvm",), "rustup" + ) self.assertEqual( [command.args[0] for command in run_checked.call_args_list], @@ -3706,6 +3734,512 @@ def test_openvmm_tests_reject_unknown_backend(self): with self.assertRaisesRegex(common.ScriptError, "unsupported.*backend"): ci.run_openvmm_tests("unknown") + def test_openvmm_test_build_filter_selects_every_backends_tests(self): + kvm = ci.OPENVMM_TEST_FILTERS["kvm"] + mshv = ci.OPENVMM_TEST_FILTERS["mshv"] + # One Linux build serves both Linux backends: a test of either + # backend's filter matches the union, which nextest reads as "or". + self.assertEqual( + ci.openvmm_test_build_filter(("mshv", "kvm")), + f"({kvm}) | ({mshv})", + ) + for backend, test_filter in ci.OPENVMM_TEST_FILTERS.items(): + with self.subTest(backend=backend): + self.assertEqual(ci.openvmm_test_build_filter((backend,)), test_filter) + # The run script quotes the filter without escaping quotes. + self.assertNotIn("'", test_filter) + self.assertEqual( + set(ci.OPENVMM_TEST_OPERATING_SYSTEMS), set(ci.OPENVMM_TEST_BACKENDS) + ) + + @staticmethod + def _openvmm_run_script(operating_system: str, test_filter: str) -> str: + """Return the run script that the pinned flowey writes for a build.""" + if operating_system == "windows": + prefix = "cd $PSScriptRoot ; .\\x86_64-pc-windows-msvc\\flowey_hvlite.exe" + target = "windows-x64" + else: + prefix = ( + 'cd "$(dirname "${BASH_SOURCE[0]}")" ; ' + "./x86_64-unknown-linux-gnu/flowey_hvlite" + ) + target = "linux-x64" + return ( + f"{prefix} pipeline run vmm-tests-run-target --target {target} " + f"--dir . --filter '{test_filter}' --repetitions 1 " + "--artifacts VMGS_WITH_BOOT_ENTRY --skip-vhd-prompt --ci-profile" + ) + + def test_openvmm_test_content_paths_stay_inside_the_directory(self): + for value in ("x86_64/vmlinux", ".config/nextest.toml", "run.sh"): + with self.subTest(value=value): + self.assertEqual(ci._content_relative_path(value), PurePosixPath(value)) + # A backslash or a drive would name another path on Windows. + for value in ( + None, + 1, + "", + ".", + "/escape", + "../escape", + "target/../../escape", + "..\\escape", + "target\\..\\..\\escape", + "C:\\escape", + "C:escape", + "C:/escape", + "\\\\server\\share\\escape", + "target/file:stream", + ): + with self.subTest(value=value): + self.assertIsNone(ci._content_relative_path(value)) + + def test_openvmm_test_run_scripts_name_the_runner_and_its_options(self): + test_filter = ci.openvmm_test_build_filter(("kvm", "mshv")) + options = [ + "--target", + "linux-x64", + "--repetitions", + "1", + "--artifacts", + "VMGS_WITH_BOOT_ENTRY", + "--skip-vhd-prompt", + "--ci-profile", + ] + with tempfile.TemporaryDirectory() as temporary: + script = Path(temporary) / "run" + for operating_system, runner, target in ( + ("linux", "x86_64-unknown-linux-gnu/flowey_hvlite", "linux-x64"), + ( + "windows", + "x86_64-pc-windows-msvc/flowey_hvlite.exe", + "windows-x64", + ), + ): + with self.subTest(operating_system=operating_system): + script.write_text( + self._openvmm_run_script(operating_system, test_filter) + "\n", + encoding="utf-8", + ) + self.assertEqual( + ci._openvmm_test_run_target(script, test_filter), + (runner, ["--target", target, *options[2:]]), + ) + + valid = self._openvmm_run_script("linux", test_filter) + for name, text, message in ( + ("another filter", valid, "does not run the filter"), + ("no cd", valid.split(" ; ", 1)[1], "does not run the filter"), + ( + "another pipeline", + valid.replace("vmm-tests-run-target", "vmm-tests-run"), + "does not run vmm-tests-run-target", + ), + ( + "absolute runner", + valid.replace("./x86_64", "/x86_64"), + "outside its directory", + ), + ( + "parent runner", + valid.replace("./x86_64", "./../x86_64"), + "outside its directory", + ), + ( + "Windows parent runner", + self._openvmm_run_script("windows", test_filter).replace( + ".\\x86_64", ".\\..\\x86_64" + ), + "outside its directory", + ), + ( + "drive runner", + self._openvmm_run_script("windows", test_filter).replace( + ".\\x86_64", ".\\C:\\x86_64" + ), + "outside its directory", + ), + ( + "unknown option", + valid + " --install-missing-deps", + "unsupported option --install-missing-deps", + ), + ( + "missing value", + valid + " --repetitions", + "--repetitions without a value", + ), + ( + "another directory", + valid.replace("--dir .", "--dir /tmp"), + "does not run the tests in its own directory", + ), + ( + "no target", + valid.replace("--target linux-x64 ", ""), + "does not run the tests in its own directory", + ), + ): + with self.subTest(case=name): + script.write_text(text, encoding="utf-8") + expected_filter = ( + ci.OPENVMM_TEST_FILTERS["kvm"] + if name == "another filter" + else test_filter + ) + with self.assertRaisesRegex(common.ScriptError, message): + ci._openvmm_test_run_target(script, expected_filter) + + def test_build_openvmm_tests_builds_only_and_describes_the_content(self): + backends = ("whp",) if os.name == "nt" else ("mshv", "kvm") + operating_system = "windows" if os.name == "nt" else "linux" + build_filter = ci.openvmm_test_build_filter(backends) + targets = { + target + for backend in backends + for target in OpenVMMBuildConstants.TEST_RUST_TARGETS[backend] + } + installed_targets = common.CommandResult( + args=("rustup", "target", "list"), + returncode=0, + stdout="".join(f"{target}\n" for target in sorted(targets)).encode(), + stderr=b"", + ) + runner = ( + "x86_64-pc-windows-msvc/flowey_hvlite.exe" + if os.name == "nt" + else "x86_64-unknown-linux-gnu/flowey_hvlite" + ) + + def build(command: list[str], **_: object) -> None: + if "--build-only" not in command: + return + content = Path(command[command.index("--dir") + 1]) + files = { + runner: b"runner", + "x86_64/vmlinux": b"kernel", + ".config/nextest.toml": b"[profile.ci]\n", + } + for relative, data in files.items(): + path = content / relative + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(data) + path.chmod(0o755 if relative != ".config/nextest.toml" else 0o644) + script = content / ci.OPENVMM_TEST_RUN_SCRIPTS[operating_system] + script.write_text( + self._openvmm_run_script(operating_system, build_filter), + encoding="utf-8", + ) + script.chmod(0o755) + + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + openvmm = root / "openvmm" + openvmm.mkdir() + (openvmm / "Cargo.toml").touch() + output = root / "content" + revision = "a" * 40 + + with ( + patch.object(OpenVMMBuildConstants, "DIRECTORY", openvmm), + patch.object(ci, "require_tool", side_effect=["cargo", "rustup"]), + patch.object(ci, "run_capture", return_value=installed_targets), + patch.object( + ci, "openvmm_git_state", return_value=(revision, True) + ) as git_state, + patch.object(ci, "run_checked", side_effect=build) as run_checked, + patch.dict(os.environ, {"RUNNER_TEMP": os.fspath(root)}), + patch("sys.stdout", new_callable=io.StringIO) as stdout, + ): + ci.build_openvmm_tests(backends, output) + + git_state.assert_called_once_with(openvmm) + restore, tests = run_checked.call_args_list + self.assertEqual( + restore.args[0], + ["cargo", "xflowey", "restore-packages", "--no-compat-igvm"], + ) + self.assertEqual( + tests.args[0], + [ + "cargo", + "xflowey", + "vmm-tests-run", + "--release", + "--ci-profile", + "--skip-vhd-prompt", + "--build-only", + "--filter", + build_filter, + "--dir", + os.fspath(output.resolve()), + ], + ) + for command in (restore, tests): + self.assertEqual(command.kwargs["cwd"], openvmm) + self.assertEqual( + command.kwargs["env"]["RUSTUP_TOOLCHAIN"], + OpenVMMBuildConstants.RUST_TOOLCHAIN, + ) + # Windows links the build reproducibly for sccache (#693). + if os.name == "nt": + self.assertIn( + ci.MSVC_REPRODUCIBLE_LINK_OPTION, + command.kwargs["env"]["_LINK_"].split(), + ) + # The build boots no NVX guest, so it needs no guest artifacts. + self.assertNotIn("OPENVMM_MICROVM_TEST_KERNEL", tests.kwargs["env"]) + + manifest = json.loads( + (output / ci.OPENVMM_TEST_CONTENT_MANIFEST).read_text(encoding="utf-8") + ) + script = ci.OPENVMM_TEST_RUN_SCRIPTS[operating_system] + executable = os.name != "nt" + self.assertEqual( + manifest, + { + "format": 1, + "openvmm_revision": revision, + "openvmm_clean": True, + "rust_toolchain": OpenVMMBuildConstants.RUST_TOOLCHAIN, + "operating_system": operating_system, + "filters": { + backend: ci.OPENVMM_TEST_FILTERS[backend] + for backend in sorted(backends) + }, + "build_filter": build_filter, + "runner": runner, + "arguments": [ + "--target", + "windows-x64" if os.name == "nt" else "linux-x64", + "--repetitions", + "1", + "--artifacts", + "VMGS_WITH_BOOT_ENTRY", + "--skip-vhd-prompt", + "--ci-profile", + ], + "files": sorted( + ( + { + "path": ".config/nextest.toml", + "size": 13, + "executable": False, + }, + { + "path": script, + "size": (output / script).stat().st_size, + "executable": executable, + }, + {"path": runner, "size": 6, "executable": executable}, + { + "path": "x86_64/vmlinux", + "size": 6, + "executable": executable, + }, + ), + key=lambda entry: str(entry["path"]), + ), + }, + ) + self.assertIn("4 files", stdout.getvalue()) + + def test_build_openvmm_tests_refuse_other_hosts_and_used_directories(self): + other = ("kvm",) if os.name == "nt" else ("whp",) + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + openvmm = root / "openvmm" + openvmm.mkdir() + (openvmm / "Cargo.toml").touch() + used = root / "used" + used.mkdir() + (used / "stale").touch() + native = ("whp",) if os.name == "nt" else ("kvm",) + for backends, output, message in ( + ((), root / "content", "at least one"), + (("unknown",), root / "content", "unsupported OpenVMM test backend"), + (other, root / "content", "tests build on"), + (native, used, "not an empty directory"), + (native, used / "stale", "not an empty directory"), + ): + with ( + self.subTest(backends=backends, output=output.name), + patch.object(OpenVMMBuildConstants, "DIRECTORY", openvmm), + patch.object(ci, "run_checked") as run_checked, + self.assertRaisesRegex(common.ScriptError, message), + ): + ci.build_openvmm_tests(backends, output) + run_checked.assert_not_called() + + def _openvmm_test_content( + self, content: Path, backend: str, revision: str + ) -> tuple[Path, list[str]]: + runner = content / "target" / "flowey_hvlite" + runner.parent.mkdir(parents=True) + runner.write_bytes(b"runner") + runner.chmod(0o644) + arguments = ["--target", "linux-x64", "--ci-profile"] + manifest = { + "format": 1, + "openvmm_revision": revision, + "openvmm_clean": True, + "rust_toolchain": OpenVMMBuildConstants.RUST_TOOLCHAIN, + "operating_system": ci.OPENVMM_TEST_OPERATING_SYSTEMS[backend], + "filters": {backend: ci.OPENVMM_TEST_FILTERS[backend]}, + "build_filter": ci.OPENVMM_TEST_FILTERS[backend], + "runner": "target/flowey_hvlite", + "arguments": arguments, + "files": [{"path": "target/flowey_hvlite", "size": 6, "executable": True}], + } + (content / ci.OPENVMM_TEST_CONTENT_MANIFEST).write_text( + json.dumps(manifest), encoding="utf-8" + ) + return runner, arguments + + def test_openvmm_tests_run_built_content_with_its_runner(self): + backend = "whp" if os.name == "nt" else "kvm" + revision = "b" * 40 + gitlink = common.CommandResult( + args=("git", "rev-parse", ":openvmm"), + returncode=0, + stdout=f"{revision}\n".encode(), + stderr=b"", + ) + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + content = root / "content" + content.mkdir() + runner, arguments = self._openvmm_test_content(content, backend, revision) + kernel = root / "vmlinux" + initrd = root / "initramfs.cpio.gz" + kernel.touch() + initrd.touch() + + with ( + patch.object(ci, "artifact_path", side_effect=[kernel, initrd]), + patch.object(ci.os, "access", return_value=True), + patch.object(ci.Path, "exists", return_value=False), + patch.object(ci, "run_capture", return_value=gitlink) as run_capture, + patch.object(ci, "run_checked") as run_checked, + patch.dict(os.environ, {"RUNNER_TEMP": os.fspath(root)}), + ): + ci.run_openvmm_tests(backend, content) + + run_capture.assert_called_once_with( + ["git", "-C", BuildConstants.REPO_ROOT, "rev-parse", ":openvmm"] + ) + run_checked.assert_called_once() + command = run_checked.call_args.args[0] + resolved = content.resolve() + self.assertEqual( + command, + [ + resolved / "target" / "flowey_hvlite", + "pipeline", + "run", + "vmm-tests-run-target", + *arguments, + "--dir", + os.fspath(resolved), + "--filter", + ci.OPENVMM_TEST_FILTERS[backend], + ], + ) + self.assertEqual(run_checked.call_args.kwargs["cwd"], resolved) + environment = run_checked.call_args.kwargs["env"] + self.assertEqual( + environment["RUSTUP_TOOLCHAIN"], OpenVMMBuildConstants.RUST_TOOLCHAIN + ) + self.assertEqual( + environment["OPENVMM_MICROVM_TEST_KERNEL"], os.fspath(kernel.resolve()) + ) + self.assertEqual( + environment["OPENVMM_MICROVM_TEST_INITRD"], os.fspath(initrd.resolve()) + ) + if os.name == "nt": + self.assertNotIn("XDG_CACHE_HOME", environment) + else: + self.assertEqual( + environment["XDG_CACHE_HOME"], os.fspath(root / "openvmm-cache") + ) + # The artifact download dropped the runner's executable bit. + self.assertTrue(runner.stat().st_mode & 0o100) + + def test_openvmm_tests_refuse_content_that_is_not_this_checkouts(self): + backend = "whp" if os.name == "nt" else "kvm" + other_backend = "kvm" if os.name == "nt" else "mshv" + revision = "c" * 40 + gitlink = common.CommandResult( + args=("git", "rev-parse", ":openvmm"), + returncode=0, + stdout=f"{revision}\n".encode(), + stderr=b"", + ) + + def edit(manifest: dict[str, object], change: str, content: Path) -> None: + if change == "format": + manifest["format"] = 2 + elif change == "revision": + manifest["openvmm_revision"] = "d" * 40 + elif change == "backend": + manifest["filters"] = { + other_backend: ci.OPENVMM_TEST_FILTERS[other_backend] + } + elif change == "filter": + manifest["filters"] = {backend: "all()"} + elif change == "runner": + manifest["runner"] = "../flowey_hvlite" + elif change == "arguments": + manifest["arguments"] = ["--target", 1] + elif change == "outside": + manifest["files"] = [{"path": "../escape", "size": 0}] + elif change == "windows outside": + manifest["files"] = [{"path": "..\\escape", "size": 0}] + elif change == "drive": + manifest["files"] = [{"path": "C:escape", "size": 0}] + elif change == "missing": + (content / "target" / "flowey_hvlite").unlink() + elif change == "truncated": + (content / "target" / "flowey_hvlite").write_bytes(b"run") + + for change, message in ( + ("format", "has format 2, expected 1"), + ("revision", f"of OpenVMM d{{40}}, but this checkout pins {revision}"), + ("backend", "no tests of this checkout's"), + ("filter", "no tests of this checkout's"), + ("runner", "is malformed"), + ("arguments", "is malformed"), + ("outside", "lists a file outside its directory"), + ("windows outside", "lists a file outside its directory"), + ("drive", "lists a file outside its directory"), + ("missing", "lacks target/flowey_hvlite"), + ("truncated", "lacks target/flowey_hvlite"), + ): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + content = root / "content" + content.mkdir() + self._openvmm_test_content(content, backend, revision) + manifest_path = content / ci.OPENVMM_TEST_CONTENT_MANIFEST + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + edit(manifest, change, content) + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + kernel = root / "vmlinux" + initrd = root / "initramfs.cpio.gz" + kernel.touch() + initrd.touch() + with ( + self.subTest(change=change), + patch.object(ci, "artifact_path", side_effect=[kernel, initrd]), + patch.object(ci.os, "access", return_value=True), + patch.object(ci.Path, "exists", return_value=False), + patch.object(ci, "run_capture", return_value=gitlink), + patch.object(ci, "run_checked") as run_checked, + self.assertRaisesRegex(common.ScriptError, message), + ): + ci.run_openvmm_tests(backend, content) + run_checked.assert_not_called() + class CiConfigurationTests(unittest.TestCase): def test_ci_needs_reference_declared_jobs(self): From 07efc1873d8347df495e1d43e6204a43d23c869f Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 22 Oct 2026 08:22:46 -0300 Subject: [PATCH 2/2] ci: build the Linux OpenVMM VMM tests once for the KVM and MSHV jobs Each Linux leg of openvmm-vmm-tests built the same x86_64-unknown-linux-gnu tests before it ran its backend's filter, and every leg waited for the guest artifacts before it started its build. In reference run 39585909588, the KVM and MSHV legs each spent about five minutes on that build, and their logs resolve the same artifacts and lay out the same test content. build-openvmm-vmm-tests-linux now builds the KVM and MSHV tests once, with build-openvmm-tests, on a general KVM runner, whose sccache holds that build, and hands the content to the two openvmm-vmm-tests-linux legs as the one-day openvmm-vmm-tests-linux artifact. The build boots no guest, so it starts with the OpenVMM binary producers instead of waiting for the guest artifacts, and it takes over the vmm-tests input and Cargo caches. Each leg downloads the content, the guest kernel, and the Alpine initramfs and runs its backend's own filter with test-openvmm --content, without checking out or building OpenVMM, and it uploads its diagnostics under the same openvmm-vmm-tests---- name, now from the content's test_results. The upload keeps the hidden .config/nextest.toml and replaces an earlier attempt's content, so a rerun of the build replaces it and a rerun of a failed leg reuses it. The WHP tests stay in openvmm-vmm-tests-windows, which still builds and runs them in one job: on Windows, the pinned vmm-tests-run-target requires Hyper-V whatever tests it runs, through Get-VM and two registry values or administrator rights, and the WHP runners run jobs as Network Service. A Windows build job can follow once vmm-tests-run-target checks only what the tests of its build need. The build runs exactly when its test jobs do: whenever the change selects the VMM tests, since the classifier then also selects the guest artifacts that they boot, or when the change detector fails. It therefore reads run-openvmm-vmm-tests alone. check-required-ci expects it with them, and the development release and performance persistence, which accepted any skipped VMM-test job, now require the three jobs to succeed, or all three to be skipped because openvmm-changes selected no VMM tests, so that a failed or unexpectedly skipped build or test blocks them. Fork, draft, and path-filter gating, cancellation, and the cache saves stay as they were. Tests cover the build and test jobs, their runners, needs, and conditions, the change detector's outputs that they and the two gates read, the guest artifact components that each downloads, reads, and waits for, the artifact's name, path, hidden files, overwrite, and retention, the order of the downloads and the tests, the diagnostics of both operating systems, the required-check policy, the release and persistence gate, the cache keys, and the derivation of the VMM tests' inputs from every job that builds or runs them. doc/ci.md describes the build, its consumers, and the handoff. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/ci.yml | 376 +++++++++++++++++++++++++----------- doc/ci.md | 79 ++++++-- scripts/nvx_tools/ci.py | 22 ++- scripts/test_nvx_tools.py | 389 ++++++++++++++++++++++++++++++-------- 4 files changed, 669 insertions(+), 197 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5d95d09d4..c4783d3a4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -620,9 +620,152 @@ jobs: secrets: OPENVMM_DEPLOY_KEY: ${{ secrets.OPENVMM_DEPLOY_KEY }} - openvmm-vmm-tests: + # The KVM and MSHV VMM tests run the same Linux build, which boots no guest, + # so one job builds it beside the guest artifacts, on a KVM runner whose + # sccache holds it, and hands its test content to both backends' jobs. + build-openvmm-vmm-tests-linux: + name: OpenVMM vmm-tests build / Linux + needs: [openvmm-changes] + if: >- + ${{ + always() && + !cancelled() && + ( + github.event_name != 'pull_request' || + github.event.pull_request.draft == false + ) && + ( + github.event_name != 'pull_request' || + github.event.pull_request.head.repo.full_name == github.repository + ) && + ( + needs.openvmm-changes.result != 'success' || + needs.openvmm-changes.outputs.run-openvmm-vmm-tests == 'true' + ) + }} + runs-on: [self-hosted, linux, kvm, virtual-machine, general] + timeout-minutes: 90 + steps: + - name: Checkout + uses: actions/checkout@v5 + with: + persist-credentials: false + + - name: Validate runner + uses: ./.github/actions/validate-runner + with: + backend: kvm + + - name: Checkout OpenVMM + id: openvmm + uses: ./.github/actions/checkout-openvmm + with: + ssh-key: ${{ secrets.OPENVMM_DEPLOY_KEY }} + + - name: Set up retrying curl + uses: ./.github/actions/setup-curl + + - name: Prepare Linux cache tools + id: cache-tools-linux + shell: bash + run: | + set -euo pipefail + cache_tools="${RUNNER_TEMP}/nvx-gzip-cache-tools" + mkdir -p "${cache_tools}" + ln -sf "$(command -v tar)" "${cache_tools}/tar" + ln -sf "$(command -v gzip)" "${cache_tools}/gzip" + echo "path=${cache_tools}" >> "${GITHUB_OUTPUT}" + + - name: Restore OpenVMM inputs on Linux + id: restore-openvmm-inputs-linux + uses: actions/cache/restore@v5 + env: + PATH: ${{ steps.cache-tools-linux.outputs.path }} + with: + path: | + openvmm/flowey-persist/flowey_lib_common__download_gh_release + openvmm/flowey-persist/flowey_lib_common__cache + key: openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}-vmm-tests-${{ steps.openvmm.outputs.sha }} + restore-keys: | + openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}- + + - name: Restore shared Cargo cache on Linux + id: restore-openvmm-cargo-linux + uses: actions/cache/restore@v5 + env: + PATH: ${{ steps.cache-tools-linux.outputs.path }} + with: + path: | + ${{ format('{0}/cargo-home/registry', runner.temp) }} + ${{ format('{0}/cargo-home/git', runner.temp) }} + key: openvmm-cargo-v1-${{ runner.os }}-${{ runner.arch }}-vmm-tests-${{ hashFiles('openvmm/Cargo.lock') }} + restore-keys: | + openvmm-cargo-v1-${{ runner.os }}-${{ runner.arch }}- + + - name: Prepare sccache + uses: ./.github/actions/sccache + with: + operation: start + + - name: Build OpenVMM tests on Linux + shell: bash + env: + GH_TOKEN: ${{ github.token }} + run: >- + python3 scripts/nvx.py build-openvmm-tests + --backend kvm --backend mshv + --output "${RUNNER_TEMP}/openvmm-vmm-tests-linux" + + - name: Report sccache + if: always() + uses: ./.github/actions/sccache + with: + operation: stop + + # include-hidden-files keeps the content's .config/nextest.toml. + - name: Upload OpenVMM tests + uses: actions/upload-artifact@v7 + with: + name: openvmm-vmm-tests-linux + path: ${{ runner.temp }}/openvmm-vmm-tests-linux + include-hidden-files: true + if-no-files-found: error + overwrite: true + retention-days: 1 + + - name: Save OpenVMM inputs on Linux + if: >- + success() && + github.event_name == 'push' && + github.ref == 'refs/heads/dev' && + steps.restore-openvmm-inputs-linux.outputs.cache-hit != 'true' + uses: actions/cache/save@v5 + env: + PATH: ${{ steps.cache-tools-linux.outputs.path }} + with: + path: | + openvmm/flowey-persist/flowey_lib_common__download_gh_release + openvmm/flowey-persist/flowey_lib_common__cache + key: openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}-vmm-tests-${{ steps.openvmm.outputs.sha }} + + - name: Save shared Cargo cache on Linux + if: >- + success() && + github.event_name == 'push' && + github.ref == 'refs/heads/dev' && + steps.restore-openvmm-cargo-linux.outputs.cache-hit != 'true' + uses: actions/cache/save@v5 + env: + PATH: ${{ steps.cache-tools-linux.outputs.path }} + with: + path: | + ${{ format('{0}/cargo-home/registry', runner.temp) }} + ${{ format('{0}/cargo-home/git', runner.temp) }} + key: openvmm-cargo-v1-${{ runner.os }}-${{ runner.arch }}-vmm-tests-${{ hashFiles('openvmm/Cargo.lock') }} + + openvmm-vmm-tests-linux: name: OpenVMM vmm-tests / ${{ matrix.name }} - needs: [artifacts, openvmm-changes] + needs: [artifacts, build-openvmm-vmm-tests-linux, openvmm-changes] if: >- ${{ always() && @@ -636,6 +779,7 @@ jobs: github.event.pull_request.head.repo.full_name == github.repository ) && needs.artifacts.result == 'success' && + needs.build-openvmm-vmm-tests-linux.result == 'success' && ( needs.openvmm-changes.result != 'success' || needs.openvmm-changes.outputs.run-openvmm-vmm-tests == 'true' @@ -653,9 +797,6 @@ jobs: - name: Linux / MSHV backend: mshv runner: '["self-hosted", "linux", "mshv", "virtual-machine", "general"]' - - name: Windows / WHP - backend: whp - runner: '["self-hosted", "windows", "whp", "virtual-machine", "general"]' steps: - name: Checkout uses: actions/checkout@v5 @@ -667,6 +808,83 @@ jobs: with: backend: ${{ matrix.backend }} + - name: Set up retrying curl + uses: ./.github/actions/setup-curl + + # The VMM tests boot only the kernel and the Alpine initramfs. + - name: Download guest kernel + uses: actions/download-artifact@v8 + with: + name: guest-kernel + path: build + + - name: Download Alpine initramfs + uses: actions/download-artifact@v8 + with: + name: guest-alpine + path: build + + - name: Download OpenVMM tests + uses: actions/download-artifact@v8 + with: + name: openvmm-vmm-tests-linux + path: ${{ runner.temp }}/openvmm-vmm-tests + + - name: Run OpenVMM tests on Linux + shell: bash + env: + GH_TOKEN: ${{ github.token }} + run: >- + python3 scripts/nvx.py test-openvmm + --backend "${{ matrix.backend }}" + --content "${RUNNER_TEMP}/openvmm-vmm-tests" + + - name: Upload OpenVMM test diagnostics + if: failure() + uses: actions/upload-artifact@v7 + with: + name: openvmm-vmm-tests-${{ runner.os }}-${{ matrix.backend }}-${{ github.run_id }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/openvmm-vmm-tests/test_results + if-no-files-found: warn + retention-days: 7 + + # The pinned OpenVMM's vmm-tests-run-target checks for Hyper-V on Windows + # whatever tests it runs, which the WHP runners' Network Service account + # cannot pass, so this job still builds the WHP tests and runs them. + openvmm-vmm-tests-windows: + name: OpenVMM vmm-tests / Windows / WHP + needs: [artifacts, openvmm-changes] + if: >- + ${{ + always() && + !cancelled() && + ( + github.event_name != 'pull_request' || + github.event.pull_request.draft == false + ) && + ( + github.event_name != 'pull_request' || + github.event.pull_request.head.repo.full_name == github.repository + ) && + needs.artifacts.result == 'success' && + ( + needs.openvmm-changes.result != 'success' || + needs.openvmm-changes.outputs.run-openvmm-vmm-tests == 'true' + ) + }} + runs-on: [self-hosted, windows, whp, virtual-machine, general] + timeout-minutes: 90 + steps: + - name: Checkout + uses: actions/checkout@v5 + with: + persist-credentials: false + + - name: Validate runner + uses: ./.github/actions/validate-runner + with: + backend: whp + - name: Checkout OpenVMM id: openvmm uses: ./.github/actions/checkout-openvmm @@ -690,39 +908,11 @@ jobs: path: build - name: Prepare Windows cache tools - if: runner.os == 'Windows' shell: powershell run: python scripts\nvx.py setup-cross-os-cache - - name: Prepare Linux cache tools - id: cache-tools-linux - if: runner.os != 'Windows' - shell: bash - run: | - set -euo pipefail - cache_tools="${RUNNER_TEMP}/nvx-gzip-cache-tools" - mkdir -p "${cache_tools}" - ln -sf "$(command -v tar)" "${cache_tools}/tar" - ln -sf "$(command -v gzip)" "${cache_tools}/gzip" - echo "path=${cache_tools}" >> "${GITHUB_OUTPUT}" - - - name: Restore OpenVMM inputs on Linux - id: restore-openvmm-inputs-linux - if: runner.os != 'Windows' - uses: actions/cache/restore@v5 - env: - PATH: ${{ steps.cache-tools-linux.outputs.path }} - with: - path: | - openvmm/flowey-persist/flowey_lib_common__download_gh_release - openvmm/flowey-persist/flowey_lib_common__cache - key: openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}-vmm-tests-${{ steps.openvmm.outputs.sha }} - restore-keys: | - openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}- - - name: Restore OpenVMM inputs on Windows id: restore-openvmm-inputs-windows - if: runner.os == 'Windows' uses: actions/cache/restore@v5 with: path: | @@ -732,23 +922,8 @@ jobs: restore-keys: | openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}- - - name: Restore shared Cargo cache on Linux - id: restore-openvmm-cargo-linux - if: runner.os != 'Windows' - uses: actions/cache/restore@v5 - env: - PATH: ${{ steps.cache-tools-linux.outputs.path }} - with: - path: | - ${{ format('{0}/cargo-home/registry', runner.temp) }} - ${{ format('{0}/cargo-home/git', runner.temp) }} - key: openvmm-cargo-v1-${{ runner.os }}-${{ runner.arch }}-vmm-tests-${{ hashFiles('openvmm/Cargo.lock') }} - restore-keys: | - openvmm-cargo-v1-${{ runner.os }}-${{ runner.arch }}- - - name: Restore shared Cargo cache on Windows id: restore-openvmm-cargo-windows - if: runner.os == 'Windows' uses: actions/cache/restore@v5 with: path: | @@ -763,33 +938,18 @@ jobs: with: operation: start - - name: Run OpenVMM tests on KVM - if: matrix.backend == 'kvm' - shell: bash - env: - GH_TOKEN: ${{ github.token }} - run: python3 scripts/nvx.py test-openvmm --backend "${{ matrix.backend }}" - - - name: Run OpenVMM tests on MSHV - if: matrix.backend == 'mshv' - shell: bash - env: - GH_TOKEN: ${{ github.token }} - run: python3 scripts/nvx.py test-openvmm --backend "${{ matrix.backend }}" - - name: Run OpenVMM tests on Windows - if: runner.os == 'Windows' shell: powershell env: GH_TOKEN: ${{ github.token }} - run: python scripts\nvx.py test-openvmm --backend "${{ matrix.backend }}" + run: python scripts\nvx.py test-openvmm --backend whp - name: Upload OpenVMM test diagnostics if: failure() uses: actions/upload-artifact@v7 with: - name: openvmm-vmm-tests-${{ runner.os }}-${{ matrix.backend }}-${{ github.run_id }}-${{ github.run_attempt }} - path: ${{ runner.os == 'Windows' && format('{0}/{1}/test_results', runner.temp, matrix.backend) || 'openvmm/target/vmm_tests/test_results' }} + name: openvmm-vmm-tests-${{ runner.os }}-whp-${{ github.run_id }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/whp/test_results if-no-files-found: warn retention-days: 7 @@ -799,28 +959,11 @@ jobs: with: operation: stop - - name: Save OpenVMM inputs on Linux - if: >- - success() && - github.event_name == 'push' && - github.ref == 'refs/heads/dev' && - runner.os != 'Windows' && - steps.restore-openvmm-inputs-linux.outputs.cache-hit != 'true' - uses: actions/cache/save@v5 - env: - PATH: ${{ steps.cache-tools-linux.outputs.path }} - with: - path: | - openvmm/flowey-persist/flowey_lib_common__download_gh_release - openvmm/flowey-persist/flowey_lib_common__cache - key: openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}-vmm-tests-${{ steps.openvmm.outputs.sha }} - - name: Save OpenVMM inputs on Windows if: >- success() && github.event_name == 'push' && github.ref == 'refs/heads/dev' && - runner.os == 'Windows' && steps.restore-openvmm-inputs-windows.outputs.cache-hit != 'true' uses: actions/cache/save@v5 with: @@ -829,28 +972,11 @@ jobs: openvmm/flowey-persist/flowey_lib_common__cache key: openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}-vmm-tests-${{ steps.openvmm.outputs.sha }} - - name: Save shared Cargo cache on Linux - if: >- - success() && - github.event_name == 'push' && - github.ref == 'refs/heads/dev' && - runner.os != 'Windows' && - steps.restore-openvmm-cargo-linux.outputs.cache-hit != 'true' - uses: actions/cache/save@v5 - env: - PATH: ${{ steps.cache-tools-linux.outputs.path }} - with: - path: | - ${{ format('{0}/cargo-home/registry', runner.temp) }} - ${{ format('{0}/cargo-home/git', runner.temp) }} - key: openvmm-cargo-v1-${{ runner.os }}-${{ runner.arch }}-vmm-tests-${{ hashFiles('openvmm/Cargo.lock') }} - - name: Save shared Cargo cache on Windows if: >- success() && github.event_name == 'push' && github.ref == 'refs/heads/dev' && - runner.os == 'Windows' && steps.restore-openvmm-cargo-windows.outputs.cache-hit != 'true' uses: actions/cache/save@v5 with: @@ -1447,7 +1573,10 @@ jobs: - nvx-rs - nvx-cli-linux - nvx-cli-windows - - openvmm-vmm-tests + - openvmm-changes + - build-openvmm-vmm-tests-linux + - openvmm-vmm-tests-linux + - openvmm-vmm-tests-windows - openvmm-unit-tests - nvx-microvm-tests-kvm - nvx-microvm-tests-mshv @@ -1458,6 +1587,8 @@ jobs: - platform-kvm - platform-mshv - platform-whp + # The VMM-test jobs must all succeed, or all be skipped because the change + # selects no VMM tests: a failed or unexpectedly skipped one blocks this. if: >- ${{ always() && @@ -1471,8 +1602,18 @@ jobs: needs.nvx-cli-linux.result == 'success' && needs.nvx-cli-windows.result == 'success' && ( - needs.openvmm-vmm-tests.result == 'success' || - needs.openvmm-vmm-tests.result == 'skipped' + ( + needs.build-openvmm-vmm-tests-linux.result == 'success' && + needs.openvmm-vmm-tests-linux.result == 'success' && + needs.openvmm-vmm-tests-windows.result == 'success' + ) || + ( + needs.openvmm-changes.result == 'success' && + needs.openvmm-changes.outputs.run-openvmm-vmm-tests != 'true' && + needs.build-openvmm-vmm-tests-linux.result == 'skipped' && + needs.openvmm-vmm-tests-linux.result == 'skipped' && + needs.openvmm-vmm-tests-windows.result == 'skipped' + ) ) && ( needs.openvmm-unit-tests.result == 'success' || @@ -1564,7 +1705,9 @@ jobs: - build-openvmm-linux-gnu - build-openvmm-linux-musl - build-openvmm-windows-msvc - - openvmm-vmm-tests + - build-openvmm-vmm-tests-linux + - openvmm-vmm-tests-linux + - openvmm-vmm-tests-windows - openvmm-unit-tests - nvx-microvm-tests-kvm - nvx-microvm-tests-mshv @@ -1612,7 +1755,9 @@ jobs: BUILD_LINUX_GNU_RESULT: ${{ needs.build-openvmm-linux-gnu.result }} BUILD_LINUX_MUSL_RESULT: ${{ needs.build-openvmm-linux-musl.result }} BUILD_WINDOWS_MSVC_RESULT: ${{ needs.build-openvmm-windows-msvc.result }} - VMM_TESTS_RESULT: ${{ needs.openvmm-vmm-tests.result }} + BUILD_VMM_TESTS_LINUX_RESULT: ${{ needs.build-openvmm-vmm-tests-linux.result }} + VMM_TESTS_LINUX_RESULT: ${{ needs.openvmm-vmm-tests-linux.result }} + VMM_TESTS_WINDOWS_RESULT: ${{ needs.openvmm-vmm-tests-windows.result }} UNIT_TESTS_RESULT: ${{ needs.openvmm-unit-tests.result }} MICROVM_KVM_RESULT: ${{ needs.nvx-microvm-tests-kvm.result }} MICROVM_MSHV_RESULT: ${{ needs.nvx-microvm-tests-mshv.result }} @@ -1641,7 +1786,10 @@ jobs: - debug-kernel - nvx-cli-linux - nvx-cli-windows - - openvmm-vmm-tests + - openvmm-changes + - build-openvmm-vmm-tests-linux + - openvmm-vmm-tests-linux + - openvmm-vmm-tests-windows - openvmm-unit-tests - nvx-microvm-tests-kvm - nvx-microvm-tests-mshv @@ -1652,6 +1800,8 @@ jobs: - platform-kvm - platform-mshv - platform-whp + # The VMM-test jobs must all succeed, or all be skipped because the change + # selects no VMM tests: a failed or unexpectedly skipped one blocks this. if: >- ${{ always() && @@ -1662,8 +1812,18 @@ jobs: needs.nvx-cli-linux.result == 'success' && needs.nvx-cli-windows.result == 'success' && ( - needs.openvmm-vmm-tests.result == 'success' || - needs.openvmm-vmm-tests.result == 'skipped' + ( + needs.build-openvmm-vmm-tests-linux.result == 'success' && + needs.openvmm-vmm-tests-linux.result == 'success' && + needs.openvmm-vmm-tests-windows.result == 'success' + ) || + ( + needs.openvmm-changes.result == 'success' && + needs.openvmm-changes.outputs.run-openvmm-vmm-tests != 'true' && + needs.build-openvmm-vmm-tests-linux.result == 'skipped' && + needs.openvmm-vmm-tests-linux.result == 'skipped' && + needs.openvmm-vmm-tests-windows.result == 'skipped' + ) ) && ( needs.openvmm-unit-tests.result == 'success' || diff --git a/doc/ci.md b/doc/ci.md index 944e247f8..60e592f67 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -14,9 +14,9 @@ its performance series on the same VM (see [Benchmarks](benchmarks.md#ci-collection)). Before it measures, each platform job also checks that its runner is still the host that the series' history comes from (see -[Expected benchmark hosts](benchmarks.md#expected-benchmark-hosts)). `openvmm-vmm-tests` downloads only the NVX +[Expected benchmark hosts](benchmarks.md#expected-benchmark-hosts)). The OpenVMM VMM tests download only the NVX Linux-direct kernel and Alpine initramfs (see -[Guest artifact handoff](#guest-artifact-handoff)) and uses them to +[Guest artifact handoff](#guest-artifact-handoff)) and use them to exercise OpenVMM's Linux MP-table lifecycle, TTRPC, and snapshot contracts on each backend. `openvmm-unit-tests` runs the OpenVMM unit and documentation tests once per operating system, on a GitHub-hosted Ubuntu runner and a WHP @@ -24,13 +24,14 @@ runner. The tests that need `/dev/kvm` or `/dev/mshv` are ignored, so the KVM and MSHV runners used to run the same 4,820 tests, and the Linux leg needs no hypervisor runner; see [Job graph and runner capacity](#job-graph-and-runner-capacity). -Failed `openvmm-vmm-tests` jobs upload Petri's `test_results` directory, +Failed VMM-test jobs, which [OpenVMM VMM tests](#openvmm-vmm-tests) describes, +upload Petri's `test_results` directory, including guest and VMM logs, screenshots, and watchdog inspection data. These seven-day artifacts are named `openvmm-vmm-tests----`, so a successful rerun does not replace the failed attempt's diagnostics. Linux collects them from -`openvmm/target/vmm_tests/test_results`; Windows uses -`//test_results`. +`/openvmm-vmm-tests/test_results`, in the test content that the +job runs; Windows uses `/whp/test_results`. The `nvx-microvm-tests-{kvm,mshv,whp}` jobs consume the NVX Linux kernel and the NVX Linux kernel plus the selected Alpine or Ubuntu initramfs and exercises Linux, SMP, virtio, sandbox, and snapshot behavior through the public OpenVMM @@ -467,11 +468,13 @@ request and selects every suite for a full run (see [Change classification](#change-classification)). When it finishes, the jobs that it selects start: `artifacts`, `distro-artifacts`, and `debug-kernel` restore or build the guest artifacts on GitHub-hosted runners (see -[Guest artifact handoff](#guest-artifact-handoff)), the three `build-openvmm-*` -producers restore or build OpenVMM on a KVM, an MSHV, and a WHP runner, and +[Guest artifact handoff](#guest-artifact-handoff)), the three OpenVMM binary +producers restore or build OpenVMM on a KVM, an MSHV, and a WHP runner, +`build-openvmm-vmm-tests-linux` builds the Linux VMM tests on a KVM runner, and `openvmm-unit-tests` starts on a GitHub-hosted and a WHP runner. -`openvmm-vmm-tests` waits for `artifacts`, which builds the kernel and the -Alpine initramfs. Each microVM test and platform job waits for `artifacts`, +`openvmm-vmm-tests-linux` waits for `artifacts`, which builds the kernel and the +Alpine initramfs, and the Linux VMM-test build, and `openvmm-vmm-tests-windows` +for `artifacts`. Each microVM test and platform job waits for `artifacts`, `distro-artifacts`, its OpenVMM producer, and the NVX CLI job of its operating system, and the debug-kernel jobs wait for `debug-kernel` instead of `distro-artifacts`. `Performance regression gate`, `Publish development @@ -564,7 +567,8 @@ components that its steps read, and waits only for the jobs that upload them: | `guest-azurelinux` | `initramfs-azurelinux.cpio.gz` and its `.packages.json` | `distro-artifacts` | microVM tests, platform jobs | | `guest-debug-kernel` | `vmlinux-debug`, `vmlinux-debug.config`, `vmlinux-debug.provenance.json` | `debug-kernel` | debug-kernel jobs | -The OpenVMM VMM tests boot only the kernel and the Alpine initramfs. The +The OpenVMM VMM tests boot only the kernel and the Alpine initramfs, and their +Linux build boots no guest, so it downloads no component. The debug-kernel jobs run the Alpine scenarios on the debug kernel and qualify their runner, through the doctor's preflight (H3), with the kernel and the Alpine initramfs. The microVM test jobs boot every guest and run sandboxes @@ -618,6 +622,59 @@ for those that were cancelled, half would have started at once, one in ten would have waited at least 50 minutes, and the longest wait, on October 20, would have been about three hours. +## OpenVMM VMM tests + +OpenVMM's `vmm-tests-run` builds the tests that a filter selects and then runs +them. In build-only mode, it instead writes everything that the tests need to +a test content directory, from which `vmm-tests-run-target` runs a filter's +tests on another host: the tests' nextest archive, OpenVMM, the guest agents, +the test firmware and kernels, and the `flowey_hvlite` that runs them. The +content is relocatable, because `vmm-tests-run-target` remaps the archive's +workspace to the directory, which also serves as the repository root whose +nextest configuration the tests read. The KVM and MSHV tests run the same +`x86_64-unknown-linux-gnu` build: their jobs resolved the same artifacts, and +the GNU OpenVMM that a KVM runner builds already runs the MSHV microVM tests +on the same Azure Linux 3 release. CI therefore builds the Linux tests once +and hands them to both backends: + +- `build-openvmm-vmm-tests-linux` runs + [`build-openvmm-tests`](usage.md#build-openvmm-tests) for KVM and MSHV, + which builds the union of their filters, on a general KVM runner, whose + sccache holds the Linux test build. It restores, and on `dev` pushes saves, + the `vmm-tests` OpenVMM input and Cargo caches, and uploads the content as + the one-day `openvmm-vmm-tests-linux` artifact, which keeps the content's + hidden `.config` and replaces an earlier attempt's content. It needs no + guest artifacts, so it starts with the OpenVMM binary producers instead of + waiting for `artifacts`. +- `openvmm-vmm-tests-linux` runs a KVM and an MSHV leg once the build and + `artifacts` succeed. Each downloads the content, the kernel, and the Alpine + initramfs, and runs [`test-openvmm`](usage.md#test-openvmm) on the content, + which runs its backend's own filter with the content's + `vmm-tests-run-target`. The legs check out no OpenVMM, restore no cache, and + compile nothing, and a rerun of a failed leg downloads the build of the + earlier attempt instead of rebuilding. +- `openvmm-vmm-tests-windows` still builds and runs the WHP tests in one job, + after `artifacts`. On Windows, the pinned `vmm-tests-run-target` checks for + Hyper-V whatever tests it runs: it requires `Get-VM` to succeed and the + `AllowFirmwareLoadFromFile` and `EnableAdditionalComPorts` registry values + to be set, or administrator rights to set them. The WHP runners run jobs as + Network Service, which is not a Hyper-V administrator, on hosts whose setup + sets neither value. The WHP tests can move to a Windows build job once + `vmm-tests-run-target` checks only the dependencies that the tests of its + build need. + +The content's manifest, `nvx-openvmm-tests.json`, records the OpenVMM revision, +the Rust release, each backend's filter, the runner and its options, and every +file, so that a test job refuses content built from another OpenVMM revision +or for another filter, and so that the content describes the inputs on which +reusing it would depend. The build and its test jobs run together: a run that +selects the VMM tests, and so builds the guest artifacts, runs all three VMM +jobs, and any other run skips all three. `Required status check` expects +exactly that, and `Publish development release` and `Persist performance +baseline` require the three jobs to succeed, or all three to be skipped +because the change selected no VMM tests, so that a failed or unexpectedly +skipped build or test blocks them. + ## Change classification `openvmm-changes` lists the files that changed between the pull request's base @@ -660,7 +717,7 @@ other path selects each suite that takes it as an input: and the platform jobs but not the VMM tests. - `run-guest-artifacts` enables `artifacts`, `distro-artifacts`, and `debug-kernel`, which build the guest artifacts and the CI debug kernel, and - `run-openvmm-binaries` the three `build-openvmm-*` producers. Neither has a + `run-openvmm-binaries` the three OpenVMM binary producers. Neither has a use of its own, so the classifier sets each exactly when a job that uses its artifacts runs: the guest artifacts for the OpenVMM VMM tests, the NVX microVM tests, or the platform jobs, and the OpenVMM binaries for the diff --git a/scripts/nvx_tools/ci.py b/scripts/nvx_tools/ci.py index 7185d3066..180446159 100644 --- a/scripts/nvx_tools/ci.py +++ b/scripts/nvx_tools/ci.py @@ -72,7 +72,9 @@ "build-openvmm-linux-gnu": "BUILD_LINUX_GNU_RESULT", "build-openvmm-linux-musl": "BUILD_LINUX_MUSL_RESULT", "build-openvmm-windows-msvc": "BUILD_WINDOWS_MSVC_RESULT", - "openvmm-vmm-tests": "VMM_TESTS_RESULT", + "build-openvmm-vmm-tests-linux": "BUILD_VMM_TESTS_LINUX_RESULT", + "openvmm-vmm-tests-linux": "VMM_TESTS_LINUX_RESULT", + "openvmm-vmm-tests-windows": "VMM_TESTS_WINDOWS_RESULT", "openvmm-unit-tests": "UNIT_TESTS_RESULT", "nvx-microvm-tests-kvm": "MICROVM_KVM_RESULT", "nvx-microvm-tests-mshv": "MICROVM_MSHV_RESULT", @@ -96,7 +98,13 @@ REQUIRED_CI_OPENVMM_TEST_JOBS = ("openvmm-unit-tests",) # OpenVMM VMM tests boot the NVX guest artifacts, so they need the job that # builds them. -REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS = ("openvmm-vmm-tests",) +REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS = ( + "openvmm-vmm-tests-linux", + "openvmm-vmm-tests-windows", +) +# Each builds, without the guest artifacts, the VMM tests that the test jobs of +# its OS run, so it runs whenever a change selects the VMM tests. +REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS = ("build-openvmm-vmm-tests-linux",) REQUIRED_CI_MICROVM_TEST_JOBS = ( "nvx-microvm-tests-kvm", "nvx-microvm-tests-mshv", @@ -482,6 +490,16 @@ def required_ci_expected_results( for job in REQUIRED_CI_OPENVMM_TEST_JOBS } ) + expected.update( + { + job: ( + "success" + if repository_jobs_enabled and changes.run_openvmm_vmm_tests + else "skipped" + ) + for job in REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS + } + ) expected.update( { job: ( diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index da4e616b7..89b842afa 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -4414,6 +4414,8 @@ def load(name: str) -> Any: "setup-cross-os-cache": set[str](), "verify": set[str](), "test-openvmm": kernel_alpine, + # The VMM tests' build boots no guest. + "build-openvmm-tests": set[str](), "test-aci-edge-sandboxes": kernel_alpine, # A sandbox boots the Alpine control initramfs under its layers. "sandbox": kernel_alpine, @@ -4589,15 +4591,28 @@ def consume( self.assertEqual( consumers, { - "openvmm-vmm-tests", + *ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS, *ci.REQUIRED_CI_MICROVM_TEST_JOBS, *ci.REQUIRED_CI_MICROVM_DEBUG_JOBS, *ci.REQUIRED_CI_MICROVM_DEBUG_PUSH_JOBS, *ci.REQUIRED_CI_PLATFORM_JOBS, }, ) + # The VMM tests' build downloads no guest artifact, none of its steps + # reads one, and it waits for no job that builds one. + for job_name in ci.REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS: + job = jobs[job_name] + context = {"runner.os": "Linux"} + with self.subTest(build=job_name): + self.assertIsNone(consume(job["steps"], context)) + self.assertEqual( + set[str]().union(*(reads(step, context) for step in job["steps"])), + set(), + ) + self.assertFalse(set(job["needs"]) & set(producers.values())) - # test-openvmm boots the kernel and the Alpine initramfs. + # test-openvmm boots the kernel and the Alpine initramfs, whether it + # builds the tests or runs the content of a build. requested: list[str] = [] class Stop(Exception): @@ -4610,16 +4625,22 @@ def artifact(name: str) -> Path: def present(path: Path, _description: str) -> Path: return path - with ( - patch.object(ci, "validate_openvmm_test_backend"), - patch.object(ci, "require_file", side_effect=present), - patch.object(ci, "require_tool", return_value="tool"), - patch.object(ci, "artifact_path", side_effect=artifact), - patch.object(ci, "_prepare_openvmm_test_environment", side_effect=Stop), - self.assertRaises(Stop), + for content, stop in ( + (None, "_prepare_openvmm_test_environment"), + (Path("content"), "_openvmm_test_content"), ): - ci.run_openvmm_tests("kvm") - self.assertEqual(set(requested), commands["test-openvmm"]) + requested.clear() + with ( + self.subTest(content=content), + patch.object(ci, "validate_openvmm_test_backend"), + patch.object(ci, "require_file", side_effect=present), + patch.object(ci, "require_tool", return_value="tool"), + patch.object(ci, "artifact_path", side_effect=artifact), + patch.object(ci, stop, side_effect=Stop), + self.assertRaises(Stop), + ): + ci.run_openvmm_tests("kvm", content) + self.assertEqual(set(requested), commands["test-openvmm"]) def test_development_release_requires_successful_crate_checks(self): workflow = ( @@ -4789,7 +4810,13 @@ def test_required_ci_result_policy(self): repository = always_successful | set(ci.REQUIRED_CI_CLI_JOBS) builds = set(ci.REQUIRED_CI_BUILD_JOBS) openvmm_tests = set(ci.REQUIRED_CI_OPENVMM_TEST_JOBS) - openvmm_artifact_tests = set(ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS) + # The VMM tests' build boots no guest, so it runs whenever a change + # selects the VMM tests, which the classifier only does together with + # the guest artifacts that the tests boot. + openvmm_test_builds = set(ci.REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS) + openvmm_artifact_tests = openvmm_test_builds | set( + ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS + ) microvm_tests = set(ci.REQUIRED_CI_MICROVM_TEST_JOBS) debug_kernel = set(ci.REQUIRED_CI_MICROVM_DEBUG_JOBS) debug_kernel_push = set(ci.REQUIRED_CI_MICROVM_DEBUG_PUSH_JOBS) @@ -4885,7 +4912,7 @@ def test_required_ci_result_policy(self): "pull_request", True, (False, True, True, True, True, True), - repository | builds | openvmm_tests, + repository | builds | openvmm_tests | openvmm_test_builds, ), ( "pull_request", @@ -5036,7 +5063,9 @@ def test_ci_jobs_follow_their_change_detector_output(self): # Each job that the change detector selects runs when its output is # true, or when the detector did not succeed, so that a failed # classification runs the full matrix. Required status check passes - # every output to check-required-ci, and no other job reads one. + # every output to check-required-ci, the release and the performance + # persistence read whether the VMM tests run, and no other job reads + # one. workflow = ( BuildConstants.REPO_ROOT / ".github" / "workflows" / "ci.yml" ).read_text(encoding="utf-8") @@ -5047,7 +5076,11 @@ def test_ci_jobs_follow_their_change_detector_output(self): **dict.fromkeys(ci.REQUIRED_CI_BUILD_JOBS, "run-openvmm-binaries"), **dict.fromkeys(ci.REQUIRED_CI_OPENVMM_TEST_JOBS, "run-openvmm-unit-tests"), **dict.fromkeys( - ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS, "run-openvmm-vmm-tests" + ( + *ci.REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS, + *ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS, + ), + "run-openvmm-vmm-tests", ), **dict.fromkeys( ( @@ -5077,6 +5110,18 @@ def test_ci_jobs_follow_their_change_detector_output(self): re.findall(r"needs\.openvmm-changes\.outputs\.([a-z-]+)", job), [output], ) + # The release and the performance persistence accept skipped VMM-test + # jobs only when the change selected no VMM tests (#690). + gates = ("release", "performance-persist") + for job_name in gates: + with self.subTest(job=job_name): + self.assertEqual( + re.findall( + r"needs\.openvmm-changes\.outputs\.([a-z-]+)", + _workflow_job(workflow, job_name), + ), + ["run-openvmm-vmm-tests"], + ) jobs = re.findall( r"^ ([A-Za-z0-9_-]+):$", workflow.split("\njobs:\n", 1)[1], re.MULTILINE ) @@ -5086,7 +5131,7 @@ def test_ci_jobs_follow_their_change_detector_output(self): for job_name in jobs if "needs.openvmm-changes.outputs." in _workflow_job(workflow, job_name) }, - set(selections) | {"required-status-check"}, + set(selections) | {"required-status-check", *gates}, ) # The detector checks and forwards exactly the classifier's outputs. script = _workflow_step(workflow, "openvmm-changes", "Detect relevant changes") @@ -5172,6 +5217,7 @@ def test_ci_repository_jobs_run_for_every_non_pull_request_event(self): *ci.REQUIRED_CI_CLI_JOBS, *ci.REQUIRED_CI_BUILD_JOBS, *ci.REQUIRED_CI_OPENVMM_TEST_JOBS, + *ci.REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS, *ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS, *ci.REQUIRED_CI_MICROVM_TEST_JOBS, *ci.REQUIRED_CI_MICROVM_DEBUG_JOBS, @@ -5580,6 +5626,7 @@ def test_ci_change_classification_runs_microvm_tests_for_the_runtime_adapter(sel self.assertEqual(expected[job], "success", job) for job in ( *ci.REQUIRED_CI_OPENVMM_TEST_JOBS, + *ci.REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS, *ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS, *ci.REQUIRED_CI_PLATFORM_JOBS, "performance-gate", @@ -5652,7 +5699,11 @@ def imported_modules(module: str) -> set[str]: } vmm_inputs = command_inputs | { f".github/actions/{name}/action.yml" - for job in ("openvmm-vmm-tests", "artifacts") + for job in ( + *ci.REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS, + *ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS, + "artifacts", + ) for name in local_actions(_workflow_job(workflow, job)) } # Every cache key input of the kernel and Alpine initramfs that the @@ -6030,10 +6081,13 @@ def test_only_guest_binary_inputs_rebuild_guests_and_run_vmm_tests(self): *ci.REQUIRED_CI_PLATFORM_JOBS, ): self.assertEqual(expected[job], "success", job) - self.assertEqual( - expected["openvmm-vmm-tests"], - "success" if guest else "skipped", - ) + for job in ( + *ci.REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS, + *ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS, + ): + self.assertEqual( + expected[job], "success" if guest else "skipped", job + ) with self.subTest(changed=paths): contents = {path: (checkout / path).read_bytes() for path in paths} for path, data in contents.items(): @@ -6530,7 +6584,8 @@ def test_flowey_downloads_use_retrying_curl(self): self.assertIn("Get-Command rustc.exe", action) self.assertNotIn("curl.cmd", action) for job_name in ( - "openvmm-vmm-tests", + *ci.REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS, + *ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS, "openvmm-unit-tests", ): with self.subTest(job_name=job_name): @@ -6835,7 +6890,8 @@ def test_ci_shares_openvmm_inputs_and_binary_artifacts(self): 2, ) for job_name, cache_namespace in ( - ("openvmm-vmm-tests", "vmm-tests"), + ("build-openvmm-vmm-tests-linux", "vmm-tests"), + ("openvmm-vmm-tests-windows", "vmm-tests"), ("openvmm-unit-tests", "unit-tests"), ): with self.subTest(job_name=job_name): @@ -6868,6 +6924,12 @@ def test_ci_shares_openvmm_inputs_and_binary_artifacts(self): "${{ runner.arch }}-", job, ) + # The Linux VMM-test jobs run the build's test content and compile + # nothing, so they neither start sccache nor restore OpenVMM's inputs. + consumer_job = _workflow_job(workflow, "openvmm-vmm-tests-linux") + self.assertNotIn("uses: ./.github/actions/sccache", consumer_job) + self.assertNotIn("uses: ./.github/actions/checkout-openvmm", consumer_job) + self.assertNotIn("actions/cache/", consumer_job) self.assertNotIn("uses: actions/cache@v5", workflow) self.assertNotIn("uses: actions/cache@v5", build_workflow) self.assertNotIn("uses: actions/cache@v5", build_action) @@ -7170,7 +7232,9 @@ def test_ci_runs_openvmm_vmm_tests_per_backend_and_unit_tests_per_os(self): workflow = ( BuildConstants.REPO_ROOT / ".github" / "workflows" / "ci.yml" ).read_text(encoding="utf-8") - vmm_tests_job = _workflow_job(workflow, "openvmm-vmm-tests") + build_job = _workflow_job(workflow, "build-openvmm-vmm-tests-linux") + linux_job = _workflow_job(workflow, "openvmm-vmm-tests-linux") + windows_job = _workflow_job(workflow, "openvmm-vmm-tests-windows") unit_tests_job = _workflow_job(workflow, "openvmm-unit-tests") def legs(job: str) -> list[tuple[str, str, str]]: @@ -7183,7 +7247,7 @@ def legs(job: str) -> list[tuple[str, str, str]]: ) self.assertEqual( - legs(vmm_tests_job), + legs(linux_job), [ ( "Linux / KVM", @@ -7195,13 +7259,22 @@ def legs(job: str) -> list[tuple[str, str, str]]: "mshv", '["self-hosted", "linux", "mshv", "virtual-machine", "general"]', ), - ( - "Windows / WHP", - "whp", - '["self-hosted", "windows", "whp", "virtual-machine", "general"]', - ), ], ) + self.assertIn(" name: OpenVMM vmm-tests / ${{ matrix.name }}\n", linux_job) + # One KVM runner builds the tests of both Linux backends. + self.assertIn(" name: OpenVMM vmm-tests build / Linux\n", build_job) + self.assertIn( + " runs-on: [self-hosted, linux, kvm, virtual-machine, general]\n", + build_job, + ) + self.assertIn(" backend: kvm\n", build_job) + self.assertIn(" name: OpenVMM vmm-tests / Windows / WHP\n", windows_job) + self.assertIn( + " runs-on: [self-hosted, windows, whp, virtual-machine, general]\n", + windows_job, + ) + self.assertIn(" backend: whp\n", windows_job) # The unit tests that need /dev/kvm or /dev/mshv are ignored, so they # run once per OS, and Linux needs no hypervisor runner (#697). self.assertEqual( @@ -7235,18 +7308,30 @@ def legs(job: str) -> list[tuple[str, str, str]]: unit_tests_job, ) self.assertEqual( - vmm_tests_job.count( - 'scripts/nvx.py test-openvmm --backend "${{ matrix.backend }}"' + build_job.count( + " python3 scripts/nvx.py build-openvmm-tests\n" + " --backend kvm --backend mshv\n" + ' --output "${RUNNER_TEMP}/openvmm-vmm-tests-linux"\n' ), - 2, + 1, + ) + self.assertEqual( + linux_job.count( + " python3 scripts/nvx.py test-openvmm\n" + ' --backend "${{ matrix.backend }}"\n' + ' --content "${RUNNER_TEMP}/openvmm-vmm-tests"\n' + ), + 1, ) self.assertEqual( - vmm_tests_job.count( - 'scripts\\nvx.py test-openvmm --backend "${{ matrix.backend }}"' + windows_job.count( + " run: python scripts\\nvx.py test-openvmm --backend whp\n" ), 1, ) - self.assertNotIn("test-openvmm-unit", vmm_tests_job) + for job in (build_job, linux_job, windows_job): + self.assertNotIn("test-openvmm-unit", job) + self.assertNotIn("test-openvmm --backend", build_job) self.assertEqual(unit_tests_job.count("scripts/nvx.py test-openvmm-unit"), 1) self.assertEqual( unit_tests_job.count("scripts\\nvx.py test-openvmm-unit"), @@ -7314,6 +7399,130 @@ def test_ci_runs_the_linux_openvmm_unit_tests_on_a_github_hosted_runner(self): with self.subTest(path=path): self.assertIsNotNone(ci.OPENVMM_UNIT_TEST_INPUT_PATHS.match(path)) + def test_ci_hands_the_linux_vmm_test_build_to_both_linux_backends(self): + # The Linux build boots no guest, so it starts beside the guest + # artifacts; each Linux backend's job runs its filter from the build's + # same-run artifact once both are ready (#690). + workflow = ( + BuildConstants.REPO_ROOT / ".github" / "workflows" / "ci.yml" + ).read_text(encoding="utf-8") + build_job = _workflow_job(workflow, "build-openvmm-vmm-tests-linux") + linux_job = _workflow_job(workflow, "openvmm-vmm-tests-linux") + windows_job = _workflow_job(workflow, "openvmm-vmm-tests-windows") + build_steps = [ + _yaml_scalar(step, "name") + for step in _workflow_steps(workflow, "build-openvmm-vmm-tests-linux") + ] + linux_steps = [ + _yaml_scalar(step, "name") + for step in _workflow_steps(workflow, "openvmm-vmm-tests-linux") + ] + + self.assertIn(" needs: [openvmm-changes]\n", build_job) + self.assertNotIn("artifacts", build_job.split(" steps:", 1)[0]) + # A change that selects the VMM tests also builds the guest artifacts, + # so the build needs no other output to run exactly with its tests. + self.assertIn( + " (\n" + " needs.openvmm-changes.result != 'success' ||\n" + " needs.openvmm-changes.outputs.run-openvmm-vmm-tests == 'true'\n" + " )\n", + build_job, + ) + self.assertIn( + " needs: [artifacts, build-openvmm-vmm-tests-linux, openvmm-changes]\n", + linux_job, + ) + self.assertIn(" needs.artifacts.result == 'success' &&\n", linux_job) + self.assertIn( + " needs.build-openvmm-vmm-tests-linux.result == 'success' &&\n", + linux_job, + ) + # The WHP job waits for neither Linux job. + self.assertIn(" needs: [artifacts, openvmm-changes]\n", windows_job) + self.assertNotIn("build-openvmm-vmm-tests-linux", windows_job) + + upload = _workflow_step( + workflow, "build-openvmm-vmm-tests-linux", "Upload OpenVMM tests" + ) + self.assertIn("uses: actions/upload-artifact@v7", upload) + self.assertIn(" name: openvmm-vmm-tests-linux\n", upload) + self.assertIn( + " path: ${{ runner.temp }}/openvmm-vmm-tests-linux\n", upload + ) + # The content's nextest configuration is under .config. + self.assertIn(" include-hidden-files: true\n", upload) + self.assertIn(" if-no-files-found: error\n", upload) + # A rerun of the build replaces the earlier attempt's content. + self.assertIn(" overwrite: true\n", upload) + self.assertIn(" retention-days: 1\n", upload) + self.assertLess( + build_steps.index("Build OpenVMM tests on Linux"), + build_steps.index("Upload OpenVMM tests"), + ) + + download = _workflow_step( + workflow, "openvmm-vmm-tests-linux", "Download OpenVMM tests" + ) + self.assertIn("uses: actions/download-artifact@v8", download) + self.assertIn(" name: openvmm-vmm-tests-linux\n", download) + self.assertIn( + " path: ${{ runner.temp }}/openvmm-vmm-tests\n", download + ) + for step in ( + "Download guest kernel", + "Download Alpine initramfs", + "Download OpenVMM tests", + ): + with self.subTest(step=step): + self.assertLess( + linux_steps.index(step), + linux_steps.index("Run OpenVMM tests on Linux"), + ) + + # Every VMM-test job must succeed, or all must be skipped because the + # change selects no VMM tests, before a release or persistence. + jobs = ( + "build-openvmm-vmm-tests-linux", + "openvmm-vmm-tests-linux", + "openvmm-vmm-tests-windows", + ) + self.assertEqual( + set(jobs), + { + *ci.REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS, + *ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS, + }, + ) + gate = ( + " (\n" + " (\n" + + "".join( + f" needs.{job}.result == 'success'" + + (" &&\n" if job != jobs[-1] else "\n") + for job in jobs + ) + + " ) ||\n" + " (\n" + " needs.openvmm-changes.result == 'success' &&\n" + " needs.openvmm-changes.outputs.run-openvmm-vmm-tests != 'true' &&\n" + + "".join( + f" needs.{job}.result == 'skipped'" + + (" &&\n" if job != jobs[-1] else "\n") + for job in jobs + ) + + " )\n" + " ) &&\n" + ) + for consumer in ("release", "performance-persist"): + job = _workflow_job(workflow, consumer) + with self.subTest(consumer=consumer): + self.assertIn(" - openvmm-changes\n", job) + for name in jobs: + self.assertIn(f" - {name}\n", job) + self.assertEqual(job.count(gate), 1) + self.assertNotIn("needs.openvmm-vmm-tests.result", job) + def test_ci_measures_each_performance_series_on_its_benchmark_runner(self): # Each backend's pool mixes VMs whose results differ by more than the # run-to-run noise (#383, #415). The platform jobs, and only they, @@ -7493,7 +7702,11 @@ def test_ci_validates_the_cli_once_per_os_after_checking_out_openvmm(self): self.assertIn(f" needs.{cli}.result == 'success' &&\n", job) for other in set(ci.REQUIRED_CI_CLI_JOBS) - {cli}: self.assertNotIn(other, job) - for job_name in ("openvmm-vmm-tests", "openvmm-unit-tests"): + for job_name in ( + *ci.REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS, + *ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS, + "openvmm-unit-tests", + ): with self.subTest(job=job_name): self.assertNotIn("nvx-cli", _workflow_job(workflow, job_name)) for consumer in ("release", "performance-persist", "required-status-check"): @@ -7723,29 +7936,37 @@ def test_ci_preserves_failed_openvmm_test_diagnostics(self): workflow = ( BuildConstants.REPO_ROOT / ".github" / "workflows" / "ci.yml" ).read_text(encoding="utf-8") - job = _workflow_job(workflow, "openvmm-vmm-tests") step_name = " - name: Upload OpenVMM test diagnostics" - self.assertIn(step_name, job) - upload = job.split(step_name, 1)[1].split("\n - name:", 1)[0] + # Each backend's artifact names its OS, backend, run, and attempt, and + # holds the test_results directory that Petri writes in the content + # directory that its job runs. + for job_name, backend, path in ( + ( + "openvmm-vmm-tests-linux", + "${{ matrix.backend }}", + "${{ runner.temp }}/openvmm-vmm-tests/test_results", + ), + ("openvmm-vmm-tests-windows", "whp", "${{ runner.temp }}/whp/test_results"), + ): + with self.subTest(job=job_name): + job = _workflow_job(workflow, job_name) + self.assertIn(step_name, job) + upload = job.split(step_name, 1)[1].split("\n - name:", 1)[0] - self.assertIn("\n if: failure()", upload) - self.assertIn("uses: actions/upload-artifact@v7", upload) - self.assertIn( - "name: openvmm-vmm-tests-${{ runner.os }}-${{ matrix.backend }}-" - "${{ github.run_id }}-${{ github.run_attempt }}", - upload, - ) - self.assertIn( - "path: ${{ runner.os == 'Windows' && " - "format('{0}/{1}/test_results', runner.temp, matrix.backend) || " - "'openvmm/target/vmm_tests/test_results' }}", - upload, - ) - self.assertIn("if-no-files-found: warn", upload) - self.assertIn("retention-days: 7", upload) + self.assertIn("\n if: failure()", upload) + self.assertIn("uses: actions/upload-artifact@v7", upload) + self.assertIn( + f"name: openvmm-vmm-tests-${{{{ runner.os }}}}-{backend}-" + "${{ github.run_id }}-${{ github.run_attempt }}\n", + upload, + ) + self.assertIn(f"path: {path}\n", upload) + self.assertIn("if-no-files-found: warn", upload) + self.assertIn("retention-days: 7", upload) + windows_job = _workflow_job(workflow, "openvmm-vmm-tests-windows") self.assertLess( - job.index(step_name), - job.index(" - name: Report sccache"), + windows_job.index(step_name), + windows_job.index(" - name: Report sccache"), ) def test_runner_setup_pins_and_validates_sccache(self): @@ -12674,27 +12895,43 @@ def test_openvmm_ci_downloads_guest_artifacts(self): workflow = ( BuildConstants.REPO_ROOT / ".github" / "workflows" / "ci.yml" ).read_text(encoding="utf-8") - openvmm_tests = _workflow_job(workflow, "openvmm-vmm-tests") - self.assertIn("needs: [artifacts, openvmm-changes]", openvmm_tests) - self.assertIn("needs.artifacts.result == 'success'", openvmm_tests) - # The VMM tests boot only the kernel and the Alpine initramfs. + # Only the jobs that run the VMM tests wait for the guest artifacts, + # which the microVM TTRPC test boots; the Linux build does not. + for job_name in ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS: + with self.subTest(job=job_name): + openvmm_tests = _workflow_job(workflow, job_name) + self.assertRegex( + openvmm_tests, + r"\n needs: \[artifacts, [a-z, -]*openvmm-changes\]", + ) + self.assertIn("needs.artifacts.result == 'success'", openvmm_tests) + # The VMM tests boot only the kernel and the Alpine initramfs. + self.assertEqual( + re.findall( + r"^ - name: (Download .+)\n" + r" uses: actions/download-artifact@v8\n" + r" with:\n" + r" name: (\S+)\n" + r" path: build$", + openvmm_tests, + re.MULTILINE, + ), + [ + ("Download guest kernel", "guest-kernel"), + ("Download Alpine initramfs", "guest-alpine"), + ], + ) + self.assertNotIn("guest-artifacts", openvmm_tests) + self.assertNotIn("distro-artifacts", openvmm_tests) + for job_name in ci.REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS: + with self.subTest(job=job_name): + self.assertNotIn( + "actions/download-artifact", _workflow_job(workflow, job_name) + ) self.assertEqual( - re.findall( - r"^ - name: (Download .+)\n" - r" uses: actions/download-artifact@v8\n" - r" with:\n" - r" name: (\S+)\n" - r" path: build$", - openvmm_tests, - re.MULTILINE, - ), - [ - ("Download guest kernel", "guest-kernel"), - ("Download Alpine initramfs", "guest-alpine"), - ], + set(ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS), + {"openvmm-vmm-tests-linux", "openvmm-vmm-tests-windows"}, ) - self.assertNotIn("guest-artifacts", openvmm_tests) - self.assertIn("openvmm-vmm-tests", ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS) def test_apk_add_uses_host_ca_bundle_without_overriding_configuration(self): root = Path("root")