[Pedro Henrique Penna (ppenna)]

Closes #695.
Part of #430.

Summary

Each backend's NVX microVM tests ran in one job: the default Alpine suite, then the guest smoke and sandbox steps, none of which needs a result of the suite. This PR runs those steps in a second job per backend, beside the suite, with the same inputs and on the same runner pool, and requires both jobs in the required status check, the development release, and performance persistence. Every step still runs exactly once per backend, and the debug-kernel jobs are unchanged.

Why

• The WHP microVM job bounds every run that skips both OpenVMM suites (CI: maximize workflow parallelism and reduce turnaround time #430). In reference run 39585909588, it took 15.1 min, of which the default suite took 10.1 min and the guest and sandbox steps 4.2 min:

  Steps in run 39585909588	WHP	KVM	MSHV
  Setup: checkouts, downloads, and runner validation	46 s	28 s	46 s
  Default Alpine suite	603 s	459 s	413 s
  Guest smoke and sandbox steps	249 s	192 s	125 s
  Whole job	905 s	681 s	588 s

• Over the 41 successful first-attempt runs with microVM jobs from 2026-10-20 21:05 to 2026-10-21 13:24 UTC, the WHP job took a median of 15.1 min (p10 14.3, p90 16.1), after a median queue of 0.1 min (max 8.4), and finished a median of 20.4 min after its run started. KVM took a median of 11.3 min and MSHV 9.8 min.

Changes

• run-nvx-microvm-tests.yml: a new boolean input, guests-and-sandboxes, selects the steps that follow the default suite instead of the suite: the public managed sandbox acceptance, the Ubuntu and Azure Linux guest smoke tests, the extended Ubuntu tests (KVM only), the Ubuntu sandbox steps, the aci_edge_sandboxes lifecycle, and the runtime adapter scenarios. Each later step keeps its !inputs.debug-kernel guard. That job uploads its failure logs as nvx-microvm-tests-<backend>-guests, because a second upload of nvx-microvm-tests-<backend> in a run would fail.
• ci.yml: three new jobs, nvx-microvm-guests-{kvm,mshv,whp} (NVX microVM tests / <platform> / Guests and sandboxes), each a copy of its backend's nvx-microvm-tests-* job apart from its name and the new input: the same needs, condition, runner labels, artifacts, and secret. release, required-status-check, and performance-persist list them in needs, and release and persistence accept only success or skipped from them, as from the default jobs.
• check-required-ci (scripts/nvx_tools/ci.py): the new jobs join REQUIRED_CI_MICROVM_TEST_JOBS and REQUIRED_CI_RESULT_ENVIRONMENTS, so the required status check expects each to succeed when the change schedules the microVM tests and to be skipped otherwise.
• doc/ci.md: describes the two jobs, their artifacts, the gates, and the general runners that they hold.

Fork, draft, path-filter, and cancellation behavior is unchanged, because the new jobs copy the conditions of the default ones. A failed part now reruns alone.

Design notes:

• Top-level jobs, rather than two jobs inside the reusable workflow, give each part its own result in the gates, so a part skipped by mistake fails the required status check instead of hiding behind its sibling's success.
• One job definition with a step selector, like debug-kernel, keeps the setup steps in one place and limits the diff to per-step conditions, so that it combines easily with the other CI: maximize workflow parallelism and reduce turnaround time #430 pull requests. Trial merges with their current heads show what the second to merge must change:
  • microvm tests: record each scenario's duration in a JSON Lines report #707 (ci: record each microVM scenario's duration in a machine-readable report #694): its always-run nvx-microvm-durations-* upload runs in both jobs of a backend, so its name needs the ${{ inputs.guests-and-sandboxes && '-guests' || '' }} suffix, and its test must derive that suffix; its doc/ci.md sentence conflicts with this PR's paragraph. With those changes, the CI tests pass on the combination.
  • ci: classify microVM tests, benchmarks, and artifact builds separately #712 (ci: classify the microVM tests, benchmarks, guest artifacts, and OpenVMM binaries separately #698): no textual conflict, but the three nvx-microvm-guests-* jobs must test run-microvm-tests instead of run-tests, like the default jobs. Until then, ci: classify microVM tests, benchmarks, and artifact builds separately #712's detector-output test and this PR's job-equality test fail. With that change, the CI tests pass on the combination.
  • ci: build the Linux OpenVMM VMM tests once for the KVM and MSHV jobs #708 (ci: build the OpenVMM VMM tests once per target and hand them to the backend jobs #690): only positional conflicts, in the constants of scripts/nvx_tools/ci.py and in two places of doc/ci.md; keeping both sides passes the CI tests.

Tests

• New test_ci_runs_the_steps_after_the_default_microvm_suite_beside_it evaluates every step condition for each backend and checks that:
  • both jobs set up the same inputs, and every other step of the former single job runs in exactly one of them;
  • the default job runs only the default suite, and the KVM-only extended Ubuntu tests stay on KVM;
  • the debug-kernel jobs select the same steps as before;
  • every upload step has a distinct artifact name in the default, guest and sandbox, and debug-kernel jobs;
  • each guest and sandbox job equals its backend's suite job apart from its name and the input.
• test_required_ci_job_uses_tested_policy now also checks that each job that the policy checks is in needs and reaches it as that job's result.
• Updated: the public sandbox acceptance test, which now looks in the guest and sandbox jobs, and the OpenVMM consumer test. The policy, gate, repository-gate, CLI, and nvx-rs handoff tests cover the new jobs through REQUIRED_CI_MICROVM_TEST_JOBS.

Validation

• Windows, Python 3.14, on 6c6be5644, with the CI tests, Ruff, and the link check rerun after rebasing onto 5418be9ab:
  • compileall and every scripts/test_*.py module that validate-nvx runs pass, except 18 Linux-shell subtests in test_setup_azure.py, which fail identically on dev under Windows and run on Linux in CI;
  • test_adversarial.py, test_hosts.py, and the CLI --help checks;
  • ruff check, ruff format --check, pyright for Linux and Windows, check_markdown_links.py, and git diff --check.
• actionlint 1.7.12 (release binary) on ci.yml and run-nvx-microvm-tests.yml: clean; it rejects a misspelled guests-and-sandboxes input.
• Seeded regressions: the tests fail for each of seven: a later step without the selector, a suite step without it, an artifact name without the suffix, a guest and sandbox job whose needs differ, a debug-kernel job that sets the selector, a guest and sandbox job missing from the release's needs, and a wrong required-status result mapping.
• Left to CI: every hypervisor job, including the new ones, on KVM, MSHV, and WHP.

Expected effect

• The WHP microVM lane shortens from about 15 to about 11 min, the default suite and its setup, while the guest and sandbox job takes about 5 min beside it. KVM drops from about 11.4 to 8.2 min and MSHV from about 9.8 to 7.7 min.
• Runs that skip both OpenVMM suites should finish about 4 min sooner whenever a second general WHP runner is free.
• Cost: one more general runner per backend for about 3 to 5 min per run, including 0.5 to 0.8 min of repeated setup. With both OpenVMM suites running, a run then fills KVM's and MSHV's four general runners, and on dev pushes and full runs the MSHV debug-kernel job waits about 3 min for one; WHP uses at most five of its eight.
• This PR's own CI runs the full matrix, because ci.yml and ci.py are inputs of every suite. Its default suite jobs took 10.4 to 10.7 min on WHP, 8.2 to 8.6 on KVM, and 7.8 to 7.9 on MSHV, and its guest and sandbox jobs 5.2, 3.8 to 3.9, and 3.4 to 3.5 min. In run 39622674984, both WHP jobs started at once, and the WHP lane took 10.7 min, 4.5 min below the baseline median. In run 39617140157, only one general WHP runner was free, so the two WHP jobs ran one after the other, in 15.8 min. Both runs overlapped heavy load from the other CI: maximize workflow parallelism and reduce turnaround time #430 pull requests, which made the KVM guest and sandbox job of the second run wait 72.5 min, while its KVM debug-kernel job, which predates this change, waited 83.4. The measurements on #695 give the details, the queue time that the extra job adds, and which backends gain from the split.
• Risk: when only one general runner of a pool is free, either job of a backend can take it first, as on WHP in run 39617140157. The lane is then about as long as one job was.

Post-merge verification

• Compare the WHP microVM lane, from the jobs' creation to the later completion of the two WHP jobs, over at least five comparable successful runs after the merge with the 41-run baseline above, including queue time, and record it on ci: run the guest smoke and sandbox steps in a job beside the default microVM suite #695.

[Copilot]

🔵 Needs a closer look

No concrete defects were identified, but cross-backend execution and runner contention still need CI validation and human confirmation.

No open issues 🎉

What changed in this PR

Splits guest and sandbox tests from the default microVM suite to reduce CI turnaround while preserving coverage across KVM, MSHV, and WHP.

Changes:

• Adds parallel guest-and-sandbox jobs with distinct failure-log artifacts.
• Integrates both job groups into required-status, release, and performance-persistence gates.
• Adds regression checks and documents runner-capacity implications.

File	Description
scripts/​test_nvx_tools.py	Checks step selection, job parity, artifact names, and gate wiring.
scripts/​nvx_tools/​ci.py	Adds guest jobs to required-result policy.
doc/​ci.md	Documents parallel jobs, artifacts, gates, and runner usage.
.github/​workflows/​run-nvx-microvm-tests.yml	Selects suite or guest steps and separates artifact names.
.github/​workflows/​ci.yml	Adds three parallel jobs and downstream gate dependencies.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-platform validation and queue-inclusive performance confirmation remain outstanding for the new CI topology.

No open issues 🎉

_{Review effort: Balanced}