WIP: Add ASAN-enabled OpenHCL IGVM build and test pipeline by will-j-wright · Pull Request #3072 · microsoft/openvmm

will-j-wright · 2026-03-19T22:28:55Z

Add AddressSanitizer (ASAN) support to the OpenHCL build pipeline and CI.

Changes

Sanitizer feature: New OpenvmmHclFeature::Sanitizer variant that injects ASAN RUSTFLAGS and RUSTC_BOOTSTRAP=1
X64Asan recipe: Dedicated IGVM recipe with 2GB VTL2 memory manifest, rootfs.asan.config (musl shared libs for dynamic linking), and the sanitizer cargo feature
extra_rootfs_configs field: New field on OpenhclIgvmRecipeDetails for per-recipe rootfs additions
Petri artifact: LATEST_ASAN_X64 artifact and resolver for ASAN IGVM
ASAN tests: Boot tests (UEFI + Linux-direct) using with_custom_openhcl to swap in the ASAN IGVM
CI: X64Asan added to checkin gates IGVM recipe list
CLI: cargo xflowey build-igvm x64-asan support
Guide: ASAN documentation under reference/openhcl/debugging/

Copilot

Pull request overview

Adds AddressSanitizer (ASAN) support for OpenHCL IGVM builds, integrates it into flowey recipes/artifact resolution, and introduces boot tests that exercise OpenHCL using the ASAN-instrumented IGVM—along with CI and Guide updates to document and gate the new build.

Changes:

Add a sanitizer feature path (propagated through openvmm_hcl/underhill_entry) and a dedicated X64Asan IGVM recipe (manifest + rootfs additions).
Add LATEST_ASAN_X64 artifact plumbing and new VMM tests that boot OpenHCL using the ASAN IGVM.
Update flowey pipelines/CI wiring and add Guide documentation for building ASAN-enabled OpenHCL IGVMs.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`vmm_tests/vmm_tests/tests/tests/x86_64/openhcl_asan.rs`	Adds new OpenHCL ASAN boot tests (UEFI + linux-direct) using `with_custom_openhcl`.
`vmm_tests/vmm_tests/tests/tests/x86_64.rs`	Registers the new `openhcl_asan` test module.
`vmm_tests/petri_artifacts_vmm_test/src/lib.rs`	Declares `LATEST_ASAN_X64` artifact and tags it as an x86_64 OpenHCL IGVM.
`vmm_tests/petri_artifact_resolver_openvmm_known_paths/src/lib.rs`	Resolves `LATEST_ASAN_X64` to the flowey `x64-asan` IGVM output path.
`vm/loader/manifests/openhcl-x64-asan.json`	Adds ASAN-specific OpenHCL manifest with increased VTL2 memory.
`openhcl/underhill_entry/src/lib.rs`	Gates mimalloc/fast_memcpy usage off when building with the `sanitizer` feature.
`openhcl/underhill_entry/Cargo.toml`	Introduces a `sanitizer` feature flag for underhill entrypoint behavior.
`openhcl/rootfs.asan.config`	Adds musl shared libs + dynamic linker symlink needed for ASAN dynamic linking.
`openhcl/openvmm_hcl/Cargo.toml`	Forwards `sanitizer` feature to `underhill_entry`.
`flowey/flowey_lib_hvlite/src/init_vmm_tests_env.rs`	Teaches vmm test env init about the `X64Asan` IGVM filename.
`flowey/flowey_lib_hvlite/src/build_openvmm_hcl.rs`	Adds `OpenvmmHclFeature::Sanitizer` and injects ASAN `RUSTFLAGS`/`RUSTC_BOOTSTRAP`.
`flowey/flowey_lib_hvlite/src/build_openhcl_igvm_from_recipe.rs`	Adds `X64Asan` recipe and `extra_rootfs_configs` support (plus sysroot env injection for initrd build).
`flowey/flowey_lib_hvlite/src/artifact_openhcl_igvm_from_recipe.rs`	Adds filename mapping for the `X64Asan` recipe.
`flowey/flowey_lib_hvlite/src/_jobs/local_build_igvm.rs`	Updates local IGVM build job destructuring to include the new recipe details field.
`flowey/flowey_hvlite/src/pipelines/checkin_gates.rs`	Includes `X64Asan` in checkin gate IGVM recipe list.
`flowey/flowey_hvlite/src/pipelines/build_igvm.rs`	Adds `x64-asan` CLI option and maps it to the new recipe.
`ci-flowey/openvmm-pr.yaml`	Updates generated CI flow to include new ASAN-related steps/indices.
`Guide/src/reference/openhcl/debugging/asan.md`	Adds documentation for building and using an ASAN-enabled OpenHCL IGVM.
`Guide/src/SUMMARY.md`	Adds the new ASAN Guide page to the book TOC.
`.github/workflows/openvmm-pr.yaml`	Updates GitHub PR workflow steps (flowey execution indices) to match new pipeline graph.
`.github/workflows/openvmm-pr-release.yaml`	Updates GitHub PR-release workflow steps to match new pipeline graph.
`.github/workflows/openvmm-ci.yaml`	Updates GitHub CI workflow steps to match new pipeline graph.

Comments suppressed due to low confidence (1)

openhcl/underhill_entry/src/lib.rs:18

The comment says that when sanitizer is active the #[global_allocator] is omitted so ASAN can use its allocator, but the mem-profile-tracing allocator is still enabled even if sanitizer is also set. This makes the behavior ambiguous and can silently disable ASAN’s intended allocation interception when both features are enabled. Consider making these features mutually exclusive (e.g., cfg(all(...)) + compile_error!) or gating the mem-profile-tracing allocator with not(feature = "sanitizer"), and update the comment accordingly.

// Use mimalloc instead of the system malloc for performance.
// For memory profiling, DHAT allocator is needed.
// When "sanitizer" is active, omit the #[global_allocator] entirely
// so ASAN's instrumented allocator (via musl malloc) is used.
#[global_allocator]
#[cfg(not(any(feature = "mem-profile-tracing", feature = "sanitizer")))]
static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
#[global_allocator]
#[cfg(feature = "mem-profile-tracing")]
static GLOBAL: dhat::Alloc = dhat::Alloc;

github-actions · 2026-03-19T23:52:45Z

At least one Petri test failed.

smalis-msft · 2026-03-21T17:11:48Z

+                            "-Ctarget-feature=-crt-static",
+                            "-Clink-self-contained=n",
+                            "-Cforce-frame-pointers=yes",
+                            "-Csymbol-mangling-version=v0",


Aren't most of these flags already set in our root .cargo/config.toml?

Yes, but doing this replaces RUSTFLAGS; it's not additive

Can we not do that? We shouldn't be overwriting rustflags anywhere, that's gonna break people trying to add something on the command line

Okay I tried a bunch of different approaches and none of them worked. I had claude make a writeup of the solutions I tried:

"ASAN needs to flip two flags from the musl rustflags section in .cargo/config.toml: +crt-static →
-crt-static and force-unwind-tables=no → yes. Unfortunately Cargo doesn't support additive or
per-flag overrides for rustflags arrays. We explored three approaches:

RUSTFLAGS env var: Replaces all flags from config.toml for all targets — exactly the problem
you flagged.

--config file.toml: Appends to the array instead of replacing it, so both +crt-static and
-crt-static end up in the flags. Rustc's target-feature doesn't use "last wins" semantics —
+crt-static always wins regardless of order, so the override is silently ignored.

--config 'key = value' (inline): The only form that actually replaces the array. This is what
we use.

The downside is that the inline override duplicates the musl flags. There's a sync comment in
.cargo/config.toml pointing to build_openvmm_hcl.rs. A proper fix would need Cargo to support
additive rustflags (rust-lang/cargo#5376), or significant flowey plumbing to dynamically read and
patch the config file at build time."

Basically it doesn't seem possible without some large flowey architectural changes. I realize how ugly it is to have them in the file, but I doubt anyone is going to try to make a flowey ASAN build with custom rustflags

If we do a musl based build with -crt-static how does the resulting binary function? We don't include a dynamically linkable musl in the vtl2 filesystem do we?

If we do a musl based build with -crt-static how does the resulting binary function? We don't include a dynamically linkable musl in the vtl2 filesystem do we?

Not normally, but I made it so the ASAN build includes them. See openhcl/rootfs.asan.config

Oh cool, that'll save me a little time, we'll need similar functionality for other stuff coming soon.

As for RUSTFLAGS, I can totally imagine wanting test scenarios with custom RUSTFLAGS and ASAN coverage, like say https://doc.rust-lang.org/stable/unstable-book/compiler-flags/randomize-layout.html. Can we use --config file.toml to prepend additional arguments instead of appending? Or some other solution?

The --config file.toml doesn’t work because -crt-static and +crt-static have some weird behavior where + always overrides - no matter the order. I’ll keep iterating on different possibilities

Copilot

Pull request overview

Copilot reviewed 39 out of 39 changed files in this pull request and generated 6 comments.

Copilot

Pull request overview

Copilot reviewed 39 out of 39 changed files in this pull request and generated 2 comments.

Add AddressSanitizer (ASAN) support to the OpenHCL build pipeline and CI: - Add `Sanitizer` variant to `OpenvmmHclFeature` that injects ASAN RUSTFLAGS (-Zsanitizer=address, -Ctarget-feature=-crt-static, etc.) and RUSTC_BOOTSTRAP=1 - Add `X64Asan` IGVM recipe with dedicated manifest (2GB VTL2 memory), rootfs.asan.config (musl shared libs for dynamic linking), and the sanitizer cargo feature - Add `extra_rootfs_configs` field to `OpenhclIgvmRecipeDetails` for per-recipe rootfs additions - Add petri artifact (LATEST_ASAN_X64) and resolver for ASAN IGVM - Add ASAN boot tests (UEFI + Linux-direct) using `with_custom_openhcl` - Add `X64Asan` to CI checkin gates IGVM recipe list - Add `cargo xflowey build-igvm x64-asan` CLI support - Revert dev manifest memory_page_count to 512MB (was incorrectly bumped to 2GB for all builds) - Fix ld-musl dynamic linker symlink name (ld-musl-x86_64.so.1) - Add ASAN guide documentation under reference/openhcl/debugging/ Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Fix table: sanitizer feature gates via #[cfg], deps are not optional - Remove incorrect note about needing to cd into crate directory - Use x64-asan recipe for manual IGVM build (includes ASAN manifest/rootfs) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ASAN builds previously set the RUSTFLAGS environment variable, which overwrites all flags from .cargo/config.toml and breaks anyone trying to add flags on the command line. Instead, pass ASAN-specific rustflags via cargo --config to override only the [target.'cfg(target_env = "musl")'] section. This preserves the common flags from .cargo/config.toml (frame pointers, symbol mangling, etc.) and only replaces the musl-specific flags that conflict with ASAN (e.g., +crt-static -> -crt-static, unwind-tables=no -> yes). Add extra_cargo_config field to run_cargo_build::Request to support passing --config overrides from build nodes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove the X64Asan recipe variant from OpenhclIgvmRecipe. Instead, add a --with-asan flag to the build-igvm CLI and a with_asan field on the Customizations struct. When with_asan is set, the recipe details are modified in place to: - Add the Sanitizer feature - Switch the IGVM manifest to openhcl-x64-asan.json (2 GB VTL2 memory) - Include rootfs.asan.config (musl shared libs for dynamic linking) This allows any recipe to be built with ASAN (e.g., x64 --with-asan, x64-cvm --with-asan) rather than requiring a dedicated recipe variant for each combination. For CI, the ASAN IGVM is built via LocalOnlyCustom with the same modifications applied to the X64 base recipe. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Remove stale troubleshooting entry about 'outside of workspace' - Update guide to use --with-asan flag instead of x64-asan recipe - Guard --with-asan to bail on non-x86_64 recipes - Update petri artifact resolver to match --with-asan output path Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Remove misleading --no-default-features from sanitizer feature comment - Reflow ASAN guide prose to ~80 columns per OSS Guide style Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The extra_cargo_config field was added for a --config-based ASAN rustflags approach that didn't work (Cargo appends array values from --config instead of replacing them). ASAN now uses RUSTFLAGS env var Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

will-j-wright · 2026-03-25T21:51:37Z

Pending rust-lang/rust#154388

will-j-wright requested review from a team as code owners March 19, 2026 22:28

Copilot AI review requested due to automatic review settings March 19, 2026 22:28

github-actions bot added the Guide label Mar 19, 2026

Copilot started reviewing on behalf of will-j-wright March 19, 2026 22:32 View session

Copilot AI reviewed Mar 19, 2026

View reviewed changes

Comment thread Guide/src/reference/openhcl/debugging/asan.md Outdated

Comment thread Guide/src/reference/openhcl/debugging/asan.md Outdated

Comment thread Guide/src/reference/openhcl/debugging/asan.md

Comment thread Guide/src/reference/openhcl/debugging/asan.md

smalis-msft reviewed Mar 21, 2026

View reviewed changes

Comment thread flowey/flowey_hvlite/src/pipelines/build_igvm.rs Outdated

smalis-msft reviewed Mar 21, 2026

View reviewed changes

Comment thread flowey/flowey_lib_hvlite/src/build_openvmm_hcl.rs

smalis-msft reviewed Mar 21, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings March 24, 2026 21:07

Copilot started reviewing on behalf of will-j-wright March 24, 2026 21:08 View session

will-j-wright force-pushed the user/wwright/ohcl_asan_ci branch 2 times, most recently from 27f14ed to 6c7d81e Compare March 24, 2026 21:13

Copilot AI reviewed Mar 24, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings March 24, 2026 21:37

will-j-wright force-pushed the user/wwright/ohcl_asan_ci branch from 36c0bb6 to c445e6e Compare March 24, 2026 21:37

Copilot started reviewing on behalf of will-j-wright March 24, 2026 21:38 View session

Copilot AI reviewed Mar 24, 2026

View reviewed changes

Comment thread openhcl/underhill_entry/Cargo.toml

Comment thread Guide/src/reference/openhcl/debugging/asan.md Outdated

Copilot AI review requested due to automatic review settings March 25, 2026 17:44

will-j-wright marked this pull request as draft March 25, 2026 17:46

Copilot started reviewing on behalf of will-j-wright March 25, 2026 17:50 View session

will-j-wright and others added 8 commits March 25, 2026 17:55

Enable ASAN in OHCL

e312f7d

Enable for ARM

c382c76

fix: label bare code fences in ASAN guide to fix mdbook test

2d2cb10

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

fix: remove stale --no-default-features comment, reflow guide prose

6c36179

- Remove misleading --no-default-features from sanitizer feature comment - Reflow ASAN guide prose to ~80 columns per OSS Guide style Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

will-j-wright force-pushed the user/wwright/ohcl_asan_ci branch 2 times, most recently from 5f1fa3c to 4f1f386 Compare March 25, 2026 18:11

will-j-wright force-pushed the user/wwright/ohcl_asan_ci branch from 4f1f386 to 9db6ee8 Compare March 25, 2026 18:28

will-j-wright closed this Mar 25, 2026

Conversation

will-j-wright commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Mar 19, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

will-j-wright Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

will-j-wright commented Mar 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

will-j-wright commented Mar 19, 2026 •

edited

Loading

will-j-wright Mar 24, 2026 •

edited

Loading