Running sanitizer tests fails with mixed ABI errors

[djc]

I tried this code:

RUSTFLAGS="-Z sanitizer=address -Cunsafe-allow-abi-mismatch=sanitizer" cargo +nightly -Z build-std test --target x86_64-unknown-linux-gnu --all-features

I expected to see this happen: tests pass/fail

Instead, this happened:

error: mixing `-Zsanitizer` will cause an ABI mismatch in crate `hashlink`
  |
  = help: the `-Zsanitizer` flag modifies the ABI so Rust crates compiled with different values of this flag cannot be used together safely
  = note: unset `-Zsanitizer` in this crate is incompatible with `-Zsanitizer=memory` in dependency `core`
  = help: set `-Zsanitizer=memory` in this crate or unset `-Zsanitizer` in `core`
  = help: if you are sure this will not cause problems, you may use `-Cunsafe-allow-abi-mismatch=sanitizer` to silence this error

Note that I already added the suppression option, but it doesn't seem to take effect here?

Meta

rustc --version --verbose:

rustc 1.92.0-nightly (839222065 2025-10-05)

(See CI: https://app.circleci.com/pipelines/github/djc/hashlink/189/workflows/58588dc4-d323-428f-a309-d950c5fa2b0f/jobs/180.)

[djc]

I cannot reproduce this on my own ARM64 Linux server. This just works:

RUSTFLAGS="-Z sanitizer=address" cargo +nightly -Z build-std test --all-features

(I also tried the CI without --target, and that still fails the same way as before.)

[saethlin]

You're trying to use sanitizer=address but getting an error about sanitizer=memory? That's a transcription error or something is deeply wrong with your build environment.

[djc]

@saethlin thanks for having a look! If you look at the original CI run, it seems to be similar:

https://app.circleci.com/pipelines/github/djc/hashlink/189/workflows/58588dc4-d323-428f-a309-d950c5fa2b0f/jobs/180

Maybe it doesn't rebuild dependencies correctly, so that an earlier sanitizer=address build is reused when I run with sanitizer=memory later?

[saethlin]

Yeah, that's what it seems like to me. You should be able to pass --verbose to Cargo to see the exact flags being used. I usually also add -j1 so the output arrives in a simpler order. At least if you can see all the flags for the ASan build that should indicate if MSan crates are being pulled in from another build... Somehow.

[saethlin]

I can repro locally. The fact that this only happens with doctests seems relevant.

[saethlin]

Repros with an empty library crate. What in tarnation...

[saethlin]

This bisects to #138736 but I think that PR just uncovered the problem

[saethlin]

To repro, on an empty library crate run

RUSTFLAGS="-Z sanitizer=memory" cargo test -Z build-std --target=x86_64-unknown-linux-gnu

The trick is this note in the PR:

For now, we should mark all sanitizers as target modifiers except for these: AddressSanitizer, LeakSanitizer

And this diagnostic:

= note: unset `-Zsanitizer` in this crate is incompatible with `-Zsanitizer=memory` in dependency `std`

The problem is that to set compiler flags in doctests you need to set RUSTDOCFLAGS. This is a pretty classic footgun, and a number of my scripts have RUSTDOCFLAGS=$RUSTFLAGS to prevent this.

This also explains why adding -Cunsafe-allow-abi-mismatch=sanitizer to RUSTFLAGS doesn't help. This does though:

RUSTDOCFLAGS=-Cunsafe-allow-abi-mismatch=sanitizer RUSTFLAGS="-Z sanitizer=memory" cargo test -Z build-std --target=x86_64-unknown-linux-gnu

So this is a diagnostic problem.