Malformed coverage data when using llvm-cov

[StackOverflowExcept1on]

I tried to fuzz our substrate runtime but getting error:

error: Failed to load coverage: 'target/x86_64-unknown-linux-gnu/coverage/x86_64-unknown-linux-gnu/release/main': Malformed coverage data

How to reproduce (on our large repo)

git clone --branch av/rust-1.76-support https://github.com/gear-tech/gear.git
cd gear
git checkout 849dbb301c751c951754b73b39a50a02e7296bef

cd utils/runtime-fuzzer

mkdir -p fuzz/corpus/main
dd if=/dev/urandom of=fuzz/corpus/main/fuzzer-seed-corpus bs=1 count=350000

# Run fuzzer for at least for 3 minutes and then press Ctrl-C to stop fuzzing.
cargo fuzz run \
    --release \
    --sanitizer=none \
    main \
    fuzz/corpus/main \
    -- \
        -rss_limit_mb=8192 \
        -max_len=450000 \
        -len_control=0

cargo fuzz coverage \
    --release \
    --sanitizer=none \
    main \
    fuzz/corpus/main \
    -- \
        -rss_limit_mb=8192 \
        -max_len=450000 \
        -len_control=0

HOST_TARGET=$(rustc -Vv | grep "host: " | sed "s/^host: \(.*\)$/\1/")
cargo cov -- show target/$HOST_TARGET/coverage/$HOST_TARGET/release/main \
  --format=text \
  --show-line-counts \
  --Xdemangler=rustfilt \
  --instr-profile=fuzz/coverage/main/coverage.profdata \
  --ignore-filename-regex=/rustc/ \
  --ignore-filename-regex=.cargo/ &> fuzz/coverage/main/coverage.txt

Meta

rustc --version --verbose:

rustc 1.77.0-nightly (3cdd004e5 2023-12-29)
binary: rustc
commit-hash: 3cdd004e55c869faa2b7b25efd3becf50346e7d6
commit-date: 2023-12-29
host: x86_64-unknown-linux-gnu
release: 1.77.0-nightly
LLVM version: 17.0.6

[StackOverflowExcept1on]

@Zalathar I don't know how to reproduce this with a minimal example, but would appreciate it if you could somehow debug this in LLVM code. As you wrote earlier this comes from llvm-cov. It throws coveragemap_error::malformed somewhere but without any backtrace.

[StackOverflowExcept1on]

I debugged LLVM and this comes from the same place: https://github.com/rust-lang/llvm-project/blob/fef3d7b14ede45d051dc688aae0bb8c8b02a0566/llvm/lib/ProfileData/Coverage/CoverageMappingReader.cpp#L340-L344
But I have no idea how to get source file by ExpandedFileID to provide more info.

[Zalathar]

Pinpointing the LLVM error was very helpful, thanks. Without that I wouldn’t be able to do much.

Can you give me an example of the LineStart/ColumnStart/NumLines/ColumnEnd values that are present when it fails? That should give me a better idea of what’s going wrong on the Rust side.

[Zalathar]

(I might not be able to track down the underlying issue, but hopefully I should at least be able to add some extra checks to the compiler to make sure LLVM doesn’t encounter a fatal error.)

[StackOverflowExcept1on]

@Zalathar

LineStart=652 LineStartDelta=0 ColumnStart=9 NumLines=0 ColumnEnd=10
Counter in file 0 652:9 -> 652:10, #44
LineStart=1112 LineStartDelta=460 ColumnStart=1 NumLines=4294966836 ColumnEnd=10
Counter in file 0 1112:1 -> 4294967948:10, #0
here 1112, 1652, 10
error: Failed to load coverage: 'target/x86_64-unknown-linux-gnu/coverage/x86_64-unknown-linux-gnu/release/main': malformed coverage data
#10

[Zalathar]

I've submitted a possible workaround in #119460.

It doesn't address the underlying question of why we were producing improper regions in the first place, but it does at least mean that we will detect and discard those regions early, instead of emitting them and having llvm-cov fail.

[Zalathar]

NumLines=4294966836 is 0xFFFF_FE34, which is -460 (relative to LineStart=1112).

This suggests that the original coordinates were 1112:1 -> 652:10.

[Zalathar]

Spans within the compiler are always properly-ordered in terms of byte positions (enforced in Span::new), so I believe the only way to end up with improper line/column coordinates is if the span starts and ends in different files.

I'm not sure how we're ending up with a span like that. I initially suspected fn_sig_span, but it turns out that we require fn_sig_span and body_span to start in the same file, so that probably isn't the cause.

[Zalathar]

I wonder if the span adjustment in filtered_terminator_span for TerminatorKind::Call could be responsible, as it combines the endpoints of two potentially-unrelated spans.

Though if the resulting span crosses file boundaries, I would expect it to be discarded by unexpand_into_body_span, unless the body span also crosses file boundaries. But we don't do any ad-hoc manipulation of the endpoints of the body span itself, so the file-crossing span would have to already be present in MIR before coverage gets involved. I don't know whether that's possible or not.