Windows CI runners are sporadically failing with os error 32

[ehuss]

The Windows CI runners have recently started failing with:

thread 'main' panicked at 'failed to copy D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage1-rustc\x86_64-pc-windows-msvc\release\rustc-main.exe to D:\a\rust\rust\build\x86_64-pc-windows-msvc\stage2\bin\rustc.exe: The process cannot access the file because it is being used by another process. (os error 32)', src\bootstrap\lib.rs:1335:17

os error 32 is ERROR_SHARING_VIOLATION. It is not clear to me what could possibly be locking the file in such a way to prevent reading it.

Recent instances:

• 2021-09-13T20:36:18 Rollup of 7 pull requests #88914: https://github.com/rust-lang-ci/rust/actions/runs/1231002021
• 2021-09-13T21:36:02 Enum should prefer discriminant zero for niche #87794: https://github.com/rust-lang-ci/rust/actions/runs/1231146440
• 2021-09-14T02:08:15 Use a separate interner type for UniqueTypeId #87867: https://github.com/rust-lang-ci/rust/actions/runs/1231817053
• 2021-09-14T03:42:19 Rollup of 6 pull requests #88921: https://github.com/rust-lang-ci/rust/runs/3594233900
• 2021-09-14T04:44:27 Use a separate interner type for UniqueTypeId #87867: https://github.com/rust-lang-ci/rust/runs/3594557656

[ehuss]

Another curious error in https://github.com/rust-lang-ci/rust/runs/3592562094:

note: LINK : fatal error LNK1104: cannot open file 'D:\a\rust\rust\build\i686-pc-windows-msvc\stage2-std\release\build\compiler_builtins-3ad280178e43f3ea\build_script_build-3ad280178e43f3ea.exe'

That error does not make much sense.

[workingjubilee]

Also see #88921

[Mark-Simulacrum]

@rustbot ping windows

This error is causing lots of spurious failures in our CI, maybe there's some steps to diagnose that folks could suggest (or it's well known and there's a clear fix).

[rustbot]

Hey Windows Group! This bug has been identified as a good "Windows candidate".
In case it's useful, here are some instructions for tackling these sorts of
bugs. Maybe take a look?
Thanks! <3

cc @arlosi @danielframpton @gdr-at-ms @kennykerr @luqmana @lzybkr @nico-abram @retep998 @rylev @sivadeilra @wesleywiser

[ehuss]

Another related error message:

rm: cannot remove '/d/a/rust/rust/build/x86_64-pc-windows-msvc/test/run-make-fulldeps/reproducible-build/reproducible-build/linker.exe': Device or resource busy

https://github.com/rust-lang-ci/rust/runs/3600286433

There haven't been any image changes since the September 9th change that broke CI.

A rough timeline:

• Sep 8: stage 0 was bumped (probably unrelated)
• 2 successful merges, 1 failed (unrelated)
• Sep 9: GitHub image update halts CI
• Sep 10: Windows builds resolved
• Things look "normal" with 22 successful merges and 11 failed (unrelated to this issue).
• Sep 13: First failure shows up on Rollup of 7 pull requests #88914.
• Since then (to now), there have been 2 successful merges and 16 failures (~11 due to this issue).

I didn't see any changes to bootstrap or CI that look fishy (except for #88797, but it seems strange that there would be a couple days without problems).

[CryZe]

It's a change to Github Actions that caused this. We also encountered it here: Lokathor/wide@6419c50

And someone else encountered it here with Python: https://github.com/PyO3/pyo3/runs/3601829130?check_suite_focus=true

tl;dr: executables have started to become effectively read only after creating (and executing?) them. So any subsequent modification / deletion fails. This is entirely unrelated to Rust.

[mejrs]

And someone else encountered it here with Python: https://github.com/PyO3/pyo3/runs/3601829130?check_suite_focus=true

I got another one: https://github.com/PyO3/pyo3/runs/3590459443?check_suite_focus=true which seems to be the earliest failure so far (at 13th Sept 20:30 CEST).

[ehuss]

I have opened actions/runner-images#4086 to see if maybe we can get some input about the configuration of the windows images.