[rust] Allow cross-compilation of selenium-manager on all platforms#17586
Open
AutomatedTester wants to merge 7 commits into
Open
[rust] Allow cross-compilation of selenium-manager on all platforms#17586AutomatedTester wants to merge 7 commits into
AutomatedTester wants to merge 7 commits into
Conversation
selenium-ci
added
C-py
4 tasks
5 tasks
Upgrades the Bazel Rust toolchain stack to enable hermetic cross-compilation of selenium-manager for every supported OS/arch combination: Build system changes: - Upgrade llvm 0.7.9 → 0.8.0 and rules_rs 0.0.61 → 0.0.76 - Register hermetic LLVM exec toolchains for Linux, macOS, and Windows so all C/C++ compilation in the exec config (process_wrapper, build scripts) is version-pinned and reproducible across CI runners - Add `experimental_stub_libgcc_s` patch for llvm 0.8.0 so the flag propagates to exec-config builds (fixes process_wrapper link on Linux) - Disable local_config_cc auto-detection globally; LLVM toolchain now covers all platforms (macOS is viable because the ring fix removed aws-lc-sys) - Windows uses the gnullvm ABI — the hermetic LLVM package produces correct PE binaries without a separately-provisioned Visual Studio installation - Add bzip2, xz, zstd BCR deps so *-sys crates use the hermetic CC toolchain - Patch windows-link and windows-targets to remove include_str! doc attrs that fail in Bazel's source-only sandbox Cross-compilation targets in rust/BUILD.bazel: - platform_transition_binary for each (os, arch) pair using musl on Linux (self-contained, no glibc dependency) and gnullvm on Windows - selenium-manager-bundle collects all variants; stub binaries stand in for targets that are skipped on the current host Binding updates (Java, Python, Ruby, .NET, JS): - All selenium_manager helpers updated to probe per-(os, arch) paths from the bundle and fall back to the legacy single-file location - .NET nuspec/csproj updated with per-RID native asset paths
AutomatedTester
force-pushed
the
worktree-pr-17427-cross-compile
branch
from
ebcdc24 to
f1cc57a
Compare
Member
|
It's suboptimal, but I'm ok with Windows or Mac not being able to cross compile everything so long as Linux can do it all. The primary missing piece as I see it is the lack of x86 support for Mac. Ideally we can continue to release a fat binary instead of requiring the bindings to pick between them, but I guess the bindings fix is easy enough if we need to go that route. |
…nager - Add x86_64-apple-darwin as a cross-compile target in rust/BUILD.bazel, common/BUILD.bazel, and common/manager/BUILD.bazel - Remove the hard rejection of macOS x86_64 from all language bindings (Java, Python, Ruby, JavaScript, .NET) - Add macos-x86_64 binary paths to .NET nuspec and props files - Add unit tests for macOS x86_64 path resolution in Python and Ruby - Fix MODULE.bazel formatting (blank line after llvm_toolchains extension)
Rename unused os_name parameter to _os_name and reformat case statement to rubocop-preferred style.
Switching from local_windows_gnullvm to local_windows_msvc on Windows CI. The gnullvm host platform triggers LLVM's bootstrap-process-wrapper Starlark transition, placing process_wrapper in a different exec config than proc-macro dependencies (time_macros, scroll_derive, etc.). This causes rustc E0463 "can't find crate" errors for every proc-macro. With local_windows_msvc as the host, native Rust builds go through x86_64-pc-windows-msvc (present on all GitHub-hosted Windows runners). Cross-compile targets in //rust still transition to x86_64-pc-windows-gnullvm via platform_transition_binary so release binaries keep the gnullvm ABI.
…ion_binary On Windows hosts, platform_transition_binary causes process_wrapper and proc-macro DLLs (time_macros, scroll_derive) to land in different exec-config output directories (-ST-<hash> vs non-ST), so rustc cannot find the DLLs. The bundle already handles this via _bundle_native_windows_* targets and the _selenium-manager-stub genrule; only the selenium-manager alias was still pointing at the transitioned targets. On a local_windows_gnullvm host the native rust_binary already compiles as x86_64-pc-windows-gnullvm, so pointing the alias at _selenium-manager-bin directly produces the correct gnullvm PE binary without any transition. Also reverts the local_windows_msvc host_platform change (that caused a different failure: LLVM clang++ receiving MSVC-style linker arguments).
…ergence local_windows_gnullvm as host drags in LLVM's bootstrap-process-wrapper Starlark transition, which puts process_wrapper in a different exec config (with -ST-<hash> suffix) from proc-macro DLLs (time_macros, scroll_derive, etc.), causing E0463 on all Windows builds including unit tests. local_windows_msvc avoids this bootstrap transition entirely. The Windows selenium-manager alias already points to _selenium-manager-bin directly (no platform_transition_binary), so no ABI mismatch occurs: the binary builds natively for x86_64-pc-windows-msvc without any toolchain mixing. The previous local_windows_msvc attempt (f5ed58f) failed because the alias still pointed to platform_transition_binary to gnullvm, mixing MSVC exec toolchain with gnullvm target. With the alias now using _selenium- manager-bin directly that cross-ABI path is gone.
Windows CI pinned only --host_platform=//:local_windows_gnullvm, leaving the target platform as Bazel's auto-detected x64_windows. That platform lacks the @llvm//constraints/windows/abi:gnullvm and crt:msvcrt constraints, so the Rust toolchain resolved for the target config was a different variant than the one resolved for the exec (host) config. The visible symptom: compiling crates like `scroll` (target config x64_windows-fastbuild) referenced their proc-macro DLLs in the exec config (local_windows_gnullvm-opt-exec). Because the two configs resolved mismatched toolchains, the gnullvm proc-macro DLL (scroll_derive, time_macros) could not be loaded by the target rustc and failed with E0463: can't find crate. Pinning --platforms to //:local_windows_gnullvm as well puts the whole build on a single gnullvm toolchain, exactly as the RBE config already pins both --host_platform and --platforms. The extra abi/crt constraints only add to the host-inherited platform, so non-Rust toolchains (Java, .NET, Python) resolve unchanged. Also removes the unused local_windows_msvc platform: the hermetic LLVM Windows linker is hardwired to gnu-mode clang++ (-target x86_64-w64-windows-gnu) regardless of the abi constraint, so an msvc host can never link rustc's MSVC-style output. gnullvm is the only viable host, making that platform dead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Taking over #17427 from @shs96c who was unable to complete it.
This PR enables cross-compilation of
selenium-managerfrom any host platform to all target platforms (Linux x86_64/aarch64, macOS aarch64, Windows x86_64/aarch64) using the hermetic LLVM toolchain.Key changes (from Simon's original work)
rules_rs0.0.61 → 0.0.76 andllvm0.7.9 → 0.8.0rust/BUILD.bazelFixes applied in this PR (on top of Simon's work)
windows-link/windows-targetspatches: The original patches changedinclude_str!("../readme.md")→include_str!("../README.md")but the crate only shipsreadme.mdand Bazel's Rust sandbox doesn't include non-source files anyway. Fixed by removing the doc attribute entirely.clang++.exewhich can't handle MSVC-style linker flags (/NOLOGO,.libfiles). Re-enabled MSVC CC auto-detection on Windows so Visual Studio'slink.exeis used for MSVC targets.Known remaining issues
aws-lc-sys(pulled in transitively throughreqwest→rustls→rustls-platform-verifier) requiresCoreServices.frameworkwhich is not in the hermetic LLVM macOS SDK sysroot. Adding the aarch64 macOS exec LLVM toolchain (new in this PR) causes this to surface. Fixing requires either a different TLS cert backend or a richer macOS sysroot — tracked as a follow-up.Closes #17427