Dist Enzyme for macOS with unversioned LLVM dylib

[sgasho]

View all comments

Follow-up to #152768.

• Link LLVM dynamically on MacOS
• Fix a macOS LLVM dylib name mismatch
• Dist Enzyme for MacOS

[ZuseZ4]

@bors delegate try

[rust-bors]

Unknown command "delegate". Run @bors help to see available commands.

[ZuseZ4]

Hmm, not sure about the exact syntax, but delegate itself should have been recognized, maybe not ported to the new bots yet? I'll let Jakub handle it.
For now

@bors try jobs=dist-aarch64-apple

[Kobzol]

@bors delegate=try

[rust-bors]

✌️ @sgasho, you can now perform try builds on this pull request!

You can now post @bors try to start a try build.

[rust-bors]

☀️ Try build successful (CI)
Build commit: a658070 (a65807068ca7fef2605b24756206b9e5dfa49bbd, parent: db1484bdee1cb065b322a5bd512b39c49dadc9e5)

[rust-bors]

☔ The latest upstream changes (presumably #150541) made this pull request unmergeable. Please resolve the merge conflicts.

[sgasho]

@bors try jobs=dist-aarch64-apple

[sgasho]

@bors try jobs=dist-aarch64-apple

[sgasho]

@bors try jobs=dist-aarch64-apple

[rust-bors]

💔 Test for 2c9be7e failed: CI. Failed job:

• try - dist-aarch64-apple (web logs, enhanced plaintext logs)

[sgasho]

@bors try jobs=dist-aarch64-apple

[rust-bors]

☀️ Try build successful (CI)
Build commit: 99e2a89 (99e2a898c1d8463abd8f9fcc03f162d20deb0286, parent: 9f0615c4301206eeba76468f78d34876c4a2997e)

[ZuseZ4]

(please feel free to ping Ralf jung whenever you think you have an artifact that you think should work for them)

[sgasho]

@RalfJung
related: #153077 (comment)
Could you check if Miri builds successfully using the artifact below?

https://github.com/rust-lang/rust/actions/runs/23104042131

[ZuseZ4]

Nice, it seems like Miri passed.

Can you please run some (or all if not too slow) of the UI or pretty tests a few times under the old and new tool chain? I'd expect a diff to be more noise than anything based on what was discussed previously, but we should confirm that the diff isn't too big. Just post all times, but the minimal times are likely the most relevant ones.

[RalfJung]

Yeah looks like linking with rustc_driver still works, nice.

There were I think some other concerns about dynamic linking, Cc @bjorn3 @madsmtm

[bjorn3]

The comment above needs to be updated with an explanation for why we also do this for macOS+libLLVM.dylib.

[sgasho]

Can you please run some (or all if not too slow) of the UI or pretty tests a few times under the old and new tool chain? I'd expect a diff to be more noise than anything based on what was discussed previously, but we should confirm that the diff isn't too big. Just post all times, but the minimal times are likely the most relevant ones.

ah, I made a mistake around how to use new toolchain. I only got pretty test results today.
Here's the results.

try	static linking(old)	toolchain from the latest commit of this PR(new)
1	2.57sec	7.82sec
2	2.89sec	6.63sec
3	2.25sec	7.07sec

I hope I can share the results of ui tests by tomorrow.

[sgasho]

mmm....Am I doing the right thing.
There are no time diffs between old and new ui tests. Both about 20min.

What is the right command I should run?
I ran

./x test --stage 0 \
      --force-rerun \
      --set build.compiletest-allow-stage0=true \
      --set "build.rustc=$(rustup which --toolchain (old or new toolchain name. new one is from CI tarballs from the latest commit) rustc) \
      --set "build.cargo=$(rustup which --toolchain (old or new toolchain name) cargo)" \
      --build-dir /tmp/somewhere \
      tests/ui

[ZuseZ4]

@jieyouxu did you add some logic (like fusing of multiple tests) to the UI test suite? I wouldn't expect it, but I'm also a bit surprised that UI times stayed the same, while pretty tests got so much slower.

@sgasho can you maybe also test some small hello world project or something similar as a tie breaker?

Edit: wait, the new tool chain is probably in a different folder, so I'd guess some MacOS Anti-Virus is checking it and killing performance. I vaguely remember these issues, there should be something in a dev guide about it.

https://nnethercote.github.io/2025/09/04/faster-rust-builds-on-mac.html

Still not sure why it's only showing up in pretty tests, but I guess that could explain the 3x perf difference between the official nightly and your manually downloaded one?

[jieyouxu]

@jieyouxu did you add some logic (like fusing of multiple tests) to the UI test suite? I wouldn't expect it, but I'm also a bit surprised that UI times stayed the same, while pretty tests got so much slower.

Not that I know of, at least not for non-parallel ui tests

[sgasho]

I reran tests/pretty and realized the time difference is likely dominated by failure output.
Both runs executed the same suite (99 tests) and both had 3 failures, but the failing cases were different and I did not notice that.

I did not so much care about the failures because the number of failures were same...I thought it was just due to the time lag between this branch and the old toolchain(nightly). But that was the problem and I overlooked it.

So, I updated the nightly and pull rebased main into this branch. I checked that all 102 tests have been passed in old toolchain(around 2.5sec). However, 4 tests failed in new one(around 6sec).

I'm checking the failures.

[sgasho]

@bors try jobs=dist-aarch64-apple,aarch64-apple,

[sgasho]

However, 4 tests failed in new one(around 6sec).

./x test tests/pretty locally on this branch has been passed in 7sec

[sgasho]

well, I git cloned r-l/r into another directory and tested it with ./x test tests/pretty. It also took 7sec, which means there are no time diffs between these branches.

[ZuseZ4]

Add a comment explaining why we overwrite it here.

[ZuseZ4]

Yeah, I was expecting perf differences between local builds vs CI artifacts, only local vs local or CI vs CI makes sense, with the latter being the source of truth since that's what we distribute. Thanks for investigating and testing all combinations. Seems like we have no significant perf change in these stress tests, and real-world application would mask any latency increase (if it even exists) even better.

A few local test failures are to be expected. If you download a CI artifact and test it on your local clone, then all those tests will fail that were modified between your last sync and the artifact creation time. As long as it's just a few and CI is happy then the PR should be good.

Mads had a few weeks and multiple pings but didn't specify what he meant with "multiple reasons" for not doing this change here: #152768 (comment). However, IIUC he reverted his opinion in #153077 (comment) and only cared about the perf, for which we have new numbers here (aka no change). The miri / downstream failure seems to also be revolved, so I am not aware of any other blockers. The change however unblocks distributing autodiff without increasing rustc artifact size for all users (requested by the infra team), and it also unblocks further projects like distributing clang (or lld) without repeatedly including LLVM.

@sgasho can you address Bjorn and my feedback, squash the commits and remove the enzye line for now? Then Ralf can run a final miri check if he wants and we can approve it in a few days if no one raises any new concerns.

[rust-bors]

☀️ Try build successful (CI)
Build commit: e9082b0 (e9082b0b5e7003739ca4f66d9151b541644e4398, parent: 7218b7fa17e8ef5e99525ae25e6e82224c27039d)