Ship the CUDA delegate as its own library - #21531
Open
shoumikhin wants to merge 12 commits into
Open
Conversation
Contributor
Author
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21531
Note: Links to docs will display an error until the docs builds have been completed. ❌ 271 New Failures, 4 Cancelled Jobs, 3 Pending, 1 Unrelated Failure, 71 Unclassified FailuresAs of commit d46bb95 with merge base d632341 ( NEW FAILURES - The following jobs have failed:
UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:
CANCELLED JOBS - The following jobs were cancelled. Please retry:
FLAKY - The following job failed but was likely due to flakiness present on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
This was referenced Jul 31, 2026
shoumikhin
added a commit
that referenced
this pull request
Aug 1, 2026
The CUDA delegate is compiled into whichever component links it, so a C++ application cannot use it without building from source, and the shim layer it depends on ends up duplicated across several shipped libraries. Build the delegate as a shared library and ship it in the wheel, so a process has one copy and both the Python bindings and a C++ application can link the same one. The CUDA runtime itself is not bundled; it continues to come from the environment. Two things were needed to make that actually reduce duplication: The platform helper resolved the ExecuTorch core statically, which would give the delegate its own copy of the backend registry. It now resolves the runtime from the shared runtime library instead. The shim library force-links its shim objects so their symbols survive, which is correct, but it did so as a public link option. That propagated to every consumer, so each one embedded another copy of the same shim code. Making it private keeps the symbols in the library that owns them while consumers resolve against it, which takes a representative shim symbol from three definitions down to one. Registration still happens through a static initializer, and the delegate is retained on the link line so that initializer runs even though no symbol from it is referenced directly. This is gated on the existing `EXECUTORCH_BUILD_SHARED` option, so only the Linux wheel changes; every other build keeps linking static libraries exactly as before. Test plan: The wheel smoke test now asserts that exactly one shipped library defines the CUDA delegate, alongside the existing backend-registry, thread-pool, CPU kernel, and XNNPACK assertions. Because the delegate is only present in wheels built with CUDA, that assertion skips cleanly when it is absent rather than failing. All three behaviors were confirmed: it passes on a wheel built with this change, it fails on a wheel built before it (correctly reporting three definers by name), and it skips on a wheel built without CUDA. The delegate's own methods are weak symbols, so the assertion checks a strong symbol from the shim layer instead. Built the wheel with CUDA enabled from a clean checkout and verified against a fresh virtual environment with a normal dependency-resolving install: - The wheel ships the delegate as its own versioned library next to the runtime, the thread pool, the CPU kernels, and the XNNPACK delegate. - `nm -DC` across every shipped shared object shows exactly one definition of a representative delegate symbol, down from three. - Only the runtime library defines the backend registry, so the delegate does not introduce a second one. - The thread-pool, CPU kernel, and XNNPACK assertions all still hold with the CUDA delegate loaded. - With `EXECUTORCH_BUILD_SHARED` off, the delegate remains a static library and no new shared object is produced, so every build that does not opt in is unaffected. Note for anyone building with CUDA: the build needs an explicit `CMAKE_CUDA_ARCHITECTURES` for compute capability 6.1 or newer, otherwise an integer dot-product intrinsic used by one of the kernels does not compile. That is independent of this change. ghstack-source-id: 964e8af ghstack-comment-id: 5147959548 Pull-Request: #21531
shoumikhin
added a commit
that referenced
this pull request
Aug 1, 2026
The CUDA delegate is compiled into whichever component links it, so a C++ application cannot use it without building from source, and the shim layer it depends on ends up duplicated across several shipped libraries. Build the delegate as a shared library and ship it in the wheel, so a process has one copy and both the Python bindings and a C++ application can link the same one. The CUDA runtime itself is not bundled; it continues to come from the environment. Two things were needed to make that actually reduce duplication: The platform helper resolved the ExecuTorch core statically, which would give the delegate its own copy of the backend registry. It now resolves the runtime from the shared runtime library instead. The shim library force-links its shim objects so their symbols survive, which is correct, but it did so as a public link option. That propagated to every consumer, so each one embedded another copy of the same shim code. Making it private keeps the symbols in the library that owns them while consumers resolve against it, which takes a representative shim symbol from three definitions down to one. Registration still happens through a static initializer, and the delegate is retained on the link line so that initializer runs even though no symbol from it is referenced directly. This is gated on the existing `EXECUTORCH_BUILD_SHARED` option, so only the Linux wheel changes; every other build keeps linking static libraries exactly as before. Test plan: The wheel smoke test now asserts that exactly one shipped library defines the CUDA delegate, alongside the existing backend-registry, thread-pool, CPU kernel, and XNNPACK assertions. Because the delegate is only present in wheels built with CUDA, that assertion skips cleanly when it is absent rather than failing. All three behaviors were confirmed: it passes on a wheel built with this change, it fails on a wheel built before it (correctly reporting three definers by name), and it skips on a wheel built without CUDA. The delegate's own methods are weak symbols, so the assertion checks a strong symbol from the shim layer instead. Built the wheel with CUDA enabled from a clean checkout and verified against a fresh virtual environment with a normal dependency-resolving install: - The wheel ships the delegate as its own versioned library next to the runtime, the thread pool, the CPU kernels, and the XNNPACK delegate. - `nm -DC` across every shipped shared object shows exactly one definition of a representative delegate symbol, down from three. - Only the runtime library defines the backend registry, so the delegate does not introduce a second one. - The thread-pool, CPU kernel, and XNNPACK assertions all still hold with the CUDA delegate loaded. - With `EXECUTORCH_BUILD_SHARED` off, the delegate remains a static library and no new shared object is produced, so every build that does not opt in is unaffected. Note for anyone building with CUDA: the build needs an explicit `CMAKE_CUDA_ARCHITECTURES` for compute capability 6.1 or newer, otherwise an integer dot-product intrinsic used by one of the kernels does not compile. That is independent of this change. ghstack-source-id: e5585aa ghstack-comment-id: 5147959548 Pull-Request: #21531
shoumikhin
added a commit
that referenced
this pull request
Aug 1, 2026
The CUDA delegate is compiled into whichever component links it, so a C++ application cannot use it without building from source, and the shim layer it depends on ends up duplicated across several shipped libraries. Build the delegate as a shared library and ship it in the wheel, so a process has one copy and both the Python bindings and a C++ application can link the same one. The CUDA runtime itself is not bundled; it continues to come from the environment. Two things were needed to make that actually reduce duplication: The platform helper resolved the ExecuTorch core statically, which would give the delegate its own copy of the backend registry. It now resolves the runtime from the shared runtime library instead. The shim library force-links its shim objects so their symbols survive, which is correct, but it did so as a public link option. That propagated to every consumer, so each one embedded another copy of the same shim code. Making it private keeps the symbols in the library that owns them while consumers resolve against it, which takes a representative shim symbol from three definitions down to one. Registration still happens through a static initializer, and the delegate is retained on the link line so that initializer runs even though no symbol from it is referenced directly. This is gated on the existing `EXECUTORCH_BUILD_SHARED` option, so only the Linux wheel changes; every other build keeps linking static libraries exactly as before. Test plan: The wheel smoke test now asserts that exactly one shipped library defines the CUDA delegate, alongside the existing backend-registry, thread-pool, CPU kernel, and XNNPACK assertions. Because the delegate is only present in wheels built with CUDA, that assertion skips cleanly when it is absent rather than failing. All three behaviors were confirmed: it passes on a wheel built with this change, it fails on a wheel built before it (correctly reporting three definers by name), and it skips on a wheel built without CUDA. The delegate's own methods are weak symbols, so the assertion checks a strong symbol from the shim layer instead. Built the wheel with CUDA enabled from a clean checkout and verified against a fresh virtual environment with a normal dependency-resolving install: - The wheel ships the delegate as its own versioned library next to the runtime, the thread pool, the CPU kernels, and the XNNPACK delegate. - `nm -DC` across every shipped shared object shows exactly one definition of a representative delegate symbol, down from three. - Only the runtime library defines the backend registry, so the delegate does not introduce a second one. - The thread-pool, CPU kernel, and XNNPACK assertions all still hold with the CUDA delegate loaded. - With `EXECUTORCH_BUILD_SHARED` off, the delegate remains a static library and no new shared object is produced, so every build that does not opt in is unaffected. Note for anyone building with CUDA: the build needs an explicit `CMAKE_CUDA_ARCHITECTURES` for compute capability 6.1 or newer, otherwise an integer dot-product intrinsic used by one of the kernels does not compile. That is independent of this change. ghstack-source-id: 475d170 ghstack-comment-id: 5147959548 Pull-Request: #21531
shoumikhin
added a commit
that referenced
this pull request
Aug 1, 2026
The CUDA delegate is compiled into whichever component links it, so a C++ application cannot use it without building from source, and the shim layer it depends on ends up duplicated across several shipped libraries. Build the delegate as a shared library and ship it in the wheel, so a process has one copy and both the Python bindings and a C++ application can link the same one. The CUDA runtime itself is not bundled; it continues to come from the environment. Two things were needed to make that actually reduce duplication: The platform helper resolved the ExecuTorch core statically, which would give the delegate its own copy of the backend registry. It now resolves the runtime from the shared runtime library instead. The shim library force-links its shim objects so their symbols survive, which is correct, but it did so as a public link option. That propagated to every consumer, so each one embedded another copy of the same shim code. Making it private keeps the symbols in the library that owns them while consumers resolve against it, which takes a representative shim symbol from three definitions down to one. Registration still happens through a static initializer, and the delegate is retained on the link line so that initializer runs even though no symbol from it is referenced directly. This is gated on the existing `EXECUTORCH_BUILD_SHARED` option, so only the Linux wheel changes; every other build keeps linking static libraries exactly as before. Test plan: The wheel smoke test now asserts that exactly one shipped library defines the CUDA delegate, alongside the existing backend-registry, thread-pool, CPU kernel, and XNNPACK assertions. Because the delegate is only present in wheels built with CUDA, that assertion skips cleanly when it is absent rather than failing. All three behaviors were confirmed: it passes on a wheel built with this change, it fails on a wheel built before it (correctly reporting three definers by name), and it skips on a wheel built without CUDA. The delegate's own methods are weak symbols, so the assertion checks a strong symbol from the shim layer instead. Built the wheel with CUDA enabled from a clean checkout and verified against a fresh virtual environment with a normal dependency-resolving install: - The wheel ships the delegate as its own versioned library next to the runtime, the thread pool, the CPU kernels, and the XNNPACK delegate. - `nm -DC` across every shipped shared object shows exactly one definition of a representative delegate symbol, down from three. - Only the runtime library defines the backend registry, so the delegate does not introduce a second one. - The thread-pool, CPU kernel, and XNNPACK assertions all still hold with the CUDA delegate loaded. - With `EXECUTORCH_BUILD_SHARED` off, the delegate remains a static library and no new shared object is produced, so every build that does not opt in is unaffected. Note for anyone building with CUDA: the build needs an explicit `CMAKE_CUDA_ARCHITECTURES` for compute capability 6.1 or newer, otherwise an integer dot-product intrinsic used by one of the kernels does not compile. That is independent of this change. ghstack-source-id: 5fb1483 ghstack-comment-id: 5147959548 Pull-Request: #21531
shoumikhin
added a commit
that referenced
this pull request
Aug 1, 2026
The CUDA delegate is compiled into whichever component links it, so a C++ application cannot use it without building from source, and the shim layer it depends on ends up duplicated across several shipped libraries. Build the delegate as a shared library and ship it in the wheel, so a process has one copy and both the Python bindings and a C++ application can link the same one. The CUDA runtime itself is not bundled; it continues to come from the environment. Two things were needed to make that actually reduce duplication: The platform helper resolved the ExecuTorch core statically, which would give the delegate its own copy of the backend registry. It now resolves the runtime from the shared runtime library instead. The shim library force-links its shim objects so their symbols survive, which is correct, but it did so as a public link option. That propagated to every consumer, so each one embedded another copy of the same shim code. Making it private keeps the symbols in the library that owns them while consumers resolve against it, which takes a representative shim symbol from three definitions down to one. Registration still happens through a static initializer, and the delegate is retained on the link line so that initializer runs even though no symbol from it is referenced directly. This is gated on the existing `EXECUTORCH_BUILD_SHARED` option, so only the Linux wheel changes; every other build keeps linking static libraries exactly as before. Test plan: The wheel smoke test now asserts that exactly one shipped library defines the CUDA delegate, alongside the existing backend-registry, thread-pool, CPU kernel, and XNNPACK assertions. Because the delegate is only present in wheels built with CUDA, that assertion skips cleanly when it is absent rather than failing. All three behaviors were confirmed: it passes on a wheel built with this change, it fails on a wheel built before it (correctly reporting three definers by name), and it skips on a wheel built without CUDA. The delegate's own methods are weak symbols, so the assertion checks a strong symbol from the shim layer instead. Built the wheel with CUDA enabled from a clean checkout and verified against a fresh virtual environment with a normal dependency-resolving install: - The wheel ships the delegate as its own versioned library next to the runtime, the thread pool, the CPU kernels, and the XNNPACK delegate. - `nm -DC` across every shipped shared object shows exactly one definition of a representative delegate symbol, down from three. - Only the runtime library defines the backend registry, so the delegate does not introduce a second one. - The thread-pool, CPU kernel, and XNNPACK assertions all still hold with the CUDA delegate loaded. - With `EXECUTORCH_BUILD_SHARED` off, the delegate remains a static library and no new shared object is produced, so every build that does not opt in is unaffected. Note for anyone building with CUDA: the build needs an explicit `CMAKE_CUDA_ARCHITECTURES` for compute capability 6.1 or newer, otherwise an integer dot-product intrinsic used by one of the kernels does not compile. That is independent of this change. ghstack-source-id: 1abba89 ghstack-comment-id: 5147959548 Pull-Request: #21531
shoumikhin
added a commit
that referenced
this pull request
Aug 1, 2026
The CUDA delegate is compiled into whichever component links it, so a C++ application cannot use it without building from source, and the shim layer it depends on ends up duplicated across several shipped libraries. Build the delegate as a shared library and ship it in the wheel, so a process has one copy and both the Python bindings and a C++ application can link the same one. The CUDA runtime itself is not bundled; it continues to come from the environment. Two things were needed to make that actually reduce duplication: The platform helper resolved the ExecuTorch core statically, which would give the delegate its own copy of the backend registry. It now resolves the runtime from the shared runtime library instead. The shim library force-links its shim objects so their symbols survive, which is correct, but it did so as a public link option. That propagated to every consumer, so each one embedded another copy of the same shim code. Making it private keeps the symbols in the library that owns them while consumers resolve against it, which takes a representative shim symbol from three definitions down to one. Registration still happens through a static initializer, and the delegate is retained on the link line so that initializer runs even though no symbol from it is referenced directly. This is gated on the existing `EXECUTORCH_BUILD_SHARED` option, so only the Linux wheel changes; every other build keeps linking static libraries exactly as before. Test plan: The wheel smoke test now asserts that exactly one shipped library defines the CUDA delegate, alongside the existing backend-registry, thread-pool, CPU kernel, and XNNPACK assertions. Because the delegate is only present in wheels built with CUDA, that assertion skips cleanly when it is absent rather than failing. All three behaviors were confirmed: it passes on a wheel built with this change, it fails on a wheel built before it (correctly reporting three definers by name), and it skips on a wheel built without CUDA. The delegate's own methods are weak symbols, so the assertion checks a strong symbol from the shim layer instead. Built the wheel with CUDA enabled from a clean checkout and verified against a fresh virtual environment with a normal dependency-resolving install: - The wheel ships the delegate as its own versioned library next to the runtime, the thread pool, the CPU kernels, and the XNNPACK delegate. - `nm -DC` across every shipped shared object shows exactly one definition of a representative delegate symbol, down from three. - Only the runtime library defines the backend registry, so the delegate does not introduce a second one. - The thread-pool, CPU kernel, and XNNPACK assertions all still hold with the CUDA delegate loaded. - With `EXECUTORCH_BUILD_SHARED` off, the delegate remains a static library and no new shared object is produced, so every build that does not opt in is unaffected. Note for anyone building with CUDA: the build needs an explicit `CMAKE_CUDA_ARCHITECTURES` for compute capability 6.1 or newer, otherwise an integer dot-product intrinsic used by one of the kernels does not compile. That is independent of this change. ghstack-source-id: 6d6da98 ghstack-comment-id: 5147959548 Pull-Request: #21531
Contributor
There was a problem hiding this comment.
Pull request overview
This PR changes the CUDA delegate packaging so it is built as its own shared library (when EXECUTORCH_BUILD_SHARED is enabled) and shipped in the Linux wheel, reducing duplicated shim/runtime code across shipped shared objects and enabling both Python and standalone C++ apps to link the same delegate binary.
Changes:
- Ship
libexecutorch_cuda_backend.so.<major>in the wheel alongside the runtime/XNNPACK/CPU-kernel shared libraries (when CUDA + shared build are enabled). - Adjust CUDA backend CMake to build the delegate as SHARED under
EXECUTORCH_BUILD_SHAREDand set versioning/RPATH suitable for wheel layout. - Extend the wheel smoke test to (a) report shipped library composition, (b) verify shipped
.sodependencies are satisfiable by the wheel/environment, and (c) assert single-definer behavior for the CUDA delegate when present.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| setup.py | Adds wheel installation rule to ship the CUDA delegate shared library when built. |
| backends/cuda/CMakeLists.txt | Builds the CUDA delegate as a shared library under EXECUTORCH_BUILD_SHARED, sets SONAME/version and RPATH for wheel placement. |
| .ci/scripts/wheel/test_cpp_sdk.py | Adds wheel composition/dependency reporting and a single-definer assertion for the CUDA delegate (optional when absent). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The CUDA delegate is compiled into whichever component links it, so a C++
application cannot use it without building from source, and the shim layer it
depends on ends up duplicated across several shipped libraries.
Build the delegate as a shared library and ship it in the wheel, so a process
has one copy and both the Python bindings and a C++ application can link the
same one. The CUDA runtime itself is not bundled; it continues to come from the
environment.
Two things were needed to make that actually reduce duplication:
The platform helper resolved the ExecuTorch core statically, which would give
the delegate its own copy of the backend registry. It now resolves the runtime
from the shared runtime library instead.
The shim library force-links its shim objects so their symbols survive, which is
correct, but it did so as a public link option. That propagated to every
consumer, so each one embedded another copy of the same shim code. Making it
private keeps the symbols in the library that owns them while consumers resolve
against it, which takes a representative shim symbol from three definitions down
to one.
Registration still happens through a static initializer, and the delegate is
retained on the link line so that initializer runs even though no symbol from it
is referenced directly.
This is gated on the existing
EXECUTORCH_BUILD_SHAREDoption, so only theLinux wheel changes; every other build keeps linking static libraries exactly as
before.
Test plan:
The wheel smoke test now asserts that exactly one shipped library defines the
CUDA delegate, alongside the existing backend-registry, thread-pool, CPU kernel,
and XNNPACK assertions. Because the delegate is only present in wheels built
with CUDA, that assertion skips cleanly when it is absent rather than failing.
All three behaviors were confirmed: it passes on a wheel built with this change,
it fails on a wheel built before it (correctly reporting three definers by
name), and it skips on a wheel built without CUDA.
The delegate's own methods are weak symbols, so the assertion checks a strong
symbol from the shim layer instead.
Built the wheel with CUDA enabled from a clean checkout and verified against a
fresh virtual environment with a normal dependency-resolving install:
the thread pool, the CPU kernels, and the XNNPACK delegate.
nm -DCacross every shipped shared object shows exactly one definition of arepresentative delegate symbol, down from three.
not introduce a second one.
CUDA delegate loaded.
EXECUTORCH_BUILD_SHAREDoff, the delegate remains a static library andno new shared object is produced, so every build that does not opt in is
unaffected.
Note for anyone building with CUDA: the build needs an explicit
CMAKE_CUDA_ARCHITECTURESfor compute capability 6.1 or newer, otherwise aninteger dot-product intrinsic used by one of the kernels does not compile. That
is independent of this change.