Fix cuPointerGetAttribute(s) returning 0 for CU_POINTER_ATTRIBUTE_DEVICE_ORDINAL - #2752
Open
fnrizzi wants to merge 1 commit into
Open
Fix cuPointerGetAttribute(s) returning 0 for CU_POINTER_ATTRIBUTE_DEVICE_ORDINAL#2752fnrizzi wants to merge 1 commit into
fnrizzi wants to merge 1 commit into
Conversation
Contributor
…ICE_ORDINAL _HelperCUpointer_attribute writes DEVICE_ORDINAL into the signed _int member in __cinit__ but pyObj() read it back from the unsigned _uint member, which is never set, so the binding always returned 0 instead of the real device. This makes any pointer look like it lives on device 0, breaking device resolution for CAI consumers (e.g. cuda.core / nccl.core) on multi-GPU systems. Read _int for DEVICE_ORDINAL to match the write path, and add a multi-GPU regression test asserting the reported ordinal matches the allocation device (existing tests missed it: they only checked call success or used ptr=0). Signed-off-by: Francesco Rizzi <frizzi@nvidia.com>
fnrizzi
force-pushed
the
fix/pointer-device-ordinal
branch
from
September 2, 2026 15:36
65c66ea to
d0543fc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cuPointerGetAttributeandcuPointerGetAttributesalways return0forCU_POINTER_ATTRIBUTE_DEVICE_ORDINAL, regardless of which device the pointer was actually allocated on.The root cause is I think a write/read field mismatch in
_HelperCUpointer_attribute(cuda_bindings/cuda/bindings/_lib/utils.pxi):__cinit__allocates storage forDEVICE_ORDINALin the signed_intmember and points
_cptrat it, so the driver writes the ordinal there.pyObj()readsDEVICE_ORDINALback from the unsigned_uintmember,which is never written and stays
0.Because the value is read from the wrong member, the ordinal is always
0.Fix
Read
DEVICE_ORDINALfrom_intinpyObj()so the read path matches the write path in__cinit__.Test
Adds a regression test that allocates on each visible GPU and asserts the
ordinal reported by both the singular and plural bindings matches the
allocation device (ground truth from
cudaPointerGetAttributes).Existing tests missed this because they only asserted call success / that
singular and plural agree (both share the buggy read path and both returned 0).
The test needs >= 2 GPUs and skips otherwise, since on device 0 a wrong
0isindistinguishable from the correct value.
Below are also minimal reproducers in c and python . the c one works correctly , the python does not and motivated this MR.
when i run these, before the fix of this MR, I get:
and