Add CUDA GPU coredump support to nvproxy#13650
Open
copybara-service[bot] wants to merge 1 commit into
Open
Conversation
Adds the five missing `ioctls` required to generate [CUDA GPU coredumps](https://docs.nvidia.com/cuda/cuda-gdb/index.html#gpu-core-dump-support) using `CUDA_ENABLE_COREDUMP_ON_EXCEPTION=1`, all on the `GT200_DEBUGGER` class. All the new ioctls simple pass-through controls except `DEBUG_READ_MEMORY`. They are gated on `compute,utility`, consistent with the existing `NV83DE` debug controls (`SET_EXCEPTION_MASK`, `READ_ALL_SM_ERROR_STATES`, `CLEAR_ALL_SM_ERROR_STATES`). Coredump generation additionally uses `NV2080_CTRL_CMD_GPU_EXEC_REG_OPS` (already supported), so the feature requires running with `--nvproxy-allowed-driver-capabilities=all,profiling`. In order for this to work, I added a `ctrlDebugReadMemory` handler that translates the embedded `Buffer` pointer in `NV83DE_CTRL_DEBUG_READ_MEMORY_PARAMS` by substituting a sentry-allocated buffer of the requested `Length` before invoking the host driver and copying the result back out to the application, mirroring the driver's own embedded-pointer copy logic in `embedded_param_copy.c` (which disables the `RMAPI_PARAM_COPY_MAX_PARAMS_SIZE` cap for this control). Tested on an A10G with a faulting Triton kernel: the generated `.nvcudmp` matches native (runc) byte-for-byte in size, and `cuda-gdb` fully decodes it (`CUDA Exception: Warp Illegal Address`, faulting PC/kernel/warp/lane and backtrace). Without this change, coredump generation silently produces an empty dump (`No exception was found on any device`). Assisted-by: Claude FUTURE_COPYBARA_INTEGRATE_REVIEW=#13644 from luiscape:luis/gpucoredump-support f41356e PiperOrigin-RevId: 944582537
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add CUDA GPU coredump support to nvproxy
Adds the five missing
ioctlsrequired to generate CUDA GPU coredumps usingCUDA_ENABLE_COREDUMP_ON_EXCEPTION=1, all on theGT200_DEBUGGERclass.All the new ioctls simple pass-through controls except
DEBUG_READ_MEMORY. They are gated oncompute,utility, consistent with the existingNV83DEdebug controls (SET_EXCEPTION_MASK,READ_ALL_SM_ERROR_STATES,CLEAR_ALL_SM_ERROR_STATES).Coredump generation additionally uses
NV2080_CTRL_CMD_GPU_EXEC_REG_OPS(already supported), so the feature requires running with--nvproxy-allowed-driver-capabilities=all,profiling.In order for this to work, I added a
ctrlDebugReadMemoryhandler that translates the embeddedBufferpointer inNV83DE_CTRL_DEBUG_READ_MEMORY_PARAMSby substituting a sentry-allocated buffer of the requestedLengthbefore invoking the host driver and copying the result back out to the application, mirroring the driver's own embedded-pointer copy logic inembedded_param_copy.c(which disables theRMAPI_PARAM_COPY_MAX_PARAMS_SIZEcap for this control).Tested on an A10G with a faulting Triton kernel: the generated
.nvcudmpmatches native (runc) byte-for-byte in size, andcuda-gdbfully decodes it (CUDA Exception: Warp Illegal Address, faulting PC/kernel/warp/lane and backtrace). Without this change, coredump generation silently produces an empty dump (No exception was found on any device).Assisted-by: Claude
FUTURE_COPYBARA_INTEGRATE_REVIEW=#13644 from luiscape:luis/gpucoredump-support f41356e