cuda.core: define the error handling policy and report failures that cannot be raised - #2759
Draft
Andy-Jost wants to merge 2 commits into
Draft
cuda.core: define the error handling policy and report failures that cannot be raised#2759Andy-Jost wants to merge 2 commits into
Andy-Jost wants to merge 2 commits into
Conversation
Run context-sensitive Device operations against the Device's bound context while preserving caller state. Centralize context-aware cleanup and synchronous allocation handling so resource lifetimes remain correct.
Contributor
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Andy-Jost
commented
Sep 3, 2026
Comment on lines
-1311
to
+1316
| HANDLE_RETURN(cydriver.cuCtxPopCurrent(&prev_ctx)) | ||
| HANDLE_RETURN(cydriver.cuCtxPushCurrent(curr_ctx)) | ||
| # cuCtxSetCurrent replaces the top of the thread's context stack | ||
| # in one driver call (or binds ctx when nothing is current), so | ||
| # a failure leaves the previous context current instead of | ||
| # leaving the thread with no context, as a failed pop-then-push | ||
| # would. | ||
| HANDLE_RETURN(cydriver.cuCtxSetCurrent(curr_ctx)) |
Contributor
Author
There was a problem hiding this comment.
I'm not certain whether the two-call sequence here is needed for a reason I don't see. It would be good to confirm this change.
Contributor
Author
|
/ok to test |
…cannot be raised Write down how cuda.core handles CUDA failures (docs/source/error_handling.rst for users, a "Failure handling" section in AGENTS.md and _cpp/DESIGN.md for contributors) and bring the code into line with it: - Add cuda.core.CUDAWarning, emitted for CUDA errors that cannot be raised (destructors, CUDA callbacks, cleanup after an earlier failure). The C++ handle layer reports through one helper that uses the Python warnings machinery when the interpreter is usable, delivers an escalated warning as an unraisable exception, and falls back to stderr otherwise. CUDA_ERROR_DEINITIALIZED is not reported. - Wrap every destroy call made from a deleter (pw_*) so its failure is reported instead of discarded, including memory pools, green contexts, graphs, graph execs, graphics resources, the linker, user objects, the NVRTC/NVVM/nvJitLink handles and file descriptors; release the GIL around the compiler-handle destroys like the CUDA ones. - When the caller's context cannot be restored after a successful operation, undo the creation and raise a CUDAError that says which context is current; report the same failure as a warning in deleters; report a skipped context-sensitive undo instead of leaking silently. - Add context_get_device and graph_node_set_params so Stream_get_ctx_device and _set_definition_node_params stop hand-rolling cuCtxPush/Pop/SetCurrent. The node update now publishes its attachment before raising a restoration failure, closing a window that left the node referencing released owners. - Device.set_current(ctx) switches with a single cuCtxSetCurrent, so a failure leaves the previous context current and the call works without one. - Report failed cuStreamEndCapture in GraphBuilder.__dealloc__ and failed child-graph rollbacks; warn from _mr_dealloc_callback instead of printing. - Add a test hook that makes the next context restoration fail, tests for the policy, and release notes for 1.3.0. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Andy-Jost
added
documentation
Andy-Jost
force-pushed
the
ajost/error-handling-policy
branch
from
September 3, 2026 03:24
8719151 to
5e8f7e9
Compare
Contributor
Author
|
/ok to test |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Closes #2758 (the error handling policy RFC). Stacked on #2750: the first commit is #2750's head; review the last commit only.
Writes down
cuda.core's failure-handling policy and brings the code into line with it.Policy (docs):
docs/source/error_handling.rst: what an exception fromcuda.coreguarantees (nothing created, caller's context unchanged), how failures that cannot be raised are reported, context restoration failures, sticky errors, interpreter shutdown, and thatcuda.coredoes not abort the process for CUDA errors.cuda_core/AGENTS.md(contributor rules: raise by default, publish before you raise, leak rather than dangle, one reporting channel, no hand-rolled context push/pop, when an abort would be justified and what it must do)._cpp/DESIGN.md: context-scoped operations and non-propagating reporting.Code:
cuda.core.CUDAWarning(aRuntimeWarning), emitted for CUDA errors that cannot be raised. The C++ handle layer reports through one helper (report_cuda_error/report_message) that goes through the Python warnings machinery when the interpreter is usable, delivers an escalated warning as an unraisable exception, and falls back to stderr otherwise.CUDA_ERROR_DEINITIALIZEDis filtered.pw_*) and reported on failure, including the ones cuda.core: make Device methods use their bound context #2750 left silent (memory pools, green contexts, graphs, graph execs, graphics, linker, user objects, NVRTC/NVVM/nvJitLink handles, file descriptors). The compiler-handle deleters now release the GIL like the CUDA ones.CUDAErrornow explains that the caller's context could not be restored and which context is current (thread-local detail consumed by_check_driver_error). In deleters it is aCUDAWarning; a skipped context-sensitive undo is reported instead of leaking silently.context_get_deviceandgraph_node_set_params;Stream_get_ctx_deviceand_set_definition_node_paramsuse them instead of hand-rolled push/pop. The node update now publishes its attachment before raising a restoration failure, closing a dangling-owner window.Device.set_current(ctx)switches with a singlecuCtxSetCurrent(a failure leaves the previous context current; works with no context current).GraphBuilder.__dealloc__and the child-graph embed rollbacks report failures that were silent._mr_dealloc_callbackwarns instead of printing to stderr.cuda.core._resource_handles._set_context_restore_fault_for_testing; newtests/test_error_handling.py;tests/test_memory.pyasserts onCUDAWarninginstead of stderr text.docs/source/release/1.3.0-notes.rst(assumes 1.2.0 has shipped).Checklist
🤖 Generated with Claude Code