[RFC] cuda.core: a written policy for failures that cannot be raised, cascading failures, and `std::abort`

[Andy-Jost]

Summary

cuda.core has no written policy for what happens when a CUDA call fails in a place where no
Python exception can propagate (a destructor, a CUDA callback, cleanup after an earlier failure),
or when a failure cascades to the point where an invariant such as "the caller's current context is
unchanged" can no longer be maintained. Today the code base answers those questions in at least five
different ways (silently discarded statuses, fprintf(stderr), print(file=sys.stderr),
warnings.warn, raise-after-success), and the review of #2750 raised the question of whether some of
these cases should instead terminate the process with std::abort.

This issue proposes the policy below, records the analysis behind it, and tracks the PR that writes
it into the docs and brings the code into line with it. The short version:

1. Raise whenever an exception can propagate, preserving the original CUresult.
2. Report, do not discard when it cannot: one channel, a new cuda.core.CUDAWarning
   (a RuntimeWarning subclass) that users can filter or escalate.
3. Leak rather than dangle when ownership of a live CUDA resource cannot be established.
4. Publish before you raise: after a driver mutation succeeds, commit whatever keeps it
   memory-safe before raising a later failure.
5. Never abort on a CUDA error, including a failed context restoration. std::abort is reserved
   for internal invariant violations where continuing could corrupt memory and no leak-based fallback
   exists. No such path exists today, and the policy says what one must look like if it is ever added.

Background

The design discussion started from the context save/restore code introduced by #2750, which has to
(1) make the target device's context current, (2) create the resource, (3) restore the caller's
context. Step 3 can fail after step 2 succeeded; the code then tries to undo step 2, which can also
fail; and whether or not the undo succeeds, the caller's context is no longer current, violating the
method's contract. The sequences in the graph code are longer still. The question asked in the
#2750 review thread and in a design discussion with the CCCL and RAPIDS teams was: when failures
cascade like this, should cuda.core ever call std::abort?

The position taken by the CCCL and RAPIDS engineers in that discussion:

• Try hard to give the strong exception guarantee (as if the call never happened); always give at
  least the basic guarantee.
• "If you think the user could resolve the error programmatically, recover gracefully. If your
  callers have no chance of correcting the mistake without rerunning the program, abort is OK."
• Failing to restore the current context was considered by some to be in the second category.

And the Python-side nuance raised in the same discussion: core dumps and gdb are not tools most
Python users reach for, so if a proper Python exception can still be raised, it should be; if it
cannot, faulthandler can at least print a Python traceback on the way out.

What the audit found

Reporting channels in use today, the three context-switch implementations, and the other sites the policy has to cover (expand)

A full audit of cuda_core/cuda/core (C++ handle layer, graph code, memory / device / stream /
event / texture code, and the remaining subsystems) plus the open #2750 branch. Highlights:

Reporting channels in use today

channel	where
status silently discarded in a shared_ptr deleter	cuStreamDestroy, cuEventDestroy, cuMemFree*, cuMemPoolDestroy, cuGreenCtxDestroy, cuDevicePrimaryCtxRelease, cuGraphDestroy, cuGraphExecDestroy, cuGraphicsUnregisterResource, cuLinkDestroy, cuArrayDestroy, cuTexObjectDestroy, cuSurfObjectDestroy, nvrtc/nvvm/nvJitLink destroy, close(fd), user-object releases in rollbacks (resource_handles.cpp)
std::fprintf(stderr, "Warning: ...")	context-restore failure in ~ScopedCurrentContext, PTDS cross-thread deallocation, with_deallocation_context (all from #2526); #2750 adds warn_on_cuda_error and WarnOnFailure wrappers for eight destroy calls, still via stderr
print(..., file=sys.stderr)	_mr_dealloc_callback in _memory/_buffer.pyx (tests assert on the exact text with capfd)
warnings.warn	deprecations and compatibility notices only
raise after the operation already succeeded	GraphNode.destroy, definition node updates, instantiate/update when releasing a staging user object fails
intentional leak	everything during interpreter finalization (py_is_finalizing()), cuLibraryUnload (disabled), deferred cleanup when Py_AddPendingCall is saturated

There is no abort, terminate, exit or Py_FatalError anywhere in cuda_core. The project has
twice treated an unintended std::terminate as a bug (#1489, #2417).

Three independently written scoped-context switches

1. ScopedCurrentContext (main, deleters only): restore failure prints to stderr and continues.
2. cuda.core: make Device methods use their bound context #2750's enter_context/exit_context/invoke_in_context_or_undo/cleanup_in_context:
   restore failure after a successful creation undoes the creation and raises the restore status as
   a bare CUDAError (the message names cuCtxSetCurrent's error code, not what happened); in
   deleters it prints to stderr.
3. _set_definition_node_params (graph/_subclasses.pyx): raw cuCtxGetCurrent/cuCtxSetCurrent
   with the restore in a finally: that HANDLE_RETURNs.

Number 3 has a real hazard: if cuGraphNodeSetParams succeeds and the restore fails, the exception
skips graph_commit_attachment, the prepared attachment is rolled back, and the graph's only
retention of the node's new owners is released while the driver node still references them. A
later instantiate or launch dereferences freed memory. This is the sharpest case in the tree of
"rollback is the wrong recovery"; the right recovery is to publish the attachment first (or leak
it), then raise.

Other findings the policy has to cover:
Device.set_current(ctx) does pop-then-push, so a failed push leaves
the thread with no context; Stream_get_ctx_device does push/query/pop with no try/finally;
GraphBuilder.__dealloc__ discards a failed cuStreamEndCapture; child-graph embed rollbacks
discard a failed cuGraphDestroyNode; a CUresult is sometimes replaced by a generic
RuntimeError; and the texture/surface undo in #2750 can skip cleanup with no report at all.

Why not abort on a failed context restoration

This was the concrete proposal on the table, so it deserves a concrete answer. cuCtxSetCurrent is
documented to fail only with:

• CUDA_ERROR_DEINITIALIZED: the driver is shutting down. Nothing after this point matters, and
  aborting would turn every interpreter exit that races driver teardown into a crash.
• CUDA_ERROR_INVALID_CONTEXT: the caller's context was destroyed while we held its raw value (a
  third party called cuCtxDestroy / cuDevicePrimaryCtxReset). The caller's context cannot be
  current anyway; any work the caller does on it fails loudly with CUDA_ERROR_CONTEXT_IS_DESTROYED.
• a deferred asynchronous launch error (the "may also return error codes from previous, asynchronous
  launches" note). Those are exactly the sticky errors (ILLEGAL_ADDRESS, LAUNCH_FAILED, ...)
  whose documentation says the process must be relaunched and every later call returns the same
  error. A retry therefore cannot succeed, and the wrong-context state is unobservable because no
  later CUDA call succeeds either.

A driver probe on H100 (driver 610.57, CUDA 13.3 bindings 13.2) confirms the picture. After a kernel
was made to fault with CUDA_ERROR_ILLEGAL_ADDRESS, cuCtxGetCurrent, cuCtxSetCurrent (to the
same context, to NULL, and to another device's context), cuCtxPushCurrent and cuCtxPopCurrent
all kept returning CUDA_SUCCESS, while cuMemAlloc, cuStreamCreate, cuEventCreate and
cuCtxSynchronize returned the fault on both devices, and cuDevicePrimaryCtxReset followed by
cuDevicePrimaryCtxRetain returned the fault as well. So the sticky case does not even break
restoration in practice; it makes everything else fail loudly, and only a process restart recovers.
The destroyed-context case does not surface as a restoration failure either: after
cuDevicePrimaryCtxReset, cuCtxSetCurrent on the stale primary handle succeeded and work on it
returned CUDA_ERROR_CONTEXT_IS_DESTROYED until the context was retained again; and after
cuCtxDestroy, cuCtxSetCurrent on the destroyed non-primary handle also returned CUDA_SUCCESS
(its later use is undefined behaviour per the driver docs, and a second bind terminated the process).
On this driver, cuCtxSetCurrent did not fail for any documented reason other than, presumably,
CUDA_ERROR_DEINITIALIZED.

In none of these cases does aborting protect the user from silently wrong results, which is the only
thing an abort buys. What it costs is the Python traceback, orderly shutdown of the application,
pytest's report (the whole session dies with exit 134), and the ability of a Jupyter kernel to
survive. Raising an exception whose message says "your context could not be restored; context X is
now current; call Device.set_current()" gives the user everything an abort would, plus the choice.
In a destructor, where nothing can be raised, a warning through the same channel is the equivalent.

Peer libraries land on both sides of this exact question (verified against current sources):
XLA CHECK-fails on restore failure except for CUDA_ERROR_DEINITIALIZED; CCCL throws from a
noexcept(false) destructor and accepts std::terminate while unwinding; PyTorch's CUDAGuard
destructor explicitly catches and warns "instead of std::terminate"; RMM ignores it in release
builds and asserts in debug; CuPy raises from __exit__. Every library with a Python surface
(PyTorch, RMM, CuPy, numba-cuda) avoids process termination in release builds. The proposal follows
them.

Proposed policy

User-facing (new docs page error_handling.rst):

• Failed CUDA calls raise; the message carries the CUDA error name and description.
• After an exception: nothing was created; the caller's current context is unchanged (except for
  Device.set_current); objects stay usable. The one exception is a context restoration failure,
  which is raised with a message that says which context is now current.
• Failures that cannot be raised (GC-driven or deferred cleanup, CUDA callbacks, cleanup after an
  earlier failure) are reported as cuda.core.CUDAWarning. CUDA_ERROR_DEINITIALIZED is not
  reported. warnings.filterwarnings("error", category=cuda.core.CUDAWarning) escalates them; an
  escalated report from a destructor is delivered through sys.unraisablehook, which pytest turns
  into PytestUnraisableExceptionWarning.
• cuda.core does not retry or hide sticky errors and does not terminate the process for them.
• During interpreter finalization cuda.core does no Python work from destructors or callbacks and
  leaks what it cannot release.
• cuda.core does not abort the process in response to a CUDA error. Abort is reserved for an
  internal invariant violation where continuing could corrupt memory; no such path exists.

Contributor-facing (new "Failure handling" section in cuda_core/AGENTS.md, plus _cpp/DESIGN.md):
raise by default and preserve the CUresult; every call except set_current leaves the current
context as it found it and uses the handle layer's scoped-context helpers rather than hand-rolled
push/pop; publish before you raise; non-propagating paths never raise and never discard a status
and report through one helper; rollback failures are reported out of band while the original
exception propagates; leak on finalization; the abort tier's exact preconditions and required
diagnostics; how to test (a fault-injection hook for restoration failures, pytest.warns(CUDAWarning)).

Proposed code changes (PR to follow, based on #2750)

Change list (expand)

• C++ handle layer: one reporting helper (report_cuda_error / report_message) that emits
  CUDAWarning via PyErr_WarnEx when the interpreter is usable, delivers an escalated warning as an
  unraisable exception, and falls back to stderr when the GIL cannot be taken; pw_* wrappers on
  every destroy call made from a deleter (streams, events, memory, pools, green contexts, graphs,
  graph execs, graphics, arrays, textures, surfaces, linker, user objects, compiler handles, fds);
  CUDA_ERROR_DEINITIALIZED filtered.
• Context restoration failure in a propagating path: creation undone (as in cuda.core: make Device methods use their bound context #2750) and the raised
  CUDAError carries an explanation of what happened and which context is current (a thread-local
  detail consumed by the Cython error path). In a deleter: CUDAWarning. The skipped-undo leak in
  the texture/surface path is reported.
• New helpers context_get_device and graph_node_set_params so the remaining hand-rolled
  push/pop sequences (Stream_get_ctx_device, _set_definition_node_params) go through the handle
  layer; the node update publishes its attachment before raising a restoration failure (fixes the
  dangling-owner hazard).
• Device.set_current(ctx) switches with a single cuCtxSetCurrent instead of pop-then-push, so a
  failure leaves the previous context current, and it works when no context is current.
• GraphBuilder.__dealloc__ and the child-graph embed rollbacks report their failures.
• _mr_dealloc_callback warns instead of printing; tests switch from capfd string matching to
  pytest.warns(CUDAWarning) / a warnings-based helper.
• Public API: cuda.core.CUDAWarning; docs page; API reference entry; release notes.
• Test hook: cuda.core._resource_handles._set_context_restore_fault_for_testing(status) makes the
  next context restoration on the calling thread fail (leaving the target context current, as a real
  failure would).

Out of scope / follow-ups

Items deliberately left for separate issues (expand)

• Exposing CUDAError from cuda.core (today it lives in cuda.core._utils.cuda_utils).
• The primary-context TLS deleter ([BUG]: Primary-context TLS destructor segfaults after Python finalization #2743 / Fix primary context cleanup during Python shutdown #2744) is left alone here so the two changes do not
  conflict; once Fix primary context cleanup during Python shutdown #2744 lands, its release call should use the same reporting wrapper.
• cuLibraryUnload remains disabled (needs the owning context, same shape as the texture fix).
• Deciding whether post-success cleanup failures (GraphNode.destroy, instantiate, update)
  should keep raising or move to report-and-return-success; the policy allows both and the PR does
  not change them.
• A logging-based channel (Support the driver error logging system #671) could later replace or supplement CUDAWarning.
• The p_cu* function pointers used by the handle layer are cuda.bindings Cython wrappers
  (cydriver.__pyx_capi__), not raw libcuda entry points. They can take the GIL (lazy driver
  init), and on a missing symbol they raise FunctionNotFoundError under the GIL and return
  CUDA_ERROR_NOT_FOUND with a Python exception left set. Every "driver call from a deleter"
  inherits that behaviour (it is the mechanism behind [BUG]: Primary-context TLS destructor segfaults after Python finalization #2743). Deciding whether teardown paths should
  use raw cuGetProcAddress pointers is a separate issue.
• Implicit std::terminate paths on out-of-memory: make_deallocation_stream is noexcept but
  allocates (new StreamBox, registry insert), as does invalidate_child_graph_state on some
  standard libraries. The policy classifies these as bugs (like fix: remove incorrect noexcept from resource handle functions #1489), to be fixed separately.
• The IPC-imported memory pool gets the same owning deleter as a locally created pool
  (clear_mempool_peer_access + cuMemPoolDestroy on a pool created by another process); whether
  that is intended has not been established.

Related

#2750 (motivating PR; author's inline question about std::abort), #2311, #1586 (+ #1597), #548,
#2497 / #2526 (origin of the stderr-warning convention and the "explicit deallocate() raises,
automatic cleanup warns" release note), #2371, #2357, #2280 / #2317, #2395, #2473, #2008 (graph
rollback and prepare/commit precedents), #2074, #1754, #2743 / #2744 (finalization leaks), #1118,
#1063, #141 ("destructors never raise" lineage), #1489, #2416 / #2417 (unintended terminate treated
as a bug), #1951 (prior error-handling audit), #2084, #2719, #2460 / #2461 (preserve the original
CUresult), #2610, #1632, #2627 / #2635, #2344, #2388, #2345 (VMM rollback semantics), #671.