GB10 (DGX Spark): silent whole-host wedge after NVRM: NV_ERR_NO_MEMORY (_memdescAllocInternal) under sustained unified-memory allocation pressure — no OOM kill, no panic, empty pstore, power cycle required

[wydi42]

NVIDIA Open GPU Kernel Modules Version

595.84

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Ubuntu 24.04.5 LTS (DGX OS flavour), aarch64

Kernel Release

Linux 6.17.0-1032-nvidia (aarch64)

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

GPU 0: NVIDIA GB10 (UUID: GPU-1e6e8ddd-283c-2fcf-cd6f-c00b19eb3090)

Describe the bug

Under sustained unified-memory allocation pressure from a userspace workload (vLLM
serving NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 per NVIDIA's own DGX Spark deployment
guide: ModelOpt NVFP4 + Marlin GEMM + Mamba-2 hybrid + baked-in MTP speculative
decoding, async-scheduling, chunked prefill, fp8 KV cache), a single long-prefill
burst (~40k tokens, one sequential request) makes the driver log

NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY]
(0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359

and then the ENTIRE HOST wedges silently: no kernel OOM kill (10-12 GiB RAM still
free), no panic, /sys/fs/pstore/ is empty, the journal stops abruptly right after the
NVRM lines. SSH/ICMP stop responding; the machine requires a manual power cycle.
Reproduced 3 times across driver 580.173.02 and 595.84, with and without
PYTORCH_ALLOC_CONF=expandable_segments:True (the mitigation from vLLM issue #55569,
same signature on GLM-NVFP4, does not hold for this NemotronH pipeline). Lighter
inference load (5k-token prompts) is stable.

An allocation failure must fail the userspace allocation (CUDA OOM error), never
silently wedge the host; at minimum a panic with a pstore/ramoops dump is needed as
a debug surface.

To Reproduce

1. On a DGX Spark (GB10, 121 GiB unified memory, Ubuntu 24.04.5 DGX OS, driver 595.84,
   Secure Boot on), serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 with vLLM
   cu130-nightly per NVIDIA's "Nemotron 3 Super - DGX Spark Deployment Guide"
   (vLLM section): gpu-memory-utilization 0.85 (NOTE: the guide value 0.90 does NOT
   reach a serving state on GB10 - see "More Info"), kv-cache fp8, moe-backend marlin,
   async-scheduling, chunked prefill, spec-decode MTP.
2. Startup completes cleanly (weights 75 GiB, KV pool ~32 GiB, autotuning done, API OK).
3. Light load is fine: 8 sequential ~5k-token prompts -> 8/8 HTTP 200.
4. Send ONE ~40k-token prompt (sequential, no concurrency): prefill runs for minutes,
   then NVRM NV_ERR_NO_MEMORY lines appear and the host wedges silently (no OOM kill,
   no panic, empty pstore) -> power cycle required.

Bug Incidence

Always

nvidia-bug-report.log.gz

Report generated after a reboot following the last wedge (system running normally).
The crash evidence is in the description: the journal of the crashed boot ends with
the quoted NVRM NV_ERR_NO_MEMORY lines from _memdescAllocInternal; no pstore entry
exists (part of the bug).

nvidia-bug-report.log.gz.gz

More Info

Additional data points from the same host (2026-09-12):

• gpu_memory_utilization 0.90 (the deployment guide value) OOM-killed the host during
  engine startup in 3/3 runs - system-wide kernel OOM kills (the vLLM container
  processes AND unrelated daemons: firewalld, sshd user managers, unbound, exporters),
  no panic, no pstore. The serving state is only reachable at 0.85. This matches vLLM
  issue #46307 (gpu_memory_utilization not respected on GB10 during startup profiling).
• All crashes occurred with no desktop/GDM session active (headless server).
• vLLM-side thread collecting the same signature (GLM-NVFP4, 2x Spark TP):
  [Bug]: GLM-5.3-Flash (glm5next) on DGX Spark / GB10: a ~230K-token prefill exhausts unified memory and kills the engine (sparse indexer logits churn); 64 MiB logits budget or expandable_segments fixes it vllm-project/vllm#55569.

[reinaertvdc]

Hitting the same signature on a DGX Spark (GB10), but from a completely different trigger — a local hashcat session, not vLLM — which suggests this isn't specific to any one CUDA workload, just to sustained/oversized unified-memory allocation pressure in general.

Setup: Ubuntu 24.04, DGX Spark (GB10, 119.6 GiB unified memory), hashcat running inside a Docker container with GPU passthrough (--gpus all equivalent via compose). Note nvidia-smi reports Memory-Usage: Not Supported from inside the container on this platform, so anything that sizes buffers off a queried "free VRAM" figure has nothing accurate to size against.

First occurrence (driver 580.159.03, kernel 6.17.0-1018-nvidia): running hashcat -I — supposedly just a device-identify query, not even a real cracking run — was enough to trigger it. RAM usage spiked immediately, the host thrashed, and it went completely unresponsive. journalctl -k for that boot shows the same call site repeating dozens of times in under a second, then nothing further reaches disk before a hard power cycle was required:

NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359

(38 consecutive occurrences logged within ~1 second, then journald itself appears to have been unable to persist further entries before the reset — later kernel hung-task warnings for unrelated processes were visible on the physical console but never made it into the journal.)

Mitigation tried: upgraded to driver 580.173.02 / kernel 7.0.0-1019-nvidia, and set nvidia_uvm.uvm_global_oversubscription=0 (was 1 by default) via modprobe.d, on the theory that letting the UVM allocator admit a request beyond physical availability — expecting to reconcile it later — is what turns a should-be-clean allocation failure into a wedge.

Second occurrence, same repro (hashcat -I), post-mitigation: the host did not lock up this time. Instead the kernel's own global OOM killer engaged and killed a wide, unrelated spread of processes (sshd, NetworkManager, avahi-daemon, an unrelated container's process, plus repeated respawns inside the hashcat container's own cgroup) before finally killing the container:

oom-kill:constraint=CONSTRAINT_NONE,...,global_oom,task_memcg=/system.slice/docker-<id>.scope,task=MainThread,...
Out of memory: Killed process ... (MainThread) ...

So: disabling uvm_global_oversubscription seems to convert this from an unrecoverable silent wedge into a recoverable but still-global OOM sweep — better, but still wrong. None of the individually-killed processes show anywhere near the RAM actually being consumed (largest killed RSS was under 200 MB), which matches the pattern in this issue and in #97: the real memory pressure doesn't seem to be attributed to any single process's accounted RSS, consistent with it living in GPU/unified-memory allocations that aren't charged the normal way.

A container-level memory.max cgroup limit on the process's own cgroup did not contain the blast radius either — the global OOM sweep happened regardless of the container's own memcg ceiling, which is a separate concern from the underlying allocation failure: cgroup accounting doesn't seem to see this memory either.

Given it reproduces from something as minimal as a device-identify call, this doesn't look tied to any single framework's memory-sizing logic (vLLM's gpu_memory_utilization, hashcat's autotune, etc.) — it looks like it can be provoked by very little on this platform.

[fuomag9]

Reproduced this exact signature on a different workload and found a variable this report didn't control for: kernel version, independent of driver package version.

Setup: 2x DGX Spark (GB10), Ubuntu 24.04.5 LTS, nvidia-driver-580-open 580.173.02-0ubuntu0.24.04.1 on both nodes (unchanged throughout). Serving deepseek-ai/DeepSeek-V4-Flash-Vision-Exp via vLLM 0.25.2.dev0+g752a3a504.d20260714 (ghcr.io/anemll/dspark-vllm-gx10:0.1.1), --tensor-parallel-size 2 --nnodes 2 over RoCE (NCCL 2.30.7+cuda13.3, ConnectX/mlx5 between the two nodes), DSpark speculative decoding.

Under kernel 7.0.0-1019-nvidia (7.0.0-1019.19~24.04.2, pulled in by a routine apt upgrade the same day the failures started), every boot attempt — 7 across roughly 5 hours — hit this deterministically, within seconds of the two ranks completing NCCL rendezvous, before any inference request and well before the sustained-load trigger described in the linked vLLM thread (vllm-project/vllm#55569). journalctl -k showed:

NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359

at the same timestamp NCCL logged:

misc/ibvwrap.cc:213 (wrap_ibv_reg_mr_iova2) NCCL WARN Call to ibv_reg_mr_iova2 failed with error Cannot allocate memory

surfaced by vLLM/PyTorch as:

torch.distributed.DistBackendError: NCCL error in: .../NCCLUtils.cpp:93, unhandled system error
ncclSystemError: ... Last error: Call to ibv_reg_mr_iova2 failed with error Cannot allocate memory

None of the following changed the outcome while still on 7.0.0-1019-nvidia:

• a full power-cycle reboot (rules out state accumulated from earlier crashes)
• --gpu-memory-utilization 0.835 → 0.78
• --max-num-batched-tokens 8192 → 2048 — this actually moved the failure earlier, into NCCL's connection-setup phase instead of the profiling forward pass, which reads more like a fixed/count-based resource limit than one proportional to allocation size
• VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=64, the mitigation verified in #55569 for its differently-triggered variant of this bug

Booting the same two nodes back into the immediately prior kernel, 6.17.0-1032-nvidia (6.17.0-1032.32) — same driver package, same container image digest, unchanged application config — fixed it on the first attempt: clean boot, no crash, served correctly, 6/6 on the recipe's smoke test, stable since.

This report already shows the bug reproduces across driver package versions 580.173.02 and 595.84. What I'm adding is that it also depends on kernel version while the driver package version is held constant: the nvidia-580-open kernel module is rebuilt per kernel (linux-modules-nvidia-580-open-<kernel-version>), and the identical 580.173.02 driver source built against 7.0.0-1019-nvidia fails where the same source built against 6.17.0-1032-nvidia does not. That points at something in the kernel itself — plausibly the DMA/IOMMU or memory-management path the out-of-tree module depends on — rather than the driver source alone, at least as a contributing factor. Might be worth a kernel diff between those two DGX OS kernel builds for GB10/Tegra-specific changes.

Happy to pull a full nvidia-bug-report.log.gz from either kernel, or the complete container logs, if useful.