Fabric-wide Xid 145 (NVLink RLW) on 8x B300 HGX: vLLM TP workers killed, NCCL collectives hang on any GPU pair until node reboot

[ennste]

NVIDIA Open GPU Kernel Modules Version

595.71.05 (Ubuntu packages: nvidia-dkms-open 595.71.05-1ubuntu1, nvidia-kernel-source-open 595.71.05-1ubuntu1)

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Ubuntu 26.04.1 LTS

Kernel Release

Linux 7.0.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Aug 1 04:26:38 UTC 2026 x86_64 GNU/Linux

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

8x NVIDIA B300 SXM6 AC (HGX 8-GPU node with NVSwitch fabric); GPU UUIDs of the affected pair: GPU-03460ed0-0b72-7fcc-9e49-a215c3e33c58 (PCI 66:00.0) and GPU-0c239aa8-f014-5baa-5147-e54ded720f61 (PCI 76:00.0)

Describe the bug

While a vLLM engine (TP=2, 2 GPUs) was serving inference traffic on an
8x B300 SXM6 node, the NVIDIA driver tore down the engine's CUDA
channels with an uncorrectable NVLink error. The failure hit both
GPUs of the TP group simultaneously, across all NVLink ports, and
was detected first by the Fabric Manager as a fabric-wide event.

After the event, the node's NVLink fabric remained broken for TP
workloads: two subsequent pod restarts hung indefinitely at NCCL
init (first collective) — including one attempt rescheduled to a
different, previously untouched GPU pair (GPUs 0+1 instead of 2+3).
Only a full node reboot recovered the fabric.

Key log lines (all within ~170 ms):

Fabric Manager (first detector, 06:18:54.930 UTC):
"Fabric Manager detected GPU NVL Non Fatal error on:
gpuGuid: 0x96c8640003542574, moduleId: 5, gpuId: 6 [and gpuId: 8 / moduleId: 7],
errorCode: 0x02 (RLW (Xid 145)), errorSubcode: 0x07 (RLW_SRC_TRACK),
portStatus: 2 (Up), portDownReasonCode: 0x00 (No_link_down_indication),
isErrorFirst: 1"
→ reported on ALL ports of both GPUs, peer ports on two different NVSwitches

Kernel (NVRM, 06:18:55.084 UTC):
NVRM: Xid (PCI:0000:66:00): 145, RLW_SRC_TRACK Nonfatal XC1 i0 Link 00 … Link 17
NVRM: Xid (PCI:0000:76:00): 145, RLW_SRC_TRACK Nonfatal XC1 i0 Link 00 … Link 17
(all 18 links on both GPUs within ~200 ms)
NVRM: Xid (PCI:0000:66:00): 45, pid=137971, name=python3, channel … caused by previous Xid 145
NVRM: Xid (PCI:0000:76:00): 45, pid=138234, name=python3, channel … caused by previous Xid 145
(same for nvidia-device-plugin pid and cuda-EvtHandlr threads)
NVRM: knvlinkSendInbandData_IMPL: Failed to send inband data: 0

Application (vLLM, 06:18:55 UTC):
RuntimeError: CUDA error code=220 (CUDA_ERROR_NVLINK_UNCORRECTABLE)
torch.AcceleratorError: CUDA error: uncorrectable NVLink error detected during the execution
Exception raised from currentStreamCaptureStatusMayInitCtx at c10/cuda/CUDAGraphsC10Utils.h:73

Context worth noting:

• The engine had been up and serving for ~20 min (weights fully loaded,
  ~212 GB VRAM per GPU) when the error hit — this was NOT a startup failure.
• The error burst covered ALL links on BOTH GPUs of the TP group at once —
  not a single link or a single GPU — suggesting a fabric-level (NVSwitch
  side or fabric state) event rather than one degraded link.
• Links never went down (portStatus stayed "Up", "No_link_down_indication"),
  FM reported isErrorFirst=1 for all affected ports.
• After the crash, TP collectives hung on ANY GPU pair on the node (also
  GPUs 0+1), while a second vLLM engine (TP=4) on GPUs 4–7 of the same
  node continued serving without interruption.
• A node reboot fully recovered the fabric. Pod/Fabric Manager restarts
  alone did not (verified 3x).
• DCGM showed no thermal or power anomaly around the event.

To Reproduce

Preconditions: HGX 8x B300 SXM6 node, driver 595.71.05 (open kernel module),
Fabric Manager 595.71.05, vLLM engine with tensor_parallel_size=2 running
on GPUs 2+3 (or any pair), CUDA graphs enabled, NCCL collectives over NVLink.

1. Start vLLM with TP=2 and let it fully load and serve (~20 min in our case)
2. Observed once: fabric-wide RLW error event (Xid 145 on all links of both
   GPUs of the TP group, reported by Fabric Manager first)
3. Driver tears down all CUDA channels of the TP workers (Xid 45)
4. vLLM engine dies with CUDA_ERROR_NVLINK_UNCORRECTABLE
5. Restart the workload on the SAME GPU pair → hangs at NCCL init
   (first collective) indefinitely, no error, VRAM stays at ~1 GB
6. Restart the workload on a DIFFERENT GPU pair (GPUs 0+1) → same hang
7. Full node reboot → fabric recovers, workload starts and serves normally

Incidence: Once so far (single event, node recovered via reboot).
We cannot deliberately re-trigger the initial fabric event.

Bug Incidence

Once

nvidia-bug-report.log.gz

To be clear about the nature of this file for this report: the initial
fabric event was a one-time transient occurrence (see "Bug Incidence:
Once") and cannot be deliberately re-triggered — so the bug report is
an environment snapshot, not a reproduction capture. The event itself
is documented by the log excerpts quoted inline in the description
above (Fabric Manager events, kernel Xid lines, vLLM traceback).

More Info

Note on the proprietary-driver checkbox: the proprietary kernel module
flavor is not available for this platform (B300 requires the open kernel
module), so a cross-check against the proprietary flavor was not possible.

Related report: #1118 (NVLink GSP crash on 8x A100 SXM4, vLLM workload)
shows the same failure signature on the previous platform generation:
NVLink fatal error → Xid 45 channel teardown on vLLM workers → node
unusable for TP workloads until reset. In that case the reporter concluded
a node-local NVLink hardware error after the same model ran fine on an
identical node.

Our case differs in three ways:

• Platform: B300 (Blackwell Ultra) instead of A100, Xid 145 (RLW) instead
  of Xid 74
• The fabric recovered fully after a node reboot (transient event), and
  FM reported isErrorFirst=1 with all links staying "Up" — which is why
  we are unsure whether this points to hardware, GSP/firmware state, or
  a driver issue in 595.71.05
• The post-event state blocked TP collectives on ANY GPU pair on the node,
  suggesting a fabric/NVSwitch-level stuck state rather than one bad link