GB202GL (RTX PRO 6000 Blackwell): iovaspaceDestruct_IMPL assertion failures (pIOVAS != NULL) after vLLM tensor-parallel Docker stop, leading to GSP_LOCKDOWN_NOTICE + Xid 119 on subsequent GPU open

[guenhter]

NVIDIA Open GPU Kernel Modules Version

580.159.03

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Ubuntu 26.04 LTS

Kernel Release

Linux vnvbrwgpu1 7.0.0-28-generic #28-Ubuntu SMP PREEMPT_DYNAMIC Sun Jun 21 01:01:36 UTC 2026 x86_64 GNU/Linux

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

GPU 0: NVIDIA RTX PRO 6000 Blackwell Server Edition (UUID: GPU-17a7ccd5-72c9-d924-c600-7a2be09787aa)
GPU 1: NVIDIA RTX PRO 6000 Blackwell Server Edition (UUID: GPU-75858999-047a-0c66-932f-c5a89028f2e5)
GPU 2: NVIDIA RTX PRO 6000 Blackwell Server Edition (UUID: GPU-f4702d06-6501-fe66-c6e7-541574891c65)
GPU 3: NVIDIA RTX PRO 6000 Blackwell Server Edition (UUID: GPU-97e94ff4-1f3b-6f88-6e58-eee43fbfa8ac)

Describe the bug

Disclaimer: The bug analysis and report was not done by me personally but an AI, as my knowledge in that area is limited and I honestly couldn't have done a better job I believe. We just had issues on our server and I let Warp on it to analyse the issue. This is the (reproducable) output.

---

When a Docker container running vLLM with tensor parallelism (--tensor-parallel-size 2)
and --ipc host is stopped via docker stop or docker compose down, the NVIDIA
driver logs iovaspaceDestruct_IMPL failures with left-over mappings, followed by
assertion failures at io_vaspace.c:592 and :601:

NVRM: iovaspaceDestruct_IMPL: 2 left-over mappings in IOVAS 0xe300
NVRM: nvAssertFailedNoLog: Assertion failed: pIOVAS != NULL @ io_vaspace.c:592
NVRM: nvAssertFailedNoLog: Assertion failed: pIOVAS != NULL @ io_vaspace.c:601
NVRM: nvAssertFailedNoLog: Assertion failed: pIOVAS != NULL @ io_vaspace.c:592
NVRM: nvAssertFailedNoLog: Assertion failed: pIOVAS != NULL @ io_vaspace.c:601

This reproduces reliably in a single start/stop cycle with the exact configuration
described below.

After multiple stop/start cycles (e.g. a nightly maintenance cron job), the accumulated
IOVAS corruption causes the GPU's GSP firmware to enter a GSP_LOCKDOWN_NOTICE state.
The next time the NVIDIA driver attempts to re-initialize that GPU
(rm_init_adapter → kgspBootstrap_GH100), it times out with Xid 119:

NVRM: Xid (PCI:0000:03:00): 119, Timeout after 6s waiting for RPC response
      from GPU2 GSP! Expected function 4097 (GSP_INIT_DONE).
NVRM: GPU2 GSP RPC buffer contains function 4124 (GSP_LOCKDOWN_NOTICE)
NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
NVRM: gpuHandleSanityCheckRegReadError_GH100: regvalue: 0xbadf5720

The GPU is then permanently unrecoverable until a full system reboot. The other GPUs
involved in the tensor-parallel group are affected in the same way.

Deceptive symptom: After IOVAS corruption, the affected GPUs continue to respond
normally to nvidia-smi queries (basic register reads still work). The failure only
manifests hours later when something triggers a fresh GSP bootstrap, masking the
corruption for an extended period.

To Reproduce

Preconditions:

• NVIDIA GB202GL GPU(s) (RTX PRO 6000 Blackwell Server Edition)
• Driver 580.159.03 open kernel module
• GPU persistence mode disabled (default — confirm with nvidia-smi -q | grep Persistence)
• Docker with NVIDIA Container Toolkit
• vLLM v0.25.1 container configured with:
  • --tensor-parallel-size 2 (NVLink P2P between 2 GPUs)
  • --gpu-memory-utilization 0.90 (~88 GB allocated per GPU)
  • --ipc host (host IPC namespace — enables persistent CUDA IPC handles)
  • A large model (tested with Qwen/Qwen3.5-122B-A10B-GPTQ-Int4, 73 GB)

---

Steps to reproduce IOVAS corruption (single cycle — always reproduces)

1. Start vLLM with tensor parallelism on 2 GPUs:

docker run --detach --name vllm-test \
  --ipc host \
  --device nvidia.com/gpu=0 \
  --device nvidia.com/gpu=1 \
  -v <path-to-hf-cache>:/root/.cache/huggingface \
  -e HF_HUB_OFFLINE=1 \
  vllm/vllm-openai:v0.25.1 \
  Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.90

2. Wait for NCCL initialization (~90 seconds). Confirm with:

docker logs vllm-test 2>&1 | grep "world_size"

Expected output (both workers must appear):

(Worker pid=...) INFO [parallel_state.py] world_size=2 rank=0 ... backend=nccl
(Worker pid=...) INFO [parallel_state.py] world_size=2 rank=1 ... backend=nccl

Also confirm GPU memory is allocated (~38 GB per GPU):

nvidia-smi --query-gpu=index,pci.bus_id,memory.used --format=csv,noheader

3. Stop the container:

docker stop --timeout 10 vllm-test && docker rm vllm-test

4. Check dmesg immediately:

sudo dmesg | grep -iE "iovaspaceDestruct|pIOVAS"

Expected output (bug):

NVRM: iovaspaceDestruct_IMPL: N left-over mappings in IOVAS 0x<bus>
NVRM: nvAssertFailedNoLog: Assertion failed: pIOVAS != NULL @ io_vaspace.c:592
NVRM: nvAssertFailedNoLog: Assertion failed: pIOVAS != NULL @ io_vaspace.c:601

---

Steps to reproduce full GPU failure (multiple cycles → Xid 119)

5. Repeat steps 1–3 three or more times without GPU persistence mode enabled.
Each cycle produces additional IOVAS corruption on different GPU buses.

6. After the final stop (container NOT restarted), run:

nvidia-smi -q -x

7. Observe in dmesg:

NVRM: Xid (PCI:0000:<bus>): 119, Timeout after 6s waiting for RPC response
NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
NVRM: GPU 0000:<bus>.0: RmInitAdapter failed! (0x62:0x65:2028)

The GPU is now unrecoverable without a full system reboot.

---

Important notes:

• The bug does not reproduce with simplified synthetic workloads (plain PyTorch
  torch.zeros allocations without --ipc host and without a real tensor-parallel
  vLLM stack).
• --ipc host is critical — it is what causes CUDA IPC handles to persist on the
  host after the container exits.
• GPU persistence mode (nvidia-smi -pm 1) prevents the issue by keeping the GPU
  initialized between restarts, bypassing the GSP re-bootstrap path entirely.

Bug Incidence

Always

nvidia-bug-report.log.gz

nvidia-bug-report-repro.gz

More Info

1. Proprietary driver not tested.
The system runs exclusively with the open kernel module. Cannot confirm whether
the issue is specific to the open module.

2. Critical factors required for reproduction (all must be present):

• --ipc host: host IPC namespace — creates persistent CUDA IPC handles visible to
  the host that are not cleaned up when the container exits
• --tensor-parallel-size 2: NVLink P2P memory mappings between GPUs via NCCL
• --gpu-memory-utilization 0.90: ~88 GB GPU memory allocated per GPU
• A large model requiring full initialization (tested: Qwen/Qwen3.5-122B-A10B-GPTQ-Int4)

Simplified workloads (plain torch.zeros, no --ipc host, single GPU) do not reproduce.

3. IOVAS errors captured in the attached bug report:

[ 3468.890720] NVRM: iovaspaceDestruct_IMPL: 2 left-over mappings in IOVAS 0xe300
[ 3478.578451] NVRM: nvAssertFailedNoLog: Assertion failed: pIOVAS != NULL @ io_vaspace.c:592
[ 3478.578467] NVRM: nvAssertFailedNoLog: Assertion failed: pIOVAS != NULL @ io_vaspace.c:601
[ 4391.210104] NVRM: iovaspaceDestruct_IMPL: 1 left-over mappings in IOVAS 0x300
[ 4391.221416] NVRM: nvAssertFailedNoLog: Assertion failed: pIOVAS != NULL @ io_vaspace.c:592
[ 4391.221424] NVRM: nvAssertFailedNoLog: Assertion failed: pIOVAS != NULL @ io_vaspace.c:601

Two cycles, two different GPU buses affected (0xe300 = GPU 2 on first stop,
0x300 = GPU 0 on second stop).

4. Fatal Xid 119 — from production (previous boot, after 3 nightly cycles):

NVRM: Xid (PCI:0000:03:00): 119, Timeout after 6s of waiting for RPC response
      from GPU2 GSP! Expected function 4097 (GSP_INIT_DONE).
NVRM: GPU2 GSP RPC buffer contains function 4124 (GSP_LOCKDOWN_NOTICE)
NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
NVRM: gpuHandleSanityCheckRegReadError_GH100: addr: 0x110624,
      regvalue: 0xbadf5720, error code: Unknown SYS_PRI_ERROR_CODE
NVRM: kflcnCoreDumpNondestructive_IMPL: PRI: riscvPrivErrInfo: badf510e
NVRM: RmInitAdapter failed! (0x62:0x65:2028)

Call stack at time of Xid 119:

_kgspRpcRecvPoll+0x784/0x9a0 [nvidia]
kgspWaitForRmInitDone_IMPL+0x36/0xf8 [nvidia]
kgspBootstrap_GH100+0x385/0xaca [nvidia]
kgspInitRm_IMPL+0x979/0x1a20 [nvidia]
RmInitAdapter+0xff2/0x1e80 [nvidia]
nv_open_device+0x459/0xab0 [nvidia]
nvidia_open_deferred+0x36/0xa0 [nvidia]

5. Expected behavior:
The driver should either cleanly release all IOVAS mappings when the container exits
(even with --ipc host), or detect and recover from the corrupted state on next
initialization without entering GSP lockdown.

6. Workaround:
Enable GPU persistence mode to prevent de-initialization between container restarts:

sudo nvidia-smi -pm 1

7. Related issue:
GitHub issue #688 (closed/Implemented) covers a similar 0xbadf5xxx register pattern
on older Maxwell HAL (_GM107). The present issue is on the GH100/Blackwell HAL
(_GH100, GB202GL) and involves the FSP/WPR2 lockdown mechanism specific to
Hopper/Blackwell secure boot.

8. System details:

• Hardware: Lenovo ThinkSystem SR675 V3, BIOS QGE144E-8.41 (2026-01-22)
• vLLM version: v0.25.1
• NCCL version: 2.28.9
• Docker with --ipc host and CDI device access (nvidia.com/gpu=N)