GB203 [RTX 5070 Ti]: GSP-RM heartbeat timeout at idle → Xid 119 ×3 → Xid 154; the announced "PF FLR" recovery action is never performed, leaving WPR2 latched across warm reboots

[andrea966p]

NVIDIA Open GPU Kernel Modules Version

610.57.04 (Ubuntu nvidia-driver-610-open 610.57.04-0ubuntu0.26.04.3).
Also reproduced on 595.84 and 610.43.02.

Does this happen with the proprietary driver (of the same version) as well?

Not applicable, and I want to flag a packaging problem while I'm here. Per the
610.57.04 README (kernel_open.html): "The proprietary flavor supports GPU
architectures Turing, Ampere, Ada, and Hopper" and "Blackwell and later are
only supported by the open kernel modules." So there is no proprietary module
to compare against on GB203.

Ubuntu's packaging does not reflect this. The Modaliases fields of
nvidia-driver-610 and nvidia-driver-610-open are byte-for-byte identical
(11,875 bytes each) and both advertise pci:v000010DEd00002C05. A user
following generic "try the proprietary driver" advice will install a module that
cannot drive their GPU. Worth raising with the distro maintainers if this
mismatch originates upstream.

Operating System and Version

Ubuntu 26.04 (resolute), x86_64.

Kernel Release

7.0.0-31-generic. Also reproduced on 7.0.0-14, -29, -30 — every kernel
this machine has ever booted. Ubuntu 26.04 ships 7.0 as its GA kernel and
carries no 6.x image, so a pre-7.0 comparison has not been possible.

Please confirm you are running a supported configuration

Yes. Consumer GB203 [10de:2c05], open kernel module, x86_64, on a
distro-packaged driver.

Describe the bug

Two failure modes. The second is caused by the driver's failure to recover from
the first, and it is the one that makes the machine look like it has dead
hardware.

Mode A — GSP-RM dies at idle

At idle, with no 3D or compute load, GSP-RM stops answering. The signature is
identical every single time:

NVRM: GPU0 _kgspRpcRecvPoll: GSP RM heartbeat timed out
NVRM: Xid (PCI:0000:01:00): 119, pid=606, name=nvidia-modeset/, Timeout after 6s of waiting for RPC response from GPU0 GSP! Expected function 76 (GSP_RM_CONTROL) sequence 13416 (0x20802801 0x4).
NVRM: Xid (PCI:0000:01:00): 119, pid=95477, name=btop, Timeout after 6s of waiting for RPC response from GPU0 GSP! Expected function 76 (GSP_RM_CONTROL) sequence 13417 (0x20801819 0x20).
NVRM: Xid (PCI:0000:01:00): 119, Timeout after 6s of waiting for RPC response from GPU0 GSP! Expected function 76 (GSP_RM_CONTROL) sequence 13418 (0x2080852f 0x34c).
NVRM: Xid (PCI:0000:01:00): 154, GPU recovery action changed from 0x0 (None) to 0x1 (PF FLR)
NVRM: nvCheckOkFailedNoLog: Check failed: Reset required [NV_ERR_RESET_REQUIRED] (0x00000062)

Exactly three Xid 119 events, then one Xid 154, on all seven recorded
occurrences. The GPU is unusable from that point on.

Mode B — the GPU then will not initialise at all

This is the part I think is the actual defect. Xid 154 announces
GPU recovery action changed from 0x0 (None) to 0x1 (PF FLR) — but no FLR is
ever performed. The GPU is left in its faulted state with WPR2 still latched.

Because a warm reboot does not remove power from the card, WPR2 survives it. The
next boot therefore fails in the GSP bootloader:

NVRM: Xid (PCI:0000:01:00): 143, Error status 0x65 while polling for FSP boot complete, 0x13, 0x56, 0x0, 0x0, 0x2
NVRM: GPU0 kfspDumpDebugState_GB100: NV_PGSP_MAILBOX(0) = 0xbadf4100
NVRM: GPU0 kfspDumpDebugState_GB100: NV_PGSP_MAILBOX(1) = 0xbadf4100
NVRM: GPU0 kfspDumpDebugState_GB100: NV_PGSP_MAILBOX(2) = 0xbadf4100
NVRM: GPU0 kfspDumpDebugState_GB100: NV_PGSP_MAILBOX(3) = 0xbadf4100
NVRM: GPU0 gpuReportGspFmcErrorCode_GH100: Fatal GSP-FMC Error: version=0x1, partition=0x1, error code=0xaff, additional info=0xb
NVRM: GPU0 gpuReportGspFmcErrorCode_GH100: Fatal GSP-FMC Error: version=0x1, partition=0x1, error code=0xa9f, additional info=0x0
NVRM: GPU0 kgspInitRm_IMPL: Max GSP-RM boot attempts exceeded: 4/4
NVRM: GPU0 RmInitAdapter: Cannot initialize GSP firmware RM

nvidia-smi reports No devices were found. Every subsequent warm reboot
reproduces this within 2–3 minutes. With the monitor on the GPU, the desktop
falls back to a 1024x768 simple-framebuffer and the machine is effectively
unusable — which is what leads people to conclude the card is defective and RMA
working hardware.

A userspace FLR fully recovers the GPU

This is the evidence that the announced recovery action is both correct and
simply not being taken. With the driver unloaded:

echo 1 | sudo tee /sys/bus/pci/devices/0000:01:00.0/reset

Before (2026-09-11 09:53): nvidia-smi → No devices were found, GSP-FMC 0xaff,
Max GSP-RM boot attempts exceeded: 4/4.
After the FLR and a module reload, with no reboot (09:54):

name, pstate, temperature.gpu, power.draw [W]
NVIDIA GeForce RTX 5070 Ti, P0, 39, 32.27 W

Full sequence in kernel-recovery-via-flr.log.

Note for anyone documenting SBR as the recovery path

Several community write-ups recommend a PCIe Secondary Bus Reset for this. SBR
is not available on consumer cards and cannot be the documented fix. The
kernel permits a bus reset only when the target is the sole device on its bus:

/* drivers/pci/pci.c, pci_parent_bus_reset() */
list_for_each_entry(pdev, &dev->bus->devices, bus_list)
        if (pdev != dev)
                return -ENOTTY;

Every GeForce card exposes an HDA audio function (0000:01:00.1) on the same
bus, so this always fails:

$ echo bus | sudo tee /sys/bus/pci/devices/0000:01:00.0/reset_method
tee: ...: Invalid argument
$ dmesg | tail -1
pci 0000:01:00.0: Unsupported reset method 'bus'

Confusingly, reset_method still lists bus, because
pci_init_reset_methods() probes once during enumeration — before 00.1 has
joined bus->devices — and caches the result. It is a stale advertised
capability. FLR is the working path, and it is also exactly what Xid 154
already says the driver intends to do.

To Reproduce

1. GB203 [10de:2c05], open kernel module 595.84 / 610.43.02 / 610.57.04, Linux 7.0.
2. Leave the machine idle at the desktop. No load is required — mean GPU
   utilisation across 19,051 samples was 0.5%, and every crash occurred at P8
   idle.
3. Wait. Observed time-to-failure, measured by a 15-second nvidia-smi poll:
   43.7 h, 1.0 h, 20.0 h.
4. On Xid 154, reboot warm → the GPU now fails to initialise entirely (Mode B),
   and keeps failing on every warm reboot.
5. Either power the machine off fully (rails down ~30 s) or issue the FLR
   above; the GPU comes back.

Bug Incidence

Always, eventually. Seven Xid 119→154 incidents recorded between 2026-08-30 and
2026-09-11 (the journal retains boots back to 2026-08-25). 37.7% of all monitored
samples were in a faulted state.

More Info

Physical causes ruled out, with measurements

Hypothesis	Measurement	Verdict
Overheating	57 °C max, 41.5 °C mean (19,051 samples)	Ruled out
Power delivery	57.0 W max, 14.6 W mean, on a ~300 W board; crashes occur at P8 idle	Ruled out
PCIe signal integrity	All aer_dev_correctable / aer_dev_fatal / aer_dev_nonfatal counters zero	Ruled out
PCIe Gen5 link instability	Root port 00:01.1 caps at 16.0 GT/s; link is already Gen4	Not applicable
Resizable BAR	16 GB BAR, functioning	Ruled out
ASPM	pcie_aspm=off — no effect	Ruled out
Runtime power management	NVreg_DynamicPowerManagement=0x00 — no effect	Ruled out
Display engine involvement	Reproduces with the monitor on motherboard HDMI and the NVIDIA card driving nothing	Ruled out
Driver branch	595.84, 610.43.02, 610.57.04 all affected	Not branch-specific
Hardware degradation	NVIDIA stack first installed 2026-08-06; the card has never run stably on this host. No working baseline exists to regress from.	No supporting evidence

What I'm asking for

1. Perform the FLR that Xid 154 announces. The driver already determines
   that PF FLR is the required recovery action and logs it, then does not
   carry it out. A userspace FLR at that point fully recovers the GPU.
2. Failing that, clear WPR2 on initialisation. RmInitAdapter could detect
   a latched WPR2 from a previous unclean GSP crash and reset the device rather
   than burning all four GSP-RM boot attempts against it.
3. Document the supported recovery path — and please make it FLR, not SBR,
   for the bus-topology reason above.
4. Separately, the root cause of the Mode A heartbeat timeout at idle.

Attached data

All files here: https://gist.github.com/andrea966p/f5351a7c6f7121b0bc85cb09b3d60ad5

File	Contents
gpu-health.csv	19,051 samples at 15 s intervals — timestamp, temp, fan, power, pstate, utilisation, memory, health flag
gpu-events.log	Fault-transition log with the kernel Xid context at each transition
xid-history.txt	Every Xid across all retained boots — the 3×119→154 pattern
kernel-modeA-xid119-154.log	Full kernel log for a Mode A crash (2026-09-09 19:11, 610.57.04)
kernel-modeB-gspfmc-0xaff.log	Full kernel log for a Mode B boot failure (2026-09-11 09:33)
kernel-recovery-via-flr.log	The FLR recovery, before and after
system-info.txt	GPU/PCI topology, reset_method, AER counters, driver packages, kernel

Hostname and username are redacted. The GPU UUID is left intact as it is useful
for correlation.

nvidia-bug-report.log.gz

Not yet attached — I will add it in a follow-up comment. Happy to provide
anything else that would help; the machine is reproducing this reliably and I
can run whatever instrumentation is useful.

[andrea966p]

nvidia-bug-report-sanitized.log.gz