gsp: don't advertise RTD3 GC6/GC8/GCOFF on external (eGPU) boards

[arc-uri-el]

Summary

_kgspInitGpuProperties() advertises RTD3 GC6/GC8/GCOFF support for external (eGPU) boards straight from the GSP silicon capability, with no IS_EXTERNAL_GPU check. These are host-platform-coordinated deep power-down states (the host power-gates the GPU's PCIe slot), which is physically impossible for an external GPU in a Thunderbolt/OCuLink enclosure with its own PSU. After the RTD3 idle timeout the GPU attempts the power-down, the GSP firmware faults, and the GPU falls off the PCIe bus.

Crash (RTX 5090 / GB202 over Thunderbolt, 610.43.02)

NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed status 0x0000000f
NVRM: ... GPU_IN_FULLCHIP_RESET
nvidia-smi: No devices were found

Occurs a few minutes after the desktop goes idle. NVreg_DynamicPowerManagement=0 does not help (it does not override this GSP-set property). The only prior workaround was pinning clocks to max (nvidia-smi -lgc/-lmc) to keep the GPU out of idle, at a large constant idle-power cost.

Fix

Gate PDB_PROP_GPU_RTD3_{GC6,GC8,GCOFF}_SUPPORTED on !PDB_PROP_GPU_IS_EXTERNAL_GPU. IS_EXTERNAL_GPU is already set by RmCheckForExternalGpu before _kgspInitGpuProperties runs. Orthogonal to dynamic clock scaling: external GPUs keep full dynamic idle clocking (verified P0<->P8, ~38 W idle) while no longer attempting the impossible deep power-down. Zero GSP crashes over a sustained idle test after the change.

Relates to #979.

First-time contributor, happy to sign the CLA or adjust to your process.

[CLAassistant]

All committers have signed the CLA.

[Binary-Eater]

Hi @arc-uri-el,

Thank you for your contribution. First, I would like to go over RTD3 in a bit more detail, specifically, about D3hot and D3cold, just to make the conversation easier for myself.

RTD3 -> Runtime PCIe Device-state 3 (suspend)

D3hot -> PCIe is on but device is put into lowest power state over PCIe
D3cold -> Vcc for the PCIe line is actually controllable and disabled. ACPI on motherboards with D3cold support take care of powering back on.

These are host-platform-coordinated deep power-down states (the host power-gates the GPU's PCIe slot), which is physically impossible for an external GPU in a Thunderbolt/OCuLink enclosure with its own PSU.

My understanding is that this is true for D3cold but not D3hot. I think the problem here is that we are disabling FBSR with the current change proposal (framebuffer self refresh) by disabling GPU GC6 entry, which is a bit unfortunate.

References:

• https://download.nvidia.com/XFree86/Linux-x86_64/610.43.03/README/dynamicpowermanagement.html
• https://docs.kernel.org/gpu/xe/xe_pm.html

I think what likely needs to happen here is that the NVIDIA Linux driver actually needs to call pci_d3cold_disable for the root pcie device for the eGPU to prevent the transition from D3Hot to D3Cold for eGPU dock GPUs. I think however we can safely drop eGPUs to D3hot power savings levels when idle as long as we prevent D3cold transitions.

• https://elixir.bootlin.com/linux/v4.15/source/drivers/pci/pci.c#L2384

This would ideally be done somewhere in src/nvidia/arch/nvalloc/unix/src/dynamic-power.c.
My initial intuition, though I have not given this my full attention, would be to take care of this pci_d3cold_disable in RmInitPowerManagement, which is called per-RM device init for any eGPU devices.

Before this change

Internal GPU (if NVIDIA) on RTD3 capable laptop/system

Could enter D3hot -> Good
Could enter D3cold -> Good

NVIDIA eGPU on RTD3 capable laptop/system

Could enter D3hot -> Good
Could enter D3cold -> Bad

After this change

Internal GPU (if NVIDIA) on RTD3 capable laptop/system

Could enter D3hot -> Good
Could enter D3cold -> Good

NVIDIA eGPU on RTD3 capable laptop/system

Cannot enter D3hot -> Loses some power saving opportunity
Cannot enter D3cold -> Good

The ideal situation

Internal GPU (if NVIDIA) on RTD3 capable laptop/system

Could enter D3hot -> Good
Could enter D3cold -> Good

NVIDIA eGPU on RTD3 capable laptop/system

Could enter D3hot -> Good
Cannot enter D3cold -> Good

[Binary-Eater]

@arc-uri-el looking at this a bit more on the kernel side, I would expect the eGPU enclosure to be handled more gracefully by the Linux kernel, https://elixir.bootlin.com/linux/v7.1.3/source/drivers/pci/pci.c#L2945

Do you mind sharing what eGPU enclosure you are using to reproduce the issue?

[arc-uri-el]

Thank you for the detailed analysis.

The enclosure is a Razer Core X V2 (Thunderbolt 5) with an Intel JHL9480 bridge. The full reproduction setup is:

• GPU: NVIDIA RTX 5090 / GB202, PCI ID 10de:2b85, 32 GB
• Enclosure: Razer Core X V2, Thunderbolt 5, Intel JHL9480
• Host: ASUS ROG Flow Z13 GZ302EA, AMD Ryzen AI MAX+ 395
• OS: NixOS 26.05
• Kernel: Linux 7.0.13
• Driver: NVIDIA open kernel modules 610.43.02
• Link: PCIe Gen4 x4 through the Thunderbolt path
• Display: the external display is driven directly by the eGPU

Preserving D3hot and FBSR while preventing only D3cold would be preferable to the current proposal if it resolves the failure. We will implement an alternative patch following your suggestion, using pci_d3cold_disable() for the external-GPU path during RM power-management initialization, and test it against the current GC6/GC8/GCOFF gate under the same idle and subsequent load cycle.

We will collect the PCI power-state and link-state observations, GSP/kernel logs, idle power, and resume/load result, then report back here once we have the results.

[arc-uri-el]

Thanks for your patience. We implemented your suggestion and have been running it as the daily-driver configuration since 2026-07-13.

Implementation

Following your recommendation, instead of gating GC6/GC8/GCOFF we now call pci_d3cold_disable() for the external-GPU path during RM power-management initialization, so D3hot and FBSR remain available and only the D3cold transition is prevented:

// in RmInitPowerManagement, external-GPU path
if (nv->is_external_gpu)
{
    nv_pci_disable_d3cold(nv);   // -> pci_d3cold_disable(nvl->pci_dev)
}

Results (same setup: RTX 5090 / Razer Core X V2 TB5 / JHL9480 / ASUS ROG Flow Z13 / NixOS, kernel 7.0.13, driver 610.43.02)

• ~2 weeks of normal daily use, 49 boots, with the eGPU driving the primary 4K display.
• Zero GSP failures: no Xid 79, no _issueRpcAndWait/rpcSendMessage failed, no GspRmFree failed, no GPU_IN_FULLCHIP_RESET, no "fallen off the bus". Previously the fault reproduced after an idle period.
• GPU idles to P8 (~10.7 W, 450 MHz core / 405 MHz mem), so idle-clock power saving is preserved. PCIe link holds Gen4 x4.

One caveat on validation

On this system the eGPU drives the primary display, so the Linux RTD3 runtime-PM path stays active (runtime_status = active, "Runtime D3 status: disabled by default"). I therefore cannot independently confirm from sysfs that a full D3hot/GC6 entry+exit cycle was exercised in this window; I can only confirm the system is stable in real use with no GSP resets, which is the exact failure that previously occurred. d3cold_allowed still reads 1 as expected, since pci_d3cold_disable() sets the internal no_d3cold flag rather than the sysfs attribute.

Is there a preferred way you would like us to force or verify the D3hot entry, so we can positively confirm that blocking only D3cold while allowing D3hot is what resolves the fault? Happy to update this PR to the pci_d3cold_disable() implementation once you are satisfied with the validation approach.

[Binary-Eater]

@arc-uri-el RTD3 cannot be entered if the eGPU drives the display, in the original configuration, did you have the GPU drive the display?

[Binary-Eater]

Could you try removing any patches and see if you can repro the issue with the configuration you used?