Probe DCGM hostengine GPU visibility in the liveness check

[gengwg]

Description

The nvidia-dcgm hostengine can start while the nvidia kernel module is not loaded (an operator rollout that recreates the dcgm pod while the driver container restarts, for example). The toolkit's host runtime wrapper then execs plain runc with no injection, nv-hostengine starts with no NVML, and nothing ever restarts it. The pod stays 1/1 Running while exporting zero DCGM_FI_* series.

The tcpSocket probe on 5555 passes on a deaf hostengine because it still listens. This replaces it with an exec probe that requires dcgmi discovery -l to report at least one active GPU, so kubelet restarts the hostengine once the driver is back. The check is skipped while no driver is present (no nvidia module and no WSL2 /dev/dxg), so a planned driver restart does not loop the hostengine. The readiness probe is unchanged.

The init-container gate from the original version of this PR is no longer needed: #2881 requires a live nvidia module, or the WSL2 /dev/dxg path, before every operand that uses the NVIDIA runtime, so this PR now carries only the probe change.

Behaviour worth stating: once the driver is present, a node that genuinely loses every GPU sees its hostengine restart every ~3 minutes instead of sitting silently at 0 GPUs. That is intended; it is visible where the previous state was not. Nodes with at least one GPU visible are unaffected. When the hostengine restarts, dcgm-exporter hits DCGM_ST_CONNECTION_NOT_VALID, which it already treats as fatal and exits, so it reconnects on its own restart.

Refs #2854

Checklist

• No secrets, sensitive information, or unrelated changes
• Lint checks passing (make lint) - YAML-only change under assets/
• Generated assets in-sync (make validate-generated-assets) - no generated files touched
• Go mod artifacts in-sync (make validate-modules) - go.mod untouched
• Test cases are added for new code paths - no Go code paths changed

Testing

• Probe pattern checked against the DCGM source: dcgmi discovery -l prints N GPUs found (Active). on stdout, and 0 GPUs found (Active). when no GPU is visible, so the anchored count matches only the visible case.
• go test ./controllers/... ./internal/... passes, including TestOperandToolkitReadinessWaitsForLiveDriver from Wait for the driver before starting GPU operands #2881.
• Not end-to-end tested with a rebuilt operator image on a live cluster.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Enterprise

Run ID: 83a07265-74ca-4fdc-8c29-1942a4811943

📥 Commits

Reviewing files that changed from the base of the PR and between df32e7e and 119d901.

📒 Files selected for processing (2)

• assets/state-dcgm-exporter/0800_daemonset.yaml
• assets/state-dcgm/0400_dcgm.yml

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

📝 Walkthrough

Walkthrough

The DCGM and DCGM exporter init containers now wait for the toolkit validation marker and the loaded nvidia kernel module. The DCGM liveness probe now runs dcgmi discovery -l and requires GPU discovery. The probe uses explicit delay, period, timeout, and failure threshold settings.

Merge Risk: ⚪ Minimal · up to 119d9

DCGM and its exporter now wait for the NVIDIA driver module before startup, and DCGM restarts when it cannot discover a GPU. No concrete current-head merge-blocking risk remains.

---

_{Comment @coderabbitai help to get the list of available commands.}

[myeolenv]

Thanks for the contribution. We will review the PR soon.

[rajathagasthya]

Thanks for the contribution!

As mentioned in #2854 (comment), any fix for #2874 should also fix this. Although, I think the enhanced liveness probe could be valuable on its own.

Let's keep this open and we'll evaluate the probe changes.

[rahulait]

Hi @gengwg , we have merged changes for option 1 in #2881

Can you rebase your changes?

[rajathagasthya]

@gengwg As Rahul said, the issue is fixed by #2881. If you want to keep the liveness probe change, we can evaluate. Otherwise, close this PR. Let us know what you'd prefer.

[gengwg]

Hi @rahulait @rajathagasthya — rebased onto main and kept only the liveness probe change, so the diff is now one file.

The init-container change is dropped: #2881 already gates every NVIDIA-runtime operand on a live module, and it also covers the WSL2 /dev/dxg path that my version did not. Upstream already carries that gate, so this PR reduces to one file again and assets/state-dcgm-exporter/0800_daemonset.yaml matches upstream. (Note: the fork's main predates #2881, so a diff against the fork shows the gate as if it were ours — the PR is against NVIDIA's main.)

What remains is assets/state-dcgm/0400_dcgm.yml: the nvidia-dcgm hostengine liveness probe becomes an exec check on dcgmi discovery -l instead of tcpSocket: 5555, so a hostengine that came up without NVML is restarted by kubelet once the driver is back. Readiness is unchanged.

Two notes for the evaluation:

• The pattern ^[1-9][0-9]* GPUs? found is deliberate. dcgmi discovery -l prints 8 GPUs found (Active). (see dcgmi/Query.cpp) and 0 GPUs found (Active). when nothing is visible, so an anchored non-zero count accepts the first and rejects the second.
• The check is skipped while no driver is present (no nvidia module and no WSL2 /dev/dxg), so a planned driver restart does not loop the hostengine. Once the driver is back the probe fails and kubelet restarts the deaf hostengine within ~3 minutes. If you'd rather keep the probe unconditional, I can drop the guard — the tradeoff is a restart loop for the whole driver outage.
• With the guard, a node that genuinely loses every GPU while the driver is present still restarts its hostengine roughly every 3 minutes instead of sitting silently at 0 GPUs. That is intended — visible where the previous state was not.

Verified: TestOperandToolkitReadinessWaitsForLiveDriver and go test ./controllers/... ./internal/... pass on the rebase. I have not run this end to end on a cluster — my original image check was against dcgm:4.4.2 and values.yaml now defaults to 4.6.0, so I can re-verify against 4.6.0 if that helps.

Happy to keep it or drop it — your call.