fix: harden GPU runtime convergence

[dholt]

Summary

Harden GPU-cluster convergence across newer DGX OS, Slurm, and Ubuntu releases:

• Select the correct SSH service when applying the Slurm login GPU guard.
• Transition safely from legacy nv_peer_mem to nvidia_peermem.
• Make exclusive-job detection custom-prefix aware and fail closed on invalid Slurm query results.
• Use a command-scoped Enroot AppArmor profile without relaxing the host-wide user-namespace policy.
• Select and verify the DGX OS 7 DCGM 4 package on Ubuntu 24.04 and newer.
• Expose custom-prefix Slurm client binaries to the exporter through read-only mounts.
• Keep privileged driver/CUDA checks, GPU clock operations, and MIG administration outside the SSH login GPU cgroup using synchronous transient systemd services. This addresses subsequent full Ansible convergence after login isolation is already active.
• Require successful MIG capability detection before installation on both Debian and Red Hat families; avoid shell interpolation for GPU clock arguments.

This incorporates and hardens the useful changes proposed in #1389 through #1395. GPU power/clock batching from #1393 remains separate.

Validation

• python3 -m unittest discover scripts/validation/tests — 33 tests passed at d1f96221.
• git diff --check origin/master...HEAD passed.
• Public-content sanitizer passed for the candidate changes and this PR body.
• Exact-commit static release QA previously passed at d1f96221, including role lint, syntax checks, and behavior contracts.
• The preceding commit bc571ff9 passed GPU-backed Ubuntu 24.04 first-convergence/runtime validation: login GPU isolation, allocated srun GPU access, custom-prefix exporter metrics, Pyxis/Enroot, and exclusive/nonexclusive prolog selection. That evidence does not certify a second full convergence with the login guard already active or the new follow-up commit.

Notes

The updated head requires fresh public CI and maintainer review. Full two-OS release QA against the final merged master, including repeated full convergence with login isolation already active, remains required before release. No release certification is claimed for this candidate.

[100milliongold]

Thanks for pulling these together, and for the hardening on top — several of the
changes here fix real gaps in what I proposed. Adding apparmor to
pyxis_ubuntu_deps is one: my version would have failed on a minimal install
where apparmor_parser is absent. set -o pipefail, the fail-closed
num_jobs=1 default, and rejecting a zero node count are all stricter than what
I had.

One request on attribution. The body credits #1389–#1395, which I appreciate,
but the single squashed commit lists one author, so the contributions won't show
up in git log, git blame, or the contributor graph. Would you consider adding
Co-authored-by: trailers before this merges? For my part:

Co-authored-by: 100milliongold <je.kim@xiilab.com>

#1389, #1390 and #1394 came from a different contributor (@100-JM) — their
address is in those commits if you'd like to include them too.

Happy to close #1391, #1392 and #1395 once this lands. #1393 stays open as you
noted.

[100-JM]

Following up on the attribution point above for #1389, #1390 and #1394, which this PR incorporates (the ssh_unit selection, the nvidia_peermem probe with the legacy nv_peer_mem transition, and the DGX OS DCGM package handling).

I appreciate the extra hardening — the /sys/module/nv_peer_mem check before unloading and the post-install DCGM assert are both stricter than what I had. Since the squashed commit carries a single author, could you add a Co-authored-by: trailer so the contribution stays visible in git log and the contributor graph? The address matches the DCO sign-off on those commits:

Co-authored-by: 백지명 <wlaud9805@gmail.com>

I'll close #1389, #1390 and #1394 once this lands. #1397 (running the remaining nvidia-smi tasks outside the ssh cgroup) is newer and not covered here, so it stays open.

[lannex]

@dholt
As a fellow developer, I must express my deep disappointment with how the commits in this PR were handled.

Squashing the hard work and original code of multiple contributors (@100milliongold and @100-JM from PRs #1389 through #1395) into a single commit under one author's name goes against the fundamental principles of open-source collaboration.

While the additional hardening work is appreciated, stripping the original authors of their proper credit in the git history and contributor graph is simply unacceptable.

Please respect your fellow contributors and amend this commit to include the requested Co-authored-by trailers before merging. Proper attribution is a core pillar of the open-source community.

[dholt]

@100milliongold @100-JM, I'm sorry your contributions were incorporated here without proper commit attribution. I've corrected the commits to credit your work, including the follow-up from #1397. Thank you for contributing these fixes and for calling this out. This PR is on hold; further integration of your PRs will preserve your original commits and authorship rather than squash them, and both of you will be acknowledged in the next release notes.