ocf_hpc: add bounded CUDA driver diagnostics and maintenance guidance

[bluetopazz]

Purpose

A successful nvidia-smi query does not establish that CUDA can initialize. This replacement for #1493 installs a manual host diagnostic for basic CUDA driver availability and documents how to validate recovery through Slurm before reopening the node.

The pre-commit hook runs hardware-independent unit tests, not the live diagnostic. Deployment installs only the diagnostic file through ocf_hpc::compute; driver packages, UVM startup configuration, services, and scheduler policy are unchanged.

Changes from #1493

• Normalize timeout output before parsing or JSON serialization. Missing commands, command failures, malformed output, and timeouts become structured failures; package inventory failure remains nonfatal.
• Isolate CUDA loading/initialization in a child with a ten-second timeout. Kill and reap timed-out children; report crashes, missing symbols, and invalid child reports. Declare native function signatures.
• Reconcile NVIDIA GPU indices, physical device indices, and CUDA counts. Add optional --expected-gpus N using an administrator-confirmed inventory, with no hardcoded historical GPU count.
• Preserve the executable path, --compact, and existing report fields. Add scope: "basic_cuda_driver" and expected_gpu_count. Passing checks do not claim successful per-GPU computation, free memory, Slurm correctness, or complete inventory without an expected count.
• Keep module/device snapshots after CUDA initialization and avoid claiming that initialization has no transient effects.
• Correct both maintenance and stale-memory recovery instructions: drain and wait for existing work, repair and check the host, establish an administrator-only maintenance reservation, clear the drain for the reserved canary, and release the reservation only after validation. Failure requires re-draining before reservation cleanup.
• Make the distinction between unit tests and administrator-run host checks explicit in the hook, documentation, and this description.

The production diagnostic is 232 lines versus 226 in #1493. Most additional lines are regression tests; the patch remains confined to the original five files.

Local validation

• 30 tests passed on Python 3.9 and 3.11. Hardware-facing tests forbid real subprocess calls and CUDA loads. The process-lifecycle test starts only a harmless sleeping local Python child and checks timeout cleanup.
• Regressions cover bytes/text/absent timeout output, command failures, malformed and inconsistent inventories, missing CUDA libraries/symbols, child failures, expected counts, optional UVM-tools absence, full JSON reports, and CLI exit codes.
• The exact local hook configuration passed through pre-commit using an isolated local-hook config.
• Flake8 5.0.4, autopep8 1.5.4 diff check, and reorder-python-imports 1.9.0 passed for both Python files.
• Pre-commit configuration validation, Python 3.7 grammar validation, and git diff --check passed. Python 3.7 runtime execution has not been performed.
• Puppet 6.28.0 parser validation passed for compute.pp.
• Coverage reports 98% statement/branch coverage combined for the diagnostic in the local Python 3.11 run.

Existing full-suite bootstrap limitation

make test was attempted. The unchanged venv-update==4.0.0 bootstrap installs pip 18.1, which then fails on the local Python 3.11 environment with ImportError: cannot import name 'Mapping' from 'collections'. It stops before the repository-wide hooks. This PR does not modify that legacy tooling.

Outstanding before deployment

This is a draft because the following checks remain unverified:

• OCF Jenkins full repository checks, including its expected Python tooling.
• Personal Puppet environment and octocatalog validation: confirm the catalog change is only the diagnostic file on compute hosts.
• Administrator-run installed diagnostic on the target host with a confirmed GPU inventory.
• Administrator-run Slurm canary inside the maintenance reservation; verify computation on all allocated GPUs, process termination, memory release, and kernel logs before ordinary scheduling resumes.
• Administrator review of the reservation/recovery procedure against the deployed Slurm configuration.

No live node maintenance or GPU recovery was performed for this replacement.