build_orchestrator: merge model-card slurm/distributed config in the main build path

[i-kosarev]

Problem

_execute_with_prebuilt_image() already merged the first discovered model's slurm/distributed fields into manifest deployment_config, letting the run phase auto-detect target=slurm from the model card alone. The normal build-from-Dockerfile path (used whenever images are actually built from source, not --use-image) never did this merge -- it only called _save_deployment_config(), which populates deployment_config purely from --additional-context.

A model card with a real "slurm": {...} config but no matching --additional-context override was silently ignored: deployment_config.slurm stayed unset, _infer_deployment_target() fell back to "local", and on GPU-less nodes (e.g. a SLURM login node used only to submit jobs) this caused GPU detection failed: Unable to determine gpu vendor instead of dispatching via sbatch.

Fix

Extract the merge logic into _merge_model_config_into_deployment() and call it from both build paths right after _save_deployment_config().

[Copilot]

Pull request overview

This PR aims to prevent runtime GPU auto-detection from failing on SLURM login nodes (where no local GPU is present) by reusing GPU vendor information already recorded in the build manifest, allowing runs to proceed far enough to be dispatched to GPU compute nodes. It also reduces the risk of node-info commands hanging by bounding package-manager queries.

Changes:

• Pre-load gpu_vendor (and related GPU docker env vars) from the build manifest into additional_context before runtime context initialization.
• Add a 10-second timeout to host package-manager queries in _show_node_info().

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

src/madengine/orchestration/run_orchestrator.py:774

• _show_node_info() now relies on the external timeout binary being present. On minimal images/nodes where timeout (coreutils) isn’t installed, these commands will fail immediately and the node info output becomes less useful. Since Console.sh() already supports a timeout= parameter, consider using that and catching the timeout exception so this debug/info path remains best-effort without adding a runtime dependency on timeout.

        host_os = self.context.ctx.get("host_os", "")
        if "HOST_UBUNTU" in host_os:
            print(self.console.sh("timeout 10 apt show rocm-libs -a", canFail=True))
        elif "HOST_CENTOS" in host_os:
            print(self.console.sh("timeout 10 yum info rocm-libs", canFail=True))
        elif "HOST_SLES" in host_os:
            print(self.console.sh("timeout 10 zypper info rocm-libs", canFail=True))
        elif "HOST_AZURE" in host_os:
            print(self.console.sh("timeout 10 tdnf info rocm-libs", canFail=True))

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (6)

src/madengine/orchestration/build_orchestrator.py:1264

• deployment_config["distributed"] may already exist but not be a dict (or may be None). In that case, key not in saved_manifest["deployment_config"]["distributed"] will raise a TypeError. Ensure it's a dict before merging model_distributed.

            if "distributed" not in saved_manifest["deployment_config"]:
                saved_manifest["deployment_config"]["distributed"] = {}

src/madengine/orchestration/build_orchestrator.py:1288

• deployment_config["slurm"] may already exist but not be a dict (or may be None). In that case, assigning keys into it will raise a TypeError. Ensure it's a dict before merging model_slurm.

            if "slurm" not in saved_manifest["deployment_config"]:
                saved_manifest["deployment_config"]["slurm"] = {}

src/madengine/orchestration/run_orchestrator.py:768

• Prefer using Console.sh's built-in timeout parameter instead of shelling out to the external timeout utility (which may not exist on all distros/containers). This also keeps the printed command clearer.

            print(self.console.sh("timeout 10 apt show rocm-libs -a", canFail=True))

src/madengine/orchestration/run_orchestrator.py:770

• Same as above: use Console.sh(timeout=10) rather than prefixing the command with timeout 10.

            print(self.console.sh("timeout 10 yum info rocm-libs", canFail=True))

src/madengine/orchestration/run_orchestrator.py:772

• Same as above: use Console.sh(timeout=10) rather than prefixing the command with timeout 10.

            print(self.console.sh("timeout 10 zypper info rocm-libs", canFail=True))

src/madengine/orchestration/run_orchestrator.py:774

• Same as above: use Console.sh(timeout=10) rather than prefixing the command with timeout 10.

            print(self.console.sh("timeout 10 tdnf info rocm-libs", canFail=True))

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.