From af14769223c072cb8b1f21dff17837bc98d1144e Mon Sep 17 00:00:00 2001 From: OutisLi Date: Mon, 17 Aug 2026 12:02:55 +0800 Subject: [PATCH 01/11] feat: add DeePMD installation agent skill --- README.md | 21 +- doc/agent-skills.md | 44 +- doc/index.rst | 4 +- doc/install/easy-install.md | 2 + doc/install/index.rst | 1 + doc/install/install-from-source.md | 3 + doc/install/install-lammps.md | 4 + doc/install/install-with-agent.md | 63 ++ pyproject.toml | 2 + skills/deepmd-install/SKILL.md | 141 ++++ .../deepmd-install/references/easy-install.md | 162 +++++ .../references/failure-modes.md | 279 ++++++++ .../deepmd-install/references/plan-schema.md | 311 +++++++++ .../deepmd-install/references/source-cpp.md | 158 +++++ .../references/source-lammps.md | 237 +++++++ .../references/source-python.md | 185 +++++ .../deepmd-install/scripts/prepare_lammps.py | 169 +++++ skills/deepmd-install/scripts/probe_env.py | 426 ++++++++++++ .../deepmd-install/scripts/validate_plan.py | 652 ++++++++++++++++++ .../deepmd-install/scripts/verify_lammps.py | 208 ++++++ .../deepmd-install/scripts/verify_python.py | 341 +++++++++ source/tests/test_deepmd_install_skill.py | 402 +++++++++++ 22 files changed, 3803 insertions(+), 12 deletions(-) create mode 100644 doc/install/install-with-agent.md create mode 100644 skills/deepmd-install/SKILL.md create mode 100644 skills/deepmd-install/references/easy-install.md create mode 100644 skills/deepmd-install/references/failure-modes.md create mode 100644 skills/deepmd-install/references/plan-schema.md create mode 100644 skills/deepmd-install/references/source-cpp.md create mode 100644 skills/deepmd-install/references/source-lammps.md create mode 100644 skills/deepmd-install/references/source-python.md create mode 100644 skills/deepmd-install/scripts/prepare_lammps.py create mode 100644 skills/deepmd-install/scripts/probe_env.py create mode 100644 skills/deepmd-install/scripts/validate_plan.py create mode 100644 skills/deepmd-install/scripts/verify_lammps.py create mode 100644 skills/deepmd-install/scripts/verify_python.py create mode 100644 source/tests/test_deepmd_install_skill.py diff --git a/README.md b/README.md index 493b963ec3..606b1224bc 100644 --- a/README.md +++ b/README.md @@ -212,9 +212,27 @@ plugins. Applications can therefore open the backend required by a model without directly linking every framework. > [!NOTE] -> Working with an AI coding or scientific agent? DeePMD-kit ships +> Working with an AI coding or scientific agent? Start with +> [Install with an AI agent][agent-install], or browse the > [official Agent Skills][agent-skills] for model selection, training, > fine-tuning, Python inference, and LAMMPS workflows. +> +> ```bash +> npx -y skills add https://github.com/deepmodeling/deepmd-kit/tree/master/skills \ +> --skill deepmd-install -y +> ``` +> +> If direct GitHub access fails, use `gh-proxy.com` for a public, read-only +> clone, then install from the local checkout. Do not send credentials or +> private repository URLs through the proxy. +> +> ```bash +> git clone --depth 1 \ +> https://gh-proxy.com/https://github.com/deepmodeling/deepmd-kit.git \ +> deepmd-kit-skill-source +> npx -y skills add ./deepmd-kit-skill-source/skills \ +> --skill deepmd-install -y +> ``` ## 📚 Documentation and community @@ -251,6 +269,7 @@ that matches the version used and the method-specific papers listed in DeePMD-kit is licensed under the [GNU Lesser General Public License v3.0 or later](./LICENSE). +[agent-install]: https://docs.deepmodeling.com/projects/deepmd/en/latest/install/install-with-agent.html [agent-skills]: https://docs.deepmodeling.com/projects/deepmd/en/latest/agent-skills.html [ase]: https://docs.deepmodeling.com/projects/deepmd/en/latest/third-party/ase.html [backends]: https://docs.deepmodeling.com/projects/deepmd/en/latest/backend.html diff --git a/doc/agent-skills.md b/doc/agent-skills.md index 54abd87d2e..3074cc4527 100644 --- a/doc/agent-skills.md +++ b/doc/agent-skills.md @@ -2,7 +2,7 @@ DeePMD-kit provides official [Agent Skills](https://agentskills.io/what-are-skills) that help AI agents run DeePMD-kit workflows in a reproducible way. These skills capture -project-specific operating knowledge—such as training inputs, model +project-specific operating knowledge—such as installation, training inputs, model selection, deployment, LAMMPS integration, and Python inference patterns—so an agent can turn a high-level request into concrete files, commands, and validation steps. @@ -14,6 +14,14 @@ in the DeePMD-kit repository under `skills/`. ## List of skills +- `deepmd-install`: Install DeePMD-kit for users. The skill probes the machine, + records the selected environment and build in a validated plan, then follows + either an easy path (conda, pip, Docker, offline, `dp1s`) or a source build of + the Python package, C/C++ interface, and LAMMPS. Kokkos builds use one + `Kokkos_ARCH_*` flag per binary. DPA4/SeZM uses `deepmd/kk`; DPA4C uses + `dpa4spin/kk` with a compact canonical graph artifact. Detailed recipes live + under `skills/deepmd-install/references/` and are loaded only for the selected + path or a matching failure mode. - `deepmd-train`: Choose a DeePMD-kit model family, then train from scratch. The skill uses progressive disclosure: the top-level workflow handles common training steps and model selection, while model-specific configuration lives @@ -40,6 +48,10 @@ paper: ## Install skills +To have an agent install DeePMD-kit itself, send +[Install with an AI agent](install/install-with-agent.md) and ask it to load +`deepmd-install`. + ### If you are a user The easiest way is to send this page to your agent and ask it to install the @@ -51,7 +63,7 @@ If you already have a DeePMD-kit checkout, run this command from the repository root: ```bash -npx -y skills add ./skills -a openclaw -y +npx -y skills add ./skills --skill '*' -y ``` If you do not have a checkout, the same skills can also be installed directly @@ -59,15 +71,26 @@ from GitHub: ```bash npx -y skills add https://github.com/deepmodeling/deepmd-kit/tree/master/skills \ - -a openclaw -y + --skill '*' -y +``` + +If direct GitHub access fails, use `gh-proxy.com` for a public, read-only clone +and install from that checkout. Do not send credentials or private repository +URLs through the proxy. + +```bash +git clone --depth 1 \ + https://gh-proxy.com/https://github.com/deepmodeling/deepmd-kit.git \ + deepmd-kit-skill-source +npx -y skills add ./deepmd-kit-skill-source/skills --skill '*' -y ``` -The examples above require Node.js/npm so that `npx` is available, and they -install the skills for OpenClaw. Replace `openclaw` with the target agent name -when installing for another agent. The GitHub command lets the skill CLI fetch -the repository for you. For large repositories or slow networks, this can take -longer than installing from an existing local checkout. Refresh or restart the -session afterward so the installed skills are reloaded. +The examples require Node.js/npm so that `npx` is available. The Skills CLI +installs every official skill for the detected agent. To target one product, +add its agent name, for example `--agent cursor` or `--agent claude-code`. The +GitHub command lets the CLI fetch the repository; installing from an existing +checkout avoids that download. Refresh or restart the session afterward so the +installed skills are reloaded. ## Minimal verification @@ -80,3 +103,6 @@ without launching an expensive calculation. For example: water dataset and draft a training input, but do not start training.” - “Use the `lammps-deepmd` skill to prepare an NVT LAMMPS input file for a DeePMD-kit model, and explain each command.” +- “Use the `deepmd-install` skill to plan a PyTorch CUDA source install, but + ask me for the CUDA toolkit path, PyTorch wheel, and install prefix first. + Do not start compiling.” diff --git a/doc/index.rst b/doc/index.rst index 11d2c2e3dc..9de8ef30e5 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -276,8 +276,8 @@ New and noteworthy :link-type: doc :shadow: sm - Give AI agents reproducible guidance for training, fine-tuning, - inference, and LAMMPS workflows. + Give AI agents reproducible guidance for installation, training, + fine-tuning, inference, and LAMMPS workflows. Documentation map ================= diff --git a/doc/install/easy-install.md b/doc/install/easy-install.md index 1eef898316..23d4e82327 100644 --- a/doc/install/easy-install.md +++ b/doc/install/easy-install.md @@ -2,6 +2,8 @@ There are various easy methods to install DeePMD-kit. Choose one that you prefer. If you want to build by yourself, jump to the next two sections. +An AI agent can perform the same installation. See [Install with an AI agent](install-with-agent.md). + After your easy installation, DeePMD-kit (`dp`) and LAMMPS (`lmp`) will be available to execute. You can try `dp -h` and `lmp -h` to see the help. `mpirun` is also available considering you may want to train models or run LAMMPS in parallel. > [!NOTE] diff --git a/doc/install/index.rst b/doc/install/index.rst index 6491e6b787..32691b7e29 100644 --- a/doc/install/index.rst +++ b/doc/install/index.rst @@ -5,6 +5,7 @@ Installation :maxdepth: 1 easy-install + install-with-agent install-from-source install-from-c-library install-lammps diff --git a/doc/install/install-from-source.md b/doc/install/install-from-source.md index 8f4b933440..3d7aa75606 100644 --- a/doc/install/install-from-source.md +++ b/doc/install/install-from-source.md @@ -1,5 +1,8 @@ # Install from source code +An AI agent can walk through this source build, including the C++ interface +and LAMMPS with Kokkos. See [Install with an AI agent](install-with-agent.md). + Please follow our [GitHub](https://github.com/deepmodeling/deepmd-kit) webpage to download the source code of a specific version or the [development version](https://github.com/deepmodeling/deepmd-kit/tree/master). Or get the DeePMD-kit source code by `git clone` diff --git a/doc/install/install-lammps.md b/doc/install/install-lammps.md index d9fcbc4e7c..30828e614e 100644 --- a/doc/install/install-lammps.md +++ b/doc/install/install-lammps.md @@ -1,5 +1,9 @@ # Install LAMMPS +An AI agent can build LAMMPS with the DeePMD-kit module and Kokkos. DPA4/SeZM +uses `pair_style deepmd/kk`; DPA4C uses `pair_style dpa4spin/kk`. See +[Install with an AI agent](install-with-agent.md). + There are two ways to install LAMMPS: the built-in mode and the plugin mode. The built-in mode builds LAMMPS along with the DeePMD-kit and DeePMD-kit will be loaded automatically when running LAMMPS. The plugin mode builds LAMMPS and a plugin separately, so one needs to use `plugin load` command to load the DeePMD-kit's LAMMPS plugin library. ## Install LAMMPS's DeePMD-kit module (built-in mode) diff --git a/doc/install/install-with-agent.md b/doc/install/install-with-agent.md new file mode 100644 index 0000000000..ff48cdcb9e --- /dev/null +++ b/doc/install/install-with-agent.md @@ -0,0 +1,63 @@ +# Install with an AI agent + +DeePMD-kit ships an official Agent Skill, `deepmd-install`, that walks an AI +agent through installing the package. The skill covers easy methods (conda, +pip, Docker, offline installers, `dp1s`) and source builds of the Python +package, the C++ interface, and LAMMPS with Kokkos (`pair_style deepmd/kk`, +used by DPA4/SeZM, and `pair_style dpa4spin/kk`, used by DPA4C). + +The skill probes the machine, asks only for decisions required by the selected +installation path, and records concrete paths and versions in a validated +plan. It does not assume a host path, package mirror, CUDA toolkit, or backend +version. + +The full skill catalog is in [Agent Skills](../agent-skills.md). For a +manual install, use [Easy install](easy-install.md) or +[Install from source](install-from-source.md). + +## If you are a user + +Send this page to your agent and ask it to install the official +`deepmd-install` skill, then install DeePMD-kit. You do not need to run the +commands below yourself. + +## If you are an agent + +Install `deepmd-install` first, then follow its probe, validated plan, and +gate workflow. + +If this machine already has a DeePMD-kit checkout, run from the repository +root: + +```bash +npx -y skills add ./skills --skill deepmd-install -y +``` + +If there is no checkout, install the same skill from GitHub: + +```bash +npx -y skills add https://github.com/deepmodeling/deepmd-kit/tree/master/skills \ + --skill deepmd-install -y +``` + +If direct GitHub access fails, use `gh-proxy.com` for a public, read-only clone +and install from the local checkout. Do not send credentials or private +repository URLs through the proxy. + +```bash +git clone --depth 1 \ + https://gh-proxy.com/https://github.com/deepmodeling/deepmd-kit.git \ + deepmd-kit-skill-source +npx -y skills add ./deepmd-kit-skill-source/skills \ + --skill deepmd-install -y +``` + +The examples require Node.js/npm so that `npx` is available. The Skills CLI +uses the detected agent. To target one product, add `--agent cursor`, +`--agent claude-code`, or another supported agent name. Refresh or restart the +session afterward so the skill is reloaded. + +Then use the `deepmd-install` skill: probe the machine, ask only for missing +decisions, validate `install-plan.json`, and execute one gate at a time. A +CUDA LAMMPS build uses exactly one `Kokkos_ARCH_*` flag per binary and verifies +the pair styles required by the selected model family. diff --git a/pyproject.toml b/pyproject.toml index 3b64d3ec1b..3a7edcb221 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -493,6 +493,8 @@ runtime-evaluated-base-classes = ["torch.nn.Module"] "**/*.ipynb" = ["T20"] # printing in a nb file is expected # Example / demo scripts are run directly: top-level imports and prints are fine. "examples/**/*.py" = ["T20", "TID253"] +# Skill helper CLIs print probe and verification reports. +"skills/**/*.py" = ["T20"] [tool.pytest.ini_options] markers = "run" diff --git a/skills/deepmd-install/SKILL.md b/skills/deepmd-install/SKILL.md new file mode 100644 index 0000000000..4e7771d4be --- /dev/null +++ b/skills/deepmd-install/SKILL.md @@ -0,0 +1,141 @@ +--- +name: deepmd-install +description: Install or rebuild DeePMD-kit with conda, pip, dp1s, offline packages, Docker, or a source checkout. Use for CPU, NVIDIA CUDA, or ROCm environments; PyTorch, TensorFlow, JAX, or Paddle backends; the C/C++ interface; and DeePMD-enabled LAMMPS, including Kokkos pair styles for DPA4 and DPA4C. +--- + +# Install DeePMD-kit + +Install the smallest runtime that satisfies the user's goal. Probe first, keep +all decisions in a validated plan, execute one self-contained gate at a time, +and verify the requested public interface rather than package presence alone. + +## Supported workflows + +| Path | Scope | +| ----------------------------------------------- | ---------------------------------------------------------------------------------------------- | +| conda, pip | Stable Python installs for PyTorch, TensorFlow, JAX, and Paddle; optional packaged host LAMMPS | +| `dp1s`, offline package, Docker | Official release artifacts selected by the user or matching documentation | +| source Python | PyTorch, TensorFlow, JAX, and Paddle with backend-specific dependencies and verification | +| source C/C++ | Backend-enabled C/C++ libraries installed into a dedicated prefix | +| source LAMMPS host | Built-in DeePMD module with exact pair-style verification | +| source LAMMPS Kokkos CUDA | PyTorch graph artifacts: `deepmd/kk` for DPA4/SeZM and `dpa4spin/kk` for DPA4C | +| ROCm source, Windows source, LAMMPS plugin mode | Follow the version-matched documentation in the selected checkout | + +## Hard rules + +1. Resolve the absolute directory containing this `SKILL.md` as `SKILL_ROOT`. + Invoke every bundled script through that absolute path. +1. Run the read-only probe before choosing versions, paths, or accelerators. +1. Ask only for a missing value that changes the selected path. Do not ask + source, compiler, CUDA, or LAMMPS questions for an unrelated easy install. +1. Record all decisions in `install-plan.json` and validate it before an + install, checkout, compile, download, or environment modification. +1. Use the plan's absolute Python executable. Never substitute a bare + `python`, `pip`, or an assumed conda activation. +1. Make every shell call self-contained. Do not rely on an `export`, `cd`, or + `conda activate` from a previous agent tool call. +1. Render commands with concrete plan values. Stop if a plan placeholder, + empty required value, unexpected path, or shell variable not assigned + earlier in the same command block remains. +1. Never run `git reset --hard`, `git clean`, recursively remove an install + prefix, or edit shell rc files as part of this workflow. Use a new build + directory or versioned install prefix instead. +1. Use documentation from the selected checkout for version-sensitive source + options. Do not apply `latest` documentation blindly to an older ref. +1. Confirm before piping a remote script to a shell. Use fail-on-HTTP-error + downloads and verify a checksum whenever the plan contains one. +1. Check GPU occupancy and bind the confirmed physical device explicitly + before a GPU smoke test. +1. On failure, stop the current gate and read + [`references/failure-modes.md`](references/failure-modes.md). Do not restart + the entire install or wipe caches. + +## Workflow + +### 0. Probe + +Run with an available Python 3.10+ interpreter: + +```bash +"" "/scripts/probe_env.py" --json +``` + +Use the report to identify the OS, libc, Python environments, compilers, +toolkits, driver, GPU compute capabilities and visibility mask, disk/RAM, and +existing DeePMD-kit/PyTorch installs. The probe does not select versions. + +### 1. Select one path + +- Prefer an easy method for a stable release without local source changes, + custom C/C++ libraries, or Kokkos device pair styles. +- Use source Python for a branch/fork, unreleased feature, local toolkit build, + or customized OPs. +- Add source C/C++ only when a C/C++ client, source-built LAMMPS, or i-PI needs + it. +- Add source LAMMPS only after the C/C++ gate passes. + +### 2. Create and validate the plan + +Read [`references/plan-schema.md`](references/plan-schema.md), write the plan to +a stable scratch path outside the source/build trees, and validate it: + +```bash +"" "/scripts/validate_plan.py" \ + "" +``` + +Present a concise summary only when the request or probe leaves a meaningful +choice unresolved. If the user already specified the method, backend, version, +and target, proceed without asking them to reconfirm their own request. + +### 3. Execute the selected references + +| Selection | Read | +| ----------------------------------- | ------------------------------------------------------------ | +| conda, pip, `dp1s`, offline, Docker | [`references/easy-install.md`](references/easy-install.md) | +| source Python | [`references/source-python.md`](references/source-python.md) | +| source C/C++ | [`references/source-cpp.md`](references/source-cpp.md) | +| source LAMMPS | [`references/source-lammps.md`](references/source-lammps.md) | +| failed gate | [`references/failure-modes.md`](references/failure-modes.md) | + +Read only the references required by the plan. Within each gate, render one +command block with absolute values so that directory and environment state +cannot leak across calls. + +### 4. Enforce the gates + +| Gate | Required evidence | +| ----------- | --------------------------------------------------------------------------------------------------------------------------------- | +| Plan | `validate_plan.py` exits zero and prints the normalized plan | +| Environment | The selected package manager and absolute interpreter target the planned environment | +| Python | `verify_python.py` passes for the selected backend and accelerator; source builds also match the expected build variant | +| C/C++ | Expected libraries and headers exist, dynamic dependencies resolve, and the build cache records the requested accelerator/backend | +| LAMMPS | `verify_lammps.py` finds the exact pair styles required by the model family and no unresolved dynamic dependency | +| Smoke | The selected short example finishes on the explicitly bound device and emits its documented success signal | + +For PyTorch source builds, install the intended PyTorch before DeePMD-kit and +use `--no-build-isolation`. Set both `DP_ENABLE_PYTORCH` and +`DP_ENABLE_TENSORFLOW` explicitly in the same command that invokes pip. + +## LAMMPS model contract + +| Model/runtime | Host pair style | Kokkos CUDA pair style | Artifact requirement | +| ------------------- | --------------- | ----------------------------------------------------------- | --------------------------------------------------------- | +| conventional models | `deepmd` | `deepmd/kk` when the model supports device edge/graph input | Compatible frozen model; `/kk` requires edge/graph `.pt2` | +| DPA4 / SeZM | `deepmd` | `deepmd/kk` | Target-specific graph `.pt2` | +| DPA4C native spin | `dpa4spin` | `dpa4spin/kk` | `atom_style spin` and compact canonical graph `.pt2` | + +Do not accept `deepmd/kk` as proof that DPA4C is available. Do not substitute +LAMMPS `PKG_GPU` for Kokkos. + +## Completion report + +Report: + +- plan path and resolved source commit, when applicable; +- environment activation or absolute Python path; +- installed DeePMD-kit/backend versions and accelerator visibility; +- C/C++ prefix and LAMMPS binary, when applicable; +- each gate command and observed result; +- any validation gate that was not run and the concrete reason; +- no claim of success beyond the gates that actually passed. diff --git a/skills/deepmd-install/references/easy-install.md b/skills/deepmd-install/references/easy-install.md new file mode 100644 index 0000000000..e0b113c64e --- /dev/null +++ b/skills/deepmd-install/references/easy-install.md @@ -0,0 +1,162 @@ +# Easy installation + +Use this reference for stable package releases and official artifacts. Read the +version-matched `doc/install/easy-install.md` when package extras, platform +support, or backend installation commands differ from this routing guide. The +published documentation is +. + +## Contents + +- [Environment gate](#environment-gate) +- [Pip](#pip) +- [Conda](#conda) +- [dp1s](#dp1s) +- [Offline package](#offline-package) +- [Docker](#docker) +- [Verification](#verification) + +## Environment gate + +Do not mix conda, pip, offline, and `dp1s` installations in one prefix. Reuse +an environment only when the user selected it and the probe found no conflicting +DeePMD-kit install. + +For a new venv, create it first, resolve its absolute interpreter, update the +plan, and re-run `validate_plan.py` before installing packages: + +```bash +"" -m venv "" +"/bin/python" -c "import sys; print(sys.executable)" +``` + +Use the platform-equivalent interpreter path on Windows. + +## Pip + +Install backend packages recorded in `package.backend_packages` before +DeePMD-kit when they require a dedicated index. Render one literal pip command; +omit `--index-url` when `backend_index_url` is null. + +```bash +"" -m pip install \ + "" "" \ + --index-url "" +``` + +Choose DeePMD-kit extras from the matching checkout documentation: + +| Backend | DeePMD-kit requirement | +| -------------- | ------------------------------------------------------------------------- | +| PyTorch | `deepmd-kit[torch]` | +| TensorFlow CPU | `deepmd-kit[cpu]` | +| TensorFlow GPU | `deepmd-kit[gpu,cu12]` when the documented CUDA runtime extra is required | +| JAX | `deepmd-kit[jax]` plus the selected JAX accelerator package | +| Paddle | Install the selected Paddle package first, then `deepmd-kit` | + +Append `lmp` and/or `ipi` only when the plan enables them. Append the exact +DeePMD-kit version when `package.deepmd_version` is non-null. Example shape: + +```bash +"" -m pip install "deepmd-kit[torch,lmp]==" +``` + +The final rendered command must contain a real version or omit the version +specifier; it must not contain ``. Add `--index-url` and +`--extra-index-url` only when the corresponding DeePMD-kit index fields are +non-null. + +## Conda + +Use the absolute manager and environment name from the plan. Install only the +requested packages; do not add LAMMPS, Horovod, or MPI unless they are in scope. + +```bash +"" create -n "" \ + -c conda-forge deepmd-kit +``` + +Add `lammps` only for `goal=python+lammps`. Resolve the created interpreter +without assuming shell activation: + +```bash +"" run -n "" \ + python -c "import sys; print(sys.executable)" +``` + +Record that absolute interpreter in the plan before running the Python verifier. +Follow the conda-forge CUDA guidance referenced by +`doc/install/easy-install.md`; do not invent a `cudatoolkit` pin. + +## dp1s + +Show the exact command and obtain confirmation before piping a remote response +to a shell: + +```bash +curl -fsSL https://dp1s.deepmodeling.com | bash +``` + +Apply only options selected from the official `dp1s` documentation. After the +installer reports its prefix, resolve the installed Python and update the plan +before verification. + +## Offline package + +Use the exact release asset and SHA-256 checksum recorded in the plan. For a +remote artifact: + +```bash +curl -fL "" -o "" +printf '%s %s\n' "" "" | sha256sum --check - +bash "" +``` + +Use `shasum -a 256` on systems without `sha256sum`. When a release is split, +verify each part before concatenating it into a new file. Never execute an HTML +error page or an artifact whose checksum is unknown. + +## Docker + +Pull the exact image reference from the plan and verify it in a disposable +container: + +```bash +docker pull "" +docker run --rm "" dp --version +``` + +For CUDA, bind the physical GPU selected for the smoke test: + +```bash +docker run --rm --gpus 'device=' \ + "" \ + python -c "import torch; print(torch.cuda.get_device_name(0))" +``` + +Add only user-selected volume mounts and working directories. Do not mount a +home directory or source tree read-write merely for version verification. + +## Verification + +Run the backend-aware verifier with the absolute installed interpreter: + +```bash +"" "/scripts/verify_python.py" \ + --backend "" \ + --accelerator "" +``` + +For packaged host LAMMPS, resolve the executable and require the conventional +host pair style: + +```bash +"" "/scripts/verify_lammps.py" \ + --binary "" \ + --model-family conventional \ + --flavor host +``` + +Packaged LAMMPS does not imply Kokkos `deepmd/kk` or DPA4C +`dpa4spin/kk`. Use the source LAMMPS path when either device pair style is +required. diff --git a/skills/deepmd-install/references/failure-modes.md b/skills/deepmd-install/references/failure-modes.md new file mode 100644 index 0000000000..de3627080b --- /dev/null +++ b/skills/deepmd-install/references/failure-modes.md @@ -0,0 +1,279 @@ +# Installation failure modes + +Diagnose the current gate only. Preserve the plan and complete logs, change one +input at a time, and re-run the smallest failing command. Do not restart the +workflow, wipe caches, reset source trees, or remove install prefixes. + +## Index + +| Symptom | Section | +| ----------------------------------------------- | ----------------------------------------------------------------------- | +| `python`, pip, and `dp` disagree | [Wrong Python environment](#wrong-python-environment) | +| A plan value is empty or a placeholder remains | [Invalid or stale plan](#invalid-or-stale-plan) | +| A bundled helper cannot be found | [Wrong skill root](#wrong-skill-root) | +| Backend imports but accelerator is unavailable | [Backend accelerator failure](#backend-accelerator-failure) | +| DeePMD-kit reports the wrong build variant | [Wrong compiled variant](#wrong-compiled-variant) | +| `ENABLE_CUSTOMIZED_OP` is false | [PyTorch custom OP is unavailable](#pytorch-custom-op-is-unavailable) | +| CUDA/toolkit/compiler error | [CUDA toolchain mismatch](#cuda-toolchain-mismatch) | +| nvalchemi or vesin is unavailable | [Optional neighbor-list dependency](#optional-neighbor-list-dependency) | +| Source checkout is not the requested ref | [Source identity mismatch](#source-identity-mismatch) | +| CMake keeps an old compiler/backend | [Stale build directory](#stale-build-directory) | +| Torch, NCCL, CUPTI, or CUDA library is missing | [Native library discovery](#native-library-discovery) | +| LAMMPS has no DeePMD styles | [Built-in module is absent](#built-in-module-is-absent) | +| `deepmd/kk` or `dpa4spin/kk` is absent | [Wrong LAMMPS runtime](#wrong-lammps-runtime) | +| Kokkos reports an architecture error | [Kokkos architecture mismatch](#kokkos-architecture-mismatch) | +| LAMMPS starts but a shared library is not found | [Runtime link failure](#runtime-link-failure) | +| Build terminates from memory or disk pressure | [Insufficient build resources](#insufficient-build-resources) | +| Downloaded archive cannot be parsed or verified | [Artifact download failure](#artifact-download-failure) | + +## Wrong Python environment + +Run all checks with the absolute interpreter from the plan: + +```bash +"" -c "import sys; print(sys.executable); print(sys.prefix)" +"" -m pip --version +command -v dp +head -n 1 "$(command -v dp)" +``` + +The interpreter, pip prefix, and `dp` shebang must identify the same +environment. A previous `conda activate` does not persist across independent +agent shell calls. Re-render the failed gate with the absolute interpreter. + +## Invalid or stale plan + +Re-run validation whenever a path, version, backend, accelerator, source ref, +or build directory changes: + +```bash +"" "/scripts/validate_plan.py" \ + "" +``` + +Never repair a failed command by guessing a missing value. Update the plan, +validate it, and render the gate again. A command containing an unassigned +plan variable, an empty required argument, or `` is not +executable. + +## Wrong skill root + +Helper scripts belong to the installed skill, not the DeePMD-kit checkout or +current directory: + +```bash +test -f "/scripts/probe_env.py" +test -f "/scripts/verify_python.py" +test -f "/scripts/verify_lammps.py" +``` + +Resolve the directory containing `SKILL.md`; do not search for a same-named +script elsewhere and do not skip the gate. + +## Backend accelerator failure + +Use the backend-aware verifier and retain its complete output: + +```bash +"" "/scripts/verify_python.py" \ + --backend "" \ + --accelerator "" +``` + +Check visibility masks from the probe before replacing packages. A false GPU +availability result may come from `CUDA_VISIBLE_DEVICES`, +`HIP_VISIBLE_DEVICES`, a driver/runtime mismatch, or a CPU backend package. + +## Wrong compiled variant + +Framework GPU visibility does not prove that DeePMD-kit custom operations were +compiled for that accelerator. Inspect the recorded variant: + +```bash +"" - <<'PY' +from deepmd.env import GLOBAL_CONFIG + +print(GLOBAL_CONFIG["dp_variant"]) +PY +``` + +Return to the source Python gate when it differs from `build.variant`. Keep the +toolkit, compiler, `DP_VARIANT`, and backend switches in the same pip call. + +## PyTorch custom OP is unavailable + +Common causes are build isolation, a missing PyTorch installation during the +build, `DP_ENABLE_PYTORCH=0`, an ABI mismatch, or a runtime library that cannot +be loaded. + +Confirm the intended PyTorch and toolchain, then re-run the source install with +`--no-build-isolation` and the complete inline environment from +[`source-python.md`](source-python.md). Do not use `--force-reinstall` until the +rendered command is known to contain the correct plan values. + +## CUDA toolchain mismatch + +Compare all three layers: + +```bash +"/bin/nvcc" --version +nvidia-smi +"" -c \ + "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())" +``` + +A CUDA major-version mismatch between toolkit and PyTorch is a hard failure. +A same-major minor mismatch requires the actual configure/compile result; +PyTorch treats it as a warning in extension builds. The NVIDIA driver must +support the selected runtime. If `nvcc` rejects the host compiler, select a +compiler supported by that toolkit and update the plan rather than using +`--allow-unsupported-compiler`. + +## Optional neighbor-list dependency + +`vesin[torch]` belongs to the PyTorch extra. nvalchemi package markers depend on +the operating system and Python version. Require these integrations only when +the selected checkout and platform declare them applicable: + +```bash +"" -m pip show vesin nvalchemi-toolkit-ops +``` + +Their absence on an ineligible platform is not a core DeePMD-kit installation +failure. + +## Source identity mismatch + +Inspect without modifying the checkout: + +```bash +git -C "" status --short +git -C "" remote -v +git -C "" rev-parse HEAD +``` + +Use a separate clone when `HEAD`, remote, or local changes do not match the +plan. Do not reset, clean, or rewrite the existing checkout. + +## Stale build directory + +A CMake cache records absolute source paths, compilers, toolkit, backend, and +architecture. Compare it with the plan: + +```bash +grep -E \ + 'CMAKE_(C|CXX|CUDA)_COMPILER:|CMAKE_INSTALL_PREFIX:|ENABLE_|USE_.*TOOLKIT:|Kokkos_ARCH_' \ + "/CMakeCache.txt" +``` + +If any identity differs, create a new build directory. Preserve the old +directory for diagnosis until the replacement passes. + +## Native library discovery + +Locate PyTorch and namespace-package NCCL from the selected environment: + +```bash +"" - <<'PY' +from importlib.util import find_spec +from pathlib import Path + +import torch + +print(Path(torch.__file__).resolve().parent / "lib") +spec = find_spec("nvidia.nccl") +if spec is not None: + for root in spec.submodule_search_locations or (): + candidate = Path(root) / "lib" + if list(candidate.glob("libnccl.so*")): + print(candidate) +PY +``` + +`nvidia.nccl.__file__` may be null. Locate CUPTI independently; it may live +under `extras/CUPTI`, an environment package, or a separate toolkit component. +Add a directory to RPATH or linker flags only after the named library is found +there. + +## Built-in module is absent + +Check the managed include without changing the LAMMPS tree: + +```bash +"" "/scripts/prepare_lammps.py" \ + --lammps-source "" \ + --deepmd-source "" \ + --check +``` + +If it differs, run the preparer without `--check`, then configure a new LAMMPS +build directory. Re-running CMake in the old directory may preserve a source +state from before the include was added. + +## Wrong LAMMPS runtime + +Verify exact styles rather than grepping any occurrence of `deepmd`: + +```bash +"" "/scripts/verify_lammps.py" \ + --binary "" \ + --model-family "" \ + --flavor "" +``` + +Kokkos must be enabled for `/kk`. DPA4C requires `dpa4spin/kk`, not +`deepmd/kk`. A CPU DeePMD C/C++ library cannot serve the Kokkos CUDA pair +styles. `PKG_GPU` does not provide them. + +If the exact style exists but rejects the model, check the artifact contract: +`deepmd/kk` needs an edge/graph `.pt2`, one model, and `atom_modify map yes`; +`dpa4spin/kk` needs a DPA4C compact canonical graph and `atom_style spin`. + +## Kokkos architecture mismatch + +Query numeric capability and compare it with both the plan and selected Kokkos +tree: + +```bash +nvidia-smi --query-gpu=index,name,compute_cap --format=csv,noheader +grep -F "kokkos_arch_option(" \ + "/lib/kokkos/cmake/kokkos_arch.cmake" +``` + +Use exactly one architecture in one build directory. Do not infer it from a +GPU-name substring. If the compiler test requires `nvcc_wrapper`, use the +wrapper from that LAMMPS tree with the host compiler supported by the selected +CUDA toolkit. + +## Runtime link failure + +Inspect the exact binary: + +```bash +ldd "" +``` + +Resolve every `not found` entry through build/install RPATH. Avoid global +`LD_LIBRARY_PATH` and shell-rc changes; they can cause another environment's +CUDA, Torch, TensorFlow, or compiler runtime to shadow the planned libraries. + +## Insufficient build resources + +Compare the probe's free disk, RAM, and CPU count with `build.jobs`. Reduce the +job count and re-run the same build command. Do not change compilers, backend, +or source ref in the same experiment. Preserve the first out-of-memory or disk +error because later compiler failures may be secondary damage. + +## Artifact download failure + +Use `curl -fL` so HTTP errors fail immediately. Verify file type and checksum +before extraction or execution: + +```bash +file "" +printf '%s %s\n' "" "" | sha256sum --check - +``` + +An HTML response, truncated split archive, unexpected top-level directory, or +checksum mismatch requires a fresh artifact from the planned URL; it is not a +reason to disable verification. diff --git a/skills/deepmd-install/references/plan-schema.md b/skills/deepmd-install/references/plan-schema.md new file mode 100644 index 0000000000..5ff29e0098 --- /dev/null +++ b/skills/deepmd-install/references/plan-schema.md @@ -0,0 +1,311 @@ +# Installation plan schema + +Store every decision in one JSON document. Keep the plan outside source, +build, and install directories. The plan contains no credentials or shell +commands. + +## Contents + +- [Top-level fields](#top-level-fields) +- [Conditional objects](#conditional-objects) +- [Validation rules](#validation-rules) +- [Examples](#examples) + +## Top-level fields + +```json +{ + "schema_version": 1, + "method": "pip", + "goal": "python", + "backend": "pytorch", + "accelerator": "cuda", + "environment": {}, + "package": {}, + "source": null, + "build": null, + "cpp": null, + "lammps": null, + "smoke_test": { + "enabled": false, + "gpu": null, + "example": null + } +} +``` + +Allowed values: + +- `method`: `pip`, `conda`, `dp1s`, `offline`, `docker`, or `source`. +- `goal`: `python`, `python+lammps`, `python+cpp`, or + `python+cpp+lammps`. +- `backend`: `pytorch`, `tensorflow`, `jax`, or `paddle`. +- `accelerator`: `cpu`, `cuda`, or `rocm`. + +Use `null` for an inapplicable object. Do not add undeclared keys. + +## Conditional objects + +### `environment` + +```json +{ + "kind": "existing", + "python": "/absolute/path/to/python", + "manager": null, + "name": null, + "prefix": "/absolute/environment/prefix" +} +``` + +- `kind`: `existing`, `venv`, `conda`, `prefix`, or `container`. +- `python`: required for `pip` and `source`; resolve it after creating a new + environment and before installing DeePMD-kit. +- `manager` and `name`: required for `conda`; `manager` is the absolute + `conda` or `mamba` executable. +- `prefix`: required for `venv`, `prefix`, and offline installations. + +### `package` + +```json +{ + "deepmd_version": null, + "deepmd_index_url": null, + "deepmd_extra_index_url": null, + "backend_packages": [ + "torch" + ], + "backend_index_url": null, + "channels": [ + "conda-forge" + ], + "install_lammps": false, + "install_ipi": false, + "artifact_url": null, + "sha256": null, + "docker_image": null +} +``` + +- Record exact backend requirements selected from the backend's official + installer or the version-matched DeePMD-kit documentation. +- Keep the two DeePMD-kit index fields null for the default package index. + Record a user-selected mirror or the documented pre-release index explicitly. +- Keep `backend_index_url` null when the default package index is intended. +- Require `artifact_url` and `sha256` for `offline`. +- Require an immutable image reference or user-selected tag for `docker`. + +### `source` + +```json +{ + "directory": "/absolute/path/to/deepmd-kit", + "remote": "https://github.com/deepmodeling/deepmd-kit.git", + "ref": "master", + "editable": false +} +``` + +Treat `ref` as an opaque Git ref. Resolve it to a commit and record that commit +in the completion report. Do not reset or clean an existing checkout. + +### `build` + +```json +{ + "variant": "cuda", + "cc": "/usr/bin/gcc", + "cxx": "/usr/bin/g++", + "cuda_home": "/usr/local/cuda-12.8", + "rocm_root": null, + "native_optimization": false, + "jobs": 8 +} +``` + +`build.variant` describes DeePMD-kit's compiled custom operations. It may +differ from `accelerator` for a backend whose accelerator runtime is supplied +entirely by its Python package. Require `cuda_home` for a CUDA build and +`rocm_root` for a ROCm build. + +### `cpp` + +```json +{ + "install_prefix": "/absolute/dedicated/deepmd-prefix", + "build_directory": "/absolute/build/deepmd-cpp", + "tensorflow_root": null, + "tensorflow_c_root": null, + "paddle_inference_dir": null +} +``` + +Use a dedicated install prefix. Never choose `/`, a home directory, a conda +prefix, `/usr`, `/usr/local`, or `$HOME/.local` as a disposable prefix. + +For a JAX C++ backend, provide either `tensorflow_root` or +`tensorflow_c_root`. For Paddle C++, provide `paddle_inference_dir`. + +### `lammps` + +```json +{ + "source_directory": "/absolute/path/to/lammps", + "build_directory": "/absolute/build/lammps-blackwell120", + "version": "stable_22Jul2025_update2", + "url": "https://github.com/lammps/lammps/archive/refs/tags/stable_22Jul2025_update2.tar.gz", + "sha256": null, + "flavor": "kokkos-cuda", + "machine": "blackwell120", + "kokkos_arch": "BLACKWELL120", + "mpi": false, + "model_family": "dpa4c" +} +``` + +- `flavor`: `host` or `kokkos-cuda`. +- `model_family`: `conventional`, `dpa4`, or `dpa4c`. +- `machine` and `kokkos_arch` are required only for `kokkos-cuda`. +- Use one Kokkos GPU architecture and one build directory per binary. + +### `smoke_test` + +```json +{ + "enabled": true, + "gpu": 7, + "example": "/absolute/path/to/example" +} +``` + +Require a physical GPU index for a CUDA smoke test. Keep `gpu` null for CPU. +The example path must belong to the selected source checkout or be explicitly +provided by the user. + +## Validation rules + +The validator enforces these invariants: + +1. `pip` and `source` use an absolute Python executable. +1. C/C++ goals use `source` and provide `build` plus `cpp`. +1. Source LAMMPS provides `lammps`; Kokkos CUDA requires the PyTorch backend + and a CUDA build. +1. DPA4C maps to `dpa4spin`/`dpa4spin/kk`; other families map to + `deepmd`/`deepmd/kk`. +1. Source directories, build directories, and install prefixes are distinct. +1. Checksums contain exactly 64 hexadecimal characters. +1. Unknown keys and unsupported combinations fail before any state change. + +## Examples + +### Pip PyTorch CUDA + +```json +{ + "schema_version": 1, + "method": "pip", + "goal": "python", + "backend": "pytorch", + "accelerator": "cuda", + "environment": { + "kind": "existing", + "python": "/opt/conda/envs/deepmd/bin/python", + "manager": null, + "name": null, + "prefix": "/opt/conda/envs/deepmd" + }, + "package": { + "deepmd_version": null, + "deepmd_index_url": null, + "deepmd_extra_index_url": null, + "backend_packages": [], + "backend_index_url": null, + "channels": [], + "install_lammps": false, + "install_ipi": false, + "artifact_url": null, + "sha256": null, + "docker_image": null + }, + "source": null, + "build": null, + "cpp": null, + "lammps": null, + "smoke_test": { + "enabled": false, + "gpu": null, + "example": null + } +} +``` + +### Source PyTorch CUDA with DPA4C LAMMPS + +```json +{ + "schema_version": 1, + "method": "source", + "goal": "python+cpp+lammps", + "backend": "pytorch", + "accelerator": "cuda", + "environment": { + "kind": "existing", + "python": "/opt/conda/envs/deepmd/bin/python", + "manager": null, + "name": null, + "prefix": "/opt/conda/envs/deepmd" + }, + "package": { + "deepmd_version": null, + "deepmd_index_url": null, + "deepmd_extra_index_url": null, + "backend_packages": [], + "backend_index_url": null, + "channels": [], + "install_lammps": false, + "install_ipi": false, + "artifact_url": null, + "sha256": null, + "docker_image": null + }, + "source": { + "directory": "/work/deepmd-kit", + "remote": "https://github.com/deepmodeling/deepmd-kit.git", + "ref": "master", + "editable": false + }, + "build": { + "variant": "cuda", + "cc": "/usr/bin/gcc", + "cxx": "/usr/bin/g++", + "cuda_home": "/usr/local/cuda-12.8", + "rocm_root": null, + "native_optimization": false, + "jobs": 8 + }, + "cpp": { + "install_prefix": "/work/install/deepmd-cuda", + "build_directory": "/work/build/deepmd-cpp-cuda", + "tensorflow_root": null, + "tensorflow_c_root": null, + "paddle_inference_dir": null + }, + "lammps": { + "source_directory": "/work/lammps-stable_22Jul2025_update2", + "build_directory": "/work/build/lammps-blackwell120", + "version": "stable_22Jul2025_update2", + "url": "https://github.com/lammps/lammps/archive/refs/tags/stable_22Jul2025_update2.tar.gz", + "sha256": null, + "flavor": "kokkos-cuda", + "machine": "blackwell120", + "kokkos_arch": "BLACKWELL120", + "mpi": false, + "model_family": "dpa4c" + }, + "smoke_test": { + "enabled": true, + "gpu": 7, + "example": "/work/deepmd-kit/examples/spin/dpa4c/lmp" + } +} +``` diff --git a/skills/deepmd-install/references/source-cpp.md b/skills/deepmd-install/references/source-cpp.md new file mode 100644 index 0000000000..17fdc703af --- /dev/null +++ b/skills/deepmd-install/references/source-cpp.md @@ -0,0 +1,158 @@ +# Source installation: C/C++ interface + +Build the C/C++ interface only after the Python/backend gate passes. Use +`doc/install/install-from-source.md` in the selected checkout for the exact +CMake options supported by that ref. The published documentation is +. + +## Contents + +- [Build-directory gate](#build-directory-gate) +- [Backend configuration](#backend-configuration) +- [Configure and install](#configure-and-install) +- [C/C++ verification](#cc-verification) + +## Build-directory gate + +Require CMake 3.25.2 or newer and the exact compilers from the plan: + +```bash +"" --version +"" --version +"" --version +``` + +The source, build, and install paths must be distinct. The install prefix must +be dedicated to this build. If the build directory contains a `CMakeCache.txt` +from another source commit, compiler, backend, or accelerator, select a new +build directory. Never delete or recursively replace the install prefix. + +## Backend configuration + +Start with all backends disabled, then enable the selected backend and its +documented C API dependency. Enabling TensorFlow also enables the JAX C API in +DeePMD-kit by design. + +| Backend | Required CMake arguments | +| ----------------------- | ----------------------------------------------------------------------------------- | +| PyTorch | `-DENABLE_PYTORCH=ON -DUSE_PT_PYTHON_LIBS=ON` plus the PyTorch CMake prefix | +| TensorFlow | `-DENABLE_TENSORFLOW=ON -DUSE_TF_PYTHON_LIBS=ON` and the selected Python executable | +| JAX with TensorFlow C++ | `-DENABLE_TENSORFLOW=ON -DUSE_TF_PYTHON_LIBS=ON` | +| JAX with TensorFlow C | `-DENABLE_JAX=ON -DCMAKE_PREFIX_PATH=` | +| Paddle | `-DENABLE_PADDLE=ON -DPADDLE_INFERENCE_DIR=` | + +For PyTorch, discover the prefix from the planned interpreter in the same shell +call that configures CMake: + +```bash +"" -c "import torch; print(torch.utils.cmake_prefix_path)" +``` + +When more than one backend is explicitly requested, confirm compatible +`_GLIBCXX_USE_CXX11_ABI` settings before compiling. Do not enable an incidental +backend merely because its Python package is importable. + +Accelerator arguments: + +| Build variant | Required CMake arguments | +| ------------- | ---------------------------------------------------------------------------------------------------------------------------------- | +| CPU | `-DUSE_CUDA_TOOLKIT=OFF -DUSE_ROCM_TOOLKIT=OFF` | +| CUDA | `-DUSE_CUDA_TOOLKIT=ON -DCUDAToolkit_ROOT= -DCMAKE_CUDA_COMPILER=/bin/nvcc -DCMAKE_CUDA_HOST_COMPILER=` | +| ROCm | `-DUSE_ROCM_TOOLKIT=ON -DCMAKE_HIP_COMPILER_ROCM_ROOT=` | + +## Configure and install + +Render all plan values into one call. This PyTorch CUDA example shows the +complete shape: + +```bash +torch_cmake_prefix=$("" -c \ + "import torch; print(torch.utils.cmake_prefix_path)") && +"" \ + -S "/source" \ + -B "" \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_PREFIX="" \ + -DCMAKE_C_COMPILER="" \ + -DCMAKE_CXX_COMPILER="" \ + -DPython_EXECUTABLE="" \ + -DPython3_EXECUTABLE="" \ + -DBUILD_CPP_IF=ON \ + -DBUILD_PY_IF=OFF \ + -DENABLE_NATIVE_OPTIMIZATION=OFF \ + -DENABLE_PYTORCH=ON \ + -DUSE_PT_PYTHON_LIBS=ON \ + -DENABLE_TENSORFLOW=OFF \ + -DENABLE_JAX=OFF \ + -DENABLE_PADDLE=OFF \ + -DUSE_CUDA_TOOLKIT=ON \ + -DUSE_ROCM_TOOLKIT=OFF \ + -DCMAKE_PREFIX_PATH="$torch_cmake_prefix" \ + -DCUDAToolkit_ROOT="" \ + -DCMAKE_CUDA_COMPILER="/bin/nvcc" \ + -DCMAKE_CUDA_HOST_COMPILER="" +``` + +Replace only the backend and accelerator arguments using the tables. Set +`ENABLE_NATIVE_OPTIMIZATION=ON` only when the installed libraries remain on the +same CPU model. + +Build and install with the bounded job count from the plan: + +```bash +"" --build "" \ + --parallel "" +"" --install "" +``` + +If the prefix already contains an installation, choose a new versioned prefix +and switch consumers only after its verification passes. + +## C/C++ verification + +Check the cache against the plan. For a CUDA PyTorch build, for example: + +```bash +grep -E \ + 'BUILD_CPP_IF:|BUILD_PY_IF:|ENABLE_PYTORCH:|USE_CUDA_TOOLKIT:|CMAKE_CUDA_COMPILER:' \ + "/CMakeCache.txt" +``` + +Require the headers and libraries: + +```bash +test -f "/include/deepmd/deepmd.hpp" +test -f "/include/deepmd/c_api.h" +test -f "/lib/libdeepmd_cc.so" +test -f "/lib/libdeepmd_c.so" +``` + +Use the platform library suffix on macOS or Windows. On Linux, fail if any +installed DeePMD library has an unresolved dynamic dependency: + +```bash +for library in ""/lib/libdeepmd*.so; do + ldd "$library" +done +``` + +Finally compile and run a public C++ API probe inside the build directory: + +```bash +probe_source="/verify_deepmd.cpp" +probe_binary="/verify_deepmd" +printf '%s\n' \ + '#include "deepmd/common.h"' \ + 'int main() { deepmd::print_summary(""); return 0; }' \ + > "$probe_source" +"" -std=c++14 "$probe_source" \ + -I"/include" \ + -L"/lib" \ + -Wl,-rpath,"/lib" \ + -ldeepmd_cc \ + -o "$probe_binary" +"$probe_binary" +``` + +Do not proceed to LAMMPS when cache values, dynamic dependencies, or the public +API probe fail. diff --git a/skills/deepmd-install/references/source-lammps.md b/skills/deepmd-install/references/source-lammps.md new file mode 100644 index 0000000000..98e89274a4 --- /dev/null +++ b/skills/deepmd-install/references/source-lammps.md @@ -0,0 +1,237 @@ +# Source installation: LAMMPS + +Build LAMMPS only after the C/C++ prefix passes its public API and dynamic-link +checks. Use `doc/install/install-lammps.md` and the example README files inside +the selected DeePMD-kit checkout for version-specific commands. The published +documentation is +. + +## Contents + +- [Acquire LAMMPS](#acquire-lammps) +- [Add the built-in module](#add-the-built-in-module) +- [Select the runtime](#select-the-runtime) +- [Host build](#host-build) +- [Kokkos CUDA build](#kokkos-cuda-build) +- [Verification](#verification) +- [Smoke tests](#smoke-tests) + +## Acquire LAMMPS + +Use an existing source directory only when its version matches the plan and its +working tree is suitable for a build. Otherwise download the exact HTTPS URL +from the plan into a separate directory: + +```bash +curl -fL "" -o "" +``` + +When `lammps.sha256` is non-null, verify it before extraction: + +```bash +printf '%s %s\n' "" "" | sha256sum --check - +``` + +Extract into a directory whose resolved path equals +`lammps.source_directory`. Stop if the archive creates an unexpected directory +or contains paths outside its top-level directory. + +## Add the built-in module + +Use the bundled preparer instead of appending an unquoted CMake line: + +```bash +"" "/scripts/prepare_lammps.py" \ + --lammps-source "" \ + --deepmd-source "" +``` + +The script maintains exactly one quoted include block and replaces a single +legacy include. It fails on ambiguous duplicate includes. + +## Select the runtime + +| Selection | Required pair styles | +| ------------------------ | ------------------------- | +| conventional host | `deepmd` | +| conventional Kokkos CUDA | `deepmd`, `deepmd/kk` | +| DPA4/SeZM host | `deepmd` | +| DPA4/SeZM Kokkos CUDA | `deepmd`, `deepmd/kk` | +| DPA4C host | `dpa4spin` | +| DPA4C Kokkos CUDA | `dpa4spin`, `dpa4spin/kk` | + +`deepmd/kk` accepts a compatible edge-input or graph-input `.pt2` artifact. +It requires `atom_modify map yes` and a single model; model-deviation ensembles +use the host pair style. `dpa4spin/kk` accepts a DPA4C compact canonical graph +artifact and requires `atom_style spin`. Neither style is provided by LAMMPS +`PKG_GPU`. + +For Kokkos, select the architecture from numeric compute capability, then +confirm that the chosen LAMMPS tree defines the option: + +| Compute capability | Kokkos architecture | +| ------------------ | ------------------- | +| 7.0 | `VOLTA70` | +| 7.5 | `TURING75` | +| 8.0 | `AMPERE80` | +| 8.6 | `AMPERE86` | +| 8.9 | `ADA89` | +| 9.0 | `HOPPER90` | +| 10.0 | `BLACKWELL100` | +| 12.0 | `BLACKWELL120` | +| 12.1 | `BLACKWELL121` | + +```bash +grep -F "kokkos_arch_option(" \ + "/lib/kokkos/cmake/kokkos_arch.cmake" +``` + +If the numeric capability or the Kokkos option is unavailable, stop and select +a compatible toolkit/LAMMPS version. Use one architecture and a distinct build +directory/binary for each target GPU family. + +## Host build + +Configure the minimal host binary. Enable optional LAMMPS packages only when +the user's simulation requires them: + +```bash +"" \ + -S "/cmake" \ + -B "" \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_C_COMPILER="" \ + -DCMAKE_CXX_COMPILER="" \ + -DPython_EXECUTABLE="" \ + -DPython3_EXECUTABLE="" \ + -DCMAKE_INSTALL_PREFIX="" \ + -DBUILD_MPI="" \ + -DBUILD_SHARED_LIBS=ON \ + -DLAMMPS_INSTALL_RPATH=ON \ + -DCMAKE_PREFIX_PATH="" +``` + +Build and install with the planned job limit: + +```bash +"" --build "" --parallel "" +"" --install "" +``` + +The expected binary is `/bin/lmp` unless the plan explicitly +sets a LAMMPS machine suffix. + +## Kokkos CUDA build + +Configure one architecture. The same shell call discovers PyTorch and +namespace-package NCCL, constructs RPATH, and invokes CMake so no derived value +can disappear: + +```bash +runtime_rpath=$("" - <<'PY' +from importlib.util import find_spec +from pathlib import Path + +import torch + +paths = [Path(torch.__file__).resolve().parent / "lib"] +spec = find_spec("nvidia.nccl") +if spec is not None: + for root in spec.submodule_search_locations or (): + candidate = Path(root) / "lib" + if list(candidate.glob("libnccl.so*")): + paths.append(candidate) +print(";".join(str(path) for path in paths if path.is_dir())) +PY +) && +test -n "$runtime_rpath" && +cuda_runtime_dir=$("" - "" <<'PY' +import sys +from pathlib import Path + +root = Path(sys.argv[1]) +candidates = [root / "lib64", *root.glob("targets/*-linux/lib")] +print(next((path.resolve() for path in candidates if path.is_dir()), "")) +PY +) && +test -n "$cuda_runtime_dir" && +torch_cmake_prefix=$("" -c \ + "import torch; print(torch.utils.cmake_prefix_path)") && +runtime_rpath="/lib;$cuda_runtime_dir;$runtime_rpath" && +"" \ + -S "/cmake" \ + -B "" \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_C_COMPILER="" \ + -DCMAKE_CXX_COMPILER="" \ + -DCMAKE_CUDA_COMPILER="/bin/nvcc" \ + -DCMAKE_CUDA_HOST_COMPILER="" \ + -DPython_EXECUTABLE="" \ + -DPython3_EXECUTABLE="" \ + -DCMAKE_INSTALL_PREFIX="" \ + -DLAMMPS_MACHINE="" \ + -DBUILD_MPI="" \ + -DBUILD_SHARED_LIBS=OFF \ + -DLAMMPS_INSTALL_RPATH=ON \ + -DPKG_KOKKOS=ON \ + -DKokkos_ENABLE_CUDA=ON \ + -DKokkos_ENABLE_SERIAL=ON \ + -DKokkos_ARCH_=ON \ + -DCMAKE_PREFIX_PATH=";$torch_cmake_prefix" \ + -DCUDAToolkit_ROOT="" \ + -DCMAKE_BUILD_RPATH="$runtime_rpath" \ + -DCMAKE_INSTALL_RPATH="$runtime_rpath" +``` + +The rendered command must contain one real `Kokkos_ARCH_*` argument and no +`` placeholder. Add linker flags only in response to a concrete +link error and only after locating the named library. + +Build and install: + +```bash +"" --build "" --parallel "" +"" --install "" +``` + +The expected binary is `/bin/lmp_`. + +## Verification + +Run the exact model-family verifier. Use `--check-links` for a source binary: + +```bash +"" "/scripts/verify_lammps.py" \ + --binary "" \ + --model-family "" \ + --flavor "" \ + --check-links +``` + +Do not replace a failed exact style check with a broad `grep deepmd`. + +## Smoke tests + +Check the selected physical GPU immediately before a CUDA test: + +```bash +nvidia-smi --query-gpu=index,name,memory.used,memory.total,utilization.gpu \ + --format=csv +``` + +Bind it explicitly so the process sees that physical card as logical device 0: + +```bash +CUDA_VISIBLE_DEVICES="" \ + "" -k on g 1 -sf kk -in "" +``` + +For DPA4/SeZM, follow +`examples/water/dpa4/lmp/README.md` from the same DeePMD-kit checkout. For +DPA4C, follow `examples/spin/dpa4c/lmp/README.md`; use its `--pt-expt` +compression/freeze sequence and spin input rather than a generic freeze +command. For conventional models, use a short user-selected example compatible +with the model format. + +Success requires a completed run with `Loop time` in the LAMMPS log and no +artifact-schema, pair-style, device, or unresolved-library error. diff --git a/skills/deepmd-install/references/source-python.md b/skills/deepmd-install/references/source-python.md new file mode 100644 index 0000000000..bc9b64a57d --- /dev/null +++ b/skills/deepmd-install/references/source-python.md @@ -0,0 +1,185 @@ +# Source installation: Python interface + +Build the Python package from the exact Git commit recorded by the workflow. +Use `doc/install/install-from-source.md` inside that checkout as the authority +for version-specific backend packages and build variables. The published +counterpart is +. + +## Contents + +- [Checkout gate](#checkout-gate) +- [Interpreter and compiler gate](#interpreter-and-compiler-gate) +- [Backend dependencies](#backend-dependencies) +- [Build](#build) +- [Verification](#verification) + +## Checkout gate + +For an absent target directory, clone without checking out a moving branch, +fetch the requested opaque ref, resolve it to a commit, and check out that +commit detached. Render all values literally in one shell call: + +```bash +git clone --no-checkout "" "" && +git -C "" fetch --tags "" "" && +git -C "" checkout --detach \ + "$(git -C "" rev-parse 'FETCH_HEAD^{commit}')" && +git -C "" rev-parse HEAD +``` + +For an existing checkout, inspect it without changing its branch, remotes, or +working tree: + +```bash +git -C "" status --short +git -C "" remote -v +git -C "" rev-parse HEAD +``` + +If the existing `HEAD` is not the requested ref, or tracked/untracked changes +overlap the build, use a separate clone or ask the user which tree to use. Do +not reset, clean, or repoint `origin`. + +## Interpreter and compiler gate + +Use the plan's absolute interpreter and confirm Python 3.10 or newer: + +```bash +"" -c "import sys; print(sys.executable); print(sys.version)" +"" --version +"" --version +``` + +For a CUDA build, verify the selected toolkit and host compiler in the same +call. A CUDA major-version mismatch with the selected backend package is a hard +failure; handle a same-major minor mismatch through the actual configure/build +result rather than string equality alone. + +```bash +"/bin/nvcc" --version +"" --version +``` + +Install build requirements into the same environment: + +```bash +"" -m pip install --upgrade \ + pip scikit-build-core packaging cmake ninja dependency_groups +``` + +## Backend dependencies + +Install every entry in `package.backend_packages` before building DeePMD-kit. +Use the recorded backend index only when it is non-null. + +### PyTorch + +Select the PyTorch requirement and index from the official PyTorch installer or +the checkout documentation. Verify the wheel before compiling DeePMD-kit: + +```bash +"" -c \ + "import torch; print(torch.__version__, torch.version.cuda, torch.version.hip, torch.cuda.is_available())" +``` + +Use the `torch` extra for the source package. It supplies `vesin[torch]` and +the platform-eligible nvalchemi integration. + +### TensorFlow + +Install the TensorFlow package selected by the checkout documentation, then +verify it with the same interpreter: + +```bash +"" -c \ + "import tensorflow as tf; print(tf.__version__, tf.config.list_physical_devices('GPU'))" +``` + +A CUDA DeePMD-kit build requires a local CUDA toolkit visible to CMake. Keep +`DP_ENABLE_TENSORFLOW=1` and `DP_ENABLE_PYTORCH=0` explicit during the build. + +### JAX + +Install the JAX accelerator package selected by the official JAX instructions +and use the DeePMD-kit `jax` extra. JAX may provide the GPU runtime while the +DeePMD-kit compiled variant remains CPU; record `accelerator` and +`build.variant` separately in the plan. + +```bash +"" -c \ + "import jax; print(jax.__version__, jax.devices())" +``` + +### Paddle + +Install the Paddle package and index selected by +`doc/install/easy-install.md` in the checkout. Build the Python package with +TensorFlow and PyTorch disabled unless either backend is also requested. + +```bash +"" -c \ + "import paddle; print(paddle.__version__, paddle.device.get_device())" +``` + +## Build + +Select the source requirement by backend: + +| Backend | Local requirement | Build switches | +| ---------- | ----------------- | ----------------------------------------------- | +| PyTorch | `.[torch]` | `DP_ENABLE_PYTORCH=1`, `DP_ENABLE_TENSORFLOW=0` | +| TensorFlow | `.` | `DP_ENABLE_PYTORCH=0`, `DP_ENABLE_TENSORFLOW=1` | +| JAX | `.[jax]` | `DP_ENABLE_PYTORCH=0`, `DP_ENABLE_TENSORFLOW=0` | +| Paddle | `.` | `DP_ENABLE_PYTORCH=0`, `DP_ENABLE_TENSORFLOW=0` | + +Render a single self-contained invocation. This CUDA PyTorch template shows the +required shape: + +```bash +cd "" && +env \ + CC="" \ + CXX="" \ + CUDA_HOME="" \ + CUDA_PATH="" \ + CUDAToolkit_ROOT="" \ + CUDACXX="/bin/nvcc" \ + CUDAHOSTCXX="" \ + DP_VARIANT=cuda \ + DP_ENABLE_PYTORCH=1 \ + DP_ENABLE_TENSORFLOW=0 \ + DP_ENABLE_NATIVE_OPTIMIZATION=0 \ + CMAKE_BUILD_PARALLEL_LEVEL="" \ + "" -m pip install \ + ".[torch]" --no-build-isolation --verbose +``` + +For CPU, omit CUDA variables and set `DP_VARIANT=cpu`. For ROCm, follow the +selected checkout documentation and set `DP_VARIANT=rocm` plus the planned +`ROCM_ROOT` in the same invocation. + +When `source.editable=true`, add `--editable` to the one pip command. Otherwise +perform a regular installation. Do not make editable mode the default for a +runtime installation. + +For TensorFlow, JAX, and Paddle, replace the requirement and the two +`DP_ENABLE_*` values using the table. Do not add a backend the plan does not +request. + +## Verification + +Run outside the source directory so an import cannot succeed from the checkout +alone: + +```bash +cd "" && +"" "/scripts/verify_python.py" \ + --backend "" \ + --accelerator "" \ + --expected-build-variant "" +``` + +Add `--expect-custom-op` for a PyTorch source build. Add `--expect-nv` and +`--expect-vesin` only when their platform markers and the plan require them. +Do not continue to the C/C++ gate until every requested check passes. diff --git a/skills/deepmd-install/scripts/prepare_lammps.py b/skills/deepmd-install/scripts/prepare_lammps.py new file mode 100644 index 0000000000..7a12aea828 --- /dev/null +++ b/skills/deepmd-install/scripts/prepare_lammps.py @@ -0,0 +1,169 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: LGPL-3.0-or-later +"""Add one managed DeePMD built-in include to a LAMMPS CMake project.""" + +from __future__ import ( + annotations, +) + +import argparse +import os +import re +import tempfile +from pathlib import ( + Path, +) + +BEGIN_MARKER = "# >>> DeePMD-kit built-in module >>>" +END_MARKER = "# <<< DeePMD-kit built-in module <<<" +LEGACY_PATTERN = re.compile( + r"^\s*include\([^\n]*source/lmp/builtin\.cmake[^\n]*\)\s*$", re.MULTILINE +) + + +def _managed_block(builtin: Path) -> str: + """Return the canonical managed CMake block.""" + return "\n".join( + ( + BEGIN_MARKER, + f"include([=[{builtin}]=])", + END_MARKER, + ) + ) + + +def _replace_managed_block(text: str, block: str) -> tuple[str, bool]: + """Replace a valid managed block or reject malformed markers.""" + begin_count = text.count(BEGIN_MARKER) + end_count = text.count(END_MARKER) + if begin_count == 0 and end_count == 0: + return text, False + if begin_count != 1 or end_count != 1: + raise ValueError("LAMMPS CMakeLists.txt contains malformed DeePMD markers") + begin = text.index(BEGIN_MARKER) + end = text.index(END_MARKER, begin) + len(END_MARKER) + return text[:begin] + block + text[end:], True + + +def render_updated_text(text: str, builtin: Path) -> str: + """Return an idempotently patched CMakeLists.txt. + + Parameters + ---------- + text : str + Existing LAMMPS top-level CMake text. + builtin : Path + Absolute DeePMD-kit `source/lmp/builtin.cmake` path. + + Returns + ------- + str + Text containing exactly one managed include. + + Raises + ------ + ValueError + If multiple unmanaged includes or malformed markers are present. + """ + block = _managed_block(builtin) + updated, replaced = _replace_managed_block(text, block) + if replaced: + outside_block = updated.replace(block, "", 1) + if LEGACY_PATTERN.search(outside_block) is not None: + raise ValueError( + "LAMMPS CMakeLists.txt contains an unmanaged DeePMD include" + ) + return updated + legacy_matches = list(LEGACY_PATTERN.finditer(text)) + if len(legacy_matches) > 1: + raise ValueError("LAMMPS CMakeLists.txt contains multiple DeePMD includes") + if legacy_matches: + match = legacy_matches[0] + return updated[: match.start()] + block + updated[match.end() :] + separator = "" if updated.endswith("\n") else "\n" + return f"{updated}{separator}\n{block}\n" + + +def _write_atomic(path: Path, text: str) -> None: + """Atomically replace a text file while preserving its mode.""" + mode = path.stat().st_mode + descriptor, temporary_name = tempfile.mkstemp( + prefix=f".{path.name}.", suffix=".tmp", dir=path.parent + ) + temporary = Path(temporary_name) + try: + with os.fdopen(descriptor, "w", encoding="utf-8") as stream: + stream.write(text) + os.chmod(temporary, mode & 0o7777) + os.replace(temporary, path) + finally: + if temporary.exists(): + temporary.unlink() + + +def prepare(lammps_source: Path, deepmd_source: Path, *, check: bool) -> bool: + """Prepare or check the managed LAMMPS include. + + Parameters + ---------- + lammps_source : Path + Absolute LAMMPS source directory. + deepmd_source : Path + Absolute DeePMD-kit source directory. + check : bool + If true, compare without writing. + + Returns + ------- + bool + True when the file already had the canonical content. + """ + lammps_source = lammps_source.resolve(strict=True) + deepmd_source = deepmd_source.resolve(strict=True) + cmake_file = lammps_source / "cmake" / "CMakeLists.txt" + builtin = deepmd_source / "source" / "lmp" / "builtin.cmake" + if not cmake_file.is_file(): + raise FileNotFoundError(f"LAMMPS CMake file not found: {cmake_file}") + if not builtin.is_file(): + raise FileNotFoundError(f"DeePMD built-in module not found: {builtin}") + original = cmake_file.read_text(encoding="utf-8") + updated = render_updated_text(original, builtin) + unchanged = updated == original + if not check and not unchanged: + _write_atomic(cmake_file, updated) + return unchanged + + +def main(argv: list[str] | None = None) -> int: + """Parse arguments and prepare the LAMMPS source tree. + + Parameters + ---------- + argv : list of str, optional + Command-line arguments. Defaults to the process arguments. + + Returns + ------- + int + Zero when prepared or already correct, otherwise one. + """ + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--lammps-source", required=True, type=Path) + parser.add_argument("--deepmd-source", required=True, type=Path) + parser.add_argument("--check", action="store_true") + args = parser.parse_args(argv) + try: + unchanged = prepare(args.lammps_source, args.deepmd_source, check=args.check) + except (OSError, ValueError) as exc: + print(f"FAIL prepare_lammps: {exc}") + return 1 + if args.check and not unchanged: + print("FAIL prepare_lammps: managed include differs from the plan") + return 1 + status = "UNCHANGED" if unchanged else "UPDATED" + print(f"PASS prepare_lammps: {status}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/skills/deepmd-install/scripts/probe_env.py b/skills/deepmd-install/scripts/probe_env.py new file mode 100644 index 0000000000..c9f206c686 --- /dev/null +++ b/skills/deepmd-install/scripts/probe_env.py @@ -0,0 +1,426 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: LGPL-3.0-or-later +"""Probe machine facts required to plan a DeePMD-kit installation.""" + +from __future__ import ( + annotations, +) + +import argparse +import csv +import importlib.metadata +import json +import os +import platform +import shutil +import subprocess +import sys +from pathlib import ( + Path, +) +from typing import ( + TYPE_CHECKING, + Any, +) + +if TYPE_CHECKING: + from collections.abc import ( + Sequence, + ) + +ENVIRONMENT_VARIABLES = ( + "CUDA_HOME", + "CUDA_PATH", + "CUDAToolkit_ROOT", + "ROCM_ROOT", + "ROCM_PATH", + "CUDA_VISIBLE_DEVICES", + "HIP_VISIBLE_DEVICES", + "ROCR_VISIBLE_DEVICES", + "CONDA_PREFIX", + "CONDA_DEFAULT_ENV", + "VIRTUAL_ENV", + "CC", + "CXX", + "CUDACXX", + "CUDAHOSTCXX", + "DP_VARIANT", + "DP_ENABLE_PYTORCH", + "DP_ENABLE_TENSORFLOW", + "PATH", +) +TOOLS = ( + "python", + "python3", + "pip", + "conda", + "mamba", + "git", + "cmake", + "ninja", + "gcc", + "g++", + "clang", + "clang++", + "nvcc", + "hipcc", + "mpicxx", + "nvidia-smi", + "rocm-smi", + "rocminfo", + "docker", + "dp", + "lmp", +) +VERSION_TOOLS = { + "pip": ("--version",), + "conda": ("--version",), + "mamba": ("--version",), + "git": ("--version",), + "cmake": ("--version",), + "ninja": ("--version",), + "gcc": ("--version",), + "g++": ("--version",), + "clang": ("--version",), + "clang++": ("--version",), + "nvcc": ("--version",), + "hipcc": ("--version",), + "mpicxx": ("--version",), + "docker": ("--version",), +} +PACKAGE_DISTRIBUTIONS = { + "deepmd": ("deepmd-kit",), + "pytorch": ("torch",), + "tensorflow": ("tensorflow", "tensorflow-cpu"), + "jax": ("jax",), + "paddle": ("paddlepaddle", "paddlepaddle-gpu"), +} + + +def _run( + command: Sequence[str], *, timeout: float = 10.0, cwd: Path | None = None +) -> dict[str, Any]: + """Run a read-only command and return a structured result.""" + executable = command[0] + resolved = shutil.which(executable) + if resolved is None and not Path(executable).is_file(): + return { + "status": "unavailable", + "returncode": None, + "stdout": "", + "stderr": "", + "output": "", + } + try: + completed = subprocess.run( + list(command), + check=False, + capture_output=True, + text=True, + timeout=timeout, + cwd=cwd, + ) + except subprocess.TimeoutExpired: + return { + "status": "timeout", + "returncode": None, + "stdout": "", + "stderr": "", + "output": "", + } + except OSError as exc: + return { + "status": "error", + "returncode": None, + "stdout": "", + "stderr": f"{type(exc).__name__}: {exc}", + "output": f"{type(exc).__name__}: {exc}", + } + stdout = completed.stdout.strip() + stderr = completed.stderr.strip() + output = "\n".join(part for part in (stdout, stderr) if part) + return { + "status": "ok" if completed.returncode == 0 else "failed", + "returncode": completed.returncode, + "stdout": stdout, + "stderr": stderr, + "output": output, + } + + +def _first_line(value: str) -> str: + """Return the first non-empty output line.""" + return next((line.strip() for line in value.splitlines() if line.strip()), "") + + +def _probe_system() -> dict[str, Any]: + """Return operating-system, memory, CPU, and disk facts.""" + libc_name, libc_version = platform.libc_ver() + memory_total: int | None = None + meminfo = Path("/proc/meminfo") + if meminfo.is_file(): + try: + for line in meminfo.read_text(encoding="utf-8").splitlines(): + if line.startswith("MemTotal:"): + memory_total = int(line.split()[1]) * 1024 + break + except (OSError, ValueError, IndexError): + memory_total = None + elif platform.system() == "Darwin": + result = _run(["sysctl", "-n", "hw.memsize"]) + if result["status"] == "ok": + try: + memory_total = int(result["output"]) + except ValueError: + memory_total = None + disk = shutil.disk_usage(Path.cwd()) + return { + "platform": platform.system(), + "release": platform.release(), + "machine": platform.machine(), + "libc": {"name": libc_name or None, "version": libc_version or None}, + "cpu_count": os.cpu_count(), + "memory_total_bytes": memory_total, + "working_directory": str(Path.cwd()), + "disk_free_bytes": disk.free, + } + + +def _probe_python() -> dict[str, Any]: + """Return facts for the interpreter executing this probe.""" + pip_result = _run([sys.executable, "-m", "pip", "--version"]) + return { + "executable": sys.executable, + "version": platform.python_version(), + "prefix": sys.prefix, + "base_prefix": sys.base_prefix, + "pip": pip_result, + } + + +def _probe_tools() -> dict[str, Any]: + """Return executable paths and concise version strings.""" + result: dict[str, Any] = {} + for name in TOOLS: + path = shutil.which(name) + entry: dict[str, Any] = {"path": path} + if path is not None and name in VERSION_TOOLS: + version = _run([path, *VERSION_TOOLS[name]]) + entry["version"] = ( + _first_line(version["output"]) + if version["status"] in {"ok", "failed"} + else None + ) + entry["version_status"] = version["status"] + result[name] = entry + return result + + +def _query_nvidia(fields: Sequence[str]) -> dict[str, Any]: + """Query NVIDIA GPUs with a fixed CSV field list.""" + result = _run( + [ + "nvidia-smi", + f"--query-gpu={','.join(fields)}", + "--format=csv,noheader,nounits", + ] + ) + if result["status"] != "ok": + return {"status": result["status"], "error": result["output"], "gpus": []} + rows = list(csv.reader(result["output"].splitlines(), skipinitialspace=True)) + gpus = [dict(zip(fields, row, strict=False)) for row in rows if row] + return {"status": "ok", "error": None, "gpus": gpus} + + +def _probe_nvidia() -> dict[str, Any]: + """Return driver, occupancy, and numeric compute capability per GPU.""" + fields = ( + "index", + "uuid", + "name", + "driver_version", + "memory.used", + "memory.total", + "utilization.gpu", + "compute_cap", + ) + result = _query_nvidia(fields) + if result["status"] == "ok": + return result + fallback_fields = fields[:-1] + fallback = _query_nvidia(fallback_fields) + if fallback["status"] == "ok": + for gpu in fallback["gpus"]: + gpu["compute_cap"] = None + fallback["compute_capability_status"] = "unsupported by nvidia-smi" + return fallback + + +def _probe_rocm() -> dict[str, Any]: + """Return ROCm tool availability and concise device output.""" + hipcc = shutil.which("hipcc") + rocminfo = shutil.which("rocminfo") + rocm_smi = shutil.which("rocm-smi") + result: dict[str, Any] = { + "hipcc": hipcc, + "rocminfo": rocminfo, + "rocm_smi": rocm_smi, + } + if hipcc is not None: + output = _run([hipcc, "--version"]) + result["hipcc_version"] = _first_line(output["output"]) + if rocm_smi is not None: + output = _run([rocm_smi, "--showproductname", "--showmeminfo", "vram"]) + result["device_summary"] = output + return result + + +def _distribution_version(names: Sequence[str]) -> dict[str, str] | None: + """Return the first installed distribution name and version.""" + for name in names: + try: + version = importlib.metadata.version(name) + except importlib.metadata.PackageNotFoundError: + continue + return {"distribution": name, "version": version} + return None + + +def _isolated_python_json(code: str) -> dict[str, Any]: + """Run an isolated import probe and decode its JSON output.""" + result = _run( + [sys.executable, "-I", "-c", code], timeout=30.0, cwd=Path(sys.prefix) + ) + if result["status"] != "ok": + return {"status": result["status"], "error": result["output"]} + try: + decoded = json.loads(result["stdout"]) + except json.JSONDecodeError: + return {"status": "invalid-json", "error": result["output"]} + if not isinstance(decoded, dict): + return {"status": "invalid-json", "error": result["output"]} + return {"status": "ok", **decoded} + + +def _probe_torch_runtime() -> dict[str, Any]: + """Return PyTorch accelerator visibility in an isolated interpreter.""" + code = r""" +import json +import torch + +devices = [] +if torch.cuda.is_available(): + for index in range(torch.cuda.device_count()): + devices.append( + { + "index": index, + "name": torch.cuda.get_device_name(index), + "capability": list(torch.cuda.get_device_capability(index)), + } + ) +print( + json.dumps( + { + "version": torch.__version__, + "file": torch.__file__, + "cuda_version": torch.version.cuda, + "hip_version": torch.version.hip, + "accelerator_available": torch.cuda.is_available(), + "devices": devices, + } + ) +) +""" + return _isolated_python_json(code) + + +def _probe_deepmd_runtime() -> dict[str, Any]: + """Return installed DeePMD-kit location and compiled build variant.""" + code = r""" +import json +import deepmd +from deepmd.env import GLOBAL_CONFIG + +result = { + "version": getattr(deepmd, "__version__", "unknown"), + "file": getattr(deepmd, "__file__", None), + "build_variant": GLOBAL_CONFIG.get("dp_variant"), + "enable_pytorch": GLOBAL_CONFIG.get("enable_pytorch"), + "enable_tensorflow": GLOBAL_CONFIG.get("enable_tensorflow"), +} +if result["enable_pytorch"]: + from deepmd.pt.cxx_op import ENABLE_CUSTOMIZED_OP + result["pytorch_custom_op"] = bool(ENABLE_CUSTOMIZED_OP) +print(json.dumps(result)) +""" + return _isolated_python_json(code) + + +def _probe_packages() -> dict[str, Any]: + """Return installed backend distributions and selected runtime details.""" + packages: dict[str, Any] = { + key: _distribution_version(names) + for key, names in PACKAGE_DISTRIBUTIONS.items() + } + if packages["pytorch"] is not None: + packages["pytorch"]["runtime"] = _probe_torch_runtime() + if packages["deepmd"] is not None: + packages["deepmd"]["runtime"] = _probe_deepmd_runtime() + return packages + + +def build_report() -> dict[str, Any]: + """Build the complete read-only probe report. + + Returns + ------- + dict + Machine facts used by the installation planner. + """ + return { + "system": _probe_system(), + "environment": {name: os.environ.get(name) for name in ENVIRONMENT_VARIABLES}, + "python": _probe_python(), + "tools": _probe_tools(), + "nvidia": _probe_nvidia(), + "rocm": _probe_rocm(), + "packages": _probe_packages(), + } + + +def _print_human(report: dict[str, Any]) -> None: + """Print a readable report without discarding structured values.""" + for section, value in report.items(): + print(f"== {section} ==") + print(json.dumps(value, indent=2, sort_keys=True)) + print() + + +def main(argv: list[str] | None = None) -> int: + """Probe the machine and print JSON or human-readable output. + + Parameters + ---------- + argv : list of str, optional + Command-line arguments. Defaults to the process arguments. + + Returns + ------- + int + Always zero; unavailable optional tools are represented in the report. + """ + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--json", action="store_true", help="Emit one machine-readable JSON object." + ) + args = parser.parse_args(argv) + report = build_report() + if args.json: + print(json.dumps(report, indent=2, sort_keys=True)) + else: + _print_human(report) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/skills/deepmd-install/scripts/validate_plan.py b/skills/deepmd-install/scripts/validate_plan.py new file mode 100644 index 0000000000..8cdb0e8253 --- /dev/null +++ b/skills/deepmd-install/scripts/validate_plan.py @@ -0,0 +1,652 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: LGPL-3.0-or-later +"""Validate a DeePMD-kit installation plan before changing the machine.""" + +from __future__ import ( + annotations, +) + +import argparse +import json +import re +from pathlib import ( + Path, +) +from typing import ( + Any, +) +from urllib.parse import ( + urlparse, +) + +SCHEMA_VERSION = 1 +METHODS = {"pip", "conda", "dp1s", "offline", "docker", "source"} +GOALS = {"python", "python+lammps", "python+cpp", "python+cpp+lammps"} +BACKENDS = {"pytorch", "tensorflow", "jax", "paddle"} +ACCELERATORS = {"cpu", "cuda", "rocm"} +TOP_LEVEL_KEYS = { + "schema_version", + "method", + "goal", + "backend", + "accelerator", + "environment", + "package", + "source", + "build", + "cpp", + "lammps", + "smoke_test", +} +OBJECT_KEYS = { + "environment": {"kind", "python", "manager", "name", "prefix"}, + "package": { + "deepmd_version", + "deepmd_index_url", + "deepmd_extra_index_url", + "backend_packages", + "backend_index_url", + "channels", + "install_lammps", + "install_ipi", + "artifact_url", + "sha256", + "docker_image", + }, + "source": {"directory", "remote", "ref", "editable"}, + "build": { + "variant", + "cc", + "cxx", + "cuda_home", + "rocm_root", + "native_optimization", + "jobs", + }, + "cpp": { + "install_prefix", + "build_directory", + "tensorflow_root", + "tensorflow_c_root", + "paddle_inference_dir", + }, + "lammps": { + "source_directory", + "build_directory", + "version", + "url", + "sha256", + "flavor", + "machine", + "kokkos_arch", + "mpi", + "model_family", + }, + "smoke_test": {"enabled", "gpu", "example"}, +} + + +def _is_absolute_path(value: object) -> bool: + """Return whether ``value`` is a non-empty absolute path string.""" + return isinstance(value, str) and bool(value) and Path(value).is_absolute() + + +def _canonical(value: str) -> Path: + """Return a normalized path without requiring it to exist.""" + return Path(value).resolve(strict=False) + + +def _require_keys( + data: dict[str, Any], required: set[str], context: str, errors: list[str] +) -> None: + """Append errors for missing keys in a mapping.""" + for key in sorted(required - data.keys()): + errors.append(f"{context}.{key}: missing required field") + + +def _check_unknown_keys( + data: dict[str, Any], allowed: set[str], context: str, errors: list[str] +) -> None: + """Append errors for undeclared keys in a mapping.""" + for key in sorted(data.keys() - allowed): + errors.append(f"{context}.{key}: unknown field") + + +def _as_object( + plan: dict[str, Any], key: str, *, required: bool, errors: list[str] +) -> dict[str, Any] | None: + """Return a conditional object after validating its type and keys.""" + value = plan.get(key) + if value is None: + if required: + errors.append(f"{key}: object is required for this plan") + return None + if not isinstance(value, dict): + errors.append(f"{key}: expected an object or null") + return None + _check_unknown_keys(value, OBJECT_KEYS[key], key, errors) + return value + + +def _validate_strings(value: object, path: str, errors: list[str]) -> None: + """Reject unresolved shell variables and placeholder-only strings.""" + if isinstance(value, dict): + for key, item in value.items(): + _validate_strings(item, f"{path}.{key}" if path else key, errors) + elif isinstance(value, list): + for index, item in enumerate(value): + _validate_strings(item, f"{path}[{index}]", errors) + elif isinstance(value, str): + if "$" in value: + errors.append(f"{path}: unresolved shell variable is not allowed") + if re.fullmatch(r"<[^>]+>", value): + errors.append(f"{path}: unresolved placeholder is not allowed") + + +def _validate_checksum(value: object, path: str, errors: list[str]) -> None: + """Validate an optional SHA-256 checksum.""" + if value is not None and ( + not isinstance(value, str) or re.fullmatch(r"[0-9a-fA-F]{64}", value) is None + ): + errors.append(f"{path}: expected 64 hexadecimal SHA-256 characters") + + +def _validate_url_or_path(value: object, path: str, errors: list[str]) -> None: + """Validate an HTTPS URL or an absolute local path.""" + if not isinstance(value, str) or not value: + errors.append(f"{path}: expected an HTTPS URL or absolute path") + return + parsed = urlparse(value) + if parsed.scheme == "https" and parsed.netloc: + return + if Path(value).is_absolute(): + return + errors.append(f"{path}: expected an HTTPS URL or absolute path") + + +def _validate_string_list(value: object, path: str, errors: list[str]) -> None: + """Validate a list of non-empty strings.""" + if not isinstance(value, list) or any( + not isinstance(item, str) or not item for item in value + ): + errors.append(f"{path}: expected a list of non-empty strings") + + +def _validate_environment( + environment: dict[str, Any], method: str, errors: list[str] +) -> None: + """Validate method-specific environment fields.""" + _require_keys(environment, {"kind"}, "environment", errors) + kind = environment.get("kind") + if kind not in {"existing", "venv", "conda", "prefix", "container"}: + errors.append(f"environment.kind: unsupported value {kind!r}") + if method in {"pip", "source"}: + python = environment.get("python") + if not _is_absolute_path(python): + errors.append( + "environment.python: pip and source methods require an absolute executable" + ) + if method == "conda": + if not _is_absolute_path(environment.get("manager")): + errors.append("environment.manager: conda requires an absolute executable") + if not isinstance(environment.get("name"), str) or not environment.get("name"): + errors.append( + "environment.name: conda requires a non-empty environment name" + ) + if kind in {"venv", "prefix"} and not _is_absolute_path(environment.get("prefix")): + errors.append(f"environment.prefix: {kind} requires an absolute path") + if method in {"dp1s", "offline"} and not _is_absolute_path( + environment.get("prefix") + ): + errors.append(f"environment.prefix: {method} requires an absolute path") + if method == "docker" and kind != "container": + errors.append("environment.kind: docker requires 'container'") + + +def _validate_package( + package: dict[str, Any], method: str, goal: str, errors: list[str] +) -> None: + """Validate package and artifact selection.""" + for key in ("backend_packages", "channels"): + if key in package: + _validate_string_list(package[key], f"package.{key}", errors) + for key in ("install_lammps", "install_ipi"): + if key in package and not isinstance(package[key], bool): + errors.append(f"package.{key}: expected a boolean") + if package.get("deepmd_version") is not None and ( + not isinstance(package["deepmd_version"], str) or not package["deepmd_version"] + ): + errors.append("package.deepmd_version: expected a non-empty string or null") + if package.get("backend_index_url") is not None: + _validate_url_or_path( + package["backend_index_url"], "package.backend_index_url", errors + ) + for key in ("deepmd_index_url", "deepmd_extra_index_url"): + if package.get(key) is not None: + _validate_url_or_path(package[key], f"package.{key}", errors) + _validate_checksum(package.get("sha256"), "package.sha256", errors) + if method == "offline": + _require_keys(package, {"artifact_url", "sha256"}, "package", errors) + _validate_url_or_path( + package.get("artifact_url"), "package.artifact_url", errors + ) + if package.get("sha256") is None: + errors.append("package.sha256: offline installation requires a checksum") + if method == "docker" and ( + not isinstance(package.get("docker_image"), str) + or not package.get("docker_image") + ): + errors.append("package.docker_image: docker requires an image reference") + if goal == "python+lammps" and method in {"pip", "conda"}: + if package.get("install_lammps") is not True: + errors.append( + "package.install_lammps: python+lammps requires the packaged LAMMPS runtime" + ) + + +def _validate_source( + source: dict[str, Any], + build: dict[str, Any], + backend: str, + accelerator: str, + errors: list[str], +) -> None: + """Validate source checkout and build fields.""" + _require_keys(source, {"directory", "remote", "ref", "editable"}, "source", errors) + if not _is_absolute_path(source.get("directory")): + errors.append("source.directory: expected an absolute path") + for key in ("remote", "ref"): + if not isinstance(source.get(key), str) or not source.get(key): + errors.append(f"source.{key}: expected a non-empty string") + if not isinstance(source.get("editable"), bool): + errors.append("source.editable: expected a boolean") + + _require_keys( + build, + {"variant", "cc", "cxx", "native_optimization", "jobs"}, + "build", + errors, + ) + variant = build.get("variant") + if variant not in ACCELERATORS: + errors.append(f"build.variant: unsupported value {variant!r}") + for key in ("cc", "cxx"): + if not _is_absolute_path(build.get(key)): + errors.append(f"build.{key}: expected an absolute executable") + if not isinstance(build.get("native_optimization"), bool): + errors.append("build.native_optimization: expected a boolean") + jobs = build.get("jobs") + if not isinstance(jobs, int) or isinstance(jobs, bool) or not 1 <= jobs <= 256: + errors.append("build.jobs: expected an integer from 1 through 256") + if variant == "cuda" and not _is_absolute_path(build.get("cuda_home")): + errors.append("build.cuda_home: CUDA build requires an absolute path") + if variant == "rocm" and not _is_absolute_path(build.get("rocm_root")): + errors.append("build.rocm_root: ROCm build requires an absolute path") + if backend in {"pytorch", "tensorflow"} and variant != accelerator: + errors.append( + f"build.variant: {backend} source build must match accelerator {accelerator!r}" + ) + + +def _validate_cpp( + cpp: dict[str, Any], + source: dict[str, Any], + environment: dict[str, Any], + backend: str, + errors: list[str], +) -> None: + """Validate C/C++ build paths and backend roots.""" + _require_keys(cpp, {"install_prefix", "build_directory"}, "cpp", errors) + for key in ("install_prefix", "build_directory"): + if not _is_absolute_path(cpp.get(key)): + errors.append(f"cpp.{key}: expected an absolute path") + if not all( + _is_absolute_path(cpp.get(key)) for key in ("install_prefix", "build_directory") + ) or not _is_absolute_path(source.get("directory")): + return + prefix = _canonical(cpp["install_prefix"]) + build_directory = _canonical(cpp["build_directory"]) + source_directory = _canonical(source["directory"]) + if len({prefix, build_directory, source_directory}) != 3: + errors.append("cpp: source, build, and install directories must be distinct") + forbidden = { + Path("/"), + Path("/usr"), + Path("/usr/local"), + Path.home().resolve(strict=False), + (Path.home() / ".local").resolve(strict=False), + } + environment_prefix = environment.get("prefix") + if _is_absolute_path(environment_prefix): + forbidden.add(_canonical(environment_prefix)) + if prefix in forbidden: + errors.append("cpp.install_prefix: select a dedicated, non-shared prefix") + if backend == "jax" and not ( + _is_absolute_path(cpp.get("tensorflow_root")) + or _is_absolute_path(cpp.get("tensorflow_c_root")) + ): + errors.append( + "cpp: JAX requires tensorflow_root or tensorflow_c_root for the C API" + ) + if backend == "paddle" and not _is_absolute_path(cpp.get("paddle_inference_dir")): + errors.append("cpp.paddle_inference_dir: Paddle C++ requires an absolute path") + + +def required_lammps_styles(model_family: str, flavor: str) -> list[str]: + """Return the pair styles required by a LAMMPS plan. + + Parameters + ---------- + model_family : str + `conventional`, `dpa4`, or `dpa4c`. + flavor : str + `host` or `kokkos-cuda`. + + Returns + ------- + list of str + Exact pair styles that must appear in `lmp -h`. + """ + base = "dpa4spin" if model_family == "dpa4c" else "deepmd" + styles = [base] + if flavor == "kokkos-cuda": + styles.append(f"{base}/kk") + return styles + + +def _validate_lammps( + lammps: dict[str, Any], + source: dict[str, Any], + build: dict[str, Any], + backend: str, + accelerator: str, + errors: list[str], +) -> None: + """Validate source LAMMPS and Kokkos choices.""" + _require_keys( + lammps, + { + "source_directory", + "build_directory", + "version", + "flavor", + "mpi", + "model_family", + }, + "lammps", + errors, + ) + for key in ("source_directory", "build_directory"): + if not _is_absolute_path(lammps.get(key)): + errors.append(f"lammps.{key}: expected an absolute path") + if not isinstance(lammps.get("version"), str) or not lammps.get("version"): + errors.append("lammps.version: expected a non-empty string") + if not isinstance(lammps.get("mpi"), bool): + errors.append("lammps.mpi: expected a boolean") + flavor = lammps.get("flavor") + if flavor not in {"host", "kokkos-cuda"}: + errors.append(f"lammps.flavor: unsupported value {flavor!r}") + model_family = lammps.get("model_family") + if model_family not in {"conventional", "dpa4", "dpa4c"}: + errors.append(f"lammps.model_family: unsupported value {model_family!r}") + _validate_checksum(lammps.get("sha256"), "lammps.sha256", errors) + source_directory = lammps.get("source_directory") + if _is_absolute_path(source_directory) and not Path(source_directory).exists(): + if lammps.get("url") is None: + errors.append("lammps.url: required when the source directory is absent") + if lammps.get("url") is not None: + _validate_url_or_path(lammps["url"], "lammps.url", errors) + if all( + _is_absolute_path(item) + for item in ( + source.get("directory"), + lammps.get("source_directory"), + lammps.get("build_directory"), + ) + ): + paths = { + _canonical(source["directory"]), + _canonical(lammps["source_directory"]), + _canonical(lammps["build_directory"]), + } + if len(paths) != 3: + errors.append( + "lammps: DeePMD source, LAMMPS source, and build paths must differ" + ) + if flavor == "kokkos-cuda": + if backend != "pytorch": + errors.append( + "lammps.flavor: Kokkos CUDA graph pair styles require PyTorch" + ) + if accelerator != "cuda" or build.get("variant") != "cuda": + errors.append("lammps.flavor: Kokkos CUDA requires CUDA runtime and build") + if ( + not isinstance(lammps.get("machine"), str) + or re.fullmatch(r"[a-z0-9][a-z0-9_-]*", lammps.get("machine", "")) is None + ): + errors.append("lammps.machine: expected a lowercase binary suffix") + if ( + not isinstance(lammps.get("kokkos_arch"), str) + or re.fullmatch(r"[A-Z][A-Z0-9_]*", lammps.get("kokkos_arch", "")) is None + ): + errors.append("lammps.kokkos_arch: expected a Kokkos architecture name") + if model_family in {"dpa4", "dpa4c"} and backend != "pytorch": + errors.append( + f"lammps.model_family: {model_family} requires the PyTorch backend" + ) + + +def _validate_smoke_test( + smoke_test: dict[str, Any], accelerator: str, errors: list[str] +) -> None: + """Validate smoke-test input and explicit GPU binding.""" + _require_keys(smoke_test, {"enabled"}, "smoke_test", errors) + enabled = smoke_test.get("enabled") + if not isinstance(enabled, bool): + errors.append("smoke_test.enabled: expected a boolean") + return + if not enabled: + return + if not _is_absolute_path(smoke_test.get("example")): + errors.append("smoke_test.example: enabled test requires an absolute path") + gpu = smoke_test.get("gpu") + if accelerator == "cuda" and ( + not isinstance(gpu, int) or isinstance(gpu, bool) or gpu < 0 + ): + errors.append( + "smoke_test.gpu: CUDA test requires a non-negative physical index" + ) + if accelerator == "cpu" and gpu is not None: + errors.append("smoke_test.gpu: CPU test must use null") + + +def validate_plan(plan: object) -> list[str]: + """Validate an installation plan. + + Parameters + ---------- + plan : object + Parsed JSON value. + + Returns + ------- + list of str + Validation errors. An empty list means the plan is valid. + """ + errors: list[str] = [] + if not isinstance(plan, dict): + return ["plan: expected a JSON object"] + _check_unknown_keys(plan, TOP_LEVEL_KEYS, "plan", errors) + _require_keys( + plan, + { + "schema_version", + "method", + "goal", + "backend", + "accelerator", + "environment", + "package", + "smoke_test", + }, + "plan", + errors, + ) + _validate_strings(plan, "", errors) + if plan.get("schema_version") != SCHEMA_VERSION: + errors.append(f"schema_version: expected {SCHEMA_VERSION}") + method = plan.get("method") + goal = plan.get("goal") + backend = plan.get("backend") + accelerator = plan.get("accelerator") + if method not in METHODS: + errors.append(f"method: unsupported value {method!r}") + if goal not in GOALS: + errors.append(f"goal: unsupported value {goal!r}") + if backend not in BACKENDS: + errors.append(f"backend: unsupported value {backend!r}") + if accelerator not in ACCELERATORS: + errors.append(f"accelerator: unsupported value {accelerator!r}") + + environment = _as_object(plan, "environment", required=True, errors=errors) + package = _as_object(plan, "package", required=True, errors=errors) + smoke_test = _as_object(plan, "smoke_test", required=True, errors=errors) + if environment is not None and method in METHODS: + _validate_environment(environment, method, errors) + if package is not None and method in METHODS and goal in GOALS: + _validate_package(package, method, goal, errors) + if smoke_test is not None and accelerator in ACCELERATORS: + _validate_smoke_test(smoke_test, accelerator, errors) + + source_required = method == "source" + source = _as_object(plan, "source", required=source_required, errors=errors) + build = _as_object(plan, "build", required=source_required, errors=errors) + if method != "source" and (source is not None or build is not None): + errors.append("source/build: allowed only when method is 'source'") + if ( + source is not None + and build is not None + and backend in BACKENDS + and accelerator in ACCELERATORS + ): + _validate_source(source, build, backend, accelerator, errors) + + cpp_required = goal in {"python+cpp", "python+cpp+lammps"} + cpp = _as_object(plan, "cpp", required=cpp_required, errors=errors) + if cpp_required and method != "source": + errors.append("goal: C/C++ goals require method 'source'") + if not cpp_required and cpp is not None: + errors.append("cpp: object is allowed only for a C/C++ goal") + if ( + cpp is not None + and source is not None + and environment is not None + and backend in BACKENDS + ): + _validate_cpp(cpp, source, environment, backend, errors) + + source_lammps_required = goal == "python+cpp+lammps" + lammps = _as_object(plan, "lammps", required=source_lammps_required, errors=errors) + if source_lammps_required and method != "source": + errors.append("goal: source LAMMPS requires method 'source'") + if not source_lammps_required and lammps is not None: + errors.append("lammps: object is allowed only for python+cpp+lammps") + if ( + lammps is not None + and source is not None + and build is not None + and backend in BACKENDS + and accelerator in ACCELERATORS + ): + _validate_lammps(lammps, source, build, backend, accelerator, errors) + + if goal == "python+lammps" and method == "source": + errors.append("goal: source-built LAMMPS also requires the C/C++ gate") + if goal == "python+lammps" and method not in { + "pip", + "conda", + "dp1s", + "offline", + "docker", + }: + errors.append("goal: packaged LAMMPS requires an easy-install method") + return errors + + +def _load_plan(path: Path) -> object: + """Load a JSON plan from disk.""" + with path.open(encoding="utf-8") as stream: + return json.load(stream) + + +def main(argv: list[str] | None = None) -> int: + """Validate a plan file and print a normalized summary. + + Parameters + ---------- + argv : list of str, optional + Command-line arguments. Defaults to the process arguments. + + Returns + ------- + int + Zero for a valid plan, otherwise one. + """ + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("plan", type=Path, help="Path to install-plan.json.") + parser.add_argument( + "--json", action="store_true", help="Emit a machine-readable result." + ) + args = parser.parse_args(argv) + try: + plan = _load_plan(args.plan) + except FileNotFoundError: + errors = [f"plan: file not found: {args.plan}"] + plan = None + except PermissionError: + errors = [f"plan: permission denied: {args.plan}"] + plan = None + except json.JSONDecodeError as exc: + errors = [f"plan: invalid JSON at line {exc.lineno}, column {exc.colno}"] + plan = None + else: + errors = validate_plan(plan) + + result: dict[str, Any] = {"valid": not errors, "errors": errors} + if isinstance(plan, dict): + result["method"] = plan.get("method") + result["goal"] = plan.get("goal") + result["backend"] = plan.get("backend") + result["accelerator"] = plan.get("accelerator") + lammps = plan.get("lammps") + if isinstance(lammps, dict): + family = lammps.get("model_family") + flavor = lammps.get("flavor") + if family in {"conventional", "dpa4", "dpa4c"} and flavor in { + "host", + "kokkos-cuda", + }: + result["required_lammps_styles"] = required_lammps_styles( + family, flavor + ) + if args.json: + print(json.dumps(result, indent=2, sort_keys=True)) + elif errors: + for error in errors: + print(f"FAIL {error}") + else: + print( + "PASS plan: " + f"method={result['method']} goal={result['goal']} " + f"backend={result['backend']} accelerator={result['accelerator']}" + ) + if "required_lammps_styles" in result: + print( + "PASS required_lammps_styles: " + + ", ".join(result["required_lammps_styles"]) + ) + return 0 if not errors else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/skills/deepmd-install/scripts/verify_lammps.py b/skills/deepmd-install/scripts/verify_lammps.py new file mode 100644 index 0000000000..0fc0a06d0c --- /dev/null +++ b/skills/deepmd-install/scripts/verify_lammps.py @@ -0,0 +1,208 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: LGPL-3.0-or-later +"""Verify a DeePMD-enabled LAMMPS binary and its required pair styles.""" + +from __future__ import ( + annotations, +) + +import argparse +import json +import platform +import re +import shutil +import subprocess +from dataclasses import ( + asdict, + dataclass, +) +from pathlib import ( + Path, +) +from typing import ( + Any, +) + + +@dataclass(frozen=True) +class CheckResult: + """Represent one LAMMPS verification result.""" + + name: str + passed: bool + detail: str + + +def required_styles(model_family: str, flavor: str) -> list[str]: + """Return exact pair styles required by a runtime selection. + + Parameters + ---------- + model_family : str + `conventional`, `dpa4`, or `dpa4c`. + flavor : str + `host` or `kokkos-cuda`. + + Returns + ------- + list of str + Required pair-style tokens. + """ + base = "dpa4spin" if model_family == "dpa4c" else "deepmd" + result = [base] + if flavor == "kokkos-cuda": + result.append(f"{base}/kk") + return result + + +def _resolve_binary(value: str) -> Path | None: + """Resolve an absolute path or executable name.""" + candidate = Path(value) + if candidate.is_absolute(): + return candidate + resolved = shutil.which(value) + return Path(resolved) if resolved is not None else None + + +def _run( + command: list[str], *, timeout: float = 30.0 +) -> subprocess.CompletedProcess[str]: + """Run a bounded LAMMPS verification command.""" + return subprocess.run( + command, + check=False, + capture_output=True, + text=True, + timeout=timeout, + ) + + +def _check_help(binary: Path, styles: list[str]) -> list[CheckResult]: + """Run LAMMPS help and check exact style tokens.""" + try: + completed = _run([str(binary), "-h"]) + except (OSError, subprocess.TimeoutExpired) as exc: + return [CheckResult("lammps_help", False, f"{type(exc).__name__}: {exc}")] + output = "\n".join((completed.stdout, completed.stderr)) + results = [ + CheckResult( + "lammps_help", + completed.returncode == 0, + f"binary={binary} returncode={completed.returncode}", + ) + ] + tokens = set(re.findall(r"[A-Za-z0-9_.+/-]+", output)) + for style in styles: + results.append(CheckResult(f"pair_style:{style}", style in tokens, style)) + return results + + +def _check_links(binary: Path) -> CheckResult: + """Check native library resolution on Linux or macOS.""" + system = platform.system() + if system == "Linux": + tool = shutil.which("ldd") + command = [tool, str(binary)] if tool is not None else None + elif system == "Darwin": + tool = shutil.which("otool") + command = [tool, "-L", str(binary)] if tool is not None else None + else: + return CheckResult("dynamic_links", True, f"not applicable on {system}") + if command is None: + return CheckResult("dynamic_links", False, "link inspection tool unavailable") + try: + completed = _run(command) + except (OSError, subprocess.TimeoutExpired) as exc: + return CheckResult("dynamic_links", False, f"{type(exc).__name__}: {exc}") + output = "\n".join((completed.stdout, completed.stderr)) + unresolved = [line.strip() for line in output.splitlines() if "not found" in line] + passed = completed.returncode == 0 and not unresolved + detail = ( + "; ".join(unresolved) if unresolved else f"returncode={completed.returncode}" + ) + return CheckResult("dynamic_links", passed, detail) + + +def run_checks( + *, binary: str, model_family: str, flavor: str, check_links: bool +) -> list[CheckResult]: + """Verify a LAMMPS executable. + + Parameters + ---------- + binary : str + Absolute path or executable name. + model_family : str + Model family used to derive pair styles. + flavor : str + Host or Kokkos CUDA build. + check_links : bool + Whether to inspect dynamic-library resolution. + + Returns + ------- + list of CheckResult + Ordered verification results. + """ + resolved = _resolve_binary(binary) + if resolved is None or not resolved.is_file(): + return [CheckResult("lammps_binary", False, f"not found: {binary}")] + if not resolved.stat().st_mode & 0o111: + return [CheckResult("lammps_binary", False, f"not executable: {resolved}")] + results = [CheckResult("lammps_binary", True, str(resolved))] + results.extend(_check_help(resolved, required_styles(model_family, flavor))) + if check_links: + results.append(_check_links(resolved)) + return results + + +def _print_results(checks: list[CheckResult], *, as_json: bool) -> None: + """Print verification results.""" + if as_json: + payload: dict[str, Any] = { + "passed": all(item.passed for item in checks), + "checks": [asdict(item) for item in checks], + } + print(json.dumps(payload, indent=2, sort_keys=True)) + return + for item in checks: + status = "PASS" if item.passed else "FAIL" + print(f"{status:<5} {item.name}: {item.detail}") + + +def main(argv: list[str] | None = None) -> int: + """Parse arguments and verify LAMMPS. + + Parameters + ---------- + argv : list of str, optional + Command-line arguments. Defaults to the process arguments. + + Returns + ------- + int + Zero when every requested check passes, otherwise one. + """ + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--binary", required=True) + parser.add_argument( + "--model-family", + required=True, + choices=("conventional", "dpa4", "dpa4c"), + ) + parser.add_argument("--flavor", required=True, choices=("host", "kokkos-cuda")) + parser.add_argument("--check-links", action="store_true") + parser.add_argument("--json", action="store_true") + args = parser.parse_args(argv) + checks = run_checks( + binary=args.binary, + model_family=args.model_family, + flavor=args.flavor, + check_links=args.check_links, + ) + _print_results(checks, as_json=args.json) + return 0 if all(item.passed for item in checks) else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/skills/deepmd-install/scripts/verify_python.py b/skills/deepmd-install/scripts/verify_python.py new file mode 100644 index 0000000000..86db52a439 --- /dev/null +++ b/skills/deepmd-install/scripts/verify_python.py @@ -0,0 +1,341 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: LGPL-3.0-or-later +"""Verify a DeePMD-kit Python install for one backend and accelerator.""" + +from __future__ import ( + annotations, +) + +import argparse +import json +from dataclasses import ( + asdict, + dataclass, +) +from typing import ( + Any, +) + + +@dataclass(frozen=True) +class CheckResult: + """Represent one verification result.""" + + name: str + passed: bool + detail: str + + +def _result(name: str, passed: bool, detail: object) -> CheckResult: + """Create a normalized check result.""" + return CheckResult(name=name, passed=passed, detail=str(detail)) + + +def _check_deepmd() -> CheckResult: + """Import DeePMD-kit and report its version and location.""" + try: + import deepmd + except (ImportError, OSError, RuntimeError) as exc: + return _result("deepmd", False, f"{type(exc).__name__}: {exc}") + version = getattr(deepmd, "__version__", "unknown") + location = getattr(deepmd, "__file__", "unknown") + return _result("deepmd", True, f"version={version} file={location}") + + +def _check_build_variant(expected: str) -> CheckResult: + """Check DeePMD-kit's recorded compiled build variant.""" + try: + from deepmd.env import ( + GLOBAL_CONFIG, + ) + except (ImportError, OSError, RuntimeError) as exc: + return _result("build_variant", False, f"{type(exc).__name__}: {exc}") + actual = str(GLOBAL_CONFIG.get("dp_variant", "unknown")).lower() + return _result( + "build_variant", + actual == expected, + f"expected={expected} actual={actual}", + ) + + +def _check_pytorch(accelerator: str) -> list[CheckResult]: + """Verify the PyTorch backend and a minimal tensor operation.""" + try: + import torch + + import deepmd.pt # noqa: F401 + except (ImportError, OSError, RuntimeError) as exc: + return [_result("pytorch", False, f"{type(exc).__name__}: {exc}")] + details = ( + f"version={torch.__version__} cuda={torch.version.cuda} " + f"hip={torch.version.hip} available={torch.cuda.is_available()}" + ) + results = [_result("pytorch", True, details)] + if accelerator == "cpu": + tensor = torch.ones(1, device="cpu") + 1 + results.append(_result("pytorch_tensor", bool(tensor.item() == 2), "cpu")) + return results + if accelerator == "cuda" and torch.version.cuda is None: + results.append(_result("pytorch_accelerator", False, "CUDA wheel required")) + return results + if accelerator == "rocm" and torch.version.hip is None: + results.append(_result("pytorch_accelerator", False, "ROCm wheel required")) + return results + if not torch.cuda.is_available(): + results.append(_result("pytorch_accelerator", False, "no visible GPU")) + return results + try: + tensor = torch.ones(1, device="cuda:0") + 1 + torch.cuda.synchronize(0) + except RuntimeError as exc: + results.append(_result("pytorch_tensor", False, exc)) + return results + detail = f"device={torch.cuda.get_device_name(0)} value={tensor.item()}" + results.append(_result("pytorch_tensor", bool(tensor.item() == 2), detail)) + return results + + +def _check_tensorflow(accelerator: str) -> list[CheckResult]: + """Verify the TensorFlow backend and a minimal tensor operation.""" + try: + import tensorflow as tf + + import deepmd.tf # noqa: F401 + except (ImportError, OSError, RuntimeError) as exc: + return [_result("tensorflow", False, f"{type(exc).__name__}: {exc}")] + results = [_result("tensorflow", True, f"version={tf.__version__}")] + devices = tf.config.list_physical_devices("GPU") + if accelerator == "cpu": + device = "/CPU:0" + elif not devices: + results.append(_result("tensorflow_accelerator", False, "no visible GPU")) + return results + else: + device = "/GPU:0" + try: + with tf.device(device): + value = float(tf.reduce_sum(tf.ones((2,), dtype=tf.float32)).numpy()) + except (RuntimeError, ValueError) as exc: + results.append(_result("tensorflow_tensor", False, exc)) + return results + results.append(_result("tensorflow_tensor", value == 2.0, f"device={device}")) + return results + + +def _check_jax(accelerator: str) -> list[CheckResult]: + """Verify the JAX backend and a minimal device operation.""" + try: + import jax + import jax.numpy as jnp + + import deepmd.jax # noqa: F401 + except (ImportError, OSError, RuntimeError) as exc: + return [_result("jax", False, f"{type(exc).__name__}: {exc}")] + devices = list(jax.devices()) + results = [ + _result( + "jax", + True, + f"version={jax.__version__} devices={[str(item) for item in devices]}", + ) + ] + if accelerator == "cpu": + candidates = [item for item in devices if item.platform == "cpu"] + else: + candidates = [ + item for item in devices if item.platform in {"gpu", "cuda", "rocm"} + ] + if not candidates: + results.append(_result("jax_accelerator", False, f"no {accelerator} device")) + return results + try: + value = jax.device_put(jnp.ones((2,)), candidates[0]).sum() + value.block_until_ready() + scalar = float(value) + except (RuntimeError, ValueError) as exc: + results.append(_result("jax_tensor", False, exc)) + return results + results.append(_result("jax_tensor", scalar == 2.0, f"device={candidates[0]}")) + return results + + +def _check_paddle(accelerator: str) -> list[CheckResult]: + """Verify the Paddle backend and a minimal tensor operation.""" + try: + import paddle + + import deepmd.pd # noqa: F401 + except (ImportError, OSError, RuntimeError) as exc: + return [_result("paddle", False, f"{type(exc).__name__}: {exc}")] + results = [_result("paddle", True, f"version={paddle.__version__}")] + if accelerator == "cpu": + device = "cpu" + elif accelerator == "cuda": + if ( + not paddle.device.is_compiled_with_cuda() + or paddle.device.cuda.device_count() < 1 + ): + results.append(_result("paddle_accelerator", False, "no visible CUDA GPU")) + return results + device = "gpu:0" + else: + is_rocm = getattr(paddle.device, "is_compiled_with_rocm", lambda: False) + if not is_rocm(): + results.append(_result("paddle_accelerator", False, "ROCm build required")) + return results + device = "gpu:0" + try: + paddle.set_device(device) + value = float(paddle.sum(paddle.ones([2], dtype="float32")).item()) + except (RuntimeError, ValueError) as exc: + results.append(_result("paddle_tensor", False, exc)) + return results + results.append(_result("paddle_tensor", value == 2.0, f"device={device}")) + return results + + +def _check_custom_op() -> CheckResult: + """Require the PyTorch customized operation library.""" + try: + from deepmd.pt.cxx_op import ( + ENABLE_CUSTOMIZED_OP, + ) + except (ImportError, OSError, RuntimeError) as exc: + return _result("pytorch_custom_op", False, f"{type(exc).__name__}: {exc}") + return _result( + "pytorch_custom_op", bool(ENABLE_CUSTOMIZED_OP), ENABLE_CUSTOMIZED_OP + ) + + +def _check_nv() -> CheckResult: + """Require the nvalchemi neighbor-list integration.""" + try: + from deepmd.pt.utils.nv_nlist import ( + is_nv_available, + ) + except (ImportError, OSError, RuntimeError) as exc: + return _result("nvalchemi", False, f"{type(exc).__name__}: {exc}") + available = bool(is_nv_available()) + return _result("nvalchemi", available, available) + + +def _check_vesin() -> CheckResult: + """Require the vesin.torch neighbor-list integration.""" + try: + from deepmd.pt_expt.utils.vesin_neighbor_list import ( + is_vesin_torch_available, + ) + except (ImportError, OSError, RuntimeError) as exc: + return _result("vesin", False, f"{type(exc).__name__}: {exc}") + available = bool(is_vesin_torch_available()) + return _result("vesin", available, available) + + +def run_checks( + *, + backend: str, + accelerator: str, + expected_build_variant: str | None, + expect_custom_op: bool, + expect_nv: bool, + expect_vesin: bool, +) -> list[CheckResult]: + """Run all checks requested by the installation plan. + + Parameters + ---------- + backend : str + Backend name. + accelerator : str + Runtime accelerator. + expected_build_variant : str, optional + Required DeePMD-kit compiled variant. + expect_custom_op : bool + Require the PyTorch custom operation library. + expect_nv : bool + Require nvalchemi. + expect_vesin : bool + Require vesin.torch. + + Returns + ------- + list of CheckResult + Ordered verification results. + """ + checks = [_check_deepmd()] + if expected_build_variant is not None: + checks.append(_check_build_variant(expected_build_variant)) + backend_checks = { + "pytorch": _check_pytorch, + "tensorflow": _check_tensorflow, + "jax": _check_jax, + "paddle": _check_paddle, + } + checks.extend(backend_checks[backend](accelerator)) + if expect_custom_op: + checks.append(_check_custom_op()) + if expect_nv: + checks.append(_check_nv()) + if expect_vesin: + checks.append(_check_vesin()) + return checks + + +def _print_results(checks: list[CheckResult], *, as_json: bool) -> None: + """Print verification results.""" + if as_json: + payload: dict[str, Any] = { + "passed": all(item.passed for item in checks), + "checks": [asdict(item) for item in checks], + } + print(json.dumps(payload, indent=2, sort_keys=True)) + return + for item in checks: + status = "PASS" if item.passed else "FAIL" + print(f"{status:<5} {item.name}: {item.detail}") + + +def main(argv: list[str] | None = None) -> int: + """Parse arguments and verify the requested Python runtime. + + Parameters + ---------- + argv : list of str, optional + Command-line arguments. Defaults to the process arguments. + + Returns + ------- + int + Zero when every requested check passes, otherwise one. + """ + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--backend", + required=True, + choices=("pytorch", "tensorflow", "jax", "paddle"), + ) + parser.add_argument("--accelerator", required=True, choices=("cpu", "cuda", "rocm")) + parser.add_argument("--expected-build-variant", choices=("cpu", "cuda", "rocm")) + parser.add_argument("--expect-custom-op", action="store_true") + parser.add_argument("--expect-nv", action="store_true") + parser.add_argument("--expect-vesin", action="store_true") + parser.add_argument("--json", action="store_true") + args = parser.parse_args(argv) + if args.backend != "pytorch" and any( + (args.expect_custom_op, args.expect_nv, args.expect_vesin) + ): + parser.error("PyTorch-only checks require --backend pytorch") + checks = run_checks( + backend=args.backend, + accelerator=args.accelerator, + expected_build_variant=args.expected_build_variant, + expect_custom_op=args.expect_custom_op, + expect_nv=args.expect_nv, + expect_vesin=args.expect_vesin, + ) + _print_results(checks, as_json=args.json) + return 0 if all(item.passed for item in checks) else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/source/tests/test_deepmd_install_skill.py b/source/tests/test_deepmd_install_skill.py new file mode 100644 index 0000000000..45dd67f056 --- /dev/null +++ b/source/tests/test_deepmd_install_skill.py @@ -0,0 +1,402 @@ +# SPDX-License-Identifier: LGPL-3.0-or-later +"""Tests for the DeePMD-kit installation skill helpers.""" + +from __future__ import ( + annotations, +) + +import importlib.util +import json +import subprocess +import sys +from pathlib import ( + Path, +) +from typing import ( + TYPE_CHECKING, +) + +import pytest + +if TYPE_CHECKING: + from types import ( + ModuleType, + ) + +REPOSITORY_ROOT = Path(__file__).resolve().parents[2] +SCRIPT_DIRECTORY = REPOSITORY_ROOT / "skills" / "deepmd-install" / "scripts" + + +def _load_script(name: str) -> ModuleType: + """Load one helper script as a module.""" + path = SCRIPT_DIRECTORY / f"{name}.py" + spec = importlib.util.spec_from_file_location(f"deepmd_install_{name}", path) + assert spec is not None + assert spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + + +PLAN = _load_script("validate_plan") +PREPARE = _load_script("prepare_lammps") + + +def _source_plan() -> dict[str, object]: + """Return a valid source CUDA DPA4C plan.""" + return { + "schema_version": 1, + "method": "source", + "goal": "python+cpp+lammps", + "backend": "pytorch", + "accelerator": "cuda", + "environment": { + "kind": "existing", + "python": "/opt/deepmd/bin/python", + "manager": None, + "name": None, + "prefix": "/opt/deepmd", + }, + "package": { + "deepmd_version": None, + "deepmd_index_url": None, + "deepmd_extra_index_url": None, + "backend_packages": [], + "backend_index_url": None, + "channels": [], + "install_lammps": False, + "install_ipi": False, + "artifact_url": None, + "sha256": None, + "docker_image": None, + }, + "source": { + "directory": "/work/deepmd-kit", + "remote": "https://github.com/deepmodeling/deepmd-kit.git", + "ref": "master", + "editable": False, + }, + "build": { + "variant": "cuda", + "cc": "/usr/bin/gcc", + "cxx": "/usr/bin/g++", + "cuda_home": "/usr/local/cuda", + "rocm_root": None, + "native_optimization": False, + "jobs": 8, + }, + "cpp": { + "install_prefix": "/work/install/deepmd", + "build_directory": "/work/build/deepmd-cpp", + "tensorflow_root": None, + "tensorflow_c_root": None, + "paddle_inference_dir": None, + }, + "lammps": { + "source_directory": "/work/lammps", + "build_directory": "/work/build/lammps-blackwell120", + "version": "stable_22Jul2025_update2", + "url": "https://github.com/lammps/lammps/archive/refs/tags/stable_22Jul2025_update2.tar.gz", + "sha256": None, + "flavor": "kokkos-cuda", + "machine": "blackwell120", + "kokkos_arch": "BLACKWELL120", + "mpi": False, + "model_family": "dpa4c", + }, + "smoke_test": {"enabled": False, "gpu": None, "example": None}, + } + + +def _easy_plan(method: str) -> dict[str, object]: + """Return a valid Python-only easy-install plan.""" + environment: dict[str, object] = { + "kind": "existing", + "python": "/opt/deepmd/bin/python", + "manager": None, + "name": None, + "prefix": "/opt/deepmd", + } + package: dict[str, object] = { + "deepmd_version": None, + "deepmd_index_url": None, + "deepmd_extra_index_url": None, + "backend_packages": [], + "backend_index_url": None, + "channels": [], + "install_lammps": False, + "install_ipi": False, + "artifact_url": None, + "sha256": None, + "docker_image": None, + } + if method == "conda": + environment.update( + {"kind": "conda", "python": None, "manager": "/opt/conda", "name": "deepmd"} + ) + elif method == "offline": + environment.update({"kind": "prefix", "python": None}) + package.update( + { + "artifact_url": "https://example.invalid/deepmd.sh", + "sha256": "a" * 64, + } + ) + elif method == "docker": + environment.update({"kind": "container", "python": None, "prefix": None}) + package["docker_image"] = "ghcr.io/deepmodeling/deepmd-kit:master" + return { + "schema_version": 1, + "method": method, + "goal": "python", + "backend": "tensorflow", + "accelerator": "cpu", + "environment": environment, + "package": package, + "source": None, + "build": None, + "cpp": None, + "lammps": None, + "smoke_test": {"enabled": False, "gpu": None, "example": None}, + } + + +def test_validate_source_dpa4c_plan() -> None: + """Accept a complete PyTorch CUDA DPA4C plan.""" + plan = _source_plan() + assert PLAN.validate_plan(plan) == [] + assert PLAN.required_lammps_styles("dpa4c", "kokkos-cuda") == [ + "dpa4spin", + "dpa4spin/kk", + ] + + +def test_validate_source_jax_gpu_with_cpu_custom_ops() -> None: + """Allow JAX to provide GPU execution independently of custom OPs.""" + plan = _source_plan() + plan["goal"] = "python" + plan["backend"] = "jax" + build = plan["build"] + assert isinstance(build, dict) + build["variant"] = "cpu" + build["cuda_home"] = None + plan["cpp"] = None + plan["lammps"] = None + assert PLAN.validate_plan(plan) == [] + + +def test_validate_source_tensorflow_cpu_plan() -> None: + """Accept a TensorFlow source build with matching CPU custom operations.""" + plan = _source_plan() + plan["goal"] = "python" + plan["backend"] = "tensorflow" + plan["accelerator"] = "cpu" + build = plan["build"] + assert isinstance(build, dict) + build["variant"] = "cpu" + build["cuda_home"] = None + plan["cpp"] = None + plan["lammps"] = None + assert PLAN.validate_plan(plan) == [] + + +@pytest.mark.parametrize("method", ["pip", "conda", "offline", "docker"]) +def test_validate_easy_install_plans(method: str) -> None: + """Accept the method-specific fields for each easy-install path.""" + assert PLAN.validate_plan(_easy_plan(method)) == [] + + +def test_validate_offline_plan_requires_checksum() -> None: + """Reject an offline artifact without an integrity value.""" + plan = _easy_plan("offline") + package = plan["package"] + assert isinstance(package, dict) + package["sha256"] = None + errors = PLAN.validate_plan(plan) + assert "package.sha256: offline installation requires a checksum" in errors + + +@pytest.mark.parametrize( + ("mutation", "message"), + [ + (("backend", "tensorflow"), "Kokkos CUDA graph pair styles require PyTorch"), + (("source", {"directory": "$HOME/deepmd"}), "unresolved shell variable"), + (("unknown", True), "unknown field"), + ], +) +def test_validate_plan_rejects_invalid_input( + mutation: tuple[str, object], message: str +) -> None: + """Reject unsupported combinations, placeholders, and unknown keys.""" + plan = _source_plan() + key, value = mutation + if key == "source": + source = plan["source"] + assert isinstance(source, dict) + source.update(value) + else: + plan[key] = value + errors = PLAN.validate_plan(plan) + assert any(message in error for error in errors) + + +def test_validate_plan_rejects_shared_cpp_prefix() -> None: + """Reject a C/C++ install into the selected Python environment.""" + plan = _source_plan() + cpp = plan["cpp"] + assert isinstance(cpp, dict) + cpp["install_prefix"] = "/opt/deepmd" + errors = PLAN.validate_plan(plan) + assert "cpp.install_prefix: select a dedicated, non-shared prefix" in errors + + +def test_prepare_lammps_is_idempotent(tmp_path: Path) -> None: + """Create one quoted managed include and preserve it on a second run.""" + lammps = tmp_path / "lammps tree" + deepmd = tmp_path / "deepmd tree" + cmake_file = lammps / "cmake" / "CMakeLists.txt" + builtin = deepmd / "source" / "lmp" / "builtin.cmake" + cmake_file.parent.mkdir(parents=True) + builtin.parent.mkdir(parents=True) + cmake_file.write_text("cmake_minimum_required(VERSION 3.25)\n", encoding="utf-8") + builtin.write_text("# builtin\n", encoding="utf-8") + + assert PREPARE.prepare(lammps, deepmd, check=False) is False + updated = cmake_file.read_text(encoding="utf-8") + assert updated.count(PREPARE.BEGIN_MARKER) == 1 + assert f"include([=[{builtin}]=])" in updated + assert PREPARE.prepare(lammps, deepmd, check=True) is True + + +def test_prepare_lammps_replaces_one_legacy_include() -> None: + """Replace a legacy include without retaining a duplicate.""" + builtin = Path("/work/deepmd/source/lmp/builtin.cmake") + original = "include(/old/deepmd/source/lmp/builtin.cmake)\n" + updated = PREPARE.render_updated_text(original, builtin) + assert "/old/deepmd" not in updated + assert updated.count("source/lmp/builtin.cmake") == 1 + + +def test_prepare_lammps_rejects_duplicate_legacy_includes() -> None: + """Reject an ambiguous source tree with multiple unmanaged includes.""" + original = "\n".join( + ( + "include(/one/source/lmp/builtin.cmake)", + "include(/two/source/lmp/builtin.cmake)", + ) + ) + with pytest.raises(ValueError, match="multiple DeePMD includes"): + PREPARE.render_updated_text( + original, Path("/work/deepmd/source/lmp/builtin.cmake") + ) + + +def test_prepare_lammps_rejects_managed_and_unmanaged_includes() -> None: + """Reject an extra include outside the managed block.""" + builtin = Path("/work/deepmd/source/lmp/builtin.cmake") + original = "\n".join( + ( + PREPARE.BEGIN_MARKER, + f"include([=[{builtin}]=])", + PREPARE.END_MARKER, + "include(/other/source/lmp/builtin.cmake)", + ) + ) + with pytest.raises(ValueError, match="unmanaged DeePMD include"): + PREPARE.render_updated_text(original, builtin) + + +def _write_fake_lammps(path: Path, styles: str) -> None: + """Write an executable that emits a deterministic LAMMPS help surface.""" + path.write_text(f"#!/usr/bin/env python3\nprint({styles!r})\n", encoding="utf-8") + path.chmod(0o755) + + +def test_verify_lammps_dpa4c_cli(tmp_path: Path) -> None: + """Accept the exact DPA4C host and Kokkos pair styles.""" + binary = tmp_path / "lmp_dpa4c" + _write_fake_lammps(binary, "Pair styles: dpa4spin dpa4spin/kk") + completed = subprocess.run( + [ + sys.executable, + str(SCRIPT_DIRECTORY / "verify_lammps.py"), + "--binary", + str(binary), + "--model-family", + "dpa4c", + "--flavor", + "kokkos-cuda", + ], + check=False, + capture_output=True, + text=True, + ) + assert completed.returncode == 0, completed.stdout + completed.stderr + assert "PASS pair_style:dpa4spin/kk" in completed.stdout + + +def test_verify_lammps_requires_exact_style_token(tmp_path: Path) -> None: + """Reject a near-name token instead of accepting a substring.""" + binary = tmp_path / "lmp_near_name" + _write_fake_lammps(binary, "Pair styles: deepmd deepmd/kkx") + completed = subprocess.run( + [ + sys.executable, + str(SCRIPT_DIRECTORY / "verify_lammps.py"), + "--binary", + str(binary), + "--model-family", + "dpa4", + "--flavor", + "kokkos-cuda", + ], + check=False, + capture_output=True, + text=True, + ) + assert completed.returncode == 1 + assert "FAIL pair_style:deepmd/kk" in completed.stdout + + +def test_verify_python_rejects_backend_incompatible_checks() -> None: + """Reject PyTorch-only flags before importing another backend.""" + completed = subprocess.run( + [ + sys.executable, + str(SCRIPT_DIRECTORY / "verify_python.py"), + "--backend", + "jax", + "--accelerator", + "cpu", + "--expect-custom-op", + ], + check=False, + capture_output=True, + text=True, + ) + assert completed.returncode == 2 + assert "PyTorch-only checks require --backend pytorch" in completed.stderr + + +def test_probe_cli_emits_json() -> None: + """Emit the stable top-level probe schema without requiring optional tools.""" + completed = subprocess.run( + [sys.executable, str(SCRIPT_DIRECTORY / "probe_env.py"), "--json"], + check=False, + capture_output=True, + text=True, + timeout=60, + ) + assert completed.returncode == 0, completed.stderr + report = json.loads(completed.stdout) + assert set(report) == { + "system", + "environment", + "python", + "tools", + "nvidia", + "rocm", + "packages", + } + assert Path(report["python"]["executable"]).is_absolute() From 4acc858facd91c6c227320ea52d346fdf2a7b757 Mon Sep 17 00:00:00 2001 From: OutisLi Date: Mon, 17 Aug 2026 13:08:30 +0800 Subject: [PATCH 02/11] fix: address installation skill review feedback --- README.md | 7 +- doc/agent-skills.md | 7 +- doc/install/install-with-agent.md | 7 +- skills/deepmd-install/SKILL.md | 11 +- .../deepmd-install/references/easy-install.md | 40 +- .../references/failure-modes.md | 28 +- .../deepmd-install/references/plan-schema.md | 35 +- .../deepmd-install/references/source-cpp.md | 10 +- .../references/source-lammps.md | 2 +- .../references/source-python.md | 13 +- .../deepmd-install/scripts/validate_plan.py | 240 ++++++++--- .../deepmd-install/scripts/verify_lammps.py | 36 +- .../deepmd-install/scripts/verify_native.py | 121 ++++++ .../deepmd-install/scripts/verify_python.py | 133 +++++- source/tests/test_deepmd_install_skill.py | 400 +++++++++++++++++- 15 files changed, 939 insertions(+), 151 deletions(-) create mode 100644 skills/deepmd-install/scripts/verify_native.py diff --git a/README.md b/README.md index 606b1224bc..c7497b7860 100644 --- a/README.md +++ b/README.md @@ -222,13 +222,12 @@ without directly linking every framework. > --skill deepmd-install -y > ``` > -> If direct GitHub access fails, use `gh-proxy.com` for a public, read-only -> clone, then install from the local checkout. Do not send credentials or -> private repository URLs through the proxy. +> If direct GitHub access fails, clone the official Gitee mirror and install +> from the local checkout: > > ```bash > git clone --depth 1 \ -> https://gh-proxy.com/https://github.com/deepmodeling/deepmd-kit.git \ +> https://gitee.com/deepmodeling/deepmd-kit.git \ > deepmd-kit-skill-source > npx -y skills add ./deepmd-kit-skill-source/skills \ > --skill deepmd-install -y diff --git a/doc/agent-skills.md b/doc/agent-skills.md index 3074cc4527..7a9b0a3d4e 100644 --- a/doc/agent-skills.md +++ b/doc/agent-skills.md @@ -74,13 +74,12 @@ npx -y skills add https://github.com/deepmodeling/deepmd-kit/tree/master/skills --skill '*' -y ``` -If direct GitHub access fails, use `gh-proxy.com` for a public, read-only clone -and install from that checkout. Do not send credentials or private repository -URLs through the proxy. +If direct GitHub access fails, clone the official Gitee mirror and install from +that checkout: ```bash git clone --depth 1 \ - https://gh-proxy.com/https://github.com/deepmodeling/deepmd-kit.git \ + https://gitee.com/deepmodeling/deepmd-kit.git \ deepmd-kit-skill-source npx -y skills add ./deepmd-kit-skill-source/skills --skill '*' -y ``` diff --git a/doc/install/install-with-agent.md b/doc/install/install-with-agent.md index ff48cdcb9e..3c1ed4336c 100644 --- a/doc/install/install-with-agent.md +++ b/doc/install/install-with-agent.md @@ -40,13 +40,12 @@ npx -y skills add https://github.com/deepmodeling/deepmd-kit/tree/master/skills --skill deepmd-install -y ``` -If direct GitHub access fails, use `gh-proxy.com` for a public, read-only clone -and install from the local checkout. Do not send credentials or private -repository URLs through the proxy. +If direct GitHub access fails, clone the official Gitee mirror and install from +the local checkout: ```bash git clone --depth 1 \ - https://gh-proxy.com/https://github.com/deepmodeling/deepmd-kit.git \ + https://gitee.com/deepmodeling/deepmd-kit.git \ deepmd-kit-skill-source npx -y skills add ./deepmd-kit-skill-source/skills \ --skill deepmd-install -y diff --git a/skills/deepmd-install/SKILL.md b/skills/deepmd-install/SKILL.md index 4e7771d4be..5fd84e15cc 100644 --- a/skills/deepmd-install/SKILL.md +++ b/skills/deepmd-install/SKILL.md @@ -1,6 +1,6 @@ --- name: deepmd-install -description: Install or rebuild DeePMD-kit with conda, pip, dp1s, offline packages, Docker, or a source checkout. Use for CPU, NVIDIA CUDA, or ROCm environments; PyTorch, TensorFlow, JAX, or Paddle backends; the C/C++ interface; and DeePMD-enabled LAMMPS, including Kokkos pair styles for DPA4 and DPA4C. +description: Install or rebuild DeePMD-kit with conda, pip, dp1s, offline packages, Docker, or a source checkout. Use for CPU, NVIDIA CUDA, or ROCm environments; PyTorch, TensorFlow, JAX, or Paddle backends; backend-enabled C/C++ interfaces; and DeePMD-enabled LAMMPS, including Kokkos pair styles for DPA4 and DPA4C. --- # Install DeePMD-kit @@ -21,6 +21,10 @@ and verify the requested public interface rather than package presence alone. | source LAMMPS Kokkos CUDA | PyTorch graph artifacts: `deepmd/kk` for DPA4/SeZM and `dpa4spin/kk` for DPA4C | | ROCm source, Windows source, LAMMPS plugin mode | Follow the version-matched documentation in the selected checkout | +Backend-neutral C/C++ libraries with `ALLOW_NO_BACKEND=ON` are outside the +automated plan. Follow `doc/install/install-from-source.md` in the selected +checkout for that layout. + ## Hard rules 1. Resolve the absolute directory containing this `SKILL.md` as `SKILL_ROOT`. @@ -36,7 +40,8 @@ and verify the requested public interface rather than package presence alone. `conda activate` from a previous agent tool call. 1. Render commands with concrete plan values. Stop if a plan placeholder, empty required value, unexpected path, or shell variable not assigned - earlier in the same command block remains. + earlier in the same command block remains. Keep each plan string in the + quoted argument position shown by the selected reference. 1. Never run `git reset --hard`, `git clean`, recursively remove an install prefix, or edit shell rc files as part of this workflow. Use a new build directory or versioned install prefix instead. @@ -106,7 +111,7 @@ cannot leak across calls. | Gate | Required evidence | | ----------- | --------------------------------------------------------------------------------------------------------------------------------- | -| Plan | `validate_plan.py` exits zero and prints the normalized plan | +| Plan | `validate_plan.py` exits zero and prints a validation summary | | Environment | The selected package manager and absolute interpreter target the planned environment | | Python | `verify_python.py` passes for the selected backend and accelerator; source builds also match the expected build variant | | C/C++ | Expected libraries and headers exist, dynamic dependencies resolve, and the build cache records the requested accelerator/backend | diff --git a/skills/deepmd-install/references/easy-install.md b/skills/deepmd-install/references/easy-install.md index e0b113c64e..7453fc212d 100644 --- a/skills/deepmd-install/references/easy-install.md +++ b/skills/deepmd-install/references/easy-install.md @@ -22,6 +22,9 @@ Do not mix conda, pip, offline, and `dp1s` installations in one prefix. Reuse an environment only when the user selected it and the probe found no conflicting DeePMD-kit install. +ROCm installations use the source workflow. Packaged LAMMPS and i-PI support +TensorFlow, PyTorch, and JAX; do not add either extra to a Paddle environment. + For a new venv, create it first, resolve its absolute interpreter, update the plan, and re-run `validate_plan.py` before installing packages: @@ -112,30 +115,43 @@ printf '%s %s\n' "" "" | sha256sum --check - bash "" ``` +For `package.artifact_path`, skip curl and verify the local file directly: + +```bash +printf '%s %s\n' "" "" | sha256sum --check - +bash "" +``` + Use `shasum -a 256` on systems without `sha256sum`. When a release is split, verify each part before concatenating it into a new file. Never execute an HTML error page or an artifact whose checksum is unknown. ## Docker -Pull the exact image reference from the plan and verify it in a disposable -container: +Pull the exact image reference from the plan: ```bash docker pull "" -docker run --rm "" dp --version ``` -For CUDA, bind the physical GPU selected for the smoke test: +Mount the verifier read-only and invoke the backend and accelerator selected by +the plan: ```bash -docker run --rm --gpus 'device=' \ +docker run --rm \ + --mount \ + type=bind,src="/scripts/verify_python.py",dst=/opt/deepmd-install/verify_python.py,readonly \ "" \ - python -c "import torch; print(torch.cuda.get_device_name(0))" + python /opt/deepmd-install/verify_python.py \ + --backend "" \ + --accelerator "" \ + --expected-version "" ``` -Add only user-selected volume mounts and working directories. Do not mount a -home directory or source tree read-write merely for version verification. +For CUDA, add `--gpus 'device='` from the plan before the +read-only mount. Omit `--expected-version` when the plan does not pin one. Add +only user-selected volume mounts and working directories; never mount a home +directory or source tree read-write for verification. ## Verification @@ -144,16 +160,20 @@ Run the backend-aware verifier with the absolute installed interpreter: ```bash "" "/scripts/verify_python.py" \ --backend "" \ - --accelerator "" + --accelerator "" \ + --expected-version "" \ + --expected-prefix "" ``` +Omit `--expected-version` when `package.deepmd_version` is null. + For packaged host LAMMPS, resolve the executable and require the conventional host pair style: ```bash "" "/scripts/verify_lammps.py" \ --binary "" \ - --model-family conventional \ + --model-family "" \ --flavor host ``` diff --git a/skills/deepmd-install/references/failure-modes.md b/skills/deepmd-install/references/failure-modes.md index de3627080b..9534322e55 100644 --- a/skills/deepmd-install/references/failure-modes.md +++ b/skills/deepmd-install/references/failure-modes.md @@ -65,6 +65,7 @@ current directory: test -f "/scripts/probe_env.py" test -f "/scripts/verify_python.py" test -f "/scripts/verify_lammps.py" +test -f "/scripts/verify_native.py" ``` Resolve the directory containing `SKILL.md`; do not search for a same-named @@ -77,12 +78,27 @@ Use the backend-aware verifier and retain its complete output: ```bash "" "/scripts/verify_python.py" \ --backend "" \ - --accelerator "" + --accelerator "" \ + --expected-version "" \ + --expected-prefix "" ``` +Omit `--expected-version` when the plan does not pin a release. + Check visibility masks from the probe before replacing packages. A false GPU availability result may come from `CUDA_VISIBLE_DEVICES`, `HIP_VISIBLE_DEVICES`, a driver/runtime mismatch, or a CPU backend package. +For a ROCm smoke test, inspect occupancy with `rocm-smi`, then bind the planned +physical index in the same command: + +```bash +HIP_VISIBLE_DEVICES="" \ + ROCR_VISIBLE_DEVICES="" \ + "" "/scripts/verify_python.py" \ + --backend "" \ + --accelerator rocm \ + --expected-prefix "" +``` ## Wrong compiled variant @@ -153,7 +169,10 @@ git -C "" rev-parse HEAD ``` Use a separate clone when `HEAD`, remote, or local changes do not match the -plan. Do not reset, clean, or rewrite the existing checkout. +plan. Store the resolved SHA in `source.commit`, revalidate with +`--require-resolved-source`, and pass the same SHA to +`verify_python.py --expected-source-commit`. Do not reset, clean, or rewrite +the existing checkout. ## Stale build directory @@ -181,7 +200,7 @@ from pathlib import Path import torch print(Path(torch.__file__).resolve().parent / "lib") -spec = find_spec("nvidia.nccl") +spec = find_spec("nvidia.nccl") if find_spec("nvidia") is not None else None if spec is not None: for root in spec.submodule_search_locations or (): candidate = Path(root) / "lib" @@ -277,3 +296,6 @@ printf '%s %s\n' "" "" | sha256sum --check - An HTML response, truncated split archive, unexpected top-level directory, or checksum mismatch requires a fresh artifact from the planned URL; it is not a reason to disable verification. + +For `package.artifact_path`, skip curl and run the same file-type and checksum +checks directly on the absolute local path. diff --git a/skills/deepmd-install/references/plan-schema.md b/skills/deepmd-install/references/plan-schema.md index 5ff29e0098..088897884c 100644 --- a/skills/deepmd-install/references/plan-schema.md +++ b/skills/deepmd-install/references/plan-schema.md @@ -82,8 +82,10 @@ Use `null` for an inapplicable object. Do not add undeclared keys. "install_lammps": false, "install_ipi": false, "artifact_url": null, + "artifact_path": null, "sha256": null, - "docker_image": null + "docker_image": null, + "lammps_model_family": null } ``` @@ -92,8 +94,11 @@ Use `null` for an inapplicable object. Do not add undeclared keys. - Keep the two DeePMD-kit index fields null for the default package index. Record a user-selected mirror or the documented pre-release index explicitly. - Keep `backend_index_url` null when the default package index is intended. -- Require `artifact_url` and `sha256` for `offline`. +- Package indexes and download URLs use HTTPS. +- Require exactly one of HTTPS `artifact_url` or absolute `artifact_path`, plus + `sha256`, for `offline`. - Require an immutable image reference or user-selected tag for `docker`. +- Require `lammps_model_family` for packaged LAMMPS verification. ### `source` @@ -102,12 +107,14 @@ Use `null` for an inapplicable object. Do not add undeclared keys. "directory": "/absolute/path/to/deepmd-kit", "remote": "https://github.com/deepmodeling/deepmd-kit.git", "ref": "master", + "commit": null, "editable": false } ``` -Treat `ref` as an opaque Git ref. Resolve it to a commit and record that commit -in the completion report. Do not reset or clean an existing checkout. +Treat `ref` as an opaque Git ref. Resolve it to a commit, store the SHA in +`commit`, and revalidate the plan before building. Do not reset or clean an +existing checkout. ### `build` @@ -178,9 +185,9 @@ For a JAX C++ backend, provide either `tensorflow_root` or } ``` -Require a physical GPU index for a CUDA smoke test. Keep `gpu` null for CPU. -The example path must belong to the selected source checkout or be explicitly -provided by the user. +Require a physical GPU index for CUDA and ROCm smoke tests. Keep `gpu` null for +CPU. The example path must belong to the selected source checkout or be +explicitly provided by the user. ## Validation rules @@ -192,8 +199,11 @@ The validator enforces these invariants: and a CUDA build. 1. DPA4C maps to `dpa4spin`/`dpa4spin/kk`; other families map to `deepmd`/`deepmd/kk`. -1. Source directories, build directories, and install prefixes are distinct. +1. DeePMD, C/C++, and LAMMPS source/build/install paths are distinct. 1. Checksums contain exactly 64 hexadecimal characters. +1. Easy-install methods reject ROCm and Paddle packaged LAMMPS/i-PI. +1. Embedded placeholders, control characters, and POSIX-template escape + characters fail before command rendering. 1. Unknown keys and unsupported combinations fail before any state change. ## Examples @@ -224,8 +234,10 @@ The validator enforces these invariants: "install_lammps": false, "install_ipi": false, "artifact_url": null, + "artifact_path": null, "sha256": null, - "docker_image": null + "docker_image": null, + "lammps_model_family": null }, "source": null, "build": null, @@ -265,13 +277,16 @@ The validator enforces these invariants: "install_lammps": false, "install_ipi": false, "artifact_url": null, + "artifact_path": null, "sha256": null, - "docker_image": null + "docker_image": null, + "lammps_model_family": null }, "source": { "directory": "/work/deepmd-kit", "remote": "https://github.com/deepmodeling/deepmd-kit.git", "ref": "master", + "commit": null, "editable": false }, "build": { diff --git a/skills/deepmd-install/references/source-cpp.md b/skills/deepmd-install/references/source-cpp.md index 17fdc703af..f0928f58f1 100644 --- a/skills/deepmd-install/references/source-cpp.md +++ b/skills/deepmd-install/references/source-cpp.md @@ -5,6 +5,10 @@ Build the C/C++ interface only after the Python/backend gate passes. Use CMake options supported by that ref. The published documentation is . +This workflow builds a backend-enabled C/C++ interface. For backend-neutral +`libdeepmd_cc`/`libdeepmd_c`, follow the checkout documentation and its explicit +`ALLOW_NO_BACKEND=ON` contract. + ## Contents - [Build-directory gate](#build-directory-gate) @@ -131,9 +135,9 @@ Use the platform library suffix on macOS or Windows. On Linux, fail if any installed DeePMD library has an unresolved dynamic dependency: ```bash -for library in ""/lib/libdeepmd*.so; do - ldd "$library" -done +"" "/scripts/verify_native.py" \ + --directory "/lib" \ + --pattern 'libdeepmd*.so*' ``` Finally compile and run a public C++ API probe inside the build directory: diff --git a/skills/deepmd-install/references/source-lammps.md b/skills/deepmd-install/references/source-lammps.md index 98e89274a4..1182c39c8f 100644 --- a/skills/deepmd-install/references/source-lammps.md +++ b/skills/deepmd-install/references/source-lammps.md @@ -135,7 +135,7 @@ from pathlib import Path import torch paths = [Path(torch.__file__).resolve().parent / "lib"] -spec = find_spec("nvidia.nccl") +spec = find_spec("nvidia.nccl") if find_spec("nvidia") is not None else None if spec is not None: for root in spec.submodule_search_locations or (): candidate = Path(root) / "lib" diff --git a/skills/deepmd-install/references/source-python.md b/skills/deepmd-install/references/source-python.md index bc9b64a57d..25ca220526 100644 --- a/skills/deepmd-install/references/source-python.md +++ b/skills/deepmd-install/references/source-python.md @@ -28,6 +28,10 @@ git -C "" checkout --detach \ git -C "" rev-parse HEAD ``` +Store the resolved SHA in `source.commit` and re-run `validate_plan.py` with +`--require-resolved-source` before installing backend packages or building +DeePMD-kit. + For an existing checkout, inspect it without changing its branch, remotes, or working tree: @@ -177,9 +181,12 @@ cd "" && "" "/scripts/verify_python.py" \ --backend "" \ --accelerator "" \ + --expected-prefix "" \ + --expected-source-commit "" \ --expected-build-variant "" ``` -Add `--expect-custom-op` for a PyTorch source build. Add `--expect-nv` and -`--expect-vesin` only when their platform markers and the plan require them. -Do not continue to the C/C++ gate until every requested check passes. +Add `--expected-version ""` when the plan pins a release. Add +`--expect-custom-op` for a PyTorch source build. Add `--expect-nv` and +`--expect-vesin` only when their platform markers and the plan require them. Do +not continue to the C/C++ gate until every requested check passes. diff --git a/skills/deepmd-install/scripts/validate_plan.py b/skills/deepmd-install/scripts/validate_plan.py index 8cdb0e8253..fda4a797c9 100644 --- a/skills/deepmd-install/scripts/validate_plan.py +++ b/skills/deepmd-install/scripts/validate_plan.py @@ -24,6 +24,9 @@ GOALS = {"python", "python+lammps", "python+cpp", "python+cpp+lammps"} BACKENDS = {"pytorch", "tensorflow", "jax", "paddle"} ACCELERATORS = {"cpu", "cuda", "rocm"} +ENVIRONMENT_KINDS = {"existing", "venv", "conda", "prefix", "container"} +LAMMPS_FLAVORS = {"host", "kokkos-cuda"} +MODEL_FAMILIES = {"conventional", "dpa4", "dpa4c"} TOP_LEVEL_KEYS = { "schema_version", "method", @@ -50,10 +53,12 @@ "install_lammps", "install_ipi", "artifact_url", + "artifact_path", "sha256", "docker_image", + "lammps_model_family", }, - "source": {"directory", "remote", "ref", "editable"}, + "source": {"directory", "remote", "ref", "commit", "editable"}, "build": { "variant", "cc", @@ -128,8 +133,18 @@ def _as_object( return value +def _validate_choice( + value: object, choices: set[str], path: str, errors: list[str] +) -> str | None: + """Validate a string enum and return the accepted value.""" + if not isinstance(value, str) or value not in choices: + errors.append(f"{path}: unsupported value {value!r}") + return None + return value + + def _validate_strings(value: object, path: str, errors: list[str]) -> None: - """Reject unresolved shell variables and placeholder-only strings.""" + """Reject unresolved placeholders and unsafe POSIX-template characters.""" if isinstance(value, dict): for key, item in value.items(): _validate_strings(item, f"{path}.{key}" if path else key, errors) @@ -139,8 +154,12 @@ def _validate_strings(value: object, path: str, errors: list[str]) -> None: elif isinstance(value, str): if "$" in value: errors.append(f"{path}: unresolved shell variable is not allowed") - if re.fullmatch(r"<[^>]+>", value): + if re.search(r"<[^<>\r\n]+>", value): errors.append(f"{path}: unresolved placeholder is not allowed") + if any(character in value for character in ('"', "`", "\\")): + errors.append(f"{path}: unsafe shell-template character is not allowed") + if any(character in value for character in ("\0", "\n", "\r")): + errors.append(f"{path}: control character is not allowed") def _validate_checksum(value: object, path: str, errors: list[str]) -> None: @@ -151,17 +170,21 @@ def _validate_checksum(value: object, path: str, errors: list[str]) -> None: errors.append(f"{path}: expected 64 hexadecimal SHA-256 characters") -def _validate_url_or_path(value: object, path: str, errors: list[str]) -> None: - """Validate an HTTPS URL or an absolute local path.""" +def _validate_https_url(value: object, path: str, errors: list[str]) -> None: + """Validate an HTTPS URL without embedded credentials.""" if not isinstance(value, str) or not value: - errors.append(f"{path}: expected an HTTPS URL or absolute path") + errors.append(f"{path}: expected an HTTPS URL") return parsed = urlparse(value) - if parsed.scheme == "https" and parsed.netloc: - return - if Path(value).is_absolute(): + if ( + parsed.scheme == "https" + and parsed.netloc + and parsed.username is None + and parsed.password is None + and not any(character.isspace() for character in value) + ): return - errors.append(f"{path}: expected an HTTPS URL or absolute path") + errors.append(f"{path}: expected an HTTPS URL without credentials") def _validate_string_list(value: object, path: str, errors: list[str]) -> None: @@ -177,9 +200,9 @@ def _validate_environment( ) -> None: """Validate method-specific environment fields.""" _require_keys(environment, {"kind"}, "environment", errors) - kind = environment.get("kind") - if kind not in {"existing", "venv", "conda", "prefix", "container"}: - errors.append(f"environment.kind: unsupported value {kind!r}") + kind = _validate_choice( + environment.get("kind"), ENVIRONMENT_KINDS, "environment.kind", errors + ) if method in {"pip", "source"}: python = environment.get("python") if not _is_absolute_path(python): @@ -204,7 +227,12 @@ def _validate_environment( def _validate_package( - package: dict[str, Any], method: str, goal: str, errors: list[str] + package: dict[str, Any], + method: str, + goal: str, + backend: str, + accelerator: str, + errors: list[str], ) -> None: """Validate package and artifact selection.""" for key in ("backend_packages", "channels"): @@ -218,30 +246,68 @@ def _validate_package( ): errors.append("package.deepmd_version: expected a non-empty string or null") if package.get("backend_index_url") is not None: - _validate_url_or_path( + _validate_https_url( package["backend_index_url"], "package.backend_index_url", errors ) for key in ("deepmd_index_url", "deepmd_extra_index_url"): if package.get(key) is not None: - _validate_url_or_path(package[key], f"package.{key}", errors) + _validate_https_url(package[key], f"package.{key}", errors) _validate_checksum(package.get("sha256"), "package.sha256", errors) if method == "offline": - _require_keys(package, {"artifact_url", "sha256"}, "package", errors) - _validate_url_or_path( - package.get("artifact_url"), "package.artifact_url", errors - ) + _require_keys(package, {"sha256"}, "package", errors) + artifact_url = package.get("artifact_url") + artifact_path = package.get("artifact_path") + if (artifact_url is None) == (artifact_path is None): + errors.append( + "package: offline installation requires exactly one of " + "artifact_url or artifact_path" + ) + elif artifact_url is not None: + _validate_https_url(artifact_url, "package.artifact_url", errors) + elif not _is_absolute_path(artifact_path): + errors.append("package.artifact_path: expected an absolute path") if package.get("sha256") is None: errors.append("package.sha256: offline installation requires a checksum") + elif ( + package.get("artifact_url") is not None + or package.get("artifact_path") is not None + ): + errors.append("package.artifact_url/artifact_path: allowed only for offline") if method == "docker" and ( not isinstance(package.get("docker_image"), str) or not package.get("docker_image") ): errors.append("package.docker_image: docker requires an image reference") - if goal == "python+lammps" and method in {"pip", "conda"}: - if package.get("install_lammps") is not True: + if goal == "python+lammps" and package.get("install_lammps") is not True: + errors.append( + "package.install_lammps: python+lammps requires the packaged LAMMPS runtime" + ) + if goal != "python+lammps" and package.get("install_lammps") is True: + errors.append("package.install_lammps: requires goal 'python+lammps'") + if method != "source" and accelerator == "rocm": + errors.append("accelerator: ROCm requires method 'source'") + packaged_lammps = goal == "python+lammps" or package.get("install_lammps") is True + if packaged_lammps: + family = _validate_choice( + package.get("lammps_model_family"), + MODEL_FAMILIES, + "package.lammps_model_family", + errors, + ) + if family in {"dpa4", "dpa4c"} and backend != "pytorch": errors.append( - "package.install_lammps: python+lammps requires the packaged LAMMPS runtime" + f"package.lammps_model_family: {family} requires the PyTorch backend" ) + elif package.get("lammps_model_family") is not None: + errors.append( + "package.lammps_model_family: allowed only when packaged LAMMPS is requested" + ) + if ( + backend == "paddle" + and method != "source" + and (packaged_lammps or package.get("install_ipi") is True) + ): + errors.append("package: Paddle does not support packaged LAMMPS or i-PI") def _validate_source( @@ -258,6 +324,12 @@ def _validate_source( for key in ("remote", "ref"): if not isinstance(source.get(key), str) or not source.get(key): errors.append(f"source.{key}: expected a non-empty string") + commit = source.get("commit") + if commit is not None and ( + not isinstance(commit, str) + or re.fullmatch(r"[0-9a-fA-F]{7,40}", commit) is None + ): + errors.append("source.commit: expected a 7-40 character commit SHA or null") if not isinstance(source.get("editable"), bool): errors.append("source.editable: expected a boolean") @@ -267,9 +339,9 @@ def _validate_source( "build", errors, ) - variant = build.get("variant") - if variant not in ACCELERATORS: - errors.append(f"build.variant: unsupported value {variant!r}") + variant = _validate_choice( + build.get("variant"), ACCELERATORS, "build.variant", errors + ) for key in ("cc", "cxx"): if not _is_absolute_path(build.get(key)): errors.append(f"build.{key}: expected an absolute executable") @@ -358,6 +430,7 @@ def _validate_lammps( lammps: dict[str, Any], source: dict[str, Any], build: dict[str, Any], + cpp: dict[str, Any], backend: str, accelerator: str, errors: list[str], @@ -383,35 +456,43 @@ def _validate_lammps( errors.append("lammps.version: expected a non-empty string") if not isinstance(lammps.get("mpi"), bool): errors.append("lammps.mpi: expected a boolean") - flavor = lammps.get("flavor") - if flavor not in {"host", "kokkos-cuda"}: - errors.append(f"lammps.flavor: unsupported value {flavor!r}") - model_family = lammps.get("model_family") - if model_family not in {"conventional", "dpa4", "dpa4c"}: - errors.append(f"lammps.model_family: unsupported value {model_family!r}") + flavor = _validate_choice( + lammps.get("flavor"), LAMMPS_FLAVORS, "lammps.flavor", errors + ) + model_family = _validate_choice( + lammps.get("model_family"), + MODEL_FAMILIES, + "lammps.model_family", + errors, + ) _validate_checksum(lammps.get("sha256"), "lammps.sha256", errors) source_directory = lammps.get("source_directory") if _is_absolute_path(source_directory) and not Path(source_directory).exists(): if lammps.get("url") is None: errors.append("lammps.url: required when the source directory is absent") if lammps.get("url") is not None: - _validate_url_or_path(lammps["url"], "lammps.url", errors) + _validate_https_url(lammps["url"], "lammps.url", errors) if all( _is_absolute_path(item) for item in ( source.get("directory"), lammps.get("source_directory"), lammps.get("build_directory"), + cpp.get("build_directory"), + cpp.get("install_prefix"), ) ): paths = { _canonical(source["directory"]), _canonical(lammps["source_directory"]), _canonical(lammps["build_directory"]), + _canonical(cpp["build_directory"]), + _canonical(cpp["install_prefix"]), } - if len(paths) != 3: + if len(paths) != 5: errors.append( - "lammps: DeePMD source, LAMMPS source, and build paths must differ" + "lammps: DeePMD source, C/C++ build/install, and LAMMPS " + "source/build paths must differ" ) if flavor == "kokkos-cuda": if backend != "pytorch": @@ -450,23 +531,26 @@ def _validate_smoke_test( if not _is_absolute_path(smoke_test.get("example")): errors.append("smoke_test.example: enabled test requires an absolute path") gpu = smoke_test.get("gpu") - if accelerator == "cuda" and ( + if accelerator in {"cuda", "rocm"} and ( not isinstance(gpu, int) or isinstance(gpu, bool) or gpu < 0 ): errors.append( - "smoke_test.gpu: CUDA test requires a non-negative physical index" + f"smoke_test.gpu: {accelerator.upper()} test requires a non-negative " + "physical index" ) if accelerator == "cpu" and gpu is not None: errors.append("smoke_test.gpu: CPU test must use null") -def validate_plan(plan: object) -> list[str]: +def validate_plan(plan: object, *, require_resolved_source: bool = False) -> list[str]: """Validate an installation plan. Parameters ---------- plan : object Parsed JSON value. + require_resolved_source : bool, optional + Require `source.commit` for a source build gate. Returns ------- @@ -493,41 +577,53 @@ def validate_plan(plan: object) -> list[str]: errors, ) _validate_strings(plan, "", errors) - if plan.get("schema_version") != SCHEMA_VERSION: + schema_version = plan.get("schema_version") + if ( + not isinstance(schema_version, int) + or isinstance(schema_version, bool) + or schema_version != SCHEMA_VERSION + ): errors.append(f"schema_version: expected {SCHEMA_VERSION}") - method = plan.get("method") - goal = plan.get("goal") - backend = plan.get("backend") - accelerator = plan.get("accelerator") - if method not in METHODS: - errors.append(f"method: unsupported value {method!r}") - if goal not in GOALS: - errors.append(f"goal: unsupported value {goal!r}") - if backend not in BACKENDS: - errors.append(f"backend: unsupported value {backend!r}") - if accelerator not in ACCELERATORS: - errors.append(f"accelerator: unsupported value {accelerator!r}") + method = _validate_choice(plan.get("method"), METHODS, "method", errors) + goal = _validate_choice(plan.get("goal"), GOALS, "goal", errors) + backend = _validate_choice(plan.get("backend"), BACKENDS, "backend", errors) + accelerator = _validate_choice( + plan.get("accelerator"), ACCELERATORS, "accelerator", errors + ) environment = _as_object(plan, "environment", required=True, errors=errors) package = _as_object(plan, "package", required=True, errors=errors) smoke_test = _as_object(plan, "smoke_test", required=True, errors=errors) - if environment is not None and method in METHODS: + if environment is not None and method is not None: _validate_environment(environment, method, errors) - if package is not None and method in METHODS and goal in GOALS: - _validate_package(package, method, goal, errors) - if smoke_test is not None and accelerator in ACCELERATORS: + if ( + package is not None + and method is not None + and goal is not None + and backend is not None + and accelerator is not None + ): + _validate_package(package, method, goal, backend, accelerator, errors) + if smoke_test is not None and accelerator is not None: _validate_smoke_test(smoke_test, accelerator, errors) source_required = method == "source" source = _as_object(plan, "source", required=source_required, errors=errors) build = _as_object(plan, "build", required=source_required, errors=errors) + if ( + require_resolved_source + and method == "source" + and source is not None + and source.get("commit") is None + ): + errors.append("source.commit: resolved source SHA is required for this gate") if method != "source" and (source is not None or build is not None): errors.append("source/build: allowed only when method is 'source'") if ( source is not None and build is not None - and backend in BACKENDS - and accelerator in ACCELERATORS + and backend is not None + and accelerator is not None ): _validate_source(source, build, backend, accelerator, errors) @@ -541,7 +637,7 @@ def validate_plan(plan: object) -> list[str]: cpp is not None and source is not None and environment is not None - and backend in BACKENDS + and backend is not None ): _validate_cpp(cpp, source, environment, backend, errors) @@ -555,10 +651,11 @@ def validate_plan(plan: object) -> list[str]: lammps is not None and source is not None and build is not None - and backend in BACKENDS - and accelerator in ACCELERATORS + and cpp is not None + and backend is not None + and accelerator is not None ): - _validate_lammps(lammps, source, build, backend, accelerator, errors) + _validate_lammps(lammps, source, build, cpp, backend, accelerator, errors) if goal == "python+lammps" and method == "source": errors.append("goal: source-built LAMMPS also requires the C/C++ gate") @@ -580,7 +677,7 @@ def _load_plan(path: Path) -> object: def main(argv: list[str] | None = None) -> int: - """Validate a plan file and print a normalized summary. + """Validate a plan file and print a validation summary. Parameters ---------- @@ -597,6 +694,11 @@ def main(argv: list[str] | None = None) -> int: parser.add_argument( "--json", action="store_true", help="Emit a machine-readable result." ) + parser.add_argument( + "--require-resolved-source", + action="store_true", + help="Require source.commit before a source build gate.", + ) args = parser.parse_args(argv) try: plan = _load_plan(args.plan) @@ -610,7 +712,9 @@ def main(argv: list[str] | None = None) -> int: errors = [f"plan: invalid JSON at line {exc.lineno}, column {exc.colno}"] plan = None else: - errors = validate_plan(plan) + errors = validate_plan( + plan, require_resolved_source=args.require_resolved_source + ) result: dict[str, Any] = {"valid": not errors, "errors": errors} if isinstance(plan, dict): @@ -622,10 +726,12 @@ def main(argv: list[str] | None = None) -> int: if isinstance(lammps, dict): family = lammps.get("model_family") flavor = lammps.get("flavor") - if family in {"conventional", "dpa4", "dpa4c"} and flavor in { - "host", - "kokkos-cuda", - }: + if ( + isinstance(family, str) + and family in MODEL_FAMILIES + and isinstance(flavor, str) + and flavor in LAMMPS_FLAVORS + ): result["required_lammps_styles"] = required_lammps_styles( family, flavor ) diff --git a/skills/deepmd-install/scripts/verify_lammps.py b/skills/deepmd-install/scripts/verify_lammps.py index 0fc0a06d0c..25bdb5e604 100644 --- a/skills/deepmd-install/scripts/verify_lammps.py +++ b/skills/deepmd-install/scripts/verify_lammps.py @@ -8,7 +8,6 @@ import argparse import json -import platform import re import shutil import subprocess @@ -23,6 +22,10 @@ Any, ) +from verify_native import ( + check_dynamic_links, +) + @dataclass(frozen=True) class CheckResult: @@ -97,32 +100,6 @@ def _check_help(binary: Path, styles: list[str]) -> list[CheckResult]: return results -def _check_links(binary: Path) -> CheckResult: - """Check native library resolution on Linux or macOS.""" - system = platform.system() - if system == "Linux": - tool = shutil.which("ldd") - command = [tool, str(binary)] if tool is not None else None - elif system == "Darwin": - tool = shutil.which("otool") - command = [tool, "-L", str(binary)] if tool is not None else None - else: - return CheckResult("dynamic_links", True, f"not applicable on {system}") - if command is None: - return CheckResult("dynamic_links", False, "link inspection tool unavailable") - try: - completed = _run(command) - except (OSError, subprocess.TimeoutExpired) as exc: - return CheckResult("dynamic_links", False, f"{type(exc).__name__}: {exc}") - output = "\n".join((completed.stdout, completed.stderr)) - unresolved = [line.strip() for line in output.splitlines() if "not found" in line] - passed = completed.returncode == 0 and not unresolved - detail = ( - "; ".join(unresolved) if unresolved else f"returncode={completed.returncode}" - ) - return CheckResult("dynamic_links", passed, detail) - - def run_checks( *, binary: str, model_family: str, flavor: str, check_links: bool ) -> list[CheckResult]: @@ -152,7 +129,10 @@ def run_checks( results = [CheckResult("lammps_binary", True, str(resolved))] results.extend(_check_help(resolved, required_styles(model_family, flavor))) if check_links: - results.append(_check_links(resolved)) + link_result = check_dynamic_links(resolved) + results.append( + CheckResult("dynamic_links", link_result.passed, link_result.detail) + ) return results diff --git a/skills/deepmd-install/scripts/verify_native.py b/skills/deepmd-install/scripts/verify_native.py new file mode 100644 index 0000000000..0cc68a2088 --- /dev/null +++ b/skills/deepmd-install/scripts/verify_native.py @@ -0,0 +1,121 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: LGPL-3.0-or-later +"""Verify dynamic-library resolution for native binaries and libraries.""" + +from __future__ import ( + annotations, +) + +import argparse +import platform +import shutil +import subprocess +from dataclasses import ( + dataclass, +) +from pathlib import ( + Path, +) + + +@dataclass(frozen=True) +class LinkResult: + """Represent one dynamic-link inspection result.""" + + path: Path + passed: bool + detail: str + + +def _run(command: list[str]) -> subprocess.CompletedProcess[str]: + """Run a bounded native-link inspection command.""" + return subprocess.run( + command, + check=False, + capture_output=True, + text=True, + timeout=30, + ) + + +def check_dynamic_links(path: Path) -> LinkResult: + """Check one native file for unresolved dynamic dependencies. + + Parameters + ---------- + path : Path + Native executable or library to inspect. + + Returns + ------- + LinkResult + Inspection status and actionable detail. + """ + path = path.resolve(strict=False) + if not path.is_file(): + return LinkResult(path, False, "file not found") + system = platform.system() + if system == "Linux": + tool = shutil.which("ldd") + command = [tool, str(path)] if tool is not None else None + elif system == "Darwin": + tool = shutil.which("otool") + command = [tool, "-L", str(path)] if tool is not None else None + else: + return LinkResult(path, True, f"not applicable on {system}") + if command is None: + return LinkResult(path, False, "link inspection tool unavailable") + try: + completed = _run(command) + except (OSError, subprocess.TimeoutExpired) as exc: + return LinkResult(path, False, f"{type(exc).__name__}: {exc}") + output = "\n".join((completed.stdout, completed.stderr)) + unresolved = [line.strip() for line in output.splitlines() if "not found" in line] + if unresolved: + return LinkResult(path, False, "; ".join(unresolved)) + if completed.returncode != 0: + return LinkResult(path, False, f"returncode={completed.returncode}") + return LinkResult(path, True, "all dynamic dependencies resolved") + + +def _collect_paths( + explicit: list[Path], directory: Path | None, pattern: str +) -> list[Path]: + """Collect unique native paths from CLI selectors.""" + paths = [item.resolve(strict=False) for item in explicit] + if directory is not None: + paths.extend(item.resolve(strict=False) for item in directory.glob(pattern)) + return list(dict.fromkeys(paths)) + + +def main(argv: list[str] | None = None) -> int: + """Parse arguments and inspect native files. + + Parameters + ---------- + argv : list of str, optional + Command-line arguments. Defaults to the process arguments. + + Returns + ------- + int + Zero when every selected file resolves, otherwise one. + """ + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--path", action="append", default=[], type=Path) + parser.add_argument("--directory", type=Path) + parser.add_argument("--pattern", default="libdeepmd*.so") + args = parser.parse_args(argv) + paths = _collect_paths(args.path, args.directory, args.pattern) + if not paths: + print("FAIL dynamic_links: no native files matched") + return 1 + results = [check_dynamic_links(path) for path in paths] + for result in results: + status = "PASS" if result.passed else "FAIL" + print(f"{status:<5} dynamic_links:{result.path}: {result.detail}") + return 0 if all(result.passed for result in results) else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/skills/deepmd-install/scripts/verify_python.py b/skills/deepmd-install/scripts/verify_python.py index 86db52a439..615e1b455c 100644 --- a/skills/deepmd-install/scripts/verify_python.py +++ b/skills/deepmd-install/scripts/verify_python.py @@ -8,10 +8,15 @@ import argparse import json +import re +import sys from dataclasses import ( asdict, dataclass, ) +from pathlib import ( + Path, +) from typing import ( Any, ) @@ -31,15 +36,60 @@ def _result(name: str, passed: bool, detail: object) -> CheckResult: return CheckResult(name=name, passed=passed, detail=str(detail)) -def _check_deepmd() -> CheckResult: - """Import DeePMD-kit and report its version and location.""" +def _check_deepmd( + expected_version: str | None, expected_prefix: str | None +) -> CheckResult: + """Import DeePMD-kit and verify its requested identity.""" try: import deepmd except (ImportError, OSError, RuntimeError) as exc: return _result("deepmd", False, f"{type(exc).__name__}: {exc}") version = getattr(deepmd, "__version__", "unknown") location = getattr(deepmd, "__file__", "unknown") - return _result("deepmd", True, f"version={version} file={location}") + actual_prefix = Path(sys.prefix).resolve(strict=False) + passed = True + identity = [f"version={version}", f"file={location}", f"prefix={actual_prefix}"] + if expected_version is not None: + passed = passed and version == expected_version + identity.append(f"expected_version={expected_version}") + if expected_prefix is not None: + expected = Path(expected_prefix).resolve(strict=False) + passed = passed and actual_prefix == expected + identity.append(f"expected_prefix={expected}") + return _result("deepmd", passed, " ".join(identity)) + + +def _tensorflow_accelerator(tf: Any) -> str | None: + """Return TensorFlow's compiled GPU runtime from build metadata.""" + try: + build_info = tf.sysconfig.get_build_info() + except (AttributeError, RuntimeError, TypeError): + return None + if not isinstance(build_info, dict): + return None + if build_info.get("is_rocm_build") is True: + return "rocm" + if build_info.get("is_cuda_build") is True: + return "cuda" + return None + + +def _jax_accelerator(devices: list[Any]) -> str | None: + """Return JAX's GPU runtime from device and client metadata.""" + metadata: list[str] = [] + for device in devices: + platform_name = str(getattr(device, "platform", "")).lower() + if platform_name in {"cuda", "rocm"}: + return platform_name + metadata.append(str(getattr(device, "device_kind", ""))) + client = getattr(device, "client", None) + metadata.append(str(getattr(client, "platform_version", ""))) + combined = " ".join(metadata).lower() + if "rocm" in combined or "hip" in combined or "amd" in combined: + return "rocm" + if "cuda" in combined or "nvidia" in combined: + return "cuda" + return None def _check_build_variant(expected: str) -> CheckResult: @@ -58,6 +108,33 @@ def _check_build_variant(expected: str) -> CheckResult: ) +def _commits_match(actual: str, expected: str) -> bool: + """Return whether full or abbreviated hexadecimal commits identify one SHA.""" + actual = actual.strip().lower() + expected = expected.strip().lower() + if re.fullmatch(r"[0-9a-f]{7,40}", actual) is None: + return False + if re.fullmatch(r"[0-9a-f]{7,40}", expected) is None: + return False + return actual.startswith(expected) or expected.startswith(actual) + + +def _check_source_commit(expected: str) -> CheckResult: + """Check the source commit recorded in DeePMD-kit's build metadata.""" + try: + from deepmd.env import ( + GLOBAL_CONFIG, + ) + except (ImportError, OSError, RuntimeError) as exc: + return _result("source_commit", False, f"{type(exc).__name__}: {exc}") + actual = str(GLOBAL_CONFIG.get("git_hash", "")) + return _result( + "source_commit", + _commits_match(actual, expected), + f"expected={expected} actual={actual or 'unknown'}", + ) + + def _check_pytorch(accelerator: str) -> list[CheckResult]: """Verify the PyTorch backend and a minimal tensor operation.""" try: @@ -111,6 +188,16 @@ def _check_tensorflow(accelerator: str) -> list[CheckResult]: results.append(_result("tensorflow_accelerator", False, "no visible GPU")) return results else: + runtime = _tensorflow_accelerator(tf) + if runtime != accelerator: + results.append( + _result( + "tensorflow_accelerator", + False, + f"expected={accelerator} actual={runtime or 'unknown'}", + ) + ) + return results device = "/GPU:0" try: with tf.device(device): @@ -148,6 +235,17 @@ def _check_jax(accelerator: str) -> list[CheckResult]: if not candidates: results.append(_result("jax_accelerator", False, f"no {accelerator} device")) return results + if accelerator != "cpu": + runtime = _jax_accelerator(candidates) + if runtime != accelerator: + results.append( + _result( + "jax_accelerator", + False, + f"expected={accelerator} actual={runtime or 'unknown'}", + ) + ) + return results try: value = jax.device_put(jnp.ones((2,)), candidates[0]).sum() value.block_until_ready() @@ -235,6 +333,9 @@ def run_checks( *, backend: str, accelerator: str, + expected_version: str | None, + expected_prefix: str | None, + expected_source_commit: str | None, expected_build_variant: str | None, expect_custom_op: bool, expect_nv: bool, @@ -248,6 +349,12 @@ def run_checks( Backend name. accelerator : str Runtime accelerator. + expected_version : str, optional + Required DeePMD-kit version. + expected_prefix : str, optional + Required Python environment prefix. + expected_source_commit : str, optional + Required source commit recorded at build time. expected_build_variant : str, optional Required DeePMD-kit compiled variant. expect_custom_op : bool @@ -262,7 +369,9 @@ def run_checks( list of CheckResult Ordered verification results. """ - checks = [_check_deepmd()] + checks = [_check_deepmd(expected_version, expected_prefix)] + if expected_source_commit is not None: + checks.append(_check_source_commit(expected_source_commit)) if expected_build_variant is not None: checks.append(_check_build_variant(expected_build_variant)) backend_checks = { @@ -315,6 +424,9 @@ def main(argv: list[str] | None = None) -> int: choices=("pytorch", "tensorflow", "jax", "paddle"), ) parser.add_argument("--accelerator", required=True, choices=("cpu", "cuda", "rocm")) + parser.add_argument("--expected-version") + parser.add_argument("--expected-prefix") + parser.add_argument("--expected-source-commit") parser.add_argument("--expected-build-variant", choices=("cpu", "cuda", "rocm")) parser.add_argument("--expect-custom-op", action="store_true") parser.add_argument("--expect-nv", action="store_true") @@ -325,9 +437,22 @@ def main(argv: list[str] | None = None) -> int: (args.expect_custom_op, args.expect_nv, args.expect_vesin) ): parser.error("PyTorch-only checks require --backend pytorch") + if ( + args.expected_prefix is not None + and not Path(args.expected_prefix).is_absolute() + ): + parser.error("--expected-prefix must be absolute") + if ( + args.expected_source_commit is not None + and re.fullmatch(r"[0-9a-fA-F]{7,40}", args.expected_source_commit) is None + ): + parser.error("--expected-source-commit must be a 7-40 character commit SHA") checks = run_checks( backend=args.backend, accelerator=args.accelerator, + expected_version=args.expected_version, + expected_prefix=args.expected_prefix, + expected_source_commit=args.expected_source_commit, expected_build_variant=args.expected_build_variant, expect_custom_op=args.expect_custom_op, expect_nv=args.expect_nv, diff --git a/source/tests/test_deepmd_install_skill.py b/source/tests/test_deepmd_install_skill.py index 45dd67f056..3eec0a8026 100644 --- a/source/tests/test_deepmd_install_skill.py +++ b/source/tests/test_deepmd_install_skill.py @@ -12,17 +12,13 @@ from pathlib import ( Path, ) -from typing import ( - TYPE_CHECKING, +from types import ( + ModuleType, + SimpleNamespace, ) import pytest -if TYPE_CHECKING: - from types import ( - ModuleType, - ) - REPOSITORY_ROOT = Path(__file__).resolve().parents[2] SCRIPT_DIRECTORY = REPOSITORY_ROOT / "skills" / "deepmd-install" / "scripts" @@ -41,6 +37,8 @@ def _load_script(name: str) -> ModuleType: PLAN = _load_script("validate_plan") PREPARE = _load_script("prepare_lammps") +NATIVE = _load_script("verify_native") +VERIFY = _load_script("verify_python") def _source_plan() -> dict[str, object]: @@ -68,13 +66,16 @@ def _source_plan() -> dict[str, object]: "install_lammps": False, "install_ipi": False, "artifact_url": None, + "artifact_path": None, "sha256": None, "docker_image": None, + "lammps_model_family": None, }, "source": { "directory": "/work/deepmd-kit", "remote": "https://github.com/deepmodeling/deepmd-kit.git", "ref": "master", + "commit": None, "editable": False, }, "build": { @@ -128,8 +129,10 @@ def _easy_plan(method: str) -> dict[str, object]: "install_lammps": False, "install_ipi": False, "artifact_url": None, + "artifact_path": None, "sha256": None, "docker_image": None, + "lammps_model_family": None, } if method == "conda": environment.update( @@ -172,6 +175,17 @@ def test_validate_source_dpa4c_plan() -> None: ] +def test_validate_source_build_gate_requires_resolved_commit() -> None: + """Require the resolved source identity before a source build gate.""" + plan = _source_plan() + errors = PLAN.validate_plan(plan, require_resolved_source=True) + assert "source.commit: resolved source SHA is required for this gate" in errors + source = plan["source"] + assert isinstance(source, dict) + source["commit"] = "ed691aab147d9d7686d296e30f46902c08e9fb68" + assert PLAN.validate_plan(plan, require_resolved_source=True) == [] + + def test_validate_source_jax_gpu_with_cpu_custom_ops() -> None: """Allow JAX to provide GPU execution independently of custom OPs.""" plan = _source_plan() @@ -217,6 +231,213 @@ def test_validate_offline_plan_requires_checksum() -> None: assert "package.sha256: offline installation requires a checksum" in errors +def test_validate_offline_local_artifact() -> None: + """Accept a local offline artifact only through its dedicated path field.""" + plan = _easy_plan("offline") + package = plan["package"] + assert isinstance(package, dict) + package["artifact_url"] = None + package["artifact_path"] = "/opt/artifacts/deepmd.sh" + assert PLAN.validate_plan(plan) == [] + + +def test_validate_offline_artifact_url_requires_https() -> None: + """Reject a local path routed through the offline curl branch.""" + plan = _easy_plan("offline") + package = plan["package"] + assert isinstance(package, dict) + package["artifact_url"] = "/opt/artifacts/deepmd.sh" + errors = PLAN.validate_plan(plan) + assert any( + "package.artifact_url: expected an HTTPS URL" in error for error in errors + ) + + +@pytest.mark.parametrize( + ("path", "value"), + [ + (("method",), []), + (("goal",), {}), + (("environment", "kind"), []), + (("build", "variant"), {}), + (("lammps", "flavor"), []), + (("lammps", "model_family"), {}), + ], +) +def test_validate_plan_rejects_non_string_enums( + path: tuple[str, ...], value: object +) -> None: + """Convert malformed JSON enum types into validation errors.""" + plan = _source_plan() + target: dict[str, object] = plan + for key in path[:-1]: + nested = target[key] + assert isinstance(nested, dict) + target = nested + target[path[-1]] = value + errors = PLAN.validate_plan(plan) + assert any("unsupported value" in error for error in errors) + + +@pytest.mark.parametrize( + ("section", "field", "value"), + [ + ("source", "ref", "v"), + ("package", "backend_packages", ["torch=="]), + ("package", "docker_image", "image:"), + ("environment", "python", "/opt//python"), + ], +) +def test_validate_plan_rejects_embedded_placeholders( + section: str, field: str, value: object +) -> None: + """Reject placeholder tokens embedded inside otherwise valid values.""" + plan = _source_plan() + target = plan[section] + assert isinstance(target, dict) + target[field] = value + errors = PLAN.validate_plan(plan) + assert any("unresolved placeholder" in error for error in errors) + + +@pytest.mark.parametrize( + "value", ['bad"value', "bad`value", "bad\\value", "bad\nvalue"] +) +def test_validate_plan_rejects_unsafe_shell_template_values(value: str) -> None: + """Reject values that can escape the documented POSIX templates.""" + plan = _source_plan() + source = plan["source"] + assert isinstance(source, dict) + source["remote"] = value + errors = PLAN.validate_plan(plan) + assert any( + "unsafe shell-template character" in error or "control character" in error + for error in errors + ) + + +def test_validate_plan_keeps_quoted_semicolon_as_data() -> None: + """Allow semicolons that remain inside the documented quoted argument.""" + plan = _source_plan() + source = plan["source"] + assert isinstance(source, dict) + source["remote"] = "https://example.invalid/repository;mirror.git" + assert PLAN.validate_plan(plan) == [] + + +@pytest.mark.parametrize( + ("field", "value"), + [ + ("backend_index_url", "/mirror"), + ("deepmd_index_url", "file:///mirror"), + ("deepmd_extra_index_url", "http://example.invalid/simple"), + ], +) +def test_validate_plan_requires_https_package_indexes(field: str, value: str) -> None: + """Reject local paths and non-HTTPS package indexes.""" + plan = _easy_plan("pip") + package = plan["package"] + assert isinstance(package, dict) + package[field] = value + errors = PLAN.validate_plan(plan) + assert any("expected an HTTPS URL" in error for error in errors) + + +def test_validate_plan_requires_https_lammps_url() -> None: + """Reject a local path rendered through the LAMMPS curl branch.""" + plan = _source_plan() + lammps = plan["lammps"] + assert isinstance(lammps, dict) + lammps["url"] = "/opt/lammps.tar.gz" + errors = PLAN.validate_plan(plan) + assert any("lammps.url: expected an HTTPS URL" in error for error in errors) + + +@pytest.mark.parametrize("field", ["build_directory", "install_prefix"]) +def test_validate_plan_rejects_lammps_cpp_path_collisions(field: str) -> None: + """Keep LAMMPS source/build paths out of the C/C++ build and prefix.""" + plan = _source_plan() + cpp = plan["cpp"] + lammps = plan["lammps"] + assert isinstance(cpp, dict) + assert isinstance(lammps, dict) + lammps["build_directory"] = cpp[field] + errors = PLAN.validate_plan(plan) + assert any("C/C++ build/install" in error for error in errors) + + +@pytest.mark.parametrize("cpp_field", ["build_directory", "install_prefix"]) +def test_validate_plan_rejects_lammps_source_in_cpp_paths(cpp_field: str) -> None: + """Keep the LAMMPS source tree out of C/C++ build and install paths.""" + plan = _source_plan() + cpp = plan["cpp"] + lammps = plan["lammps"] + assert isinstance(cpp, dict) + assert isinstance(lammps, dict) + lammps["source_directory"] = cpp[cpp_field] + errors = PLAN.validate_plan(plan) + assert any("C/C++ build/install" in error for error in errors) + + +def test_validate_plan_rejects_easy_rocm() -> None: + """Route ROCm installations through the source workflow.""" + plan = _easy_plan("pip") + plan["accelerator"] = "rocm" + errors = PLAN.validate_plan(plan) + assert "accelerator: ROCm requires method 'source'" in errors + + +@pytest.mark.parametrize("feature", ["lammps", "ipi"]) +def test_validate_plan_rejects_paddle_packaged_native_tools(feature: str) -> None: + """Reject packaged LAMMPS and i-PI for the unsupported Paddle backend.""" + plan = _easy_plan("pip") + plan["backend"] = "paddle" + package = plan["package"] + assert isinstance(package, dict) + if feature == "lammps": + plan["goal"] = "python+lammps" + package["install_lammps"] = True + package["lammps_model_family"] = "conventional" + else: + package["install_ipi"] = True + errors = PLAN.validate_plan(plan) + assert "package: Paddle does not support packaged LAMMPS or i-PI" in errors + + +def test_validate_packaged_lammps_requires_model_family() -> None: + """Require exact pair-style identity for packaged LAMMPS.""" + plan = _easy_plan("pip") + plan["goal"] = "python+lammps" + package = plan["package"] + assert isinstance(package, dict) + package["install_lammps"] = True + errors = PLAN.validate_plan(plan) + assert any("package.lammps_model_family" in error for error in errors) + package["lammps_model_family"] = "dpa4c" + plan["backend"] = "pytorch" + assert PLAN.validate_plan(plan) == [] + + +def test_validate_rocm_smoke_test_requires_physical_gpu() -> None: + """Require explicit ROCm device binding for an enabled smoke test.""" + plan = _source_plan() + plan["goal"] = "python" + plan["accelerator"] = "rocm" + plan["backend"] = "jax" + build = plan["build"] + smoke_test = plan["smoke_test"] + assert isinstance(build, dict) + assert isinstance(smoke_test, dict) + build.update({"variant": "cpu", "cuda_home": None}) + plan["cpp"] = None + plan["lammps"] = None + smoke_test.update({"enabled": True, "gpu": None, "example": "/work/example"}) + errors = PLAN.validate_plan(plan) + assert any( + "ROCM test requires a non-negative physical index" in error for error in errors + ) + + @pytest.mark.parametrize( ("mutation", "message"), [ @@ -313,6 +534,171 @@ def _write_fake_lammps(path: Path, styles: str) -> None: path.chmod(0o755) +def test_native_link_check_rejects_not_found_with_zero_exit( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """Fail when ldd reports an unresolved library despite return code zero.""" + binary = tmp_path / "libexample.so" + binary.write_bytes(b"native") + monkeypatch.setattr(NATIVE.platform, "system", lambda: "Linux") + monkeypatch.setattr(NATIVE.shutil, "which", lambda _name: "/usr/bin/ldd") + monkeypatch.setattr( + NATIVE, + "_run", + lambda _command: subprocess.CompletedProcess( + args=[], returncode=0, stdout="libmissing.so => not found\n", stderr="" + ), + ) + result = NATIVE.check_dynamic_links(binary) + assert result.passed is False + assert "libmissing.so => not found" in result.detail + + +def test_verify_python_checks_version_and_prefix( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + """Reject an importable DeePMD-kit with the wrong requested identity.""" + fake_deepmd = SimpleNamespace( + __version__="3.2.0", __file__="/opt/deepmd/deepmd/__init__.py" + ) + monkeypatch.setitem(sys.modules, "deepmd", fake_deepmd) + monkeypatch.setattr(VERIFY.sys, "prefix", str(tmp_path / "environment")) + passing = VERIFY._check_deepmd("3.2.0", str(tmp_path / "environment")) + wrong_version = VERIFY._check_deepmd("3.1.0", str(tmp_path / "environment")) + wrong_prefix = VERIFY._check_deepmd("3.2.0", str(tmp_path / "other")) + assert passing.passed is True + assert wrong_version.passed is False + assert wrong_prefix.passed is False + + +def test_verify_python_matches_abbreviated_source_commit() -> None: + """Match the abbreviated commit stored in build metadata to a full SHA.""" + assert VERIFY._commits_match("e59966be", "e59966be1234567890abcdef1234567890abcdef") + assert not VERIFY._commits_match( + "e59966be", "a59966be1234567890abcdef1234567890abcdef" + ) + + +@pytest.mark.parametrize( + ("build_info", "expected"), + [ + ({"is_cuda_build": True, "is_rocm_build": False}, "cuda"), + ({"is_cuda_build": False, "is_rocm_build": True}, "rocm"), + ], +) +def test_tensorflow_runtime_detection( + build_info: dict[str, bool], expected: str +) -> None: + """Distinguish TensorFlow CUDA and ROCm build metadata.""" + tensorflow = SimpleNamespace( + sysconfig=SimpleNamespace(get_build_info=lambda: build_info) + ) + assert VERIFY._tensorflow_accelerator(tensorflow) == expected + assert VERIFY._tensorflow_accelerator(tensorflow) != ( + "rocm" if expected == "cuda" else "cuda" + ) + + +@pytest.mark.parametrize( + ("requested", "build_info"), + [ + ("cuda", {"is_cuda_build": False, "is_rocm_build": True}), + ("rocm", {"is_cuda_build": True, "is_rocm_build": False}), + ], +) +def test_tensorflow_rejects_wrong_gpu_runtime( + requested: str, + build_info: dict[str, bool], + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Fail TensorFlow verification when GPU presence masks a runtime mismatch.""" + deepmd = ModuleType("deepmd") + deepmd.__path__ = [] # type: ignore[attr-defined] + deepmd_tf = ModuleType("deepmd.tf") + tensorflow = ModuleType("tensorflow") + tensorflow.__version__ = "test" # type: ignore[attr-defined] + tensorflow.sysconfig = SimpleNamespace( # type: ignore[attr-defined] + get_build_info=lambda: build_info + ) + tensorflow.config = SimpleNamespace( # type: ignore[attr-defined] + list_physical_devices=lambda _kind: [object()] + ) + monkeypatch.setitem(sys.modules, "deepmd", deepmd) + monkeypatch.setitem(sys.modules, "deepmd.tf", deepmd_tf) + monkeypatch.setitem(sys.modules, "tensorflow", tensorflow) + checks = VERIFY._check_tensorflow(requested) + assert checks[-1].passed is False + assert f"expected={requested}" in checks[-1].detail + + +@pytest.mark.parametrize( + ("device_kind", "platform_version", "expected"), + [ + ("NVIDIA RTX PRO 6000", "CUDA 13.0", "cuda"), + ("AMD Instinct MI300X", "ROCm 7.0", "rocm"), + ], +) +def test_jax_runtime_detection( + device_kind: str, platform_version: str, expected: str +) -> None: + """Distinguish JAX CUDA and ROCm client metadata.""" + device = SimpleNamespace( + platform="gpu", + device_kind=device_kind, + client=SimpleNamespace(platform_version=platform_version), + ) + assert VERIFY._jax_accelerator([device]) == expected + assert VERIFY._jax_accelerator([device]) != ( + "rocm" if expected == "cuda" else "cuda" + ) + + +@pytest.mark.parametrize( + ("requested", "device_kind", "platform_version"), + [ + ("cuda", "AMD Instinct MI300X", "ROCm 7.0"), + ("rocm", "NVIDIA RTX PRO 6000", "CUDA 13.0"), + ], +) +def test_jax_rejects_wrong_gpu_runtime( + requested: str, + device_kind: str, + platform_version: str, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Fail JAX verification when a GPU uses the other runtime.""" + deepmd = ModuleType("deepmd") + deepmd.__path__ = [] # type: ignore[attr-defined] + deepmd_jax = ModuleType("deepmd.jax") + jax = ModuleType("jax") + jax.__path__ = [] # type: ignore[attr-defined] + jax.__version__ = "test" # type: ignore[attr-defined] + device = SimpleNamespace( + platform="gpu", + device_kind=device_kind, + client=SimpleNamespace(platform_version=platform_version), + ) + jax.devices = lambda: [device] # type: ignore[attr-defined] + jax_numpy = ModuleType("jax.numpy") + monkeypatch.setitem(sys.modules, "deepmd", deepmd) + monkeypatch.setitem(sys.modules, "deepmd.jax", deepmd_jax) + monkeypatch.setitem(sys.modules, "jax", jax) + monkeypatch.setitem(sys.modules, "jax.numpy", jax_numpy) + checks = VERIFY._check_jax(requested) + assert checks[-1].passed is False + assert f"expected={requested}" in checks[-1].detail + + +def test_docker_reference_uses_backend_aware_verifier() -> None: + """Keep Docker verification backend-aware and read-only mounted.""" + reference = ( + REPOSITORY_ROOT / "skills" / "deepmd-install" / "references" / "easy-install.md" + ).read_text(encoding="utf-8") + assert "verify_python.py" in reference + assert '--backend ""' in reference + assert "readonly" in reference + + def test_verify_lammps_dpa4c_cli(tmp_path: Path) -> None: """Accept the exact DPA4C host and Kokkos pair styles.""" binary = tmp_path / "lmp_dpa4c" From a10ba14094a0cb17ffdc16a01ba2cfb4dfdf1395 Mon Sep 17 00:00:00 2001 From: OutisLi Date: Mon, 17 Aug 2026 13:10:35 +0800 Subject: [PATCH 03/11] fix: preserve Windows plan paths --- skills/deepmd-install/scripts/validate_plan.py | 4 +++- source/tests/test_deepmd_install_skill.py | 10 ++++++++++ 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/skills/deepmd-install/scripts/validate_plan.py b/skills/deepmd-install/scripts/validate_plan.py index fda4a797c9..d1c46706fb 100644 --- a/skills/deepmd-install/scripts/validate_plan.py +++ b/skills/deepmd-install/scripts/validate_plan.py @@ -8,6 +8,7 @@ import argparse import json +import os import re from pathlib import ( Path, @@ -156,7 +157,8 @@ def _validate_strings(value: object, path: str, errors: list[str]) -> None: errors.append(f"{path}: unresolved shell variable is not allowed") if re.search(r"<[^<>\r\n]+>", value): errors.append(f"{path}: unresolved placeholder is not allowed") - if any(character in value for character in ('"', "`", "\\")): + unsafe_characters = ('"', "`") + (("\\",) if os.name != "nt" else ()) + if any(character in value for character in unsafe_characters): errors.append(f"{path}: unsafe shell-template character is not allowed") if any(character in value for character in ("\0", "\n", "\r")): errors.append(f"{path}: control character is not allowed") diff --git a/source/tests/test_deepmd_install_skill.py b/source/tests/test_deepmd_install_skill.py index 3eec0a8026..31f0f8034e 100644 --- a/source/tests/test_deepmd_install_skill.py +++ b/source/tests/test_deepmd_install_skill.py @@ -325,6 +325,16 @@ def test_validate_plan_keeps_quoted_semicolon_as_data() -> None: assert PLAN.validate_plan(plan) == [] +def test_validate_plan_allows_windows_path_separator_on_windows( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Preserve Windows paths while keeping POSIX backslash escaping blocked.""" + errors: list[str] = [] + monkeypatch.setattr(PLAN.os, "name", "nt") + PLAN._validate_strings(r"C:\DeePMD\python.exe", "environment.python", errors) + assert errors == [] + + @pytest.mark.parametrize( ("field", "value"), [ From 90e4625ed4b62bb9cd38bc89800d2d56172a629b Mon Sep 17 00:00:00 2001 From: OutisLi Date: Mon, 17 Aug 2026 20:49:46 +0800 Subject: [PATCH 04/11] fix: address follow-up installation skill review --- .../references/failure-modes.md | 40 +++- .../deepmd-install/references/plan-schema.md | 21 +- .../deepmd-install/references/source-cpp.md | 7 +- .../references/source-lammps.md | 6 +- .../references/source-python.md | 9 +- .../deepmd-install/scripts/validate_plan.py | 38 ++-- .../deepmd-install/scripts/verify_lammps.py | 2 +- .../deepmd-install/scripts/verify_native.py | 63 ++++-- .../deepmd-install/scripts/verify_python.py | 54 +++-- source/tests/test_deepmd_install_skill.py | 184 +++++++++++++++--- 10 files changed, 335 insertions(+), 89 deletions(-) diff --git a/skills/deepmd-install/references/failure-modes.md b/skills/deepmd-install/references/failure-modes.md index 9534322e55..ef987b0c24 100644 --- a/skills/deepmd-install/references/failure-modes.md +++ b/skills/deepmd-install/references/failure-modes.md @@ -38,8 +38,14 @@ head -n 1 "$(command -v dp)" ``` The interpreter, pip prefix, and `dp` shebang must identify the same -environment. A previous `conda activate` does not persist across independent -agent shell calls. Re-render the failed gate with the absolute interpreter. +environment. Resolve `deepmd.__file__` as well; it must exist below the planned +prefix. If it points into a checkout or another environment, run the verifier +from a neutral directory without an injected `PYTHONPATH`. A previous +`conda activate` does not persist across independent agent shell calls. +If a neutral-directory import still points into the checkout, replace the +editable installation with the regular source installation documented in +[`source-python.md`](source-python.md). Re-render the failed gate with the +absolute interpreter. ## Invalid or stale plan @@ -88,6 +94,18 @@ Omit `--expected-version` when the plan does not pin a release. Check visibility masks from the probe before replacing packages. A false GPU availability result may come from `CUDA_VISIBLE_DEVICES`, `HIP_VISIBLE_DEVICES`, a driver/runtime mismatch, or a CPU backend package. +For JAX, inspect backend metadata rather than the device vendor label: + +```bash +"" - <<'PY' +import jax + +for device in jax.devices(): + client = device.client + print(device.platform, client.platform, client.platform_version) +PY +``` + For a ROCm smoke test, inspect occupancy with `rocm-smi`, then bind the planned physical index in the same command: @@ -266,15 +284,23 @@ CUDA toolkit. ## Runtime link failure -Inspect the exact binary: +Use the bundled verifier so Linux checks `ldd` output and macOS asks `dyld` to +load the library instead of trusting Mach-O metadata alone: ```bash -ldd "" +"" "/scripts/verify_native.py" \ + --path "" +"" "/scripts/verify_lammps.py" \ + --binary "" \ + --model-family "" \ + --flavor "" \ + --check-links ``` Resolve every `not found` entry through build/install RPATH. Avoid global `LD_LIBRARY_PATH` and shell-rc changes; they can cause another environment's CUDA, Torch, TensorFlow, or compiler runtime to shadow the planned libraries. +On macOS, `otool -L` lists install names but does not prove they resolve. ## Insufficient build resources @@ -293,9 +319,9 @@ file "" printf '%s %s\n' "" "" | sha256sum --check - ``` -An HTML response, truncated split archive, unexpected top-level directory, or -checksum mismatch requires a fresh artifact from the planned URL; it is not a -reason to disable verification. +An absent checksum, HTML response, truncated split archive, unexpected +top-level directory, or checksum mismatch blocks extraction. Obtain a fresh +artifact and trusted checksum for the planned URL; do not disable verification. For `package.artifact_path`, skip curl and run the same file-type and checksum checks directly on the absolute local path. diff --git a/skills/deepmd-install/references/plan-schema.md b/skills/deepmd-install/references/plan-schema.md index 088897884c..fe15657427 100644 --- a/skills/deepmd-install/references/plan-schema.md +++ b/skills/deepmd-install/references/plan-schema.md @@ -107,8 +107,7 @@ Use `null` for an inapplicable object. Do not add undeclared keys. "directory": "/absolute/path/to/deepmd-kit", "remote": "https://github.com/deepmodeling/deepmd-kit.git", "ref": "master", - "commit": null, - "editable": false + "commit": null } ``` @@ -161,7 +160,7 @@ For a JAX C++ backend, provide either `tensorflow_root` or "build_directory": "/absolute/build/lammps-blackwell120", "version": "stable_22Jul2025_update2", "url": "https://github.com/lammps/lammps/archive/refs/tags/stable_22Jul2025_update2.tar.gz", - "sha256": null, + "sha256": "", "flavor": "kokkos-cuda", "machine": "blackwell120", "kokkos_arch": "BLACKWELL120", @@ -174,6 +173,9 @@ For a JAX C++ backend, provide either `tensorflow_root` or - `model_family`: `conventional`, `dpa4`, or `dpa4c`. - `machine` and `kokkos_arch` are required only for `kokkos-cuda`. - Use one Kokkos GPU architecture and one build directory per binary. +- Keep `url` and `sha256` null only when `source_directory` is an existing + directory whose LAMMPS version has been verified. Otherwise provide both an + HTTPS archive URL and its trusted 64-character SHA-256 checksum. ### `smoke_test` @@ -200,7 +202,8 @@ The validator enforces these invariants: 1. DPA4C maps to `dpa4spin`/`dpa4spin/kk`; other families map to `deepmd`/`deepmd/kk`. 1. DeePMD, C/C++, and LAMMPS source/build/install paths are distinct. -1. Checksums contain exactly 64 hexadecimal characters. +1. Checksums contain exactly 64 hexadecimal characters; offline artifacts and + downloaded LAMMPS archives require a checksum. 1. Easy-install methods reject ROCm and Paddle packaged LAMMPS/i-PI. 1. Embedded placeholders, control characters, and POSIX-template escape characters fail before command rendering. @@ -253,6 +256,11 @@ The validator enforces these invariants: ### Source PyTorch CUDA with DPA4C LAMMPS +This is a pre-resolution template. Replace the LAMMPS checksum placeholder +before initial validation. Keep `source.commit` null until the checkout gate +resolves `source.ref`; then record the SHA and run the resolved-source gate +before installing dependencies or building. + ```json { "schema_version": 1, @@ -286,8 +294,7 @@ The validator enforces these invariants: "directory": "/work/deepmd-kit", "remote": "https://github.com/deepmodeling/deepmd-kit.git", "ref": "master", - "commit": null, - "editable": false + "commit": null }, "build": { "variant": "cuda", @@ -310,7 +317,7 @@ The validator enforces these invariants: "build_directory": "/work/build/lammps-blackwell120", "version": "stable_22Jul2025_update2", "url": "https://github.com/lammps/lammps/archive/refs/tags/stable_22Jul2025_update2.tar.gz", - "sha256": null, + "sha256": "", "flavor": "kokkos-cuda", "machine": "blackwell120", "kokkos_arch": "BLACKWELL120", diff --git a/skills/deepmd-install/references/source-cpp.md b/skills/deepmd-install/references/source-cpp.md index f0928f58f1..c6d0d37d99 100644 --- a/skills/deepmd-install/references/source-cpp.md +++ b/skills/deepmd-install/references/source-cpp.md @@ -131,13 +131,12 @@ test -f "/lib/libdeepmd_cc.so" test -f "/lib/libdeepmd_c.so" ``` -Use the platform library suffix on macOS or Windows. On Linux, fail if any -installed DeePMD library has an unresolved dynamic dependency: +On Linux and macOS, fail if any installed DeePMD library has an unresolved +dynamic dependency. The verifier selects the platform library suffix: ```bash "" "/scripts/verify_native.py" \ - --directory "/lib" \ - --pattern 'libdeepmd*.so*' + --directory "/lib" ``` Finally compile and run a public C++ API probe inside the build directory: diff --git a/skills/deepmd-install/references/source-lammps.md b/skills/deepmd-install/references/source-lammps.md index 1182c39c8f..d0a698c734 100644 --- a/skills/deepmd-install/references/source-lammps.md +++ b/skills/deepmd-install/references/source-lammps.md @@ -26,12 +26,16 @@ from the plan into a separate directory: curl -fL "" -o "" ``` -When `lammps.sha256` is non-null, verify it before extraction: +For a download, require `lammps.sha256` and verify it before extraction: ```bash printf '%s %s\n' "" "" | sha256sum --check - ``` +Stop on a missing or mismatched checksum. A plan may omit the URL and checksum +only when `lammps.source_directory` already exists and its version has been +verified. + Extract into a directory whose resolved path equals `lammps.source_directory`. Stop if the archive creates an unexpected directory or contains paths outside its top-level directory. diff --git a/skills/deepmd-install/references/source-python.md b/skills/deepmd-install/references/source-python.md index 25ca220526..9dd340639e 100644 --- a/skills/deepmd-install/references/source-python.md +++ b/skills/deepmd-install/references/source-python.md @@ -32,6 +32,11 @@ Store the resolved SHA in `source.commit` and re-run `validate_plan.py` with `--require-resolved-source` before installing backend packages or building DeePMD-kit. +```bash +"" "/scripts/validate_plan.py" \ + "" --require-resolved-source +``` + For an existing checkout, inspect it without changing its branch, remotes, or working tree: @@ -163,10 +168,6 @@ For CPU, omit CUDA variables and set `DP_VARIANT=cpu`. For ROCm, follow the selected checkout documentation and set `DP_VARIANT=rocm` plus the planned `ROCM_ROOT` in the same invocation. -When `source.editable=true`, add `--editable` to the one pip command. Otherwise -perform a regular installation. Do not make editable mode the default for a -runtime installation. - For TensorFlow, JAX, and Paddle, replace the requirement and the two `DP_ENABLE_*` values using the table. Do not add a backend the plan does not request. diff --git a/skills/deepmd-install/scripts/validate_plan.py b/skills/deepmd-install/scripts/validate_plan.py index d1c46706fb..5b5d5221e6 100644 --- a/skills/deepmd-install/scripts/validate_plan.py +++ b/skills/deepmd-install/scripts/validate_plan.py @@ -59,7 +59,7 @@ "docker_image", "lammps_model_family", }, - "source": {"directory", "remote", "ref", "commit", "editable"}, + "source": {"directory", "remote", "ref", "commit"}, "build": { "variant", "cc", @@ -92,6 +92,11 @@ } +def _platform_name() -> str: + """Return the operating-system family used for command rendering.""" + return os.name + + def _is_absolute_path(value: object) -> bool: """Return whether ``value`` is a non-empty absolute path string.""" return isinstance(value, str) and bool(value) and Path(value).is_absolute() @@ -157,7 +162,7 @@ def _validate_strings(value: object, path: str, errors: list[str]) -> None: errors.append(f"{path}: unresolved shell variable is not allowed") if re.search(r"<[^<>\r\n]+>", value): errors.append(f"{path}: unresolved placeholder is not allowed") - unsafe_characters = ('"', "`") + (("\\",) if os.name != "nt" else ()) + unsafe_characters = ('"', "`") + (("\\",) if _platform_name() != "nt" else ()) if any(character in value for character in unsafe_characters): errors.append(f"{path}: unsafe shell-template character is not allowed") if any(character in value for character in ("\0", "\n", "\r")): @@ -320,7 +325,7 @@ def _validate_source( errors: list[str], ) -> None: """Validate source checkout and build fields.""" - _require_keys(source, {"directory", "remote", "ref", "editable"}, "source", errors) + _require_keys(source, {"directory", "remote", "ref"}, "source", errors) if not _is_absolute_path(source.get("directory")): errors.append("source.directory: expected an absolute path") for key in ("remote", "ref"): @@ -332,9 +337,6 @@ def _validate_source( or re.fullmatch(r"[0-9a-fA-F]{7,40}", commit) is None ): errors.append("source.commit: expected a 7-40 character commit SHA or null") - if not isinstance(source.get("editable"), bool): - errors.append("source.editable: expected a boolean") - _require_keys( build, {"variant", "cc", "cxx", "native_optimization", "jobs"}, @@ -467,13 +469,25 @@ def _validate_lammps( "lammps.model_family", errors, ) - _validate_checksum(lammps.get("sha256"), "lammps.sha256", errors) + checksum = lammps.get("sha256") + _validate_checksum(checksum, "lammps.sha256", errors) source_directory = lammps.get("source_directory") - if _is_absolute_path(source_directory) and not Path(source_directory).exists(): - if lammps.get("url") is None: - errors.append("lammps.url: required when the source directory is absent") - if lammps.get("url") is not None: - _validate_https_url(lammps["url"], "lammps.url", errors) + url = lammps.get("url") + if _is_absolute_path(source_directory): + source_path = Path(source_directory) + if source_path.exists() and not source_path.is_dir(): + errors.append("lammps.source_directory: existing path is not a directory") + elif not source_path.is_dir(): + if url is None: + errors.append( + "lammps.url: required when the source directory is absent" + ) + if checksum is None: + errors.append( + "lammps.sha256: required when the source directory is absent" + ) + if url is not None: + _validate_https_url(url, "lammps.url", errors) if all( _is_absolute_path(item) for item in ( diff --git a/skills/deepmd-install/scripts/verify_lammps.py b/skills/deepmd-install/scripts/verify_lammps.py index 25bdb5e604..cc8d59be7b 100644 --- a/skills/deepmd-install/scripts/verify_lammps.py +++ b/skills/deepmd-install/scripts/verify_lammps.py @@ -129,7 +129,7 @@ def run_checks( results = [CheckResult("lammps_binary", True, str(resolved))] results.extend(_check_help(resolved, required_styles(model_family, flavor))) if check_links: - link_result = check_dynamic_links(resolved) + link_result = check_dynamic_links(resolved, darwin_probe=[str(resolved), "-h"]) results.append( CheckResult("dynamic_links", link_result.passed, link_result.detail) ) diff --git a/skills/deepmd-install/scripts/verify_native.py b/skills/deepmd-install/scripts/verify_native.py index 0cc68a2088..dd6e4fb4e0 100644 --- a/skills/deepmd-install/scripts/verify_native.py +++ b/skills/deepmd-install/scripts/verify_native.py @@ -1,6 +1,6 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: LGPL-3.0-or-later -"""Verify dynamic-library resolution for native binaries and libraries.""" +"""Verify dynamic-library resolution for native libraries.""" from __future__ import ( annotations, @@ -10,6 +10,7 @@ import platform import shutil import subprocess +import sys from dataclasses import ( dataclass, ) @@ -27,6 +28,32 @@ class LinkResult: detail: str +def _system_name() -> str: + """Return the host operating-system name.""" + return platform.system() + + +def _find_executable(name: str) -> str | None: + """Return the absolute path of an executable available on PATH.""" + return shutil.which(name) + + +def _darwin_load_command(path: Path) -> list[str]: + """Build an isolated dyld probe for a loadable Mach-O library.""" + loader = ( + "import ctypes, os, sys; " + "mode = getattr(os, 'RTLD_LOCAL', 0) | getattr(os, 'RTLD_NOW', 0); " + "ctypes.CDLL(sys.argv[1], mode=mode)" + ) + return [sys.executable, "-c", loader, str(path)] + + +def _default_pattern(system_name: str | None = None) -> str: + """Return the platform-specific DeePMD native-library glob.""" + selected_system = _system_name() if system_name is None else system_name + return "libdeepmd*.dylib" if selected_system == "Darwin" else "libdeepmd*.so" + + def _run(command: list[str]) -> subprocess.CompletedProcess[str]: """Run a bounded native-link inspection command.""" return subprocess.run( @@ -38,13 +65,18 @@ def _run(command: list[str]) -> subprocess.CompletedProcess[str]: ) -def check_dynamic_links(path: Path) -> LinkResult: +def check_dynamic_links( + path: Path, *, darwin_probe: list[str] | None = None +) -> LinkResult: """Check one native file for unresolved dynamic dependencies. Parameters ---------- path : Path Native executable or library to inspect. + darwin_probe : list of str, optional + Command that loads a Mach-O executable. Libraries use an isolated + ``ctypes.CDLL`` probe when this argument is omitted. Returns ------- @@ -54,15 +86,14 @@ def check_dynamic_links(path: Path) -> LinkResult: path = path.resolve(strict=False) if not path.is_file(): return LinkResult(path, False, "file not found") - system = platform.system() + system = _system_name() if system == "Linux": - tool = shutil.which("ldd") + tool = _find_executable("ldd") command = [tool, str(path)] if tool is not None else None elif system == "Darwin": - tool = shutil.which("otool") - command = [tool, "-L", str(path)] if tool is not None else None + command = darwin_probe or _darwin_load_command(path) else: - return LinkResult(path, True, f"not applicable on {system}") + return LinkResult(path, False, f"unsupported platform: {system}") if command is None: return LinkResult(path, False, "link inspection tool unavailable") try: @@ -70,11 +101,15 @@ def check_dynamic_links(path: Path) -> LinkResult: except (OSError, subprocess.TimeoutExpired) as exc: return LinkResult(path, False, f"{type(exc).__name__}: {exc}") output = "\n".join((completed.stdout, completed.stderr)) - unresolved = [line.strip() for line in output.splitlines() if "not found" in line] + unresolved = [ + line.strip() for line in output.splitlines() if "not found" in line.lower() + ] if unresolved: return LinkResult(path, False, "; ".join(unresolved)) if completed.returncode != 0: - return LinkResult(path, False, f"returncode={completed.returncode}") + details = [line.strip() for line in output.splitlines() if line.strip()] + suffix = f": {'; '.join(details[-4:])}" if details else "" + return LinkResult(path, False, f"returncode={completed.returncode}{suffix}") return LinkResult(path, True, "all dynamic dependencies resolved") @@ -102,9 +137,15 @@ def main(argv: list[str] | None = None) -> int: Zero when every selected file resolves, otherwise one. """ parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--path", action="append", default=[], type=Path) + parser.add_argument( + "--path", + action="append", + default=[], + type=Path, + help="Native library path.", + ) parser.add_argument("--directory", type=Path) - parser.add_argument("--pattern", default="libdeepmd*.so") + parser.add_argument("--pattern", default=_default_pattern()) args = parser.parse_args(argv) paths = _collect_paths(args.path, args.directory, args.pattern) if not paths: diff --git a/skills/deepmd-install/scripts/verify_python.py b/skills/deepmd-install/scripts/verify_python.py index 615e1b455c..8f59b78557 100644 --- a/skills/deepmd-install/scripts/verify_python.py +++ b/skills/deepmd-install/scripts/verify_python.py @@ -36,6 +36,11 @@ def _result(name: str, passed: bool, detail: object) -> CheckResult: return CheckResult(name=name, passed=passed, detail=str(detail)) +def _interpreter_prefix() -> Path: + """Return the resolved prefix of the running interpreter.""" + return Path(sys.prefix).resolve(strict=False) + + def _check_deepmd( expected_version: str | None, expected_prefix: str | None ) -> CheckResult: @@ -45,8 +50,18 @@ def _check_deepmd( except (ImportError, OSError, RuntimeError) as exc: return _result("deepmd", False, f"{type(exc).__name__}: {exc}") version = getattr(deepmd, "__version__", "unknown") - location = getattr(deepmd, "__file__", "unknown") - actual_prefix = Path(sys.prefix).resolve(strict=False) + location_value = getattr(deepmd, "__file__", None) + if not isinstance(location_value, str) or not location_value: + return _result("deepmd", False, f"version={version} file=missing") + try: + location = Path(location_value).resolve(strict=True) + except (OSError, RuntimeError) as exc: + return _result( + "deepmd", + False, + f"version={version} file={location_value} {type(exc).__name__}: {exc}", + ) + actual_prefix = _interpreter_prefix() passed = True identity = [f"version={version}", f"file={location}", f"prefix={actual_prefix}"] if expected_version is not None: @@ -54,7 +69,8 @@ def _check_deepmd( identity.append(f"expected_version={expected_version}") if expected_prefix is not None: expected = Path(expected_prefix).resolve(strict=False) - passed = passed and actual_prefix == expected + imported_from_prefix = location == expected or expected in location.parents + passed = passed and actual_prefix == expected and imported_from_prefix identity.append(f"expected_prefix={expected}") return _result("deepmd", passed, " ".join(identity)) @@ -75,21 +91,27 @@ def _tensorflow_accelerator(tf: Any) -> str | None: def _jax_accelerator(devices: list[Any]) -> str | None: - """Return JAX's GPU runtime from device and client metadata.""" - metadata: list[str] = [] + """Return JAX's unambiguous GPU runtime from backend metadata.""" + runtimes: set[str] = set() for device in devices: - platform_name = str(getattr(device, "platform", "")).lower() - if platform_name in {"cuda", "rocm"}: - return platform_name - metadata.append(str(getattr(device, "device_kind", ""))) client = getattr(device, "client", None) - metadata.append(str(getattr(client, "platform_version", ""))) - combined = " ".join(metadata).lower() - if "rocm" in combined or "hip" in combined or "amd" in combined: - return "rocm" - if "cuda" in combined or "nvidia" in combined: - return "cuda" - return None + canonical_platforms = ( + str(getattr(client, "platform", "")).lower(), + str(getattr(device, "platform", "")).lower(), + ) + runtime = next( + (item for item in canonical_platforms if item in {"cuda", "rocm"}), + None, + ) + if runtime is None: + platform_version = str(getattr(client, "platform_version", "")).lower() + is_cuda = "cuda" in platform_version + is_rocm = "rocm" in platform_version or "hip" in platform_version + if is_cuda != is_rocm: + runtime = "cuda" if is_cuda else "rocm" + if runtime is not None: + runtimes.add(runtime) + return next(iter(runtimes)) if len(runtimes) == 1 else None def _check_build_variant(expected: str) -> CheckResult: diff --git a/source/tests/test_deepmd_install_skill.py b/source/tests/test_deepmd_install_skill.py index 31f0f8034e..dcd4de3ed3 100644 --- a/source/tests/test_deepmd_install_skill.py +++ b/source/tests/test_deepmd_install_skill.py @@ -76,7 +76,6 @@ def _source_plan() -> dict[str, object]: "remote": "https://github.com/deepmodeling/deepmd-kit.git", "ref": "master", "commit": None, - "editable": False, }, "build": { "variant": "cuda", @@ -99,7 +98,7 @@ def _source_plan() -> dict[str, object]: "build_directory": "/work/build/lammps-blackwell120", "version": "stable_22Jul2025_update2", "url": "https://github.com/lammps/lammps/archive/refs/tags/stable_22Jul2025_update2.tar.gz", - "sha256": None, + "sha256": "a" * 64, "flavor": "kokkos-cuda", "machine": "blackwell120", "kokkos_arch": "BLACKWELL120", @@ -300,9 +299,7 @@ def test_validate_plan_rejects_embedded_placeholders( assert any("unresolved placeholder" in error for error in errors) -@pytest.mark.parametrize( - "value", ['bad"value', "bad`value", "bad\\value", "bad\nvalue"] -) +@pytest.mark.parametrize("value", ['bad"value', "bad`value", "bad\nvalue"]) def test_validate_plan_rejects_unsafe_shell_template_values(value: str) -> None: """Reject values that can escape the documented POSIX templates.""" plan = _source_plan() @@ -325,14 +322,21 @@ def test_validate_plan_keeps_quoted_semicolon_as_data() -> None: assert PLAN.validate_plan(plan) == [] -def test_validate_plan_allows_windows_path_separator_on_windows( +def test_validate_plan_applies_platform_specific_backslash_policy( monkeypatch: pytest.MonkeyPatch, ) -> None: - """Preserve Windows paths while keeping POSIX backslash escaping blocked.""" - errors: list[str] = [] - monkeypatch.setattr(PLAN.os, "name", "nt") - PLAN._validate_strings(r"C:\DeePMD\python.exe", "environment.python", errors) - assert errors == [] + """Preserve Windows paths and reject POSIX shell-escape backslashes.""" + windows_errors: list[str] = [] + monkeypatch.setattr(PLAN, "_platform_name", lambda: "nt") + PLAN._validate_strings( + r"C:\DeePMD\python.exe", "environment.python", windows_errors + ) + assert windows_errors == [] + + posix_errors: list[str] = [] + monkeypatch.setattr(PLAN, "_platform_name", lambda: "posix") + PLAN._validate_strings(r"C:\DeePMD\python.exe", "environment.python", posix_errors) + assert any("unsafe shell-template character" in item for item in posix_errors) @pytest.mark.parametrize( @@ -363,6 +367,44 @@ def test_validate_plan_requires_https_lammps_url() -> None: assert any("lammps.url: expected an HTTPS URL" in error for error in errors) +def test_validate_plan_requires_checksum_for_lammps_download(tmp_path: Path) -> None: + """Reject an unverified archive when the LAMMPS source is absent.""" + plan = _source_plan() + lammps = plan["lammps"] + assert isinstance(lammps, dict) + lammps["source_directory"] = str(tmp_path / "missing-lammps") + lammps["sha256"] = None + errors = PLAN.validate_plan(plan) + assert "lammps.sha256: required when the source directory is absent" in errors + + +def test_validate_plan_allows_existing_lammps_without_archive( + tmp_path: Path, +) -> None: + """Allow a verified existing source directory without download fields.""" + source_directory = tmp_path / "lammps" + source_directory.mkdir() + plan = _source_plan() + lammps = plan["lammps"] + assert isinstance(lammps, dict) + lammps.update( + {"source_directory": str(source_directory), "url": None, "sha256": None} + ) + assert PLAN.validate_plan(plan) == [] + + +def test_validate_plan_rejects_lammps_source_file(tmp_path: Path) -> None: + """Reject an existing file where a LAMMPS source directory is required.""" + source_path = tmp_path / "lammps" + source_path.write_text("not a source tree", encoding="utf-8") + plan = _source_plan() + lammps = plan["lammps"] + assert isinstance(lammps, dict) + lammps["source_directory"] = str(source_path) + errors = PLAN.validate_plan(plan) + assert "lammps.source_directory: existing path is not a directory" in errors + + @pytest.mark.parametrize("field", ["build_directory", "install_prefix"]) def test_validate_plan_rejects_lammps_cpp_path_collisions(field: str) -> None: """Keep LAMMPS source/build paths out of the C/C++ build and prefix.""" @@ -550,8 +592,8 @@ def test_native_link_check_rejects_not_found_with_zero_exit( """Fail when ldd reports an unresolved library despite return code zero.""" binary = tmp_path / "libexample.so" binary.write_bytes(b"native") - monkeypatch.setattr(NATIVE.platform, "system", lambda: "Linux") - monkeypatch.setattr(NATIVE.shutil, "which", lambda _name: "/usr/bin/ldd") + monkeypatch.setattr(NATIVE, "_system_name", lambda: "Linux") + monkeypatch.setattr(NATIVE, "_find_executable", lambda _name: "/usr/bin/ldd") monkeypatch.setattr( NATIVE, "_run", @@ -564,23 +606,94 @@ def test_native_link_check_rejects_not_found_with_zero_exit( assert "libmissing.so => not found" in result.detail +def test_native_link_check_uses_dyld_loader( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """Fail when the isolated Darwin loader cannot resolve a dependency.""" + library = tmp_path / "libexample.dylib" + library.write_bytes(b"native") + commands: list[list[str]] = [] + + def run(command: list[str]) -> subprocess.CompletedProcess[str]: + commands.append(command) + return subprocess.CompletedProcess( + args=command, + returncode=1, + stdout="", + stderr="OSError: Library not loaded: @rpath/libmissing.dylib", + ) + + monkeypatch.setattr(NATIVE, "_system_name", lambda: "Darwin") + monkeypatch.setattr(NATIVE, "_run", run) + result = NATIVE.check_dynamic_links(library) + assert result.passed is False + assert "Library not loaded" in result.detail + assert commands[0][:2] == [sys.executable, "-c"] + assert "otool" not in commands[0] + + +def test_native_link_default_pattern_matches_platform() -> None: + """Select the native-library suffix without changing explicit patterns.""" + assert NATIVE._default_pattern("Linux") == "libdeepmd*.so" + assert NATIVE._default_pattern("Darwin") == "libdeepmd*.dylib" + + +def test_native_link_collection_preserves_explicit_pattern(tmp_path: Path) -> None: + """Use a caller-provided directory pattern without platform substitution.""" + selected = tmp_path / "libdeepmd.custom" + ignored = tmp_path / "libdeepmd.so" + selected.write_bytes(b"native") + ignored.write_bytes(b"native") + assert NATIVE._collect_paths([], tmp_path, "*.custom") == [selected.resolve()] + + +def test_native_link_check_fails_closed_on_unsupported_platform( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """Reject an unimplemented platform instead of reporting false success.""" + library = tmp_path / "deepmd.dll" + library.write_bytes(b"native") + monkeypatch.setattr(NATIVE, "_system_name", lambda: "Windows") + result = NATIVE.check_dynamic_links(library) + assert result.passed is False + assert result.detail == "unsupported platform: Windows" + + def test_verify_python_checks_version_and_prefix( monkeypatch: pytest.MonkeyPatch, tmp_path: Path ) -> None: - """Reject an importable DeePMD-kit with the wrong requested identity.""" - fake_deepmd = SimpleNamespace( - __version__="3.2.0", __file__="/opt/deepmd/deepmd/__init__.py" - ) + """Require both interpreter and imported module to use the planned prefix.""" + environment = tmp_path / "environment" + module_file = environment / "lib" / "deepmd" / "__init__.py" + module_file.parent.mkdir(parents=True) + module_file.write_text("", encoding="utf-8") + fake_deepmd = SimpleNamespace(__version__="3.2.0", __file__=str(module_file)) monkeypatch.setitem(sys.modules, "deepmd", fake_deepmd) - monkeypatch.setattr(VERIFY.sys, "prefix", str(tmp_path / "environment")) - passing = VERIFY._check_deepmd("3.2.0", str(tmp_path / "environment")) - wrong_version = VERIFY._check_deepmd("3.1.0", str(tmp_path / "environment")) + monkeypatch.setattr(VERIFY, "_interpreter_prefix", lambda: environment.resolve()) + passing = VERIFY._check_deepmd("3.2.0", str(environment)) + wrong_version = VERIFY._check_deepmd("3.1.0", str(environment)) wrong_prefix = VERIFY._check_deepmd("3.2.0", str(tmp_path / "other")) assert passing.passed is True assert wrong_version.passed is False assert wrong_prefix.passed is False +def test_verify_python_rejects_shadowed_or_missing_module( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + """Reject a checkout import even when the interpreter prefix is correct.""" + environment = tmp_path / "environment" + checkout_file = tmp_path / "checkout" / "deepmd" / "__init__.py" + checkout_file.parent.mkdir(parents=True) + checkout_file.write_text("", encoding="utf-8") + fake_deepmd = SimpleNamespace(__version__="3.2.0", __file__=str(checkout_file)) + monkeypatch.setitem(sys.modules, "deepmd", fake_deepmd) + monkeypatch.setattr(VERIFY, "_interpreter_prefix", lambda: environment.resolve()) + assert VERIFY._check_deepmd("3.2.0", str(environment)).passed is False + fake_deepmd.__file__ = None + assert VERIFY._check_deepmd("3.2.0", str(environment)).passed is False + + def test_verify_python_matches_abbreviated_source_commit() -> None: """Match the abbreviated commit stored in build metadata to a full SHA.""" assert VERIFY._commits_match("e59966be", "e59966be1234567890abcdef1234567890abcdef") @@ -642,20 +755,27 @@ def test_tensorflow_rejects_wrong_gpu_runtime( @pytest.mark.parametrize( - ("device_kind", "platform_version", "expected"), + ("client_platform", "device_kind", "platform_version", "expected"), [ - ("NVIDIA RTX PRO 6000", "CUDA 13.0", "cuda"), - ("AMD Instinct MI300X", "ROCm 7.0", "rocm"), + ("gpu", "NVIDIA RTX PRO 6000", "CUDA 13.0", "cuda"), + ("gpu", "AMD Instinct MI300X", "ROCm 7.0", "rocm"), + ("gpu", "AMD-compatible adapter", "CUDA 13.0", "cuda"), + ("cuda", "vendor text is not authoritative", "unknown", "cuda"), ], ) def test_jax_runtime_detection( - device_kind: str, platform_version: str, expected: str + client_platform: str, + device_kind: str, + platform_version: str, + expected: str, ) -> None: """Distinguish JAX CUDA and ROCm client metadata.""" device = SimpleNamespace( platform="gpu", device_kind=device_kind, - client=SimpleNamespace(platform_version=platform_version), + client=SimpleNamespace( + platform=client_platform, platform_version=platform_version + ), ) assert VERIFY._jax_accelerator([device]) == expected assert VERIFY._jax_accelerator([device]) != ( @@ -663,6 +783,18 @@ def test_jax_runtime_detection( ) +def test_jax_runtime_detection_fails_closed_on_conflicting_backends() -> None: + """Reject a device set that reports both CUDA and ROCm clients.""" + devices = [ + SimpleNamespace( + platform="gpu", + client=SimpleNamespace(platform=runtime, platform_version=runtime), + ) + for runtime in ("cuda", "rocm") + ] + assert VERIFY._jax_accelerator(devices) is None + + @pytest.mark.parametrize( ("requested", "device_kind", "platform_version"), [ @@ -686,7 +818,7 @@ def test_jax_rejects_wrong_gpu_runtime( device = SimpleNamespace( platform="gpu", device_kind=device_kind, - client=SimpleNamespace(platform_version=platform_version), + client=SimpleNamespace(platform="gpu", platform_version=platform_version), ) jax.devices = lambda: [device] # type: ignore[attr-defined] jax_numpy = ModuleType("jax.numpy") From cb1e44f42b04459f2183535312e889fd0cd96070 Mon Sep 17 00:00:00 2001 From: OutisLi Date: Tue, 18 Aug 2026 10:26:27 +0800 Subject: [PATCH 05/11] fix: align installation skill plan contracts --- .../deepmd-install/references/easy-install.md | 11 +++++ .../references/failure-modes.md | 6 +++ .../deepmd-install/references/plan-schema.md | 15 +++++- .../deepmd-install/references/source-cpp.md | 24 +++++---- .../deepmd-install/scripts/validate_plan.py | 13 +++-- source/tests/test_deepmd_install_skill.py | 49 +++++++++++++++++++ 6 files changed, 103 insertions(+), 15 deletions(-) diff --git a/skills/deepmd-install/references/easy-install.md b/skills/deepmd-install/references/easy-install.md index 7453fc212d..104e650f78 100644 --- a/skills/deepmd-install/references/easy-install.md +++ b/skills/deepmd-install/references/easy-install.md @@ -73,6 +73,17 @@ non-null. Use the absolute manager and environment name from the plan. Install only the requested packages; do not add LAMMPS, Horovod, or MPI unless they are in scope. +Treat `package.channels` as authoritative when it is non-empty and preserve its +order by rendering one `-c` option per entry. This includes selected mirrors and +pre-release channels such as `conda-forge/label/deepmd-kit_dev` or +`conda-forge/label/deepmd-kit_rc`. Use `conda-forge` only when the list is empty. + +```bash +"" create -n "" \ + -c "" -c "" deepmd-kit +``` + +For an empty `package.channels` list, render the stable default explicitly: ```bash "" create -n "" \ diff --git a/skills/deepmd-install/references/failure-modes.md b/skills/deepmd-install/references/failure-modes.md index ef987b0c24..36b247ea2f 100644 --- a/skills/deepmd-install/references/failure-modes.md +++ b/skills/deepmd-install/references/failure-modes.md @@ -62,6 +62,12 @@ validate it, and render the gate again. A command containing an unassigned plan variable, an empty required argument, or `` is not executable. +For conda, compare every rendered `-c` argument with `package.channels`. An +empty list uses only the stable `conda-forge` default; a non-empty list replaces +that default and preserves the recorded order. For JAX C/C++, two null +TensorFlow roots select the Python-library route. A single root selects its +corresponding external library route; setting both is ambiguous and invalid. + ## Wrong skill root Helper scripts belong to the installed skill, not the DeePMD-kit checkout or diff --git a/skills/deepmd-install/references/plan-schema.md b/skills/deepmd-install/references/plan-schema.md index fe15657427..422d9f38c8 100644 --- a/skills/deepmd-install/references/plan-schema.md +++ b/skills/deepmd-install/references/plan-schema.md @@ -94,6 +94,9 @@ Use `null` for an inapplicable object. Do not add undeclared keys. - Keep the two DeePMD-kit index fields null for the default package index. Record a user-selected mirror or the documented pre-release index explicitly. - Keep `backend_index_url` null when the default package index is intended. +- For conda, an empty `channels` list selects the stable `conda-forge` + default. A non-empty list is authoritative and is rendered in order, with + one `-c` option per channel. Keep this list empty for other methods. - Package indexes and download URLs use HTTPS. - Require exactly one of HTTPS `artifact_url` or absolute `artifact_path`, plus `sha256`, for `offline`. @@ -149,8 +152,16 @@ entirely by its Python package. Require `cuda_home` for a CUDA build and Use a dedicated install prefix. Never choose `/`, a home directory, a conda prefix, `/usr`, `/usr/local`, or `$HOME/.local` as a disposable prefix. -For a JAX C++ backend, provide either `tensorflow_root` or -`tensorflow_c_root`. For Paddle C++, provide `paddle_inference_dir`. +For a JAX C++ backend, the TensorFlow dependency is selected by the two +nullable roots: + +- Keep both null to use the TensorFlow C++ libraries from the selected Python + environment with `USE_TF_PYTHON_LIBS=ON`. +- Set only `tensorflow_root` to use an external TensorFlow C++ installation. +- Set only `tensorflow_c_root` to use the TensorFlow C library. + +The two TensorFlow roots are mutually exclusive. For Paddle C++, provide +`paddle_inference_dir`. ### `lammps` diff --git a/skills/deepmd-install/references/source-cpp.md b/skills/deepmd-install/references/source-cpp.md index c6d0d37d99..3b643974f3 100644 --- a/skills/deepmd-install/references/source-cpp.md +++ b/skills/deepmd-install/references/source-cpp.md @@ -35,15 +35,21 @@ build directory. Never delete or recursively replace the install prefix. Start with all backends disabled, then enable the selected backend and its documented C API dependency. Enabling TensorFlow also enables the JAX C API in -DeePMD-kit by design. - -| Backend | Required CMake arguments | -| ----------------------- | ----------------------------------------------------------------------------------- | -| PyTorch | `-DENABLE_PYTORCH=ON -DUSE_PT_PYTHON_LIBS=ON` plus the PyTorch CMake prefix | -| TensorFlow | `-DENABLE_TENSORFLOW=ON -DUSE_TF_PYTHON_LIBS=ON` and the selected Python executable | -| JAX with TensorFlow C++ | `-DENABLE_TENSORFLOW=ON -DUSE_TF_PYTHON_LIBS=ON` | -| JAX with TensorFlow C | `-DENABLE_JAX=ON -DCMAKE_PREFIX_PATH=` | -| Paddle | `-DENABLE_PADDLE=ON -DPADDLE_INFERENCE_DIR=` | +DeePMD-kit by design. For JAX, select exactly one route from the state of the +nullable roots in the plan. + +| Backend dependency | Plan state | Required CMake arguments | +| ---------------------------------------- | ------------------------------- | ------------------------------------------------------------------------------------- | +| PyTorch | Not applicable | `-DENABLE_PYTORCH=ON -DUSE_PT_PYTHON_LIBS=ON` plus the PyTorch CMake prefix | +| TensorFlow | Not applicable | `-DENABLE_TENSORFLOW=ON -DUSE_TF_PYTHON_LIBS=ON` and the selected Python executable | +| JAX with Python TensorFlow C++ libraries | Both TensorFlow roots are null | `-DENABLE_TENSORFLOW=ON -DUSE_TF_PYTHON_LIBS=ON` and the selected Python executable | +| JAX with external TensorFlow C++ | Only `tensorflow_root` is set | `-DENABLE_TENSORFLOW=ON -DUSE_TF_PYTHON_LIBS=OFF -DTENSORFLOW_ROOT=` | +| JAX with TensorFlow C | Only `tensorflow_c_root` is set | `-DENABLE_JAX=ON -DCMAKE_PREFIX_PATH=` | +| Paddle | Not applicable | `-DENABLE_PADDLE=ON -DPADDLE_INFERENCE_DIR=` | + +The Python TensorFlow C++ route requires the planned interpreter to import a +TensorFlow package that provides `libtensorflow_cc`. Record and install that +package during the Python gate before configuring CMake. For PyTorch, discover the prefix from the planned interpreter in the same shell call that configures CMake: diff --git a/skills/deepmd-install/scripts/validate_plan.py b/skills/deepmd-install/scripts/validate_plan.py index 5b5d5221e6..10f6a7df60 100644 --- a/skills/deepmd-install/scripts/validate_plan.py +++ b/skills/deepmd-install/scripts/validate_plan.py @@ -245,6 +245,9 @@ def _validate_package( for key in ("backend_packages", "channels"): if key in package: _validate_string_list(package[key], f"package.{key}", errors) + channels = package.get("channels") + if method != "conda" and isinstance(channels, list) and channels: + errors.append("package.channels: non-empty channels require method 'conda'") for key in ("install_lammps", "install_ipi"): if key in package and not isinstance(package[key], bool): errors.append(f"package.{key}: expected a boolean") @@ -376,6 +379,9 @@ def _validate_cpp( for key in ("install_prefix", "build_directory"): if not _is_absolute_path(cpp.get(key)): errors.append(f"cpp.{key}: expected an absolute path") + for key in ("tensorflow_root", "tensorflow_c_root", "paddle_inference_dir"): + if cpp.get(key) is not None and not _is_absolute_path(cpp.get(key)): + errors.append(f"cpp.{key}: expected an absolute path or null") if not all( _is_absolute_path(cpp.get(key)) for key in ("install_prefix", "build_directory") ) or not _is_absolute_path(source.get("directory")): @@ -397,12 +403,11 @@ def _validate_cpp( forbidden.add(_canonical(environment_prefix)) if prefix in forbidden: errors.append("cpp.install_prefix: select a dedicated, non-shared prefix") - if backend == "jax" and not ( - _is_absolute_path(cpp.get("tensorflow_root")) - or _is_absolute_path(cpp.get("tensorflow_c_root")) + if backend == "jax" and all( + cpp.get(key) is not None for key in ("tensorflow_root", "tensorflow_c_root") ): errors.append( - "cpp: JAX requires tensorflow_root or tensorflow_c_root for the C API" + "cpp: JAX tensorflow_root and tensorflow_c_root are mutually exclusive" ) if backend == "paddle" and not _is_absolute_path(cpp.get("paddle_inference_dir")): errors.append("cpp.paddle_inference_dir: Paddle C++ requires an absolute path") diff --git a/source/tests/test_deepmd_install_skill.py b/source/tests/test_deepmd_install_skill.py index dcd4de3ed3..9692790f66 100644 --- a/source/tests/test_deepmd_install_skill.py +++ b/source/tests/test_deepmd_install_skill.py @@ -214,12 +214,51 @@ def test_validate_source_tensorflow_cpu_plan() -> None: assert PLAN.validate_plan(plan) == [] +def test_validate_jax_cpp_with_tensorflow_python_libraries() -> None: + """Allow JAX C++ to use TensorFlow libraries from the Python environment.""" + plan = _source_plan() + plan["goal"] = "python+cpp" + plan["backend"] = "jax" + plan["lammps"] = None + assert PLAN.validate_plan(plan) == [] + + +def test_validate_jax_cpp_rejects_ambiguous_tensorflow_roots() -> None: + """Require one unambiguous TensorFlow dependency route for JAX C++.""" + plan = _source_plan() + plan["goal"] = "python+cpp" + plan["backend"] = "jax" + plan["lammps"] = None + cpp = plan["cpp"] + assert isinstance(cpp, dict) + cpp["tensorflow_root"] = "/opt/tensorflow-cpp" + cpp["tensorflow_c_root"] = "/opt/tensorflow-c" + errors = PLAN.validate_plan(plan) + assert any("mutually exclusive" in error for error in errors) + + @pytest.mark.parametrize("method", ["pip", "conda", "offline", "docker"]) def test_validate_easy_install_plans(method: str) -> None: """Accept the method-specific fields for each easy-install path.""" assert PLAN.validate_plan(_easy_plan(method)) == [] +def test_validate_conda_channels_are_method_specific() -> None: + """Accept selected conda channels without ignoring them for other methods.""" + plan = _easy_plan("conda") + package = plan["package"] + assert isinstance(package, dict) + package["channels"] = ["conda-forge/label/deepmd-kit_rc", "conda-forge"] + assert PLAN.validate_plan(plan) == [] + + pip_plan = _easy_plan("pip") + pip_package = pip_plan["package"] + assert isinstance(pip_package, dict) + pip_package["channels"] = ["conda-forge"] + errors = PLAN.validate_plan(pip_plan) + assert "package.channels: non-empty channels require method 'conda'" in errors + + def test_validate_offline_plan_requires_checksum() -> None: """Reject an offline artifact without an integrity value.""" plan = _easy_plan("offline") @@ -841,6 +880,16 @@ def test_docker_reference_uses_backend_aware_verifier() -> None: assert "readonly" in reference +def test_conda_reference_renders_planned_channels() -> None: + """Keep selected conda channels authoritative with a stable default.""" + reference = ( + REPOSITORY_ROOT / "skills" / "deepmd-install" / "references" / "easy-install.md" + ).read_text(encoding="utf-8") + assert "package.channels" in reference + assert '-c "" -c ""' in reference + assert "Use `conda-forge` only when the list is empty" in reference + + def test_verify_lammps_dpa4c_cli(tmp_path: Path) -> None: """Accept the exact DPA4C host and Kokkos pair styles.""" binary = tmp_path / "lmp_dpa4c" From 0c1e3778f09df1d6640e8a3d8904a306b79c0119 Mon Sep 17 00:00:00 2001 From: OutisLi Date: Tue, 18 Aug 2026 15:15:07 +0800 Subject: [PATCH 06/11] fix: preserve easy-install plan identity --- .../deepmd-install/references/easy-install.md | 42 +++++++++----- .../references/failure-modes.md | 23 +++++--- .../deepmd-install/references/plan-schema.md | 7 ++- .../references/source-python.md | 7 ++- .../deepmd-install/scripts/validate_plan.py | 15 +++++ source/tests/test_deepmd_install_skill.py | 55 ++++++++++++++++++- 6 files changed, 122 insertions(+), 27 deletions(-) diff --git a/skills/deepmd-install/references/easy-install.md b/skills/deepmd-install/references/easy-install.md index 104e650f78..f970be43a6 100644 --- a/skills/deepmd-install/references/easy-install.md +++ b/skills/deepmd-install/references/easy-install.md @@ -77,17 +77,25 @@ Treat `package.channels` as authoritative when it is non-empty and preserve its order by rendering one `-c` option per entry. This includes selected mirrors and pre-release channels such as `conda-forge/label/deepmd-kit_dev` or `conda-forge/label/deepmd-kit_rc`. Use `conda-forge` only when the list is empty. +Pass `--override-channels` so configured defaults cannot supply packages outside +the plan. + +Render the DeePMD-kit package argument as `deepmd-kit=` when +`package.deepmd_version` is non-null, or as `deepmd-kit` when it is null. The +following non-null example preserves the recorded channel order: ```bash "" create -n "" \ - -c "" -c "" deepmd-kit + --override-channels \ + -c "" -c "" "deepmd-kit=" ``` -For an empty `package.channels` list, render the stable default explicitly: +For null `package.deepmd_version` and an empty `package.channels` list, render +the stable unversioned default explicitly: ```bash "" create -n "" \ - -c conda-forge deepmd-kit + --override-channels -c conda-forge deepmd-kit ``` Add `lammps` only for `goal=python+lammps`. Resolve the created interpreter @@ -95,11 +103,11 @@ without assuming shell activation: ```bash "" run -n "" \ - python -c "import sys; print(sys.executable)" + python -c "import sys; print(sys.executable); print(sys.prefix)" ``` -Record that absolute interpreter in the plan before running the Python verifier. -Follow the conda-forge CUDA guidance referenced by +Record both absolute paths in the plan, re-run `validate_plan.py`, and then run +the Python verifier. Follow the conda-forge CUDA guidance referenced by `doc/install/easy-install.md`; do not invent a `cudatoolkit` pin. ## dp1s @@ -108,12 +116,18 @@ Show the exact command and obtain confirmation before piping a remote response to a shell: ```bash -curl -fsSL https://dp1s.deepmodeling.com | bash +curl -fsSL https://dp1s.deepmodeling.com | env \ + DP1S_HOME="" \ + DP1S_NO_PATH_UPDATE=1 \ + DEEPMD_VERSION="" \ + bash ``` -Apply only options selected from the official `dp1s` documentation. After the -installer reports its prefix, resolve the installed Python and update the plan -before verification. +Omit the `DEEPMD_VERSION` assignment when `package.deepmd_version` is null. +`DP1S_NO_PATH_UPDATE` prevents the installer from editing shell startup files. +Apply only additional options selected from the official `dp1s` documentation. +After installation, resolve the absolute interpreter from the installed `dp` +entry point, record it in the plan, and re-run validation before verification. ## Offline package @@ -133,9 +147,11 @@ printf '%s %s\n' "" "" | sha256sum --check - bash "" ``` -Use `shasum -a 256` on systems without `sha256sum`. When a release is split, -verify each part before concatenating it into a new file. Never execute an HTML -error page or an artifact whose checksum is unknown. +Use `shasum -a 256` on systems without `sha256sum`. This workflow accepts one +complete installer only. For a split release, follow the version-matched manual +instructions to produce an already-assembled local installer, and use this +workflow only when a trusted final SHA-256 is available for that complete file. +Never execute an HTML error page or an artifact whose checksum is unknown. ## Docker diff --git a/skills/deepmd-install/references/failure-modes.md b/skills/deepmd-install/references/failure-modes.md index 36b247ea2f..45fa4682f5 100644 --- a/skills/deepmd-install/references/failure-modes.md +++ b/skills/deepmd-install/references/failure-modes.md @@ -62,11 +62,16 @@ validate it, and render the gate again. A command containing an unassigned plan variable, an empty required argument, or `` is not executable. -For conda, compare every rendered `-c` argument with `package.channels`. An -empty list uses only the stable `conda-forge` default; a non-empty list replaces -that default and preserves the recorded order. For JAX C/C++, two null -TensorFlow roots select the Python-library route. A single root selects its -corresponding external library route; setting both is ambiguous and invalid. +For conda, require `--override-channels` and compare every rendered `-c` +argument with `package.channels`. An empty list uses only the stable +`conda-forge` default; a non-empty list replaces that default and preserves the +recorded order. A non-null `package.deepmd_version` produces the exact +`deepmd-kit=` package constraint. + +For `dp1s`, compare `DP1S_HOME` and the optional `DEEPMD_VERSION` with the plan, +and require `DP1S_NO_PATH_UPDATE=1`. For JAX C/C++, two null TensorFlow roots +select the Python-library route. A single root selects its corresponding +external library route; setting both is ambiguous and invalid. ## Wrong skill root @@ -325,9 +330,11 @@ file "" printf '%s %s\n' "" "" | sha256sum --check - ``` -An absent checksum, HTML response, truncated split archive, unexpected -top-level directory, or checksum mismatch blocks extraction. Obtain a fresh -artifact and trusted checksum for the planned URL; do not disable verification. +An absent checksum, HTML response, unexpected top-level directory, or checksum +mismatch blocks extraction. Obtain a fresh artifact and trusted checksum for +the planned URL; do not disable verification. Split parts are not valid plan +artifacts; supply one already-assembled installer with a trusted checksum for +the complete file. For `package.artifact_path`, skip curl and run the same file-type and checksum checks directly on the absolute local path. diff --git a/skills/deepmd-install/references/plan-schema.md b/skills/deepmd-install/references/plan-schema.md index 422d9f38c8..d722260317 100644 --- a/skills/deepmd-install/references/plan-schema.md +++ b/skills/deepmd-install/references/plan-schema.md @@ -63,7 +63,9 @@ Use `null` for an inapplicable object. Do not add undeclared keys. environment and before installing DeePMD-kit. - `manager` and `name`: required for `conda`; `manager` is the absolute `conda` or `mamba` executable. -- `prefix`: required for `venv`, `prefix`, and offline installations. +- `prefix`: required for `pip`, `source`, `venv`, `prefix`, `dp1s`, and offline + installations. For conda, resolve and record both `sys.executable` and + `sys.prefix` after creating the environment and before verification. ### `package` @@ -100,6 +102,9 @@ Use `null` for an inapplicable object. Do not add undeclared keys. - Package indexes and download URLs use HTTPS. - Require exactly one of HTTPS `artifact_url` or absolute `artifact_path`, plus `sha256`, for `offline`. +- An offline plan represents one complete installer. `artifact_path` must be an + existing assembled file with a trusted SHA-256 for that complete file. Split + release parts and per-part checksums are outside this plan contract. - Require an immutable image reference or user-selected tag for `docker`. - Require `lammps_model_family` for packaged LAMMPS verification. diff --git a/skills/deepmd-install/references/source-python.md b/skills/deepmd-install/references/source-python.md index 9dd340639e..1cc6adf561 100644 --- a/skills/deepmd-install/references/source-python.md +++ b/skills/deepmd-install/references/source-python.md @@ -52,10 +52,13 @@ not reset, clean, or repoint `origin`. ## Interpreter and compiler gate -Use the plan's absolute interpreter and confirm Python 3.10 or newer: +Use the plan's absolute interpreter and confirm Python 3.10 or newer. Record +the resolved `sys.prefix` as `environment.prefix` and revalidate the plan if it +differs from the recorded value: ```bash -"" -c "import sys; print(sys.executable); print(sys.version)" +"" -c \ + "import sys; print(sys.executable); print(sys.prefix); print(sys.version)" "" --version "" --version ``` diff --git a/skills/deepmd-install/scripts/validate_plan.py b/skills/deepmd-install/scripts/validate_plan.py index 10f6a7df60..d9cb5e2b62 100644 --- a/skills/deepmd-install/scripts/validate_plan.py +++ b/skills/deepmd-install/scripts/validate_plan.py @@ -216,6 +216,10 @@ def _validate_environment( errors.append( "environment.python: pip and source methods require an absolute executable" ) + if not _is_absolute_path(environment.get("prefix")): + errors.append( + "environment.prefix: pip and source methods require an absolute prefix" + ) if method == "conda": if not _is_absolute_path(environment.get("manager")): errors.append("environment.manager: conda requires an absolute executable") @@ -223,6 +227,13 @@ def _validate_environment( errors.append( "environment.name: conda requires a non-empty environment name" ) + for key in ("python", "prefix"): + if environment.get(key) is not None and not _is_absolute_path( + environment.get(key) + ): + errors.append( + f"environment.{key}: conda requires an absolute path or null" + ) if kind in {"venv", "prefix"} and not _is_absolute_path(environment.get("prefix")): errors.append(f"environment.prefix: {kind} requires an absolute path") if method in {"dp1s", "offline"} and not _is_absolute_path( @@ -276,6 +287,10 @@ def _validate_package( _validate_https_url(artifact_url, "package.artifact_url", errors) elif not _is_absolute_path(artifact_path): errors.append("package.artifact_path: expected an absolute path") + elif not Path(artifact_path).is_file(): + errors.append( + "package.artifact_path: expected an existing complete artifact file" + ) if package.get("sha256") is None: errors.append("package.sha256: offline installation requires a checksum") elif ( diff --git a/source/tests/test_deepmd_install_skill.py b/source/tests/test_deepmd_install_skill.py index 9692790f66..85ba12c5dd 100644 --- a/source/tests/test_deepmd_install_skill.py +++ b/source/tests/test_deepmd_install_skill.py @@ -243,6 +243,19 @@ def test_validate_easy_install_plans(method: str) -> None: assert PLAN.validate_plan(_easy_plan(method)) == [] +@pytest.mark.parametrize("method", ["pip", "source"]) +def test_validate_interpreter_methods_require_prefix(method: str) -> None: + """Require a planned interpreter prefix before pip or source execution.""" + plan = _easy_plan("pip") if method == "pip" else _source_plan() + environment = plan["environment"] + assert isinstance(environment, dict) + environment["prefix"] = None + errors = PLAN.validate_plan(plan) + assert any( + "pip and source methods require an absolute prefix" in error for error in errors + ) + + def test_validate_conda_channels_are_method_specific() -> None: """Accept selected conda channels without ignoring them for other methods.""" plan = _easy_plan("conda") @@ -259,6 +272,17 @@ def test_validate_conda_channels_are_method_specific() -> None: assert "package.channels: non-empty channels require method 'conda'" in errors +@pytest.mark.parametrize("field", ["python", "prefix"]) +def test_validate_conda_resolved_paths_must_be_absolute(field: str) -> None: + """Reject a relative Conda interpreter or prefix after environment creation.""" + plan = _easy_plan("conda") + environment = plan["environment"] + assert isinstance(environment, dict) + environment[field] = "relative/path" + errors = PLAN.validate_plan(plan) + assert f"environment.{field}: conda requires an absolute path or null" in errors + + def test_validate_offline_plan_requires_checksum() -> None: """Reject an offline artifact without an integrity value.""" plan = _easy_plan("offline") @@ -269,16 +293,29 @@ def test_validate_offline_plan_requires_checksum() -> None: assert "package.sha256: offline installation requires a checksum" in errors -def test_validate_offline_local_artifact() -> None: +def test_validate_offline_local_artifact(tmp_path: Path) -> None: """Accept a local offline artifact only through its dedicated path field.""" + artifact = tmp_path / "deepmd.sh" + artifact.write_text("installer", encoding="utf-8") plan = _easy_plan("offline") package = plan["package"] assert isinstance(package, dict) package["artifact_url"] = None - package["artifact_path"] = "/opt/artifacts/deepmd.sh" + package["artifact_path"] = str(artifact) assert PLAN.validate_plan(plan) == [] +def test_validate_offline_local_artifact_must_exist(tmp_path: Path) -> None: + """Reject a missing file in the single-artifact offline contract.""" + plan = _easy_plan("offline") + package = plan["package"] + assert isinstance(package, dict) + package["artifact_url"] = None + package["artifact_path"] = str(tmp_path / "missing-deepmd.sh") + errors = PLAN.validate_plan(plan) + assert any("existing complete artifact file" in error for error in errors) + + def test_validate_offline_artifact_url_requires_https() -> None: """Reject a local path routed through the offline curl branch.""" plan = _easy_plan("offline") @@ -887,7 +924,19 @@ def test_conda_reference_renders_planned_channels() -> None: ).read_text(encoding="utf-8") assert "package.channels" in reference assert '-c "" -c ""' in reference - assert "Use `conda-forge` only when the list is empty" in reference + assert reference.count("--override-channels") >= 2 + assert '"deepmd-kit="' in reference + assert "print(sys.executable); print(sys.prefix)" in reference + + +def test_dp1s_reference_preserves_planned_identity() -> None: + """Pass the planned dp1s prefix and version without editing shell startup.""" + reference = ( + REPOSITORY_ROOT / "skills" / "deepmd-install" / "references" / "easy-install.md" + ).read_text(encoding="utf-8") + assert 'DP1S_HOME=""' in reference + assert 'DEEPMD_VERSION=""' in reference + assert "DP1S_NO_PATH_UPDATE=1" in reference def test_verify_lammps_dpa4c_cli(tmp_path: Path) -> None: From 3cdeb630b51946016b856d4b60d765e93c971586 Mon Sep 17 00:00:00 2001 From: OutisLi Date: Tue, 18 Aug 2026 20:52:53 +0800 Subject: [PATCH 07/11] fix: separate dp1s home from Python prefix --- .../deepmd-install/references/easy-install.md | 9 ++-- .../references/failure-modes.md | 13 +++-- .../deepmd-install/references/plan-schema.md | 15 ++++-- .../deepmd-install/scripts/validate_plan.py | 36 ++++++++++--- source/tests/test_deepmd_install_skill.py | 51 ++++++++++++++++++- 5 files changed, 105 insertions(+), 19 deletions(-) diff --git a/skills/deepmd-install/references/easy-install.md b/skills/deepmd-install/references/easy-install.md index f970be43a6..e576f81460 100644 --- a/skills/deepmd-install/references/easy-install.md +++ b/skills/deepmd-install/references/easy-install.md @@ -117,7 +117,7 @@ to a shell: ```bash curl -fsSL https://dp1s.deepmodeling.com | env \ - DP1S_HOME="" \ + DP1S_HOME="" \ DP1S_NO_PATH_UPDATE=1 \ DEEPMD_VERSION="" \ bash @@ -126,8 +126,11 @@ curl -fsSL https://dp1s.deepmodeling.com | env \ Omit the `DEEPMD_VERSION` assignment when `package.deepmd_version` is null. `DP1S_NO_PATH_UPDATE` prevents the installer from editing shell startup files. Apply only additional options selected from the official `dp1s` documentation. -After installation, resolve the absolute interpreter from the installed `dp` -entry point, record it in the plan, and re-run validation before verification. +After installation, keep `environment.dp1s_home` unchanged. Resolve the absolute +interpreter from the installed `dp` entry point, run that interpreter to obtain +its `sys.prefix`, record both as `environment.python` and `environment.prefix`, +and re-run validation before verification. The Python prefix normally differs +from `dp1s_home`; pass only `environment.prefix` to `--expected-prefix`. ## Offline package diff --git a/skills/deepmd-install/references/failure-modes.md b/skills/deepmd-install/references/failure-modes.md index 45fa4682f5..642c64c2cb 100644 --- a/skills/deepmd-install/references/failure-modes.md +++ b/skills/deepmd-install/references/failure-modes.md @@ -68,10 +68,15 @@ argument with `package.channels`. An empty list uses only the stable recorded order. A non-null `package.deepmd_version` produces the exact `deepmd-kit=` package constraint. -For `dp1s`, compare `DP1S_HOME` and the optional `DEEPMD_VERSION` with the plan, -and require `DP1S_NO_PATH_UPDATE=1`. For JAX C/C++, two null TensorFlow roots -select the Python-library route. A single root selects its corresponding -external library route; setting both is ambiguous and invalid. +For `dp1s`, compare `DP1S_HOME` with `environment.dp1s_home`, compare the +optional `DEEPMD_VERSION` with the plan, and require +`DP1S_NO_PATH_UPDATE=1`. Preserve `dp1s_home` after installation; record the +installed interpreter's distinct `sys.prefix` as `environment.prefix` for the +Python identity gate. + +For JAX C/C++, two null TensorFlow roots select the Python-library route. A +single root selects its corresponding external library route; setting both is +ambiguous and invalid. ## Wrong skill root diff --git a/skills/deepmd-install/references/plan-schema.md b/skills/deepmd-install/references/plan-schema.md index d722260317..1f4451cd8a 100644 --- a/skills/deepmd-install/references/plan-schema.md +++ b/skills/deepmd-install/references/plan-schema.md @@ -54,7 +54,8 @@ Use `null` for an inapplicable object. Do not add undeclared keys. "python": "/absolute/path/to/python", "manager": null, "name": null, - "prefix": "/absolute/environment/prefix" + "prefix": "/absolute/environment/prefix", + "dp1s_home": null } ``` @@ -63,9 +64,13 @@ Use `null` for an inapplicable object. Do not add undeclared keys. environment and before installing DeePMD-kit. - `manager` and `name`: required for `conda`; `manager` is the absolute `conda` or `mamba` executable. -- `prefix`: required for `pip`, `source`, `venv`, `prefix`, `dp1s`, and offline +- `prefix`: required for `pip`, `source`, `venv`, `prefix`, and offline installations. For conda, resolve and record both `sys.executable` and `sys.prefix` after creating the environment and before verification. +- `dp1s_home`: required only for `dp1s`; it is the Pixi and exposed-binary root + passed as `DP1S_HOME`, not the installed Python prefix. Before installation, + keep `python` and `prefix` null. After installation, preserve `dp1s_home` and + record the resolved interpreter and its `sys.prefix` together. ### `package` @@ -241,7 +246,8 @@ The validator enforces these invariants: "python": "/opt/conda/envs/deepmd/bin/python", "manager": null, "name": null, - "prefix": "/opt/conda/envs/deepmd" + "prefix": "/opt/conda/envs/deepmd", + "dp1s_home": null }, "package": { "deepmd_version": null, @@ -289,7 +295,8 @@ before installing dependencies or building. "python": "/opt/conda/envs/deepmd/bin/python", "manager": null, "name": null, - "prefix": "/opt/conda/envs/deepmd" + "prefix": "/opt/conda/envs/deepmd", + "dp1s_home": null }, "package": { "deepmd_version": null, diff --git a/skills/deepmd-install/scripts/validate_plan.py b/skills/deepmd-install/scripts/validate_plan.py index d9cb5e2b62..c86457fbb4 100644 --- a/skills/deepmd-install/scripts/validate_plan.py +++ b/skills/deepmd-install/scripts/validate_plan.py @@ -43,7 +43,7 @@ "smoke_test", } OBJECT_KEYS = { - "environment": {"kind", "python", "manager", "name", "prefix"}, + "environment": {"kind", "python", "manager", "name", "prefix", "dp1s_home"}, "package": { "deepmd_version", "deepmd_index_url", @@ -234,12 +234,36 @@ def _validate_environment( errors.append( f"environment.{key}: conda requires an absolute path or null" ) - if kind in {"venv", "prefix"} and not _is_absolute_path(environment.get("prefix")): - errors.append(f"environment.prefix: {kind} requires an absolute path") - if method in {"dp1s", "offline"} and not _is_absolute_path( - environment.get("prefix") + if method == "dp1s": + if not _is_absolute_path(environment.get("dp1s_home")): + errors.append("environment.dp1s_home: dp1s requires an absolute path") + python = environment.get("python") + prefix = environment.get("prefix") + unresolved = python is None and prefix is None + resolved = _is_absolute_path(python) and _is_absolute_path(prefix) + if not (unresolved or resolved): + errors.append( + "environment: dp1s python and prefix must both be null or absolute paths" + ) + if ( + resolved + and _is_absolute_path(environment.get("dp1s_home")) + and _canonical(environment["dp1s_home"]) == _canonical(prefix) + ): + errors.append( + "environment: dp1s_home and resolved Python prefix must be distinct" + ) + elif environment.get("dp1s_home") is not None: + errors.append("environment.dp1s_home: allowed only for method 'dp1s'") + dp1s_prefix_pending = method == "dp1s" and environment.get("prefix") is None + if ( + kind in {"venv", "prefix"} + and not dp1s_prefix_pending + and not _is_absolute_path(environment.get("prefix")) ): - errors.append(f"environment.prefix: {method} requires an absolute path") + errors.append(f"environment.prefix: {kind} requires an absolute path") + if method == "offline" and not _is_absolute_path(environment.get("prefix")): + errors.append("environment.prefix: offline requires an absolute path") if method == "docker" and kind != "container": errors.append("environment.kind: docker requires 'container'") diff --git a/source/tests/test_deepmd_install_skill.py b/source/tests/test_deepmd_install_skill.py index 85ba12c5dd..5a4fe3b865 100644 --- a/source/tests/test_deepmd_install_skill.py +++ b/source/tests/test_deepmd_install_skill.py @@ -55,6 +55,7 @@ def _source_plan() -> dict[str, object]: "manager": None, "name": None, "prefix": "/opt/deepmd", + "dp1s_home": None, }, "package": { "deepmd_version": None, @@ -117,6 +118,7 @@ def _easy_plan(method: str) -> dict[str, object]: "manager": None, "name": None, "prefix": "/opt/deepmd", + "dp1s_home": None, } package: dict[str, object] = { "deepmd_version": None, @@ -137,6 +139,15 @@ def _easy_plan(method: str) -> dict[str, object]: environment.update( {"kind": "conda", "python": None, "manager": "/opt/conda", "name": "deepmd"} ) + elif method == "dp1s": + environment.update( + { + "kind": "prefix", + "python": None, + "prefix": None, + "dp1s_home": "/opt/dp1s", + } + ) elif method == "offline": environment.update({"kind": "prefix", "python": None}) package.update( @@ -237,7 +248,7 @@ def test_validate_jax_cpp_rejects_ambiguous_tensorflow_roots() -> None: assert any("mutually exclusive" in error for error in errors) -@pytest.mark.parametrize("method", ["pip", "conda", "offline", "docker"]) +@pytest.mark.parametrize("method", ["pip", "conda", "dp1s", "offline", "docker"]) def test_validate_easy_install_plans(method: str) -> None: """Accept the method-specific fields for each easy-install path.""" assert PLAN.validate_plan(_easy_plan(method)) == [] @@ -272,6 +283,41 @@ def test_validate_conda_channels_are_method_specific() -> None: assert "package.channels: non-empty channels require method 'conda'" in errors +def test_validate_dp1s_lifecycle_keeps_home_and_prefix_distinct(tmp_path: Path) -> None: + """Accept unresolved and resolved dp1s identities without conflating paths.""" + plan = _easy_plan("dp1s") + environment = plan["environment"] + assert isinstance(environment, dict) + dp1s_home = tmp_path / "dp1s-home" + python_prefix = dp1s_home / "envs" / "dp1s" + environment["dp1s_home"] = str(dp1s_home) + assert PLAN.validate_plan(plan) == [] + + environment["python"] = str(python_prefix / "bin" / "python") + environment["prefix"] = str(python_prefix) + assert environment["dp1s_home"] != environment["prefix"] + assert PLAN.validate_plan(plan) == [] + + environment["prefix"] = environment["dp1s_home"] + errors = PLAN.validate_plan(plan) + assert any("must be distinct" in error for error in errors) + + environment["prefix"] = str(python_prefix) + environment["python"] = None + errors = PLAN.validate_plan(plan) + assert any("must both be null or absolute paths" in error for error in errors) + + +def test_validate_dp1s_home_is_method_specific() -> None: + """Reject a dp1s installer root that another method would ignore.""" + plan = _easy_plan("pip") + environment = plan["environment"] + assert isinstance(environment, dict) + environment["dp1s_home"] = "/opt/dp1s" + errors = PLAN.validate_plan(plan) + assert "environment.dp1s_home: allowed only for method 'dp1s'" in errors + + @pytest.mark.parametrize("field", ["python", "prefix"]) def test_validate_conda_resolved_paths_must_be_absolute(field: str) -> None: """Reject a relative Conda interpreter or prefix after environment creation.""" @@ -934,9 +980,10 @@ def test_dp1s_reference_preserves_planned_identity() -> None: reference = ( REPOSITORY_ROOT / "skills" / "deepmd-install" / "references" / "easy-install.md" ).read_text(encoding="utf-8") - assert 'DP1S_HOME=""' in reference + assert 'DP1S_HOME=""' in reference assert 'DEEPMD_VERSION=""' in reference assert "DP1S_NO_PATH_UPDATE=1" in reference + assert '--expected-prefix ""' in reference def test_verify_lammps_dpa4c_cli(tmp_path: Path) -> None: From 200dd38c740357125445b18440d96c0b3c2dce3c Mon Sep 17 00:00:00 2001 From: OutisLi Date: Tue, 18 Aug 2026 21:38:28 +0800 Subject: [PATCH 08/11] refactor: simplify DeePMD installation skill --- doc/agent-skills.md | 19 +- doc/install/install-with-agent.md | 24 +- pyproject.toml | 2 - skills/deepmd-install/SKILL.md | 315 +++-- .../deepmd-install/references/easy-install.md | 212 ---- .../references/failure-modes.md | 400 ++---- .../references/official-docs.md | 107 ++ .../deepmd-install/references/plan-schema.md | 356 ------ .../deepmd-install/references/source-cpp.md | 167 --- .../references/source-lammps.md | 241 ---- .../references/source-python.md | 196 --- .../deepmd-install/scripts/prepare_lammps.py | 169 --- skills/deepmd-install/scripts/probe_env.py | 426 ------- .../deepmd-install/scripts/validate_plan.py | 818 ------------- .../deepmd-install/scripts/verify_lammps.py | 188 --- .../deepmd-install/scripts/verify_native.py | 162 --- .../deepmd-install/scripts/verify_python.py | 488 -------- source/tests/test_deepmd_install_skill.py | 1075 ----------------- 18 files changed, 408 insertions(+), 4957 deletions(-) delete mode 100644 skills/deepmd-install/references/easy-install.md create mode 100644 skills/deepmd-install/references/official-docs.md delete mode 100644 skills/deepmd-install/references/plan-schema.md delete mode 100644 skills/deepmd-install/references/source-cpp.md delete mode 100644 skills/deepmd-install/references/source-lammps.md delete mode 100644 skills/deepmd-install/references/source-python.md delete mode 100644 skills/deepmd-install/scripts/prepare_lammps.py delete mode 100644 skills/deepmd-install/scripts/probe_env.py delete mode 100644 skills/deepmd-install/scripts/validate_plan.py delete mode 100644 skills/deepmd-install/scripts/verify_lammps.py delete mode 100644 skills/deepmd-install/scripts/verify_native.py delete mode 100644 skills/deepmd-install/scripts/verify_python.py delete mode 100644 source/tests/test_deepmd_install_skill.py diff --git a/doc/agent-skills.md b/doc/agent-skills.md index 7a9b0a3d4e..6e488a32de 100644 --- a/doc/agent-skills.md +++ b/doc/agent-skills.md @@ -14,14 +14,11 @@ in the DeePMD-kit repository under `skills/`. ## List of skills -- `deepmd-install`: Install DeePMD-kit for users. The skill probes the machine, - records the selected environment and build in a validated plan, then follows - either an easy path (conda, pip, Docker, offline, `dp1s`) or a source build of - the Python package, C/C++ interface, and LAMMPS. Kokkos builds use one - `Kokkos_ARCH_*` flag per binary. DPA4/SeZM uses `deepmd/kk`; DPA4C uses - `dpa4spin/kk` with a compact canonical graph artifact. Detailed recipes live - under `skills/deepmd-install/references/` and are loaded only for the selected - path or a matching failure mode. +- `deepmd-install`: Select a pip, conda, `dp1s`, offline, Docker, or source + installation path, load the official documentation matching the requested + version, and verify the requested Python, C/C++, or LAMMPS interface. A + compact failure-mode reference covers issues not resolved by the install + pages. - `deepmd-train`: Choose a DeePMD-kit model family, then train from scratch. The skill uses progressive disclosure: the top-level workflow handles common training steps and model selection, while model-specific configuration lives @@ -102,6 +99,6 @@ without launching an expensive calculation. For example: water dataset and draft a training input, but do not start training.” - “Use the `lammps-deepmd` skill to prepare an NVT LAMMPS input file for a DeePMD-kit model, and explain each command.” -- “Use the `deepmd-install` skill to plan a PyTorch CUDA source install, but - ask me for the CUDA toolkit path, PyTorch wheel, and install prefix first. - Do not start compiling.” +- “Use the `deepmd-install` skill to install the current stable release with + PyTorch and CUDA. Read the matching official documentation before choosing + packages, then verify a tensor operation on the GPU.” diff --git a/doc/install/install-with-agent.md b/doc/install/install-with-agent.md index 3c1ed4336c..f73fb6b49b 100644 --- a/doc/install/install-with-agent.md +++ b/doc/install/install-with-agent.md @@ -3,13 +3,13 @@ DeePMD-kit ships an official Agent Skill, `deepmd-install`, that walks an AI agent through installing the package. The skill covers easy methods (conda, pip, Docker, offline installers, `dp1s`) and source builds of the Python -package, the C++ interface, and LAMMPS with Kokkos (`pair_style deepmd/kk`, -used by DPA4/SeZM, and `pair_style dpa4spin/kk`, used by DPA4C). +package, the C++ interface, and LAMMPS. -The skill probes the machine, asks only for decisions required by the selected -installation path, and records concrete paths and versions in a validated -plan. It does not assume a host path, package mirror, CUDA toolkit, or backend -version. +The skill inspects the target machine, asks only for decisions required by the +selected installation path, reads the official documentation matching the +requested version, and verifies the requested interface. Version-specific +package and build commands remain in the installation documentation instead +of being duplicated in the skill. The full skill catalog is in [Agent Skills](../agent-skills.md). For a manual install, use [Easy install](easy-install.md) or @@ -23,8 +23,8 @@ commands below yourself. ## If you are an agent -Install `deepmd-install` first, then follow its probe, validated plan, and -gate workflow. +Install `deepmd-install` first, then load it and install DeePMD-kit with the +required backend, accelerator, and optional native interfaces. If this machine already has a DeePMD-kit checkout, run from the repository root: @@ -56,7 +56,7 @@ uses the detected agent. To target one product, add `--agent cursor`, `--agent claude-code`, or another supported agent name. Refresh or restart the session afterward so the skill is reloaded. -Then use the `deepmd-install` skill: probe the machine, ask only for missing -decisions, validate `install-plan.json`, and execute one gate at a time. A -CUDA LAMMPS build uses exactly one `Kokkos_ARCH_*` flag per binary and verifies -the pair styles required by the selected model family. +Then use the `deepmd-install` skill. The agent selects one installation method, +loads the version-matched official documentation, executes its commands in the +chosen environment, and verifies the requested Python, C/C++, or LAMMPS +interface. diff --git a/pyproject.toml b/pyproject.toml index 3a7edcb221..3b64d3ec1b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -493,8 +493,6 @@ runtime-evaluated-base-classes = ["torch.nn.Module"] "**/*.ipynb" = ["T20"] # printing in a nb file is expected # Example / demo scripts are run directly: top-level imports and prints are fine. "examples/**/*.py" = ["T20", "TID253"] -# Skill helper CLIs print probe and verification reports. -"skills/**/*.py" = ["T20"] [tool.pytest.ini_options] markers = "run" diff --git a/skills/deepmd-install/SKILL.md b/skills/deepmd-install/SKILL.md index 5fd84e15cc..330a6bab69 100644 --- a/skills/deepmd-install/SKILL.md +++ b/skills/deepmd-install/SKILL.md @@ -1,146 +1,205 @@ --- name: deepmd-install -description: Install or rebuild DeePMD-kit with conda, pip, dp1s, offline packages, Docker, or a source checkout. Use for CPU, NVIDIA CUDA, or ROCm environments; PyTorch, TensorFlow, JAX, or Paddle backends; backend-enabled C/C++ interfaces; and DeePMD-enabled LAMMPS, including Kokkos pair styles for DPA4 and DPA4C. +description: Install DeePMD-kit with pip, conda, dp1s, an offline package, Docker, or source code. Use for PyTorch, TensorFlow, JAX, or Paddle on CPU, CUDA, or ROCm, and for the C/C++ interface or DeePMD-enabled LAMMPS. --- # Install DeePMD-kit -Install the smallest runtime that satisfies the user's goal. Probe first, keep -all decisions in a validated plan, execute one self-contained gate at a time, -and verify the requested public interface rather than package presence alone. - -## Supported workflows - -| Path | Scope | -| ----------------------------------------------- | ---------------------------------------------------------------------------------------------- | -| conda, pip | Stable Python installs for PyTorch, TensorFlow, JAX, and Paddle; optional packaged host LAMMPS | -| `dp1s`, offline package, Docker | Official release artifacts selected by the user or matching documentation | -| source Python | PyTorch, TensorFlow, JAX, and Paddle with backend-specific dependencies and verification | -| source C/C++ | Backend-enabled C/C++ libraries installed into a dedicated prefix | -| source LAMMPS host | Built-in DeePMD module with exact pair-style verification | -| source LAMMPS Kokkos CUDA | PyTorch graph artifacts: `deepmd/kk` for DPA4/SeZM and `dpa4spin/kk` for DPA4C | -| ROCm source, Windows source, LAMMPS plugin mode | Follow the version-matched documentation in the selected checkout | - -Backend-neutral C/C++ libraries with `ALLOW_NO_BACKEND=ON` are outside the -automated plan. Follow `doc/install/install-from-source.md` in the selected -checkout for that layout. - -## Hard rules - -1. Resolve the absolute directory containing this `SKILL.md` as `SKILL_ROOT`. - Invoke every bundled script through that absolute path. -1. Run the read-only probe before choosing versions, paths, or accelerators. -1. Ask only for a missing value that changes the selected path. Do not ask - source, compiler, CUDA, or LAMMPS questions for an unrelated easy install. -1. Record all decisions in `install-plan.json` and validate it before an - install, checkout, compile, download, or environment modification. -1. Use the plan's absolute Python executable. Never substitute a bare - `python`, `pip`, or an assumed conda activation. -1. Make every shell call self-contained. Do not rely on an `export`, `cd`, or - `conda activate` from a previous agent tool call. -1. Render commands with concrete plan values. Stop if a plan placeholder, - empty required value, unexpected path, or shell variable not assigned - earlier in the same command block remains. Keep each plan string in the - quoted argument position shown by the selected reference. -1. Never run `git reset --hard`, `git clean`, recursively remove an install - prefix, or edit shell rc files as part of this workflow. Use a new build - directory or versioned install prefix instead. -1. Use documentation from the selected checkout for version-sensitive source - options. Do not apply `latest` documentation blindly to an older ref. -1. Confirm before piping a remote script to a shell. Use fail-on-HTTP-error - downloads and verify a checksum whenever the plan contains one. -1. Check GPU occupancy and bind the confirmed physical device explicitly - before a GPU smoke test. -1. On failure, stop the current gate and read - [`references/failure-modes.md`](references/failure-modes.md). Do not restart - the entire install or wipe caches. - -## Workflow - -### 0. Probe - -Run with an available Python 3.10+ interpreter: +Use the official documentation as the source of version-specific commands. +This skill selects the shortest suitable installation path, applies a small +set of safety rules, and verifies the requested runtime. It does not duplicate +the full installation manual. + +## 1. Establish the target + +Determine only the choices that affect installation: + +- DeePMD-kit release or Git ref; +- PyTorch, TensorFlow, JAX, or Paddle backend; +- CPU, NVIDIA CUDA, or ROCm runtime; +- Python only, packaged LAMMPS, C/C++, or source-built LAMMPS; +- existing environment or a new user-approved environment. + +If the user does not request a development version, prefer the current stable +release. When no method is specified, recommend a source install if the +machine has the required compiler and the user accepts the build time; +otherwise choose the shortest supported package path. Do not create a new +environment when the user requires an existing one. Ask only for a missing +choice that changes the installation path. + +Inspect the OS, architecture, Python version and prefix, package manager, and +requested accelerator before changing the machine. Resolve the selected +Python executable to an absolute path and use `"" -m pip`; +do not rely on a bare `pip` or shell activation persisting between commands. + +## 2. Read the matching documentation + +Assume no local DeePMD-kit checkout exists. Fetch the documentation before +rendering an install command: + +1. Use the agent's browser or web-fetch tool to open the direct official page, + not a search-result summary. + +1. For a release, open the Releases page, record its tag, and use the matching + versioned documentation. + +1. If the page is missing or no web-fetch tool is available, fetch the needed + Markdown file from that exact tag or commit: + + ```bash + curl -fsSL --retry 2 \ + "https://raw.githubusercontent.com/deepmodeling/deepmd-kit//doc/install/.md" + ``` + +1. If direct GitHub access fails, retry that same public URL through the + documented `gh-proxy.com` form. Reject an HTTP failure, empty response, or + HTML error page. + +Read [`references/official-docs.md`](references/official-docs.md) for the URL +map, version-selection rules, exact repository paths, and network fallback. +Use the docs matching the selected version, not `latest` for an older release. + +The commands below are route summaries. Before execution, replace every +placeholder with an observed or user-selected value and apply any changed +requirements from the matching official page. + +## 3. Choose one installation path + +### pip + +Use pip for a released Python package and optional packaged LAMMPS. Start from +the backend tab in the official easy-install page: + +| Backend | Minimal package shape | +| ---------- | ------------------------------------------------------------------------------------------ | +| PyTorch | Install the selected PyTorch build, then `deepmd-kit` or `deepmd-kit[torch]` as documented | +| TensorFlow | `deepmd-kit[cpu]` for CPU or the documented GPU/CUDA extras | +| JAX | `deepmd-kit[jax]`, plus the documented JAX accelerator package | +| Paddle | Install the documented Paddle package first, then `deepmd-kit` | + +Add `lmp` or `ipi` to the extras only when requested and supported by the +selected backend and platform. Install through the absolute target Python: + +```bash +"" -m pip install "" +``` + +### conda + +Use conda-forge for a released package when the user prefers conda: + +```bash +"" create -n "" \ + -c conda-forge deepmd-kit +``` + +Add `lammps` or distributed-training packages only when requested. Follow the +linked conda-forge CUDA guidance rather than inventing a toolkit pin. After +creation, resolve the environment's absolute Python before verification. + +### dp1s + +Use the official one-second installer when the user selects it. Show the +remote script command and obtain confirmation before piping it to a shell: + +```bash +curl -fsSL https://dp1s.deepmodeling.com | bash +``` + +Read the dp1s repository for `DP1S_HOME`, version selection, release-candidate, +and PATH-update options. Resolve the installed `dp` entry point and its Python +prefix instead of assuming that `DP1S_HOME` is the Python environment. + +### Offline package + +Use the exact asset for the selected release, OS, architecture, and runtime +from the official GitHub Releases page. Follow the release instructions to +assemble split files, verify the published checksum when available, and run +the completed installer. Never execute a partial download, an HTML response, +or an asset selected only by a similar filename. + +### Docker + +Pull the exact official image tag selected from the package page. For current +official images, the DeePMD environment is under `/opt/deepmd-kit`; confirm the +selected image's absolute `sys.executable` and `sys.prefix` before using it. +The minimal CPU check is: + +```bash +docker pull "" +docker run --rm --entrypoint /opt/deepmd-kit/bin/python \ + "" -c \ + "import sys, deepmd; print(sys.executable, sys.prefix, deepmd.__version__)" +``` + +Use a different absolute interpreter only when the selected image definition +documents it. For packaged LAMMPS, run the binary inside the same container: ```bash -"" "/scripts/probe_env.py" --json +docker run --rm --entrypoint /opt/deepmd-kit/bin/lmp \ + "" -h ``` -Use the report to identify the OS, libc, Python environments, compilers, -toolkits, driver, GPU compute capabilities and visibility mask, disk/RAM, and -existing DeePMD-kit/PyTorch installs. The probe does not select versions. +A host-side `lmp` does not verify the image. Mount inputs read-only, and add +explicit GPU device selection for CUDA. -### 1. Select one path +### Source Python (recommended) -- Prefer an easy method for a stable release without local source changes, - custom C/C++ libraries, or Kokkos device pair styles. -- Use source Python for a branch/fork, unreleased feature, local toolkit build, - or customized OPs. -- Add source C/C++ only when a C/C++ client, source-built LAMMPS, or i-PI needs - it. -- Add source LAMMPS only after the C/C++ gate passes. +Use source installation for a reproducible build from a selected stable tag, +an unreleased feature, a custom build, or ROCm. Reject a remote or ref beginning +with `-`, then clone and resolve the selected ref safely: -### 2. Create and validate the plan +```bash +git clone --no-checkout -- \ + https://github.com/deepmodeling/deepmd-kit.git "" +git -C "" fetch --tags -- origin "" +git -C "" checkout --detach FETCH_HEAD +git -C "" rev-parse HEAD +``` -Read [`references/plan-schema.md`](references/plan-schema.md), write the plan to -a stable scratch path outside the source/build trees, and validate it: +Read `doc/install/install-from-source.md` at that exact commit, install the +selected backend first, and apply only the documented build variables. The +basic Python build is: ```bash -"" "/scripts/validate_plan.py" \ - "" +"" -m pip install "" ``` -Present a concise summary only when the request or probe leaves a meaningful -choice unresolved. If the user already specified the method, backend, version, -and target, proceed without asking them to reconfirm their own request. - -### 3. Execute the selected references - -| Selection | Read | -| ----------------------------------- | ------------------------------------------------------------ | -| conda, pip, `dp1s`, offline, Docker | [`references/easy-install.md`](references/easy-install.md) | -| source Python | [`references/source-python.md`](references/source-python.md) | -| source C/C++ | [`references/source-cpp.md`](references/source-cpp.md) | -| source LAMMPS | [`references/source-lammps.md`](references/source-lammps.md) | -| failed gate | [`references/failure-modes.md`](references/failure-modes.md) | - -Read only the references required by the plan. Within each gate, render one -command block with absolute values so that directory and environment state -cannot leak across calls. - -### 4. Enforce the gates - -| Gate | Required evidence | -| ----------- | --------------------------------------------------------------------------------------------------------------------------------- | -| Plan | `validate_plan.py` exits zero and prints a validation summary | -| Environment | The selected package manager and absolute interpreter target the planned environment | -| Python | `verify_python.py` passes for the selected backend and accelerator; source builds also match the expected build variant | -| C/C++ | Expected libraries and headers exist, dynamic dependencies resolve, and the build cache records the requested accelerator/backend | -| LAMMPS | `verify_lammps.py` finds the exact pair styles required by the model family and no unresolved dynamic dependency | -| Smoke | The selected short example finishes on the explicitly bound device and emits its documented success signal | - -For PyTorch source builds, install the intended PyTorch before DeePMD-kit and -use `--no-build-isolation`. Set both `DP_ENABLE_PYTORCH` and -`DP_ENABLE_TENSORFLOW` explicitly in the same command that invokes pip. - -## LAMMPS model contract - -| Model/runtime | Host pair style | Kokkos CUDA pair style | Artifact requirement | -| ------------------- | --------------- | ----------------------------------------------------------- | --------------------------------------------------------- | -| conventional models | `deepmd` | `deepmd/kk` when the model supports device edge/graph input | Compatible frozen model; `/kk` requires edge/graph `.pt2` | -| DPA4 / SeZM | `deepmd` | `deepmd/kk` | Target-specific graph `.pt2` | -| DPA4C native spin | `dpa4spin` | `dpa4spin/kk` | `atom_style spin` and compact canonical graph `.pt2` | - -Do not accept `deepmd/kk` as proof that DPA4C is available. Do not substitute -LAMMPS `PKG_GPU` for Kokkos. - -## Completion report - -Report: - -- plan path and resolved source commit, when applicable; -- environment activation or absolute Python path; -- installed DeePMD-kit/backend versions and accelerator visibility; -- C/C++ prefix and LAMMPS binary, when applicable; -- each gate command and observed result; -- any validation gate that was not run and the concrete reason; -- no claim of success beyond the gates that actually passed. +Keep source, build, and install locations distinct. + +### Pre-compiled C library + +Use this route only when the official page provides an artifact for the +selected version, platform, and backend. Download and unpack it into a +dedicated prefix, then follow the same page for CMake discovery and optional +LAMMPS plugin use. Do not substitute a Python wheel or a C library from another +release. + +### C/C++ interface and LAMMPS + +For C/C++, follow the backend-specific section of the matching source-install +page; do not guess CMake options or backend library roots. For LAMMPS, use a +packaged `lmp` when it satisfies the request. Otherwise follow the matching +built-in or plugin instructions after the C/C++ interface succeeds. Enable +Kokkos only when the requested LAMMPS runtime requires it, and select the +architecture supported by that exact LAMMPS/Kokkos source tree. + +## 4. Verify the requested interface + +An installation is complete only after the requested public interface runs: + +1. Print `sys.executable`, `sys.prefix`, `deepmd.__version__`, and + `deepmd.__file__` with the selected absolute Python. +1. Import the selected backend (`deepmd.pt`, `deepmd.tf`, `deepmd.jax`, or + `deepmd.pd`) and run one minimal tensor operation on the requested device. +1. Run the installed `dp --version` and backend-specific help. +1. For C/C++, confirm the installed headers/libraries and unresolved dynamic + dependencies before testing a client. +1. For LAMMPS, run the selected binary with `-h`, require the exact DeePMD pair + style needed by the model, and run a short documented example. +1. For Docker, perform all applicable checks inside the selected image. + +On failure, stop at the first failing check and read +[`references/failure-modes.md`](references/failure-modes.md). Report the +selected method, version or commit, environment identity, commands executed, +observed verification results, and anything that remains unverified. diff --git a/skills/deepmd-install/references/easy-install.md b/skills/deepmd-install/references/easy-install.md deleted file mode 100644 index e576f81460..0000000000 --- a/skills/deepmd-install/references/easy-install.md +++ /dev/null @@ -1,212 +0,0 @@ -# Easy installation - -Use this reference for stable package releases and official artifacts. Read the -version-matched `doc/install/easy-install.md` when package extras, platform -support, or backend installation commands differ from this routing guide. The -published documentation is -. - -## Contents - -- [Environment gate](#environment-gate) -- [Pip](#pip) -- [Conda](#conda) -- [dp1s](#dp1s) -- [Offline package](#offline-package) -- [Docker](#docker) -- [Verification](#verification) - -## Environment gate - -Do not mix conda, pip, offline, and `dp1s` installations in one prefix. Reuse -an environment only when the user selected it and the probe found no conflicting -DeePMD-kit install. - -ROCm installations use the source workflow. Packaged LAMMPS and i-PI support -TensorFlow, PyTorch, and JAX; do not add either extra to a Paddle environment. - -For a new venv, create it first, resolve its absolute interpreter, update the -plan, and re-run `validate_plan.py` before installing packages: - -```bash -"" -m venv "" -"/bin/python" -c "import sys; print(sys.executable)" -``` - -Use the platform-equivalent interpreter path on Windows. - -## Pip - -Install backend packages recorded in `package.backend_packages` before -DeePMD-kit when they require a dedicated index. Render one literal pip command; -omit `--index-url` when `backend_index_url` is null. - -```bash -"" -m pip install \ - "" "" \ - --index-url "" -``` - -Choose DeePMD-kit extras from the matching checkout documentation: - -| Backend | DeePMD-kit requirement | -| -------------- | ------------------------------------------------------------------------- | -| PyTorch | `deepmd-kit[torch]` | -| TensorFlow CPU | `deepmd-kit[cpu]` | -| TensorFlow GPU | `deepmd-kit[gpu,cu12]` when the documented CUDA runtime extra is required | -| JAX | `deepmd-kit[jax]` plus the selected JAX accelerator package | -| Paddle | Install the selected Paddle package first, then `deepmd-kit` | - -Append `lmp` and/or `ipi` only when the plan enables them. Append the exact -DeePMD-kit version when `package.deepmd_version` is non-null. Example shape: - -```bash -"" -m pip install "deepmd-kit[torch,lmp]==" -``` - -The final rendered command must contain a real version or omit the version -specifier; it must not contain ``. Add `--index-url` and -`--extra-index-url` only when the corresponding DeePMD-kit index fields are -non-null. - -## Conda - -Use the absolute manager and environment name from the plan. Install only the -requested packages; do not add LAMMPS, Horovod, or MPI unless they are in scope. -Treat `package.channels` as authoritative when it is non-empty and preserve its -order by rendering one `-c` option per entry. This includes selected mirrors and -pre-release channels such as `conda-forge/label/deepmd-kit_dev` or -`conda-forge/label/deepmd-kit_rc`. Use `conda-forge` only when the list is empty. -Pass `--override-channels` so configured defaults cannot supply packages outside -the plan. - -Render the DeePMD-kit package argument as `deepmd-kit=` when -`package.deepmd_version` is non-null, or as `deepmd-kit` when it is null. The -following non-null example preserves the recorded channel order: - -```bash -"" create -n "" \ - --override-channels \ - -c "" -c "" "deepmd-kit=" -``` - -For null `package.deepmd_version` and an empty `package.channels` list, render -the stable unversioned default explicitly: - -```bash -"" create -n "" \ - --override-channels -c conda-forge deepmd-kit -``` - -Add `lammps` only for `goal=python+lammps`. Resolve the created interpreter -without assuming shell activation: - -```bash -"" run -n "" \ - python -c "import sys; print(sys.executable); print(sys.prefix)" -``` - -Record both absolute paths in the plan, re-run `validate_plan.py`, and then run -the Python verifier. Follow the conda-forge CUDA guidance referenced by -`doc/install/easy-install.md`; do not invent a `cudatoolkit` pin. - -## dp1s - -Show the exact command and obtain confirmation before piping a remote response -to a shell: - -```bash -curl -fsSL https://dp1s.deepmodeling.com | env \ - DP1S_HOME="" \ - DP1S_NO_PATH_UPDATE=1 \ - DEEPMD_VERSION="" \ - bash -``` - -Omit the `DEEPMD_VERSION` assignment when `package.deepmd_version` is null. -`DP1S_NO_PATH_UPDATE` prevents the installer from editing shell startup files. -Apply only additional options selected from the official `dp1s` documentation. -After installation, keep `environment.dp1s_home` unchanged. Resolve the absolute -interpreter from the installed `dp` entry point, run that interpreter to obtain -its `sys.prefix`, record both as `environment.python` and `environment.prefix`, -and re-run validation before verification. The Python prefix normally differs -from `dp1s_home`; pass only `environment.prefix` to `--expected-prefix`. - -## Offline package - -Use the exact release asset and SHA-256 checksum recorded in the plan. For a -remote artifact: - -```bash -curl -fL "" -o "" -printf '%s %s\n' "" "" | sha256sum --check - -bash "" -``` - -For `package.artifact_path`, skip curl and verify the local file directly: - -```bash -printf '%s %s\n' "" "" | sha256sum --check - -bash "" -``` - -Use `shasum -a 256` on systems without `sha256sum`. This workflow accepts one -complete installer only. For a split release, follow the version-matched manual -instructions to produce an already-assembled local installer, and use this -workflow only when a trusted final SHA-256 is available for that complete file. -Never execute an HTML error page or an artifact whose checksum is unknown. - -## Docker - -Pull the exact image reference from the plan: - -```bash -docker pull "" -``` - -Mount the verifier read-only and invoke the backend and accelerator selected by -the plan: - -```bash -docker run --rm \ - --mount \ - type=bind,src="/scripts/verify_python.py",dst=/opt/deepmd-install/verify_python.py,readonly \ - "" \ - python /opt/deepmd-install/verify_python.py \ - --backend "" \ - --accelerator "" \ - --expected-version "" -``` - -For CUDA, add `--gpus 'device='` from the plan before the -read-only mount. Omit `--expected-version` when the plan does not pin one. Add -only user-selected volume mounts and working directories; never mount a home -directory or source tree read-write for verification. - -## Verification - -Run the backend-aware verifier with the absolute installed interpreter: - -```bash -"" "/scripts/verify_python.py" \ - --backend "" \ - --accelerator "" \ - --expected-version "" \ - --expected-prefix "" -``` - -Omit `--expected-version` when `package.deepmd_version` is null. - -For packaged host LAMMPS, resolve the executable and require the conventional -host pair style: - -```bash -"" "/scripts/verify_lammps.py" \ - --binary "" \ - --model-family "" \ - --flavor host -``` - -Packaged LAMMPS does not imply Kokkos `deepmd/kk` or DPA4C -`dpa4spin/kk`. Use the source LAMMPS path when either device pair style is -required. diff --git a/skills/deepmd-install/references/failure-modes.md b/skills/deepmd-install/references/failure-modes.md index 642c64c2cb..510f7067b6 100644 --- a/skills/deepmd-install/references/failure-modes.md +++ b/skills/deepmd-install/references/failure-modes.md @@ -1,345 +1,133 @@ # Installation failure modes -Diagnose the current gate only. Preserve the plan and complete logs, change one -input at a time, and re-run the smallest failing command. Do not restart the -workflow, wipe caches, reset source trees, or remove install prefixes. - -## Index - -| Symptom | Section | -| ----------------------------------------------- | ----------------------------------------------------------------------- | -| `python`, pip, and `dp` disagree | [Wrong Python environment](#wrong-python-environment) | -| A plan value is empty or a placeholder remains | [Invalid or stale plan](#invalid-or-stale-plan) | -| A bundled helper cannot be found | [Wrong skill root](#wrong-skill-root) | -| Backend imports but accelerator is unavailable | [Backend accelerator failure](#backend-accelerator-failure) | -| DeePMD-kit reports the wrong build variant | [Wrong compiled variant](#wrong-compiled-variant) | -| `ENABLE_CUSTOMIZED_OP` is false | [PyTorch custom OP is unavailable](#pytorch-custom-op-is-unavailable) | -| CUDA/toolkit/compiler error | [CUDA toolchain mismatch](#cuda-toolchain-mismatch) | -| nvalchemi or vesin is unavailable | [Optional neighbor-list dependency](#optional-neighbor-list-dependency) | -| Source checkout is not the requested ref | [Source identity mismatch](#source-identity-mismatch) | -| CMake keeps an old compiler/backend | [Stale build directory](#stale-build-directory) | -| Torch, NCCL, CUPTI, or CUDA library is missing | [Native library discovery](#native-library-discovery) | -| LAMMPS has no DeePMD styles | [Built-in module is absent](#built-in-module-is-absent) | -| `deepmd/kk` or `dpa4spin/kk` is absent | [Wrong LAMMPS runtime](#wrong-lammps-runtime) | -| Kokkos reports an architecture error | [Kokkos architecture mismatch](#kokkos-architecture-mismatch) | -| LAMMPS starts but a shared library is not found | [Runtime link failure](#runtime-link-failure) | -| Build terminates from memory or disk pressure | [Insufficient build resources](#insufficient-build-resources) | -| Downloaded archive cannot be parsed or verified | [Artifact download failure](#artifact-download-failure) | - -## Wrong Python environment - -Run all checks with the absolute interpreter from the plan: +Diagnose the first failing step. Keep its complete output, change one input at +a time, and rerun the smallest command that distinguishes the cause. Do not +restart the whole installation, erase caches, reset a source tree, or remove a +working environment without the user's approval. -```bash -"" -c "import sys; print(sys.executable); print(sys.prefix)" -"" -m pip --version -command -v dp -head -n 1 "$(command -v dp)" -``` - -The interpreter, pip prefix, and `dp` shebang must identify the same -environment. Resolve `deepmd.__file__` as well; it must exist below the planned -prefix. If it points into a checkout or another environment, run the verifier -from a neutral directory without an injected `PYTHONPATH`. A previous -`conda activate` does not persist across independent agent shell calls. -If a neutral-directory import still points into the checkout, replace the -editable installation with the regular source installation documented in -[`source-python.md`](source-python.md). Re-render the failed gate with the -absolute interpreter. - -## Invalid or stale plan - -Re-run validation whenever a path, version, backend, accelerator, source ref, -or build directory changes: - -```bash -"" "/scripts/validate_plan.py" \ - "" -``` - -Never repair a failed command by guessing a missing value. Update the plan, -validate it, and render the gate again. A command containing an unassigned -plan variable, an empty required argument, or `` is not -executable. - -For conda, require `--override-channels` and compare every rendered `-c` -argument with `package.channels`. An empty list uses only the stable -`conda-forge` default; a non-empty list replaces that default and preserves the -recorded order. A non-null `package.deepmd_version` produces the exact -`deepmd-kit=` package constraint. - -For `dp1s`, compare `DP1S_HOME` with `environment.dp1s_home`, compare the -optional `DEEPMD_VERSION` with the plan, and require -`DP1S_NO_PATH_UPDATE=1`. Preserve `dp1s_home` after installation; record the -installed interpreter's distinct `sys.prefix` as `environment.prefix` for the -Python identity gate. - -For JAX C/C++, two null TensorFlow roots select the Python-library route. A -single root selects its corresponding external library route; setting both is -ambiguous and invalid. - -## Wrong skill root - -Helper scripts belong to the installed skill, not the DeePMD-kit checkout or -current directory: - -```bash -test -f "/scripts/probe_env.py" -test -f "/scripts/verify_python.py" -test -f "/scripts/verify_lammps.py" -test -f "/scripts/verify_native.py" -``` - -Resolve the directory containing `SKILL.md`; do not search for a same-named -script elsewhere and do not skip the gate. - -## Backend accelerator failure - -Use the backend-aware verifier and retain its complete output: - -```bash -"" "/scripts/verify_python.py" \ - --backend "" \ - --accelerator "" \ - --expected-version "" \ - --expected-prefix "" -``` - -Omit `--expected-version` when the plan does not pin a release. - -Check visibility masks from the probe before replacing packages. A false GPU -availability result may come from `CUDA_VISIBLE_DEVICES`, -`HIP_VISIBLE_DEVICES`, a driver/runtime mismatch, or a CPU backend package. -For JAX, inspect backend metadata rather than the device vendor label: - -```bash -"" - <<'PY' -import jax - -for device in jax.devices(): - client = device.client - print(device.platform, client.platform, client.platform_version) -PY -``` - -For a ROCm smoke test, inspect occupancy with `rocm-smi`, then bind the planned -physical index in the same command: - -```bash -HIP_VISIBLE_DEVICES="" \ - ROCR_VISIBLE_DEVICES="" \ - "" "/scripts/verify_python.py" \ - --backend "" \ - --accelerator rocm \ - --expected-prefix "" -``` - -## Wrong compiled variant - -Framework GPU visibility does not prove that DeePMD-kit custom operations were -compiled for that accelerator. Inspect the recorded variant: - -```bash -"" - <<'PY' -from deepmd.env import GLOBAL_CONFIG +## Documentation or asset is unavailable -print(GLOBAL_CONFIG["dp_variant"]) -PY -``` - -Return to the source Python gate when it differs from `build.variant`. Keep the -toolkit, compiler, `DP_VARIANT`, and backend switches in the same pip call. - -## PyTorch custom OP is unavailable +Confirm the requested version or ref and the exact URL. Try the versioned docs, +then the same file at the exact GitHub ref as described in +[`official-docs.md`](official-docs.md). For a blocked public GitHub URL, use the +documented `gh-proxy.com` fallback. Do not silently substitute a newer release, +different CUDA build, or similarly named asset. -Common causes are build isolation, a missing PyTorch installation during the -build, `DP_ENABLE_PYTORCH=0`, an ABI mismatch, or a runtime library that cannot -be loaded. +For downloads, retain the HTTP status, file size, and content type. An HTML +error page is not an installer or archive. Reassemble split offline assets in +the documented order and verify a published checksum when one exists. -Confirm the intended PyTorch and toolchain, then re-run the source install with -`--no-build-isolation` and the complete inline environment from -[`source-python.md`](source-python.md). Do not use `--force-reinstall` until the -rendered command is known to contain the correct plan values. +## Python, pip, and dp identify different environments -## CUDA toolchain mismatch - -Compare all three layers: +Run these checks with the selected absolute interpreter: ```bash -"/bin/nvcc" --version -nvidia-smi "" -c \ - "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())" -``` - -A CUDA major-version mismatch between toolkit and PyTorch is a hard failure. -A same-major minor mismatch requires the actual configure/compile result; -PyTorch treats it as a warning in extension builds. The NVIDIA driver must -support the selected runtime. If `nvcc` rejects the host compiler, select a -compiler supported by that toolkit and update the plan rather than using -`--allow-unsupported-compiler`. - -## Optional neighbor-list dependency - -`vesin[torch]` belongs to the PyTorch extra. nvalchemi package markers depend on -the operating system and Python version. Require these integrations only when -the selected checkout and platform declare them applicable: - -```bash -"" -m pip show vesin nvalchemi-toolkit-ops -``` - -Their absence on an ineligible platform is not a core DeePMD-kit installation -failure. - -## Source identity mismatch - -Inspect without modifying the checkout: - -```bash -git -C "" status --short -git -C "" remote -v -git -C "" rev-parse HEAD -``` - -Use a separate clone when `HEAD`, remote, or local changes do not match the -plan. Store the resolved SHA in `source.commit`, revalidate with -`--require-resolved-source`, and pass the same SHA to -`verify_python.py --expected-source-commit`. Do not reset, clean, or rewrite -the existing checkout. - -## Stale build directory - -A CMake cache records absolute source paths, compilers, toolkit, backend, and -architecture. Compare it with the plan: - -```bash -grep -E \ - 'CMAKE_(C|CXX|CUDA)_COMPILER:|CMAKE_INSTALL_PREFIX:|ENABLE_|USE_.*TOOLKIT:|Kokkos_ARCH_' \ - "/CMakeCache.txt" + "import sys; print(sys.executable); print(sys.prefix)" +"" -m pip --version +command -v dp ``` -If any identity differs, create a new build directory. Preserve the old -directory for diagnosis until the replacement passes. +The Python prefix, pip location, `dp` entry point, and `deepmd.__file__` must +identify the intended environment. If an import resolves into a source tree, +repeat it from a neutral directory and check `PYTHONPATH`. Do not repair an +identity mismatch by installing the package again with a different `pip`. -## Native library discovery +## Dependency resolution or backend import fails -Locate PyTorch and namespace-package NCCL from the selected environment: +Read the first resolver conflict or import error. Compare the selected Python, +OS, architecture, DeePMD-kit version, backend package, and package index with +the matching documentation. Install only the requested backend; adding every +extra often creates conflicts and does not diagnose the missing dependency. -```bash -"" - <<'PY' -from importlib.util import find_spec -from pathlib import Path - -import torch - -print(Path(torch.__file__).resolve().parent / "lib") -spec = find_spec("nvidia.nccl") if find_spec("nvidia") is not None else None -if spec is not None: - for root in spec.submodule_search_locations or (): - candidate = Path(root) / "lib" - if list(candidate.glob("libnccl.so*")): - print(candidate) -PY -``` - -`nvidia.nccl.__file__` may be null. Locate CUPTI independently; it may live -under `extras/CUPTI`, an environment package, or a separate toolkit component. -Add a directory to RPATH or linker flags only after the named library is found -there. +For PyTorch, TensorFlow, JAX, and Paddle, verify the framework itself before +testing DeePMD-kit. A successful `import deepmd` does not prove that the +selected backend is installed. -## Built-in module is absent +## CPU, CUDA, or ROCm is not available -Check the managed include without changing the LAMMPS tree: +Separate three facts: the accelerator is visible to the host, the framework +was built for the requested runtime, and a tensor operation executes on that +device. Check device visibility variables and compare the framework runtime +with the driver/toolkit reported by the system. Do not replace packages until +the failing layer is identified. -```bash -"" "/scripts/prepare_lammps.py" \ - --lammps-source "" \ - --deepmd-source "" \ - --check -``` +A source compiler toolkit and a framework wheel runtime are related but not +identical. Treat a major-version incompatibility as a hard failure; use the +official compatibility guidance for minor-version combinations. -If it differs, run the preparer without `--check`, then configure a new LAMMPS -build directory. Re-running CMake in the old directory may preserve a source -state from before the include was added. +## Source ref or build identity is wrong -## Wrong LAMMPS runtime - -Verify exact styles rather than grepping any occurrence of `deepmd`: +Inspect without modifying the checkout: ```bash -"" "/scripts/verify_lammps.py" \ - --binary "" \ - --model-family "" \ - --flavor "" +git -C "" status --short +git -C "" remote -v +git -C "" rev-parse HEAD ``` -Kokkos must be enabled for `/kk`. DPA4C requires `dpa4spin/kk`, not -`deepmd/kk`. A CPU DeePMD C/C++ library cannot serve the Kokkos CUDA pair -styles. `PKG_GPU` does not provide them. +Use a separate clone when the remote, commit, or local changes do not match the +request. Never use `git reset --hard` or `git clean` on a user tree. Reject a +remote or ref beginning with `-`; option-like input must not reach `git fetch` +or `git checkout`. -If the exact style exists but rejects the model, check the artifact contract: -`deepmd/kk` needs an edge/graph `.pt2`, one model, and `atom_modify map yes`; -`dpa4spin/kk` needs a DPA4C compact canonical graph and `atom_style spin`. +## A source build fails -## Kokkos architecture mismatch +Preserve the first compiler or CMake error. Verify the absolute compiler, +Python, backend installation, toolkit root, and free disk/RAM. Re-read the +source-install page and build options at the exact commit. If CMake cached a +different compiler, source path, backend, or install prefix, create a new build +directory instead of layering more flags onto the stale cache. -Query numeric capability and compare it with both the plan and selected Kokkos -tree: +For PyTorch custom operations, confirm that the intended PyTorch is visible to +the build and that build isolation behavior matches the selected version's +documentation. For TensorFlow, JAX, or Paddle C/C++, use the documented library +interface and ABI for that backend; do not substitute a Python package root +unless the docs explicitly support it. -```bash -nvidia-smi --query-gpu=index,name,compute_cap --format=csv,noheader -grep -F "kokkos_arch_option(" \ - "/lib/kokkos/cmake/kokkos_arch.cmake" -``` +## A native library cannot be loaded -Use exactly one architecture in one build directory. Do not infer it from a -GPU-name substring. If the compiler test requires `nvcc_wrapper`, use the -wrapper from that LAMMPS tree with the host compiler supported by the selected -CUDA toolkit. +Use `ldd` on Linux or `otool -L` on macOS for the failing library or executable. +Locate the named dependency in the selected environment before changing +`RPATH`, `LD_LIBRARY_PATH`, or linker flags. Framework import success does not +prove that a separately built C++ client or LAMMPS binary uses the same ABI and +libraries. -## Runtime link failure +## LAMMPS lacks the required DeePMD pair style -Use the bundled verifier so Linux checks `ldd` output and macOS asks `dyld` to -load the library instead of trusting Mach-O metadata alone: +Run the exact selected binary with `-h`. Distinguish packaged, built-in, and +plugin installations; for plugin mode, confirm that the plugin was loaded. +Require the pair style used by the target model rather than any occurrence of +the word `deepmd`. Kokkos styles require a Kokkos-enabled LAMMPS build and an +architecture supported by that LAMMPS source tree. -```bash -"" "/scripts/verify_native.py" \ - --path "" -"" "/scripts/verify_lammps.py" \ - --binary "" \ - --model-family "" \ - --flavor "" \ - --check-links -``` - -Resolve every `not found` entry through build/install RPATH. Avoid global -`LD_LIBRARY_PATH` and shell-rc changes; they can cause another environment's -CUDA, Torch, TensorFlow, or compiler runtime to shadow the planned libraries. -On macOS, `otool -L` lists install names but does not prove they resolve. +If the style exists but rejects a model, diagnose model/runtime compatibility +separately from installation. Rebuild only after confirming that the binary, +DeePMD C/C++ libraries, backend, and model family are the intended combination. -## Insufficient build resources +## Docker verification disagrees with the host -Compare the probe's free disk, RAM, and CPU count with `build.jobs`. Reduce the -job count and re-run the same build command. Do not change compilers, backend, -or source ref in the same experiment. Preserve the first out-of-memory or disk -error because later compiler failures may be secondary damage. +Host executables do not verify a container. Run all checks inside the exact +image tag, record the container's absolute `sys.executable` and `sys.prefix`, +and invoke that interpreter directly. For packaged LAMMPS, run the in-container +binary. Mount verification inputs read-only and pass an explicit device with +Docker's GPU option when CUDA is requested. -## Artifact download failure +If the image contains multiple Python installations, do not fall back to a +bare `python`; inspect the image definition or documented environment prefix +and verify `deepmd.__file__` against it. -Use `curl -fL` so HTTP errors fail immediately. Verify file type and checksum -before extraction or execution: +## Permission, disk, or memory failure -```bash -file "" -printf '%s %s\n' "" "" | sha256sum --check - -``` +Do not switch silently to `sudo`, a shared system prefix, or a different disk. +Report the required permission or resource and use a user-approved dedicated +prefix. For compilation memory pressure, reduce parallel jobs and rerun the +same build so that compiler, backend, toolkit, and source commit remain fixed. -An absent checksum, HTML response, unexpected top-level directory, or checksum -mismatch blocks extraction. Obtain a fresh artifact and trusted checksum for -the planned URL; do not disable verification. Split parts are not valid plan -artifacts; supply one already-assembled installer with a trusted checksum for -the complete file. +## Reporting a blocked installation -For `package.artifact_path`, skip curl and run the same file-type and checksum -checks directly on the absolute local path. +Report the selected method and version, the exact failing command, its first +actionable error, environment identity, and checks that passed. State what +additional user choice or external change is required. Do not report an +installation as successful when only package resolution or file creation +completed. diff --git a/skills/deepmd-install/references/official-docs.md b/skills/deepmd-install/references/official-docs.md new file mode 100644 index 0000000000..0f12fddfc1 --- /dev/null +++ b/skills/deepmd-install/references/official-docs.md @@ -0,0 +1,107 @@ +# Official documentation routing + +Fetch documentation before choosing version-sensitive packages, image tags, +build variables, or LAMMPS options. This reference assumes there is no local +DeePMD-kit checkout. + +## Fetch procedure + +1. Use the agent's browser, URL reader, or web-fetch tool to open a direct URL + from this file. Read the page body and relevant tabs; a search snippet is + not documentation. + +1. Determine the target release tag or Git ref before selecting commands. For + a stable release, obtain the tag from the official Releases page. + +1. Open the versioned documentation. If it is missing or incomplete, retrieve + the corresponding raw Markdown from the exact tag or commit. + +1. If the agent has no web-fetch tool, use `curl` and keep the response in the + tool output: + + ```bash + curl -fsSL --retry 2 \ + "https://raw.githubusercontent.com/deepmodeling/deepmd-kit//doc/install/.md" + ``` + +1. Stop on a nonzero curl exit, empty response, or HTML error page. Do not + generate commands from partial or unverified content. + +## Primary pages + +- Easy install: +- Source Python and C/C++: +- Pre-compiled C library: +- LAMMPS: +- Development packages: +- Releases: +- Container images: +- dp1s options: + +Use official backend installers when the DeePMD-kit page links to them: + +- PyTorch: +- TensorFlow: +- JAX: +- Paddle: + +## Match the target version + +The `latest` documentation follows the development branch and may not describe +an older release. For a release, first try the versioned documentation root: + +```text +https://docs.deepmodeling.com/projects/deepmd/en/v/ +``` + +If that build is unavailable or incomplete, fetch the relevant Markdown file +from the exact tag or commit in the official repository. Typical paths are: + +```text +doc/install/easy-install.md +doc/install/easy-install-dev.md +doc/install/install-from-source.md +doc/install/install-from-c-library.md +doc/install/install-lammps.md +pyproject.toml +``` + +For example, replace `` with a validated tag or commit in either form: + +```text +https://github.com/deepmodeling/deepmd-kit/blob//doc/install/install-from-source.md +https://raw.githubusercontent.com/deepmodeling/deepmd-kit//doc/install/install-from-source.md +``` + +Resolve a branch or tag to a commit SHA before building. Treat a Git ref as +data: reject values beginning with `-`, quote it, and use Git's `--` option +boundary where supported. + +## When a page lacks the required detail + +Inspect authoritative files at the same ref instead of borrowing commands from +another version: + +1. `pyproject.toml` for Python requirements and extras; +1. `source/CMakeLists.txt` and included CMake modules for current build options; +1. `.github/workflows/` and `source/install/docker/Dockerfile` for maintained + package and image build examples; +1. the selected LAMMPS tree for available Kokkos architecture names. + +State clearly when a conclusion is inferred from maintained build +configuration rather than stated in the user documentation. Do not invent a +version, wheel index, image tag, CMake option, or Kokkos architecture. + +## Network fallback + +If a GitHub repository or raw-file URL is unreachable because of network +restrictions, retry the same public URL through `gh-proxy.com`, for example: + +```text +https://gh-proxy.com/https://github.com/deepmodeling/deepmd-kit.git +https://gh-proxy.com/https://raw.githubusercontent.com/deepmodeling/deepmd-kit//doc/install/install-from-source.md +``` + +Use the proxy only for public GitHub content. Never send credentials through +it. After cloning through a proxy, verify the configured upstream URL and the +resolved commit SHA before building. diff --git a/skills/deepmd-install/references/plan-schema.md b/skills/deepmd-install/references/plan-schema.md deleted file mode 100644 index 1f4451cd8a..0000000000 --- a/skills/deepmd-install/references/plan-schema.md +++ /dev/null @@ -1,356 +0,0 @@ -# Installation plan schema - -Store every decision in one JSON document. Keep the plan outside source, -build, and install directories. The plan contains no credentials or shell -commands. - -## Contents - -- [Top-level fields](#top-level-fields) -- [Conditional objects](#conditional-objects) -- [Validation rules](#validation-rules) -- [Examples](#examples) - -## Top-level fields - -```json -{ - "schema_version": 1, - "method": "pip", - "goal": "python", - "backend": "pytorch", - "accelerator": "cuda", - "environment": {}, - "package": {}, - "source": null, - "build": null, - "cpp": null, - "lammps": null, - "smoke_test": { - "enabled": false, - "gpu": null, - "example": null - } -} -``` - -Allowed values: - -- `method`: `pip`, `conda`, `dp1s`, `offline`, `docker`, or `source`. -- `goal`: `python`, `python+lammps`, `python+cpp`, or - `python+cpp+lammps`. -- `backend`: `pytorch`, `tensorflow`, `jax`, or `paddle`. -- `accelerator`: `cpu`, `cuda`, or `rocm`. - -Use `null` for an inapplicable object. Do not add undeclared keys. - -## Conditional objects - -### `environment` - -```json -{ - "kind": "existing", - "python": "/absolute/path/to/python", - "manager": null, - "name": null, - "prefix": "/absolute/environment/prefix", - "dp1s_home": null -} -``` - -- `kind`: `existing`, `venv`, `conda`, `prefix`, or `container`. -- `python`: required for `pip` and `source`; resolve it after creating a new - environment and before installing DeePMD-kit. -- `manager` and `name`: required for `conda`; `manager` is the absolute - `conda` or `mamba` executable. -- `prefix`: required for `pip`, `source`, `venv`, `prefix`, and offline - installations. For conda, resolve and record both `sys.executable` and - `sys.prefix` after creating the environment and before verification. -- `dp1s_home`: required only for `dp1s`; it is the Pixi and exposed-binary root - passed as `DP1S_HOME`, not the installed Python prefix. Before installation, - keep `python` and `prefix` null. After installation, preserve `dp1s_home` and - record the resolved interpreter and its `sys.prefix` together. - -### `package` - -```json -{ - "deepmd_version": null, - "deepmd_index_url": null, - "deepmd_extra_index_url": null, - "backend_packages": [ - "torch" - ], - "backend_index_url": null, - "channels": [ - "conda-forge" - ], - "install_lammps": false, - "install_ipi": false, - "artifact_url": null, - "artifact_path": null, - "sha256": null, - "docker_image": null, - "lammps_model_family": null -} -``` - -- Record exact backend requirements selected from the backend's official - installer or the version-matched DeePMD-kit documentation. -- Keep the two DeePMD-kit index fields null for the default package index. - Record a user-selected mirror or the documented pre-release index explicitly. -- Keep `backend_index_url` null when the default package index is intended. -- For conda, an empty `channels` list selects the stable `conda-forge` - default. A non-empty list is authoritative and is rendered in order, with - one `-c` option per channel. Keep this list empty for other methods. -- Package indexes and download URLs use HTTPS. -- Require exactly one of HTTPS `artifact_url` or absolute `artifact_path`, plus - `sha256`, for `offline`. -- An offline plan represents one complete installer. `artifact_path` must be an - existing assembled file with a trusted SHA-256 for that complete file. Split - release parts and per-part checksums are outside this plan contract. -- Require an immutable image reference or user-selected tag for `docker`. -- Require `lammps_model_family` for packaged LAMMPS verification. - -### `source` - -```json -{ - "directory": "/absolute/path/to/deepmd-kit", - "remote": "https://github.com/deepmodeling/deepmd-kit.git", - "ref": "master", - "commit": null -} -``` - -Treat `ref` as an opaque Git ref. Resolve it to a commit, store the SHA in -`commit`, and revalidate the plan before building. Do not reset or clean an -existing checkout. - -### `build` - -```json -{ - "variant": "cuda", - "cc": "/usr/bin/gcc", - "cxx": "/usr/bin/g++", - "cuda_home": "/usr/local/cuda-12.8", - "rocm_root": null, - "native_optimization": false, - "jobs": 8 -} -``` - -`build.variant` describes DeePMD-kit's compiled custom operations. It may -differ from `accelerator` for a backend whose accelerator runtime is supplied -entirely by its Python package. Require `cuda_home` for a CUDA build and -`rocm_root` for a ROCm build. - -### `cpp` - -```json -{ - "install_prefix": "/absolute/dedicated/deepmd-prefix", - "build_directory": "/absolute/build/deepmd-cpp", - "tensorflow_root": null, - "tensorflow_c_root": null, - "paddle_inference_dir": null -} -``` - -Use a dedicated install prefix. Never choose `/`, a home directory, a conda -prefix, `/usr`, `/usr/local`, or `$HOME/.local` as a disposable prefix. - -For a JAX C++ backend, the TensorFlow dependency is selected by the two -nullable roots: - -- Keep both null to use the TensorFlow C++ libraries from the selected Python - environment with `USE_TF_PYTHON_LIBS=ON`. -- Set only `tensorflow_root` to use an external TensorFlow C++ installation. -- Set only `tensorflow_c_root` to use the TensorFlow C library. - -The two TensorFlow roots are mutually exclusive. For Paddle C++, provide -`paddle_inference_dir`. - -### `lammps` - -```json -{ - "source_directory": "/absolute/path/to/lammps", - "build_directory": "/absolute/build/lammps-blackwell120", - "version": "stable_22Jul2025_update2", - "url": "https://github.com/lammps/lammps/archive/refs/tags/stable_22Jul2025_update2.tar.gz", - "sha256": "", - "flavor": "kokkos-cuda", - "machine": "blackwell120", - "kokkos_arch": "BLACKWELL120", - "mpi": false, - "model_family": "dpa4c" -} -``` - -- `flavor`: `host` or `kokkos-cuda`. -- `model_family`: `conventional`, `dpa4`, or `dpa4c`. -- `machine` and `kokkos_arch` are required only for `kokkos-cuda`. -- Use one Kokkos GPU architecture and one build directory per binary. -- Keep `url` and `sha256` null only when `source_directory` is an existing - directory whose LAMMPS version has been verified. Otherwise provide both an - HTTPS archive URL and its trusted 64-character SHA-256 checksum. - -### `smoke_test` - -```json -{ - "enabled": true, - "gpu": 7, - "example": "/absolute/path/to/example" -} -``` - -Require a physical GPU index for CUDA and ROCm smoke tests. Keep `gpu` null for -CPU. The example path must belong to the selected source checkout or be -explicitly provided by the user. - -## Validation rules - -The validator enforces these invariants: - -1. `pip` and `source` use an absolute Python executable. -1. C/C++ goals use `source` and provide `build` plus `cpp`. -1. Source LAMMPS provides `lammps`; Kokkos CUDA requires the PyTorch backend - and a CUDA build. -1. DPA4C maps to `dpa4spin`/`dpa4spin/kk`; other families map to - `deepmd`/`deepmd/kk`. -1. DeePMD, C/C++, and LAMMPS source/build/install paths are distinct. -1. Checksums contain exactly 64 hexadecimal characters; offline artifacts and - downloaded LAMMPS archives require a checksum. -1. Easy-install methods reject ROCm and Paddle packaged LAMMPS/i-PI. -1. Embedded placeholders, control characters, and POSIX-template escape - characters fail before command rendering. -1. Unknown keys and unsupported combinations fail before any state change. - -## Examples - -### Pip PyTorch CUDA - -```json -{ - "schema_version": 1, - "method": "pip", - "goal": "python", - "backend": "pytorch", - "accelerator": "cuda", - "environment": { - "kind": "existing", - "python": "/opt/conda/envs/deepmd/bin/python", - "manager": null, - "name": null, - "prefix": "/opt/conda/envs/deepmd", - "dp1s_home": null - }, - "package": { - "deepmd_version": null, - "deepmd_index_url": null, - "deepmd_extra_index_url": null, - "backend_packages": [], - "backend_index_url": null, - "channels": [], - "install_lammps": false, - "install_ipi": false, - "artifact_url": null, - "artifact_path": null, - "sha256": null, - "docker_image": null, - "lammps_model_family": null - }, - "source": null, - "build": null, - "cpp": null, - "lammps": null, - "smoke_test": { - "enabled": false, - "gpu": null, - "example": null - } -} -``` - -### Source PyTorch CUDA with DPA4C LAMMPS - -This is a pre-resolution template. Replace the LAMMPS checksum placeholder -before initial validation. Keep `source.commit` null until the checkout gate -resolves `source.ref`; then record the SHA and run the resolved-source gate -before installing dependencies or building. - -```json -{ - "schema_version": 1, - "method": "source", - "goal": "python+cpp+lammps", - "backend": "pytorch", - "accelerator": "cuda", - "environment": { - "kind": "existing", - "python": "/opt/conda/envs/deepmd/bin/python", - "manager": null, - "name": null, - "prefix": "/opt/conda/envs/deepmd", - "dp1s_home": null - }, - "package": { - "deepmd_version": null, - "deepmd_index_url": null, - "deepmd_extra_index_url": null, - "backend_packages": [], - "backend_index_url": null, - "channels": [], - "install_lammps": false, - "install_ipi": false, - "artifact_url": null, - "artifact_path": null, - "sha256": null, - "docker_image": null, - "lammps_model_family": null - }, - "source": { - "directory": "/work/deepmd-kit", - "remote": "https://github.com/deepmodeling/deepmd-kit.git", - "ref": "master", - "commit": null - }, - "build": { - "variant": "cuda", - "cc": "/usr/bin/gcc", - "cxx": "/usr/bin/g++", - "cuda_home": "/usr/local/cuda-12.8", - "rocm_root": null, - "native_optimization": false, - "jobs": 8 - }, - "cpp": { - "install_prefix": "/work/install/deepmd-cuda", - "build_directory": "/work/build/deepmd-cpp-cuda", - "tensorflow_root": null, - "tensorflow_c_root": null, - "paddle_inference_dir": null - }, - "lammps": { - "source_directory": "/work/lammps-stable_22Jul2025_update2", - "build_directory": "/work/build/lammps-blackwell120", - "version": "stable_22Jul2025_update2", - "url": "https://github.com/lammps/lammps/archive/refs/tags/stable_22Jul2025_update2.tar.gz", - "sha256": "", - "flavor": "kokkos-cuda", - "machine": "blackwell120", - "kokkos_arch": "BLACKWELL120", - "mpi": false, - "model_family": "dpa4c" - }, - "smoke_test": { - "enabled": true, - "gpu": 7, - "example": "/work/deepmd-kit/examples/spin/dpa4c/lmp" - } -} -``` diff --git a/skills/deepmd-install/references/source-cpp.md b/skills/deepmd-install/references/source-cpp.md deleted file mode 100644 index 3b643974f3..0000000000 --- a/skills/deepmd-install/references/source-cpp.md +++ /dev/null @@ -1,167 +0,0 @@ -# Source installation: C/C++ interface - -Build the C/C++ interface only after the Python/backend gate passes. Use -`doc/install/install-from-source.md` in the selected checkout for the exact -CMake options supported by that ref. The published documentation is -. - -This workflow builds a backend-enabled C/C++ interface. For backend-neutral -`libdeepmd_cc`/`libdeepmd_c`, follow the checkout documentation and its explicit -`ALLOW_NO_BACKEND=ON` contract. - -## Contents - -- [Build-directory gate](#build-directory-gate) -- [Backend configuration](#backend-configuration) -- [Configure and install](#configure-and-install) -- [C/C++ verification](#cc-verification) - -## Build-directory gate - -Require CMake 3.25.2 or newer and the exact compilers from the plan: - -```bash -"" --version -"" --version -"" --version -``` - -The source, build, and install paths must be distinct. The install prefix must -be dedicated to this build. If the build directory contains a `CMakeCache.txt` -from another source commit, compiler, backend, or accelerator, select a new -build directory. Never delete or recursively replace the install prefix. - -## Backend configuration - -Start with all backends disabled, then enable the selected backend and its -documented C API dependency. Enabling TensorFlow also enables the JAX C API in -DeePMD-kit by design. For JAX, select exactly one route from the state of the -nullable roots in the plan. - -| Backend dependency | Plan state | Required CMake arguments | -| ---------------------------------------- | ------------------------------- | ------------------------------------------------------------------------------------- | -| PyTorch | Not applicable | `-DENABLE_PYTORCH=ON -DUSE_PT_PYTHON_LIBS=ON` plus the PyTorch CMake prefix | -| TensorFlow | Not applicable | `-DENABLE_TENSORFLOW=ON -DUSE_TF_PYTHON_LIBS=ON` and the selected Python executable | -| JAX with Python TensorFlow C++ libraries | Both TensorFlow roots are null | `-DENABLE_TENSORFLOW=ON -DUSE_TF_PYTHON_LIBS=ON` and the selected Python executable | -| JAX with external TensorFlow C++ | Only `tensorflow_root` is set | `-DENABLE_TENSORFLOW=ON -DUSE_TF_PYTHON_LIBS=OFF -DTENSORFLOW_ROOT=` | -| JAX with TensorFlow C | Only `tensorflow_c_root` is set | `-DENABLE_JAX=ON -DCMAKE_PREFIX_PATH=` | -| Paddle | Not applicable | `-DENABLE_PADDLE=ON -DPADDLE_INFERENCE_DIR=` | - -The Python TensorFlow C++ route requires the planned interpreter to import a -TensorFlow package that provides `libtensorflow_cc`. Record and install that -package during the Python gate before configuring CMake. - -For PyTorch, discover the prefix from the planned interpreter in the same shell -call that configures CMake: - -```bash -"" -c "import torch; print(torch.utils.cmake_prefix_path)" -``` - -When more than one backend is explicitly requested, confirm compatible -`_GLIBCXX_USE_CXX11_ABI` settings before compiling. Do not enable an incidental -backend merely because its Python package is importable. - -Accelerator arguments: - -| Build variant | Required CMake arguments | -| ------------- | ---------------------------------------------------------------------------------------------------------------------------------- | -| CPU | `-DUSE_CUDA_TOOLKIT=OFF -DUSE_ROCM_TOOLKIT=OFF` | -| CUDA | `-DUSE_CUDA_TOOLKIT=ON -DCUDAToolkit_ROOT= -DCMAKE_CUDA_COMPILER=/bin/nvcc -DCMAKE_CUDA_HOST_COMPILER=` | -| ROCm | `-DUSE_ROCM_TOOLKIT=ON -DCMAKE_HIP_COMPILER_ROCM_ROOT=` | - -## Configure and install - -Render all plan values into one call. This PyTorch CUDA example shows the -complete shape: - -```bash -torch_cmake_prefix=$("" -c \ - "import torch; print(torch.utils.cmake_prefix_path)") && -"" \ - -S "/source" \ - -B "" \ - -DCMAKE_BUILD_TYPE=Release \ - -DCMAKE_INSTALL_PREFIX="" \ - -DCMAKE_C_COMPILER="" \ - -DCMAKE_CXX_COMPILER="" \ - -DPython_EXECUTABLE="" \ - -DPython3_EXECUTABLE="" \ - -DBUILD_CPP_IF=ON \ - -DBUILD_PY_IF=OFF \ - -DENABLE_NATIVE_OPTIMIZATION=OFF \ - -DENABLE_PYTORCH=ON \ - -DUSE_PT_PYTHON_LIBS=ON \ - -DENABLE_TENSORFLOW=OFF \ - -DENABLE_JAX=OFF \ - -DENABLE_PADDLE=OFF \ - -DUSE_CUDA_TOOLKIT=ON \ - -DUSE_ROCM_TOOLKIT=OFF \ - -DCMAKE_PREFIX_PATH="$torch_cmake_prefix" \ - -DCUDAToolkit_ROOT="" \ - -DCMAKE_CUDA_COMPILER="/bin/nvcc" \ - -DCMAKE_CUDA_HOST_COMPILER="" -``` - -Replace only the backend and accelerator arguments using the tables. Set -`ENABLE_NATIVE_OPTIMIZATION=ON` only when the installed libraries remain on the -same CPU model. - -Build and install with the bounded job count from the plan: - -```bash -"" --build "" \ - --parallel "" -"" --install "" -``` - -If the prefix already contains an installation, choose a new versioned prefix -and switch consumers only after its verification passes. - -## C/C++ verification - -Check the cache against the plan. For a CUDA PyTorch build, for example: - -```bash -grep -E \ - 'BUILD_CPP_IF:|BUILD_PY_IF:|ENABLE_PYTORCH:|USE_CUDA_TOOLKIT:|CMAKE_CUDA_COMPILER:' \ - "/CMakeCache.txt" -``` - -Require the headers and libraries: - -```bash -test -f "/include/deepmd/deepmd.hpp" -test -f "/include/deepmd/c_api.h" -test -f "/lib/libdeepmd_cc.so" -test -f "/lib/libdeepmd_c.so" -``` - -On Linux and macOS, fail if any installed DeePMD library has an unresolved -dynamic dependency. The verifier selects the platform library suffix: - -```bash -"" "/scripts/verify_native.py" \ - --directory "/lib" -``` - -Finally compile and run a public C++ API probe inside the build directory: - -```bash -probe_source="/verify_deepmd.cpp" -probe_binary="/verify_deepmd" -printf '%s\n' \ - '#include "deepmd/common.h"' \ - 'int main() { deepmd::print_summary(""); return 0; }' \ - > "$probe_source" -"" -std=c++14 "$probe_source" \ - -I"/include" \ - -L"/lib" \ - -Wl,-rpath,"/lib" \ - -ldeepmd_cc \ - -o "$probe_binary" -"$probe_binary" -``` - -Do not proceed to LAMMPS when cache values, dynamic dependencies, or the public -API probe fail. diff --git a/skills/deepmd-install/references/source-lammps.md b/skills/deepmd-install/references/source-lammps.md deleted file mode 100644 index d0a698c734..0000000000 --- a/skills/deepmd-install/references/source-lammps.md +++ /dev/null @@ -1,241 +0,0 @@ -# Source installation: LAMMPS - -Build LAMMPS only after the C/C++ prefix passes its public API and dynamic-link -checks. Use `doc/install/install-lammps.md` and the example README files inside -the selected DeePMD-kit checkout for version-specific commands. The published -documentation is -. - -## Contents - -- [Acquire LAMMPS](#acquire-lammps) -- [Add the built-in module](#add-the-built-in-module) -- [Select the runtime](#select-the-runtime) -- [Host build](#host-build) -- [Kokkos CUDA build](#kokkos-cuda-build) -- [Verification](#verification) -- [Smoke tests](#smoke-tests) - -## Acquire LAMMPS - -Use an existing source directory only when its version matches the plan and its -working tree is suitable for a build. Otherwise download the exact HTTPS URL -from the plan into a separate directory: - -```bash -curl -fL "" -o "" -``` - -For a download, require `lammps.sha256` and verify it before extraction: - -```bash -printf '%s %s\n' "" "" | sha256sum --check - -``` - -Stop on a missing or mismatched checksum. A plan may omit the URL and checksum -only when `lammps.source_directory` already exists and its version has been -verified. - -Extract into a directory whose resolved path equals -`lammps.source_directory`. Stop if the archive creates an unexpected directory -or contains paths outside its top-level directory. - -## Add the built-in module - -Use the bundled preparer instead of appending an unquoted CMake line: - -```bash -"" "/scripts/prepare_lammps.py" \ - --lammps-source "" \ - --deepmd-source "" -``` - -The script maintains exactly one quoted include block and replaces a single -legacy include. It fails on ambiguous duplicate includes. - -## Select the runtime - -| Selection | Required pair styles | -| ------------------------ | ------------------------- | -| conventional host | `deepmd` | -| conventional Kokkos CUDA | `deepmd`, `deepmd/kk` | -| DPA4/SeZM host | `deepmd` | -| DPA4/SeZM Kokkos CUDA | `deepmd`, `deepmd/kk` | -| DPA4C host | `dpa4spin` | -| DPA4C Kokkos CUDA | `dpa4spin`, `dpa4spin/kk` | - -`deepmd/kk` accepts a compatible edge-input or graph-input `.pt2` artifact. -It requires `atom_modify map yes` and a single model; model-deviation ensembles -use the host pair style. `dpa4spin/kk` accepts a DPA4C compact canonical graph -artifact and requires `atom_style spin`. Neither style is provided by LAMMPS -`PKG_GPU`. - -For Kokkos, select the architecture from numeric compute capability, then -confirm that the chosen LAMMPS tree defines the option: - -| Compute capability | Kokkos architecture | -| ------------------ | ------------------- | -| 7.0 | `VOLTA70` | -| 7.5 | `TURING75` | -| 8.0 | `AMPERE80` | -| 8.6 | `AMPERE86` | -| 8.9 | `ADA89` | -| 9.0 | `HOPPER90` | -| 10.0 | `BLACKWELL100` | -| 12.0 | `BLACKWELL120` | -| 12.1 | `BLACKWELL121` | - -```bash -grep -F "kokkos_arch_option(" \ - "/lib/kokkos/cmake/kokkos_arch.cmake" -``` - -If the numeric capability or the Kokkos option is unavailable, stop and select -a compatible toolkit/LAMMPS version. Use one architecture and a distinct build -directory/binary for each target GPU family. - -## Host build - -Configure the minimal host binary. Enable optional LAMMPS packages only when -the user's simulation requires them: - -```bash -"" \ - -S "/cmake" \ - -B "" \ - -DCMAKE_BUILD_TYPE=Release \ - -DCMAKE_C_COMPILER="" \ - -DCMAKE_CXX_COMPILER="" \ - -DPython_EXECUTABLE="" \ - -DPython3_EXECUTABLE="" \ - -DCMAKE_INSTALL_PREFIX="" \ - -DBUILD_MPI="" \ - -DBUILD_SHARED_LIBS=ON \ - -DLAMMPS_INSTALL_RPATH=ON \ - -DCMAKE_PREFIX_PATH="" -``` - -Build and install with the planned job limit: - -```bash -"" --build "" --parallel "" -"" --install "" -``` - -The expected binary is `/bin/lmp` unless the plan explicitly -sets a LAMMPS machine suffix. - -## Kokkos CUDA build - -Configure one architecture. The same shell call discovers PyTorch and -namespace-package NCCL, constructs RPATH, and invokes CMake so no derived value -can disappear: - -```bash -runtime_rpath=$("" - <<'PY' -from importlib.util import find_spec -from pathlib import Path - -import torch - -paths = [Path(torch.__file__).resolve().parent / "lib"] -spec = find_spec("nvidia.nccl") if find_spec("nvidia") is not None else None -if spec is not None: - for root in spec.submodule_search_locations or (): - candidate = Path(root) / "lib" - if list(candidate.glob("libnccl.so*")): - paths.append(candidate) -print(";".join(str(path) for path in paths if path.is_dir())) -PY -) && -test -n "$runtime_rpath" && -cuda_runtime_dir=$("" - "" <<'PY' -import sys -from pathlib import Path - -root = Path(sys.argv[1]) -candidates = [root / "lib64", *root.glob("targets/*-linux/lib")] -print(next((path.resolve() for path in candidates if path.is_dir()), "")) -PY -) && -test -n "$cuda_runtime_dir" && -torch_cmake_prefix=$("" -c \ - "import torch; print(torch.utils.cmake_prefix_path)") && -runtime_rpath="/lib;$cuda_runtime_dir;$runtime_rpath" && -"" \ - -S "/cmake" \ - -B "" \ - -DCMAKE_BUILD_TYPE=Release \ - -DCMAKE_C_COMPILER="" \ - -DCMAKE_CXX_COMPILER="" \ - -DCMAKE_CUDA_COMPILER="/bin/nvcc" \ - -DCMAKE_CUDA_HOST_COMPILER="" \ - -DPython_EXECUTABLE="" \ - -DPython3_EXECUTABLE="" \ - -DCMAKE_INSTALL_PREFIX="" \ - -DLAMMPS_MACHINE="" \ - -DBUILD_MPI="" \ - -DBUILD_SHARED_LIBS=OFF \ - -DLAMMPS_INSTALL_RPATH=ON \ - -DPKG_KOKKOS=ON \ - -DKokkos_ENABLE_CUDA=ON \ - -DKokkos_ENABLE_SERIAL=ON \ - -DKokkos_ARCH_=ON \ - -DCMAKE_PREFIX_PATH=";$torch_cmake_prefix" \ - -DCUDAToolkit_ROOT="" \ - -DCMAKE_BUILD_RPATH="$runtime_rpath" \ - -DCMAKE_INSTALL_RPATH="$runtime_rpath" -``` - -The rendered command must contain one real `Kokkos_ARCH_*` argument and no -`` placeholder. Add linker flags only in response to a concrete -link error and only after locating the named library. - -Build and install: - -```bash -"" --build "" --parallel "" -"" --install "" -``` - -The expected binary is `/bin/lmp_`. - -## Verification - -Run the exact model-family verifier. Use `--check-links` for a source binary: - -```bash -"" "/scripts/verify_lammps.py" \ - --binary "" \ - --model-family "" \ - --flavor "" \ - --check-links -``` - -Do not replace a failed exact style check with a broad `grep deepmd`. - -## Smoke tests - -Check the selected physical GPU immediately before a CUDA test: - -```bash -nvidia-smi --query-gpu=index,name,memory.used,memory.total,utilization.gpu \ - --format=csv -``` - -Bind it explicitly so the process sees that physical card as logical device 0: - -```bash -CUDA_VISIBLE_DEVICES="" \ - "" -k on g 1 -sf kk -in "" -``` - -For DPA4/SeZM, follow -`examples/water/dpa4/lmp/README.md` from the same DeePMD-kit checkout. For -DPA4C, follow `examples/spin/dpa4c/lmp/README.md`; use its `--pt-expt` -compression/freeze sequence and spin input rather than a generic freeze -command. For conventional models, use a short user-selected example compatible -with the model format. - -Success requires a completed run with `Loop time` in the LAMMPS log and no -artifact-schema, pair-style, device, or unresolved-library error. diff --git a/skills/deepmd-install/references/source-python.md b/skills/deepmd-install/references/source-python.md deleted file mode 100644 index 1cc6adf561..0000000000 --- a/skills/deepmd-install/references/source-python.md +++ /dev/null @@ -1,196 +0,0 @@ -# Source installation: Python interface - -Build the Python package from the exact Git commit recorded by the workflow. -Use `doc/install/install-from-source.md` inside that checkout as the authority -for version-specific backend packages and build variables. The published -counterpart is -. - -## Contents - -- [Checkout gate](#checkout-gate) -- [Interpreter and compiler gate](#interpreter-and-compiler-gate) -- [Backend dependencies](#backend-dependencies) -- [Build](#build) -- [Verification](#verification) - -## Checkout gate - -For an absent target directory, clone without checking out a moving branch, -fetch the requested opaque ref, resolve it to a commit, and check out that -commit detached. Render all values literally in one shell call: - -```bash -git clone --no-checkout "" "" && -git -C "" fetch --tags "" "" && -git -C "" checkout --detach \ - "$(git -C "" rev-parse 'FETCH_HEAD^{commit}')" && -git -C "" rev-parse HEAD -``` - -Store the resolved SHA in `source.commit` and re-run `validate_plan.py` with -`--require-resolved-source` before installing backend packages or building -DeePMD-kit. - -```bash -"" "/scripts/validate_plan.py" \ - "" --require-resolved-source -``` - -For an existing checkout, inspect it without changing its branch, remotes, or -working tree: - -```bash -git -C "" status --short -git -C "" remote -v -git -C "" rev-parse HEAD -``` - -If the existing `HEAD` is not the requested ref, or tracked/untracked changes -overlap the build, use a separate clone or ask the user which tree to use. Do -not reset, clean, or repoint `origin`. - -## Interpreter and compiler gate - -Use the plan's absolute interpreter and confirm Python 3.10 or newer. Record -the resolved `sys.prefix` as `environment.prefix` and revalidate the plan if it -differs from the recorded value: - -```bash -"" -c \ - "import sys; print(sys.executable); print(sys.prefix); print(sys.version)" -"" --version -"" --version -``` - -For a CUDA build, verify the selected toolkit and host compiler in the same -call. A CUDA major-version mismatch with the selected backend package is a hard -failure; handle a same-major minor mismatch through the actual configure/build -result rather than string equality alone. - -```bash -"/bin/nvcc" --version -"" --version -``` - -Install build requirements into the same environment: - -```bash -"" -m pip install --upgrade \ - pip scikit-build-core packaging cmake ninja dependency_groups -``` - -## Backend dependencies - -Install every entry in `package.backend_packages` before building DeePMD-kit. -Use the recorded backend index only when it is non-null. - -### PyTorch - -Select the PyTorch requirement and index from the official PyTorch installer or -the checkout documentation. Verify the wheel before compiling DeePMD-kit: - -```bash -"" -c \ - "import torch; print(torch.__version__, torch.version.cuda, torch.version.hip, torch.cuda.is_available())" -``` - -Use the `torch` extra for the source package. It supplies `vesin[torch]` and -the platform-eligible nvalchemi integration. - -### TensorFlow - -Install the TensorFlow package selected by the checkout documentation, then -verify it with the same interpreter: - -```bash -"" -c \ - "import tensorflow as tf; print(tf.__version__, tf.config.list_physical_devices('GPU'))" -``` - -A CUDA DeePMD-kit build requires a local CUDA toolkit visible to CMake. Keep -`DP_ENABLE_TENSORFLOW=1` and `DP_ENABLE_PYTORCH=0` explicit during the build. - -### JAX - -Install the JAX accelerator package selected by the official JAX instructions -and use the DeePMD-kit `jax` extra. JAX may provide the GPU runtime while the -DeePMD-kit compiled variant remains CPU; record `accelerator` and -`build.variant` separately in the plan. - -```bash -"" -c \ - "import jax; print(jax.__version__, jax.devices())" -``` - -### Paddle - -Install the Paddle package and index selected by -`doc/install/easy-install.md` in the checkout. Build the Python package with -TensorFlow and PyTorch disabled unless either backend is also requested. - -```bash -"" -c \ - "import paddle; print(paddle.__version__, paddle.device.get_device())" -``` - -## Build - -Select the source requirement by backend: - -| Backend | Local requirement | Build switches | -| ---------- | ----------------- | ----------------------------------------------- | -| PyTorch | `.[torch]` | `DP_ENABLE_PYTORCH=1`, `DP_ENABLE_TENSORFLOW=0` | -| TensorFlow | `.` | `DP_ENABLE_PYTORCH=0`, `DP_ENABLE_TENSORFLOW=1` | -| JAX | `.[jax]` | `DP_ENABLE_PYTORCH=0`, `DP_ENABLE_TENSORFLOW=0` | -| Paddle | `.` | `DP_ENABLE_PYTORCH=0`, `DP_ENABLE_TENSORFLOW=0` | - -Render a single self-contained invocation. This CUDA PyTorch template shows the -required shape: - -```bash -cd "" && -env \ - CC="" \ - CXX="" \ - CUDA_HOME="" \ - CUDA_PATH="" \ - CUDAToolkit_ROOT="" \ - CUDACXX="/bin/nvcc" \ - CUDAHOSTCXX="" \ - DP_VARIANT=cuda \ - DP_ENABLE_PYTORCH=1 \ - DP_ENABLE_TENSORFLOW=0 \ - DP_ENABLE_NATIVE_OPTIMIZATION=0 \ - CMAKE_BUILD_PARALLEL_LEVEL="" \ - "" -m pip install \ - ".[torch]" --no-build-isolation --verbose -``` - -For CPU, omit CUDA variables and set `DP_VARIANT=cpu`. For ROCm, follow the -selected checkout documentation and set `DP_VARIANT=rocm` plus the planned -`ROCM_ROOT` in the same invocation. - -For TensorFlow, JAX, and Paddle, replace the requirement and the two -`DP_ENABLE_*` values using the table. Do not add a backend the plan does not -request. - -## Verification - -Run outside the source directory so an import cannot succeed from the checkout -alone: - -```bash -cd "" && -"" "/scripts/verify_python.py" \ - --backend "" \ - --accelerator "" \ - --expected-prefix "" \ - --expected-source-commit "" \ - --expected-build-variant "" -``` - -Add `--expected-version ""` when the plan pins a release. Add -`--expect-custom-op` for a PyTorch source build. Add `--expect-nv` and -`--expect-vesin` only when their platform markers and the plan require them. Do -not continue to the C/C++ gate until every requested check passes. diff --git a/skills/deepmd-install/scripts/prepare_lammps.py b/skills/deepmd-install/scripts/prepare_lammps.py deleted file mode 100644 index 7a12aea828..0000000000 --- a/skills/deepmd-install/scripts/prepare_lammps.py +++ /dev/null @@ -1,169 +0,0 @@ -#!/usr/bin/env python3 -# SPDX-License-Identifier: LGPL-3.0-or-later -"""Add one managed DeePMD built-in include to a LAMMPS CMake project.""" - -from __future__ import ( - annotations, -) - -import argparse -import os -import re -import tempfile -from pathlib import ( - Path, -) - -BEGIN_MARKER = "# >>> DeePMD-kit built-in module >>>" -END_MARKER = "# <<< DeePMD-kit built-in module <<<" -LEGACY_PATTERN = re.compile( - r"^\s*include\([^\n]*source/lmp/builtin\.cmake[^\n]*\)\s*$", re.MULTILINE -) - - -def _managed_block(builtin: Path) -> str: - """Return the canonical managed CMake block.""" - return "\n".join( - ( - BEGIN_MARKER, - f"include([=[{builtin}]=])", - END_MARKER, - ) - ) - - -def _replace_managed_block(text: str, block: str) -> tuple[str, bool]: - """Replace a valid managed block or reject malformed markers.""" - begin_count = text.count(BEGIN_MARKER) - end_count = text.count(END_MARKER) - if begin_count == 0 and end_count == 0: - return text, False - if begin_count != 1 or end_count != 1: - raise ValueError("LAMMPS CMakeLists.txt contains malformed DeePMD markers") - begin = text.index(BEGIN_MARKER) - end = text.index(END_MARKER, begin) + len(END_MARKER) - return text[:begin] + block + text[end:], True - - -def render_updated_text(text: str, builtin: Path) -> str: - """Return an idempotently patched CMakeLists.txt. - - Parameters - ---------- - text : str - Existing LAMMPS top-level CMake text. - builtin : Path - Absolute DeePMD-kit `source/lmp/builtin.cmake` path. - - Returns - ------- - str - Text containing exactly one managed include. - - Raises - ------ - ValueError - If multiple unmanaged includes or malformed markers are present. - """ - block = _managed_block(builtin) - updated, replaced = _replace_managed_block(text, block) - if replaced: - outside_block = updated.replace(block, "", 1) - if LEGACY_PATTERN.search(outside_block) is not None: - raise ValueError( - "LAMMPS CMakeLists.txt contains an unmanaged DeePMD include" - ) - return updated - legacy_matches = list(LEGACY_PATTERN.finditer(text)) - if len(legacy_matches) > 1: - raise ValueError("LAMMPS CMakeLists.txt contains multiple DeePMD includes") - if legacy_matches: - match = legacy_matches[0] - return updated[: match.start()] + block + updated[match.end() :] - separator = "" if updated.endswith("\n") else "\n" - return f"{updated}{separator}\n{block}\n" - - -def _write_atomic(path: Path, text: str) -> None: - """Atomically replace a text file while preserving its mode.""" - mode = path.stat().st_mode - descriptor, temporary_name = tempfile.mkstemp( - prefix=f".{path.name}.", suffix=".tmp", dir=path.parent - ) - temporary = Path(temporary_name) - try: - with os.fdopen(descriptor, "w", encoding="utf-8") as stream: - stream.write(text) - os.chmod(temporary, mode & 0o7777) - os.replace(temporary, path) - finally: - if temporary.exists(): - temporary.unlink() - - -def prepare(lammps_source: Path, deepmd_source: Path, *, check: bool) -> bool: - """Prepare or check the managed LAMMPS include. - - Parameters - ---------- - lammps_source : Path - Absolute LAMMPS source directory. - deepmd_source : Path - Absolute DeePMD-kit source directory. - check : bool - If true, compare without writing. - - Returns - ------- - bool - True when the file already had the canonical content. - """ - lammps_source = lammps_source.resolve(strict=True) - deepmd_source = deepmd_source.resolve(strict=True) - cmake_file = lammps_source / "cmake" / "CMakeLists.txt" - builtin = deepmd_source / "source" / "lmp" / "builtin.cmake" - if not cmake_file.is_file(): - raise FileNotFoundError(f"LAMMPS CMake file not found: {cmake_file}") - if not builtin.is_file(): - raise FileNotFoundError(f"DeePMD built-in module not found: {builtin}") - original = cmake_file.read_text(encoding="utf-8") - updated = render_updated_text(original, builtin) - unchanged = updated == original - if not check and not unchanged: - _write_atomic(cmake_file, updated) - return unchanged - - -def main(argv: list[str] | None = None) -> int: - """Parse arguments and prepare the LAMMPS source tree. - - Parameters - ---------- - argv : list of str, optional - Command-line arguments. Defaults to the process arguments. - - Returns - ------- - int - Zero when prepared or already correct, otherwise one. - """ - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--lammps-source", required=True, type=Path) - parser.add_argument("--deepmd-source", required=True, type=Path) - parser.add_argument("--check", action="store_true") - args = parser.parse_args(argv) - try: - unchanged = prepare(args.lammps_source, args.deepmd_source, check=args.check) - except (OSError, ValueError) as exc: - print(f"FAIL prepare_lammps: {exc}") - return 1 - if args.check and not unchanged: - print("FAIL prepare_lammps: managed include differs from the plan") - return 1 - status = "UNCHANGED" if unchanged else "UPDATED" - print(f"PASS prepare_lammps: {status}") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/skills/deepmd-install/scripts/probe_env.py b/skills/deepmd-install/scripts/probe_env.py deleted file mode 100644 index c9f206c686..0000000000 --- a/skills/deepmd-install/scripts/probe_env.py +++ /dev/null @@ -1,426 +0,0 @@ -#!/usr/bin/env python3 -# SPDX-License-Identifier: LGPL-3.0-or-later -"""Probe machine facts required to plan a DeePMD-kit installation.""" - -from __future__ import ( - annotations, -) - -import argparse -import csv -import importlib.metadata -import json -import os -import platform -import shutil -import subprocess -import sys -from pathlib import ( - Path, -) -from typing import ( - TYPE_CHECKING, - Any, -) - -if TYPE_CHECKING: - from collections.abc import ( - Sequence, - ) - -ENVIRONMENT_VARIABLES = ( - "CUDA_HOME", - "CUDA_PATH", - "CUDAToolkit_ROOT", - "ROCM_ROOT", - "ROCM_PATH", - "CUDA_VISIBLE_DEVICES", - "HIP_VISIBLE_DEVICES", - "ROCR_VISIBLE_DEVICES", - "CONDA_PREFIX", - "CONDA_DEFAULT_ENV", - "VIRTUAL_ENV", - "CC", - "CXX", - "CUDACXX", - "CUDAHOSTCXX", - "DP_VARIANT", - "DP_ENABLE_PYTORCH", - "DP_ENABLE_TENSORFLOW", - "PATH", -) -TOOLS = ( - "python", - "python3", - "pip", - "conda", - "mamba", - "git", - "cmake", - "ninja", - "gcc", - "g++", - "clang", - "clang++", - "nvcc", - "hipcc", - "mpicxx", - "nvidia-smi", - "rocm-smi", - "rocminfo", - "docker", - "dp", - "lmp", -) -VERSION_TOOLS = { - "pip": ("--version",), - "conda": ("--version",), - "mamba": ("--version",), - "git": ("--version",), - "cmake": ("--version",), - "ninja": ("--version",), - "gcc": ("--version",), - "g++": ("--version",), - "clang": ("--version",), - "clang++": ("--version",), - "nvcc": ("--version",), - "hipcc": ("--version",), - "mpicxx": ("--version",), - "docker": ("--version",), -} -PACKAGE_DISTRIBUTIONS = { - "deepmd": ("deepmd-kit",), - "pytorch": ("torch",), - "tensorflow": ("tensorflow", "tensorflow-cpu"), - "jax": ("jax",), - "paddle": ("paddlepaddle", "paddlepaddle-gpu"), -} - - -def _run( - command: Sequence[str], *, timeout: float = 10.0, cwd: Path | None = None -) -> dict[str, Any]: - """Run a read-only command and return a structured result.""" - executable = command[0] - resolved = shutil.which(executable) - if resolved is None and not Path(executable).is_file(): - return { - "status": "unavailable", - "returncode": None, - "stdout": "", - "stderr": "", - "output": "", - } - try: - completed = subprocess.run( - list(command), - check=False, - capture_output=True, - text=True, - timeout=timeout, - cwd=cwd, - ) - except subprocess.TimeoutExpired: - return { - "status": "timeout", - "returncode": None, - "stdout": "", - "stderr": "", - "output": "", - } - except OSError as exc: - return { - "status": "error", - "returncode": None, - "stdout": "", - "stderr": f"{type(exc).__name__}: {exc}", - "output": f"{type(exc).__name__}: {exc}", - } - stdout = completed.stdout.strip() - stderr = completed.stderr.strip() - output = "\n".join(part for part in (stdout, stderr) if part) - return { - "status": "ok" if completed.returncode == 0 else "failed", - "returncode": completed.returncode, - "stdout": stdout, - "stderr": stderr, - "output": output, - } - - -def _first_line(value: str) -> str: - """Return the first non-empty output line.""" - return next((line.strip() for line in value.splitlines() if line.strip()), "") - - -def _probe_system() -> dict[str, Any]: - """Return operating-system, memory, CPU, and disk facts.""" - libc_name, libc_version = platform.libc_ver() - memory_total: int | None = None - meminfo = Path("/proc/meminfo") - if meminfo.is_file(): - try: - for line in meminfo.read_text(encoding="utf-8").splitlines(): - if line.startswith("MemTotal:"): - memory_total = int(line.split()[1]) * 1024 - break - except (OSError, ValueError, IndexError): - memory_total = None - elif platform.system() == "Darwin": - result = _run(["sysctl", "-n", "hw.memsize"]) - if result["status"] == "ok": - try: - memory_total = int(result["output"]) - except ValueError: - memory_total = None - disk = shutil.disk_usage(Path.cwd()) - return { - "platform": platform.system(), - "release": platform.release(), - "machine": platform.machine(), - "libc": {"name": libc_name or None, "version": libc_version or None}, - "cpu_count": os.cpu_count(), - "memory_total_bytes": memory_total, - "working_directory": str(Path.cwd()), - "disk_free_bytes": disk.free, - } - - -def _probe_python() -> dict[str, Any]: - """Return facts for the interpreter executing this probe.""" - pip_result = _run([sys.executable, "-m", "pip", "--version"]) - return { - "executable": sys.executable, - "version": platform.python_version(), - "prefix": sys.prefix, - "base_prefix": sys.base_prefix, - "pip": pip_result, - } - - -def _probe_tools() -> dict[str, Any]: - """Return executable paths and concise version strings.""" - result: dict[str, Any] = {} - for name in TOOLS: - path = shutil.which(name) - entry: dict[str, Any] = {"path": path} - if path is not None and name in VERSION_TOOLS: - version = _run([path, *VERSION_TOOLS[name]]) - entry["version"] = ( - _first_line(version["output"]) - if version["status"] in {"ok", "failed"} - else None - ) - entry["version_status"] = version["status"] - result[name] = entry - return result - - -def _query_nvidia(fields: Sequence[str]) -> dict[str, Any]: - """Query NVIDIA GPUs with a fixed CSV field list.""" - result = _run( - [ - "nvidia-smi", - f"--query-gpu={','.join(fields)}", - "--format=csv,noheader,nounits", - ] - ) - if result["status"] != "ok": - return {"status": result["status"], "error": result["output"], "gpus": []} - rows = list(csv.reader(result["output"].splitlines(), skipinitialspace=True)) - gpus = [dict(zip(fields, row, strict=False)) for row in rows if row] - return {"status": "ok", "error": None, "gpus": gpus} - - -def _probe_nvidia() -> dict[str, Any]: - """Return driver, occupancy, and numeric compute capability per GPU.""" - fields = ( - "index", - "uuid", - "name", - "driver_version", - "memory.used", - "memory.total", - "utilization.gpu", - "compute_cap", - ) - result = _query_nvidia(fields) - if result["status"] == "ok": - return result - fallback_fields = fields[:-1] - fallback = _query_nvidia(fallback_fields) - if fallback["status"] == "ok": - for gpu in fallback["gpus"]: - gpu["compute_cap"] = None - fallback["compute_capability_status"] = "unsupported by nvidia-smi" - return fallback - - -def _probe_rocm() -> dict[str, Any]: - """Return ROCm tool availability and concise device output.""" - hipcc = shutil.which("hipcc") - rocminfo = shutil.which("rocminfo") - rocm_smi = shutil.which("rocm-smi") - result: dict[str, Any] = { - "hipcc": hipcc, - "rocminfo": rocminfo, - "rocm_smi": rocm_smi, - } - if hipcc is not None: - output = _run([hipcc, "--version"]) - result["hipcc_version"] = _first_line(output["output"]) - if rocm_smi is not None: - output = _run([rocm_smi, "--showproductname", "--showmeminfo", "vram"]) - result["device_summary"] = output - return result - - -def _distribution_version(names: Sequence[str]) -> dict[str, str] | None: - """Return the first installed distribution name and version.""" - for name in names: - try: - version = importlib.metadata.version(name) - except importlib.metadata.PackageNotFoundError: - continue - return {"distribution": name, "version": version} - return None - - -def _isolated_python_json(code: str) -> dict[str, Any]: - """Run an isolated import probe and decode its JSON output.""" - result = _run( - [sys.executable, "-I", "-c", code], timeout=30.0, cwd=Path(sys.prefix) - ) - if result["status"] != "ok": - return {"status": result["status"], "error": result["output"]} - try: - decoded = json.loads(result["stdout"]) - except json.JSONDecodeError: - return {"status": "invalid-json", "error": result["output"]} - if not isinstance(decoded, dict): - return {"status": "invalid-json", "error": result["output"]} - return {"status": "ok", **decoded} - - -def _probe_torch_runtime() -> dict[str, Any]: - """Return PyTorch accelerator visibility in an isolated interpreter.""" - code = r""" -import json -import torch - -devices = [] -if torch.cuda.is_available(): - for index in range(torch.cuda.device_count()): - devices.append( - { - "index": index, - "name": torch.cuda.get_device_name(index), - "capability": list(torch.cuda.get_device_capability(index)), - } - ) -print( - json.dumps( - { - "version": torch.__version__, - "file": torch.__file__, - "cuda_version": torch.version.cuda, - "hip_version": torch.version.hip, - "accelerator_available": torch.cuda.is_available(), - "devices": devices, - } - ) -) -""" - return _isolated_python_json(code) - - -def _probe_deepmd_runtime() -> dict[str, Any]: - """Return installed DeePMD-kit location and compiled build variant.""" - code = r""" -import json -import deepmd -from deepmd.env import GLOBAL_CONFIG - -result = { - "version": getattr(deepmd, "__version__", "unknown"), - "file": getattr(deepmd, "__file__", None), - "build_variant": GLOBAL_CONFIG.get("dp_variant"), - "enable_pytorch": GLOBAL_CONFIG.get("enable_pytorch"), - "enable_tensorflow": GLOBAL_CONFIG.get("enable_tensorflow"), -} -if result["enable_pytorch"]: - from deepmd.pt.cxx_op import ENABLE_CUSTOMIZED_OP - result["pytorch_custom_op"] = bool(ENABLE_CUSTOMIZED_OP) -print(json.dumps(result)) -""" - return _isolated_python_json(code) - - -def _probe_packages() -> dict[str, Any]: - """Return installed backend distributions and selected runtime details.""" - packages: dict[str, Any] = { - key: _distribution_version(names) - for key, names in PACKAGE_DISTRIBUTIONS.items() - } - if packages["pytorch"] is not None: - packages["pytorch"]["runtime"] = _probe_torch_runtime() - if packages["deepmd"] is not None: - packages["deepmd"]["runtime"] = _probe_deepmd_runtime() - return packages - - -def build_report() -> dict[str, Any]: - """Build the complete read-only probe report. - - Returns - ------- - dict - Machine facts used by the installation planner. - """ - return { - "system": _probe_system(), - "environment": {name: os.environ.get(name) for name in ENVIRONMENT_VARIABLES}, - "python": _probe_python(), - "tools": _probe_tools(), - "nvidia": _probe_nvidia(), - "rocm": _probe_rocm(), - "packages": _probe_packages(), - } - - -def _print_human(report: dict[str, Any]) -> None: - """Print a readable report without discarding structured values.""" - for section, value in report.items(): - print(f"== {section} ==") - print(json.dumps(value, indent=2, sort_keys=True)) - print() - - -def main(argv: list[str] | None = None) -> int: - """Probe the machine and print JSON or human-readable output. - - Parameters - ---------- - argv : list of str, optional - Command-line arguments. Defaults to the process arguments. - - Returns - ------- - int - Always zero; unavailable optional tools are represented in the report. - """ - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument( - "--json", action="store_true", help="Emit one machine-readable JSON object." - ) - args = parser.parse_args(argv) - report = build_report() - if args.json: - print(json.dumps(report, indent=2, sort_keys=True)) - else: - _print_human(report) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/skills/deepmd-install/scripts/validate_plan.py b/skills/deepmd-install/scripts/validate_plan.py deleted file mode 100644 index c86457fbb4..0000000000 --- a/skills/deepmd-install/scripts/validate_plan.py +++ /dev/null @@ -1,818 +0,0 @@ -#!/usr/bin/env python3 -# SPDX-License-Identifier: LGPL-3.0-or-later -"""Validate a DeePMD-kit installation plan before changing the machine.""" - -from __future__ import ( - annotations, -) - -import argparse -import json -import os -import re -from pathlib import ( - Path, -) -from typing import ( - Any, -) -from urllib.parse import ( - urlparse, -) - -SCHEMA_VERSION = 1 -METHODS = {"pip", "conda", "dp1s", "offline", "docker", "source"} -GOALS = {"python", "python+lammps", "python+cpp", "python+cpp+lammps"} -BACKENDS = {"pytorch", "tensorflow", "jax", "paddle"} -ACCELERATORS = {"cpu", "cuda", "rocm"} -ENVIRONMENT_KINDS = {"existing", "venv", "conda", "prefix", "container"} -LAMMPS_FLAVORS = {"host", "kokkos-cuda"} -MODEL_FAMILIES = {"conventional", "dpa4", "dpa4c"} -TOP_LEVEL_KEYS = { - "schema_version", - "method", - "goal", - "backend", - "accelerator", - "environment", - "package", - "source", - "build", - "cpp", - "lammps", - "smoke_test", -} -OBJECT_KEYS = { - "environment": {"kind", "python", "manager", "name", "prefix", "dp1s_home"}, - "package": { - "deepmd_version", - "deepmd_index_url", - "deepmd_extra_index_url", - "backend_packages", - "backend_index_url", - "channels", - "install_lammps", - "install_ipi", - "artifact_url", - "artifact_path", - "sha256", - "docker_image", - "lammps_model_family", - }, - "source": {"directory", "remote", "ref", "commit"}, - "build": { - "variant", - "cc", - "cxx", - "cuda_home", - "rocm_root", - "native_optimization", - "jobs", - }, - "cpp": { - "install_prefix", - "build_directory", - "tensorflow_root", - "tensorflow_c_root", - "paddle_inference_dir", - }, - "lammps": { - "source_directory", - "build_directory", - "version", - "url", - "sha256", - "flavor", - "machine", - "kokkos_arch", - "mpi", - "model_family", - }, - "smoke_test": {"enabled", "gpu", "example"}, -} - - -def _platform_name() -> str: - """Return the operating-system family used for command rendering.""" - return os.name - - -def _is_absolute_path(value: object) -> bool: - """Return whether ``value`` is a non-empty absolute path string.""" - return isinstance(value, str) and bool(value) and Path(value).is_absolute() - - -def _canonical(value: str) -> Path: - """Return a normalized path without requiring it to exist.""" - return Path(value).resolve(strict=False) - - -def _require_keys( - data: dict[str, Any], required: set[str], context: str, errors: list[str] -) -> None: - """Append errors for missing keys in a mapping.""" - for key in sorted(required - data.keys()): - errors.append(f"{context}.{key}: missing required field") - - -def _check_unknown_keys( - data: dict[str, Any], allowed: set[str], context: str, errors: list[str] -) -> None: - """Append errors for undeclared keys in a mapping.""" - for key in sorted(data.keys() - allowed): - errors.append(f"{context}.{key}: unknown field") - - -def _as_object( - plan: dict[str, Any], key: str, *, required: bool, errors: list[str] -) -> dict[str, Any] | None: - """Return a conditional object after validating its type and keys.""" - value = plan.get(key) - if value is None: - if required: - errors.append(f"{key}: object is required for this plan") - return None - if not isinstance(value, dict): - errors.append(f"{key}: expected an object or null") - return None - _check_unknown_keys(value, OBJECT_KEYS[key], key, errors) - return value - - -def _validate_choice( - value: object, choices: set[str], path: str, errors: list[str] -) -> str | None: - """Validate a string enum and return the accepted value.""" - if not isinstance(value, str) or value not in choices: - errors.append(f"{path}: unsupported value {value!r}") - return None - return value - - -def _validate_strings(value: object, path: str, errors: list[str]) -> None: - """Reject unresolved placeholders and unsafe POSIX-template characters.""" - if isinstance(value, dict): - for key, item in value.items(): - _validate_strings(item, f"{path}.{key}" if path else key, errors) - elif isinstance(value, list): - for index, item in enumerate(value): - _validate_strings(item, f"{path}[{index}]", errors) - elif isinstance(value, str): - if "$" in value: - errors.append(f"{path}: unresolved shell variable is not allowed") - if re.search(r"<[^<>\r\n]+>", value): - errors.append(f"{path}: unresolved placeholder is not allowed") - unsafe_characters = ('"', "`") + (("\\",) if _platform_name() != "nt" else ()) - if any(character in value for character in unsafe_characters): - errors.append(f"{path}: unsafe shell-template character is not allowed") - if any(character in value for character in ("\0", "\n", "\r")): - errors.append(f"{path}: control character is not allowed") - - -def _validate_checksum(value: object, path: str, errors: list[str]) -> None: - """Validate an optional SHA-256 checksum.""" - if value is not None and ( - not isinstance(value, str) or re.fullmatch(r"[0-9a-fA-F]{64}", value) is None - ): - errors.append(f"{path}: expected 64 hexadecimal SHA-256 characters") - - -def _validate_https_url(value: object, path: str, errors: list[str]) -> None: - """Validate an HTTPS URL without embedded credentials.""" - if not isinstance(value, str) or not value: - errors.append(f"{path}: expected an HTTPS URL") - return - parsed = urlparse(value) - if ( - parsed.scheme == "https" - and parsed.netloc - and parsed.username is None - and parsed.password is None - and not any(character.isspace() for character in value) - ): - return - errors.append(f"{path}: expected an HTTPS URL without credentials") - - -def _validate_string_list(value: object, path: str, errors: list[str]) -> None: - """Validate a list of non-empty strings.""" - if not isinstance(value, list) or any( - not isinstance(item, str) or not item for item in value - ): - errors.append(f"{path}: expected a list of non-empty strings") - - -def _validate_environment( - environment: dict[str, Any], method: str, errors: list[str] -) -> None: - """Validate method-specific environment fields.""" - _require_keys(environment, {"kind"}, "environment", errors) - kind = _validate_choice( - environment.get("kind"), ENVIRONMENT_KINDS, "environment.kind", errors - ) - if method in {"pip", "source"}: - python = environment.get("python") - if not _is_absolute_path(python): - errors.append( - "environment.python: pip and source methods require an absolute executable" - ) - if not _is_absolute_path(environment.get("prefix")): - errors.append( - "environment.prefix: pip and source methods require an absolute prefix" - ) - if method == "conda": - if not _is_absolute_path(environment.get("manager")): - errors.append("environment.manager: conda requires an absolute executable") - if not isinstance(environment.get("name"), str) or not environment.get("name"): - errors.append( - "environment.name: conda requires a non-empty environment name" - ) - for key in ("python", "prefix"): - if environment.get(key) is not None and not _is_absolute_path( - environment.get(key) - ): - errors.append( - f"environment.{key}: conda requires an absolute path or null" - ) - if method == "dp1s": - if not _is_absolute_path(environment.get("dp1s_home")): - errors.append("environment.dp1s_home: dp1s requires an absolute path") - python = environment.get("python") - prefix = environment.get("prefix") - unresolved = python is None and prefix is None - resolved = _is_absolute_path(python) and _is_absolute_path(prefix) - if not (unresolved or resolved): - errors.append( - "environment: dp1s python and prefix must both be null or absolute paths" - ) - if ( - resolved - and _is_absolute_path(environment.get("dp1s_home")) - and _canonical(environment["dp1s_home"]) == _canonical(prefix) - ): - errors.append( - "environment: dp1s_home and resolved Python prefix must be distinct" - ) - elif environment.get("dp1s_home") is not None: - errors.append("environment.dp1s_home: allowed only for method 'dp1s'") - dp1s_prefix_pending = method == "dp1s" and environment.get("prefix") is None - if ( - kind in {"venv", "prefix"} - and not dp1s_prefix_pending - and not _is_absolute_path(environment.get("prefix")) - ): - errors.append(f"environment.prefix: {kind} requires an absolute path") - if method == "offline" and not _is_absolute_path(environment.get("prefix")): - errors.append("environment.prefix: offline requires an absolute path") - if method == "docker" and kind != "container": - errors.append("environment.kind: docker requires 'container'") - - -def _validate_package( - package: dict[str, Any], - method: str, - goal: str, - backend: str, - accelerator: str, - errors: list[str], -) -> None: - """Validate package and artifact selection.""" - for key in ("backend_packages", "channels"): - if key in package: - _validate_string_list(package[key], f"package.{key}", errors) - channels = package.get("channels") - if method != "conda" and isinstance(channels, list) and channels: - errors.append("package.channels: non-empty channels require method 'conda'") - for key in ("install_lammps", "install_ipi"): - if key in package and not isinstance(package[key], bool): - errors.append(f"package.{key}: expected a boolean") - if package.get("deepmd_version") is not None and ( - not isinstance(package["deepmd_version"], str) or not package["deepmd_version"] - ): - errors.append("package.deepmd_version: expected a non-empty string or null") - if package.get("backend_index_url") is not None: - _validate_https_url( - package["backend_index_url"], "package.backend_index_url", errors - ) - for key in ("deepmd_index_url", "deepmd_extra_index_url"): - if package.get(key) is not None: - _validate_https_url(package[key], f"package.{key}", errors) - _validate_checksum(package.get("sha256"), "package.sha256", errors) - if method == "offline": - _require_keys(package, {"sha256"}, "package", errors) - artifact_url = package.get("artifact_url") - artifact_path = package.get("artifact_path") - if (artifact_url is None) == (artifact_path is None): - errors.append( - "package: offline installation requires exactly one of " - "artifact_url or artifact_path" - ) - elif artifact_url is not None: - _validate_https_url(artifact_url, "package.artifact_url", errors) - elif not _is_absolute_path(artifact_path): - errors.append("package.artifact_path: expected an absolute path") - elif not Path(artifact_path).is_file(): - errors.append( - "package.artifact_path: expected an existing complete artifact file" - ) - if package.get("sha256") is None: - errors.append("package.sha256: offline installation requires a checksum") - elif ( - package.get("artifact_url") is not None - or package.get("artifact_path") is not None - ): - errors.append("package.artifact_url/artifact_path: allowed only for offline") - if method == "docker" and ( - not isinstance(package.get("docker_image"), str) - or not package.get("docker_image") - ): - errors.append("package.docker_image: docker requires an image reference") - if goal == "python+lammps" and package.get("install_lammps") is not True: - errors.append( - "package.install_lammps: python+lammps requires the packaged LAMMPS runtime" - ) - if goal != "python+lammps" and package.get("install_lammps") is True: - errors.append("package.install_lammps: requires goal 'python+lammps'") - if method != "source" and accelerator == "rocm": - errors.append("accelerator: ROCm requires method 'source'") - packaged_lammps = goal == "python+lammps" or package.get("install_lammps") is True - if packaged_lammps: - family = _validate_choice( - package.get("lammps_model_family"), - MODEL_FAMILIES, - "package.lammps_model_family", - errors, - ) - if family in {"dpa4", "dpa4c"} and backend != "pytorch": - errors.append( - f"package.lammps_model_family: {family} requires the PyTorch backend" - ) - elif package.get("lammps_model_family") is not None: - errors.append( - "package.lammps_model_family: allowed only when packaged LAMMPS is requested" - ) - if ( - backend == "paddle" - and method != "source" - and (packaged_lammps or package.get("install_ipi") is True) - ): - errors.append("package: Paddle does not support packaged LAMMPS or i-PI") - - -def _validate_source( - source: dict[str, Any], - build: dict[str, Any], - backend: str, - accelerator: str, - errors: list[str], -) -> None: - """Validate source checkout and build fields.""" - _require_keys(source, {"directory", "remote", "ref"}, "source", errors) - if not _is_absolute_path(source.get("directory")): - errors.append("source.directory: expected an absolute path") - for key in ("remote", "ref"): - if not isinstance(source.get(key), str) or not source.get(key): - errors.append(f"source.{key}: expected a non-empty string") - commit = source.get("commit") - if commit is not None and ( - not isinstance(commit, str) - or re.fullmatch(r"[0-9a-fA-F]{7,40}", commit) is None - ): - errors.append("source.commit: expected a 7-40 character commit SHA or null") - _require_keys( - build, - {"variant", "cc", "cxx", "native_optimization", "jobs"}, - "build", - errors, - ) - variant = _validate_choice( - build.get("variant"), ACCELERATORS, "build.variant", errors - ) - for key in ("cc", "cxx"): - if not _is_absolute_path(build.get(key)): - errors.append(f"build.{key}: expected an absolute executable") - if not isinstance(build.get("native_optimization"), bool): - errors.append("build.native_optimization: expected a boolean") - jobs = build.get("jobs") - if not isinstance(jobs, int) or isinstance(jobs, bool) or not 1 <= jobs <= 256: - errors.append("build.jobs: expected an integer from 1 through 256") - if variant == "cuda" and not _is_absolute_path(build.get("cuda_home")): - errors.append("build.cuda_home: CUDA build requires an absolute path") - if variant == "rocm" and not _is_absolute_path(build.get("rocm_root")): - errors.append("build.rocm_root: ROCm build requires an absolute path") - if backend in {"pytorch", "tensorflow"} and variant != accelerator: - errors.append( - f"build.variant: {backend} source build must match accelerator {accelerator!r}" - ) - - -def _validate_cpp( - cpp: dict[str, Any], - source: dict[str, Any], - environment: dict[str, Any], - backend: str, - errors: list[str], -) -> None: - """Validate C/C++ build paths and backend roots.""" - _require_keys(cpp, {"install_prefix", "build_directory"}, "cpp", errors) - for key in ("install_prefix", "build_directory"): - if not _is_absolute_path(cpp.get(key)): - errors.append(f"cpp.{key}: expected an absolute path") - for key in ("tensorflow_root", "tensorflow_c_root", "paddle_inference_dir"): - if cpp.get(key) is not None and not _is_absolute_path(cpp.get(key)): - errors.append(f"cpp.{key}: expected an absolute path or null") - if not all( - _is_absolute_path(cpp.get(key)) for key in ("install_prefix", "build_directory") - ) or not _is_absolute_path(source.get("directory")): - return - prefix = _canonical(cpp["install_prefix"]) - build_directory = _canonical(cpp["build_directory"]) - source_directory = _canonical(source["directory"]) - if len({prefix, build_directory, source_directory}) != 3: - errors.append("cpp: source, build, and install directories must be distinct") - forbidden = { - Path("/"), - Path("/usr"), - Path("/usr/local"), - Path.home().resolve(strict=False), - (Path.home() / ".local").resolve(strict=False), - } - environment_prefix = environment.get("prefix") - if _is_absolute_path(environment_prefix): - forbidden.add(_canonical(environment_prefix)) - if prefix in forbidden: - errors.append("cpp.install_prefix: select a dedicated, non-shared prefix") - if backend == "jax" and all( - cpp.get(key) is not None for key in ("tensorflow_root", "tensorflow_c_root") - ): - errors.append( - "cpp: JAX tensorflow_root and tensorflow_c_root are mutually exclusive" - ) - if backend == "paddle" and not _is_absolute_path(cpp.get("paddle_inference_dir")): - errors.append("cpp.paddle_inference_dir: Paddle C++ requires an absolute path") - - -def required_lammps_styles(model_family: str, flavor: str) -> list[str]: - """Return the pair styles required by a LAMMPS plan. - - Parameters - ---------- - model_family : str - `conventional`, `dpa4`, or `dpa4c`. - flavor : str - `host` or `kokkos-cuda`. - - Returns - ------- - list of str - Exact pair styles that must appear in `lmp -h`. - """ - base = "dpa4spin" if model_family == "dpa4c" else "deepmd" - styles = [base] - if flavor == "kokkos-cuda": - styles.append(f"{base}/kk") - return styles - - -def _validate_lammps( - lammps: dict[str, Any], - source: dict[str, Any], - build: dict[str, Any], - cpp: dict[str, Any], - backend: str, - accelerator: str, - errors: list[str], -) -> None: - """Validate source LAMMPS and Kokkos choices.""" - _require_keys( - lammps, - { - "source_directory", - "build_directory", - "version", - "flavor", - "mpi", - "model_family", - }, - "lammps", - errors, - ) - for key in ("source_directory", "build_directory"): - if not _is_absolute_path(lammps.get(key)): - errors.append(f"lammps.{key}: expected an absolute path") - if not isinstance(lammps.get("version"), str) or not lammps.get("version"): - errors.append("lammps.version: expected a non-empty string") - if not isinstance(lammps.get("mpi"), bool): - errors.append("lammps.mpi: expected a boolean") - flavor = _validate_choice( - lammps.get("flavor"), LAMMPS_FLAVORS, "lammps.flavor", errors - ) - model_family = _validate_choice( - lammps.get("model_family"), - MODEL_FAMILIES, - "lammps.model_family", - errors, - ) - checksum = lammps.get("sha256") - _validate_checksum(checksum, "lammps.sha256", errors) - source_directory = lammps.get("source_directory") - url = lammps.get("url") - if _is_absolute_path(source_directory): - source_path = Path(source_directory) - if source_path.exists() and not source_path.is_dir(): - errors.append("lammps.source_directory: existing path is not a directory") - elif not source_path.is_dir(): - if url is None: - errors.append( - "lammps.url: required when the source directory is absent" - ) - if checksum is None: - errors.append( - "lammps.sha256: required when the source directory is absent" - ) - if url is not None: - _validate_https_url(url, "lammps.url", errors) - if all( - _is_absolute_path(item) - for item in ( - source.get("directory"), - lammps.get("source_directory"), - lammps.get("build_directory"), - cpp.get("build_directory"), - cpp.get("install_prefix"), - ) - ): - paths = { - _canonical(source["directory"]), - _canonical(lammps["source_directory"]), - _canonical(lammps["build_directory"]), - _canonical(cpp["build_directory"]), - _canonical(cpp["install_prefix"]), - } - if len(paths) != 5: - errors.append( - "lammps: DeePMD source, C/C++ build/install, and LAMMPS " - "source/build paths must differ" - ) - if flavor == "kokkos-cuda": - if backend != "pytorch": - errors.append( - "lammps.flavor: Kokkos CUDA graph pair styles require PyTorch" - ) - if accelerator != "cuda" or build.get("variant") != "cuda": - errors.append("lammps.flavor: Kokkos CUDA requires CUDA runtime and build") - if ( - not isinstance(lammps.get("machine"), str) - or re.fullmatch(r"[a-z0-9][a-z0-9_-]*", lammps.get("machine", "")) is None - ): - errors.append("lammps.machine: expected a lowercase binary suffix") - if ( - not isinstance(lammps.get("kokkos_arch"), str) - or re.fullmatch(r"[A-Z][A-Z0-9_]*", lammps.get("kokkos_arch", "")) is None - ): - errors.append("lammps.kokkos_arch: expected a Kokkos architecture name") - if model_family in {"dpa4", "dpa4c"} and backend != "pytorch": - errors.append( - f"lammps.model_family: {model_family} requires the PyTorch backend" - ) - - -def _validate_smoke_test( - smoke_test: dict[str, Any], accelerator: str, errors: list[str] -) -> None: - """Validate smoke-test input and explicit GPU binding.""" - _require_keys(smoke_test, {"enabled"}, "smoke_test", errors) - enabled = smoke_test.get("enabled") - if not isinstance(enabled, bool): - errors.append("smoke_test.enabled: expected a boolean") - return - if not enabled: - return - if not _is_absolute_path(smoke_test.get("example")): - errors.append("smoke_test.example: enabled test requires an absolute path") - gpu = smoke_test.get("gpu") - if accelerator in {"cuda", "rocm"} and ( - not isinstance(gpu, int) or isinstance(gpu, bool) or gpu < 0 - ): - errors.append( - f"smoke_test.gpu: {accelerator.upper()} test requires a non-negative " - "physical index" - ) - if accelerator == "cpu" and gpu is not None: - errors.append("smoke_test.gpu: CPU test must use null") - - -def validate_plan(plan: object, *, require_resolved_source: bool = False) -> list[str]: - """Validate an installation plan. - - Parameters - ---------- - plan : object - Parsed JSON value. - require_resolved_source : bool, optional - Require `source.commit` for a source build gate. - - Returns - ------- - list of str - Validation errors. An empty list means the plan is valid. - """ - errors: list[str] = [] - if not isinstance(plan, dict): - return ["plan: expected a JSON object"] - _check_unknown_keys(plan, TOP_LEVEL_KEYS, "plan", errors) - _require_keys( - plan, - { - "schema_version", - "method", - "goal", - "backend", - "accelerator", - "environment", - "package", - "smoke_test", - }, - "plan", - errors, - ) - _validate_strings(plan, "", errors) - schema_version = plan.get("schema_version") - if ( - not isinstance(schema_version, int) - or isinstance(schema_version, bool) - or schema_version != SCHEMA_VERSION - ): - errors.append(f"schema_version: expected {SCHEMA_VERSION}") - method = _validate_choice(plan.get("method"), METHODS, "method", errors) - goal = _validate_choice(plan.get("goal"), GOALS, "goal", errors) - backend = _validate_choice(plan.get("backend"), BACKENDS, "backend", errors) - accelerator = _validate_choice( - plan.get("accelerator"), ACCELERATORS, "accelerator", errors - ) - - environment = _as_object(plan, "environment", required=True, errors=errors) - package = _as_object(plan, "package", required=True, errors=errors) - smoke_test = _as_object(plan, "smoke_test", required=True, errors=errors) - if environment is not None and method is not None: - _validate_environment(environment, method, errors) - if ( - package is not None - and method is not None - and goal is not None - and backend is not None - and accelerator is not None - ): - _validate_package(package, method, goal, backend, accelerator, errors) - if smoke_test is not None and accelerator is not None: - _validate_smoke_test(smoke_test, accelerator, errors) - - source_required = method == "source" - source = _as_object(plan, "source", required=source_required, errors=errors) - build = _as_object(plan, "build", required=source_required, errors=errors) - if ( - require_resolved_source - and method == "source" - and source is not None - and source.get("commit") is None - ): - errors.append("source.commit: resolved source SHA is required for this gate") - if method != "source" and (source is not None or build is not None): - errors.append("source/build: allowed only when method is 'source'") - if ( - source is not None - and build is not None - and backend is not None - and accelerator is not None - ): - _validate_source(source, build, backend, accelerator, errors) - - cpp_required = goal in {"python+cpp", "python+cpp+lammps"} - cpp = _as_object(plan, "cpp", required=cpp_required, errors=errors) - if cpp_required and method != "source": - errors.append("goal: C/C++ goals require method 'source'") - if not cpp_required and cpp is not None: - errors.append("cpp: object is allowed only for a C/C++ goal") - if ( - cpp is not None - and source is not None - and environment is not None - and backend is not None - ): - _validate_cpp(cpp, source, environment, backend, errors) - - source_lammps_required = goal == "python+cpp+lammps" - lammps = _as_object(plan, "lammps", required=source_lammps_required, errors=errors) - if source_lammps_required and method != "source": - errors.append("goal: source LAMMPS requires method 'source'") - if not source_lammps_required and lammps is not None: - errors.append("lammps: object is allowed only for python+cpp+lammps") - if ( - lammps is not None - and source is not None - and build is not None - and cpp is not None - and backend is not None - and accelerator is not None - ): - _validate_lammps(lammps, source, build, cpp, backend, accelerator, errors) - - if goal == "python+lammps" and method == "source": - errors.append("goal: source-built LAMMPS also requires the C/C++ gate") - if goal == "python+lammps" and method not in { - "pip", - "conda", - "dp1s", - "offline", - "docker", - }: - errors.append("goal: packaged LAMMPS requires an easy-install method") - return errors - - -def _load_plan(path: Path) -> object: - """Load a JSON plan from disk.""" - with path.open(encoding="utf-8") as stream: - return json.load(stream) - - -def main(argv: list[str] | None = None) -> int: - """Validate a plan file and print a validation summary. - - Parameters - ---------- - argv : list of str, optional - Command-line arguments. Defaults to the process arguments. - - Returns - ------- - int - Zero for a valid plan, otherwise one. - """ - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("plan", type=Path, help="Path to install-plan.json.") - parser.add_argument( - "--json", action="store_true", help="Emit a machine-readable result." - ) - parser.add_argument( - "--require-resolved-source", - action="store_true", - help="Require source.commit before a source build gate.", - ) - args = parser.parse_args(argv) - try: - plan = _load_plan(args.plan) - except FileNotFoundError: - errors = [f"plan: file not found: {args.plan}"] - plan = None - except PermissionError: - errors = [f"plan: permission denied: {args.plan}"] - plan = None - except json.JSONDecodeError as exc: - errors = [f"plan: invalid JSON at line {exc.lineno}, column {exc.colno}"] - plan = None - else: - errors = validate_plan( - plan, require_resolved_source=args.require_resolved_source - ) - - result: dict[str, Any] = {"valid": not errors, "errors": errors} - if isinstance(plan, dict): - result["method"] = plan.get("method") - result["goal"] = plan.get("goal") - result["backend"] = plan.get("backend") - result["accelerator"] = plan.get("accelerator") - lammps = plan.get("lammps") - if isinstance(lammps, dict): - family = lammps.get("model_family") - flavor = lammps.get("flavor") - if ( - isinstance(family, str) - and family in MODEL_FAMILIES - and isinstance(flavor, str) - and flavor in LAMMPS_FLAVORS - ): - result["required_lammps_styles"] = required_lammps_styles( - family, flavor - ) - if args.json: - print(json.dumps(result, indent=2, sort_keys=True)) - elif errors: - for error in errors: - print(f"FAIL {error}") - else: - print( - "PASS plan: " - f"method={result['method']} goal={result['goal']} " - f"backend={result['backend']} accelerator={result['accelerator']}" - ) - if "required_lammps_styles" in result: - print( - "PASS required_lammps_styles: " - + ", ".join(result["required_lammps_styles"]) - ) - return 0 if not errors else 1 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/skills/deepmd-install/scripts/verify_lammps.py b/skills/deepmd-install/scripts/verify_lammps.py deleted file mode 100644 index cc8d59be7b..0000000000 --- a/skills/deepmd-install/scripts/verify_lammps.py +++ /dev/null @@ -1,188 +0,0 @@ -#!/usr/bin/env python3 -# SPDX-License-Identifier: LGPL-3.0-or-later -"""Verify a DeePMD-enabled LAMMPS binary and its required pair styles.""" - -from __future__ import ( - annotations, -) - -import argparse -import json -import re -import shutil -import subprocess -from dataclasses import ( - asdict, - dataclass, -) -from pathlib import ( - Path, -) -from typing import ( - Any, -) - -from verify_native import ( - check_dynamic_links, -) - - -@dataclass(frozen=True) -class CheckResult: - """Represent one LAMMPS verification result.""" - - name: str - passed: bool - detail: str - - -def required_styles(model_family: str, flavor: str) -> list[str]: - """Return exact pair styles required by a runtime selection. - - Parameters - ---------- - model_family : str - `conventional`, `dpa4`, or `dpa4c`. - flavor : str - `host` or `kokkos-cuda`. - - Returns - ------- - list of str - Required pair-style tokens. - """ - base = "dpa4spin" if model_family == "dpa4c" else "deepmd" - result = [base] - if flavor == "kokkos-cuda": - result.append(f"{base}/kk") - return result - - -def _resolve_binary(value: str) -> Path | None: - """Resolve an absolute path or executable name.""" - candidate = Path(value) - if candidate.is_absolute(): - return candidate - resolved = shutil.which(value) - return Path(resolved) if resolved is not None else None - - -def _run( - command: list[str], *, timeout: float = 30.0 -) -> subprocess.CompletedProcess[str]: - """Run a bounded LAMMPS verification command.""" - return subprocess.run( - command, - check=False, - capture_output=True, - text=True, - timeout=timeout, - ) - - -def _check_help(binary: Path, styles: list[str]) -> list[CheckResult]: - """Run LAMMPS help and check exact style tokens.""" - try: - completed = _run([str(binary), "-h"]) - except (OSError, subprocess.TimeoutExpired) as exc: - return [CheckResult("lammps_help", False, f"{type(exc).__name__}: {exc}")] - output = "\n".join((completed.stdout, completed.stderr)) - results = [ - CheckResult( - "lammps_help", - completed.returncode == 0, - f"binary={binary} returncode={completed.returncode}", - ) - ] - tokens = set(re.findall(r"[A-Za-z0-9_.+/-]+", output)) - for style in styles: - results.append(CheckResult(f"pair_style:{style}", style in tokens, style)) - return results - - -def run_checks( - *, binary: str, model_family: str, flavor: str, check_links: bool -) -> list[CheckResult]: - """Verify a LAMMPS executable. - - Parameters - ---------- - binary : str - Absolute path or executable name. - model_family : str - Model family used to derive pair styles. - flavor : str - Host or Kokkos CUDA build. - check_links : bool - Whether to inspect dynamic-library resolution. - - Returns - ------- - list of CheckResult - Ordered verification results. - """ - resolved = _resolve_binary(binary) - if resolved is None or not resolved.is_file(): - return [CheckResult("lammps_binary", False, f"not found: {binary}")] - if not resolved.stat().st_mode & 0o111: - return [CheckResult("lammps_binary", False, f"not executable: {resolved}")] - results = [CheckResult("lammps_binary", True, str(resolved))] - results.extend(_check_help(resolved, required_styles(model_family, flavor))) - if check_links: - link_result = check_dynamic_links(resolved, darwin_probe=[str(resolved), "-h"]) - results.append( - CheckResult("dynamic_links", link_result.passed, link_result.detail) - ) - return results - - -def _print_results(checks: list[CheckResult], *, as_json: bool) -> None: - """Print verification results.""" - if as_json: - payload: dict[str, Any] = { - "passed": all(item.passed for item in checks), - "checks": [asdict(item) for item in checks], - } - print(json.dumps(payload, indent=2, sort_keys=True)) - return - for item in checks: - status = "PASS" if item.passed else "FAIL" - print(f"{status:<5} {item.name}: {item.detail}") - - -def main(argv: list[str] | None = None) -> int: - """Parse arguments and verify LAMMPS. - - Parameters - ---------- - argv : list of str, optional - Command-line arguments. Defaults to the process arguments. - - Returns - ------- - int - Zero when every requested check passes, otherwise one. - """ - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--binary", required=True) - parser.add_argument( - "--model-family", - required=True, - choices=("conventional", "dpa4", "dpa4c"), - ) - parser.add_argument("--flavor", required=True, choices=("host", "kokkos-cuda")) - parser.add_argument("--check-links", action="store_true") - parser.add_argument("--json", action="store_true") - args = parser.parse_args(argv) - checks = run_checks( - binary=args.binary, - model_family=args.model_family, - flavor=args.flavor, - check_links=args.check_links, - ) - _print_results(checks, as_json=args.json) - return 0 if all(item.passed for item in checks) else 1 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/skills/deepmd-install/scripts/verify_native.py b/skills/deepmd-install/scripts/verify_native.py deleted file mode 100644 index dd6e4fb4e0..0000000000 --- a/skills/deepmd-install/scripts/verify_native.py +++ /dev/null @@ -1,162 +0,0 @@ -#!/usr/bin/env python3 -# SPDX-License-Identifier: LGPL-3.0-or-later -"""Verify dynamic-library resolution for native libraries.""" - -from __future__ import ( - annotations, -) - -import argparse -import platform -import shutil -import subprocess -import sys -from dataclasses import ( - dataclass, -) -from pathlib import ( - Path, -) - - -@dataclass(frozen=True) -class LinkResult: - """Represent one dynamic-link inspection result.""" - - path: Path - passed: bool - detail: str - - -def _system_name() -> str: - """Return the host operating-system name.""" - return platform.system() - - -def _find_executable(name: str) -> str | None: - """Return the absolute path of an executable available on PATH.""" - return shutil.which(name) - - -def _darwin_load_command(path: Path) -> list[str]: - """Build an isolated dyld probe for a loadable Mach-O library.""" - loader = ( - "import ctypes, os, sys; " - "mode = getattr(os, 'RTLD_LOCAL', 0) | getattr(os, 'RTLD_NOW', 0); " - "ctypes.CDLL(sys.argv[1], mode=mode)" - ) - return [sys.executable, "-c", loader, str(path)] - - -def _default_pattern(system_name: str | None = None) -> str: - """Return the platform-specific DeePMD native-library glob.""" - selected_system = _system_name() if system_name is None else system_name - return "libdeepmd*.dylib" if selected_system == "Darwin" else "libdeepmd*.so" - - -def _run(command: list[str]) -> subprocess.CompletedProcess[str]: - """Run a bounded native-link inspection command.""" - return subprocess.run( - command, - check=False, - capture_output=True, - text=True, - timeout=30, - ) - - -def check_dynamic_links( - path: Path, *, darwin_probe: list[str] | None = None -) -> LinkResult: - """Check one native file for unresolved dynamic dependencies. - - Parameters - ---------- - path : Path - Native executable or library to inspect. - darwin_probe : list of str, optional - Command that loads a Mach-O executable. Libraries use an isolated - ``ctypes.CDLL`` probe when this argument is omitted. - - Returns - ------- - LinkResult - Inspection status and actionable detail. - """ - path = path.resolve(strict=False) - if not path.is_file(): - return LinkResult(path, False, "file not found") - system = _system_name() - if system == "Linux": - tool = _find_executable("ldd") - command = [tool, str(path)] if tool is not None else None - elif system == "Darwin": - command = darwin_probe or _darwin_load_command(path) - else: - return LinkResult(path, False, f"unsupported platform: {system}") - if command is None: - return LinkResult(path, False, "link inspection tool unavailable") - try: - completed = _run(command) - except (OSError, subprocess.TimeoutExpired) as exc: - return LinkResult(path, False, f"{type(exc).__name__}: {exc}") - output = "\n".join((completed.stdout, completed.stderr)) - unresolved = [ - line.strip() for line in output.splitlines() if "not found" in line.lower() - ] - if unresolved: - return LinkResult(path, False, "; ".join(unresolved)) - if completed.returncode != 0: - details = [line.strip() for line in output.splitlines() if line.strip()] - suffix = f": {'; '.join(details[-4:])}" if details else "" - return LinkResult(path, False, f"returncode={completed.returncode}{suffix}") - return LinkResult(path, True, "all dynamic dependencies resolved") - - -def _collect_paths( - explicit: list[Path], directory: Path | None, pattern: str -) -> list[Path]: - """Collect unique native paths from CLI selectors.""" - paths = [item.resolve(strict=False) for item in explicit] - if directory is not None: - paths.extend(item.resolve(strict=False) for item in directory.glob(pattern)) - return list(dict.fromkeys(paths)) - - -def main(argv: list[str] | None = None) -> int: - """Parse arguments and inspect native files. - - Parameters - ---------- - argv : list of str, optional - Command-line arguments. Defaults to the process arguments. - - Returns - ------- - int - Zero when every selected file resolves, otherwise one. - """ - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument( - "--path", - action="append", - default=[], - type=Path, - help="Native library path.", - ) - parser.add_argument("--directory", type=Path) - parser.add_argument("--pattern", default=_default_pattern()) - args = parser.parse_args(argv) - paths = _collect_paths(args.path, args.directory, args.pattern) - if not paths: - print("FAIL dynamic_links: no native files matched") - return 1 - results = [check_dynamic_links(path) for path in paths] - for result in results: - status = "PASS" if result.passed else "FAIL" - print(f"{status:<5} dynamic_links:{result.path}: {result.detail}") - return 0 if all(result.passed for result in results) else 1 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/skills/deepmd-install/scripts/verify_python.py b/skills/deepmd-install/scripts/verify_python.py deleted file mode 100644 index 8f59b78557..0000000000 --- a/skills/deepmd-install/scripts/verify_python.py +++ /dev/null @@ -1,488 +0,0 @@ -#!/usr/bin/env python3 -# SPDX-License-Identifier: LGPL-3.0-or-later -"""Verify a DeePMD-kit Python install for one backend and accelerator.""" - -from __future__ import ( - annotations, -) - -import argparse -import json -import re -import sys -from dataclasses import ( - asdict, - dataclass, -) -from pathlib import ( - Path, -) -from typing import ( - Any, -) - - -@dataclass(frozen=True) -class CheckResult: - """Represent one verification result.""" - - name: str - passed: bool - detail: str - - -def _result(name: str, passed: bool, detail: object) -> CheckResult: - """Create a normalized check result.""" - return CheckResult(name=name, passed=passed, detail=str(detail)) - - -def _interpreter_prefix() -> Path: - """Return the resolved prefix of the running interpreter.""" - return Path(sys.prefix).resolve(strict=False) - - -def _check_deepmd( - expected_version: str | None, expected_prefix: str | None -) -> CheckResult: - """Import DeePMD-kit and verify its requested identity.""" - try: - import deepmd - except (ImportError, OSError, RuntimeError) as exc: - return _result("deepmd", False, f"{type(exc).__name__}: {exc}") - version = getattr(deepmd, "__version__", "unknown") - location_value = getattr(deepmd, "__file__", None) - if not isinstance(location_value, str) or not location_value: - return _result("deepmd", False, f"version={version} file=missing") - try: - location = Path(location_value).resolve(strict=True) - except (OSError, RuntimeError) as exc: - return _result( - "deepmd", - False, - f"version={version} file={location_value} {type(exc).__name__}: {exc}", - ) - actual_prefix = _interpreter_prefix() - passed = True - identity = [f"version={version}", f"file={location}", f"prefix={actual_prefix}"] - if expected_version is not None: - passed = passed and version == expected_version - identity.append(f"expected_version={expected_version}") - if expected_prefix is not None: - expected = Path(expected_prefix).resolve(strict=False) - imported_from_prefix = location == expected or expected in location.parents - passed = passed and actual_prefix == expected and imported_from_prefix - identity.append(f"expected_prefix={expected}") - return _result("deepmd", passed, " ".join(identity)) - - -def _tensorflow_accelerator(tf: Any) -> str | None: - """Return TensorFlow's compiled GPU runtime from build metadata.""" - try: - build_info = tf.sysconfig.get_build_info() - except (AttributeError, RuntimeError, TypeError): - return None - if not isinstance(build_info, dict): - return None - if build_info.get("is_rocm_build") is True: - return "rocm" - if build_info.get("is_cuda_build") is True: - return "cuda" - return None - - -def _jax_accelerator(devices: list[Any]) -> str | None: - """Return JAX's unambiguous GPU runtime from backend metadata.""" - runtimes: set[str] = set() - for device in devices: - client = getattr(device, "client", None) - canonical_platforms = ( - str(getattr(client, "platform", "")).lower(), - str(getattr(device, "platform", "")).lower(), - ) - runtime = next( - (item for item in canonical_platforms if item in {"cuda", "rocm"}), - None, - ) - if runtime is None: - platform_version = str(getattr(client, "platform_version", "")).lower() - is_cuda = "cuda" in platform_version - is_rocm = "rocm" in platform_version or "hip" in platform_version - if is_cuda != is_rocm: - runtime = "cuda" if is_cuda else "rocm" - if runtime is not None: - runtimes.add(runtime) - return next(iter(runtimes)) if len(runtimes) == 1 else None - - -def _check_build_variant(expected: str) -> CheckResult: - """Check DeePMD-kit's recorded compiled build variant.""" - try: - from deepmd.env import ( - GLOBAL_CONFIG, - ) - except (ImportError, OSError, RuntimeError) as exc: - return _result("build_variant", False, f"{type(exc).__name__}: {exc}") - actual = str(GLOBAL_CONFIG.get("dp_variant", "unknown")).lower() - return _result( - "build_variant", - actual == expected, - f"expected={expected} actual={actual}", - ) - - -def _commits_match(actual: str, expected: str) -> bool: - """Return whether full or abbreviated hexadecimal commits identify one SHA.""" - actual = actual.strip().lower() - expected = expected.strip().lower() - if re.fullmatch(r"[0-9a-f]{7,40}", actual) is None: - return False - if re.fullmatch(r"[0-9a-f]{7,40}", expected) is None: - return False - return actual.startswith(expected) or expected.startswith(actual) - - -def _check_source_commit(expected: str) -> CheckResult: - """Check the source commit recorded in DeePMD-kit's build metadata.""" - try: - from deepmd.env import ( - GLOBAL_CONFIG, - ) - except (ImportError, OSError, RuntimeError) as exc: - return _result("source_commit", False, f"{type(exc).__name__}: {exc}") - actual = str(GLOBAL_CONFIG.get("git_hash", "")) - return _result( - "source_commit", - _commits_match(actual, expected), - f"expected={expected} actual={actual or 'unknown'}", - ) - - -def _check_pytorch(accelerator: str) -> list[CheckResult]: - """Verify the PyTorch backend and a minimal tensor operation.""" - try: - import torch - - import deepmd.pt # noqa: F401 - except (ImportError, OSError, RuntimeError) as exc: - return [_result("pytorch", False, f"{type(exc).__name__}: {exc}")] - details = ( - f"version={torch.__version__} cuda={torch.version.cuda} " - f"hip={torch.version.hip} available={torch.cuda.is_available()}" - ) - results = [_result("pytorch", True, details)] - if accelerator == "cpu": - tensor = torch.ones(1, device="cpu") + 1 - results.append(_result("pytorch_tensor", bool(tensor.item() == 2), "cpu")) - return results - if accelerator == "cuda" and torch.version.cuda is None: - results.append(_result("pytorch_accelerator", False, "CUDA wheel required")) - return results - if accelerator == "rocm" and torch.version.hip is None: - results.append(_result("pytorch_accelerator", False, "ROCm wheel required")) - return results - if not torch.cuda.is_available(): - results.append(_result("pytorch_accelerator", False, "no visible GPU")) - return results - try: - tensor = torch.ones(1, device="cuda:0") + 1 - torch.cuda.synchronize(0) - except RuntimeError as exc: - results.append(_result("pytorch_tensor", False, exc)) - return results - detail = f"device={torch.cuda.get_device_name(0)} value={tensor.item()}" - results.append(_result("pytorch_tensor", bool(tensor.item() == 2), detail)) - return results - - -def _check_tensorflow(accelerator: str) -> list[CheckResult]: - """Verify the TensorFlow backend and a minimal tensor operation.""" - try: - import tensorflow as tf - - import deepmd.tf # noqa: F401 - except (ImportError, OSError, RuntimeError) as exc: - return [_result("tensorflow", False, f"{type(exc).__name__}: {exc}")] - results = [_result("tensorflow", True, f"version={tf.__version__}")] - devices = tf.config.list_physical_devices("GPU") - if accelerator == "cpu": - device = "/CPU:0" - elif not devices: - results.append(_result("tensorflow_accelerator", False, "no visible GPU")) - return results - else: - runtime = _tensorflow_accelerator(tf) - if runtime != accelerator: - results.append( - _result( - "tensorflow_accelerator", - False, - f"expected={accelerator} actual={runtime or 'unknown'}", - ) - ) - return results - device = "/GPU:0" - try: - with tf.device(device): - value = float(tf.reduce_sum(tf.ones((2,), dtype=tf.float32)).numpy()) - except (RuntimeError, ValueError) as exc: - results.append(_result("tensorflow_tensor", False, exc)) - return results - results.append(_result("tensorflow_tensor", value == 2.0, f"device={device}")) - return results - - -def _check_jax(accelerator: str) -> list[CheckResult]: - """Verify the JAX backend and a minimal device operation.""" - try: - import jax - import jax.numpy as jnp - - import deepmd.jax # noqa: F401 - except (ImportError, OSError, RuntimeError) as exc: - return [_result("jax", False, f"{type(exc).__name__}: {exc}")] - devices = list(jax.devices()) - results = [ - _result( - "jax", - True, - f"version={jax.__version__} devices={[str(item) for item in devices]}", - ) - ] - if accelerator == "cpu": - candidates = [item for item in devices if item.platform == "cpu"] - else: - candidates = [ - item for item in devices if item.platform in {"gpu", "cuda", "rocm"} - ] - if not candidates: - results.append(_result("jax_accelerator", False, f"no {accelerator} device")) - return results - if accelerator != "cpu": - runtime = _jax_accelerator(candidates) - if runtime != accelerator: - results.append( - _result( - "jax_accelerator", - False, - f"expected={accelerator} actual={runtime or 'unknown'}", - ) - ) - return results - try: - value = jax.device_put(jnp.ones((2,)), candidates[0]).sum() - value.block_until_ready() - scalar = float(value) - except (RuntimeError, ValueError) as exc: - results.append(_result("jax_tensor", False, exc)) - return results - results.append(_result("jax_tensor", scalar == 2.0, f"device={candidates[0]}")) - return results - - -def _check_paddle(accelerator: str) -> list[CheckResult]: - """Verify the Paddle backend and a minimal tensor operation.""" - try: - import paddle - - import deepmd.pd # noqa: F401 - except (ImportError, OSError, RuntimeError) as exc: - return [_result("paddle", False, f"{type(exc).__name__}: {exc}")] - results = [_result("paddle", True, f"version={paddle.__version__}")] - if accelerator == "cpu": - device = "cpu" - elif accelerator == "cuda": - if ( - not paddle.device.is_compiled_with_cuda() - or paddle.device.cuda.device_count() < 1 - ): - results.append(_result("paddle_accelerator", False, "no visible CUDA GPU")) - return results - device = "gpu:0" - else: - is_rocm = getattr(paddle.device, "is_compiled_with_rocm", lambda: False) - if not is_rocm(): - results.append(_result("paddle_accelerator", False, "ROCm build required")) - return results - device = "gpu:0" - try: - paddle.set_device(device) - value = float(paddle.sum(paddle.ones([2], dtype="float32")).item()) - except (RuntimeError, ValueError) as exc: - results.append(_result("paddle_tensor", False, exc)) - return results - results.append(_result("paddle_tensor", value == 2.0, f"device={device}")) - return results - - -def _check_custom_op() -> CheckResult: - """Require the PyTorch customized operation library.""" - try: - from deepmd.pt.cxx_op import ( - ENABLE_CUSTOMIZED_OP, - ) - except (ImportError, OSError, RuntimeError) as exc: - return _result("pytorch_custom_op", False, f"{type(exc).__name__}: {exc}") - return _result( - "pytorch_custom_op", bool(ENABLE_CUSTOMIZED_OP), ENABLE_CUSTOMIZED_OP - ) - - -def _check_nv() -> CheckResult: - """Require the nvalchemi neighbor-list integration.""" - try: - from deepmd.pt.utils.nv_nlist import ( - is_nv_available, - ) - except (ImportError, OSError, RuntimeError) as exc: - return _result("nvalchemi", False, f"{type(exc).__name__}: {exc}") - available = bool(is_nv_available()) - return _result("nvalchemi", available, available) - - -def _check_vesin() -> CheckResult: - """Require the vesin.torch neighbor-list integration.""" - try: - from deepmd.pt_expt.utils.vesin_neighbor_list import ( - is_vesin_torch_available, - ) - except (ImportError, OSError, RuntimeError) as exc: - return _result("vesin", False, f"{type(exc).__name__}: {exc}") - available = bool(is_vesin_torch_available()) - return _result("vesin", available, available) - - -def run_checks( - *, - backend: str, - accelerator: str, - expected_version: str | None, - expected_prefix: str | None, - expected_source_commit: str | None, - expected_build_variant: str | None, - expect_custom_op: bool, - expect_nv: bool, - expect_vesin: bool, -) -> list[CheckResult]: - """Run all checks requested by the installation plan. - - Parameters - ---------- - backend : str - Backend name. - accelerator : str - Runtime accelerator. - expected_version : str, optional - Required DeePMD-kit version. - expected_prefix : str, optional - Required Python environment prefix. - expected_source_commit : str, optional - Required source commit recorded at build time. - expected_build_variant : str, optional - Required DeePMD-kit compiled variant. - expect_custom_op : bool - Require the PyTorch custom operation library. - expect_nv : bool - Require nvalchemi. - expect_vesin : bool - Require vesin.torch. - - Returns - ------- - list of CheckResult - Ordered verification results. - """ - checks = [_check_deepmd(expected_version, expected_prefix)] - if expected_source_commit is not None: - checks.append(_check_source_commit(expected_source_commit)) - if expected_build_variant is not None: - checks.append(_check_build_variant(expected_build_variant)) - backend_checks = { - "pytorch": _check_pytorch, - "tensorflow": _check_tensorflow, - "jax": _check_jax, - "paddle": _check_paddle, - } - checks.extend(backend_checks[backend](accelerator)) - if expect_custom_op: - checks.append(_check_custom_op()) - if expect_nv: - checks.append(_check_nv()) - if expect_vesin: - checks.append(_check_vesin()) - return checks - - -def _print_results(checks: list[CheckResult], *, as_json: bool) -> None: - """Print verification results.""" - if as_json: - payload: dict[str, Any] = { - "passed": all(item.passed for item in checks), - "checks": [asdict(item) for item in checks], - } - print(json.dumps(payload, indent=2, sort_keys=True)) - return - for item in checks: - status = "PASS" if item.passed else "FAIL" - print(f"{status:<5} {item.name}: {item.detail}") - - -def main(argv: list[str] | None = None) -> int: - """Parse arguments and verify the requested Python runtime. - - Parameters - ---------- - argv : list of str, optional - Command-line arguments. Defaults to the process arguments. - - Returns - ------- - int - Zero when every requested check passes, otherwise one. - """ - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument( - "--backend", - required=True, - choices=("pytorch", "tensorflow", "jax", "paddle"), - ) - parser.add_argument("--accelerator", required=True, choices=("cpu", "cuda", "rocm")) - parser.add_argument("--expected-version") - parser.add_argument("--expected-prefix") - parser.add_argument("--expected-source-commit") - parser.add_argument("--expected-build-variant", choices=("cpu", "cuda", "rocm")) - parser.add_argument("--expect-custom-op", action="store_true") - parser.add_argument("--expect-nv", action="store_true") - parser.add_argument("--expect-vesin", action="store_true") - parser.add_argument("--json", action="store_true") - args = parser.parse_args(argv) - if args.backend != "pytorch" and any( - (args.expect_custom_op, args.expect_nv, args.expect_vesin) - ): - parser.error("PyTorch-only checks require --backend pytorch") - if ( - args.expected_prefix is not None - and not Path(args.expected_prefix).is_absolute() - ): - parser.error("--expected-prefix must be absolute") - if ( - args.expected_source_commit is not None - and re.fullmatch(r"[0-9a-fA-F]{7,40}", args.expected_source_commit) is None - ): - parser.error("--expected-source-commit must be a 7-40 character commit SHA") - checks = run_checks( - backend=args.backend, - accelerator=args.accelerator, - expected_version=args.expected_version, - expected_prefix=args.expected_prefix, - expected_source_commit=args.expected_source_commit, - expected_build_variant=args.expected_build_variant, - expect_custom_op=args.expect_custom_op, - expect_nv=args.expect_nv, - expect_vesin=args.expect_vesin, - ) - _print_results(checks, as_json=args.json) - return 0 if all(item.passed for item in checks) else 1 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/source/tests/test_deepmd_install_skill.py b/source/tests/test_deepmd_install_skill.py deleted file mode 100644 index 5a4fe3b865..0000000000 --- a/source/tests/test_deepmd_install_skill.py +++ /dev/null @@ -1,1075 +0,0 @@ -# SPDX-License-Identifier: LGPL-3.0-or-later -"""Tests for the DeePMD-kit installation skill helpers.""" - -from __future__ import ( - annotations, -) - -import importlib.util -import json -import subprocess -import sys -from pathlib import ( - Path, -) -from types import ( - ModuleType, - SimpleNamespace, -) - -import pytest - -REPOSITORY_ROOT = Path(__file__).resolve().parents[2] -SCRIPT_DIRECTORY = REPOSITORY_ROOT / "skills" / "deepmd-install" / "scripts" - - -def _load_script(name: str) -> ModuleType: - """Load one helper script as a module.""" - path = SCRIPT_DIRECTORY / f"{name}.py" - spec = importlib.util.spec_from_file_location(f"deepmd_install_{name}", path) - assert spec is not None - assert spec.loader is not None - module = importlib.util.module_from_spec(spec) - sys.modules[spec.name] = module - spec.loader.exec_module(module) - return module - - -PLAN = _load_script("validate_plan") -PREPARE = _load_script("prepare_lammps") -NATIVE = _load_script("verify_native") -VERIFY = _load_script("verify_python") - - -def _source_plan() -> dict[str, object]: - """Return a valid source CUDA DPA4C plan.""" - return { - "schema_version": 1, - "method": "source", - "goal": "python+cpp+lammps", - "backend": "pytorch", - "accelerator": "cuda", - "environment": { - "kind": "existing", - "python": "/opt/deepmd/bin/python", - "manager": None, - "name": None, - "prefix": "/opt/deepmd", - "dp1s_home": None, - }, - "package": { - "deepmd_version": None, - "deepmd_index_url": None, - "deepmd_extra_index_url": None, - "backend_packages": [], - "backend_index_url": None, - "channels": [], - "install_lammps": False, - "install_ipi": False, - "artifact_url": None, - "artifact_path": None, - "sha256": None, - "docker_image": None, - "lammps_model_family": None, - }, - "source": { - "directory": "/work/deepmd-kit", - "remote": "https://github.com/deepmodeling/deepmd-kit.git", - "ref": "master", - "commit": None, - }, - "build": { - "variant": "cuda", - "cc": "/usr/bin/gcc", - "cxx": "/usr/bin/g++", - "cuda_home": "/usr/local/cuda", - "rocm_root": None, - "native_optimization": False, - "jobs": 8, - }, - "cpp": { - "install_prefix": "/work/install/deepmd", - "build_directory": "/work/build/deepmd-cpp", - "tensorflow_root": None, - "tensorflow_c_root": None, - "paddle_inference_dir": None, - }, - "lammps": { - "source_directory": "/work/lammps", - "build_directory": "/work/build/lammps-blackwell120", - "version": "stable_22Jul2025_update2", - "url": "https://github.com/lammps/lammps/archive/refs/tags/stable_22Jul2025_update2.tar.gz", - "sha256": "a" * 64, - "flavor": "kokkos-cuda", - "machine": "blackwell120", - "kokkos_arch": "BLACKWELL120", - "mpi": False, - "model_family": "dpa4c", - }, - "smoke_test": {"enabled": False, "gpu": None, "example": None}, - } - - -def _easy_plan(method: str) -> dict[str, object]: - """Return a valid Python-only easy-install plan.""" - environment: dict[str, object] = { - "kind": "existing", - "python": "/opt/deepmd/bin/python", - "manager": None, - "name": None, - "prefix": "/opt/deepmd", - "dp1s_home": None, - } - package: dict[str, object] = { - "deepmd_version": None, - "deepmd_index_url": None, - "deepmd_extra_index_url": None, - "backend_packages": [], - "backend_index_url": None, - "channels": [], - "install_lammps": False, - "install_ipi": False, - "artifact_url": None, - "artifact_path": None, - "sha256": None, - "docker_image": None, - "lammps_model_family": None, - } - if method == "conda": - environment.update( - {"kind": "conda", "python": None, "manager": "/opt/conda", "name": "deepmd"} - ) - elif method == "dp1s": - environment.update( - { - "kind": "prefix", - "python": None, - "prefix": None, - "dp1s_home": "/opt/dp1s", - } - ) - elif method == "offline": - environment.update({"kind": "prefix", "python": None}) - package.update( - { - "artifact_url": "https://example.invalid/deepmd.sh", - "sha256": "a" * 64, - } - ) - elif method == "docker": - environment.update({"kind": "container", "python": None, "prefix": None}) - package["docker_image"] = "ghcr.io/deepmodeling/deepmd-kit:master" - return { - "schema_version": 1, - "method": method, - "goal": "python", - "backend": "tensorflow", - "accelerator": "cpu", - "environment": environment, - "package": package, - "source": None, - "build": None, - "cpp": None, - "lammps": None, - "smoke_test": {"enabled": False, "gpu": None, "example": None}, - } - - -def test_validate_source_dpa4c_plan() -> None: - """Accept a complete PyTorch CUDA DPA4C plan.""" - plan = _source_plan() - assert PLAN.validate_plan(plan) == [] - assert PLAN.required_lammps_styles("dpa4c", "kokkos-cuda") == [ - "dpa4spin", - "dpa4spin/kk", - ] - - -def test_validate_source_build_gate_requires_resolved_commit() -> None: - """Require the resolved source identity before a source build gate.""" - plan = _source_plan() - errors = PLAN.validate_plan(plan, require_resolved_source=True) - assert "source.commit: resolved source SHA is required for this gate" in errors - source = plan["source"] - assert isinstance(source, dict) - source["commit"] = "ed691aab147d9d7686d296e30f46902c08e9fb68" - assert PLAN.validate_plan(plan, require_resolved_source=True) == [] - - -def test_validate_source_jax_gpu_with_cpu_custom_ops() -> None: - """Allow JAX to provide GPU execution independently of custom OPs.""" - plan = _source_plan() - plan["goal"] = "python" - plan["backend"] = "jax" - build = plan["build"] - assert isinstance(build, dict) - build["variant"] = "cpu" - build["cuda_home"] = None - plan["cpp"] = None - plan["lammps"] = None - assert PLAN.validate_plan(plan) == [] - - -def test_validate_source_tensorflow_cpu_plan() -> None: - """Accept a TensorFlow source build with matching CPU custom operations.""" - plan = _source_plan() - plan["goal"] = "python" - plan["backend"] = "tensorflow" - plan["accelerator"] = "cpu" - build = plan["build"] - assert isinstance(build, dict) - build["variant"] = "cpu" - build["cuda_home"] = None - plan["cpp"] = None - plan["lammps"] = None - assert PLAN.validate_plan(plan) == [] - - -def test_validate_jax_cpp_with_tensorflow_python_libraries() -> None: - """Allow JAX C++ to use TensorFlow libraries from the Python environment.""" - plan = _source_plan() - plan["goal"] = "python+cpp" - plan["backend"] = "jax" - plan["lammps"] = None - assert PLAN.validate_plan(plan) == [] - - -def test_validate_jax_cpp_rejects_ambiguous_tensorflow_roots() -> None: - """Require one unambiguous TensorFlow dependency route for JAX C++.""" - plan = _source_plan() - plan["goal"] = "python+cpp" - plan["backend"] = "jax" - plan["lammps"] = None - cpp = plan["cpp"] - assert isinstance(cpp, dict) - cpp["tensorflow_root"] = "/opt/tensorflow-cpp" - cpp["tensorflow_c_root"] = "/opt/tensorflow-c" - errors = PLAN.validate_plan(plan) - assert any("mutually exclusive" in error for error in errors) - - -@pytest.mark.parametrize("method", ["pip", "conda", "dp1s", "offline", "docker"]) -def test_validate_easy_install_plans(method: str) -> None: - """Accept the method-specific fields for each easy-install path.""" - assert PLAN.validate_plan(_easy_plan(method)) == [] - - -@pytest.mark.parametrize("method", ["pip", "source"]) -def test_validate_interpreter_methods_require_prefix(method: str) -> None: - """Require a planned interpreter prefix before pip or source execution.""" - plan = _easy_plan("pip") if method == "pip" else _source_plan() - environment = plan["environment"] - assert isinstance(environment, dict) - environment["prefix"] = None - errors = PLAN.validate_plan(plan) - assert any( - "pip and source methods require an absolute prefix" in error for error in errors - ) - - -def test_validate_conda_channels_are_method_specific() -> None: - """Accept selected conda channels without ignoring them for other methods.""" - plan = _easy_plan("conda") - package = plan["package"] - assert isinstance(package, dict) - package["channels"] = ["conda-forge/label/deepmd-kit_rc", "conda-forge"] - assert PLAN.validate_plan(plan) == [] - - pip_plan = _easy_plan("pip") - pip_package = pip_plan["package"] - assert isinstance(pip_package, dict) - pip_package["channels"] = ["conda-forge"] - errors = PLAN.validate_plan(pip_plan) - assert "package.channels: non-empty channels require method 'conda'" in errors - - -def test_validate_dp1s_lifecycle_keeps_home_and_prefix_distinct(tmp_path: Path) -> None: - """Accept unresolved and resolved dp1s identities without conflating paths.""" - plan = _easy_plan("dp1s") - environment = plan["environment"] - assert isinstance(environment, dict) - dp1s_home = tmp_path / "dp1s-home" - python_prefix = dp1s_home / "envs" / "dp1s" - environment["dp1s_home"] = str(dp1s_home) - assert PLAN.validate_plan(plan) == [] - - environment["python"] = str(python_prefix / "bin" / "python") - environment["prefix"] = str(python_prefix) - assert environment["dp1s_home"] != environment["prefix"] - assert PLAN.validate_plan(plan) == [] - - environment["prefix"] = environment["dp1s_home"] - errors = PLAN.validate_plan(plan) - assert any("must be distinct" in error for error in errors) - - environment["prefix"] = str(python_prefix) - environment["python"] = None - errors = PLAN.validate_plan(plan) - assert any("must both be null or absolute paths" in error for error in errors) - - -def test_validate_dp1s_home_is_method_specific() -> None: - """Reject a dp1s installer root that another method would ignore.""" - plan = _easy_plan("pip") - environment = plan["environment"] - assert isinstance(environment, dict) - environment["dp1s_home"] = "/opt/dp1s" - errors = PLAN.validate_plan(plan) - assert "environment.dp1s_home: allowed only for method 'dp1s'" in errors - - -@pytest.mark.parametrize("field", ["python", "prefix"]) -def test_validate_conda_resolved_paths_must_be_absolute(field: str) -> None: - """Reject a relative Conda interpreter or prefix after environment creation.""" - plan = _easy_plan("conda") - environment = plan["environment"] - assert isinstance(environment, dict) - environment[field] = "relative/path" - errors = PLAN.validate_plan(plan) - assert f"environment.{field}: conda requires an absolute path or null" in errors - - -def test_validate_offline_plan_requires_checksum() -> None: - """Reject an offline artifact without an integrity value.""" - plan = _easy_plan("offline") - package = plan["package"] - assert isinstance(package, dict) - package["sha256"] = None - errors = PLAN.validate_plan(plan) - assert "package.sha256: offline installation requires a checksum" in errors - - -def test_validate_offline_local_artifact(tmp_path: Path) -> None: - """Accept a local offline artifact only through its dedicated path field.""" - artifact = tmp_path / "deepmd.sh" - artifact.write_text("installer", encoding="utf-8") - plan = _easy_plan("offline") - package = plan["package"] - assert isinstance(package, dict) - package["artifact_url"] = None - package["artifact_path"] = str(artifact) - assert PLAN.validate_plan(plan) == [] - - -def test_validate_offline_local_artifact_must_exist(tmp_path: Path) -> None: - """Reject a missing file in the single-artifact offline contract.""" - plan = _easy_plan("offline") - package = plan["package"] - assert isinstance(package, dict) - package["artifact_url"] = None - package["artifact_path"] = str(tmp_path / "missing-deepmd.sh") - errors = PLAN.validate_plan(plan) - assert any("existing complete artifact file" in error for error in errors) - - -def test_validate_offline_artifact_url_requires_https() -> None: - """Reject a local path routed through the offline curl branch.""" - plan = _easy_plan("offline") - package = plan["package"] - assert isinstance(package, dict) - package["artifact_url"] = "/opt/artifacts/deepmd.sh" - errors = PLAN.validate_plan(plan) - assert any( - "package.artifact_url: expected an HTTPS URL" in error for error in errors - ) - - -@pytest.mark.parametrize( - ("path", "value"), - [ - (("method",), []), - (("goal",), {}), - (("environment", "kind"), []), - (("build", "variant"), {}), - (("lammps", "flavor"), []), - (("lammps", "model_family"), {}), - ], -) -def test_validate_plan_rejects_non_string_enums( - path: tuple[str, ...], value: object -) -> None: - """Convert malformed JSON enum types into validation errors.""" - plan = _source_plan() - target: dict[str, object] = plan - for key in path[:-1]: - nested = target[key] - assert isinstance(nested, dict) - target = nested - target[path[-1]] = value - errors = PLAN.validate_plan(plan) - assert any("unsupported value" in error for error in errors) - - -@pytest.mark.parametrize( - ("section", "field", "value"), - [ - ("source", "ref", "v"), - ("package", "backend_packages", ["torch=="]), - ("package", "docker_image", "image:"), - ("environment", "python", "/opt//python"), - ], -) -def test_validate_plan_rejects_embedded_placeholders( - section: str, field: str, value: object -) -> None: - """Reject placeholder tokens embedded inside otherwise valid values.""" - plan = _source_plan() - target = plan[section] - assert isinstance(target, dict) - target[field] = value - errors = PLAN.validate_plan(plan) - assert any("unresolved placeholder" in error for error in errors) - - -@pytest.mark.parametrize("value", ['bad"value', "bad`value", "bad\nvalue"]) -def test_validate_plan_rejects_unsafe_shell_template_values(value: str) -> None: - """Reject values that can escape the documented POSIX templates.""" - plan = _source_plan() - source = plan["source"] - assert isinstance(source, dict) - source["remote"] = value - errors = PLAN.validate_plan(plan) - assert any( - "unsafe shell-template character" in error or "control character" in error - for error in errors - ) - - -def test_validate_plan_keeps_quoted_semicolon_as_data() -> None: - """Allow semicolons that remain inside the documented quoted argument.""" - plan = _source_plan() - source = plan["source"] - assert isinstance(source, dict) - source["remote"] = "https://example.invalid/repository;mirror.git" - assert PLAN.validate_plan(plan) == [] - - -def test_validate_plan_applies_platform_specific_backslash_policy( - monkeypatch: pytest.MonkeyPatch, -) -> None: - """Preserve Windows paths and reject POSIX shell-escape backslashes.""" - windows_errors: list[str] = [] - monkeypatch.setattr(PLAN, "_platform_name", lambda: "nt") - PLAN._validate_strings( - r"C:\DeePMD\python.exe", "environment.python", windows_errors - ) - assert windows_errors == [] - - posix_errors: list[str] = [] - monkeypatch.setattr(PLAN, "_platform_name", lambda: "posix") - PLAN._validate_strings(r"C:\DeePMD\python.exe", "environment.python", posix_errors) - assert any("unsafe shell-template character" in item for item in posix_errors) - - -@pytest.mark.parametrize( - ("field", "value"), - [ - ("backend_index_url", "/mirror"), - ("deepmd_index_url", "file:///mirror"), - ("deepmd_extra_index_url", "http://example.invalid/simple"), - ], -) -def test_validate_plan_requires_https_package_indexes(field: str, value: str) -> None: - """Reject local paths and non-HTTPS package indexes.""" - plan = _easy_plan("pip") - package = plan["package"] - assert isinstance(package, dict) - package[field] = value - errors = PLAN.validate_plan(plan) - assert any("expected an HTTPS URL" in error for error in errors) - - -def test_validate_plan_requires_https_lammps_url() -> None: - """Reject a local path rendered through the LAMMPS curl branch.""" - plan = _source_plan() - lammps = plan["lammps"] - assert isinstance(lammps, dict) - lammps["url"] = "/opt/lammps.tar.gz" - errors = PLAN.validate_plan(plan) - assert any("lammps.url: expected an HTTPS URL" in error for error in errors) - - -def test_validate_plan_requires_checksum_for_lammps_download(tmp_path: Path) -> None: - """Reject an unverified archive when the LAMMPS source is absent.""" - plan = _source_plan() - lammps = plan["lammps"] - assert isinstance(lammps, dict) - lammps["source_directory"] = str(tmp_path / "missing-lammps") - lammps["sha256"] = None - errors = PLAN.validate_plan(plan) - assert "lammps.sha256: required when the source directory is absent" in errors - - -def test_validate_plan_allows_existing_lammps_without_archive( - tmp_path: Path, -) -> None: - """Allow a verified existing source directory without download fields.""" - source_directory = tmp_path / "lammps" - source_directory.mkdir() - plan = _source_plan() - lammps = plan["lammps"] - assert isinstance(lammps, dict) - lammps.update( - {"source_directory": str(source_directory), "url": None, "sha256": None} - ) - assert PLAN.validate_plan(plan) == [] - - -def test_validate_plan_rejects_lammps_source_file(tmp_path: Path) -> None: - """Reject an existing file where a LAMMPS source directory is required.""" - source_path = tmp_path / "lammps" - source_path.write_text("not a source tree", encoding="utf-8") - plan = _source_plan() - lammps = plan["lammps"] - assert isinstance(lammps, dict) - lammps["source_directory"] = str(source_path) - errors = PLAN.validate_plan(plan) - assert "lammps.source_directory: existing path is not a directory" in errors - - -@pytest.mark.parametrize("field", ["build_directory", "install_prefix"]) -def test_validate_plan_rejects_lammps_cpp_path_collisions(field: str) -> None: - """Keep LAMMPS source/build paths out of the C/C++ build and prefix.""" - plan = _source_plan() - cpp = plan["cpp"] - lammps = plan["lammps"] - assert isinstance(cpp, dict) - assert isinstance(lammps, dict) - lammps["build_directory"] = cpp[field] - errors = PLAN.validate_plan(plan) - assert any("C/C++ build/install" in error for error in errors) - - -@pytest.mark.parametrize("cpp_field", ["build_directory", "install_prefix"]) -def test_validate_plan_rejects_lammps_source_in_cpp_paths(cpp_field: str) -> None: - """Keep the LAMMPS source tree out of C/C++ build and install paths.""" - plan = _source_plan() - cpp = plan["cpp"] - lammps = plan["lammps"] - assert isinstance(cpp, dict) - assert isinstance(lammps, dict) - lammps["source_directory"] = cpp[cpp_field] - errors = PLAN.validate_plan(plan) - assert any("C/C++ build/install" in error for error in errors) - - -def test_validate_plan_rejects_easy_rocm() -> None: - """Route ROCm installations through the source workflow.""" - plan = _easy_plan("pip") - plan["accelerator"] = "rocm" - errors = PLAN.validate_plan(plan) - assert "accelerator: ROCm requires method 'source'" in errors - - -@pytest.mark.parametrize("feature", ["lammps", "ipi"]) -def test_validate_plan_rejects_paddle_packaged_native_tools(feature: str) -> None: - """Reject packaged LAMMPS and i-PI for the unsupported Paddle backend.""" - plan = _easy_plan("pip") - plan["backend"] = "paddle" - package = plan["package"] - assert isinstance(package, dict) - if feature == "lammps": - plan["goal"] = "python+lammps" - package["install_lammps"] = True - package["lammps_model_family"] = "conventional" - else: - package["install_ipi"] = True - errors = PLAN.validate_plan(plan) - assert "package: Paddle does not support packaged LAMMPS or i-PI" in errors - - -def test_validate_packaged_lammps_requires_model_family() -> None: - """Require exact pair-style identity for packaged LAMMPS.""" - plan = _easy_plan("pip") - plan["goal"] = "python+lammps" - package = plan["package"] - assert isinstance(package, dict) - package["install_lammps"] = True - errors = PLAN.validate_plan(plan) - assert any("package.lammps_model_family" in error for error in errors) - package["lammps_model_family"] = "dpa4c" - plan["backend"] = "pytorch" - assert PLAN.validate_plan(plan) == [] - - -def test_validate_rocm_smoke_test_requires_physical_gpu() -> None: - """Require explicit ROCm device binding for an enabled smoke test.""" - plan = _source_plan() - plan["goal"] = "python" - plan["accelerator"] = "rocm" - plan["backend"] = "jax" - build = plan["build"] - smoke_test = plan["smoke_test"] - assert isinstance(build, dict) - assert isinstance(smoke_test, dict) - build.update({"variant": "cpu", "cuda_home": None}) - plan["cpp"] = None - plan["lammps"] = None - smoke_test.update({"enabled": True, "gpu": None, "example": "/work/example"}) - errors = PLAN.validate_plan(plan) - assert any( - "ROCM test requires a non-negative physical index" in error for error in errors - ) - - -@pytest.mark.parametrize( - ("mutation", "message"), - [ - (("backend", "tensorflow"), "Kokkos CUDA graph pair styles require PyTorch"), - (("source", {"directory": "$HOME/deepmd"}), "unresolved shell variable"), - (("unknown", True), "unknown field"), - ], -) -def test_validate_plan_rejects_invalid_input( - mutation: tuple[str, object], message: str -) -> None: - """Reject unsupported combinations, placeholders, and unknown keys.""" - plan = _source_plan() - key, value = mutation - if key == "source": - source = plan["source"] - assert isinstance(source, dict) - source.update(value) - else: - plan[key] = value - errors = PLAN.validate_plan(plan) - assert any(message in error for error in errors) - - -def test_validate_plan_rejects_shared_cpp_prefix() -> None: - """Reject a C/C++ install into the selected Python environment.""" - plan = _source_plan() - cpp = plan["cpp"] - assert isinstance(cpp, dict) - cpp["install_prefix"] = "/opt/deepmd" - errors = PLAN.validate_plan(plan) - assert "cpp.install_prefix: select a dedicated, non-shared prefix" in errors - - -def test_prepare_lammps_is_idempotent(tmp_path: Path) -> None: - """Create one quoted managed include and preserve it on a second run.""" - lammps = tmp_path / "lammps tree" - deepmd = tmp_path / "deepmd tree" - cmake_file = lammps / "cmake" / "CMakeLists.txt" - builtin = deepmd / "source" / "lmp" / "builtin.cmake" - cmake_file.parent.mkdir(parents=True) - builtin.parent.mkdir(parents=True) - cmake_file.write_text("cmake_minimum_required(VERSION 3.25)\n", encoding="utf-8") - builtin.write_text("# builtin\n", encoding="utf-8") - - assert PREPARE.prepare(lammps, deepmd, check=False) is False - updated = cmake_file.read_text(encoding="utf-8") - assert updated.count(PREPARE.BEGIN_MARKER) == 1 - assert f"include([=[{builtin}]=])" in updated - assert PREPARE.prepare(lammps, deepmd, check=True) is True - - -def test_prepare_lammps_replaces_one_legacy_include() -> None: - """Replace a legacy include without retaining a duplicate.""" - builtin = Path("/work/deepmd/source/lmp/builtin.cmake") - original = "include(/old/deepmd/source/lmp/builtin.cmake)\n" - updated = PREPARE.render_updated_text(original, builtin) - assert "/old/deepmd" not in updated - assert updated.count("source/lmp/builtin.cmake") == 1 - - -def test_prepare_lammps_rejects_duplicate_legacy_includes() -> None: - """Reject an ambiguous source tree with multiple unmanaged includes.""" - original = "\n".join( - ( - "include(/one/source/lmp/builtin.cmake)", - "include(/two/source/lmp/builtin.cmake)", - ) - ) - with pytest.raises(ValueError, match="multiple DeePMD includes"): - PREPARE.render_updated_text( - original, Path("/work/deepmd/source/lmp/builtin.cmake") - ) - - -def test_prepare_lammps_rejects_managed_and_unmanaged_includes() -> None: - """Reject an extra include outside the managed block.""" - builtin = Path("/work/deepmd/source/lmp/builtin.cmake") - original = "\n".join( - ( - PREPARE.BEGIN_MARKER, - f"include([=[{builtin}]=])", - PREPARE.END_MARKER, - "include(/other/source/lmp/builtin.cmake)", - ) - ) - with pytest.raises(ValueError, match="unmanaged DeePMD include"): - PREPARE.render_updated_text(original, builtin) - - -def _write_fake_lammps(path: Path, styles: str) -> None: - """Write an executable that emits a deterministic LAMMPS help surface.""" - path.write_text(f"#!/usr/bin/env python3\nprint({styles!r})\n", encoding="utf-8") - path.chmod(0o755) - - -def test_native_link_check_rejects_not_found_with_zero_exit( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch -) -> None: - """Fail when ldd reports an unresolved library despite return code zero.""" - binary = tmp_path / "libexample.so" - binary.write_bytes(b"native") - monkeypatch.setattr(NATIVE, "_system_name", lambda: "Linux") - monkeypatch.setattr(NATIVE, "_find_executable", lambda _name: "/usr/bin/ldd") - monkeypatch.setattr( - NATIVE, - "_run", - lambda _command: subprocess.CompletedProcess( - args=[], returncode=0, stdout="libmissing.so => not found\n", stderr="" - ), - ) - result = NATIVE.check_dynamic_links(binary) - assert result.passed is False - assert "libmissing.so => not found" in result.detail - - -def test_native_link_check_uses_dyld_loader( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch -) -> None: - """Fail when the isolated Darwin loader cannot resolve a dependency.""" - library = tmp_path / "libexample.dylib" - library.write_bytes(b"native") - commands: list[list[str]] = [] - - def run(command: list[str]) -> subprocess.CompletedProcess[str]: - commands.append(command) - return subprocess.CompletedProcess( - args=command, - returncode=1, - stdout="", - stderr="OSError: Library not loaded: @rpath/libmissing.dylib", - ) - - monkeypatch.setattr(NATIVE, "_system_name", lambda: "Darwin") - monkeypatch.setattr(NATIVE, "_run", run) - result = NATIVE.check_dynamic_links(library) - assert result.passed is False - assert "Library not loaded" in result.detail - assert commands[0][:2] == [sys.executable, "-c"] - assert "otool" not in commands[0] - - -def test_native_link_default_pattern_matches_platform() -> None: - """Select the native-library suffix without changing explicit patterns.""" - assert NATIVE._default_pattern("Linux") == "libdeepmd*.so" - assert NATIVE._default_pattern("Darwin") == "libdeepmd*.dylib" - - -def test_native_link_collection_preserves_explicit_pattern(tmp_path: Path) -> None: - """Use a caller-provided directory pattern without platform substitution.""" - selected = tmp_path / "libdeepmd.custom" - ignored = tmp_path / "libdeepmd.so" - selected.write_bytes(b"native") - ignored.write_bytes(b"native") - assert NATIVE._collect_paths([], tmp_path, "*.custom") == [selected.resolve()] - - -def test_native_link_check_fails_closed_on_unsupported_platform( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch -) -> None: - """Reject an unimplemented platform instead of reporting false success.""" - library = tmp_path / "deepmd.dll" - library.write_bytes(b"native") - monkeypatch.setattr(NATIVE, "_system_name", lambda: "Windows") - result = NATIVE.check_dynamic_links(library) - assert result.passed is False - assert result.detail == "unsupported platform: Windows" - - -def test_verify_python_checks_version_and_prefix( - monkeypatch: pytest.MonkeyPatch, tmp_path: Path -) -> None: - """Require both interpreter and imported module to use the planned prefix.""" - environment = tmp_path / "environment" - module_file = environment / "lib" / "deepmd" / "__init__.py" - module_file.parent.mkdir(parents=True) - module_file.write_text("", encoding="utf-8") - fake_deepmd = SimpleNamespace(__version__="3.2.0", __file__=str(module_file)) - monkeypatch.setitem(sys.modules, "deepmd", fake_deepmd) - monkeypatch.setattr(VERIFY, "_interpreter_prefix", lambda: environment.resolve()) - passing = VERIFY._check_deepmd("3.2.0", str(environment)) - wrong_version = VERIFY._check_deepmd("3.1.0", str(environment)) - wrong_prefix = VERIFY._check_deepmd("3.2.0", str(tmp_path / "other")) - assert passing.passed is True - assert wrong_version.passed is False - assert wrong_prefix.passed is False - - -def test_verify_python_rejects_shadowed_or_missing_module( - monkeypatch: pytest.MonkeyPatch, tmp_path: Path -) -> None: - """Reject a checkout import even when the interpreter prefix is correct.""" - environment = tmp_path / "environment" - checkout_file = tmp_path / "checkout" / "deepmd" / "__init__.py" - checkout_file.parent.mkdir(parents=True) - checkout_file.write_text("", encoding="utf-8") - fake_deepmd = SimpleNamespace(__version__="3.2.0", __file__=str(checkout_file)) - monkeypatch.setitem(sys.modules, "deepmd", fake_deepmd) - monkeypatch.setattr(VERIFY, "_interpreter_prefix", lambda: environment.resolve()) - assert VERIFY._check_deepmd("3.2.0", str(environment)).passed is False - fake_deepmd.__file__ = None - assert VERIFY._check_deepmd("3.2.0", str(environment)).passed is False - - -def test_verify_python_matches_abbreviated_source_commit() -> None: - """Match the abbreviated commit stored in build metadata to a full SHA.""" - assert VERIFY._commits_match("e59966be", "e59966be1234567890abcdef1234567890abcdef") - assert not VERIFY._commits_match( - "e59966be", "a59966be1234567890abcdef1234567890abcdef" - ) - - -@pytest.mark.parametrize( - ("build_info", "expected"), - [ - ({"is_cuda_build": True, "is_rocm_build": False}, "cuda"), - ({"is_cuda_build": False, "is_rocm_build": True}, "rocm"), - ], -) -def test_tensorflow_runtime_detection( - build_info: dict[str, bool], expected: str -) -> None: - """Distinguish TensorFlow CUDA and ROCm build metadata.""" - tensorflow = SimpleNamespace( - sysconfig=SimpleNamespace(get_build_info=lambda: build_info) - ) - assert VERIFY._tensorflow_accelerator(tensorflow) == expected - assert VERIFY._tensorflow_accelerator(tensorflow) != ( - "rocm" if expected == "cuda" else "cuda" - ) - - -@pytest.mark.parametrize( - ("requested", "build_info"), - [ - ("cuda", {"is_cuda_build": False, "is_rocm_build": True}), - ("rocm", {"is_cuda_build": True, "is_rocm_build": False}), - ], -) -def test_tensorflow_rejects_wrong_gpu_runtime( - requested: str, - build_info: dict[str, bool], - monkeypatch: pytest.MonkeyPatch, -) -> None: - """Fail TensorFlow verification when GPU presence masks a runtime mismatch.""" - deepmd = ModuleType("deepmd") - deepmd.__path__ = [] # type: ignore[attr-defined] - deepmd_tf = ModuleType("deepmd.tf") - tensorflow = ModuleType("tensorflow") - tensorflow.__version__ = "test" # type: ignore[attr-defined] - tensorflow.sysconfig = SimpleNamespace( # type: ignore[attr-defined] - get_build_info=lambda: build_info - ) - tensorflow.config = SimpleNamespace( # type: ignore[attr-defined] - list_physical_devices=lambda _kind: [object()] - ) - monkeypatch.setitem(sys.modules, "deepmd", deepmd) - monkeypatch.setitem(sys.modules, "deepmd.tf", deepmd_tf) - monkeypatch.setitem(sys.modules, "tensorflow", tensorflow) - checks = VERIFY._check_tensorflow(requested) - assert checks[-1].passed is False - assert f"expected={requested}" in checks[-1].detail - - -@pytest.mark.parametrize( - ("client_platform", "device_kind", "platform_version", "expected"), - [ - ("gpu", "NVIDIA RTX PRO 6000", "CUDA 13.0", "cuda"), - ("gpu", "AMD Instinct MI300X", "ROCm 7.0", "rocm"), - ("gpu", "AMD-compatible adapter", "CUDA 13.0", "cuda"), - ("cuda", "vendor text is not authoritative", "unknown", "cuda"), - ], -) -def test_jax_runtime_detection( - client_platform: str, - device_kind: str, - platform_version: str, - expected: str, -) -> None: - """Distinguish JAX CUDA and ROCm client metadata.""" - device = SimpleNamespace( - platform="gpu", - device_kind=device_kind, - client=SimpleNamespace( - platform=client_platform, platform_version=platform_version - ), - ) - assert VERIFY._jax_accelerator([device]) == expected - assert VERIFY._jax_accelerator([device]) != ( - "rocm" if expected == "cuda" else "cuda" - ) - - -def test_jax_runtime_detection_fails_closed_on_conflicting_backends() -> None: - """Reject a device set that reports both CUDA and ROCm clients.""" - devices = [ - SimpleNamespace( - platform="gpu", - client=SimpleNamespace(platform=runtime, platform_version=runtime), - ) - for runtime in ("cuda", "rocm") - ] - assert VERIFY._jax_accelerator(devices) is None - - -@pytest.mark.parametrize( - ("requested", "device_kind", "platform_version"), - [ - ("cuda", "AMD Instinct MI300X", "ROCm 7.0"), - ("rocm", "NVIDIA RTX PRO 6000", "CUDA 13.0"), - ], -) -def test_jax_rejects_wrong_gpu_runtime( - requested: str, - device_kind: str, - platform_version: str, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """Fail JAX verification when a GPU uses the other runtime.""" - deepmd = ModuleType("deepmd") - deepmd.__path__ = [] # type: ignore[attr-defined] - deepmd_jax = ModuleType("deepmd.jax") - jax = ModuleType("jax") - jax.__path__ = [] # type: ignore[attr-defined] - jax.__version__ = "test" # type: ignore[attr-defined] - device = SimpleNamespace( - platform="gpu", - device_kind=device_kind, - client=SimpleNamespace(platform="gpu", platform_version=platform_version), - ) - jax.devices = lambda: [device] # type: ignore[attr-defined] - jax_numpy = ModuleType("jax.numpy") - monkeypatch.setitem(sys.modules, "deepmd", deepmd) - monkeypatch.setitem(sys.modules, "deepmd.jax", deepmd_jax) - monkeypatch.setitem(sys.modules, "jax", jax) - monkeypatch.setitem(sys.modules, "jax.numpy", jax_numpy) - checks = VERIFY._check_jax(requested) - assert checks[-1].passed is False - assert f"expected={requested}" in checks[-1].detail - - -def test_docker_reference_uses_backend_aware_verifier() -> None: - """Keep Docker verification backend-aware and read-only mounted.""" - reference = ( - REPOSITORY_ROOT / "skills" / "deepmd-install" / "references" / "easy-install.md" - ).read_text(encoding="utf-8") - assert "verify_python.py" in reference - assert '--backend ""' in reference - assert "readonly" in reference - - -def test_conda_reference_renders_planned_channels() -> None: - """Keep selected conda channels authoritative with a stable default.""" - reference = ( - REPOSITORY_ROOT / "skills" / "deepmd-install" / "references" / "easy-install.md" - ).read_text(encoding="utf-8") - assert "package.channels" in reference - assert '-c "" -c ""' in reference - assert reference.count("--override-channels") >= 2 - assert '"deepmd-kit="' in reference - assert "print(sys.executable); print(sys.prefix)" in reference - - -def test_dp1s_reference_preserves_planned_identity() -> None: - """Pass the planned dp1s prefix and version without editing shell startup.""" - reference = ( - REPOSITORY_ROOT / "skills" / "deepmd-install" / "references" / "easy-install.md" - ).read_text(encoding="utf-8") - assert 'DP1S_HOME=""' in reference - assert 'DEEPMD_VERSION=""' in reference - assert "DP1S_NO_PATH_UPDATE=1" in reference - assert '--expected-prefix ""' in reference - - -def test_verify_lammps_dpa4c_cli(tmp_path: Path) -> None: - """Accept the exact DPA4C host and Kokkos pair styles.""" - binary = tmp_path / "lmp_dpa4c" - _write_fake_lammps(binary, "Pair styles: dpa4spin dpa4spin/kk") - completed = subprocess.run( - [ - sys.executable, - str(SCRIPT_DIRECTORY / "verify_lammps.py"), - "--binary", - str(binary), - "--model-family", - "dpa4c", - "--flavor", - "kokkos-cuda", - ], - check=False, - capture_output=True, - text=True, - ) - assert completed.returncode == 0, completed.stdout + completed.stderr - assert "PASS pair_style:dpa4spin/kk" in completed.stdout - - -def test_verify_lammps_requires_exact_style_token(tmp_path: Path) -> None: - """Reject a near-name token instead of accepting a substring.""" - binary = tmp_path / "lmp_near_name" - _write_fake_lammps(binary, "Pair styles: deepmd deepmd/kkx") - completed = subprocess.run( - [ - sys.executable, - str(SCRIPT_DIRECTORY / "verify_lammps.py"), - "--binary", - str(binary), - "--model-family", - "dpa4", - "--flavor", - "kokkos-cuda", - ], - check=False, - capture_output=True, - text=True, - ) - assert completed.returncode == 1 - assert "FAIL pair_style:deepmd/kk" in completed.stdout - - -def test_verify_python_rejects_backend_incompatible_checks() -> None: - """Reject PyTorch-only flags before importing another backend.""" - completed = subprocess.run( - [ - sys.executable, - str(SCRIPT_DIRECTORY / "verify_python.py"), - "--backend", - "jax", - "--accelerator", - "cpu", - "--expect-custom-op", - ], - check=False, - capture_output=True, - text=True, - ) - assert completed.returncode == 2 - assert "PyTorch-only checks require --backend pytorch" in completed.stderr - - -def test_probe_cli_emits_json() -> None: - """Emit the stable top-level probe schema without requiring optional tools.""" - completed = subprocess.run( - [sys.executable, str(SCRIPT_DIRECTORY / "probe_env.py"), "--json"], - check=False, - capture_output=True, - text=True, - timeout=60, - ) - assert completed.returncode == 0, completed.stderr - report = json.loads(completed.stdout) - assert set(report) == { - "system", - "environment", - "python", - "tools", - "nvidia", - "rocm", - "packages", - } - assert Path(report["python"]["executable"]).is_absolute() From 39a24f808f2ae530f96d5ffea9952fb819ce2f1a Mon Sep 17 00:00:00 2001 From: OutisLi Date: Tue, 18 Aug 2026 22:41:33 +0800 Subject: [PATCH 09/11] fix(skills): harden DeePMD installation guidance --- skills/deepmd-install/SKILL.md | 52 +++++++++++++------ .../references/failure-modes.md | 24 ++++++--- .../references/official-docs.md | 28 ++++++---- 3 files changed, 73 insertions(+), 31 deletions(-) diff --git a/skills/deepmd-install/SKILL.md b/skills/deepmd-install/SKILL.md index 330a6bab69..1d5ec5914b 100644 --- a/skills/deepmd-install/SKILL.md +++ b/skills/deepmd-install/SKILL.md @@ -1,6 +1,6 @@ --- name: deepmd-install -description: Install DeePMD-kit with pip, conda, dp1s, an offline package, Docker, or source code. Use for PyTorch, TensorFlow, JAX, or Paddle on CPU, CUDA, or ROCm, and for the C/C++ interface or DeePMD-enabled LAMMPS. +description: Install DeePMD-kit with pip, conda, dp1s, an offline package, Docker, or source code. Use for PyTorch, TensorFlow, JAX, or Paddle on CPU, CUDA, or ROCm, and for backend-enabled or backend-neutral C/C++ interfaces and DeePMD-enabled LAMMPS. --- # Install DeePMD-kit @@ -15,7 +15,8 @@ the full installation manual. Determine only the choices that affect installation: - DeePMD-kit release or Git ref; -- PyTorch, TensorFlow, JAX, or Paddle backend; +- PyTorch, TensorFlow, JAX, or Paddle backend, or a backend-neutral C/C++ + library; - CPU, NVIDIA CUDA, or ROCm runtime; - Python only, packaged LAMMPS, C/C++, or source-built LAMMPS; - existing environment or a new user-approved environment. @@ -27,6 +28,9 @@ otherwise choose the shortest supported package path. Do not create a new environment when the user requires an existing one. Ask only for a missing choice that changes the installation path. +The backend-neutral choice applies only to the C/C++ interface and requires a +compatible backend plugin at runtime. It is not a Python installation target. + Inspect the OS, architecture, Python version and prefix, package manager, and requested accelerator before changing the machine. Resolve the selected Python executable to an absolute path and use `"" -m pip`; @@ -51,9 +55,9 @@ rendering an install command: "https://raw.githubusercontent.com/deepmodeling/deepmd-kit//doc/install/.md" ``` -1. If direct GitHub access fails, retry that same public URL through the - documented `gh-proxy.com` form. Reject an HTTP failure, empty response, or - HTML error page. +1. If direct GitHub access fails, fetch the same path and exact ref from the + official Gitee mirror. Reject an HTTP failure, empty response, HTML error + page, or missing ref. Read [`references/official-docs.md`](references/official-docs.md) for the URL map, version-selection rules, exact repository paths, and network fallback. @@ -157,14 +161,29 @@ git -C "" checkout --detach FETCH_HEAD git -C "" rev-parse HEAD ``` +If GitHub is unavailable, use the official mirror at +`https://gitee.com/deepmodeling/deepmd-kit.git` and resolve the same validated +ref. Do not obtain installation instructions or source through an +unauthenticated proxy. + Read `doc/install/install-from-source.md` at that exact commit, install the -selected backend first, and apply only the documented build variables. The -basic Python build is: +selected backend first, and apply only the documented build variables. Render +the target-defining variables in the same invocation so build defaults cannot +select another runtime or backend: ```bash +DP_VARIANT="" \ +DP_ENABLE_TENSORFLOW="<0|1>" \ +DP_ENABLE_PYTORCH="<0|1>" \ "" -m pip install "" ``` +Enable only the TensorFlow or PyTorch compiled support requested by the user. +For a Python-only JAX or Paddle installation, set both backend variables to +`0` unless the matching documentation requires compiled support. Add the +documented `CUDAToolkit_ROOT` or `ROCM_ROOT` to the same invocation when the +selected runtime requires an explicit toolkit root. + Keep source, build, and install locations distinct. ### Pre-compiled C library @@ -177,12 +196,14 @@ release. ### C/C++ interface and LAMMPS -For C/C++, follow the backend-specific section of the matching source-install -page; do not guess CMake options or backend library roots. For LAMMPS, use a -packaged `lmp` when it satisfies the request. Otherwise follow the matching -built-in or plugin instructions after the C/C++ interface succeeds. Enable -Kokkos only when the requested LAMMPS runtime requires it, and select the -architecture supported by that exact LAMMPS/Kokkos source tree. +For C/C++, choose the backend-enabled or backend-neutral section of the +matching source-install page; do not guess CMake options or backend library +roots. A backend-neutral build must set `ALLOW_NO_BACKEND=ON`, build only the +C/C++ libraries, and provide a compatible backend plugin at runtime. For +LAMMPS, use a packaged `lmp` when it satisfies the request. Otherwise follow +the matching built-in or plugin instructions after the C/C++ interface +succeeds. Enable Kokkos only when the requested LAMMPS runtime requires it, +and select the architecture supported by that exact LAMMPS/Kokkos source tree. ## 4. Verify the requested interface @@ -193,8 +214,9 @@ An installation is complete only after the requested public interface runs: 1. Import the selected backend (`deepmd.pt`, `deepmd.tf`, `deepmd.jax`, or `deepmd.pd`) and run one minimal tensor operation on the requested device. 1. Run the installed `dp --version` and backend-specific help. -1. For C/C++, confirm the installed headers/libraries and unresolved dynamic - dependencies before testing a client. +1. For C/C++, confirm the installed headers/libraries, then load a built + library or run a linked client so the platform loader resolves its dynamic + dependencies. 1. For LAMMPS, run the selected binary with `-h`, require the exact DeePMD pair style needed by the model, and run a short documented example. 1. For Docker, perform all applicable checks inside the selected image. diff --git a/skills/deepmd-install/references/failure-modes.md b/skills/deepmd-install/references/failure-modes.md index 510f7067b6..5f07f761c5 100644 --- a/skills/deepmd-install/references/failure-modes.md +++ b/skills/deepmd-install/references/failure-modes.md @@ -9,9 +9,9 @@ working environment without the user's approval. Confirm the requested version or ref and the exact URL. Try the versioned docs, then the same file at the exact GitHub ref as described in -[`official-docs.md`](official-docs.md). For a blocked public GitHub URL, use the -documented `gh-proxy.com` fallback. Do not silently substitute a newer release, -different CUDA build, or similarly named asset. +[`official-docs.md`](official-docs.md). If GitHub is blocked, use the exact ref +and path on the official Gitee mirror. Do not silently substitute a newer +release, different CUDA build, or similarly named asset. For downloads, retain the HTTP status, file size, and content type. An HTML error page is not an installer or archive. Reassemble split offline assets in @@ -87,10 +87,20 @@ unless the docs explicitly support it. ## A native library cannot be loaded -Use `ldd` on Linux or `otool -L` on macOS for the failing library or executable. -Locate the named dependency in the selected environment before changing -`RPATH`, `LD_LIBRARY_PATH`, or linker flags. Framework import success does not -prove that a separately built C++ client or LAMMPS binary uses the same ABI and +Use `ldd` on Linux or `otool -L` on macOS to identify dependencies of the +failing library or executable. These listings are diagnostic; `otool -L` does +not prove that dyld can resolve its install names. Require the platform loader +to load the built library or execute a linked client. A minimal dylib probe is: + +```bash +"" -c \ + 'import ctypes; ctypes.CDLL(".dylib")' +``` + +Treat any loader error or Linux `not found` entry as a failed gate. Locate the +named dependency in the selected environment before changing `RPATH`, +`LD_LIBRARY_PATH`, or linker flags. Framework import success does not prove +that a separately built C++ client or LAMMPS binary uses the same ABI and libraries. ## LAMMPS lacks the required DeePMD pair style diff --git a/skills/deepmd-install/references/official-docs.md b/skills/deepmd-install/references/official-docs.md index 0f12fddfc1..474a36d58a 100644 --- a/skills/deepmd-install/references/official-docs.md +++ b/skills/deepmd-install/references/official-docs.md @@ -24,8 +24,16 @@ DeePMD-kit checkout. "https://raw.githubusercontent.com/deepmodeling/deepmd-kit//doc/install/.md" ``` -1. Stop on a nonzero curl exit, empty response, or HTML error page. Do not - generate commands from partial or unverified content. +1. If GitHub is unavailable, fetch the same exact ref and path from the + official Gitee mirror: + + ```bash + curl -fsSL --retry 2 \ + "https://gitee.com/deepmodeling/deepmd-kit/raw//doc/install/.md" + ``` + +1. Stop on a nonzero curl exit, empty response, HTML error page, or missing + ref. Do not generate commands from partial or unverified content. ## Primary pages @@ -36,6 +44,7 @@ DeePMD-kit checkout. - Development packages: - Releases: - Container images: +- Official Gitee mirror: - dp1s options: Use official backend installers when the DeePMD-kit page links to them: @@ -71,6 +80,7 @@ For example, replace `` with a validated tag or commit in either form: ```text https://github.com/deepmodeling/deepmd-kit/blob//doc/install/install-from-source.md https://raw.githubusercontent.com/deepmodeling/deepmd-kit//doc/install/install-from-source.md +https://gitee.com/deepmodeling/deepmd-kit/raw//doc/install/install-from-source.md ``` Resolve a branch or tag to a commit SHA before building. Treat a Git ref as @@ -94,14 +104,14 @@ version, wheel index, image tag, CMake option, or Kokkos architecture. ## Network fallback -If a GitHub repository or raw-file URL is unreachable because of network -restrictions, retry the same public URL through `gh-proxy.com`, for example: +If GitHub is unreachable, use the official Gitee mirror for both source and +raw documentation: ```text -https://gh-proxy.com/https://github.com/deepmodeling/deepmd-kit.git -https://gh-proxy.com/https://raw.githubusercontent.com/deepmodeling/deepmd-kit//doc/install/install-from-source.md +https://gitee.com/deepmodeling/deepmd-kit.git +https://gitee.com/deepmodeling/deepmd-kit/raw//doc/install/install-from-source.md ``` -Use the proxy only for public GitHub content. Never send credentials through -it. After cloning through a proxy, verify the configured upstream URL and the -resolved commit SHA before building. +Require the requested tag or commit to exist on the mirror. After cloning, +record the configured remote and resolved commit SHA before building; do not +silently substitute the mirror's default branch for a missing ref. From 666339dd0fdea3f8090a6bdfe4054d86196f59e9 Mon Sep 17 00:00:00 2001 From: OutisLi Date: Tue, 18 Aug 2026 23:33:50 +0800 Subject: [PATCH 10/11] style(skills): format DeePMD installation skill --- skills/deepmd-install/SKILL.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/skills/deepmd-install/SKILL.md b/skills/deepmd-install/SKILL.md index 1d5ec5914b..7634302e38 100644 --- a/skills/deepmd-install/SKILL.md +++ b/skills/deepmd-install/SKILL.md @@ -173,9 +173,9 @@ select another runtime or backend: ```bash DP_VARIANT="" \ -DP_ENABLE_TENSORFLOW="<0|1>" \ -DP_ENABLE_PYTORCH="<0|1>" \ -"" -m pip install "" + DP_ENABLE_TENSORFLOW="<0|1>" \ + DP_ENABLE_PYTORCH="<0|1>" \ + "" -m pip install "" ``` Enable only the TensorFlow or PyTorch compiled support requested by the user. From cece07d32f6ee904df79dde0d5f925959459627b Mon Sep 17 00:00:00 2001 From: OutisLi Date: Wed, 19 Aug 2026 00:18:17 +0800 Subject: [PATCH 11/11] fix(skills): secure install rendering and version pins --- skills/deepmd-install/SKILL.md | 34 ++++++++++++++----- .../references/failure-modes.md | 10 ++++++ 2 files changed, 35 insertions(+), 9 deletions(-) diff --git a/skills/deepmd-install/SKILL.md b/skills/deepmd-install/SKILL.md index 7634302e38..89245bf0bc 100644 --- a/skills/deepmd-install/SKILL.md +++ b/skills/deepmd-install/SKILL.md @@ -63,9 +63,15 @@ Read [`references/official-docs.md`](references/official-docs.md) for the URL map, version-selection rules, exact repository paths, and network fallback. Use the docs matching the selected version, not `latest` for an older release. -The commands below are route summaries. Before execution, replace every -placeholder with an observed or user-selected value and apply any changed -requirements from the matching official page. +The commands below describe executable and argument boundaries; they are not +text-substitution templates. Pass dynamic values through an argument array +with shell evaluation disabled, such as a process API's `shell=False`, and +pass environment variables through a process environment map. If only a shell +string is available, escape each complete value with the target shell's +canonical quoting (`shlex.quote` for POSIX). Reject NUL, newline, and other +control characters; double quotes alone are not a safety boundary. Keep +option-looking inputs behind the documented option boundary. Apply any changed +requirements from the matching official page before execution. ## 3. Choose one installation path @@ -82,10 +88,14 @@ the backend tab in the official easy-install page: | Paddle | Install the documented Paddle package first, then `deepmd-kit` | Add `lmp` or `ipi` to the extras only when requested and supported by the -selected backend and platform. Install through the absolute target Python: +selected backend and platform. Normalize the selected release tag to its +package version, for example `v3.1.3` to `3.1.3`, and append an exact `==` +constraint to the DeePMD-kit requirement after any extras. Install through the +absolute target Python: ```bash -"" -m pip install "" +"" -m pip install \ + "==" ``` ### conda @@ -94,12 +104,14 @@ Use conda-forge for a released package when the user prefers conda: ```bash "" create -n "" \ - -c conda-forge deepmd-kit + -c conda-forge "deepmd-kit==" ``` -Add `lammps` or distributed-training packages only when requested. Follow the -linked conda-forge CUDA guidance rather than inventing a toolkit pin. After -creation, resolve the environment's absolute Python before verification. +Normalize the selected release tag as for pip and use Conda's exact `==` +MatchSpec. Add `lammps` or distributed-training packages only when requested. +Follow the linked conda-forge CUDA guidance rather than inventing a toolkit +pin. After creation, resolve the environment's absolute Python before +verification. ### dp1s @@ -211,6 +223,10 @@ An installation is complete only after the requested public interface runs: 1. Print `sys.executable`, `sys.prefix`, `deepmd.__version__`, and `deepmd.__file__` with the selected absolute Python. +1. For pip or conda, normalize the selected release tag and + `deepmd.__version__` with `packaging.version.Version(...).public`; fail if + they differ. For source, verify the resolved Git commit and installation + origin separately. 1. Import the selected backend (`deepmd.pt`, `deepmd.tf`, `deepmd.jax`, or `deepmd.pd`) and run one minimal tensor operation on the requested device. 1. Run the installed `dp --version` and backend-specific help. diff --git a/skills/deepmd-install/references/failure-modes.md b/skills/deepmd-install/references/failure-modes.md index 5f07f761c5..bd081df8b2 100644 --- a/skills/deepmd-install/references/failure-modes.md +++ b/skills/deepmd-install/references/failure-modes.md @@ -17,6 +17,16 @@ For downloads, retain the HTTP status, file size, and content type. An HTML error page is not an installer or archive. Reassemble split offline assets in the documented order and verify a published checksum when one exists. +## A rendered value changes the command + +Stop if a dynamic value is interpolated directly into shell source. Prefer an +argument array with shell evaluation disabled. To check a shell-string +renderer, print its resulting argument vector instead of executing the target +and test literal values such as `path with spaces`, `$(printf injected)`, +`` `printf injected` ``, `x;printf injected`, and `-n`. Each value must remain +one unchanged argument, and no nested command may execute. Reject NUL, newline, +and other control characters rather than trying to quote them. + ## Python, pip, and dp identify different environments Run these checks with the selected absolute interpreter: