[BUG][pt_expt][DPA1] Automatic graph-lower export changes predictions for dense-trained se_atten_v2 models with nonzero davg

[hcustc]

Bug summary

I trained a DPA1-L0 model with the standard PyTorch backend (dp --pt train) and converted the
frozen .pth model to .pt2 using dp convert-backend. The descriptor used the default
set_davg_zero: false and has nonzero davg.

The automatically selected graph-lower model loads and runs, but it is not numerically
equivalent to the source model. On the same six-frame system, the force-component difference
is:

• MAE: 1.1381067 eV/Angstrom
• RMSE: 1.7619205 eV/Angstrom
• Maximum absolute difference: 6.7789032 eV/Angstrom

In contrast, a manually exported nlist/dense-lower .pt2 model reproduces the source model
with a force-component MAE of 5.13e-6 eV/Angstrom and a maximum difference of 1.91e-5 eV/ Angstrom.

The discrepancy is reproduced by an uncompiled CPU eager comparison between
forward_common_lower and forward_common_lower_graph, and is already present in the
descriptor output. As a diagnostic, setting the descriptor mean (davg) to zero in memory
reduces the descriptor discrepancy to numerical noise.

I also independently retrained four models with set_davg_zero: true. After automatic graph-
lower conversion, their force-component MAE differences are 1.76e-5–1.84e-5 eV/Angstrom,
with a maximum observed difference of 1.33e-4 eV/Angstrom.

These results indicate that automatic conversion accepts a dense-trained se_atten_v2 model
with nonzero davg, even though its graph-lower representation is not functionally equivalent.

DeePMD-kit Version

0.1.dev1+gbc902daf7.d20260809

Backend and its version

PyTorch 2.11.0+cu128

How did you download the software?

Built from source

Input Files, Running Commands, Error Log, etc.

A minimal reproducer archive is attached containing:

reproducer/
├── frozen_model.pth
├── input.json
├── compare_predictions.py
└── test_system/
    ├── type.raw
    ├── type_map.raw
    ├── nopbc
    └── set.000/
        ├── box.npy
        ├── coord.npy
        ├── energy.npy
        └── force.npy

The test system is nonperiodic and contains six frames with seven atoms per frame.

Relevant model settings:

{
  "type_map": ["C", "Cl", "H", "O"],
  "descriptor": {
    "type": "se_atten_v2",
    "sel": 250,
    "rcut": 6.0,
    "attn": 128,
    "attn_layer": 0,
    "tebd_input_mode": "strip",
    "precision": "float32"
  }
}

set_davg_zero was not specified during training, so its effective value is false. The trained
descriptor has nonzero davg; its largest absolute value is approximately 0.01418.

Environment:

DeePMD-kit: 0.1.dev1+gbc902daf7.d20260809
Git commit: bc902daf747d93c5d7314338d287fd2dd0a7e32c
PyTorch: 2.11.0+cu128
Python: 3.12.12
OS: Linux x86_64

The build is based on the official master branch. The only local source change preserves an
explicitly requested `CMAKE_CUDA_ARCHITECTURES=120`; no Python, model, descriptor,
serialization, or inference code was modified.

There is no exception or error traceback. Conversion and inference both exit successfully. This
is a silent numerical correctness failure.

Observed source .pth versus automatic graph-lower .pt2 difference:

Energy MAE:                  1.3291797 eV/frame
Energy maximum difference:  2.0507687 eV/frame
Force-component MAE:        1.1381067 eV/Angstrom
Force-component RMSE:       1.7619205 eV/Angstrom
Force maximum difference:   6.7789032 eV/Angstrom

For comparison, manually exporting the same model through nlist/dense lower gives a force-
component MAE of 5.13e-6 eV/Angstrom and a maximum difference of 1.91e-5 eV/Angstrom.

### Steps to Reproduce

[deepmd-dpa1-graph-lower-reproducer.zip](https://github.com/user-attachments/files/31117915/deepmd-dpa1-graph-lower-reproducer.zip)

1. Extract the attached minimal reproducer and enter its directory:

   ```bash
   unzip deepmd-dpa1-graph-lower-reproducer.zip
   cd deepmd-dpa1-graph-lower-reproducer

2. Convert the supplied source model. dp convert-backend automatically selects graph lower:

   dp convert-backend frozen_model.pth model_graph.pt2

3. Evaluate the source and converted models on the same six-frame system:

  dp --pt test -m frozen_model.pth -s test_system -n 32 -d pth
  dp --pt-expt test -m model_graph.pt2 -s test_system -n 32 -d graph

4. Compare their predictions:

   python compare_predictions.py pth graph

   Expected reproduction output:

   Energy MAE difference:              1.3291797 eV/frame
   Energy maximum absolute difference: 2.0507687 eV/frame
   Force-component MAE:                1.1381067 eV/Angstrom
   Force-component RMSE:               1.7619205 eV/Angstrom
   Force-component maximum difference: 6.7789032 eV/Angstrom

No retraining, LAMMPS, or Kokkos is required. The same underlying dense-versus-graph
discrepancy was also reproduced independently in uncompiled CPU eager execution.

### Further Information, Files, and Links

_No response_

[dosubot]

Hi @hcustc! I'm Dosu and I'm helping the DeepModeling team.

This is a variant of a previously reported bug. The root cause is the same phantom padding-neighbor residual documented in deepmd/dpmodel/descriptor/dpa1.py: the dense lower path (forward_common_lower) retains a -davg/dstd contribution for every vacant padding slot in the fixed-capacity neighbor list, while the graph lower path (forward_common_lower_graph) omits padding neighbors entirely [1]. When davg ≠ 0, the two paths are not numerically equivalent.

Your case differs from issue #5862 (which covers .pt checkpoint loading): here the discrepancy appears during dp convert-backend to .pt2. The export gate model_uses_graph_lower in deepmd/pt_expt/utils/serialization.py does not check whether the model has nonzero davg, so a dense-trained model with set_davg_zero: false gets silently exported into a graph-lower .pt2 whose descriptor semantics differ from the source.

Your set_davg_zero: true retrained models round-trip cleanly (MAE ~1.8×10⁻⁵ eV/Å) because with davg = 0 the padding residual vanishes and the two paths produce identical outputs [2].

Two potential fixes to consider:

1. Block automatic graph-lower export when davg ≠ 0 — make dp convert-backend fall back to the nlist/dense-lower format and warn the user, rather than silently producing an inequivalent model.
2. Zero out davg in the graph-lower export path — mathematically valid if the graph path has already absorbed the normalization elsewhere, but this needs verification against the DPA1 aggregation semantics.

The companion issue #5824 proposes decoupling DPA1 graph normalization and statistics from the dense sel-based representation [3], which is the structural prerequisite for option 2. Issue #5862 tracks the related .pt checkpoint path mismatch [1].

Immediate workaround: Manually export via the nlist/dense-lower path (as you've already done), which you confirmed gives a force-component MAE of ~5×10⁻⁶ eV/Å. Until the export gate logic is fixed, models trained with set_davg_zero: false should not use the automatically selected graph-lower .pt2 format.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.