From 1bbc5c44ae4d185fac09ae6066f66949e7519ff9 Mon Sep 17 00:00:00 2001 From: Jiahui Cao Date: Thu, 13 Aug 2026 06:30:47 +0000 Subject: [PATCH 1/3] perf(dsv4-fp4-mi355x-vllm-agentic-mtp): repin to the 08-09 nightly and complete the DP-attention curve MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Repin to the 08-09 nightly (f8d03e77) so the config can start at all: the previously pinned tag no longer serves this recipe, and the 08-12 nightly (3ee2df30) memory-faults during the profile run. Two arms: pure TP8 through its measured knee at 48, and DP-attention with the experts left TP-sharded (ep 1) extended from a single conc-64 point to 32/48/64/96. No EP8 arm -- EP8 only pays off with the MegaMoE backend, which needs mori.ir.flydsl, absent from this image. Recipe side, all DP-attention-conditional: cap --max-num-seqs at CONC rather than 2*CONC (the limit is per scheduler and DP-attention runs one per rank), set --max-num-batched-tokens 8192 with --prefill-schedule-interval 8 and --long-prefill-token-threshold 16384, and raise --gpu-memory-utilization from 0.8 to 0.86. 重新钉到 08-09 nightly（f8d03e77），配置才能起得来：原先钉的 tag 已经跑不动这个 recipe，而 08-12 nightly（3ee2df30）在 profile run 阶段会 memory fault。 两条 arm：纯 TP8 跑到实测拐点 48，以及专家仍按 TP 切分（ep 1）的 DP-attention arm，从原来单点 conc 64 扩展到 32/48/64/96。不加 EP8 arm —— EP8 只有配 MegaMoE 后端才划算，而它需要 mori.ir.flydsl，该镜像里没有。 recipe 侧改动均只在 DP-attention 下生效：--max-num-seqs 从 2*CONC 收到 CONC （该上限是每个 scheduler 生效的，而 DP-attention 每个 rank 一个 scheduler）， 设置 --max-num-batched-tokens 8192 配合 --prefill-schedule-interval 8 与 --long-prefill-token-threshold 16384，并把 --gpu-memory-utilization 从 0.8 提到 0.86。 --- .../agentic/dsv4_fp4_mi355x_vllm_mtp.sh | 15 ++++++++++++++- configs/amd-master.yaml | 4 ++-- perf-changelog.yaml | 14 ++++++++++++++ 3 files changed, 30 insertions(+), 3 deletions(-) diff --git a/benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh b/benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh index 1b9311eda..dca301717 100644 --- a/benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh +++ b/benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh @@ -371,10 +371,22 @@ if [ "$EP_SIZE" -gt 1 ]; then EP_ARGS=(--enable-expert-parallel) fi +DP_SCHED_ARGS=() +if [ "$DP_ATTENTION" = "true" ]; then + DP_SCHED_ARGS=( + --prefill-schedule-interval 8 + --max-num-batched-tokens 8192 + --long-prefill-token-threshold 16384 + ) +fi + # AgentX concurrency counts live session trees, not individual requests. # Subagent fan-out can push instantaneous request concurrency above CONC, so # leave 2x headroom rather than clipping those bursts at the scheduler. MAX_NUM_SEQS=$((2 * CONC)) +if [ "$DP_ATTENTION" = "true" ]; then + MAX_NUM_SEQS="$CONC" +fi # DeepSeek-V4-Pro ships a native MTP head. AgentX throughput pins its # three-token draft to the committed thinking-on golden acceptance length; @@ -406,7 +418,8 @@ VLLM_CMD=( --kv-cache-dtype fp8 "${PARALLEL_ARGS[@]}" "${EP_ARGS[@]}" - --gpu-memory-utilization 0.8 + "${DP_SCHED_ARGS[@]}" + --gpu-memory-utilization 0.86 --moe-backend aiter --compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}' --speculative-config "$SPEC_CONFIG" diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index c94029484..b406168e3 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1301,7 +1301,7 @@ qwen3.5-fp8-mi355x-sglang-agentic-hicache: # is no longer available from Docker Hub. Throughput uses the three-token golden # synthetic acceptance length, while eval-only runs retain real verification. dsv4-fp4-mi355x-vllm-agentic-mtp: - image: vllm/vllm-openai-rocm:nightly-821717118fc26667dd474b9b0ab81d29259dfc5c + image: vllm/vllm-openai-rocm:nightly-f8d03e77416bf90c49acbe50e233275722f02c4b model: deepseek-ai/DeepSeek-V4-Pro model-prefix: dsv4 runner: cluster:mi355x-amds @@ -1313,7 +1313,7 @@ dsv4-fp4-mi355x-vllm-agentic-mtp: - dram-utilization: 0.60 search-space: - { tp: 8, ep: 1, dp-attn: false, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 4, 8, 16, 32, 40, 48] } - - { tp: 8, ep: 1, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [64], router: { name: vllm-router, version: "0.1.14" } } + - { tp: 8, ep: 1, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [32, 48, 64, 96], router: { name: vllm-router, version: "0.1.14" } } # LMCache invalid-block recovery currently assumes one KV-cache group, # while MTP creates two. Restore these points after the upstream hybrid # KV recovery fix lands: https://github.com/vllm-project/vllm/pull/45497 diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 6082f2b86..97bc7bdfb 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -5918,3 +5918,17 @@ - "Add a TEP2 arm (tp 2, ep 2) to the qwen3.5-fp4-b200-sglang-mtp 8k/1k sweep at concurrency 16, 32, and 64" - "Rides on the NVFP4-V2 checkpoint switch from #2205" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2550 + +- config-keys: + - dsv4-fp4-mi355x-vllm-agentic-mtp + scenario-type: + - agentic-coding + description: + - "Repin the image to the 08-09 nightly (f8d03e77) so the config can start at all: the previously pinned nightly no longer serves this recipe, and the 08-12 nightly (3ee2df30) memory-faults during the profile run -- eight TP workers serialize on /tmp/aiter_configs/bf16_tuned_gemm.csv.lock and the worker holding the baton dies in the post-MoE bf16 GEMM" + - "Extend the dp-attn arm (tp 8, ep 1, dp-attn true) from a single concurrency 64 point to 32, 48, 64, and 96. Experts stay TP-sharded; one scheduler per rank keeps the per-rank batch small where the single TP8 scheduler collapses, so this arm carries the high-concurrency half of the curve. Concurrency 64 overlaps the previously published point" + - "Keep pure TP8 at 1, 4, 8, 16, 32, 40, and 48. Measured on this base it peaks at 48 (7,259.7 tok/s/GPU) and falls off a cliff past it -- 2,372.4 at 64 and 1,609.3 at 96 -- so the ladder stops at the knee rather than spending runner hours below it" + - "No EP8 arm. EP8 only pays off on this model with the MegaMoE backend, which needs mori.ir.flydsl -- absent from the pinned image. On the aiter backend this config uses, EP8 has no measured advantage over the two arms above" + - "Cap --max-num-seqs at CONC rather than 2*CONC under DP-attention. The limit is per scheduler and DP-attention runs one scheduler per rank, so the existing 2x headroom admitted 16x the intended batch across the eight ranks" + - "Set --max-num-batched-tokens 8192 with --prefill-schedule-interval 8 and --long-prefill-token-threshold 16384 under DP-attention. 16384 measured 23% below 8192 at concurrency 96 (9,424.1 against 12,244.0 tok/s/GPU) on the DP-attention topology" + - "Raise --gpu-memory-utilization from 0.8 to 0.86. Weights alone take 138.85 GiB per rank on this base, leaving the KV pool the binding constraint at high concurrency" + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2590 From 95f98ead3dab5cbc488d3451faac029e0445cfd1 Mon Sep 17 00:00:00 2001 From: Jiahui Cao Date: Thu, 13 Aug 2026 06:46:47 +0000 Subject: [PATCH 2/3] perf(dsv4): enable the aiter fused shared-expert path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit DSv4-Pro is a mixed checkpoint: MXFP4 routed experts with an FP8 shared expert. vllm/models/deepseek_v4/amd/model.py gates the fused shared-expert path on VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS, which defaults to False, so the checked-in recipe was not running the configuration the validated manual runs used. The aiter side (aiter/fused_moe_dp_shared_expert.py) is present on the pinned base. The flag is mutually exclusive with expert parallelism -- _fuse_shared_experts_enabled() returns False when enable_expert_parallel is set -- which is consistent with both arms here running ep 1. DSv4-Pro 是混合精度检查点:routed experts 为 MXFP4,shared expert 为 FP8。 vllm/models/deepseek_v4/amd/model.py 通过 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS 控制 fused shared-expert 路径,该开关默认为 False,因此签入的 recipe 与已验证的 手工运行并非同一配置。aiter 侧(aiter/fused_moe_dp_shared_expert.py)在所钉基座中存在。 该开关与专家并行互斥 —— enable_expert_parallel 开启时 _fuse_shared_experts_enabled() 返回 False —— 这与本 PR 两条 arm 均为 ep 1 一致。 --- benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh | 1 + perf-changelog.yaml | 1 + 2 files changed, 2 insertions(+) diff --git a/benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh b/benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh index dca301717..20d5f6280 100644 --- a/benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh +++ b/benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh @@ -404,6 +404,7 @@ set -x export VLLM_ROCM_USE_AITER=1 #export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 export VLLM_ROCM_USE_AITER_MOE=1 +export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 sleep 180 diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 97bc7bdfb..ed6201f72 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -5930,5 +5930,6 @@ - "No EP8 arm. EP8 only pays off on this model with the MegaMoE backend, which needs mori.ir.flydsl -- absent from the pinned image. On the aiter backend this config uses, EP8 has no measured advantage over the two arms above" - "Cap --max-num-seqs at CONC rather than 2*CONC under DP-attention. The limit is per scheduler and DP-attention runs one scheduler per rank, so the existing 2x headroom admitted 16x the intended batch across the eight ranks" - "Set --max-num-batched-tokens 8192 with --prefill-schedule-interval 8 and --long-prefill-token-threshold 16384 under DP-attention. 16384 measured 23% below 8192 at concurrency 96 (9,424.1 against 12,244.0 tok/s/GPU) on the DP-attention topology" + - "Export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1. DSv4-Pro is a mixed checkpoint (MXFP4 routed experts, FP8 shared expert) and vllm/models/deepseek_v4/amd/model.py gates the fused shared-expert path on this flag; the aiter side is aiter/fused_moe_dp_shared_expert.py, present on this base. The flag defaults to False, so the checked-in recipe was not running the configuration every validated manual run of this recipe used. Note it is mutually exclusive with expert parallelism -- _fuse_shared_experts_enabled() returns False when enable_expert_parallel is set -- which is consistent with both arms here running ep 1" - "Raise --gpu-memory-utilization from 0.8 to 0.86. Weights alone take 138.85 GiB per rank on this base, leaving the KV pool the binding constraint at high concurrency" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2590 From 28c9a87ebb384fc2ada921d517343f10067fba71 Mon Sep 17 00:00:00 2001 From: Jiahui Cao Date: Thu, 13 Aug 2026 06:58:19 +0000 Subject: [PATCH 3/3] docs(dsv4): source the gmu and MNBT rationale to the runs they came from MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gpu-memory-utilization bullet cited 138.85 GiB per rank, which was measured on a different build than the pinned nightly; the figure for this topology on a pre-#51473 base is 160.66 GiB with 34.17 GiB of KV left at 0.86. The max-num-batched-tokens comparison was taken on the expert-parallel MegaMoE topology, not on the dp-attn arm, so it motivates the value rather than establishing it. Also correct the dp-attn description: dp-attn true maps to --tensor-parallel-size 1 --data-parallel-size 8, so the weights are replicated per rank rather than TP-sharded. 将 gmu 与 MNBT 的依据回溯到其真实来源的运行。138.85 GiB 来自另一构建； 本拓扑在 pre-#51473 基座上的实测为每 rank 160.66 GiB、0.86 下仅剩 34.17 GiB KV。 MNBT 对比取自专家并行 MegaMoE 拓扑而非本 arm，只作选值依据。 同时更正 dp-attn 描述：其对应 TP=1 + DP=8，权重按 rank 复制而非 TP 切分。 Co-Authored-By: Claude Opus 5 --- perf-changelog.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/perf-changelog.yaml b/perf-changelog.yaml index ed6201f72..c0b13d2af 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -5925,11 +5925,11 @@ - agentic-coding description: - "Repin the image to the 08-09 nightly (f8d03e77) so the config can start at all: the previously pinned nightly no longer serves this recipe, and the 08-12 nightly (3ee2df30) memory-faults during the profile run -- eight TP workers serialize on /tmp/aiter_configs/bf16_tuned_gemm.csv.lock and the worker holding the baton dies in the post-MoE bf16 GEMM" - - "Extend the dp-attn arm (tp 8, ep 1, dp-attn true) from a single concurrency 64 point to 32, 48, 64, and 96. Experts stay TP-sharded; one scheduler per rank keeps the per-rank batch small where the single TP8 scheduler collapses, so this arm carries the high-concurrency half of the curve. Concurrency 64 overlaps the previously published point" + - "Extend the dp-attn arm (tp 8, ep 1, dp-attn true) from a single concurrency 64 point to 32, 48, 64, and 96. dp-attn true maps to --tensor-parallel-size 1 --data-parallel-size 8, so each rank holds a full copy of the weights; one scheduler per rank keeps the per-rank batch small where the single TP8 scheduler collapses, so this arm carries the high-concurrency half of the curve. Concurrency 64 overlaps the previously published point" - "Keep pure TP8 at 1, 4, 8, 16, 32, 40, and 48. Measured on this base it peaks at 48 (7,259.7 tok/s/GPU) and falls off a cliff past it -- 2,372.4 at 64 and 1,609.3 at 96 -- so the ladder stops at the knee rather than spending runner hours below it" - "No EP8 arm. EP8 only pays off on this model with the MegaMoE backend, which needs mori.ir.flydsl -- absent from the pinned image. On the aiter backend this config uses, EP8 has no measured advantage over the two arms above" - "Cap --max-num-seqs at CONC rather than 2*CONC under DP-attention. The limit is per scheduler and DP-attention runs one scheduler per rank, so the existing 2x headroom admitted 16x the intended batch across the eight ranks" - - "Set --max-num-batched-tokens 8192 with --prefill-schedule-interval 8 and --long-prefill-token-threshold 16384 under DP-attention. 16384 measured 23% below 8192 at concurrency 96 (9,424.1 against 12,244.0 tok/s/GPU) on the DP-attention topology" + - "Set --max-num-batched-tokens 8192 with --prefill-schedule-interval 8 and --long-prefill-token-threshold 16384 under DP-attention. 16384 measured 23% below 8192 at concurrency 96 (9,424.1 against 12,244.0 tok/s/GPU); that comparison was taken on the expert-parallel MegaMoE topology rather than on this arm, so it motivates the value without establishing it here and the sweep measures it" - "Export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1. DSv4-Pro is a mixed checkpoint (MXFP4 routed experts, FP8 shared expert) and vllm/models/deepseek_v4/amd/model.py gates the fused shared-expert path on this flag; the aiter side is aiter/fused_moe_dp_shared_expert.py, present on this base. The flag defaults to False, so the checked-in recipe was not running the configuration every validated manual run of this recipe used. Note it is mutually exclusive with expert parallelism -- _fuse_shared_experts_enabled() returns False when enable_expert_parallel is set -- which is consistent with both arms here running ep 1" - - "Raise --gpu-memory-utilization from 0.8 to 0.86. Weights alone take 138.85 GiB per rank on this base, leaving the KV pool the binding constraint at high concurrency" + - "Raise --gpu-memory-utilization from 0.8 to 0.86. This base predates vllm #51473, so the mxfp4 oracle rounds inter_dim 384 up to 512 and the weights inflate; under this topology on the previously pinned nightly, also pre-#51473, weight residency measured 160.66 GiB per rank and left only 34.17 GiB of KV at 0.86. Utilization is the only lever in the recipe that offsets any of that" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2590