[AMD] [AgentX] dsv4-fp4-mi355x-vllm-agentic-mtp, update image and extend the DP-attention concurrency curve - #2590
[AMD] [AgentX] dsv4-fp4-mi355x-vllm-agentic-mtp, update image and extend the DP-attention concurrency curve #2590jiacao-amd wants to merge 3 commits into
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
9702fd6 to
b816427
Compare
jiacao-amd
changed the title
jiacao-amd
changed the title
…d complete the DP-attention curve Repin to the 08-09 nightly (f8d03e77) so the config can start at all: the previously pinned tag no longer serves this recipe, and the 08-12 nightly (3ee2df30) memory-faults during the profile run. Two arms: pure TP8 through its measured knee at 48, and DP-attention with the experts left TP-sharded (ep 1) extended from a single conc-64 point to 32/48/64/96. No EP8 arm -- EP8 only pays off with the MegaMoE backend, which needs mori.ir.flydsl, absent from this image. Recipe side, all DP-attention-conditional: cap --max-num-seqs at CONC rather than 2*CONC (the limit is per scheduler and DP-attention runs one per rank), set --max-num-batched-tokens 8192 with --prefill-schedule-interval 8 and --long-prefill-token-threshold 16384, and raise --gpu-memory-utilization from 0.8 to 0.86. 重新钉到 08-09 nightly（f8d03e77），配置才能起得来：原先钉的 tag 已经跑不动这个 recipe，而 08-12 nightly（3ee2df30）在 profile run 阶段会 memory fault。 两条 arm：纯 TP8 跑到实测拐点 48，以及专家仍按 TP 切分（ep 1）的 DP-attention arm，从原来单点 conc 64 扩展到 32/48/64/96。不加 EP8 arm —— EP8 只有配 MegaMoE 后端才划算，而它需要 mori.ir.flydsl，该镜像里没有。 recipe 侧改动均只在 DP-attention 下生效：--max-num-seqs 从 2*CONC 收到 CONC （该上限是每个 scheduler 生效的，而 DP-attention 每个 rank 一个 scheduler）， 设置 --max-num-batched-tokens 8192 配合 --prefill-schedule-interval 8 与 --long-prefill-token-threshold 16384，并把 --gpu-memory-utilization 从 0.8 提到 0.86。
b816427 to
1bbc5c4
Compare
DSv4-Pro is a mixed checkpoint: MXFP4 routed experts with an FP8 shared expert. vllm/models/deepseek_v4/amd/model.py gates the fused shared-expert path on VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS, which defaults to False, so the checked-in recipe was not running the configuration the validated manual runs used. The aiter side (aiter/fused_moe_dp_shared_expert.py) is present on the pinned base. The flag is mutually exclusive with expert parallelism -- _fuse_shared_experts_enabled() returns False when enable_expert_parallel is set -- which is consistent with both arms here running ep 1. DSv4-Pro 是混合精度检查点:routed experts 为 MXFP4,shared expert 为 FP8。 vllm/models/deepseek_v4/amd/model.py 通过 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS 控制 fused shared-expert 路径,该开关默认为 False,因此签入的 recipe 与已验证的 手工运行并非同一配置。aiter 侧(aiter/fused_moe_dp_shared_expert.py)在所钉基座中存在。 该开关与专家并行互斥 —— enable_expert_parallel 开启时 _fuse_shared_experts_enabled() 返回 False —— 这与本 PR 两条 arm 均为 ep 1 一致。
The gpu-memory-utilization bullet cited 138.85 GiB per rank, which was measured on a different build than the pinned nightly; the figure for this topology on a pre-#51473 base is 160.66 GiB with 34.17 GiB of KV left at 0.86. The max-num-batched-tokens comparison was taken on the expert-parallel MegaMoE topology, not on the dp-attn arm, so it motivates the value rather than establishing it. Also correct the dp-attn description: dp-attn true maps to --tensor-parallel-size 1 --data-parallel-size 8, so the weights are replicated per rank rather than TP-sharded. 将 gmu 与 MNBT 的依据回溯到其真实来源的运行。138.85 GiB 来自另一构建； 本拓扑在 pre-#51473 基座上的实测为每 rank 160.66 GiB、0.86 下仅剩 34.17 GiB KV。 MNBT 对比取自专家并行 MegaMoE 拓扑而非本 arm，只作选值依据。 同时更正 dp-attn 描述：其对应 TP=1 + DP=8，权重按 rank 复制而非 TP 切分。 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Claude finished @jiacao-amd's task in 3m 27s —— View job PR Review
LGTM - no blocking issues found Checks performed:
Note: the inferencemax-repos MCP server was not available in this run, so I could not verify against vLLM source that the ROCm 08-09 nightly accepts |
jiacao-amd
changed the title
jiacao-amd
changed the title
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
LGTM — the one inline finding is a nit (an inert flag value, not a functional break), and the rest of the repin/config changes are well-reasoned and self-contained.
Extended reasoning...
Overview
This PR repins the dsv4-fp4-mi355x-vllm-agentic-mtp recipe to a new ROCm nightly and extends the DP-attention concurrency ladder. Changes are confined to a benchmark launch script, the corresponding amd-master.yaml search-space entry, and an append-only perf-changelog.yaml entry — no production serving code, auth, or shared infrastructure is touched.
Security risks
None. This is benchmark/tuning configuration for an internal sweep harness; it does not touch request handling, credentials, or externally reachable code paths.
Level of scrutiny
Low-to-moderate is appropriate here: it's a data/config change (image pin bump, concurrency list, and a handful of vLLM CLI flags gated behind DP_ATTENTION=true) rather than a logic change to shared code. The one inline finding (a --long-prefill-token-threshold value that can never bind under the accompanying --max-num-batched-tokens budget) is flagged as a Nit by the bug hunter — it doesn't crash or break the run, it just leaves a piece of the stated 'chunk long prefills' rationale as dead configuration. The sweep itself will still execute and produce valid throughput numbers for the arm.
Other factors
The PR description is thorough and grounds each flag change (max-num-seqs capping, gpu-memory-utilization bump, aiter fused shared-expert flag) in specific measurements or code references. The PR is still in draft pending sweep results per its own description, and full-sweep-enabled is applied appropriately given the image/topology changes. No unresolved reviewer threads exist in the timeline.
| DP_SCHED_ARGS=() | ||
| if [ "$DP_ATTENTION" = "true" ]; then | ||
| DP_SCHED_ARGS=( | ||
| --prefill-schedule-interval 8 | ||
| --max-num-batched-tokens 8192 | ||
| --long-prefill-token-threshold 16384 | ||
| ) |
There was a problem hiding this comment.
🟡 With DP_ATTENTION=true, --long-prefill-token-threshold 16384 is set larger than --max-num-batched-tokens 8192, making it an inert no-op: vLLM's V1 scheduler caps a request's per-step token count at the threshold before clamping to the token budget, so a threshold above the budget can never bind. This defeats the stated purpose ('long prefills are chunked rather than starving decode'); the sibling recipe dsv4_fp4_b300_vllm_mtp.sh uses the correct inverse relationship (threshold 512 << budget), and this value should likely be similarly small (e.g. 512).
Extended reasoning...
The bug. benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh:374-380 sets, under DP_ATTENTION=true:
--max-num-batched-tokens 8192
--long-prefill-token-threshold 16384
In vLLM's V1 scheduler (vllm/v1/core/sched/scheduler.py), long_prefill_token_threshold is applied as a per-step cap on a request's num_new_tokens, and that cap is applied before the min() with token_budget (which derives from --max-num-batched-tokens). Concretely, both the running-request loop and the waiting-request (prefill) loop do:
if 0 < self.scheduler_config.long_prefill_token_threshold < num_new_tokens:
num_new_tokens = self.scheduler_config.long_prefill_token_threshold
num_new_tokens = min(num_new_tokens, token_budget)Since min(min(N, 16384), 8192) == min(N, 8192) for any N, the 16384 clamp can never be the binding constraint when the budget is only 8192 — it is a pure no-op. This directly contradicts the PR description's stated rationale for adding the flag: "so long prefills are chunked rather than starving decode." As written, a single long prefill still consumes the full 8192-token budget in one scheduler step, which is exactly the decode-starvation scenario the flag is meant to prevent.
Proof by walkthrough. Take a request with a 40,000-token prompt.
num_new_tokensis initially computed as the remaining prompt tokens to process this step, e.g.40000(before any budget/threshold clamp).- Threshold check:
0 < 16384 < 40000is true, sonum_new_tokensis set to16384. - Budget clamp:
num_new_tokens = min(16384, 8192) = 8192. - The final scheduled chunk size is
8192— identical to what would have been scheduled with no threshold set at all (long_prefill_token_threshold=0disables the cap entirely, per its own docstring). The 16384 value never had any effect on the outcome for any prompt length, because step 3 always overrides step 2 whenever threshold > budget.
Why this isn't caught elsewhere. I checked whether a second mechanism (e.g., classifying a request as "long" against its total prompt length for a concurrent-long-prefill limiter, as one reviewer speculated) exists in this vLLM version that might make the 16384 value meaningful independently of the per-step cap. It does not: grepping the scheduler and scheduler-config modules for long_prefill / max_long_partial_prefills turns up only the three call sites above, all of which use the threshold identically as a per-step num_new_tokens cap, not a request-classification threshold compared against num_prompt_tokens. The field's docstring ("a request is considered long if the prompt is longer than this number of tokens") is a legacy description of intent, but the actual gating logic operates purely on per-step token counts, so the threshold's real effect is exactly what the code does with it — cap the chunk, then get overridden by the smaller budget.
Impact. Nothing crashes and the arm still runs — --prefill-schedule-interval 8 provides separate prefill-scheduling cadence throttling, and the sweep measures the arm's throughput regardless of whether this particular flag does anything. But the flag is dead configuration: it does not chunk long prefills any more tightly than --max-num-batched-tokens already does on its own, so any decode-starvation behavior from long prefills under DP-attention is unmitigated by this addition, contrary to what the PR description claims it does.
Fix. Lower --long-prefill-token-threshold below --max-num-batched-tokens (8192), e.g. to 512, mirroring the working pattern in the sibling recipe benchmarks/single_node/agentic/dsv4_fp4_b300_vllm_mtp.sh (--long-prefill-token-threshold 512 alongside --max-num-batched-tokens 8192/16384), which is commented there as keeping "decode latency bounded under load."
seungrokj
changed the title
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31676544794 |
|
/reuse-sweep-run |
2 participants
Summary
Update the
dsv4-fp4-mi355x-vllm-agentic-mtpimage tovllm/vllm-openai-rocm:nightly-f8d03e77416bf90c49acbe50e233275722f02c4band complete the DP-attention half of the concurrency curve. MI355X, 8 GPU, FP4 vLLM,deepseek-ai/DeepSeek-V4-Pro, agentic-coding trace replay with MTP (acceptance length 2.49).Arms
Changes
configs/amd-master.yaml: update the pinned image; extend thedp-attnarm from one point to four.benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh: under DP-attention cap--max-num-seqsatCONC(the limit is per scheduler) and set--max-num-batched-tokens 8192,--prefill-schedule-interval 8,--long-prefill-token-threshold 16384; exportVLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1; raise--gpu-memory-utilizationto 0.86.perf-changelog.yaml: corresponding entry.full-sweep-enabled: the image pin and parallelism arms both change, so cached results are not comparable.中文说明
将
dsv4-fp4-mi355x-vllm-agentic-mtp镜像更新为vllm/vllm-openai-rocm:nightly-f8d03e77416bf90c49acbe50e233275722f02c4b，并补齐并发曲线中 DP-attention 的一半。变更：更新所钉镜像；
dp-attn组合由单点扩展为 32/48/64/96；DP-attention 下--max-num-seqs收紧为CONC（该上限按 scheduler 生效），并设置--max-num-batched-tokens 8192、--prefill-schedule-interval 8、--long-prefill-token-threshold 16384；导出VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1；--gpu-memory-utilization提高到 0.86。申请
full-sweep-enabled：镜像 pin 与并行组合均变化，旧结果不可比。