Skip to content

[AMD] [AgentX] dsv4-fp4-mi355x-vllm-agentic-mtp, update image and extend the DP-attention concurrency curve #2590

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jiacao-amd wants to merge 3 commits into
base: main
Choose a base branch
from
+32 −3
Open

[AMD] [AgentX] dsv4-fp4-mi355x-vllm-agentic-mtp, update image and extend the DP-attention concurrency curve #2590

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1bbc5c4
perf(dsv4-fp4-mi355x-vllm-agentic-mtp): repin to the 08-09 nightly an…
Aug 13, 2026
95f98ea
perf(dsv4): enable the aiter fused shared-expert path
Aug 13, 2026
28c9a87
docs(dsv4): source the gmu and MNBT rationale to the runs they came from
Aug 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -371,10 +371,22 @@
EP_ARGS=(--enable-expert-parallel)
fi

DP_SCHED_ARGS=()
if [ "$DP_ATTENTION" = "true" ]; then
DP_SCHED_ARGS=(
--prefill-schedule-interval 8
--max-num-batched-tokens 8192
--long-prefill-token-threshold 16384
)

Check warning on line 380 in benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh

View check run for this annotation

Claude / Claude Code Review

long-prefill-token-threshold larger than max-num-batched-tokens is a no-op 

With DP_ATTENTION=true, --long-prefill-token-threshold 16384 is set larger than --max-num-batched-tokens 8192, making it an inert no-op: vLLM's V1 scheduler caps a request's per-step token count at the threshold *before* clamping to the token budget, so a threshold above the budget can never bind. This defeats the stated purpose ('long prefills are chunked rather than starving decode'); the sibling recipe dsv4_fp4_b300_vllm_mtp.sh uses the correct inverse relationship (threshold 512 << budget),
Comment on lines +374 to +380

@claude claude Bot Aug 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 With DP_ATTENTION=true, --long-prefill-token-threshold 16384 is set larger than --max-num-batched-tokens 8192, making it an inert no-op: vLLM's V1 scheduler caps a request's per-step token count at the threshold before clamping to the token budget, so a threshold above the budget can never bind. This defeats the stated purpose ('long prefills are chunked rather than starving decode'); the sibling recipe dsv4_fp4_b300_vllm_mtp.sh uses the correct inverse relationship (threshold 512 << budget), and this value should likely be similarly small (e.g. 512).

Extended reasoning...

The bug. benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh:374-380 sets, under DP_ATTENTION=true:

--max-num-batched-tokens 8192
--long-prefill-token-threshold 16384

In vLLM's V1 scheduler (vllm/v1/core/sched/scheduler.py), long_prefill_token_threshold is applied as a per-step cap on a request's num_new_tokens, and that cap is applied before the min() with token_budget (which derives from --max-num-batched-tokens). Concretely, both the running-request loop and the waiting-request (prefill) loop do:

if 0 < self.scheduler_config.long_prefill_token_threshold < num_new_tokens:
    num_new_tokens = self.scheduler_config.long_prefill_token_threshold
num_new_tokens = min(num_new_tokens, token_budget)

Since min(min(N, 16384), 8192) == min(N, 8192) for any N, the 16384 clamp can never be the binding constraint when the budget is only 8192 — it is a pure no-op. This directly contradicts the PR description's stated rationale for adding the flag: "so long prefills are chunked rather than starving decode." As written, a single long prefill still consumes the full 8192-token budget in one scheduler step, which is exactly the decode-starvation scenario the flag is meant to prevent.

Proof by walkthrough. Take a request with a 40,000-token prompt.

  1. num_new_tokens is initially computed as the remaining prompt tokens to process this step, e.g. 40000 (before any budget/threshold clamp).
  2. Threshold check: 0 < 16384 < 40000 is true, so num_new_tokens is set to 16384.
  3. Budget clamp: num_new_tokens = min(16384, 8192) = 8192.
  4. The final scheduled chunk size is 8192 — identical to what would have been scheduled with no threshold set at all (long_prefill_token_threshold=0 disables the cap entirely, per its own docstring). The 16384 value never had any effect on the outcome for any prompt length, because step 3 always overrides step 2 whenever threshold > budget.

Why this isn't caught elsewhere. I checked whether a second mechanism (e.g., classifying a request as "long" against its total prompt length for a concurrent-long-prefill limiter, as one reviewer speculated) exists in this vLLM version that might make the 16384 value meaningful independently of the per-step cap. It does not: grepping the scheduler and scheduler-config modules for long_prefill / max_long_partial_prefills turns up only the three call sites above, all of which use the threshold identically as a per-step num_new_tokens cap, not a request-classification threshold compared against num_prompt_tokens. The field's docstring ("a request is considered long if the prompt is longer than this number of tokens") is a legacy description of intent, but the actual gating logic operates purely on per-step token counts, so the threshold's real effect is exactly what the code does with it — cap the chunk, then get overridden by the smaller budget.

Impact. Nothing crashes and the arm still runs — --prefill-schedule-interval 8 provides separate prefill-scheduling cadence throttling, and the sweep measures the arm's throughput regardless of whether this particular flag does anything. But the flag is dead configuration: it does not chunk long prefills any more tightly than --max-num-batched-tokens already does on its own, so any decode-starvation behavior from long prefills under DP-attention is unmitigated by this addition, contrary to what the PR description claims it does.

Fix. Lower --long-prefill-token-threshold below --max-num-batched-tokens (8192), e.g. to 512, mirroring the working pattern in the sibling recipe benchmarks/single_node/agentic/dsv4_fp4_b300_vllm_mtp.sh (--long-prefill-token-threshold 512 alongside --max-num-batched-tokens 8192/16384), which is commented there as keeping "decode latency bounded under load."

fi

# AgentX concurrency counts live session trees, not individual requests.
# Subagent fan-out can push instantaneous request concurrency above CONC, so
# leave 2x headroom rather than clipping those bursts at the scheduler.
MAX_NUM_SEQS=$((2 * CONC))
if [ "$DP_ATTENTION" = "true" ]; then
MAX_NUM_SEQS="$CONC"
fi

# DeepSeek-V4-Pro ships a native MTP head. AgentX throughput pins its
# three-token draft to the committed thinking-on golden acceptance length;
Expand All @@ -392,6 +404,7 @@
export VLLM_ROCM_USE_AITER=1
#export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
export VLLM_ROCM_USE_AITER_MOE=1
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1

sleep 180

Expand All @@ -406,7 +419,8 @@
--kv-cache-dtype fp8
"${PARALLEL_ARGS[@]}"
"${EP_ARGS[@]}"
--gpu-memory-utilization 0.8
"${DP_SCHED_ARGS[@]}"
--gpu-memory-utilization 0.86
--moe-backend aiter
--compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}'
--speculative-config "$SPEC_CONFIG"
Expand Down
4 changes: 2 additions & 2 deletions configs/amd-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1301,7 +1301,7 @@ qwen3.5-fp8-mi355x-sglang-agentic-hicache:
# is no longer available from Docker Hub. Throughput uses the three-token golden
# synthetic acceptance length, while eval-only runs retain real verification.
dsv4-fp4-mi355x-vllm-agentic-mtp:
image: vllm/vllm-openai-rocm:nightly-821717118fc26667dd474b9b0ab81d29259dfc5c
image: vllm/vllm-openai-rocm:nightly-f8d03e77416bf90c49acbe50e233275722f02c4b
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: cluster:mi355x-amds
Expand All @@ -1313,7 +1313,7 @@ dsv4-fp4-mi355x-vllm-agentic-mtp:
- dram-utilization: 0.60
search-space:
- { tp: 8, ep: 1, dp-attn: false, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 4, 8, 16, 32, 40, 48] }
- { tp: 8, ep: 1, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [64], router: { name: vllm-router, version: "0.1.14" } }
- { tp: 8, ep: 1, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [32, 48, 64, 96], router: { name: vllm-router, version: "0.1.14" } }
# LMCache invalid-block recovery currently assumes one KV-cache group,
# while MTP creates two. Restore these points after the upstream hybrid
# KV recovery fix lands: https://github.com/vllm-project/vllm/pull/45497
Expand Down
15 changes: 15 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5918,3 +5918,18 @@
- "Add a TEP2 arm (tp 2, ep 2) to the qwen3.5-fp4-b200-sglang-mtp 8k/1k sweep at concurrency 16, 32, and 64"
- "Rides on the NVFP4-V2 checkpoint switch from #2205"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2550

- config-keys:
- dsv4-fp4-mi355x-vllm-agentic-mtp
scenario-type:
- agentic-coding
description:
- "Repin the image to the 08-09 nightly (f8d03e77) so the config can start at all: the previously pinned nightly no longer serves this recipe, and the 08-12 nightly (3ee2df30) memory-faults during the profile run -- eight TP workers serialize on /tmp/aiter_configs/bf16_tuned_gemm.csv.lock and the worker holding the baton dies in the post-MoE bf16 GEMM"
- "Extend the dp-attn arm (tp 8, ep 1, dp-attn true) from a single concurrency 64 point to 32, 48, 64, and 96. dp-attn true maps to --tensor-parallel-size 1 --data-parallel-size 8, so each rank holds a full copy of the weights; one scheduler per rank keeps the per-rank batch small where the single TP8 scheduler collapses, so this arm carries the high-concurrency half of the curve. Concurrency 64 overlaps the previously published point"
- "Keep pure TP8 at 1, 4, 8, 16, 32, 40, and 48. Measured on this base it peaks at 48 (7,259.7 tok/s/GPU) and falls off a cliff past it -- 2,372.4 at 64 and 1,609.3 at 96 -- so the ladder stops at the knee rather than spending runner hours below it"
- "No EP8 arm. EP8 only pays off on this model with the MegaMoE backend, which needs mori.ir.flydsl -- absent from the pinned image. On the aiter backend this config uses, EP8 has no measured advantage over the two arms above"
- "Cap --max-num-seqs at CONC rather than 2*CONC under DP-attention. The limit is per scheduler and DP-attention runs one scheduler per rank, so the existing 2x headroom admitted 16x the intended batch across the eight ranks"
- "Set --max-num-batched-tokens 8192 with --prefill-schedule-interval 8 and --long-prefill-token-threshold 16384 under DP-attention. 16384 measured 23% below 8192 at concurrency 96 (9,424.1 against 12,244.0 tok/s/GPU); that comparison was taken on the expert-parallel MegaMoE topology rather than on this arm, so it motivates the value without establishing it here and the sweep measures it"
- "Export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1. DSv4-Pro is a mixed checkpoint (MXFP4 routed experts, FP8 shared expert) and vllm/models/deepseek_v4/amd/model.py gates the fused shared-expert path on this flag; the aiter side is aiter/fused_moe_dp_shared_expert.py, present on this base. The flag defaults to False, so the checked-in recipe was not running the configuration every validated manual run of this recipe used. Note it is mutually exclusive with expert parallelism -- _fuse_shared_experts_enabled() returns False when enable_expert_parallel is set -- which is consistent with both arms here running ep 1"
- "Raise --gpu-memory-utilization from 0.8 to 0.86. This base predates vllm #51473, so the mxfp4 oracle rounds inter_dim 384 up to 512 and the weights inflate; under this topology on the previously pinned nightly, also pre-#51473, weight residency measured 160.66 GiB per rank and left only 34.17 GiB of KV at 0.86. Utilization is the only lever in the recipe that offsets any of that"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2590
Loading