perf(qwen3.5-fp4-b200-sglang-mtp): add TEP2 arm on NVFP4-V2 / perf(qwen3.5-fp4-b200-sglang-mtp): 基于 NVFP4-V2 添加 TEP2 组合

[hshrivastava-droid]

Summary

Add a TEP2 (tp 2, ep 2) arm to the qwen3.5-fp4-b200-sglang-mtp 8k/1k fixed-sequence-length MTP sweep at concurrency 16, 32, and 64. This arm is measured on the nvidia/Qwen3.5-397B-A17B-NVFP4-V2 checkpoint that landed via #2205; no other configs are touched.

Scope:

• Config key: qwen3.5-fp4-b200-sglang-mtp
• Hardware: NVIDIA B200
• Precision / framework: FP4 with SGLang lmsysorg/sglang:v0.5.14-cu130
• Checkpoint: nvidia/Qwen3.5-397B-A17B-NVFP4-V2 (inherited from Switch Qwen3.5 FP4 SGLang configs to NVFP4 V2 / 将 Qwen3.5 FP4 SGLang 配置切换到 NVFP4 V2 #2205, resolved via runners/launch_b200-dgxc.sh)
• Workload: aggregated, single-node, 8k/1k fixed-sequence throughput and regular evals; MTP spec-decoding with --use-chat-template
• New arm topology: tp 2, ep 2
• New arm concurrencies: 16, 32, 64

Changes

• configs/nvidia-master.yaml: append { tp: 2, ep: 2, conc-list: [16, 32, 64], spec-decoding: mtp } to the qwen3.5-fp4-b200-sglang-mtp 8k/1k search-space.
• perf-changelog.yaml: append the corresponding entry, calling out that the arm rides on the NVFP4-V2 checkpoint switch from Switch Qwen3.5 FP4 SGLang configs to NVFP4 V2 / 将 Qwen3.5 FP4 SGLang 配置切换到 NVFP4 V2 #2205, with the PR link backfilled.

Results

Numbers are from the exact-head Run Sweep on qwen3.5-fp4-b200-sglang-mtp and compared against the current published tp 2, ep 1 rows on main:

• Expert parallelism 2 measured against a same-node tp 2, ep 1 reference at concurrency 32: +1.11% tok/s/GPU and +2.28% tok/s/user, non-overlapping on both axes across two samples.
• Combined against the currently published tp 2, ep 1 rows on main:
  • c = 16: 7534.5 tok/s/GPU at 120.16 tok/s/user (+5.85% / +5.17%)
  • c = 32: 10319.8 tok/s/GPU at 80.41 tok/s/user (+8.94% / +7.26%)
  • c = 64: 13290.8 tok/s/GPU at 51.97 tok/s/user (+9.46% / +9.06%)
• Correctness held on the promoted configuration: GSM8K strict-match 0.967–0.975 and flexible-extract 0.960–0.968 against a 0.94 threshold; coherence passed; decode CUDA graph active; MTP acceptance baseline-like with no backend fallback.

Notes

• Rides on top of #2205; the model / launch-script / squash paths for NVFP4-V2 on b200-dgxc are already merged and require no further edits here.
• Single-node AGG SGLang: the upstream SGLang Qwen3.5 cookbook is the recipe-of-record; the merged recipe update from Switch Qwen3.5 FP4 SGLang configs to NVFP4 V2 / 将 Qwen3.5 FP4 SGLang 配置切换到 NVFP4 V2 #2205 covers the checkpoint. The TEP2 topology is an additional throughput arm on the same server arguments, not a new recipe.

中文说明

概要

向 qwen3.5-fp4-b200-sglang-mtp 的 8k/1k 定长序列 MTP 扫描中新增一个 TEP2（tp 2, ep 2）组合，覆盖并发度 16、32、64。本 PR 基于 #2205 落地的 nvidia/Qwen3.5-397B-A17B-NVFP4-V2 检查点进行测量；不涉及其他配置。

范围：

• 配置键：qwen3.5-fp4-b200-sglang-mtp
• 硬件：NVIDIA B200
• 精度 / 框架：FP4，SGLang lmsysorg/sglang:v0.5.14-cu130
• 检查点：nvidia/Qwen3.5-397B-A17B-NVFP4-V2（继承自 Switch Qwen3.5 FP4 SGLang configs to NVFP4 V2 / 将 Qwen3.5 FP4 SGLang 配置切换到 NVFP4 V2 #2205，通过 runners/launch_b200-dgxc.sh 路由）
• 工作负载：聚合式、单节点、8k/1k 定长序列吞吐及常规评估；MTP 推测解码并启用 --use-chat-template
• 新增组合拓扑：tp 2，ep 2
• 新增并发度：16、32、64

变更内容

• configs/nvidia-master.yaml：向 qwen3.5-fp4-b200-sglang-mtp 8k/1k 的 search-space 追加 { tp: 2, ep: 2, conc-list: [16, 32, 64], spec-decoding: mtp }。
• perf-changelog.yaml：在文件末尾追加对应条目，并注明该组合建立在 Switch Qwen3.5 FP4 SGLang configs to NVFP4 V2 / 将 Qwen3.5 FP4 SGLang 配置切换到 NVFP4 V2 #2205 切换到 NVFP4-V2 检查点之后，PR 链接已回填。

结果

数据来自 qwen3.5-fp4-b200-sglang-mtp 对应 head 的 Run Sweep，与 main 上当前发布的 tp 2, ep 1 结果对比：

• 在并发 32 上，ep 2 相对同节点 tp 2, ep 1 参考：tok/s/GPU +1.11%，tok/s/user +2.28%，两次采样在两个轴上均不重叠。
• 相对 main 上当前发布的 tp 2, ep 1 行：
  • c = 16：7534.5 tok/s/GPU、120.16 tok/s/user（+5.85% / +5.17%）
  • c = 32：10319.8 tok/s/GPU、80.41 tok/s/user（+8.94% / +7.26%）
  • c = 64：13290.8 tok/s/GPU、51.97 tok/s/user（+9.46% / +9.06%）
• 正确性在上榜配置上保持：GSM8K strict-match 0.967–0.975、flexible-extract 0.960–0.968，均高于 0.94 阈值；一致性通过；decode CUDA graph 已启用；MTP acceptance 与基线一致，未回退到后端。

备注

• 本 PR 基于 #2205；b200-dgxc 上 NVFP4-V2 的模型、启动脚本与 squash 路径已合并，此处无需额外改动。
• 单节点 AGG SGLang：以上游 SGLang Qwen3.5 cookbook 为对齐来源；Switch Qwen3.5 FP4 SGLang configs to NVFP4 V2 / 将 Qwen3.5 FP4 SGLang 配置切换到 NVFP4 V2 #2205 合并的 recipe 更新已覆盖检查点切换。本次 TEP2 拓扑是在相同 server 参数下追加的一条吞吐组合，并非新增独立 recipe。

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[claude]

LGTM — straightforward benchmark config addition.

Extended reasoning...

Overview

This PR adds a single new search-space arm (tp:2, ep:2, conc-list:[16,32,64], spec-decoding:mtp) to the existing qwen3.5-fp4-b200-sglang-mtp sweep in configs/nvidia-master.yaml, and appends a corresponding perf-changelog.yaml entry documenting the measured gains (+5-9% tok/s/GPU and tok/s/user across concurrencies 16/32/64) and correctness checks (GSM8K strict-match/flexible-extract above threshold, coherence pass, MTP acceptance baseline-like).

Security risks

None. This is a pure benchmark configuration change — no code paths, auth, or data handling are touched.

Level of scrutiny

Low. This follows the exact same mechanical pattern used throughout this file for adding sweep arms (confirmed conc-list syntax matches other entries in the config), and the changelog entry follows the established schema with the required fields (config-keys, scenario-type, description, pr-link) matching the format of adjacent entries.

Other factors

No outstanding review comments in the timeline beyond the standard automated recipe-reminder bot message. The perf-changelog description includes specific before/after benchmark numbers and correctness validation results, consistent with prior similar TEP-arm-addition PRs in the repo history.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31432304739
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=31432304739

[Ankur-singh]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/31432304739
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/31432304739
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.

Additional detail section:

Scope of the change. This PR adds a single search-space row to the existing qwen3.5-fp4-b200-sglang-mtp entry — { tp: 2, ep: 2, conc-list: [16, 32, 64], spec-decoding: mtp } — plus the corresponding perf-changelog.yaml entry. No benchmark script, recipe file, image pin, or launcher is touched.

Validation and eval evidence. Run 31432304739 ran on the exact PR head e3643999, attempt 1, and settled at 21 success / 12 skipped / 0 failures. All three new arm points executed and passed as distinct TP2/EP2 jobs at concurrency 16, 32, and 64, alongside the pre-existing TP4 c4 and TP2 c4/c8/c16/c32/c64 rows. Evals are non-skipped and green for the new arm specifically — eval / … TP2/EP2 mtp c32 and … TP2/EP2 mtp c64 — with collect-evals succeeding. The skipped lanes (multi-node, agentic, agentic-eval) have no matrix entries for this single-node config.

Speculative decoding and chat template. benchmarks/single_node/fixed_seq_len/qwen3.5_fp4_b200_mtp.sh passes --use-chat-template to the serving benchmark, so the MTP acceptance-length distribution is measured against chat-formatted prompts rather than raw tokens. The script runs EAGLE with --speculative-num-steps 3, --speculative-eagle-topk 1, --speculative-num-draft-tokens 4, unchanged by this PR. The golden-AL item is not applicable: this is a fixed-seq-len 8k/1k submission, not an agentic workload, so it uses measured acceptance rather than the simulated synthetic acceptance that item governs.

The new arm is a real configuration, not a relabel. The script consumes expert parallelism directly — --tensor-parallel-size=$TP --data-parallel-size=1 --expert-parallel-size=$EP_SIZE — so ep: 2 genuinely changes the served topology relative to the existing ep: 1 rows, and the sweep surfaced the points as separate TP2/EP2 jobs. Expert parallelism redistributes the same MoE computation across GPUs; it removes no model-architecture FLOPs, and the promoted configuration carries its own passing GSM8K evals in the linked run.

Model and scenario scope. MODELS.md lists Qwen3.5-397B-A17B as active for Single-turn 8k1k limited to fp8/fp4, and records the deprecations as Single-turn 1k1k, Single-turn 1k8k, and all bf16 recipes. This submission is 8k1k NVFP4 and adds no 1k1k lane. MODELS.md also records the engine expectation for this model as the native/upstream SGLang engine with native MTP, which is exactly what this arm exercises on the upstream lmsysorg/sglang:v0.5.14-cu130 image.

Single-node recipe publication — left unchecked, needs a scope call. This is a single-node SGLang submission, so the item is at least arguably in scope, and no upstream recipe or cookbook link has been provided on this PR. Against that: the PR introduces no recipe file and no new serving script — it extends an already-published single-node config's search space with a parallelism variant, and the underlying qwen3.5_fp4_b200_mtp.sh recipe is unchanged. I am not checking this box on my own reading of "recipe in this PR"; if the intent is that a new TP/EP arm on an existing single-node recipe requires a matching cookbook update, this PR needs that upstream link merged first.

No engine or serving-stack patching. The diff is two YAML files. No .patch, git apply, sed, site-packages edit, monkey-patch, container overwrite, or forked wheel, and the pinned upstream image is unchanged.

Signed: Ankur-singh

[Klaud-Cold]

❌❌❌ REJECTED ❌❌❌

@Ankur-singh Blocking issue: this is a single-node SGLang submission, but no merged upstream recipe PR or published cookbook page is linked in the sign-off (the recipe checklist item is unchecked). The new tp: 2, ep: 2 arm changes the deployment-defining parallelism of qwen3.5-fp4-b200-sglang-mtp, so it needs the corresponding SGLang cookbook / vLLM recipes documentation merged upstream and linked before merge.

✅ Check 0 (CODEOWNER): PASS — Ankur-singh owns configs/nvidia-master.yaml per CODEOWNERS; perf-changelog.yaml is catch-all-only and covered.
✅ Check 1 (sweep on in-PR commit): PASS — run 31432304739 on head e364399 has green, executed single-node 8k1k / (incl. TP2/EP2 c16/c32/c64) and eval / check-runs.
✅ Check 2 (evals real): PASS — GSM8K strict 0.968/0.970 on the new TP2/EP2 arm (0.971 on the EP1 reference) vs the 0.94 qwen3.5 bar, on the PR's image lmsysorg/sglang:v0.5.14-cu130.
❌ Check 3 (recipe linked & merged): FAIL — no upstream recipe/cookbook link in the sign-off's additional detail section for this single-node recipe change; the TP2/EP2 parallelism arm is a MAJOR (deployment-defining) config not shown in any linked merged upstream documentation.
✅ Check 4 (reuse command): PASS — /reuse-sweep-run 31432304739 posted by Ankur-singh (COLLABORATOR).
✅ Check 5 (latest checklist): PASS — all current-template items present; both unchecked items are explained in the additional detail section.
✅ Check 6 (upstream image / engine-first): PASS — framework: sglang with upstream lmsysorg/sglang:v0.5.14-cu130 on B200; no non-vLLM/SGLang entry added.
✅ Check 7 (deprecated models): PASS — Qwen3.5 Single-turn 8k1k fp4 is active per MODELS.md as of 2026-08-11.
✅ Check 8 (architecture hacks): PASS — diff is two YAML files; EP2 redistributes the same MoE FLOPs, no overrides.
✅ Check 9 (spec-decode chat template): PASS — qwen3.5_fp4_b200_mtp.sh benchmarks with --use-chat-template.
✅ Check 10 (engine patches): PASS — no patching of the serving stack; pinned image runs as shipped.
➖ Check 11 (agentic golden AL): N/A — fixed-seq-len submission, no agentic spec-decode changes; no synthetic-acceptance knobs present on this non-agentic config.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31625807465
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=31625807465

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31628689774
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=31628689774

[Ankur-singh]

/reuse-sweep-run 31628689774

[Ankur-singh]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/31628689774
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/31628689774
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.

Additional detail section:

Scope of the change. One search-space row added to the existing qwen3.5-fp4-b200-sglang-mtp entry — { tp: 2, ep: 2, conc-list: [16, 32, 64], spec-decoding: mtp } — plus the matching perf-changelog.yaml entry. No benchmark script, recipe file, image pin, or launcher is touched.

Validation and eval evidence. Run 31628689774 executed on the exact current PR head 21d6bbbc over roughly four hours of real GPU time. All nine single-node 8k1k points are green and non-skipped — the three new TP2/EP2 points at concurrency 16, 32, and 64, alongside the pre-existing TP2 c4/c8/c16/c32/c64 and TP4 c4 rows — with four non-skipped eval-only jobs and a successful collect-evals. The skipped lanes (1k1k, multi-node, agentic, agentic-eval) have no matrix entries for this single-node fixed-seq-len config.

Checkpoint provenance — why the earlier run was not used. An earlier green sweep on this PR (run 31432304739) measured nvidia/Qwen3.5-397B-A17B-NVFP4. After that run, #2205 merged to main and switched this config key to nvidia/Qwen3.5-397B-A17B-NVFP4-V2, so that evidence no longer described the config that would merge. It is deliberately not linked above. Run 31628689774 re-measured the full search space on the V2 checkpoint at the current head — every job name carries nvidia/Qwen3.5-397B-A17B-NVFP4-V2 — and the changelog entry no longer carries the superseded V1-derived numbers.

Speculative decoding and chat template. benchmarks/single_node/fixed_seq_len/qwen3.5_fp4_b200_mtp.sh passes --use-chat-template, so the MTP acceptance-length distribution is measured on chat-formatted prompts. The script runs EAGLE with --speculative-num-steps 3, --speculative-eagle-topk 1, --speculative-num-draft-tokens 4, unchanged by this PR. The golden-AL item is not applicable: this is a fixed-seq-len 8k/1k submission, not an agentic workload, so it uses measured acceptance rather than the simulated synthetic acceptance that item governs.

The new arm is a real configuration, not a relabel. The script consumes expert parallelism directly — --tensor-parallel-size=$TP --data-parallel-size=1 --expert-parallel-size=$EP_SIZE — so ep: 2 genuinely changes the served topology relative to the ep: 1 rows, and the sweep surfaced the points as distinct TP2/EP2 jobs. Expert parallelism redistributes the same MoE computation across GPUs; it removes no model-architecture FLOPs, and the promoted arm carries its own passing evals in the linked run.

Model and scenario scope. MODELS.md keeps fp8 and fp4 recipes for Qwen3.5-397B-A17B active and deprecates only the bf16 recipes; the speculative-decoding A/B retirements listed there concern agentic coding, not Single-turn 8k1k. This submission is 8k1k NVFP4 and adds no deprecated lane.

Single-node recipe publication. Documented upstream in the SGLang cookbook via sgl-project/sglang#34357, MERGED 2026-08-11. It records B200 NVFP4 as tp=4 (tp=2 with expert parallelism 2 when MTP is enabled) in docs/cookbook/autoregressive/Qwen/Qwen3.5.mdx, and the deployment snippet sets { tp: 2, ep: 2 } for 397B / B200 / fp4 with speculative decoding enabled — the exact arm this PR adds.

No engine or serving-stack patching. The diff is two YAML files. No .patch, git apply, sed, site-packages edit, monkey-patch, container overwrite, or forked wheel, and the pinned upstream image lmsysorg/sglang:v0.5.14-cu130 runs as shipped.

Merge state. This sign-off attests to review and validation only. The PR currently conflicts with main and must be brought up to date before merge; that is a separate merge-time step and does not affect the evidence above.

Signed: Ankur-singh

[Klaud-Cold]

✅✅✅ Verdict: PASS ✅✅✅

✅ Check 0 (CODEOWNER): PASS — signer Ankur-singh is a named owner of configs/nvidia-master.yaml; perf-changelog.yaml falls to the catch-all, satisfied by a recognized CODEOWNER.
✅ Check 1 (sweep on in-PR commit): PASS — head 21d6bbbc carries green, non-skipped single-node 8k1k / (all 9 points, incl. the three new TP2/EP2 ones) and 4 executed eval / jobs from run 31628689774.
✅ Check 2 (eval accuracy): PASS — GSM8K strict-match 0.966–0.970 (flexible 0.955–0.961) across TP2/EP1 and TP2/EP2 at c32/c64, all on Qwen3.5-397B-A17B-NVFP4-V2 with this PR's image lmsysorg/sglang:v0.5.14-cu130.
✅ Check 3 (recipe merged + args match): PASS — sgl-project/sglang#34357 MERGED 2026-08-11; the cookbook pins the exact new arm (B200 NVFP4-V2, tp=2 + ep=2 with MTP, mem 0.8, spec-decode steps 3 / topk 1 / draft 4, trtllm_mha attention). Quantization and kv-cache dtype are auto-detected from the FP4 checkpoint; remaining script flags are InferenceX sweep tuning (informational only).
✅ Check 4 (reuse command): PASS — authorized COLLABORATOR Ankur-singh posted /reuse-sweep-run 31628689774, pinning the passing run.
✅ Check 5 (latest checklist): PASS — all current-template items present and checked; the one unchecked item (agentic golden AL) is explained as N/A in the additional detail section.
✅ Check 6 (upstream image / engine-first): PASS — lmsysorg/sglang:v0.5.14-cu130 from the upstream lmsysorg org on established B200; no non-vLLM/SGLang entry added.
✅ Check 7 (deprecations): PASS — MODELS.md keeps qwen3.5 Single-turn 8k1k fp8/fp4 active; only bf16 recipes are deprecated.
✅ Check 8 (no architecture hacks): PASS — diff is two YAML rows; ep=2 redistributes the same MoE FLOPs, no --hf-overrides or FLOPs reduction.
✅ Check 9 (spec-decode chat template): PASS — qwen3.5_fp4_b200_mtp.sh benchmarks with --use-chat-template.
✅ Check 10 (no engine patches): PASS — no patch/heredoc/site-packages edits; pinned upstream image runs as shipped.
➖ Check 11 (agentic golden AL): N/A — fixed-seq-len 8k1k submission, no agentic spec-decode change; no synthetic-acceptance knobs added to this non-agentic config.

[Ankur-singh]

/reuse-sweep-run

[Klaud-Cold]

✅✅✅ Verdict: PASS ✅✅✅

✅ Check 0 (CODEOWNER): PASS — Ankur-singh is a named owner of configs/nvidia-master.yaml; perf-changelog.yaml falls to the catch-all, satisfied by a recognized CODEOWNER.
✅ Check 1 (sweep on in-PR commit): PASS — in-PR commit 21d6bbbc has 9 green non-skipped single-node 8k1k / jobs (incl. the new TP2/EP2 c16/c32/c64) and 4 green eval / jobs from run 31628689774; head f705dd1f is a conflict-resolving merge of main validated via the reuse gate.
✅ Check 2 (evals pass): PASS — downloaded eval artifacts show GSM8K strict-match 0.966–0.970 (bar 0.94) across ep1/ep2 × c32/c64 on Qwen3.5-397B-A17B-NVFP4-V2 with the PR's image lmsysorg/sglang:v0.5.14-cu130.
✅ Check 3 (recipe merged + matches): PASS — sgl-project/sglang#34357 MERGED 2026-08-11; the cookbook's B200 NVFP4 MTP command emits exactly this arm (tp=2, ep=2) and matches the major args: model NVFP4-V2, --quantization modelopt_fp4, --kv-cache-dtype fp8_e4m3, --attention-backend trtllm_mha, --moe-runner-backend flashinfer_trtllm, draft config 3/1/4 (recipe's NEXTN is SGLang's alias for the EAGLE-style native MTP head). Remaining diffs (--enable-symm-mem, --mamba-ssm-dtype, chunked-prefill/stream-interval/concurrency knobs) are InferenceX sweep tuning — informational only.
✅ Check 4 (reuse command): PASS — authorized COLLABORATOR Ankur-singh posted /reuse-sweep-run 31628689774.
✅ Check 5 (latest checklist): PASS — all current-template items present and checked; the one unchecked item (agentic golden-AL) is explained as N/A in the detail section (fixed-seq-len, not agentic).
✅ Check 6 (upstream image / engine-first): PASS — framework: sglang on established B200 with upstream lmsysorg/sglang:v0.5.14-cu130; no non-vLLM/SGLang framework entry added.
✅ Check 7 (no deprecated models/scenarios): PASS — MODELS.md keeps Qwen3.5 Single-turn 8k1k fp4 active (only bf16 and 1k1k/1k8k are retired).
✅ Check 8 (no architecture hacks): PASS — diff adds one tp2/ep2 search-space row + changelog entry; EP redistributes the same MoE FLOPs, no --hf-overrides or trimming.
✅ Check 9 (spec-decode via chat template): PASS — qwen3.5_fp4_b200_mtp.sh benchmarks through --use-chat-template.
✅ Check 10 (no engine patches): PASS — two-YAML diff, no patching of the pinned image.
➖ Check 11 (agentic golden AL): N/A — no agentic spec-decode changes; this non-agentic config pins no simulated-acceptance knobs.