[Klaud Cold] qwen3.5-fp4-mi355x-sglang-agentic-mtp: add HiCache DRAM offload arms / 为 MI355X Qwen3.5 FP4 SGLang AgentX MTP 新增 HiCache DRAM 卸载分支 - #2582
Conversation
… arms Mirror every resident TP2/EP2 and TP4 EAGLE MTP concurrency point with a HiCache DRAM KV-offload arm, so each measured resident point has a host-tier counterpart on MI355X. The HiCache arms run at page size 64 with the kernel io-backend and page_first layout -- the transfer path Qwen3.5's hybrid attention/Mamba host pools already build on gfx950 -- sized to 80% of the workflow DRAM budget under a 180 GB per-rank ceiling. Resident arms keep page size 16 and are untouched. They also patch sgl-project/sglang#34560 into the container. Qwen3.5 keeps its language-model attributes in the nested text_config and SGLang normalizes the MTP draft depth only on the parent HF config, so ModelConfig.num_nextn_predict_layers stays None, the packed/sidecar routing added in sgl-project/sglang#30393 misroutes the draft cache to the sidecar path, and the scheduler aborts with "AttributeError: 'HybridLinearKVPool' object has no attribute 'layer_num'". The patch is scoped to the HiCache arms and is a no-op once the fix ships in an MI355X image. 为 MI355X Qwen3.5 MXFP4 AgentX EAGLE MTP 配方新增 HiCache DRAM KV 卸载分支， 与现有 TP2/EP2 与 TP4 常驻并发点一一对应，使每个已测常驻点都有对应的主机层数据点。 HiCache 分支使用 page size 64、kernel io-backend 与 page_first 布局，即 Qwen3.5 混合注意力/Mamba 主机池在 gfx950 上已经能够构建的传输路径；容量按工作流 DRAM 预算的 80% 计算，并设置每 rank 180 GB 上限。常驻分支仍为 page size 16， 保持不变。 该分支同时在容器内应用 sgl-project/sglang#34560 的修复。Qwen3.5 将语言模型属性 存放在嵌套的 text_config 中，而 SGLang 仅在父 HF config 上归一化 MTP draft 深度，导致 ModelConfig.num_nextn_predict_layers 保持为 None； sgl-project/sglang#30393 引入的 packed/sidecar 路由因此将 draft cache 误判到 sidecar 路径，调度器在启动时抛出 "AttributeError: 'HybridLinearKVPool' object has no attribute 'layer_num'"。 该补丁仅作用于 HiCache 分支，待修复进入 MI355X 镜像后即为空操作。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
2 similar comments
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
将 Qwen3.5 MI355X HiCache 变更日志条目关联至 PR #2582。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| HICACHE_ALIGNMENT_RESERVE_GB=$TP | ||
| HICACHE_USABLE_TOTAL_GB=$((TOTAL_CPU_DRAM_GB - HICACHE_ALIGNMENT_RESERVE_GB)) | ||
| if [ "$HICACHE_USABLE_TOTAL_GB" -lt 1 ]; then | ||
| echo "Error: insufficient DRAM after HiCache alignment reserve." >&2 | ||
| exit 1 | ||
| fi | ||
| MAX_HICACHE_SIZE_GB=$((HICACHE_USABLE_TOTAL_GB * 80 / 100 * 60 / 61 / TP)) | ||
| # 144 GB/rank is the largest pool observed to allocate on this image; 180 | ||
| # is a bounded step up from it. Raise once a run confirms the larger pinned | ||
| # allocation stays inside the watchdog. | ||
| HICACHE_MAX_SIZE_GB_PER_RANK=${HICACHE_MAX_SIZE_GB_PER_RANK:-180} | ||
| if [ "$MAX_HICACHE_SIZE_GB" -gt "$HICACHE_MAX_SIZE_GB_PER_RANK" ]; then | ||
| MAX_HICACHE_SIZE_GB="$HICACHE_MAX_SIZE_GB_PER_RANK" | ||
| fi | ||
| HICACHE_SIZE_GB="${HICACHE_SIZE_GB:-$MAX_HICACHE_SIZE_GB}" | ||
| if [ "$HICACHE_SIZE_GB" -lt 1 ] || [ "$HICACHE_SIZE_GB" -gt "$MAX_HICACHE_SIZE_GB" ]; then | ||
| echo "Error: HICACHE_SIZE_GB=$HICACHE_SIZE_GB outside 1..$MAX_HICACHE_SIZE_GB." >&2 | ||
| exit 1 | ||
| fi | ||
| PROJECTED_HICACHE_TOTAL_GB=$(((HICACHE_SIZE_GB * TP * 61 + 59) / 60 + HICACHE_ALIGNMENT_RESERVE_GB)) | ||
| if [ "$PROJECTED_HICACHE_TOTAL_GB" -gt "$TOTAL_CPU_DRAM_GB" ]; then | ||
| echo "Error: projected HiCache use ${PROJECTED_HICACHE_TOTAL_GB} GB exceeds configured ${TOTAL_CPU_DRAM_GB} GB." >&2 | ||
| exit 1 | ||
| fi | ||
| echo "HiCache pools: ${HICACHE_SIZE_GB} GB per rank across TP=${TP}; projected node total ${PROJECTED_HICACHE_TOTAL_GB} GB of ${TOTAL_CPU_DRAM_GB} GB." |
There was a problem hiding this comment.
🟡 The HiCache reserve→scale→project sizing block (lines 112-136) is copy-pasted near-verbatim across five sibling recipes (qwen3.5 fp4/fp8 × b200/b300/mi355x mtp), differing only in the per-model layer-fraction ratio (60/61 here vs 15/31 on B300) plus this recipe's extra budget cap. Consider extracting the shared arithmetic into benchmark_lib.sh, parameterized by the layer-count fraction and reserve, so future rounding/reserve fixes and the magic ratios have one place to live instead of five.
Extended reasoning...
The HICACHE_ALIGNMENT_RESERVE_GB / HICACHE_USABLE_TOTAL_GB / MAX_HICACHE_SIZE_GB / PROJECTED_HICACHE_TOTAL_GB block in this recipe (lines 112-136) reproduces, line for line, the same reserve-then-scale-then-project skeleton found in qwen3.5_fp8_b300_sglang_mtp.sh:53-70, and — as a grep for HICACHE_ALIGNMENT_RESERVE_GB confirms — in qwen3.5_fp4_b300_sglang_mtp.sh, qwen3.5_fp8_b200_sglang_mtp.sh, and qwen3.5_fp4_b200_sglang_mtp.sh as well. That is five independent copies of the same variable names, the same DRAM-alignment-reserve subtraction, the same insufficient-DRAM guard, the same ${HICACHE_SIZE_GB:-$MAX_HICACHE_SIZE_GB} default-with-bounds-check pattern, and the same ceiling-division projection guard.
The only genuine per-recipe deltas are the two magic layer-fraction ratios (60/61 here vs 15/31 on B300, reflecting each model's target-layer count) and this recipe's two added clauses (the 80% budget cap and the HICACHE_MAX_SIZE_GB_PER_RANK ceiling). Everything else — the arithmetic shape, the guard conditions, the echo/error message structure — is identical scaffolding repeated five times. benchmark_lib.sh is already the established home for shared recipe plumbing (require_agentic_kv_offload_backend, check_env_vars, wait_for_server_ready), but it currently has no equivalent helper for this sizing math, so a future correction to the rounding or reserve model has to be hunted down and reapplied in five separate files, and the two ratios have no single documented derivation point to check for drift.
One verifier pushed back that extracting a helper would make things worse, arguing the per-model ratios are supposed to differ (they encode different layer counts) and that inline sizing math is deliberate for auditability in these self-contained recipe scripts. That is a fair engineering consideration — a helper would need to take the layer-fraction, reserve, and the two optional clauses as parameters, so it would not eliminate all of the per-recipe detail. But parameterizing by "layer-count fraction + reserve" is exactly what would let a maintainer see, in one place, that fp4_mi355x uses 60/61 while b300 uses 15/31, rather than needing to diff five files to notice a ratio drifted or a rounding fix was applied to some copies and not others. Centralizing the skeleton does not have to sacrifice per-recipe transparency: each script would still pass its own ratio and reserve inline, visible at the call site.
To reproduce the duplication: open benchmarks/single_node/agentic/qwen3.5_fp8_b300_sglang_mtp.sh:53-70 next to this PR's qwen3.5_fp4_mi355x_sglang_mtp.sh:112-136 — the variable names, guard structure, and arithmetic shape line up almost exactly, with only 60/61/61+59/60 swapped for 15/31/31+14/15 and the two added clauses in the newer file. grep -l HICACHE_ALIGNMENT_RESERVE_GB benchmarks/single_node/agentic/*.sh returns all five files, confirming this is a growing pattern rather than an isolated coincidence.
This is a code-quality/reuse observation, not a correctness bug — nothing here produces wrong output, and the PR's own arithmetic is internally consistent for this model. Recommend extracting the reserve/scale/project skeleton into a benchmark_lib.sh helper parameterized by per-rank layer-count fraction and reserve, so a future fix or ratio-drift check only needs to touch one place.
Comment out the merged resident arms and the remaining HiCache concurrency points, leaving TP4/EP1 conc 16 with kv-offloading dram + hicache. This proves whether the sgl-project/sglang#34560 container patch clears the HybridLinearKVPool draft-sidecar crash on gfx950 without spending MI355X node time on already-measured resident points. 注释掉已合入的常驻分支与其余 HiCache 并发点，仅保留 TP4/EP1 并发 16 的 kv-offloading dram + hicache 配置，用于验证 sgl-project/sglang#34560 容器补丁 能否在 gfx950 上消除 HybridLinearKVPool draft sidecar 崩溃，同时避免在已测的 常驻点上占用 MI355X 节点时间。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31624871677 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31625689397 |
1 participant
Summary / 概述
Adds HiCache DRAM KV-offload arms to
qwen3.5-fp4-mi355x-sglang-agentic-mtp, mirroring every resident concurrency point merged in #2562 so each measured resident point has a host-tier counterpart. The resident arms are unchanged.为
qwen3.5-fp4-mi355x-sglang-agentic-mtp新增 HiCache DRAM KV 卸载分支，与 #2562 中合入的每一个常驻并发点一一对应，使每个已测常驻点都有对应的主机层数据点。常驻分支保持不变。Generated matrix: 32 configs (6 + 6 + 10 + 10). / 生成的矩阵共 32 个配置（6 + 6 + 10 + 10）。
Why the HiCache arms were dropped from #2562 / #2562 中删除 HiCache 分支的原因
Both HiCache arms attempted on
v0.5.17-rocm720-mi35x-20260811(run 31579113764) died in scheduler init:在
v0.5.17-rocm720-mi35x-20260811上尝试的两个 HiCache 分支均在调度器初始化阶段失败：The hybrid stack itself is healthy on gfx950 — the same logs show
93.37 GBtarget KV and50.65 GBMamba host pools allocating per rank underpage_first, thenAttached hybrid pool stack to UnifiedRadixCache: pools=KV + MAMBA, transfer_layer_num=60. Only the MTP draft pool crashes.混合池本身在 gfx950 上是正常的：同一份日志显示每个 rank 在
page_first布局下成功分配了93.37 GB目标 KV 与50.65 GBMamba 主机池，并完成Attached hybrid pool stack to UnifiedRadixCache。仅 MTP draft 池崩溃。Root cause (confirmed by upstream sgl-project/sglang#34560): Qwen3.5 conditional-generation checkpoints keep language-model attributes in the nested
text_config. SGLang normalizes the MTP draft depth on the parent HF config only, soModelConfig.num_nextn_predict_layers(derived fromhf_text_config) staysNone. The packed-vs-sidecar draft routing added in sgl-project/sglang#30393 therefore misclassifies the draft cache as a sidecar and hitspool.layer_numon a pool that has no such attribute.根因（已由上游 sgl-project/sglang#34560 确认）：Qwen3.5 将语言模型属性存放在嵌套的
text_config中，而 SGLang 仅在父 HF config 上归一化 MTP draft 深度，导致由hf_text_config推导的ModelConfig.num_nextn_predict_layers保持为None。sgl-project/sglang#30393 引入的 packed/sidecar 路由因此将 draft cache 误判为 sidecar，并在无该属性的池上访问pool.layer_num。No image can avoid this. #30393 merged 2026-08-06; the GDN chunked-extend padding fix this recipe requires for TP2/EP2 MTP (sgl-project/sglang#33810) merged 2026-08-07. There is no
sglang-rocmmi35x build with the GDN fix and without the HiCache regression, and #34560 is still open upstream.无法通过换镜像规避。 #30393 于 2026-08-06 合入；本配方 TP2/EP2 MTP 所需的 GDN chunked-extend padding 修复（#33810）于 2026-08-07 合入。不存在既包含 GDN 修复又不含该 HiCache 回归的
sglang-rocmmi35x 构建，且 #34560 在上游仍未合入。Changes / 变更内容
configs/amd-master.yaml— two HiCache rows mirroring the resident conc-lists. Image unchanged (v0.5.17-rocm720-mi35x-20260811). / 新增两行 HiCache 配置，镜像不变。benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang_mtp.shmodel_config.py, gated to the HiCache arms so the merged resident data stays byte-identical. The patcher is idempotent, derives its own indentation, and exits non-zero if the anchor moves so an image bump cannot silently skip it. Same precedent as theqwen3.5_fp8_h200_mtp.sh/qwen3.5_fp8_h100_mtp.shin-containersed. / 仅在 HiCache 分支内对容器中的model_config.py应用 #34560 的单行修复；幂等、自动推导缩进，锚点失配时以非零码退出。--page-size 64under HiCache (resident stays16),--hicache-io-backend kernel,--hicache-mem-layout page_first,--hicache-write-policy write_through_selective— matching the validated B300 Qwen3.5 MTP sibling and the transfer path the gfx950 logs already exercise. / HiCache 分支使用 page size 64 等参数，与已验证的 B300 Qwen3.5 MTP 同类配方一致。--hicache-sizeis the per-rank total SGLang splits across the two hybrid pools (observed144→93.37 + 50.65 = 144.02GB/rank), not a per-pool figure as the earlier [AMD] [AgentX] Add MI355X Qwen3.5 FP4 SGLang MTP #2562 draft assumed. Budget is 80% of workflow DRAM, capped at 180 GB/rank, both overridable. / 依据实测修正容量计算：--hicache-size是 SGLang 在两个混合池之间切分的每 rank 总量。perf-changelog.yaml— appended entry. / 追加条目。Validation / 验证
bash -non the recipe. / 配方通过bash -n。model_config.py: produces exactly #34560's one-line diff, is idempotent on re-run, and exits 1 when the anchor is absent. / 补丁脚本已针对真实上游model_config.py验证：产生与 #34560 完全一致的单行改动、可重复执行、锚点缺失时退出码为 1。utils/validate_perf_changelog.py --base-ref origin/main --head-ref HEAD→ passed. / 通过。utils/matrix_logic/test_generate_sweep_configs.py→ 105 passed. / 105 项通过。TOTAL_CPU_DRAM_GB1199 (TP4) / 599 (TP2). / 矩阵生成结果符合预期。The HiCache arms themselves are unproven on hardware — no MI355X run has gotten past scheduler init with HiCache + MTP on this model. The full sweep is the validation. / HiCache 分支尚未经过硬件验证，以完整 sweep 作为验证手段。
🤖 Generated with Claude Code