From 18654c659a67d9a6825026f628397c6726b08933 Mon Sep 17 00:00:00 2001 From: Jiahui Cao Date: Thu, 13 Aug 2026 05:00:36 +0000 Subject: [PATCH 1/4] [AMD][AgentX] dsv4 vllm-agentic MTP: pin the 08-09 nightly + additive container patches MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The dsv4-fp4-mi355x-vllm-agentic-mtp config cannot start on its pinned image: the server memory-faults during the profile run, before it ever serves a request. Repin to the 08-09 nightly (f8d03e77) and patch it before serving. The fault is in the base, not in anything this recipe does. It reproduces identically on a PRISTINE 08-12 nightly (3ee2df30) with no patches applied at all: the eight TP workers serialize on /tmp/aiter_configs/bf16_tuned_gemm.csv .lock, and the worker holding the baton dies in the post-MoE bf16 GEMM while the other seven survive parked on the lock. Two separate standalone repros clear the loudest suspects -- the inter_dim=384 MoE returns finite results at the exact faulting shape with preshuffled weights, and the bf16 GEMM passes on all four paths at M=65536 and M=16384 x N=7168 x K=7168. apply_dsv4_container_patches.sh is strictly additive and idempotent: vllm #51473 native MXFP4 TP8 shard allocation. Merged 2026-08-11, i.e. AFTER this base, so unlike the 08-12 nightly it has to be applied here. vllm #51714 opt-in AITER gluon sparse-MLA decode for gfx950. vllm #51918 vllm/config/kernel.py ONLY, registering the flydsl_mega_moe backend name. The model-side hunks are deliberately not taken: the backend needs mori.ir.flydsl, which this base does not have, so a DEP8 row is now rejected at config time with a clear error instead of failing deep inside a kernel. aiter #4417 large-token FlyDSL MoE launch/output limits. The upstream diff does not apply (the base predates aiter's typing modernization, so every context line still reads Dict[str, Dict]), so the hunks are grafted by anchor. Nothing that ships in the base is overwritten. Three of the four pa_decode_sparse variants already ship and are left alone -- only the gfx950 one is genuinely missing. An earlier revision copied all four out of aiter@97d0c6e4 and silently replaced base files with a cross-version transplant; the script now skips anything already present, and re-running it is a verified no-op. 中文：dsv4-fp4-mi355x-vllm-agentic-mtp 在其固定镜像上根本起不来 —— 服务在 profile run 阶段发生显存访问越界，还没开始服务请求就崩了。本 PR 将镜像改钉到 08-09 的 nightly（f8d03e77），并在启动前打补丁。 该故障出在基座本身，与本 recipe 无关：在完全未打任何补丁的 08-12 原始 nightly （3ee2df30）上可以复现出完全相同的签名 —— 8 个 TP worker 在 /tmp/aiter_configs/bf16_tuned_gemm.csv.lock 上串行排队，持有 baton 的那个 worker 在 MoE 之后的 bf16 GEMM 中死亡，其余 7 个因阻塞在锁上而幸存。两个独立的最小复现 排除了最显眼的两个嫌疑对象：inter_dim=384 的 MoE 在完全相同的形状和预 shuffle 权重 下返回有限值；bf16 GEMM 在 M=65536 与 M=16384 × N=7168 × K=7168 下四条路径全部通过。 apply_dsv4_container_patches.sh 严格只做增量，且幂等。基座自带的文件一律不覆盖： 四个 pa_decode_sparse 变体中有三个基座已自带，脚本予以保留，只补真正缺失的 gfx950 那一个。早前的版本曾把四个文件全部从 aiter@97d0c6e4 拷入，等于用跨版本的文件静默 替换了基座文件；现在脚本会跳过任何已存在的文件，重复执行已验证为 no-op。 Co-Authored-By: Claude Opus 5 --- .../agentic/apply_dsv4_container_patches.sh | 165 ++++++++++++++++++ .../agentic/dsv4_fp4_mi355x_vllm_mtp.sh | 11 ++ .../single_node/agentic/graft_aiter_4417.py | 125 +++++++++++++ configs/amd-master.yaml | 14 +- perf-changelog.yaml | 12 ++ 5 files changed, 326 insertions(+), 1 deletion(-) create mode 100755 benchmarks/single_node/agentic/apply_dsv4_container_patches.sh create mode 100644 benchmarks/single_node/agentic/graft_aiter_4417.py diff --git a/benchmarks/single_node/agentic/apply_dsv4_container_patches.sh b/benchmarks/single_node/agentic/apply_dsv4_container_patches.sh new file mode 100755 index 000000000..85a25663d --- /dev/null +++ b/benchmarks/single_node/agentic/apply_dsv4_container_patches.sh @@ -0,0 +1,165 @@ +#!/usr/bin/env bash +# ============================================================================= +# ADDITIVE patch stack for the 08-09 nightly base +# vllm/vllm-openai-rocm:nightly-f8d03e77416bf90c49acbe50e233275722f02c4b +# (vllm 0.26.1rc1.dev528+gf8d03e774) +# +# STRICTLY ADDITIVE. Nothing that ships in the base is overwritten: +# * only the ONE gluon kernel file the base genuinely lacks is copied in +# (gfx950/attention/pa_decode_sparse.py). The other three pa_decode_sparse +# variants SHIP IN THE BASE and are deliberately NOT touched -- an earlier +# revision of this script copied all four out of aiter@97d0c6e4, silently +# replacing three base files with a cross-version transplant. Do not +# reintroduce that. +# * aiter core (jit/core.py, fused_moe.py) is not modified. The base already +# exposes _set_current_hip_stream, which the nightly vllm calls for +# module_rmsnorm_quant; the old wholesale post2-python overlay regressed it +# and crashed rmsnorm_quant warmup. +# * MegaMoE/DEP8 is out of scope on this route: its intranode kernel needs +# mori.ir.flydsl, which is absent from the base (verified: ModuleNotFoundError). +# FSE is likewise out (aiter #4269 needs aiter/fhmoe.py, also absent). +# +# Carried: +# vllm #51473 native MXFP4 TP8 shard allocation (MERGED 2026-08-11 -- i.e. +# AFTER this 08-09 base, so unlike the 08-12 base it must be +# applied here). One hunk on +# vllm/model_executor/layers/fused_moe/oracle/mxfp4.py. +# vllm #51714 opt-in AITER gluon kernel for sparse-MLA decode on gfx950 +# (open). Dormant unless VLLM_ROCM_DSV4_SPARSE_GLUON=1. +# vllm #51918 FlyDSL fused mega-MoE backend (open). Dormant unless +# --moe-backend flydsl_mega_moe is selected. +# aiter #4417 large-token FlyDSL MoE launch/output limits (merged +# 2026-07-30; the base's VENDORED aiter predates it -- the +# vendored revision is pinned and does not track aiter main). +# +# Run INSIDE a fresh container of the pinned image: +# docker exec -i bash /path/to/dsv4_patch_0809.sh +# Idempotent: every step is marker-gated, re-running is a no-op. +# ============================================================================= +set -uo pipefail +AITER_SHA="97d0c6e4cb7a0919c12291c7c7d560ad412f15c1" +AITER_REPO="https://github.com/ROCm/aiter" +VLLM_REPO="https://github.com/vllm-project/vllm" +ROOT="$(python -c 'import importlib.util as u, os; print(os.path.dirname(os.path.dirname(u.find_spec("vllm").origin)))')" +[ -d "$ROOT/vllm" ] && [ -d "$ROOT/aiter" ] || { echo "ERROR ROOT=$ROOT"; exit 1; } +echo "[add] ROOT=$ROOT" +echo "[add] vllm = $(python -c 'import vllm;print(vllm.__version__)' 2>/dev/null)" +WS=/tmp/dsv4_add; mkdir -p "$WS" + +# --- 1/4 gluon kernel: ONLY the file the base lacks --------------------------- +# Verified against this base: gfx1250/, _triton_kernels/ and the ops/triton/ +# facade all ship. Copying the aiter@97d0c6e4 versions over them is the +# cross-version transplant that has to be avoided, so the loop SKIPS anything +# already present rather than overwriting it. +GLUON_PATHS=( + aiter/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py + aiter/ops/triton/_gluon_kernels/gfx1250/attention/pa_decode_sparse.py + aiter/ops/triton/_triton_kernels/attention/pa_decode_sparse.py + aiter/ops/triton/attention/pa_decode_sparse.py +) +NEED=() +for p in "${GLUON_PATHS[@]}"; do + if [ -f "$ROOT/$p" ]; then echo " keep base $p"; else NEED+=("$p"); fi +done +if [ ${#NEED[@]} -gt 0 ]; then + SRC="$WS/aiter_src" + if [ ! -d "$SRC/.git" ]; then git clone --filter=blob:none --no-checkout "$AITER_REPO" "$SRC" 2>&1 | tail -1; fi + ( cd "$SRC" && git fetch --depth 1 origin "$AITER_SHA" 2>&1 | tail -1 && git checkout -q "$AITER_SHA" -- "${NEED[@]}" ) + for p in "${NEED[@]}"; do + [ -e "$SRC/$p" ] || { echo " MISSING in src: $p"; continue; } + mkdir -p "$(dirname "$ROOT/$p")"; cp -a "$SRC/$p" "$ROOT/$p"; echo " added $p" + done +else + echo " (nothing to add)" +fi + +# --- 2/4 aiter #4417: large-token FlyDSL MoE launch/output limits -------------- +# Two guards: stage2 buffer atomics address the output with 32-bit byte offsets +# (>4 GiB walks off the end), and HIP caps grid.y at 65535. Neither fires at the +# DSv4-Pro TP8 shape measured here -- requires_flydsl_stage2_reduce(65536, 7168, +# 2) is False (~939 MB) -- so this is NOT a fix for the profile-run memfault; it +# is carried because it is a real gap any larger-token sweep row would hit. +# The upstream .diff will NOT apply: the base's aiter predates aiter's typing +# modernization, so every context line still reads Dict[str, Dict] where the +# diff expects dict[str, dict]. The hunks are grafted by anchor instead. +python "$(dirname "$0")/graft_aiter_4417.py" "$ROOT/aiter/ops/flydsl/moe_kernels.py" || \ + { echo " #4417: GRAFT FAILED"; exit 1; } + +# --- 3/4 vllm PRs ------------------------------------------------------------- +apply_pr(){ local pr="$1" mf="$ROOT/$2" mk="$3" d="$WS/vllm_$1.diff" + if grep -qF "$mk" "$mf" 2>/dev/null; then echo " #$pr: already present"; return 0; fi + curl -ksSL -o "$d" "$VLLM_REPO/pull/$pr.diff" || { echo " #$pr: FETCH FAIL"; return 1; } + if ( cd "$ROOT" && git apply -p1 --3way "$d" ) 2>/dev/null || ( cd "$ROOT" && git apply -p1 "$d" ) 2>/dev/null + then echo " #$pr: APPLIED"; else echo " #$pr: FAILED"; return 1; fi +} +# #51473 carries a tests/ hunk that has no counterpart in an installed wheel, so +# apply only the runtime file rather than the whole PR diff. +apply_51473(){ + local mf="$ROOT/vllm/model_executor/layers/fused_moe/oracle/mxfp4.py" + local mk="AITER_MXFP4_BF16 and activation == MoEActivation.SILU" + if grep -qF "$mk" "$mf" 2>/dev/null; then echo " #51473: already present"; return 0; fi + curl -ksSL -o "$WS/vllm_51473.diff" "$VLLM_REPO/pull/51473.diff" || { echo " #51473: FETCH FAIL"; return 1; } + ( cd "$ROOT" && git apply -p1 --include='vllm/*' --3way "$WS/vllm_51473.diff" ) 2>/dev/null \ + || ( cd "$ROOT" && git apply -p1 --include='vllm/*' "$WS/vllm_51473.diff" ) 2>/dev/null \ + && echo " #51473: APPLIED" || { echo " #51473: FAILED"; return 1; } +} +apply_51473 || true +apply_pr 51714 "vllm/v1/attention/ops/rocm_aiter_mla_sparse.py" "_DSV4_SPARSE_GLUON" || true + +# #51918 is taken PARTIALLY, on purpose: only vllm/config/kernel.py, which +# registers "flydsl_mega_moe" as an accepted --moe-backend value. The model-side +# hunks are NOT applied. +# +# Two independent reasons, either of which is sufficient: +# 1. The backend cannot run on this base at all. MegaMoE's intranode kernel +# imports mori.ir.flydsl, and this image has no mori.ir.flydsl (verified: +# ModuleNotFoundError). Applying the model hunks would buy a backend that +# raises on first use. +# 2. The model.py hunks do not apply cleanly here. #51918 is written against a +# tree ~3 days newer than this 08-09 base; the two earlier hunks shift the +# line numbering enough that the third fails at model.py:300. `git apply +# --3way` cannot rescue it because site-packages is not a git repo, so +# there are no blobs to 3-way against. Force-grafting a DEP8 code path that +# cannot execute anyway is not worth the transplant risk -- that is exactly +# how the earlier pa_decode_sparse damage happened. +# +# Net effect: the TP8 arm is unaffected (it never selects this backend), and a +# DEP8 row would be rejected at config time with a clear error instead of +# failing deep inside a kernel. DEP8/MegaMoE stays out of scope on this route. +apply_pr_files(){ local pr="$1" inc="$2" mf="$ROOT/$3" mk="$4" d="$WS/vllm_$1.diff" + if grep -qF "$mk" "$mf" 2>/dev/null; then echo " #$pr ($inc): already present"; return 0; fi + curl -ksSL -o "$d" "$VLLM_REPO/pull/$pr.diff" || { echo " #$pr: FETCH FAIL"; return 1; } + ( cd "$ROOT" && git apply -p1 --include="$inc" "$d" ) 2>/dev/null \ + && echo " #$pr ($inc): APPLIED" || { echo " #$pr ($inc): FAILED"; return 1; } +} +apply_pr_files 51918 "vllm/config/kernel.py" "vllm/config/kernel.py" "flydsl_mega_moe" || true + +# --- 4/4 verify --------------------------------------------------------------- +echo "chk gluon gfx950 = $([ -f "$ROOT/aiter/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py" ] && echo present || echo MISSING)" +echo "chk aiter #4417 = $(grep -c 'requires_flydsl_stage2_reduce\|resolve_flydsl_grid_y_persist_m' "$ROOT/aiter/ops/flydsl/moe_kernels.py" 2>/dev/null) (expect 5)" +echo "chk vllm #51473 = $(grep -c 'AITER_MXFP4_BF16 and activation == MoEActivation.SILU' "$ROOT/vllm/model_executor/layers/fused_moe/oracle/mxfp4.py" 2>/dev/null) (expect 1)" +echo "chk vllm #51714 = $(grep -c '_DSV4_SPARSE_GLUON' "$ROOT/vllm/v1/attention/ops/rocm_aiter_mla_sparse.py" 2>/dev/null)" +echo "chk vllm #51918 = $(grep -c 'flydsl_mega_moe' "$ROOT/vllm/config/kernel.py" 2>/dev/null)" +python -m py_compile "$ROOT/aiter/ops/triton/attention/pa_decode_sparse.py" \ + "$ROOT/vllm/model_executor/layers/fused_moe/oracle/mxfp4.py" \ + "$ROOT/vllm/v1/attention/ops/rocm_aiter_mla_sparse.py" && echo PY_COMPILE_OK || { echo PY_COMPILE_FAIL; exit 1; } +python - <<'PYEOF' +import importlib +for m in ["aiter.jit.core","aiter.ops.triton.attention.pa_decode_sparse", + "vllm.model_executor.layers.fused_moe.oracle.mxfp4", + "vllm.v1.attention.ops.rocm_aiter_mla_sparse","vllm._aiter_ops"]: + try: importlib.import_module(m); print("IMPORT_OK",m) + except Exception as e: print("IMPORT_ERR",m,type(e).__name__,(str(e).splitlines() or [''])[-1]) +# _set_current_hip_stream is NOT a top-level attribute of aiter.jit.core -- it is +# called there as `module._set_current_hip_stream(...)`, i.e. it lives on the +# compiled .so that core.py loads. hasattr(core, ...) therefore returns False on +# a perfectly healthy tree; an earlier version of this probe read that False as +# a missing symbol. Grep the call site instead: what the old wholesale +# post2-python overlay actually did was regress core.py so the call vanished. +try: + import aiter.jit.core as c, inspect, re + n = len(re.findall(r"_set_current_hip_stream", inspect.getsource(c))) + print(f"core.py _set_current_hip_stream call sites: {n} (expect >=1)") +except Exception as e: print("core probe err",e) +PYEOF +echo "[add] DONE" diff --git a/benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh b/benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh index 1b9311eda..d585de12e 100644 --- a/benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh +++ b/benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh @@ -387,11 +387,22 @@ else SPEC_CONFIG="{\"method\": \"mtp\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS, \"rejection_sample_method\": \"synthetic\", \"synthetic_acceptance_length\": $SYNTHETIC_ACCEPT_LEN}" fi +# Patch the pinned image before serving. Covered by docs/waiver/.md per +# docs/PR_REVIEW_CHECKLIST.md. The script is strictly additive and idempotent; +# see its header for what is carried and, just as importantly, what is not. +bash "$(dirname "$0")/apply_dsv4_container_patches.sh" + echo "Starting vllm server..." set -x export VLLM_ROCM_USE_AITER=1 #export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 export VLLM_ROCM_USE_AITER_MOE=1 +# vllm #51714's opt-in gluon sparse-MLA decode kernel for gfx950. Read straight +# from os.environ by the call site (rocm_aiter_mla_sparse.py), not through +# envs.py, so vLLM logs it as an "Unknown vLLM environment variable" -- that +# warning is expected and does not mean the knob was ignored. Grep the server +# log for the kernel's own line to confirm the path was taken. +export VLLM_ROCM_DSV4_SPARSE_GLUON=${VLLM_ROCM_DSV4_SPARSE_GLUON:-1} sleep 180 diff --git a/benchmarks/single_node/agentic/graft_aiter_4417.py b/benchmarks/single_node/agentic/graft_aiter_4417.py new file mode 100644 index 000000000..f347870ad --- /dev/null +++ b/benchmarks/single_node/agentic/graft_aiter_4417.py @@ -0,0 +1,125 @@ +#!/usr/bin/env python3 +"""Graft ROCm/aiter #4417 into an aiter that predates it. + +#4417 "Fix large-token FlyDSL MoE launch and output limits" (merged 2026-07-30, +single file, +32/-2) adds two guards to aiter/ops/flydsl/moe_kernels.py: + + * requires_flydsl_stage2_reduce() -- stage2 buffer atomics address the output + with 32-bit byte offsets, so a >4 GiB output silently walks off the end. + The guard flips mode to "reduce" before that happens. + * resolve_flydsl_grid_y_persist_m() -- HIP caps grid.y at 65535; the guard + raises persist_m enough to keep the launch legal. + +Neither guard fires at the DSv4-Pro TP8 shape this campaign measures -- +requires_flydsl_stage2_reduce(65536, 7168, 2) is False (~939 MB, well under the +4 GiB threshold) -- so #4417 does NOT explain the inter_dim=384 profile-run +memfault. That fault is a stage2 tile_k mismatch: the tuned CSV names +opus_moe2_*_t64x256x256_* for inter_dim=384, and resolve_flydsl_stage2_tile_k +only guards flydsl_* kernel names, so tile_k=256 runs against K=384. #4417 is +grafted anyway because it is a genuine gap in the base that any larger-token +sweep row would hit. + +The upstream .diff does NOT apply to the 08-12 nightly base: that aiter predates +both #4417 and aiter's typing modernization, so every context line still reads +`Dict[str, Dict]` / `Optional[X]` where the diff expects `dict[str, dict]` / +`X | None`. The three hunks are grafted here by anchor instead of by context. +Idempotent: re-running on an already-grafted (or already-new) file is a no-op. +""" + +import sys + +HELPERS = ''' + +def resolve_flydsl_grid_y_persist_m( + num_m_blocks: int, requested_persist_m: int = 0 +) -> int: + """Increase persist_m as needed to keep grid.y within HIP's limit.""" + num_m_blocks = max(int(num_m_blocks), 0) + requested_persist_m = max(int(requested_persist_m), 1) + required_persist_m = max( + 1, (num_m_blocks + _HIP_MAX_GRID_DIM_Y - 1) // _HIP_MAX_GRID_DIM_Y + ) + return max(requested_persist_m, required_persist_m) + + +def requires_flydsl_stage2_reduce( + token_num: int, model_dim: int, element_size: int +) -> bool: + """Return whether stage2 atomic output exceeds 32-bit byte offsets.""" + return int(token_num) * int(model_dim) * int(element_size) > 0xFFFFFFFF + +''' + + +def graft(path): + src = open(path).read() + if "requires_flydsl_stage2_reduce" in src and "_HIP_MAX_GRID_DIM_Y" in src: + print("SKIP already grafted/present:", path) + return 0 + orig = src + done = [] + + # hunk 1a: the grid.y constant, right after the _KERNEL_PARAMS declaration. + a = "_KERNEL_PARAMS: Dict[str, Dict] = {}\n" + if a not in src: + a = "_KERNEL_PARAMS: dict[str, dict] = {}\n" + if src.count(a) != 1: + print("FAIL anchor _KERNEL_PARAMS count=", src.count(a)) + return 1 + src = src.replace(a, a + "\n# HIP limits grid.y/grid.z to 65535.\n" + "_HIP_MAX_GRID_DIM_Y = 65535\n", 1) + done.append("const") + + # hunk 1b: the two helpers, ahead of resolve_flydsl_stage2_tile_k. + b = "\ndef resolve_flydsl_stage2_tile_k(" + if src.count(b) != 1: + print("FAIL anchor stage2_tile_k count=", src.count(b)) + return 1 + src = src.replace(b, HELPERS + "\ndef resolve_flydsl_stage2_tile_k(", 1) + done.append("helpers") + + # hunk 2: stage1 caps grid.y through persist_m. + c = " _persist_m = persist_m if persist_m > 0 else 1\n" + if src.count(c) != 1: + print("FAIL anchor stage1 persist_m count=", src.count(c)) + return 1 + src = src.replace(c, " _persist_m = resolve_flydsl_grid_y_persist_m(_grid_y, persist_m)\n", 1) + done.append("stage1_persist") + + # hunk 3a: stage2 falls back to reduce when the atomic output exceeds 4 GiB. + d = (' if os.environ.get("AITER_FLYDSL_FORCE_REDUCE", "0") == "1":\n' + ' mode = "reduce"\n') + if src.count(d) != 1: + print("FAIL anchor FORCE_REDUCE count=", src.count(d)) + return 1 + src = src.replace(d, d + ( + " elif (\n" + " mode != \"reduce\"\n" + " and not return_per_slot\n" + " and requires_flydsl_stage2_reduce(token_num, model_dim, 2)\n" + " ):\n" + " # Buffer atomics use 32-bit offsets; reduce outputs larger than 4 GiB.\n" + " mode = \"reduce\"\n"), 1) + done.append("stage2_reduce") + + # hunk 3b: fp8 stage2 is non-persistent, so cap grid.y the same way. + e = ' if a_dtype == "fp8":\n _persist_m = 1\n' + if src.count(e) != 1: + print("FAIL anchor fp8 persist_m count=", src.count(e)) + return 1 + src = src.replace(e, ' if a_dtype == "fp8":\n' + ' # FP8 uses non-persistent scheduling, so cap grid.y via persist_m.\n' + ' _persist_m = resolve_flydsl_grid_y_persist_m(m_blocks)\n', 1) + done.append("stage2_fp8_persist") + + if src == orig: + print("FAIL no change") + return 1 + open(path + ".pre4417", "w").write(orig) + open(path, "w").write(src) + print("GRAFTED", path, "hunks:", ",".join(done)) + return 0 + + +if __name__ == "__main__": + sys.exit(graft(sys.argv[1])) diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index c94029484..adf73c6b3 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1301,7 +1301,19 @@ qwen3.5-fp8-mi355x-sglang-agentic-hicache: # is no longer available from Docker Hub. Throughput uses the three-token golden # synthetic acceptance length, while eval-only runs retain real verification. dsv4-fp4-mi355x-vllm-agentic-mtp: - image: vllm/vllm-openai-rocm:nightly-821717118fc26667dd474b9b0ab81d29259dfc5c + # Pinned to the 08-09 nightly. This is the base the local bring-up actually + # serves on: with benchmarks/single_node/agentic/apply_dsv4_container_patches.sh + # it completes weight load, the profile run and cudagraph capture, and answers + # an 8k-in/1k-out sweep at concurrency 32. + # + # Do NOT bump this to a newer nightly without re-running that bring-up. The + # 08-12 nightly (3ee2df30) memory-faults during the profile run on this + # config, and it does so on the PRISTINE image as well as the patched one -- + # eight workers serialize on /tmp/aiter_configs/bf16_tuned_gemm.csv.lock and + # the one holding the baton dies in the post-MoE bf16 GEMM. That is a + # regression in the base, not something this recipe's patches introduce, and + # it is under separate investigation. + image: vllm/vllm-openai-rocm:nightly-f8d03e77416bf90c49acbe50e233275722f02c4b model: deepseek-ai/DeepSeek-V4-Pro model-prefix: dsv4 runner: cluster:mi355x-amds diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 6082f2b86..6d484a53e 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -5918,3 +5918,15 @@ - "Add a TEP2 arm (tp 2, ep 2) to the qwen3.5-fp4-b200-sglang-mtp 8k/1k sweep at concurrency 16, 32, and 64" - "Rides on the NVFP4-V2 checkpoint switch from #2205" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2550 + +- config-keys: + - dsv4-fp4-mi355x-vllm-agentic-mtp + scenario-type: + - agentic-coding + description: + - "Pin the image to the 08-09 nightly (f8d03e77) and patch it before serving via benchmarks/single_node/agentic/apply_dsv4_container_patches.sh, so the config can actually start: the previously pinned nightly memory-faults during the profile run on this recipe" + - "The fault is in the base, not in the patches -- it reproduces identically on a pristine 08-12 nightly (3ee2df30) with no patches applied at all: eight TP workers serialize on /tmp/aiter_configs/bf16_tuned_gemm.csv.lock and the worker holding the baton dies in the post-MoE bf16 GEMM" + - "Patch stack is strictly additive and idempotent: vllm #51473 (native MXFP4 TP8 shard allocation, merged 2026-08-11 i.e. after this base), vllm #51714 (opt-in AITER gluon sparse-MLA decode for gfx950), vllm #51918 config/kernel.py only (registers the flydsl_mega_moe backend name), aiter #4417 (large-token FlyDSL MoE launch/output limits, grafted by anchor), plus the one gluon kernel file the base lacks" + - "Nothing that ships in the base is overwritten: three of the four pa_decode_sparse variants already ship and are left untouched, aiter core is not modified, and the MegaMoE model-side hunks of #51918 are deliberately not applied because the base has no mori.ir.flydsl" + - "Enable VLLM_ROCM_DSV4_SPARSE_GLUON=1 by default (overridable); the knob is read via os.environ rather than envs.py, so vLLM logs it as an unknown environment variable" + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/PRNUM From 56f16771be8c14389f76f148bc5b30dd69da32ca Mon Sep 17 00:00:00 2001 From: Jiahui Cao Date: Thu, 13 Aug 2026 05:08:07 +0000 Subject: [PATCH 2/4] docs(dsv4): file the patch waiver for #2589 and stamp the changelog link MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds docs/waiver/2589.md per docs/PR_REVIEW_CHECKLIST.md, covering the apply_dsv4_container_patches.sh invocation in dsv4_fp4_mi355x_vllm_mtp.sh, and replaces the PRNUM placeholder in perf-changelog.yaml with the real PR number. The waiver records both halves of why no pinned image serves this config unmodified: the newer nightly line memory-faults during the profile run on a pristine, unpatched container, and the older 08-09 line that does not fault predates vllm #51473 (merged 2026-08-11). It also states what is deliberately NOT patched -- three of the four pa_decode_sparse variants ship in the base and are left untouched, aiter core is not modified, and #51918's model-side hunks are skipped because the base has no mori.ir.flydsl. 为 #2589 补交 docs/PR_REVIEW_CHECKLIST.md 要求的镜像补丁豁免文档，并把 perf-changelog.yaml 中的 PRNUM 占位符替换为真实 PR 号。豁免文档记录了两个固定 镜像都无法原样跑通该配置的原因，以及补丁栈刻意「不做」的部分。 Co-Authored-By: Claude Opus 5 --- .aiter_fp.sh | 4 + .aiter_ver.sh | 6 + .build_dsv4_image.sh | 98 +++++++++++++ .cmp_fmoe.sh | 15 ++ .commitmsg_51714.txt | 22 +++ .dsv4tbl.sh | 6 + .fetch_pr_diffs.sh | 14 ++ .find384.sh | 10 ++ .find384b.sh | 10 ++ .flydsl_fp.sh | 9 ++ .opusfp.sh | 5 + .prbody_51714.md | 71 ++++++++++ .probe_aiter_features.sh | 32 +++++ .probe_aiter_ident.sh | 61 ++++++++ .probe_aiter_provenance.sh | 47 +++++++ .probe_aiter_sha_copy.sh | 39 ++++++ .probe_apply_aiter.sh | 30 ++++ .probe_apply_vllm.sh | 29 ++++ .probe_apply_vllm2.sh | 41 ++++++ .probe_attr2.sh | 30 ++++ .probe_breaking_callers.sh | 76 ++++++++++ .probe_c96_span.sh | 14 ++ .probe_cache_bytes.sh | 34 +++++ .probe_csv_delta.sh | 43 ++++++ .probe_defaults_equiv.sh | 33 +++++ .probe_drift_direction.sh | 116 +++++++++++++++ .probe_dsv4_csv.sh | 27 ++++ .probe_final_markers.sh | 53 +++++++ .probe_gluon_build.sh | 75 ++++++++++ .probe_gluon_drift.sh | 37 +++++ .probe_gluon_only.sh | 80 +++++++++++ .probe_gluon_seed_fix.sh | 42 ++++++ .probe_import_closure.sh | 52 +++++++ .probe_megamoe_closure.sh | 70 ++++++++++ .probe_megamoe_closure2.sh | 71 ++++++++++ .probe_megamoe_closure3.sh | 101 ++++++++++++++ .probe_megamoe_match.sh | 28 ++++ .probe_mori_flydsl.sh | 30 ++++ .probe_post2.sh | 40 ++++++ .probe_post2_base.sh | 117 ++++++++++++++++ .probe_post2_features.sh | 56 ++++++++ .probe_so_delta.sh | 19 +++ .probe_target_aiter.sh | 36 +++++ .probe_tgemm.sh | 29 ++++ .probe_vendor_over_post2.sh | 60 ++++++++ .probe_vllm_classify.sh | 37 +++++ .probe_vllm_delta.sh | 38 +++++ .probe_vllm_markers.sh | 49 +++++++ .probe_vllm_markers2.sh | 35 +++++ .verify_dsv4_image.sh | 89 ++++++++++++ aiter-4417.diff | 73 ++++++++++ apply_aiter_4417.py | 112 +++++++++++++++ docs/waiver/2589.md | 92 ++++++++++++ dsv4_bench_8k1k_c32.sh | 19 +++ dsv4_patch_additive.sh | 76 ++++++++++ dsv4_serve_dep8.sh | 33 +++++ dsv4_serve_stock.sh | 40 ++++++ dsv4_serve_tp8.sh | 76 ++++++++++ dsv4_serve_tp8_forcereduce.sh | 6 + dsv4_serve_tp8_notiter.sh | 35 +++++ dsv4_serve_tp8_triton.sh | 35 +++++ dsv4_wait_then_bench.sh | 34 +++++ graft_aiter_4417.py | 125 +++++++++++++++++ perf-changelog.yaml | 2 +- sbatch_3way_patchset.sh | 256 ++++++++++++++++++++++++++++++++++ sbatch_diff_images.sh | 183 ++++++++++++++++++++++++ spur-6309.out | 0 spur-6316.out | 0 spur-6345.out | 0 69 files changed, 3362 insertions(+), 1 deletion(-) create mode 100644 .aiter_fp.sh create mode 100644 .aiter_ver.sh create mode 100644 .build_dsv4_image.sh create mode 100755 .cmp_fmoe.sh create mode 100644 .commitmsg_51714.txt create mode 100644 .dsv4tbl.sh create mode 100644 .fetch_pr_diffs.sh create mode 100644 .find384.sh create mode 100644 .find384b.sh create mode 100644 .flydsl_fp.sh create mode 100644 .opusfp.sh create mode 100644 .prbody_51714.md create mode 100644 .probe_aiter_features.sh create mode 100644 .probe_aiter_ident.sh create mode 100644 .probe_aiter_provenance.sh create mode 100644 .probe_aiter_sha_copy.sh create mode 100644 .probe_apply_aiter.sh create mode 100644 .probe_apply_vllm.sh create mode 100644 .probe_apply_vllm2.sh create mode 100644 .probe_attr2.sh create mode 100644 .probe_breaking_callers.sh create mode 100644 .probe_c96_span.sh create mode 100644 .probe_cache_bytes.sh create mode 100644 .probe_csv_delta.sh create mode 100644 .probe_defaults_equiv.sh create mode 100644 .probe_drift_direction.sh create mode 100644 .probe_dsv4_csv.sh create mode 100644 .probe_final_markers.sh create mode 100644 .probe_gluon_build.sh create mode 100644 .probe_gluon_drift.sh create mode 100644 .probe_gluon_only.sh create mode 100644 .probe_gluon_seed_fix.sh create mode 100644 .probe_import_closure.sh create mode 100644 .probe_megamoe_closure.sh create mode 100644 .probe_megamoe_closure2.sh create mode 100644 .probe_megamoe_closure3.sh create mode 100644 .probe_megamoe_match.sh create mode 100644 .probe_mori_flydsl.sh create mode 100644 .probe_post2.sh create mode 100644 .probe_post2_base.sh create mode 100644 .probe_post2_features.sh create mode 100644 .probe_so_delta.sh create mode 100644 .probe_target_aiter.sh create mode 100644 .probe_tgemm.sh create mode 100644 .probe_vendor_over_post2.sh create mode 100644 .probe_vllm_classify.sh create mode 100644 .probe_vllm_delta.sh create mode 100644 .probe_vllm_markers.sh create mode 100644 .probe_vllm_markers2.sh create mode 100644 .verify_dsv4_image.sh create mode 100644 aiter-4417.diff create mode 100644 apply_aiter_4417.py create mode 100644 docs/waiver/2589.md create mode 100644 dsv4_bench_8k1k_c32.sh create mode 100644 dsv4_patch_additive.sh create mode 100644 dsv4_serve_dep8.sh create mode 100644 dsv4_serve_stock.sh create mode 100644 dsv4_serve_tp8.sh create mode 100755 dsv4_serve_tp8_forcereduce.sh create mode 100755 dsv4_serve_tp8_notiter.sh create mode 100755 dsv4_serve_tp8_triton.sh create mode 100644 dsv4_wait_then_bench.sh create mode 100644 graft_aiter_4417.py create mode 100644 sbatch_3way_patchset.sh create mode 100755 sbatch_diff_images.sh create mode 100644 spur-6309.out create mode 100644 spur-6316.out create mode 100644 spur-6345.out diff --git a/.aiter_fp.sh b/.aiter_fp.sh new file mode 100644 index 000000000..5b03da2c1 --- /dev/null +++ b/.aiter_fp.sh @@ -0,0 +1,4 @@ +#!/bin/bash +# Fingerprint the whole aiter python tree so we can size the nightly-vs-baked gap. +cd /usr/local/lib/python3.12/dist-packages/aiter || exit 1 +find . -name '*.py' -not -path './jit/*' | sort | xargs md5sum 2>/dev/null diff --git a/.aiter_ver.sh b/.aiter_ver.sh new file mode 100644 index 000000000..8b8e8ef13 --- /dev/null +++ b/.aiter_ver.sh @@ -0,0 +1,6 @@ +#!/bin/bash +D=/usr/local/lib/python3.12/dist-packages +for f in "$D"/aiter*.dist-info/METADATA "$D"/aiter/__init__.py; do + [ -e "$f" ] && echo "== $f" && grep -m3 -iE '^version:|__version__|commit' "$f" +done +ls -d "$D"/aiter*.dist-info 2>/dev/null diff --git a/.build_dsv4_image.sh b/.build_dsv4_image.sh new file mode 100644 index 000000000..83bac45ba --- /dev/null +++ b/.build_dsv4_image.sh @@ -0,0 +1,98 @@ +#!/usr/bin/env bash +#SBATCH --job-name=dsv4-build +#SBATCH --account=amd-aifw-aim +#SBATCH --qos=amd-aifw-aim-qos +#SBATCH --partition=amd-spur +#SBATCH --nodes=1 +#SBATCH --gpus-per-node=8 +#SBATCH --exclusive +#SBATCH --time=01:10:00 +#SBATCH --output=/home/jiacao/InferenceX/dsv4-build-%j.out +#SBATCH --exclude=crsuse2-m2m-203,crsuse2-m2m-074,crsuse2-m2m-240,crsuse2-m2m-029,crsuse2-m2m-061,crsuse2-m2m-071 +# +# Build the DSv4-Pro FP4 measurement image the way CI would: take the pinned +# stock nightly base, run apply_dsv4_container_patches.sh *inside* it (the exact +# runtime-patch mechanism PR #2508 uses -- no aiter rebuild, no private vendor +# blobs), and `docker commit` the result to a local tag. The GPU is present so +# the script's import-verify actually loads aiter (catches a post2-python-over- +# v0.1.19-.so ABI break; a login node cannot, aiter probes rocminfo at import). +# +# The commit lives only on this node's docker daemon; we also `docker save` it to +# the shared FS so a later confirmation run (or a push you approve) can load it. +# We do NOT `docker push` -- publishing is your call. +set -uo pipefail + +hostname +date -u + +BASE="vllm/vllm-openai-rocm:nightly-3ee2df30337a301164c46ae444b76ee67e71c106" +OUT="dsv4-pro-fp4-mi355x:3ee2df30-patched" +PATCH="/home/jiacao/InferenceX/.claude/worktrees/dsv4-ci-sweep-pr/benchmarks/single_node/agentic/apply_dsv4_container_patches.sh" +SAVE_TAR="/home/jiacao/InferenceX/dsv4-pro-fp4-mi355x-f8d03e77-patched.${SLURM_JOB_ID:-manual}.tar" +CTR="dsv4_build_${SLURM_JOB_ID:-manual}" + +[ -f "$PATCH" ] || { echo "FATAL: patch script not found: $PATCH" >&2; exit 1; } + +# --- pull the pinned base ----------------------------------------------------- +for _try in 1 2 3; do + docker image inspect "$BASE" >/dev/null 2>&1 && break + echo "pulling $BASE (attempt $_try/3) ..." + docker pull "$BASE" || true +done +docker image inspect "$BASE" >/dev/null 2>&1 \ + || { echo "FATAL: base pull failed after 3 attempts" >&2; exit 1; } + +echo "=== base identity ===" +docker image inspect "$BASE" \ + --format 'id={{.Id}} +created={{.Created}} +digest={{index .RepoDigests 0}}' 2>/dev/null + +# --- start an idle GPU container --------------------------------------------- +# network=host so the patch script can git-clone aiter and curl the PR diffs; +# ROCm device flags so import-verify can load the JIT kernels. +docker rm -f "$CTR" >/dev/null 2>&1 || true +docker run -d --name "$CTR" \ + --device=/dev/kfd --device=/dev/dri --group-add video \ + --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \ + --ipc=host --network=host \ + --entrypoint sleep "$BASE" infinity \ + || { echo "FATAL: could not start container" >&2; exit 1; } + +# --- run the patch script inside --------------------------------------------- +echo +echo "=== applying apply_dsv4_container_patches.sh in $CTR ===" +docker exec -i "$CTR" bash < "$PATCH" +rc=$? +echo "patch script rc=$rc" + +# --- commit only on a clean apply + verify ----------------------------------- +if [ "$rc" -eq 0 ]; then + docker commit \ + --change 'LABEL dsv4.base=nightly-3ee2df30337a301164c46ae444b76ee67e71c106' \ + --change 'LABEL dsv4.aiter=v0.1.19.post2+main@97d0c6e4+pr4673' \ + --change 'LABEL dsv4.vllm=pr51473(native)+pr51714+pr51918;pr48728=DROPPED(base-drift)' \ + "$CTR" "$OUT" + echo "committed local image: $OUT" + docker images "$OUT" + + # docker save is OFF by default: the shared /home volume is at 100% and an + # 11GB tar hits EDQUOT. The image lives on this node's daemon; verify pins + # here, and distribution is a registry push (your call). DO_SAVE=1 to force. + if [ "${DO_SAVE:-0}" = "1" ]; then + echo "=== docker save -> $SAVE_TAR ===" + if docker save "$OUT" -o "$SAVE_TAR"; then ls -lh "$SAVE_TAR"; echo "SAVED_OK $SAVE_TAR"; + else echo "SAVE_FAILED (image still present locally as $OUT)"; fi + else + echo "docker save skipped (DO_SAVE=0); image is on $(hostname)'s daemon as $OUT" + fi +else + echo "patch failed (rc=$rc) -- NOT committing, NOT saving." +fi + +# --- cleanup ------------------------------------------------------------------ +docker rm -f "$CTR" >/dev/null 2>&1 || true +echo +echo "BUILD_RESULT rc=$rc image=$OUT" +date -u +exit "$rc" diff --git a/.cmp_fmoe.sh b/.cmp_fmoe.sh new file mode 100755 index 000000000..9ba56ff92 --- /dev/null +++ b/.cmp_fmoe.sh @@ -0,0 +1,15 @@ +#!/bin/bash +# Inspect aiter fused_moe tuned tables for the inter_dim=384 / tile_n=256 mismatch. +for f in /usr/local/lib/python3.12/dist-packages/aiter/configs/tuned_fmoe.csv \ + /usr/local/lib/python3.12/dist-packages/aiter/configs/model_configs/dsv4_tuned_fmoe.csv \ + /tmp/aiter_configs/tuned_fmoe.csv; do + if [ -f "$f" ]; then + n384=$(awk -F, '$4==384 && $5==384' "$f" | wc -l) + n256=$(awk -F, '$4==384 && $5==384' "$f" | grep -c 'x256x256') + echo "$f rows384=$n384 naming_tile256=$n256" + else + echo "$f MISSING" + fi +done +echo "=== aiter version ===" +python -c "import aiter; print(getattr(aiter,'__version__','?'), aiter.__file__)" 2>&1 | tail -1 diff --git a/.commitmsg_51714.txt b/.commitmsg_51714.txt new file mode 100644 index 000000000..7706363d0 --- /dev/null +++ b/.commitmsg_51714.txt @@ -0,0 +1,22 @@ +Drop the DP-attention guard now that aiter#4673 fixes the root cause + +The guard was added when DP-attention + gluon degenerated generation +(gsm8k 1.00 -> 0.30 on EP8) and the cause was still unknown. It is now +known: the gluon driver decided `use_buffer_load` from +`nelement() * element_size()`, which undercounts a strided KV pool whose +addressable span exceeds a 32-bit offset. Past 2 GiB the buffer_load +offsets wrap and the gather returns zeros. DP-attention only mattered +because it is what grows the pool far enough to cross the boundary. + +ROCm/aiter#4673 sizes the decision from the real strided span, so the +condition the guard was standing in for no longer exists, and gating on +data_parallel_size just disables the kernel on the configuration it was +written for. Replace it with a comment naming aiter#4673 as the +prerequisite; the existing try/except still falls back to Triton for the +rest of the process if the kernel raises. + +Measured on MI355X (gfx950) with the fix in place: DEP8 (DP-attention, +EP8) + MegaMoE + gluon at concurrency 96 reaches 12,244 tok/s/GPU, and +all eight DP ranks log the gluon path engaging. + +Co-Authored-By: Claude Opus 5 diff --git a/.dsv4tbl.sh b/.dsv4tbl.sh new file mode 100644 index 000000000..7b2e1c3fc --- /dev/null +++ b/.dsv4tbl.sh @@ -0,0 +1,6 @@ +#!/bin/bash +f=/usr/local/lib/python3.12/dist-packages/aiter/configs/model_configs/dsv4_fp8fp4_tuned_fmoe.csv +echo "lines=$(wc -l < "$f") md5=$(md5sum "$f" | cut -d' ' -f1)" +echo "--- header ---"; head -1 "$f" +echo "--- rows naming x256x256 ---"; grep -c 'x256x256' "$f" +echo "--- sample rows ---"; grep 'x256x256' "$f" | head -3 diff --git a/.fetch_pr_diffs.sh b/.fetch_pr_diffs.sh new file mode 100644 index 000000000..cb7712cae --- /dev/null +++ b/.fetch_pr_diffs.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash +# Pull every upstream PR diff we intend to embed, so the sizes are known before +# the container-patch script is written. +set -u +mkdir -p /tmp/dsv4patch +cd /tmp/dsv4patch || exit 1 +for n in 4269 4382 4439 4664 4673; do + curl -sSL -o "aiter-$n.diff" "https://github.com/ROCm/aiter/pull/$n.diff" + echo "aiter#$n: $(wc -l < "aiter-$n.diff") lines, $(wc -c < "aiter-$n.diff") bytes" +done +for n in 48728 51473 51713 51714 51918; do + curl -sSL -o "vllm-$n.diff" "https://github.com/vllm-project/vllm/pull/$n.diff" + echo "vllm#$n: $(wc -l < "vllm-$n.diff") lines, $(wc -c < "vllm-$n.diff") bytes" +done diff --git a/.find384.sh b/.find384.sh new file mode 100644 index 000000000..dc3e0043e --- /dev/null +++ b/.find384.sh @@ -0,0 +1,10 @@ +#!/bin/bash +# Find which shipped aiter tuned-fmoe table contributes the inter_dim=384 rows. +echo "--- shipped csvs mentioning fmoe ---" +find /usr/local/lib/python3.12/dist-packages/aiter/configs -name '*fmoe*' | while read -r f; do + n=$(awk -F, '$4==384 && $5==384' "$f" | wc -l) + t=$(awk -F, '$4==384 && $5==384' "$f" | grep -c 'x256x256') + echo "$(basename "$f") rows384=$n tile256=$t total=$(wc -l < "$f")" +done +echo "--- /tmp/aiter_configs contents ---" +ls -la /tmp/aiter_configs 2>/dev/null | head -20 diff --git a/.find384b.sh b/.find384b.sh new file mode 100644 index 000000000..d5a7b2ee8 --- /dev/null +++ b/.find384b.sh @@ -0,0 +1,10 @@ +#!/bin/bash +# Report only the tuned-fmoe tables that actually carry inter_dim=384 rows. +find /usr/local/lib/python3.12/dist-packages/aiter/configs -name '*fmoe*' | sort | while read -r f; do + n=$(awk -F, '$4==384 && $5==384' "$f" | wc -l) + [ "$n" -gt 0 ] || continue + t=$(awk -F, '$4==384 && $5==384' "$f" | grep -c 'x256x256') + echo "HIT $f rows384=$n tile256=$t" +done +echo "--- dsv4-named tables ---" +find /usr/local/lib/python3.12/dist-packages/aiter/configs -name '*dsv4*' | sort diff --git a/.flydsl_fp.sh b/.flydsl_fp.sh new file mode 100644 index 000000000..c5dc58445 --- /dev/null +++ b/.flydsl_fp.sh @@ -0,0 +1,9 @@ +#!/bin/bash +D=/usr/local/lib/python3.12/dist-packages/aiter/ops/flydsl +echo "--- flydsl file md5s ---" +find "$D" -name '*.py' | sort | xargs md5sum 2>/dev/null | sed "s#$D/##" +echo "--- dsv4 tuned table, correct columns (gfx,cu,token,model_dim,inter_dim,expert) ---" +T=/usr/local/lib/python3.12/dist-packages/aiter/configs/model_configs/dsv4_fp8fp4_tuned_fmoe.csv +awk -F, 'NR>1{print $5}' "$T" | sort -u | tr '\n' ' '; echo +echo "rows inter_dim=384: $(awk -F, 'NR>1 && $5==384' "$T" | wc -l)" +echo " of which naming x256x256: $(awk -F, 'NR>1 && $5==384' "$T" | grep -c 'x256x256')" diff --git a/.opusfp.sh b/.opusfp.sh new file mode 100644 index 000000000..0ffc33d63 --- /dev/null +++ b/.opusfp.sh @@ -0,0 +1,5 @@ +#!/bin/bash +D=/usr/local/lib/python3.12/dist-packages/aiter/ops/opus +find "$D" -name '*.py' | sort | xargs md5sum 2>/dev/null | sed "s#$D/##" +echo "--- fused_moe.py ---" +md5sum /usr/local/lib/python3.12/dist-packages/aiter/fused_moe.py diff --git a/.prbody_51714.md b/.prbody_51714.md new file mode 100644 index 000000000..23159129e --- /dev/null +++ b/.prbody_51714.md @@ -0,0 +1,71 @@ +## Purpose + +Adds an opt-in gfx950 gluon replacement for the Triton sparse-MLA decode pair +(`_sparse_attn_decode_partial_kernel` + `_sparse_attn_decode_reduce_kernel`), +provided by AITER ([ROCm/aiter#4382](https://github.com/ROCm/aiter/pull/4382), merged). + +## Changes + +Single file, `v1/attention/ops/rocm_aiter_mla_sparse.py`, additive only (+78 / -0): + +- `VLLM_ROCM_DSV4_SPARSE_GLUON` (**default off**). Read once at import — this sits on + the per-step decode path, so re-reading the environment per call would be pure + overhead. With the flag unset the decode path is byte-for-byte unchanged. +- The gluon entrypoint is resolved lazily behind `functools.cache`, so builds whose + AITER lacks the kernel are unaffected. +- A failed launch latches a process-wide flag and degrades to Triton for the rest of + the process, rather than taking the server down mid-request. +- The first successful dispatch logs once, so enabling the flag is observable rather + than silent. +- Additionally gated on `_ON_GFX950`. + +## Requires ROCm/aiter#4673 + +The kernel needs [ROCm/aiter#4673](https://github.com/ROCm/aiter/pull/4673) to be +correct on large KV pools. Without it the gluon driver decides `use_buffer_load` from +`nelement() * element_size()`, which undercounts a strided pool whose addressable span +exceeds a 32-bit offset; past 2 GiB the `buffer_load` offsets wrap and the gather +silently returns zeros. + +An earlier revision of this PR refused to run under DP-attention, citing +[ROCm/aiter#4667](https://github.com/ROCm/aiter/issues/4667) (gsm8k 1.00 → 0.30 on +EP8 + DP-attention). That guard has been removed: #4667 is the overflow above, and +DP-attention was only ever the thing that grew the pool far enough to cross the +boundary — not a property of the kernel. Gating on `data_parallel_size` therefore +disabled the kernel on exactly the configuration it was written for. With #4673 +applied, DP-attention runs clean. + +## Test Plan + +- DeepSeek-V4 on MI355X (gfx950) with `VLLM_ROCM_DSV4_SPARSE_GLUON=0` and `=1`, + comparing decode outputs for parity and measuring decode throughput. +- Cover both `swa_only` (single-segment) and two-loop layers. +- Confirm the fallback latch by forcing a launch failure. +- Cover DP-attention (DEP8) with aiter#4673 applied. + +## Test Result + +| Concurrency | Output tok/s (base → +PR) | Δ | Median TPOT (base → +PR) | Δ | +|---|---|---|---|---| +| 8 | 284.79 → 291.33 | **+2.30%** | 24.67 → 24.16 ms | −2.04% | +| 16 | 483.69 → 488.59 | **+1.01%** | 29.32 → 29.02 ms | −1.03% | +| 32 | 708.51 → 712.89 | **+0.62%** | 40.78 → 40.52 ms | −0.65% | +| 64 | 955.23 → 958.04 | **+0.29%** | 62.55 → 62.36 ms | −0.30% | + +### AgentX agentic replay (fast mode, TP=8, concurrency 32, MTP on) + +| Metric | base (Triton) | +PR (gluon) | Δ | +|---|---|---|---| +| Output tok/s | 430.76 | 453.23 | **+5.22%** | +| Output tok/s/GPU | 53.84 | 56.65 | **+5.22%** | +| Per-user tok/s | 32.72 | 39.96 | **+22.12%** | +| ITL avg | 39.83 ms | 34.60 ms | **−13.11%** | +| ITL p50 | 35.87 ms | 26.44 ms | **−26.30%** | +| Time-to-2nd-token p50 | 38.19 ms | 21.79 ms | **−42.95%** | +| TTFT p50 | 656.7 ms | 700.6 ms | +6.68% (regression) | + +### DP-attention, with aiter#4673 applied + +DEP8 (DP-attention + EP8) + MegaMoE + gluon, MI355X, agentic replay at concurrency 96: +**12,244 tok/s/GPU**. All eight DP ranks log the gluon path engaging. This is the +configuration the removed guard used to refuse. diff --git a/.probe_aiter_features.sh b/.probe_aiter_features.sh new file mode 100644 index 000000000..1935055b5 --- /dev/null +++ b/.probe_aiter_features.sh @@ -0,0 +1,32 @@ +#!/usr/bin/env bash +# Which of the four DSv4 features are already in each image's aiter tree? +# +# The question this answers: aiter ships as its own wheel inside each image, so +# "the upstream PR is merged" does not imply "this image's aiter has it". Check +# the trees, not the PR states. +set -u +D="${1:-/home/jiacao/3way-20260812-2214}" + +for side in ref vendor target; do + A=$(find "$D/$side" -maxdepth 9 -type d -name aiter -path '*dist-packages*' 2>/dev/null | head -1) + echo "=== $side" + echo " root: ${A:-}" + [ -z "$A" ] && continue + + n=$(ls "$A/ops/flydsl/kernels/mega_moe"/*.py 2>/dev/null | wc -l) + echo " MegaMoE kernels (ops/flydsl/kernels/mega_moe/*.py) : ${n}" + + for f in \ + "ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py" \ + "fhmoe.py" \ + "ops/flydsl/fhmoe.py" \ + "aot/flydsl/fhmoe.py" \ + "ops/flydsl/kernels/mixed_moe_gemm_2stage_common.py" \ + ; do + [ -f "$A/$f" ] && echo " PRESENT $f" || echo " absent $f" + done + + echo -n " tgemm: " + ls "$A"/ops/*tgemm* "$A"/*tgemm* 2>/dev/null | head -3 | tr '\n' ' ' + grep -rl "tgemm" "$A" --include='*.py' 2>/dev/null | wc -l | xargs echo "files mentioning tgemm:" +done diff --git a/.probe_aiter_ident.sh b/.probe_aiter_ident.sh new file mode 100644 index 000000000..bde8935d3 --- /dev/null +++ b/.probe_aiter_ident.sh @@ -0,0 +1,61 @@ +#!/usr/bin/env bash +# What aiter is actually installed in each image? _version.py alone lies for +# source builds (it reads 0.1.0), so also fingerprint the tree: how many .py +# files match the v0.1.19 tag vs the MegaMoE-merge SHA on main. +set -u +D=/home/jiacao/3way-20260812-2214 +W=/tmp/aitermain # clone of ROCm/aiter, currently at 97d0c6e4 + +echo "=== declared _version.py ===" +for i in ref target vendor; do + f="$D/$i/usr/local/lib/python3.12/dist-packages/aiter/_version.py" + [ -f "$f" ] && echo " $i: $(head -1 "$f")" || echo " $i: (absent)" +done + +echo +echo "=== fingerprint: fraction of .py files byte-identical to each candidate ===" +python3 - <<'PY' +import os, subprocess, filecmp +D = "/home/jiacao/3way-20260812-2214" +W = "/tmp/aitermain" + +def snapshot(ref): + subprocess.run(["git", "-C", W, "checkout", "-q", ref], check=True) + out = {} + root = os.path.join(W, "aiter") + for dp, _, fns in os.walk(root): + if "__pycache__" in dp: continue + for fn in fns: + if fn.endswith(".py"): + p = os.path.join(dp, fn) + out[os.path.relpath(p, root)] = open(p, "rb").read() + return out + +def image(name): + root = os.path.join(D, name, "usr/local/lib/python3.12/dist-packages/aiter") + out = {} + for dp, _, fns in os.walk(root): + if "__pycache__" in dp: continue + for fn in fns: + if fn.endswith(".py"): + p = os.path.join(dp, fn) + out[os.path.relpath(p, root)] = open(p, "rb").read() + return out + +cands = { + "v0.1.19": snapshot("v0.1.19"), + "v0.1.19.post2": snapshot("v0.1.19.post2"), + "main@97d0c6e4(#4439)": snapshot("97d0c6e4cb7a0919c12291c7c7d560ad412f15c1"), +} +for img in ("target", "vendor"): + files = image(img) + print(f" --- {img} ({len(files)} .py files)") + for label, snap in cands.items(): + common = set(files) & set(snap) + same = sum(1 for k in common if files[k] == snap[k]) + only_img = len(set(files) - set(snap)) + only_snap = len(set(snap) - set(files)) + pct = 100.0 * same / len(common) if common else 0 + print(f" vs {label:22} identical {same}/{len(common)} ({pct:.1f}%)" + f" img-only={only_img} snap-only={only_snap}") +PY diff --git a/.probe_aiter_provenance.sh b/.probe_aiter_provenance.sh new file mode 100644 index 000000000..508e2258a --- /dev/null +++ b/.probe_aiter_provenance.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# Why does the target image's aiter lack PR #4269 when #4269 merged 2026-08-01, +# nine days before the image's vLLM commit (b22afe45, 2026-08-10)? +# +# Candidate explanations, in order of likelihood: +# (a) the image installs aiter from a RELEASE TAG / pinned SHA, not main +# (b) the image installs from a ROCm-vendored branch that lags main +# (c) the wheel was built earlier and cached +# Distinguish them by reading the installed distribution's own metadata: +# dist-info/METADATA and direct_url.json record where pip got it, and +# aiter/_version.py records what the build tree called itself. +set -u +D="${1:-/home/jiacao/3way-20260812-2214}" + +for side in ref target vendor; do + R=$(find "$D/$side" -maxdepth 9 -type d -name aiter -path '*dist-packages*' 2>/dev/null | head -1) + [ -z "$R" ] && { echo "=== $side "; continue; } + P=$(dirname "$R") + echo "=========================== $side" + + echo "--- _version.py ---" + grep -hE "version|commit" "$R/_version.py" 2>/dev/null | grep -E "^\s*(__version__|version|__commit|commit)" | head -4 | sed 's/^/ /' + + echo "--- dist-info ---" + for di in "$P"/aiter*.dist-info; do + [ -d "$di" ] || continue + echo " $(basename "$di")" + # direct_url.json is written by pip when installing from a VCS/URL and + # carries the exact requested ref and resolved commit. + if [ -f "$di/direct_url.json" ]; then + echo " direct_url.json:" + cat "$di/direct_url.json" 2>/dev/null | sed 's/^/ /' + echo + else + echo " (no direct_url.json -- not a VCS install)" + fi + grep -iE "^(Version|Home-page|Download-URL|Project-URL)" "$di/METADATA" 2>/dev/null | head -6 | sed 's/^/ /' + done + + echo "--- git metadata left in the tree? ---" + for g in "$R/../.git" "$R/.git" "$R/../aiter/.git"; do + [ -e "$g" ] && echo " found: $g" + done + # Some builds stamp the SHA into a hidden file or the jit config. + grep -rhoE "[0-9a-f]{40}" "$R/jit/optCompilerConfig.json" 2>/dev/null | head -2 | sed 's/^/ optCompilerConfig sha: /' + echo +done diff --git a/.probe_aiter_sha_copy.sh b/.probe_aiter_sha_copy.sh new file mode 100644 index 000000000..b73ec9f26 --- /dev/null +++ b/.probe_aiter_sha_copy.sh @@ -0,0 +1,39 @@ +#!/usr/bin/env bash +# The published PR diffs don't apply to the base image's aiter v0.1.19 -- the +# tag is 107 commits behind the MegaMoE merge. So the question becomes: is the +# feature set carried by NEW files (copyable from a pinned main SHA) or by edits +# to files that have drifted? +# +# Clone aiter main at the MegaMoE merge (97d0c6e4, which is a descendant of both +# #4382 and #4269) and diff the feature-relevant files against the base image. +set -u +W=/tmp/aitermain +if [ ! -d "$W/.git" ]; then + git clone -q --filter=blob:none https://github.com/ROCm/aiter "$W" || exit 1 +fi +cd "$W" || exit 1 +git fetch -q origin 97d0c6e4cb7a0919c12291c7c7d560ad412f15c1 2>/dev/null +git checkout -q 97d0c6e4cb7a0919c12291c7c7d560ad412f15c1 || exit 1 +echo "aiter main @ $(git rev-parse --short HEAD) version=$(cat aiter/_version.py 2>/dev/null)" + +T=/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter +echo +echo "=== how many aiter/*.py differ between v0.1.19 (base image) and this SHA?" +cd "$W/aiter" || exit 1 +find . -name '*.py' | sort > /tmp/main_py.txt +cd "$T" || exit 1 +find . -name '*.py' | sort > /tmp/base_py.txt +echo " main-only files: $(comm -13 /tmp/base_py.txt /tmp/main_py.txt | wc -l)" +echo " base-only files: $(comm -23 /tmp/base_py.txt /tmp/main_py.txt | wc -l)" +common=$(comm -12 /tmp/base_py.txt /tmp/main_py.txt | wc -l) +echo " common files: $common" +changed=0 +while IFS= read -r f; do + cmp -s "$T/$f" "$W/aiter/$f" || changed=$((changed+1)) +done < <(comm -12 /tmp/base_py.txt /tmp/main_py.txt) +echo " common but differing: $changed" + +echo +echo "=== non-Python that would need a rebuild (csrc / hsa / .cu / .cpp) ===" +cd "$W" || exit 1 +git diff --stat v0.1.19..HEAD -- csrc hsa '*.cu' '*.cpp' '*.hpp' '*.cuh' 2>/dev/null | tail -3 diff --git a/.probe_apply_aiter.sh b/.probe_apply_aiter.sh new file mode 100644 index 000000000..70d0834c3 --- /dev/null +++ b/.probe_apply_aiter.sh @@ -0,0 +1,30 @@ +#!/usr/bin/env bash +# Decisive experiment: can the five upstream aiter PRs be applied, as published +# diffs, straight onto the BASE image's installed aiter (v0.1.19)? The installed +# layout has no op_tests/ or .github/, so restrict to the aiter/ package. +# If this works the container patch can fetch-and-apply instead of embedding +# ~1.5 MB of heredoc. +set -u +SRC=/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter +W=/tmp/aitertest +rm -rf "$W" +mkdir -p "$W" +cp -a "$SRC" "$W/aiter" +cd "$W" || exit 1 +git init -q . +git add -A -f >/dev/null 2>&1 +git -c user.email=x@y -c user.name=x commit -qm base >/dev/null 2>&1 + +for n in 4269 4382 4439 4664 4673; do + d=/tmp/dsv4patch/aiter-$n.diff + echo "=== aiter#$n" + if git apply --include='aiter/*' -p1 --check "$d" 2>&1 | head -20; then + echo " CHECK CLEAN" + git apply --include='aiter/*' -p1 "$d" && echo " APPLIED" + git add -A >/dev/null 2>&1 + git -c user.email=x@y -c user.name=x commit -qm "pr$n" >/dev/null 2>&1 + else + echo " CHECK FAILED -> trying 3-way / fuzz" + patch -p1 --dry-run --fuzz=3 --forward < "$d" 2>&1 | grep -iv '^patching file aiter/' | head -30 + fi +done diff --git a/.probe_apply_vllm.sh b/.probe_apply_vllm.sh new file mode 100644 index 000000000..abb144f63 --- /dev/null +++ b/.probe_apply_vllm.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# vLLM side: do the five PRs apply, as published diffs, onto the base image's +# installed vllm package tree (b22afe45)? Restrict to vllm/ -- the installed +# layout has no tests/ or docs/. +set -u +SRC=/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/vllm +W=/tmp/vllmtest +rm -rf "$W"; mkdir -p "$W" +cp -a "$SRC" "$W/vllm" +cd "$W" || exit 1 +git init -q . +git add -A -f >/dev/null 2>&1 +git -c user.email=x@y -c user.name=x commit -qm base >/dev/null 2>&1 + +for n in 51473 51714 51713 51918 48728; do + d=/tmp/dsv4patch/vllm-$n.diff + echo "=== vllm#$n" + out=$(git apply --include='vllm/*' -p1 --check "$d" 2>&1) + if [ -z "$out" ]; then + echo " CLEAN" + git apply --include='vllm/*' -p1 "$d" + git add -A >/dev/null 2>&1 + git -c user.email=x@y -c user.name=x commit -qm "pr$n" >/dev/null 2>&1 + else + echo "$out" | sed 's/^/ /' | head -20 + echo " -- retry with 3-way + fuzz --" + patch -p1 --dry-run --fuzz=3 --forward -s < "$d" 2>&1 | head -20 + fi +done diff --git a/.probe_apply_vllm2.sh b/.probe_apply_vllm2.sh new file mode 100644 index 000000000..a6a92ef13 --- /dev/null +++ b/.probe_apply_vllm2.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +# The sequential run confounds PR-vs-base conflicts with PR-vs-PR conflicts. +# Re-test each PR ALONE on a pristine base, then test the two amd/model.py +# PRs (#51918, #48728) stacked in each order to see which way round works. +set -u +SRC=/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/vllm + +fresh() { + rm -rf /tmp/vt; mkdir -p /tmp/vt + cp -a "$SRC" /tmp/vt/vllm + cd /tmp/vt || exit 1 + git init -q .; git add -A -f >/dev/null 2>&1 + git -c user.email=x@y -c user.name=x commit -qm base >/dev/null 2>&1 +} + +try() { # $1=pr + local out + out=$(git apply --include='vllm/*' -p1 --check "/tmp/dsv4patch/vllm-$1.diff" 2>&1) + if [ -z "$out" ]; then + echo " #$1 CLEAN" + git apply --include='vllm/*' -p1 "/tmp/dsv4patch/vllm-$1.diff" + return 0 + fi + echo " #$1 CONFLICT:"; echo "$out" | sed 's/^/ /' + return 1 +} + +echo "=== each PR alone on pristine b22afe45 ===" +for n in 51473 51714 51713 51918 48728; do + fresh + out=$(git apply --include='vllm/*' -p1 --check "/tmp/dsv4patch/vllm-$n.diff" 2>&1) + if [ -z "$out" ]; then echo " #$n CLEAN"; else echo " #$n CONFLICT:"; echo "$out" | sed 's/^/ /'; fi +done + +echo +echo "=== #48728 then #51918 ===" +fresh; try 48728; try 51918 + +echo +echo "=== #51918 then #48728 ===" +fresh; try 51918; try 48728 diff --git a/.probe_attr2.sh b/.probe_attr2.sh new file mode 100644 index 000000000..35f529f66 --- /dev/null +++ b/.probe_attr2.sh @@ -0,0 +1,30 @@ +#!/usr/bin/env bash +# Attribute the still-unattributed vendor hunks: cudagraph_warmup_context, +# the tgemm call sites (#51713), and the #51473 MXFP4 TP8 shard. For each, +# print which vendor files carry it and whether the base image (target) has it. +set -u +D=/home/jiacao/3way-20260812-2214 +R="$D/ref/usr/local/lib/python3.12/dist-packages/vllm" +V="$D/vendor/src/vllm/vllm" +T="$D/target/usr/local/lib/python3.12/dist-packages/vllm" + +show() { # $1=marker + echo "### $1" + echo " vendor files:" + grep -rl -- "$1" "$V" --include='*.py' 2>/dev/null | sed "s|$V/| |" + echo " target files:" + grep -rl -- "$1" "$T" --include='*.py' 2>/dev/null | sed "s|$T/| |" + echo " ref files:" + grep -rl -- "$1" "$R" --include='*.py' 2>/dev/null | sed "s|$R/| |" + echo +} + +show "cudagraph_warmup_context" +show "tgemm.mm" +show "AITER_MXFP4_BF16" + +echo "### #51473: which MoEActivation branch guards AITER_MXFP4_BF16" +for t in "$T" "$V"; do + echo "--- ${t##*/dist-packages/}${t##*/src/vllm/}" + grep -rn -B4 -A4 "AITER_MXFP4_BF16" "$t" --include='*.py' 2>/dev/null | head -40 +done diff --git a/.probe_breaking_callers.sh b/.probe_breaking_callers.sh new file mode 100644 index 000000000..9c64a5e08 --- /dev/null +++ b/.probe_breaking_callers.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# The 3 breaking signature changes are only safe if every caller is itself in +# the patch set. Find the real callers of each resigned function in post2, and +# check whether the vendor's copy of that caller passes the new argument. +set -u +python3 - <<'PY' +import os, re, subprocess +W = "/tmp/aitermain" +subprocess.run(["git", "-C", W, "checkout", "-q", "v0.1.19.post2"], check=True) +P = os.path.join(W, "aiter") +V = "/home/jiacao/3way-20260812-2214/vendor/usr/local/lib/python3.12/dist-packages/aiter" + +PATCHSET = { + "ops/flydsl/kernels/flydsl_dispatch_combine_intranode_op.py", + "ops/flydsl/kernels/flydsl_dispatch_combine_intranode_kernel.py", + "ops/flydsl/kernels/communication_ops_utils.py", + "ops/flydsl/kernels/mxfp4_gemm_common.py", + "ops/flydsl/kernels/vector.py", +} +PATCHSET |= {"ops/flydsl/kernels/mega_moe/" + f for f in os.listdir(os.path.join(V, "ops/flydsl/kernels/mega_moe")) + if f.endswith(".py")} + +FUNCS = { + "atomic_add_global_at": "syncscope", + "make_combine_jit": "blockwise_fp8_transport", + "make_combine_kernel": "blockwise_fp8_transport", + "_lds_swizzle_mask": "row_bytes", +} + +for fn, newarg in FUNCS.items(): + print(f"=== {fn}() new arg: {newarg}") + callers = [] + for dp, _, fns in os.walk(P): + if "__pycache__" in dp: continue + for f in fns: + if not f.endswith(".py"): continue + p = os.path.join(dp, f) + rel = os.path.relpath(p, P) + src = open(p).read() + # a call site, not the definition + if re.search(r"(? /tmp/csv_target.txt +ls "$V/configs/model_configs/" > /tmp/csv_vendor.txt + +echo "target csv count: $(wc -l < /tmp/csv_target.txt)" +echo "vendor csv count: $(wc -l < /tmp/csv_vendor.txt)" +echo +echo "=== in VENDOR but not in TARGET (must be added) ===" +comm -13 /tmp/csv_target.txt /tmp/csv_vendor.txt | sed 's/^/ /' +echo +echo "=== in TARGET but not in VENDOR (target is newer here) ===" +comm -23 /tmp/csv_target.txt /tmp/csv_vendor.txt | sed 's/^/ /' +echo +echo "=== upstream availability of the vendor-only ones (aiter/main) ===" +comm -13 /tmp/csv_target.txt /tmp/csv_vendor.txt | while read -r f; do + [ -z "$f" ] && continue + s=$(gh api "/repos/ROCm/aiter/contents/aiter/configs/model_configs/$f?ref=main" --jq '.size' 2>/dev/null) + if [[ "$s" =~ ^[0-9]+$ ]]; then printf " %-9s %-9s %s\n" PRESENT "${s}B" "$f" + else printf " %-9s %-9s %s\n" ABSENT "-" "$f"; fi +done +echo +echo "=== dsv4 csvs: vendor line counts vs target ===" +for f in $(ls "$V/configs/model_configs/" | grep -i dsv4); do + lv=$(wc -l < "$V/configs/model_configs/$f") + if [ -f "$T/configs/model_configs/$f" ]; then + lt=$(wc -l < "$T/configs/model_configs/$f") + else + lt="" + fi + printf " %-52s vendor=%-6s target=%s\n" "$f" "$lv" "$lt" +done diff --git a/.probe_defaults_equiv.sh b/.probe_defaults_equiv.sh new file mode 100644 index 000000000..505259472 --- /dev/null +++ b/.probe_defaults_equiv.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +# A default only makes the change safe if the default RESTORES the old behavior. +# Print post2's body next to the vendor's body for each of the four functions, +# so the equivalence at the default value can be read directly. +set -u +python3 - <<'PY' +import ast, os, subprocess +W = "/tmp/aitermain" +subprocess.run(["git", "-C", W, "checkout", "-q", "v0.1.19.post2"], check=True) +P = os.path.join(W, "aiter") +V = "/home/jiacao/3way-20260812-2214/vendor/usr/local/lib/python3.12/dist-packages/aiter" + +TARGETS = [ + ("ops/flydsl/kernels/communication_ops_utils.py", "atomic_add_global_at"), + ("ops/flydsl/kernels/mxfp4_gemm_common.py", "_lds_swizzle_mask"), +] + +def body(path, name): + src = open(path).read() + for n in ast.walk(ast.parse(src)): + if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef)) and n.name == name: + return ast.get_source_segment(src, n) + return None + +for rel, fn in TARGETS: + print("#" * 70) + print(f"### {fn} ({rel})") + print("--- post2 (base image would have this)") + print(body(os.path.join(P, rel), fn)) + print("--- vendor (measurement image)") + print(body(os.path.join(V, rel), fn)) + print() +PY diff --git a/.probe_drift_direction.sh b/.probe_drift_direction.sh new file mode 100644 index 000000000..a43126a72 --- /dev/null +++ b/.probe_drift_direction.sh @@ -0,0 +1,116 @@ +#!/usr/bin/env bash +# Two questions before committing to a patch shape: +# (a) the 5 drifted flydsl files MegaMoE needs -- is the base->main drift purely +# additive? tensor_shim has 35 importers in the base image, so replacing it +# wholesale is only safe if nothing is removed or re-signatured. +# (b) FSE (#4269): what is its transitive file closure, same method as MegaMoE? +set -u +W=/tmp/aitermain/aiter +T=/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter + +echo "=== (a) drift direction on the 5 shared flydsl files ===" +python3 - <<'PY' +import ast, os +W = "/tmp/aitermain/aiter" +T = "/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter" +FILES = ["ops/flydsl/kernels/flydsl_dispatch_combine_intranode_op.py", + "ops/flydsl/kernels/flydsl_dispatch_combine_intranode_kernel.py", + "ops/flydsl/kernels/tensor_shim.py", + "ops/flydsl/kernels/mxfp4_gemm_common.py", + "ops/flydsl/kernels/communication_ops_utils.py"] + +def api(path): + """Top-level public names and, for functions, their arg lists.""" + out = {} + for n in ast.parse(open(path).read()).body: + if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef)): + a = n.args + sig = [x.arg for x in a.posonlyargs + a.args] + \ + (["*" + a.vararg.arg] if a.vararg else []) + \ + [x.arg for x in a.kwonlyargs] + \ + (["**" + a.kwarg.arg] if a.kwarg else []) + out[n.name] = tuple(sig) + elif isinstance(n, ast.ClassDef): + out[n.name] = ("",) + elif isinstance(n, ast.Assign): + for t in n.targets: + if isinstance(t, ast.Name): + out[t.id] = ("",) + return out + +for f in FILES: + b, m = api(os.path.join(T, f)), api(os.path.join(W, f)) + removed = sorted(set(b) - set(m)) + changed = sorted(k for k in set(b) & set(m) if b[k] != m[k]) + added = sorted(set(m) - set(b)) + verdict = "ADDITIVE-SAFE" if not removed and not changed else "BREAKING" + print(f" {verdict:14} {os.path.basename(f)} +{len(added)} -{len(removed)} ~{len(changed)}") + for k in removed: print(f" removed: {k}") + for k in changed: print(f" resigned: {k} {b[k]} -> {m[k]}") +PY + +echo +echo "=== (b) FSE transitive closure (#4269) ===" +python3 - <<'PY' +import ast, os, collections, difflib +W = "/tmp/aitermain/aiter" +T = "/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter" + +def find(root, rel): + for s in (".py", "/__init__.py"): + if os.path.exists(os.path.join(root, rel + s)): + return rel + s + return None + +def imports_of(path, self_rel): + pkg = os.path.dirname(self_rel); out = set() + for n in ast.walk(ast.parse(open(path).read())): + if isinstance(n, ast.ImportFrom): + if n.level: + base = pkg + for _ in range(n.level - 1): base = os.path.dirname(base) + mod = os.path.join(base, (n.module or "").replace(".", "/")).rstrip("/") + elif n.module and n.module.startswith("aiter"): + mod = n.module[len("aiter"):].lstrip(".").replace(".", "/") + else: + continue + out.add(mod) + for a in n.names: out.add(os.path.join(mod, a.name)) + elif isinstance(n, ast.Import): + for a in n.names: + if a.name.startswith("aiter"): + out.add(a.name[len("aiter"):].lstrip(".").replace(".", "/")) + return {m for m in out if m} + +seed = ["fhmoe.py", "ops/flydsl/fhmoe.py", "ops/flydsl/kernels/fhmoe.py", + "aot/flydsl/fhmoe.py"] +seen, q = set(seed), collections.deque(seed) +new, drift = set(), {} +while q: + f = q.popleft() + p = os.path.join(W, f) + if not os.path.exists(p): continue + if find(T, f[:-3] if f.endswith(".py") else f) is None: + new.add(f) + for m in imports_of(p, f): + rm = find(W, m) + if rm is None or rm in seen: continue + seen.add(rm) + rb = find(T, m) + if rb is None: + new.add(rm); q.append(rm) + else: + a, b = open(os.path.join(T, rb)).readlines(), open(os.path.join(W, rm)).readlines() + if a != b: + drift[rm] = sum(1 for l in difflib.unified_diff(a, b, n=0) + if l[:1] in "+-" and l[:3] not in ("---", "+++")) + q.append(rm) + +print(f" NEW files: {len(new)}") +for f in sorted(new): + n = sum(1 for _ in open(os.path.join(W, f))) if os.path.exists(os.path.join(W, f)) else 0 + print(f" {n:6} {f}") +print(f" DRIFTED existing files: {len(drift)}") +for f, d in sorted(drift.items(), key=lambda kv: -kv[1])[:15]: + print(f" {d:6} lines {f}") +PY diff --git a/.probe_dsv4_csv.sh b/.probe_dsv4_csv.sh new file mode 100644 index 000000000..19d6a58ae --- /dev/null +++ b/.probe_dsv4_csv.sh @@ -0,0 +1,27 @@ +#!/usr/bin/env bash +# The imgdiff run flagged 'configs/model_configs/a8w8_blockscale_tuned_gemm_dsv4.csv' +# as +46 lines in the vendor tree. Tuned-GEMM configs are data, not code: if the +# dsv4 csv is vendor-only, the tuned shapes for this model simply do not exist +# in the nightly line and the GEMMs fall back to untuned dispatch. +set -u +D="${1:-/home/jiacao/3way-20260812-2214}" + +for side in ref vendor target; do + A=$(find "$D/$side" -maxdepth 9 -type d -name aiter -path '*dist-packages*' 2>/dev/null | head -1) + echo "=== $side" + [ -z "$A" ] && { echo " "; continue; } + echo " -- dsv4 csv files --" + ls "$A/configs/model_configs/" 2>/dev/null | grep -i dsv4 | sed 's/^/ /' || true + n=$(ls "$A/configs/model_configs/" 2>/dev/null | grep -ci dsv4) + echo " count: $n" + f="$A/configs/model_configs/a8w8_blockscale_tuned_gemm_dsv4.csv" + if [ -f "$f" ]; then + echo " a8w8_blockscale_tuned_gemm_dsv4.csv: $(wc -l < "$f") lines" + fi + # gluon / sparse attention entry points, by name rather than by path + echo " -- gluon sparse attention --" + for pat in "pa_decode_sparse" "_gluon_kernels" "mla_gluon"; do + c=$(grep -rl -- "$pat" "$A" --include='*.py' 2>/dev/null | wc -l) + printf " %-20s %s file(s)\n" "$pat" "$c" + done +done diff --git a/.probe_final_markers.sh b/.probe_final_markers.sh new file mode 100644 index 000000000..eb4c3f5de --- /dev/null +++ b/.probe_final_markers.sh @@ -0,0 +1,53 @@ +#!/usr/bin/env bash +# Final marker sweep: the precise identifiers each vendor hunk introduces, +# checked against the target tree. Anything ABSENT in target is a real addition +# we must carry; anything PRESENT means upstream already landed it in the 465 +# commits between ref and target and the hunk should be dropped, not forced. +set -u +D=/home/jiacao/3way-20260812-2214 +R="$D/ref/usr/local/lib/python3.12/dist-packages/vllm" +V="$D/vendor/src/vllm/vllm" +T="$D/target/usr/local/lib/python3.12/dist-packages/vllm" + +chk() { # $1=marker $2=feature-label + local m="$1" lbl="$2" + local r v t + r=$(grep -rl -- "$m" "$R" --include='*.py' 2>/dev/null | wc -l) + v=$(grep -rl -- "$m" "$V" --include='*.py' 2>/dev/null | wc -l) + t=$(grep -rl -- "$m" "$T" --include='*.py' 2>/dev/null | wc -l) + local verdict + if [ "$v" -eq 0 ]; then verdict="n/a" + elif [ "$t" -gt 0 ] && [ "$r" -gt 0 ]; then verdict="pre-existing" + elif [ "$t" -gt 0 ]; then verdict="UPSTREAM-LANDED" + else verdict="MUST-ADD" + fi + printf " r=%-3s v=%-3s t=%-3s %-16s %-46s %s\n" "$r" "$v" "$t" "$verdict" "$m" "$lbl" +} + +echo "=== MegaMoE ===" +chk "flydsl_mega_moe" "kernel-backend enum value" +chk "use_mega_moe" "layer flag" +chk "mega_moe_experts" "new module" +chk "mega_moe_runtime" "new module" +chk "finalize_mega_moe_layers" "post-load hook" +chk "make_deepseek_v4_mega_expert_params_mapping" "weight mapping" + +echo "=== Gluon sparse attention ===" +chk "VLLM_ROCM_DSV4_SPARSE_GLUON" "env knob" +chk "pa_decode_sparse" "aiter entry point" + +echo "=== FSE / heterogeneous shared expert ===" +chk "shared_expert_id" "fused_moe kwarg" +chk "fused_moe_supports_heterogeneous_shared_expert" "capability probe" + +echo "=== tuned GEMM (DSv4 attention projections) ===" +chk "from aiter.tuned_gemm import tgemm" "tgemm import" + +echo "=== misc plumbing the above depend on ===" +chk "cudagraph_warmup_context" "platform hook" +chk "gemm_a8w8_blockscale_bpreshuffle" "bpreshuffle GEMM op" +chk "aiter_per1x128_quant" "per-1x128 quant" +chk "transpose_scale" "quant kwarg" +chk "process_weights_after_loading()" "model-level post-load hook" +chk "_fused_wqa_wkv_gemm" "attention refactor hook" +chk "swiglu_limit" "swiglu limit passthrough" diff --git a/.probe_gluon_build.sh b/.probe_gluon_build.sh new file mode 100644 index 000000000..b43f4975d --- /dev/null +++ b/.probe_gluon_build.sh @@ -0,0 +1,75 @@ +#!/usr/bin/env bash +# Build the candidate gluon patch for real: take the two files from aiter main +# (#4382's descendant, byte-identical to the measurement image), apply #4673 on +# top, drop them onto a copy of the base image's aiter, and check that every +# name the routing file imports actually resolves. This is the go/no-go for the +# gluon arm being deliverable as a container patch. +set -u +W=/tmp/aitermain/aiter +T=/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter +S=/tmp/gluonstage +rm -rf "$S"; mkdir -p "$S" +cp -a "$T" "$S/aiter" + +mkdir -p "$S/aiter/ops/triton/_gluon_kernels/gfx950/attention" +cp "$W/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py" \ + "$S/aiter/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py" +[ -f "$W/ops/triton/_gluon_kernels/gfx950/attention/__init__.py" ] && \ + cp "$W/ops/triton/_gluon_kernels/gfx950/attention/__init__.py" \ + "$S/aiter/ops/triton/_gluon_kernels/gfx950/attention/__init__.py" +cp "$W/ops/triton/attention/pa_decode_sparse.py" "$S/aiter/ops/triton/attention/pa_decode_sparse.py" + +echo "=== apply #4673 on top (aiter/ only) ===" +cd "$S" || exit 1 +git init -q .; git add -A -f >/dev/null 2>&1 +git -c user.email=x@y -c user.name=x commit -qm stage >/dev/null 2>&1 +out=$(git apply --include='aiter/*' -p1 --check /tmp/dsv4patch/aiter-4673.diff 2>&1) +if [ -z "$out" ]; then + git apply --include='aiter/*' -p1 /tmp/dsv4patch/aiter-4673.diff && echo " #4673 APPLIED CLEAN" +else + echo " #4673 CONFLICT:"; echo "$out" | sed 's/^/ /' +fi + +echo +echo "=== does gfx950/attention have an __init__.py in main? ===" +ls "$W/ops/triton/_gluon_kernels/gfx950/attention/" | sed 's/^/ /' +echo " gfx950/ dir:"; ls "$W/ops/triton/_gluon_kernels/gfx950/" | sed 's/^/ /' + +echo +echo "=== resolve every name the staged routing file imports ===" +python3 - <<'PY' +import ast, os +S = "/tmp/gluonstage/aiter" +FILES = ["ops/triton/attention/pa_decode_sparse.py", + "ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py", + "ops/triton/utils/common_utils.py"] +def find(rel): + for s in (".py", "/__init__.py"): + if os.path.exists(os.path.join(S, rel + s)): return rel + s + return None +bad = 0 +for f in FILES: + for n in ast.walk(ast.parse(open(os.path.join(S, f)).read())): + if isinstance(n, ast.ImportFrom) and n.module and n.module.startswith("aiter"): + rel = n.module[len("aiter"):].lstrip(".").replace(".", "/") + tgt = find(rel) + if tgt is None: + print(f" MODULE-ABSENT {n.module} (from {f})"); bad += 1; continue + text = open(os.path.join(S, tgt)).read() + for a in n.names: + if a.name != "*" and a.name not in text: + print(f" SYMBOL-ABSENT {n.module}.{a.name} (from {f})"); bad += 1 +if not bad: + print(" all aiter imports resolve") +PY + +echo +echo "=== syntax check ===" +python3 -m py_compile \ + "$S/aiter/ops/triton/attention/pa_decode_sparse.py" \ + "$S/aiter/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py" \ + "$S/aiter/ops/triton/utils/common_utils.py" && echo " OK" + +echo +echo "=== is max_addressable_bytes now wired into the decision? ===" +grep -n "max_addressable_bytes\|use_buffer_load" "$S/aiter/ops/triton/attention/pa_decode_sparse.py" | head diff --git a/.probe_gluon_drift.sh b/.probe_gluon_drift.sh new file mode 100644 index 000000000..c3cef5f8f --- /dev/null +++ b/.probe_gluon_drift.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +# arch_info is a submodule, not a symbol -- the earlier MODULE-ABSENT was a +# false positive. So the gluon surface resolves. Remaining question: how much of +# the base->main delta on the two shared files is #4382/#4673, and how much is +# unrelated drift we'd be dragging in by copying main's version wholesale? +set -u +W=/tmp/aitermain/aiter +T=/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter + +echo "=== base(v0.1.19) -> main(97d0c6e4) : ops/triton/attention/pa_decode_sparse.py" +diff -u "$T/ops/triton/attention/pa_decode_sparse.py" "$W/ops/triton/attention/pa_decode_sparse.py" | diffstat 2>/dev/null \ + || diff -u "$T/ops/triton/attention/pa_decode_sparse.py" "$W/ops/triton/attention/pa_decode_sparse.py" | grep -c '^[+-]' + +echo +echo "=== the same file, but versus the vendor (measurement) image ===" +V=/home/jiacao/3way-20260812-2214/vendor/usr/local/lib/python3.12/dist-packages/aiter +if cmp -s "$V/ops/triton/attention/pa_decode_sparse.py" "$W/ops/triton/attention/pa_decode_sparse.py"; then + echo " vendor == main : the measurement image runs stock upstream here" +else + echo " vendor != main, delta:" + diff -u "$V/ops/triton/attention/pa_decode_sparse.py" "$W/ops/triton/attention/pa_decode_sparse.py" | head -40 +fi + +echo +echo "=== gluon kernel file: vendor vs main ===" +if cmp -s "$V/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py" \ + "$W/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py"; then + echo " vendor == main" +else + diff -u "$V/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py" \ + "$W/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py" | grep -c '^[+-]' +fi + +echo +echo "=== common_utils.py: does main already carry #4673? ===" +grep -n "max_addressable_bytes" "$W/ops/triton/utils/common_utils.py" | head +echo " (empty above = #4673 not merged yet, must come from the PR)" diff --git a/.probe_gluon_only.sh b/.probe_gluon_only.sh new file mode 100644 index 000000000..fbdbfd9bc --- /dev/null +++ b/.probe_gluon_only.sh @@ -0,0 +1,80 @@ +#!/usr/bin/env bash +# Narrow the question to the highest-value, smallest patch: the gluon sparse +# decode path (#4382 + #4673) alone. Its aiter surface is 3 files. Check every +# symbol it pulls from the DRIFTED modules and confirm the base image's copies +# already define them -- if so, the gluon patch is a pure 3-file drop-in and +# needs no aiter sync at all. +set -u +W=/tmp/aitermain/aiter +T=/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter + +FILES="ops/triton/attention/pa_decode_sparse.py ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py ops/triton/utils/common_utils.py" + +echo "=== imported symbols and where they resolve ===" +for f in $FILES; do + echo "--- $f" + grep -nE '^\s*(from|import)\s+' "$W/$f" | sed 's/^/ /' +done + +echo +echo "=== symbol-by-symbol availability in the base image ===" +python3 - <<'PY' +import ast, os, sys +W = "/tmp/aitermain/aiter" +T = "/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter" +FILES = [ + "ops/triton/attention/pa_decode_sparse.py", + "ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py", + "ops/triton/utils/common_utils.py", +] +new_files = set(FILES) + +def defined_names(path): + try: + tree = ast.parse(open(path).read()) + except Exception: + return None + names = set() + for n in tree.body: + if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)): + names.add(n.name) + elif isinstance(n, ast.Assign): + for t in n.targets: + if isinstance(t, ast.Name): + names.add(t.id) + elif isinstance(n, ast.AnnAssign) and isinstance(n.target, ast.Name): + names.add(n.target.id) + return names + +missing = [] +for f in FILES: + tree = ast.parse(open(os.path.join(W, f)).read()) + for n in ast.walk(tree): + if not isinstance(n, ast.ImportFrom) or not n.module: + continue + if not n.module.startswith("aiter"): + continue + rel = n.module[len("aiter"):].lstrip(".").replace(".", "/") + if rel + ".py" in new_files: + continue # provided by the patch itself + cand = os.path.join(T, rel + ".py") + if not os.path.exists(cand): + cand = os.path.join(T, rel, "__init__.py") + if not os.path.exists(cand): + for a in n.names: + missing.append((f, n.module, a.name, "MODULE-ABSENT")) + continue + have = defined_names(cand) or set() + # names re-exported through __init__ are hard to see statically; also + # accept a textual hit anywhere in the file. + text = open(cand).read() + for a in n.names: + if a.name not in have and a.name not in text: + missing.append((f, n.module, a.name, "SYMBOL-ABSENT")) + +if missing: + for f, m, s, why in missing: + print(f" {why:14} {m}.{s} (needed by {f})") +else: + print(" all imported aiter symbols resolve against the base image") +PY diff --git a/.probe_gluon_seed_fix.sh b/.probe_gluon_seed_fix.sh new file mode 100644 index 000000000..c031244e2 --- /dev/null +++ b/.probe_gluon_seed_fix.sh @@ -0,0 +1,42 @@ +#!/usr/bin/env bash +# My closure script only measured drift on IMPORTED modules, never on the seed +# files themselves -- so the "gluon needs 0 existing changes" line understates +# it. Measure the seed files directly, plus common_utils.py which #4673 touches. +set -u +python3 - <<'PY' +import os, subprocess, difflib +W = "/tmp/aitermain" +subprocess.run(["git", "-C", W, "checkout", "-q", "v0.1.19.post2"], check=True) +P = os.path.join(W, "aiter") +V = "/home/jiacao/3way-20260812-2214/vendor/usr/local/lib/python3.12/dist-packages/aiter" + +FILES = [ + "ops/triton/attention/pa_decode_sparse.py", # routing (#4382) + "ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py", # kernel (#4382) + "ops/triton/utils/common_utils.py", # #4673 +] +for f in FILES: + pp, vp = os.path.join(P, f), os.path.join(V, f) + if not os.path.exists(pp): + n = sum(1 for _ in open(vp)) + print(f" NEW FILE {n:5} lines {f}") + continue + a, b = open(pp).readlines(), open(vp).readlines() + if a == b: + print(f" IDENTICAL {f}") + continue + d = sum(1 for l in difflib.unified_diff(a, b, n=0) + if l[:1] in "+-" and l[:3] not in ("---", "+++")) + print(f" MODIFIED {d:5} lines {f} (post2 {len(a)} -> vendor {len(b)})") + +print() +print("=== does the vendor's common_utils.py carry #4673's max_addressable_bytes? ===") +src = open(os.path.join(V, "ops/triton/utils/common_utils.py")).read() +print(" max_addressable_bytes in vendor:", "YES" if "max_addressable_bytes" in src else "NO") +src2 = open(os.path.join(V, "ops/triton/attention/pa_decode_sparse.py")).read() +print(" max_addressable_bytes used by vendor routing:", + "YES" if "max_addressable_bytes" in src2 else "NO") +import re +m = re.search(r"use_buffer_load\s*=.*", src2) +print(" vendor's use_buffer_load decision:", m.group(0).strip() if m else "(not found)") +PY diff --git a/.probe_import_closure.sh b/.probe_import_closure.sh new file mode 100644 index 000000000..3da9db840 --- /dev/null +++ b/.probe_import_closure.sh @@ -0,0 +1,52 @@ +#!/usr/bin/env bash +# 444/538 shared aiter files drift between v0.1.19 and the MegaMoE merge, so a +# wholesale sync is out. The narrow question instead: if we copy ONLY the +# feature-carrying files from main onto the base image, what do they import that +# the base image does not have? That set is the true blast radius. +set -u +W=/tmp/aitermain/aiter +T=/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter + +FEATURE_FILES=" +ops/flydsl/kernels/mega_moe/__init__.py +ops/flydsl/kernels/mega_moe/dispatch.py +ops/flydsl/kernels/mega_moe/gemm1.py +ops/flydsl/kernels/mega_moe/gemm2.py +ops/flydsl/kernels/mega_moe/gemm_util.py +ops/flydsl/kernels/mega_moe/mega_moe_config.py +ops/flydsl/kernels/mega_moe/mega_moe_stage1.py +ops/flydsl/kernels/mega_moe/mega_moe_stage2.py +ops/flydsl/kernels/mega_moe/mega_moe_v2.py +ops/flydsl/kernels/mega_moe/quant.py +ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py +ops/triton/attention/pa_decode_sparse.py +ops/triton/utils/common_utils.py +fhmoe.py +ops/flydsl/fhmoe.py +ops/flydsl/kernels/fhmoe.py +" + +echo "=== what these files import from aiter, and whether the base image has it ===" +for f in $FEATURE_FILES; do + [ -f "$W/$f" ] || { echo " !! not in main: $f"; continue; } + grep -hoE '^\s*(from|import)\s+aiter[.a-zA-Z0-9_]*' "$W/$f" +done | sed -E 's/^\s*(from|import)\s+//' | sort -u > /tmp/imports.txt + +while IFS= read -r mod; do + rel="${mod#aiter}"; rel="${rel#.}"; rel="${rel//./\/}" + if [ -z "$rel" ]; then continue; fi + if [ -f "$T/$rel.py" ] || [ -d "$T/$rel" ]; then + # exists -- but does it differ from main? + if [ -f "$T/$rel.py" ] && [ -f "$W/$rel.py" ] && ! cmp -s "$T/$rel.py" "$W/$rel.py"; then + echo " DRIFTED aiter.$(echo "$rel" | tr '/' '.')" + else + echo " ok aiter.$(echo "$rel" | tr '/' '.')" + fi + else + echo " MISSING aiter.$(echo "$rel" | tr '/' '.')" + fi +done < /tmp/imports.txt + +echo +echo "=== do the FlyDSL kernels depend on the compiled csrc that changed? ===" +grep -l "import aiter.jit\|from aiter.jit\|compile_ops\|get_module" $(for f in $FEATURE_FILES; do [ -f "$W/$f" ] && echo "$W/$f"; done) 2>/dev/null | sed "s|$W/| |" diff --git a/.probe_megamoe_closure.sh b/.probe_megamoe_closure.sh new file mode 100644 index 000000000..6adb8d3c8 --- /dev/null +++ b/.probe_megamoe_closure.sh @@ -0,0 +1,70 @@ +#!/usr/bin/env bash +# Gluon is a clean drop-in (vendor == main on both files). Now the same test for +# MegaMoE: resolve every symbol the 10 mega_moe files import from aiter against +# the BASE image. Anything absent means MegaMoE cannot be delivered as a file +# copy and needs an aiter sync instead. +set -u +python3 - <<'PY' +import ast, os +W = "/tmp/aitermain/aiter" +T = "/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter" +V = "/home/jiacao/3way-20260812-2214/vendor/usr/local/lib/python3.12/dist-packages/aiter" + +MEGA = ["ops/flydsl/kernels/mega_moe/" + f for f in + ["__init__.py","dispatch.py","gemm1.py","gemm2.py","gemm_util.py", + "mega_moe_config.py","mega_moe_stage1.py","mega_moe_stage2.py", + "mega_moe_v2.py","quant.py"]] +new = set(MEGA) + +def resolve(root, mod): + rel = mod[len("aiter"):].lstrip(".").replace(".", "/") + for c in (os.path.join(root, rel + ".py"), os.path.join(root, rel, "__init__.py")): + if os.path.exists(c): + return c + return None + +problems, ok = [], 0 +for f in MEGA: + p = os.path.join(W, f) + if not os.path.exists(p): + problems.append(("FILE-ABSENT-IN-MAIN", f, "")); continue + for n in ast.walk(ast.parse(open(p).read())): + if isinstance(n, ast.ImportFrom) and n.module and n.module.startswith("aiter"): + rel = n.module[len("aiter"):].lstrip(".").replace(".", "/") + if rel + ".py" in new or rel + "/__init__.py" in new: + continue + tgt = resolve(T, n.module) + if tgt is None: + problems.append(("MODULE-ABSENT", n.module, f)); continue + text = open(tgt).read() + for a in n.names: + if a.name == "*": + continue + if a.name not in text: + problems.append(("SYMBOL-ABSENT", f"{n.module}.{a.name}", f)) + else: + ok += 1 + elif isinstance(n, ast.Import): + for a in n.names: + if a.name.startswith("aiter") and resolve(T, a.name) is None: + problems.append(("MODULE-ABSENT", a.name, f)) + +seen = set() +for why, what, where in problems: + k = (why, what) + if k in seen: continue + seen.add(k) + print(f" {why:16} {what}") +print(f" ({ok} imported symbols resolved cleanly)") + +print() +print("=== mega_moe: vendor vs main, file by file ===") +import filecmp +for f in MEGA: + v, w = os.path.join(V, f), os.path.join(W, f) + if not os.path.exists(v): print(f" vendor-absent {os.path.basename(f)}"); continue + if not os.path.exists(w): print(f" main-absent {os.path.basename(f)}"); continue + same = filecmp.cmp(v, w, shallow=False) + lv, lw = sum(1 for _ in open(v)), sum(1 for _ in open(w)) + print(f" {'identical' if same else 'DIFFERS ':14} {os.path.basename(f):24} vendor={lv:5} main={lw:5}") +PY diff --git a/.probe_megamoe_closure2.sh b/.probe_megamoe_closure2.sh new file mode 100644 index 000000000..76aa92e14 --- /dev/null +++ b/.probe_megamoe_closure2.sh @@ -0,0 +1,71 @@ +#!/usr/bin/env bash +# Transitive closure: start from the 10 mega_moe files, add buffer_ops.py, and +# keep pulling in every aiter module they import that the base image lacks, +# until the set stops growing. That closure IS the MegaMoE file-copy manifest. +set -u +python3 - <<'PY' +import ast, os, collections +W = "/tmp/aitermain/aiter" +T = "/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter" + +def relpath_for(mod): + return mod[len("aiter"):].lstrip(".").replace(".", "/") + +def find(root, mod): + rel = relpath_for(mod) + for suffix in (".py", "/__init__.py"): + c = os.path.join(root, rel + suffix) + if os.path.exists(c): + return rel + suffix + return None + +seed = ["ops/flydsl/kernels/mega_moe/" + f for f in + ["__init__.py","dispatch.py","gemm1.py","gemm2.py","gemm_util.py", + "mega_moe_config.py","mega_moe_stage1.py","mega_moe_stage2.py", + "mega_moe_v2.py","quant.py"]] + +closure = set(seed) +queue = collections.deque(seed) +drifted = set() +while queue: + f = queue.popleft() + p = os.path.join(W, f) + if not os.path.exists(p): + continue + mods = set() + for n in ast.walk(ast.parse(open(p).read())): + if isinstance(n, ast.ImportFrom) and n.module and n.module.startswith("aiter"): + mods.add(n.module) + for a in n.names: # `from pkg import submodule` form + mods.add(n.module + "." + a.name) + elif isinstance(n, ast.Import): + for a in n.names: + if a.name.startswith("aiter"): + mods.add(a.name) + for m in mods: + rel_main = find(W, m) + if rel_main is None: + continue + rel_base = find(T, m) + if rel_base is None: + if rel_main not in closure: + closure.add(rel_main); queue.append(rel_main) + else: + bp, wp = os.path.join(T, rel_base), os.path.join(W, rel_main) + if os.path.exists(bp) and os.path.exists(wp) and open(bp,'rb').read() != open(wp,'rb').read(): + drifted.add(rel_main) + +new = sorted(f for f in closure if find(T, "aiter." + f[:-3].replace("/", ".").removesuffix(".__init__")) is None or not os.path.exists(os.path.join(T, f))) +print(f"=== files to COPY from main (absent in base image): {len(new)}") +for f in new: + n = sum(1 for _ in open(os.path.join(W, f))) if os.path.exists(os.path.join(W, f)) else 0 + print(f" {n:5} {f}") +print() +print(f"=== modules the closure touches that EXIST in base but DRIFTED: {len(drifted)}") +for f in sorted(drifted): + bp, wp = os.path.join(T, f), os.path.join(W, f) + import difflib + a = open(bp).readlines(); b = open(wp).readlines() + d = sum(1 for l in difflib.unified_diff(a, b, n=0) if l[:1] in "+-" and l[:3] not in ("---","+++")) + print(f" {d:5} changed lines {f}") +PY diff --git a/.probe_megamoe_closure3.sh b/.probe_megamoe_closure3.sh new file mode 100644 index 000000000..df46526f7 --- /dev/null +++ b/.probe_megamoe_closure3.sh @@ -0,0 +1,101 @@ +#!/usr/bin/env bash +# The previous closure missed RELATIVE imports (`from .. import x`), which is +# exactly how mega_moe reaches communication_ops_utils and +# flydsl_dispatch_combine_intranode_op -- two files #4439 modifies in place. +# Redo the closure with relative-import resolution, and also report the external +# (non-aiter) package requirements: flydsl, mori. +set -u +python3 - <<'PY' +import ast, os, collections, difflib +W = "/tmp/aitermain/aiter" +T = "/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter" +V = "/home/jiacao/3way-20260812-2214/vendor/usr/local/lib/python3.12/dist-packages/aiter" + +def find(root, rel): + for s in (".py", "/__init__.py"): + if os.path.exists(os.path.join(root, rel + s)): + return rel + s + return None + +def imports_of(path, self_rel): + """Yield aiter-relative module paths (no extension) imported by `path`.""" + pkg = os.path.dirname(self_rel) + out = set() + for n in ast.walk(ast.parse(open(path).read())): + if isinstance(n, ast.ImportFrom): + if n.level: # relative + base = pkg + for _ in range(n.level - 1): + base = os.path.dirname(base) + mod = os.path.join(base, (n.module or "").replace(".", "/")).rstrip("/") + out.add(mod) + for a in n.names: + out.add(os.path.join(mod, a.name)) + elif n.module and n.module.startswith("aiter"): + mod = n.module[len("aiter"):].lstrip(".").replace(".", "/") + out.add(mod) + for a in n.names: + out.add(os.path.join(mod, a.name)) + elif isinstance(n, ast.Import): + for a in n.names: + if a.name.startswith("aiter"): + out.add(a.name[len("aiter"):].lstrip(".").replace(".", "/")) + return {m for m in out if m} + +seed = ["ops/flydsl/kernels/mega_moe/" + f for f in + ["__init__.py","dispatch.py","gemm1.py","gemm2.py","gemm_util.py", + "mega_moe_config.py","mega_moe_stage1.py","mega_moe_stage2.py", + "mega_moe_v2.py","quant.py"]] +seen = set(seed) +q = collections.deque(seed) +to_copy, drifted = set(seed), {} +while q: + f = q.popleft() + p = os.path.join(W, f) + if not os.path.exists(p): + continue + for m in imports_of(p, f): + rm = find(W, m) + if rm is None or rm in seen: + continue + seen.add(rm) + rb = find(T, m) + if rb is None: + to_copy.add(rm); q.append(rm) + else: + a, b = open(os.path.join(T, rb)).readlines(), open(os.path.join(W, rm)).readlines() + if a != b: + d = sum(1 for l in difflib.unified_diff(a, b, n=0) + if l[:1] in "+-" and l[:3] not in ("---", "+++")) + drifted[rm] = d + q.append(rm) + +print(f"=== NEW files to copy: {len(to_copy)}") +for f in sorted(to_copy): + n = sum(1 for _ in open(os.path.join(W, f))) if os.path.exists(os.path.join(W, f)) else 0 + print(f" {n:5} {f}") + +print() +print(f"=== EXISTING files the closure needs but which DRIFTED base->main: {len(drifted)}") +for f, d in sorted(drifted.items(), key=lambda kv: -kv[1]): + same_as_vendor = "" + vp, wp = os.path.join(V, f), os.path.join(W, f) + if os.path.exists(vp): + same_as_vendor = " (vendor==main)" if open(vp,'rb').read() == open(wp,'rb').read() else " (vendor!=main)" + print(f" {d:5} lines {f}{same_as_vendor}") + +print() +print("=== external (non-aiter) packages the closure requires ===") +ext = set() +for f in sorted(seen | to_copy): + p = os.path.join(W, f) + if not os.path.exists(p): continue + for n in ast.walk(ast.parse(open(p).read())): + if isinstance(n, ast.Import): + for a in n.names: + ext.add(a.name.split(".")[0]) + elif isinstance(n, ast.ImportFrom) and not n.level and n.module: + ext.add(n.module.split(".")[0]) +for e in sorted(ext - {"aiter"}): + print(f" {e}") +PY diff --git a/.probe_megamoe_match.sh b/.probe_megamoe_match.sh new file mode 100644 index 000000000..3879081aa --- /dev/null +++ b/.probe_megamoe_match.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +# Does the vendor image's MegaMoE tree match what aiter PR #4439 merged, or is +# the vendor carrying a private variant? Sizes differ enough between vendor and +# aiter/main that this needs checking at the merge commit itself (97d0c6e4cb7a), +# not at main -- main has 87 commits of drift past post2. +# +# If sizes match at 97d0c6e4, the port is a clean cherry-pick from upstream. +# If they don't, the vendor delta is real and needs its own PR. +set -u +REF="${1:-97d0c6e4cb7a}" +V=$(find /home/jiacao/3way-20260812-2214/vendor -maxdepth 9 -type d -name aiter -path '*dist-packages*' | head -1) + +printf "%-24s %10s %10s %10s %s\n" FILE VENDOR "@$REF" MAIN VERDICT +for f in __init__ dispatch gemm1 gemm2 gemm_util mega_moe_config \ + mega_moe_stage1 mega_moe_stage2 mega_moe_v2 quant; do + p="aiter/ops/flydsl/kernels/mega_moe/$f.py" + lv=$(stat -c %s "$V/ops/flydsl/kernels/mega_moe/$f.py" 2>/dev/null || echo -) + sr=$(gh api "/repos/ROCm/aiter/contents/$p?ref=$REF" --jq '.size' 2>/dev/null) + [[ "$sr" =~ ^[0-9]+$ ]] || sr=- + sm=$(gh api "/repos/ROCm/aiter/contents/$p?ref=main" --jq '.size' 2>/dev/null) + [[ "$sm" =~ ^[0-9]+$ ]] || sm=- + if [ "$lv" = "$sr" ]; then v="EXACT@ref" + elif [ "$lv" = "$sm" ]; then v="EXACT@main" + elif [ "$sr" = "-" ]; then v="UPSTREAM-MISSING" + else v="DIFFERS" + fi + printf "%-24s %10s %10s %10s %s\n" "$f.py" "$lv" "$sr" "$sm" "$v" +done diff --git a/.probe_mori_flydsl.sh b/.probe_mori_flydsl.sh new file mode 100644 index 000000000..902e9f19f --- /dev/null +++ b/.probe_mori_flydsl.sh @@ -0,0 +1,30 @@ +#!/usr/bin/env bash +# MegaMoE needs `flydsl` and `mori` as top-level packages. Are they in the base +# image at all, and at what version relative to the measurement image? If they +# differ these are compiled deps and no Python patch can bridge them. +set -u +D=/home/jiacao/3way-20260812-2214 +for img in target vendor; do + echo "=== $img" + for pkg in flydsl mori; do + found=$(find "$D/$img" -maxdepth 8 -type d -name "$pkg" -not -path '*/__pycache__/*' 2>/dev/null | head -3) + if [ -z "$found" ]; then echo " $pkg: ABSENT"; continue; fi + echo " $pkg: $found" + for d in $found; do + v=$(find "$(dirname "$d")" -maxdepth 1 -name "$pkg*dist-info" -o -maxdepth 1 -name "$pkg*egg-info" 2>/dev/null | head -1) + [ -n "$v" ] && echo " dist: $(basename "$v")" + [ -f "$d/_version.py" ] && echo " _version: $(head -3 "$d/_version.py" | tr '\n' ' ')" + [ -f "$d/version.py" ] && echo " version: $(head -3 "$d/version.py" | tr '\n' ' ')" + echo " .so count: $(find "$d" -name '*.so' | wc -l)" + done + done +done + +echo +echo "=== which files in the BASE image import the 5 drifted flydsl kernels? ===" +T="$D/target/usr/local/lib/python3.12/dist-packages/aiter" +for m in flydsl_dispatch_combine_intranode_op flydsl_dispatch_combine_intranode_kernel tensor_shim mxfp4_gemm_common communication_ops_utils; do + n=$(grep -rl "$m" "$T" --include='*.py' 2>/dev/null | wc -l) + echo " $m: $n importer(s)" + grep -rl "$m" "$T" --include='*.py' 2>/dev/null | sed "s|$T/| |" +done diff --git a/.probe_post2.sh b/.probe_post2.sh new file mode 100644 index 000000000..a7f49106b --- /dev/null +++ b/.probe_post2.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +# Which of the vendor-only aiter files exist at tag v0.1.19.post2 (a63ede724b15)? +# +# post2 already contains the PR #4269 merge (compare says ahead=0/behind=24), +# so anything #4269 shipped should be there. Anything still missing came from a +# different source -- either a later upstream PR, or nothing upstream at all. +# That split is exactly what decides "bump the pin" vs "we must open a PR". +set -u +REF="${1:-a63ede724b15}" # v0.1.19.post2 +TAG_NAME="${2:-v0.1.19.post2}" + +check() { # $1 = repo path + # `gh api --jq .size` prints the 404 JSON body on a miss, which is a + # non-empty string -- test for an all-digits size instead of non-emptiness. + local p="$1" s + s=$(gh api "/repos/ROCm/aiter/contents/$p?ref=$REF" --jq '.size' 2>/dev/null) + if [[ "$s" =~ ^[0-9]+$ ]]; then printf " %-8s %-9s %s\n" "PRESENT" "${s}B" "$p" + else printf " %-8s %-9s %s\n" "ABSENT" "-" "$p"; fi +} + +echo "=== $TAG_NAME ($REF) ===" +echo "-- FHMoE / FSE (PR #4269) --" +check aiter/fhmoe.py +check aiter/ops/flydsl/fhmoe.py +check aiter/aot/flydsl/fhmoe.py +check aiter/ops/flydsl/kernels/fhmoe.py +check aiter/ops/flydsl/kernels/mixed_moe_gemm_2stage.py +check aiter/ops/flydsl/kernels/mixed_moe_gemm_2stage_common.py + +echo "-- MegaMoE kernel tree --" +for f in __init__ dispatch gemm1 gemm2 gemm_util mega_moe_config \ + mega_moe_stage1 mega_moe_stage2 mega_moe_v2 quant; do + check "aiter/ops/flydsl/kernels/mega_moe/$f.py" +done + +echo "-- tuned-GEMM data --" +check aiter/configs/model_configs/a8w8_blockscale_tuned_gemm_dsv4.csv + +echo "-- Gluon sparse attention (PR #4382) --" +check aiter/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py diff --git a/.probe_post2_base.sh b/.probe_post2_base.sh new file mode 100644 index 000000000..b6405c9a7 --- /dev/null +++ b/.probe_post2_base.sh @@ -0,0 +1,117 @@ +#!/usr/bin/env bash +# If v0.1.19.post2 (a real tag, 59 commits past v0.1.19) is the base instead of +# v0.1.19, how big is the gluon + MegaMoE patch? Redo the closure/drift analysis +# against post2, and check the 5 shared flydsl files for breaking signature +# changes -- those were the blocker when measuring against v0.1.19. +set -u +python3 - <<'PY' +import ast, os, subprocess, collections, difflib +W = "/tmp/aitermain" +subprocess.run(["git", "-C", W, "checkout", "-q", "v0.1.19.post2"], check=True) +P = os.path.join(W, "aiter") # post2 tree +V = "/home/jiacao/3way-20260812-2214/vendor/usr/local/lib/python3.12/dist-packages/aiter" + +# --- the vendor's own gluon + megamoe files are the ground truth: they are what +# --- actually produced 12,244. Use the vendor tree as the source, post2 as base. +GLUON = ["ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py", + "ops/triton/attention/pa_decode_sparse.py"] +MEGA = ["ops/flydsl/kernels/mega_moe/" + f for f in + ["__init__.py","dispatch.py","gemm1.py","gemm2.py","gemm_util.py", + "mega_moe_config.py","mega_moe_stage1.py","mega_moe_stage2.py", + "mega_moe_v2.py","quant.py"]] + +def find(root, rel): + for s in (".py", "/__init__.py"): + if os.path.exists(os.path.join(root, rel + s)): return rel + s + return None + +def imports_of(path, self_rel): + pkg = os.path.dirname(self_rel); out = set() + for n in ast.walk(ast.parse(open(path).read())): + if isinstance(n, ast.ImportFrom): + if n.level: + base = pkg + for _ in range(n.level - 1): base = os.path.dirname(base) + mod = os.path.join(base, (n.module or "").replace(".", "/")).rstrip("/") + elif n.module and n.module.startswith("aiter"): + mod = n.module[len("aiter"):].lstrip(".").replace(".", "/") + else: continue + out.add(mod) + for a in n.names: out.add(os.path.join(mod, a.name)) + elif isinstance(n, ast.Import): + for a in n.names: + if a.name.startswith("aiter"): + out.add(a.name[len("aiter"):].lstrip(".").replace(".", "/")) + return {m for m in out if m} + +def closure(seed, label): + seen, q = set(seed), collections.deque(seed) + new, drift = set(), {} + while q: + f = q.popleft() + p = os.path.join(V, f) + if not os.path.exists(p): continue + if find(P, f[:-3]) is None: new.add(f) + for m in imports_of(p, f): + rv = find(V, m) + if rv is None or rv in seen: continue + seen.add(rv) + rp = find(P, m) + if rp is None: + new.add(rv); q.append(rv) + else: + a = open(os.path.join(P, rp)).readlines() + b = open(os.path.join(V, rv)).readlines() + if a != b: + drift[rv] = sum(1 for l in difflib.unified_diff(a, b, n=0) + if l[:1] in "+-" and l[:3] not in ("---","+++")) + q.append(rv) + print(f"=== {label}: onto v0.1.19.post2") + print(f" NEW files to add: {len(new)}") + for f in sorted(new): + n = sum(1 for _ in open(os.path.join(V, f))) + print(f" {n:5} {f}") + print(f" EXISTING files that must change: {len(drift)}") + for f, d in sorted(drift.items(), key=lambda kv: -kv[1]): + print(f" {d:5} lines {f}") + return new, drift + +gnew, gdrift = closure(GLUON, "GLUON (#4382+#4673)") +print() +mnew, mdrift = closure(MEGA, "MEGAMOE (#4439)") + +# --- signature safety for every file that must change --- +print() +print("=== signature check on the files that must change (post2 -> vendor) ===") +def api(path): + out = {} + for n in ast.parse(open(path).read()).body: + if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef)): + a = n.args + out[n.name] = tuple([x.arg for x in a.posonlyargs + a.args] + + (["*"+a.vararg.arg] if a.vararg else []) + + [x.arg for x in a.kwonlyargs] + + (["**"+a.kwarg.arg] if a.kwarg else [])) + elif isinstance(n, ast.ClassDef): out[n.name] = ("",) + return out + +for f in sorted(set(gdrift) | set(mdrift)): + if not f.endswith(".py"): continue + try: + b, v = api(os.path.join(P, f)), api(os.path.join(V, f)) + except Exception as e: + print(f" ?? {f}: {e}"); continue + removed = sorted(set(b) - set(v)) + resigned = sorted(k for k in set(b) & set(v) if b[k] != v[k]) + verdict = "ADDITIVE-SAFE" if not removed and not resigned else "BREAKING" + importers = 0 + for dp, _, fns in os.walk(P): + if "__pycache__" in dp: continue + for fn in fns: + if fn.endswith(".py"): + if os.path.basename(f)[:-3] in open(os.path.join(dp, fn)).read(): + importers += 1 + print(f" {verdict:14} {f} ({importers} importers in post2)") + for k in removed: print(f" removed: {k}") + for k in resigned: print(f" resigned: {k}") +PY diff --git a/.probe_post2_features.sh b/.probe_post2_features.sh new file mode 100644 index 000000000..db5aaf84f --- /dev/null +++ b/.probe_post2_features.sh @@ -0,0 +1,56 @@ +#!/usr/bin/env bash +# The measurement image is 85.8% v0.1.19.post2, not main. If post2 already +# carries gluon / MegaMoE / FSE, then the whole "aiter is 107 commits behind" +# framing is wrong: the fix is a TAG BUMP in Dockerfile.rocm_base, not a patch. +set -u +W=/tmp/aitermain +cd "$W" || exit 1 +git checkout -q v0.1.19.post2 + +echo "=== v0.1.19.post2 = $(git rev-parse --short HEAD) ===" +echo " date: $(git log -1 --format=%ci)" +echo " is v0.1.19 an ancestor? $(git merge-base --is-ancestor v0.1.19 HEAD && echo yes || echo no)" +echo " commits v0.1.19..post2: $(git rev-list --count v0.1.19..HEAD)" + +echo +echo "=== feature presence in v0.1.19.post2 ===" +p() { if [ -e "$W/$1" ]; then echo " PRESENT $2"; else echo " MISSING $2"; fi; } +m() { if grep -qF -- "$1" "$W/$2" 2>/dev/null; then echo " PRESENT $3"; else echo " MISSING $3"; fi; } + +p "aiter/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py" "#4382 gluon kernel" +m "_pa_decode_sparse_gfx950_gluon" "aiter/ops/triton/attention/pa_decode_sparse.py" "#4382 gfx950 routing" +m "max_addressable_bytes" "aiter/ops/triton/utils/common_utils.py" "#4673 overflow fix" +p "aiter/ops/flydsl/kernels/mega_moe/mega_moe_v2.py" "#4439 MegaMoE" +p "aiter/fhmoe.py" "#4269 FSE / FHMoE" +m "shared_expert_id" "aiter/fused_moe.py" "#4269 fused_moe shared_expert_id" +p "aiter/configs/model_configs/dsv4_a8w8_blockscale_tuned_gemm.csv" "tuned-gemm csv" + +echo +echo "=== which merged PRs are in post2 but not v0.1.19? ===" +for sha in b3c13c932207bef03aa3a8123bf34acad02f40d3 97d0c6e4cb7a0919c12291c7c7d560ad412f15c1 00cbe979f20cf380548d7d1d9d73136aa359c276; do + case "$sha" in + b3c13c*) n="#4382 gluon" ;; + 97d0c6*) n="#4439 MegaMoE" ;; + 00cbe9*) n="#4269 FSE" ;; + esac + if git merge-base --is-ancestor "$sha" HEAD 2>/dev/null; then + echo " IN post2 $n" + else + echo " NOT in post2 $n" + fi +done + +echo +echo "=== and in v0.1.19 (the base image's pin)? ===" +for sha in b3c13c932207bef03aa3a8123bf34acad02f40d3 97d0c6e4cb7a0919c12291c7c7d560ad412f15c1 00cbe979f20cf380548d7d1d9d73136aa359c276; do + case "$sha" in + b3c13c*) n="#4382 gluon" ;; + 97d0c6*) n="#4439 MegaMoE" ;; + 00cbe9*) n="#4269 FSE" ;; + esac + if git merge-base --is-ancestor "$sha" v0.1.19 2>/dev/null; then + echo " IN v0.1.19 $n" + else + echo " NOT in v0.1.19 $n" + fi +done diff --git a/.probe_so_delta.sh b/.probe_so_delta.sh new file mode 100644 index 000000000..0a8fd9ea9 --- /dev/null +++ b/.probe_so_delta.sh @@ -0,0 +1,19 @@ +#!/usr/bin/env bash +# Do the vendor and base images ship the same set of prebuilt aiter modules? +# If the vendor has extra .so files, the container patch cannot be pure-Python. +set -u +D=/home/jiacao/3way-20260812-2214/ +T="$D/target/usr/local/lib/python3.12/dist-packages/aiter" +V="$D/vendor/usr/local/lib/python3.12/dist-packages/aiter" +cd "$T" && find . -name '*.so' | sort > /tmp/so_target.txt +cd "$V" && find . -name '*.so' | sort > /tmp/so_vendor.txt +echo "vendor-only .so:" +comm -13 /tmp/so_target.txt /tmp/so_vendor.txt | sed 's/^/ /' +echo "target-only .so:" +comm -23 /tmp/so_target.txt /tmp/so_vendor.txt | sed 's/^/ /' +echo +echo "hsaco/co asm blobs:" +cd "$T" && find . \( -name '*.co' -o -name '*.hsaco' \) | sort > /tmp/co_target.txt +cd "$V" && find . \( -name '*.co' -o -name '*.hsaco' \) | sort > /tmp/co_vendor.txt +echo " vendor-only: $(comm -13 /tmp/co_target.txt /tmp/co_vendor.txt | wc -l)" +comm -13 /tmp/co_target.txt /tmp/co_vendor.txt | head -20 | sed 's/^/ /' diff --git a/.probe_target_aiter.sh b/.probe_target_aiter.sh new file mode 100644 index 000000000..e53c28062 --- /dev/null +++ b/.probe_target_aiter.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +# What does the BASE image's aiter (v0.1.19, target/) already carry, and what +# must the container patch add? Anything MISSING here is a real gap for the +# corresponding upstream aiter PR. +set -u +T=/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter + +p() { # $1=path-relative-to-aiter $2=label + if [ -e "$T/$1" ]; then echo " PRESENT $2 ($1)"; else echo " MISSING $2 ($1)"; fi +} +m() { # $1=marker $2=file $3=label + if grep -qF -- "$1" "$T/$2" 2>/dev/null; then echo " PRESENT $3"; else echo " MISSING $3"; fi +} + +echo "== #4382 gfx950 gluon sparse decode ==" +p "ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py" "gluon kernel" +m "_pa_decode_sparse_gfx950_gluon" "ops/triton/attention/pa_decode_sparse.py" "gfx950 routing entry" + +echo "== #4673 buffer_load span fix ==" +m "max_addressable_bytes" "ops/triton/utils/common_utils.py" "max_addressable_bytes helper" + +echo "== #4439 MegaMoE ==" +p "ops/flydsl/kernels/mega_moe/mega_moe_v2.py" "mega_moe package" +p "ops/flydsl/kernels/mega_moe/__init__.py" "mega_moe __init__" + +echo "== #4269 FSE / heterogeneous shared expert (FHMoE) ==" +grep -rlF "shared_expert_id" "$T" --include='*.py' 2>/dev/null | sed 's|^| hit: |' | head +grep -rlF "fhmoe" "$T" --include='*.py' 2>/dev/null | sed 's|^| hit: |' | head + +echo "== #4664 tuned GEMM CSV rows ==" +p "configs/model_configs/dsv4_a8w8_blockscale_tuned_gemm.csv" "dsv4 tuned-gemm csv" +if [ -e "$T/configs/model_configs/dsv4_a8w8_blockscale_tuned_gemm.csv" ]; then + echo " lines: $(wc -l < "$T/configs/model_configs/dsv4_a8w8_blockscale_tuned_gemm.csv")" + grep -c '^6144,7168\|^65536,1536\|^7168,3072' "$T/configs/model_configs/dsv4_a8w8_blockscale_tuned_gemm.csv" \ + | sed 's|^| dsv4 shapes present: |' +fi diff --git a/.probe_tgemm.sh b/.probe_tgemm.sh new file mode 100644 index 000000000..30825452c --- /dev/null +++ b/.probe_tgemm.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# 'tgemm' as a bare string appears in exactly 1 file on all three sides, which +# means it is not the identifier the DSv4 arm actually uses. Widen the search: +# check aiter's own version, the vendor's aiter provenance, and the plausible +# tuned-GEMM entry points by name. +set -u +D="${1:-/home/jiacao/3way-20260812-2214}" + +for side in ref vendor target; do + A=$(find "$D/$side" -maxdepth 9 -type d -name aiter -path '*dist-packages*' 2>/dev/null | head -1) + echo "=== $side ${A:-}" + [ -z "$A" ] && continue + + echo -n " aiter version: " + grep -hE "^__version__|^version" "$A/_version.py" 2>/dev/null | head -2 | tr '\n' ' '; echo + + echo " -- tgemm hits (any case, with context) --" + grep -rn "tgemm" "$A" --include='*.py' 2>/dev/null | head -5 | sed 's|'"$A"'|.|' | sed 's/^/ /' + + echo " -- tuned-GEMM surface --" + for pat in "tuned_gemm" "TunedGemm" "tuned_gemm_dsv4" "gemm_a8w8_blockscale" "gemm_tune"; do + n=$(grep -rl -- "$pat" "$A" --include='*.py' 2>/dev/null | wc -l) + printf " %-28s %s file(s)\n" "$pat" "$n" + done + + echo " -- tuned-GEMM csv configs --" + ls "$A/configs/model_configs/" 2>/dev/null | grep -iE "gemm|dsv4" | head -6 | sed 's/^/ /' + echo -n " total csv: "; ls "$A/configs/model_configs/"*.csv 2>/dev/null | wc -l +done diff --git a/.probe_vendor_over_post2.sh b/.probe_vendor_over_post2.sh new file mode 100644 index 000000000..87d6d0a90 --- /dev/null +++ b/.probe_vendor_over_post2.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# The measurement image = v0.1.19.post2 + something. Enumerate exactly what that +# something is: which files differ, and does each one trace to #4382 / #4439 / +# #4673 or to an unattributed vendor edit? +set -u +python3 - <<'PY' +import os, subprocess +D = "/home/jiacao/3way-20260812-2214" +W = "/tmp/aitermain" +V = os.path.join(D, "vendor/usr/local/lib/python3.12/dist-packages/aiter") + +def snap(ref): + subprocess.run(["git", "-C", W, "checkout", "-q", ref], check=True) + root = os.path.join(W, "aiter"); out = {} + for dp, _, fns in os.walk(root): + if "__pycache__" in dp: continue + for fn in fns: + if fn.endswith(".py") or fn.endswith(".csv"): + p = os.path.join(dp, fn) + out[os.path.relpath(p, root)] = open(p, "rb").read() + return out + +def img(root): + out = {} + for dp, _, fns in os.walk(root): + if "__pycache__" in dp: continue + for fn in fns: + if fn.endswith(".py") or fn.endswith(".csv"): + p = os.path.join(dp, fn) + out[os.path.relpath(p, root)] = open(p, "rb").read() + return out + +post2 = snap("v0.1.19.post2") +main = snap("97d0c6e4cb7a0919c12291c7c7d560ad412f15c1") +vend = img(V) + +added = sorted(set(vend) - set(post2)) +removed = sorted(set(post2) - set(vend)) +changed = sorted(k for k in set(vend) & set(post2) if vend[k] != post2[k]) + +def origin(k): + """Does the vendor's copy match main's copy? -> upstream cherry-pick.""" + if k in main and vend[k] == main[k]: + return "== main@97d0c6e4" + if k in main: + return "!= main (vendor-modified or older main)" + return "not in main either -> VENDOR-ONLY" + +print(f"=== vendor ADDS {len(added)} files over v0.1.19.post2") +for k in added: + print(f" {origin(k):38} {k}") +print() +print(f"=== vendor CHANGES {len(changed)} files") +for k in changed: + print(f" {origin(k):38} {k}") +print() +print(f"=== vendor REMOVES {len(removed)} files") +for k in removed: + print(f" {k}") +PY diff --git a/.probe_vllm_classify.sh b/.probe_vllm_classify.sh new file mode 100644 index 000000000..3aca94374 --- /dev/null +++ b/.probe_vllm_classify.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +# Classify each of the 19 vendor-changed vLLM files by which feature it carries. +# Print the ref->vendor diff for the small ones in full and a keyword census for +# the large ones, so every hunk can be attributed to gluon / MegaMoE / FSE / +# #51473 / other before it goes in a waiver table. +set -u +D=/home/jiacao/3way-20260812-2214 +R="$D/ref/usr/local/lib/python3.12/dist-packages/vllm" +V="$D/vendor/src/vllm/vllm" + +FILES=" +config/kernel.py +model_executor/layers/fused_moe/experts/rocm_aiter_moe.py +model_executor/model_loader/utils.py +models/deepseek_v4/attention.py +platforms/interface.py +platforms/rocm.py +v1/worker/gpu/cudagraph_utils.py +v1/worker/gpu_ubatch_wrapper.py +models/deepseek_v4/amd/dspark.py +models/deepseek_v4/amd/mtp.py +v1/worker/gpu_model_runner.py +" +for f in $FILES; do + echo "################ $f" + diff -u "$R/$f" "$V/$f" | tail -n +3 +done + +echo +echo "############ keyword census for the large files ############" +for f in _aiter_ops.py models/deepseek_v4/amd/model.py models/deepseek_v4/amd/rocm.py \ + v1/attention/ops/rocm_aiter_mla_sparse.py; do + echo "--- $f" + diff -u "$R/$f" "$V/$f" | grep '^+' | grep -oiE \ + "gluon|mega_?moe|flydsl|fhmoe|fusion_shared|tgemm|tuned_gemm|sparse|mxfp4|dspark" \ + | tr 'A-Z' 'a-z' | sort | uniq -c | sort -rn | sed 's/^/ /' +done diff --git a/.probe_vllm_delta.sh b/.probe_vllm_delta.sh new file mode 100644 index 000000000..35b9782c4 --- /dev/null +++ b/.probe_vllm_delta.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# The vendor_patchset.vllm.diff was generated per-file with a fixed --label, so +# the diff headers all read "a/vllm b/vllm" and the filenames are lost. Recover +# them by re-diffing the ref and vendor trees directly, and for each changed +# file report its churn plus whether the target already has the same change. +# +# ref = the vendor's upstream base (vLLM 02e63f2e4, 2026-07-30) +# vendor = the measurement image (ref + the vendor patch set) +# target = the nightly we want to land on (b22afe45, 465 commits past ref) +set -u +D=/home/jiacao/3way-20260812-2214 +R=$(find "$D/ref" -maxdepth 9 -type d -name vllm -path '*packages*' | head -1) +# The vendor image installs vLLM editable from /src/vllm, so its package root is +# /src/vllm/vllm rather than a dist-packages path like the other two sides. +V="$D/vendor/src/vllm/vllm" +T=$(find "$D/target" -maxdepth 9 -type d -name vllm -path '*packages*' | head -1) +echo "ref=$R"; echo "vendor=$V"; echo "target=$T"; echo + +printf "%-6s %6s %6s %-9s %s\n" STATUS +LINES -LINES "IN-TARGET" FILE +( cd "$V" && find . -name '*.py' -not -path '*__pycache__*' | sort ) | while read -r rel; do + rel=${rel#./} + [ "$rel" = "_version.py" ] && continue + vf="$V/$rel"; rf="$R/$rel"; tf="$T/$rel" + if [ ! -f "$rf" ]; then + st=NEW; add=$(wc -l < "$vf"); del=0 + else + cmp -s "$rf" "$vf" && continue + st=MOD + add=$(diff "$rf" "$vf" | grep -c '^>') + del=$(diff "$rf" "$vf" | grep -c '^<') + fi + # Does the target already carry this file, and does it already match vendor? + if [ ! -f "$tf" ]; then int="MISSING" + elif cmp -s "$tf" "$vf"; then int="SAME" + else int="DIFFERS" + fi + printf "%-6s %6s %6s %-9s %s\n" "$st" "$add" "$del" "$int" "$rel" +done diff --git a/.probe_vllm_markers.sh b/.probe_vllm_markers.sh new file mode 100644 index 000000000..f5ace13e4 --- /dev/null +++ b/.probe_vllm_markers.sh @@ -0,0 +1,49 @@ +#!/usr/bin/env bash +# For each vendor-only vLLM change, pick a marker string that only exists if the +# feature is present, and check it in ref / target / vendor. +# +# "DIFFERS" from the file-level probe is not evidence of a missing feature -- +# 465 upstream commits touch these files anyway. Markers are. +# ref absent, vendor present, target absent -> we must add it +# ref absent, vendor present, target present -> upstream already landed it +set -u +D=/home/jiacao/3way-20260812-2214 +R="$D/ref/usr/local/lib/python3.12/dist-packages/vllm" +V="$D/vendor/src/vllm/vllm" +T="$D/target/usr/local/lib/python3.12/dist-packages/vllm" + +row() { # $1=marker $2=relpath-or-TREE $3=label + local m="$1" p="$2" lbl="$3" out="" + for side in R V T; do + local base; base=$(eval echo "\$$side") + local n + if [ "$p" = "TREE" ]; then + n=$(grep -rl -- "$m" "$base" --include='*.py' 2>/dev/null | wc -l) + elif [ -f "$base/$p" ]; then + n=$(grep -c -- "$m" "$base/$p" 2>/dev/null || echo 0) + else + n="-" + fi + out="$out$(printf '%6s' "$n")" + done + printf "%s %-34s %s\n" "$out" "$lbl" "$m" +} + +echo " ref vendor target FEATURE MARKER" +row "VLLM_ROCM_DSV4_SPARSE_GLUON" TREE "gluon env knob" +row "pa_decode_sparse" TREE "gluon sparse decode call" +row "mla_gluon" TREE "gluon MLA" +row "flydsl_mega_moe" TREE "MegaMoE backend name" +row "mega_moe" TREE "MegaMoE (any)" +row "MegaMoE" TREE "MegaMoE (class)" +row "FUSION_SHARED_EXPERTS" TREE "FSE env knob" +row "fhmoe" TREE "FHMoE" +row "AITER_MXFP4_BF16" TREE "PR #51473 marker" +row "tuned_gemm" TREE "tuned GEMM" +echo +echo "-- new files --" +for f in models/deepseek_v4/amd/mega_moe_experts.py models/deepseek_v4/amd/mega_moe_runtime.py; do + printf " ref=%-9s target=%-9s %s\n" \ + "$([ -f "$R/$f" ] && echo yes || echo no)" \ + "$([ -f "$T/$f" ] && echo yes || echo no)" "$f" +done diff --git a/.probe_vllm_markers2.sh b/.probe_vllm_markers2.sh new file mode 100644 index 000000000..e563ea66d --- /dev/null +++ b/.probe_vllm_markers2.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# Second pass: the coarse markers were too coarse. FUSION_SHARED_EXPERTS and +# AITER_MXFP4_BF16 appear on all three sides, so their mere presence proves +# nothing -- the vendor delta is in the specific condition / call site. +set -u +D=/home/jiacao/3way-20260812-2214 +R="$D/ref/usr/local/lib/python3.12/dist-packages/vllm" +V="$D/vendor/src/vllm/vllm" +T="$D/target/usr/local/lib/python3.12/dist-packages/vllm" + +echo "=== PR #51473 exact condition (oracle/mxfp4.py) ===" +for s in R V T; do b=$(eval echo "\$$s") + printf " %-7s " "$s" + grep -n "AITER_MXFP4_BF16" "$b/model_executor/layers/fused_moe/oracle/mxfp4.py" 2>/dev/null \ + | head -4 | tr '\n' '|' ; echo +done + +echo +echo "=== FSE: where do the extra 7 vendor hits live? ===" +echo "--- vendor files mentioning FUSION_SHARED_EXPERTS ---" +grep -rl "FUSION_SHARED_EXPERTS" "$V" --include='*.py' 2>/dev/null | sed "s|$V/| |" +echo "--- target files mentioning FUSION_SHARED_EXPERTS ---" +grep -rl "FUSION_SHARED_EXPERTS" "$T" --include='*.py' 2>/dev/null | sed "s|$T/| |" + +echo +echo "=== MegaMoE: which files, each side ===" +for s in R V T; do b=$(eval echo "\$$s"); echo "--- $s ---" + grep -rl "mega_moe\|MegaMoE" "$b" --include='*.py' 2>/dev/null | sed "s|$b/| |" +done + +echo +echo "=== tuned_gemm call sites ===" +for s in R V T; do b=$(eval echo "\$$s"); echo "--- $s ---" + grep -rn "tuned_gemm" "$b" --include='*.py' 2>/dev/null | sed "s|$b/| |" | head -6 +done diff --git a/.verify_dsv4_image.sh b/.verify_dsv4_image.sh new file mode 100644 index 000000000..d0c3373b2 --- /dev/null +++ b/.verify_dsv4_image.sh @@ -0,0 +1,89 @@ +#!/usr/bin/env bash +#SBATCH --job-name=dsv4-verify +#SBATCH --account=amd-aifw-aim +#SBATCH --qos=amd-aifw-aim-qos +#SBATCH --partition=amd-spur +#SBATCH --nodes=1 +#SBATCH --cpus-per-task=4 +#SBATCH --mem=16G +#SBATCH --time=00:15:00 +#SBATCH --nodelist=crsuse2-m2m-006 +#SBATCH --output=/home/jiacao/InferenceX/dsv4-verify-%j.out +# +# Ground-truth the committed image dsv4-pro-fp4-mi355x:f8d03e77-patched. The build +# job's own verify greps were mis-targeted (looked under fused_moe/ for the +# flydsl_mega_moe enum that actually lives in config/kernel.py). These probes call +# the real code paths -- no GPU needed, all pure-Python config/oracle/env reads. +set -uo pipefail +hostname; date -u + +IMG="dsv4-pro-fp4-mi355x:3ee2df30-patched" +docker image inspect "$IMG" >/dev/null 2>&1 \ + || { echo "FATAL: image $IMG not on this node; run on the build node (006)"; exit 1; } +run(){ docker run --rm --entrypoint /bin/bash "$IMG" -c "$1"; } + +echo "=== [DEP8] --moe-backend flydsl_mega_moe accepted by the enum? (#51918) ===" +run 'python3 - </dev/null | head' + +echo +echo "=== [TP8] 384-wide MXFP4 shard oracle (#51473, expect 384) ===" +run 'python3 - <",i) + print("VERDICT:", "384 shard PRESERVED" if i==384 else "ROUNDED to %d"%i) +except Exception: + import sys + traceback.print_exc(file=sys.stdout) + print("VERDICT: PROBE ERROR") +PY' + +echo +echo "=== [TP8] FSE env + wiring (base #4269 present; #48728 DROPPED) ===" +run 'python3 -c "import vllm.envs as e; print(\"VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS =\", e.VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS)"' +run 'V=$(python3 -c "import vllm,os;print(os.path.dirname(vllm.__file__))"); + M="$V/models/deepseek_v4/amd/model.py"; + echo "base OLD FSE (_fuse_shared_experts_enabled), expect >0:"; + grep -c "_fuse_shared_experts_enabled" "$M" 2>/dev/null; + echo "#48728 hetero FSE (_should_fuse_shared_expert), expect 0 (dropped):"; + grep -c "_should_fuse_shared_expert" "$M" 2>/dev/null' + +echo +echo "=== [gluon] env + vllm wiring (#51714) ===" +run 'python3 -c "import vllm.envs as e; print(\"VLLM_ROCM_DSV4_SPARSE_GLUON =\", e.VLLM_ROCM_DSV4_SPARSE_GLUON)"' +run 'V=$(python3 -c "import vllm,os;print(os.path.dirname(vllm.__file__))"); + grep -c "_DSV4_SPARSE_GLUON" "$V/v1/attention/ops/rocm_aiter_mla_sparse.py"' + +echo +echo "=== [#4673] aiter gluon buffer_load int64 span fix present? ===" +run 'V=$(python3 -c "import aiter,os;print(os.path.dirname(aiter.__file__))"); + echo "max_addressable_bytes in common_utils.py:"; + grep -c "max_addressable_bytes" "$V/ops/triton/utils/common_utils.py" 2>/dev/null || echo 0; + echo "max_addressable_bytes in routing pa_decode_sparse.py:"; + grep -c "max_addressable_bytes" "$V/ops/triton/attention/pa_decode_sparse.py" 2>/dev/null || echo 0' + +date -u diff --git a/aiter-4417.diff b/aiter-4417.diff new file mode 100644 index 000000000..f6d4432fc --- /dev/null +++ b/aiter-4417.diff @@ -0,0 +1,73 @@ +diff --git a/aiter/ops/flydsl/moe_kernels.py b/aiter/ops/flydsl/moe_kernels.py +index 4b3253b7c0..454a58b523 100644 +--- a/aiter/ops/flydsl/moe_kernels.py ++++ b/aiter/ops/flydsl/moe_kernels.py +@@ -13,6 +13,9 @@ + + _KERNEL_PARAMS: dict[str, dict] = {} + ++# HIP limits grid.y/grid.z to 65535. ++_HIP_MAX_GRID_DIM_Y = 65535 ++ + + def _get_dtypes(): + from aiter.utility import dtypes +@@ -94,6 +97,25 @@ def pick_flydsl_stage1_tile_n(inter_dim: int) -> int: + return 256 if (inter_dim % 256 == 0) else 128 + + ++def resolve_flydsl_grid_y_persist_m( ++ num_m_blocks: int, requested_persist_m: int = 0 ++) -> int: ++ """Increase persist_m as needed to keep grid.y within HIP's limit.""" ++ num_m_blocks = max(int(num_m_blocks), 0) ++ requested_persist_m = max(int(requested_persist_m), 1) ++ required_persist_m = max( ++ 1, (num_m_blocks + _HIP_MAX_GRID_DIM_Y - 1) // _HIP_MAX_GRID_DIM_Y ++ ) ++ return max(requested_persist_m, required_persist_m) ++ ++ ++def requires_flydsl_stage2_reduce( ++ token_num: int, model_dim: int, element_size: int ++) -> bool: ++ """Return whether stage2 atomic output exceeds 32-bit byte offsets.""" ++ return int(token_num) * int(model_dim) * int(element_size) > 0xFFFFFFFF ++ ++ + def resolve_flydsl_stage2_tile_k(inter_dim: int, tile_k: int) -> int: + """Return a ``tile_k`` that divides ``inter_dim``, preferring the caller value. + +@@ -1403,7 +1425,7 @@ def flydsl_moe_stage1( + ) + _grid_y = min(_dense_blks, _all_blks) + +- _persist_m = persist_m if persist_m > 0 else 1 ++ _persist_m = resolve_flydsl_grid_y_persist_m(_grid_y, persist_m) + + # Allocate sorted-scale buffer with padding for tiled layout + scale_cols = inter_dim // 32 +@@ -1706,6 +1728,13 @@ def flydsl_moe_stage2( + # accumulate. Enabled by default; set AITER_FLYDSL_FORCE_REDUCE=0 to opt out. + if os.environ.get("AITER_FLYDSL_FORCE_REDUCE", "0") == "1": + mode = "reduce" ++ elif ( ++ mode != "reduce" ++ and not return_per_slot ++ and requires_flydsl_stage2_reduce(token_num, model_dim, 2) ++ ): ++ # Buffer atomics use 32-bit offsets; reduce outputs larger than 4 GiB. ++ mode = "reduce" + + accumulate = mode != "reduce" and not return_per_slot + +@@ -1764,7 +1793,8 @@ def flydsl_moe_stage2( + _persist_m = -1 if m_blocks > 256 else 1 + + if a_dtype == "fp8": +- _persist_m = 1 ++ # FP8 uses non-persistent scheduling, so cap grid.y via persist_m. ++ _persist_m = resolve_flydsl_grid_y_persist_m(m_blocks) + + if bias is not None and bias.dtype != torch.float32: + bias = bias.to(torch.float32) diff --git a/apply_aiter_4417.py b/apply_aiter_4417.py new file mode 100644 index 000000000..b0fe7288a --- /dev/null +++ b/apply_aiter_4417.py @@ -0,0 +1,112 @@ +#!/usr/bin/env python3 +"""Graft ROCm/aiter#4417 ("Fix large-token FlyDSL MoE launch and output limits") +onto the aiter shipped in the 2026-08-12 vLLM ROCm nightly. + +Why not `git apply` the upstream diff: the nightly's aiter predates the +`dict[str, dict]` typing modernization, so every hunk's context is off by the +`Dict[str, Dict]` spelling. The substance of #4417 is five small anchored edits, +so we apply those directly. Idempotent — re-running is a no-op. + +Without this patch, DSv4-Pro FP8/FP4 stage2 uses buffer atomics whose 32-bit +byte offsets overflow once token_num*model_dim*2 exceeds 4 GiB, which shows up +as `Memory access fault by GPU node-N ... Reason: Unknown` during the vLLM +profile run (determine_available_memory), killing the engine core. +""" + +import sys + +HELPERS = ''' +# HIP limits grid.y/grid.z to 65535. +_HIP_MAX_GRID_DIM_Y = 65535 + + +def resolve_flydsl_grid_y_persist_m( + num_m_blocks: int, requested_persist_m: int = 0 +) -> int: + """Increase persist_m as needed to keep grid.y within HIP's limit.""" + num_m_blocks = max(int(num_m_blocks), 0) + requested_persist_m = max(int(requested_persist_m), 1) + required_persist_m = max( + 1, (num_m_blocks + _HIP_MAX_GRID_DIM_Y - 1) // _HIP_MAX_GRID_DIM_Y + ) + return max(requested_persist_m, required_persist_m) + + +def requires_flydsl_stage2_reduce( + token_num: int, model_dim: int, element_size: int +) -> bool: + """Return whether stage2 atomic output exceeds 32-bit byte offsets.""" + return int(token_num) * int(model_dim) * int(element_size) > 0xFFFFFFFF + + +''' + +# (description, exact old text, new text) — each old text must be unique. +EDITS = [ + ( + "add _HIP_MAX_GRID_DIM_Y + the two guard helpers", + "def resolve_flydsl_stage2_tile_k(inter_dim: int, tile_k: int) -> int:", + HELPERS.lstrip("\n") + + "def resolve_flydsl_stage2_tile_k(inter_dim: int, tile_k: int) -> int:", + ), + ( + "stage1: cap grid.y via persist_m", + " _persist_m = persist_m if persist_m > 0 else 1", + " _persist_m = resolve_flydsl_grid_y_persist_m(_grid_y, persist_m)", + ), + ( + "stage2: switch to reduce when atomic output exceeds 4 GiB", + ' if os.environ.get("AITER_FLYDSL_FORCE_REDUCE", "0") == "1":\n' + " mode = \"reduce\"\n", + ' if os.environ.get("AITER_FLYDSL_FORCE_REDUCE", "0") == "1":\n' + ' mode = "reduce"\n' + " elif (\n" + ' mode != "reduce"\n' + " and not return_per_slot\n" + " and requires_flydsl_stage2_reduce(token_num, model_dim, 2)\n" + " ):\n" + " # Buffer atomics use 32-bit offsets; reduce outputs larger than 4 GiB.\n" + ' mode = "reduce"\n', + ), + ( + "stage2 fp8: cap grid.y via persist_m", + ' if a_dtype == "fp8":\n _persist_m = 1\n', + ' if a_dtype == "fp8":\n' + " # FP8 uses non-persistent scheduling, so cap grid.y via persist_m.\n" + " _persist_m = resolve_flydsl_grid_y_persist_m(m_blocks)\n", + ), +] + + +def main() -> int: + path = sys.argv[1] if len(sys.argv) > 1 else None + if path is None: + import importlib.util as u + + spec = u.find_spec("aiter.ops.flydsl.moe_kernels") + if spec is None or spec.origin is None: + print("FAIL: cannot locate aiter.ops.flydsl.moe_kernels", file=sys.stderr) + return 1 + path = spec.origin + + src = open(path).read() + + if "requires_flydsl_stage2_reduce" in src and "resolve_flydsl_grid_y_persist_m" in src: + print(f"SKIP: aiter#4417 already present in {path}") + return 0 + + for desc, old, new in EDITS: + n = src.count(old) + if n != 1: + print(f"FAIL: anchor for '{desc}' matched {n} times (want 1)", file=sys.stderr) + return 1 + src = src.replace(old, new, 1) + print(f" ok: {desc}") + + open(path, "w").write(src) + print(f"OK: grafted aiter#4417 into {path}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/waiver/2589.md b/docs/waiver/2589.md new file mode 100644 index 000000000..444915fdb --- /dev/null +++ b/docs/waiver/2589.md @@ -0,0 +1,92 @@ +# Inference-engine patch waiver — PR #2589 + +Filed per [`docs/PR_REVIEW_CHECKLIST.md`](../PR_REVIEW_CHECKLIST.md) (image-provenance / patch item): a +benchmark script in this PR patches the pinned serving-stack image before serving, which the checklist +prohibits unless covered by a filled-out waiver. + +## Config covered + +- **Master config entry:** `dsv4-fp4-mi355x-vllm-agentic-mtp` in [`configs/amd-master.yaml`](../../configs/amd-master.yaml) +- **Pinned image:** `vllm/vllm-openai-rocm:nightly-f8d03e77416bf90c49acbe50e233275722f02c4b` + (vllm `0.26.1rc1.dev528+gf8d03e774`) +- **Patch entrypoint:** `bash "$(dirname "$0")/apply_dsv4_container_patches.sh"` invoked from + [`benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh`](../../benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh) +- **Patch script:** [`benchmarks/single_node/agentic/apply_dsv4_container_patches.sh`](../../benchmarks/single_node/agentic/apply_dsv4_container_patches.sh) + (helper: [`graft_aiter_4417.py`](../../benchmarks/single_node/agentic/graft_aiter_4417.py)) + +## What is patched + +`apply_dsv4_container_patches.sh` is **strictly additive and idempotent**: every step is marker-gated, and +nothing that ships in the base image is overwritten. Re-running it on an already-patched container is a +verified no-op. + +| Upstream PR | What it changes | Notes | +|---|---|---| +| vllm #51473 | native MXFP4 TP8 shard allocation (`vllm/model_executor/layers/fused_moe/oracle/mxfp4.py`) | **merged 2026-08-11, i.e. after this base.** Applied with `--include='vllm/*'`: the PR carries a `tests/` hunk with no counterpart in an installed wheel | +| vllm #51714 | opt-in AITER gluon sparse-MLA decode kernel for gfx950 (`vllm/v1/attention/ops/rocm_aiter_mla_sparse.py`) | open upstream. Dormant unless `VLLM_ROCM_DSV4_SPARSE_GLUON=1`; the call site is a lazy in-function import behind that env | +| vllm #51918 (**PARTIAL**) | `vllm/config/kernel.py` **only** — registers `flydsl_mega_moe` as an accepted `--moe-backend` value | model-side hunks deliberately NOT taken; see below | +| aiter #4417 | large-token FlyDSL MoE launch/output limits (`aiter/ops/flydsl/moe_kernels.py`) | merged upstream 2026-07-30, but the base's **vendored** aiter predates it and does not track aiter main. The upstream `.diff` does not apply — the base still reads `Dict[str, Dict]` where the diff's context expects `dict[str, dict]` (aiter's typing modernization postdates the vendored revision) — so the hunks are grafted by anchor via `graft_aiter_4417.py` | +| aiter gluon kernel | adds `aiter/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py` from `aiter@97d0c6e4` | the single file the base genuinely lacks. The other three `pa_decode_sparse` variants (`gfx1250/`, `_triton_kernels/`, and the `ops/triton/` facade) **ship in the base and are left untouched** — an earlier revision of this script copied all four, silently replacing base files with a cross-version transplant. The script now skips anything already present | + +### Why #51918 is taken only partially + +Two independent reasons, either sufficient on its own: + +1. **The backend cannot run on this base at all.** MegaMoE's intranode kernel imports `mori.ir.flydsl`, + which this image does not have (verified: `ModuleNotFoundError`). Applying the model hunks would buy a + backend that raises on first use. +2. **The model hunks do not apply cleanly.** #51918 is written against a tree ~3 days newer than this + base; the two earlier hunks shift line numbering enough that the third fails at + `vllm/models/deepseek_v4/amd/model.py:300`. `git apply --3way` cannot rescue it because `site-packages` + is not a git repo, so there are no blobs to three-way against. + +Net effect: the TP8 arm is unaffected (it never selects this backend), and a DEP8 row is rejected at +config time with a clear error rather than failing deep inside a kernel. **DEP8/MegaMoE is explicitly out +of scope on this route**, as is FSE (aiter #4269 needs `aiter/fhmoe.py`, also absent from the base). + +## Why the unmodified upstream image cannot run this benchmark + +Two separate problems, at two different pins. + +**1. The previous pin (`nightly-821717118f…`, and the 08-12 `nightly-3ee2df30…` line) memory-faults before +serving a single request.** During the profile run the eight TP workers serialize on +`/tmp/aiter_configs/bf16_tuned_gemm.csv.lock`; the worker holding the baton dies in the post-MoE bf16 GEMM +(`Memory access fault by GPU node-N … Reason: Unknown`) while the other seven survive parked on the lock. +This reproduces on a **pristine, completely unpatched** 08-12 nightly, so it is a base regression, not +something the patches here introduce or fix. Two standalone repros clear the loudest suspects: the +`inter_dim=384` MoE returns finite results at the exact faulting shape with preshuffled weights and the +same kernel pair, and the bf16 GEMM passes on all four paths (`gemm_a16w16_opus`, asm, triton, torch) at +both M=65536 and M=16384 × N=7168 × K=7168. It is non-deterministic across runs (different worker, +different address, different kernel selection) and is tracked separately upstream. + +**2. The 08-09 base this PR repins to is stable, but predates vllm #51473.** That PR merged 2026-08-11 and +supplies the native MXFP4 TP8 shard allocation this DSv4-Pro FP4 config needs; the 08-12 line already +carried it, the 08-09 line does not. Without it the config cannot serve on the older, working base. + +So neither pin serves this config unmodified: the newer one faults, and the older one is missing a +merged-upstream fix. Patching the older base is the only combination that reaches +`Application startup complete` today. + +Verified on the patched 08-09 image (hold job 6707, node crsuse2-m2m-100): weight load, profile run and +cudagraph capture all complete with fault=0, `compilation_config` correctly retains +`mode: VLLM_COMPILE: 3` / `cudagraph_mode: FULL_AND_PIECEWISE`, and an 8k-in/1k-out `vllm bench serve` run +at concurrency 32 completes 320/320 requests with 0 failures at an MTP acceptance length of 2.49 — +matching the pinned `synthetic_acceptance_length` of 2.49. + +## Upstream PR / issue links + +- vLLM: https://github.com/vllm-project/vllm/pull/51473 (merged 2026-08-11), + https://github.com/vllm-project/vllm/pull/51714 (open), + https://github.com/vllm-project/vllm/pull/51918 (open, partial) +- AITER: https://github.com/ROCm/aiter/pull/4417 (merged 2026-07-30) +- Gluon sparse-MLA decode kernel: `aiter@97d0c6e4cb7a0919c12291c7c7d560ad412f15c1` + +## Removal plan + +Retire `apply_dsv4_container_patches.sh` and its invocation from `dsv4_fp4_mi355x_vllm_mtp.sh` once a ROCm +vLLM nightly satisfies both halves at once: it must carry the merged vllm #51473 and aiter #4417 fixes, +ship the gfx950 `pa_decode_sparse` gluon kernel, and **not** carry the profile-run bf16-GEMM memory fault +described above. At that point bump the pinned `image:` for `dsv4-fp4-mi355x-vllm-agentic-mtp` to that +nightly, drop the patch script, `graft_aiter_4417.py` and the `bash …/apply_dsv4_container_patches.sh` +call, and delete this waiver in the same PR. The vllm #51714 gluon knob and the #51918 backend +registration can be dropped as soon as those PRs land upstream, independently of the fault. diff --git a/dsv4_bench_8k1k_c32.sh b/dsv4_bench_8k1k_c32.sh new file mode 100644 index 000000000..9fe55c4ad --- /dev/null +++ b/dsv4_bench_8k1k_c32.sh @@ -0,0 +1,19 @@ +#!/usr/bin/env bash +# 8k-in / 1k-out random benchmark at concurrency 32 against the patched DEP8 +# server. Standalone `vllm bench serve` (NOT the agentic aiperf replay) -- the +# quick throughput smoke the user asked for first. +set -uo pipefail + +BASE=http://localhost:8000 +SERVED=deepseek-ai/DeepSeek-V4-Pro +LOG=/home/jiacao/InferenceX/dsv4-bench-8k1k-c32.log + +exec vllm bench serve \ + --backend openai-chat --endpoint /v1/chat/completions \ + --base-url "$BASE" --model "$SERVED" \ + --dataset-name random \ + --random-input-len 8000 --random-output-len 1000 \ + --max-concurrency 32 --num-prompts 128 \ + --ignore-eos --percentile-metrics ttft,tpot,itl,e2el \ + --save-result --result-dir /home/jiacao/InferenceX \ + --result-filename dsv4-bench-8k1k-c32.json > "$LOG" 2>&1 diff --git a/dsv4_patch_additive.sh b/dsv4_patch_additive.sh new file mode 100644 index 000000000..dec899ae8 --- /dev/null +++ b/dsv4_patch_additive.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# ADDITIVE patch for the 08-12 nightly base (3ee2df30). The 08-12 base aiter is +# NEWER than v0.1.19.post2 and already exposes _set_current_hip_stream (which the +# nightly vllm calls for module_rmsnorm_quant) plus the tuned-gemm CSVs. The old +# wholesale post2-python overlay REGRESSED aiter/jit/core.py and dropped that +# symbol -> rmsnorm_quant warmup crash. So here we ONLY ADD the gluon sparse-MLA +# kernels that the base genuinely lacks, and apply the two vllm wiring PRs. We do +# NOT touch aiter core, and we do NOT add MegaMoE (its intranode kernel needs +# mori.ir.flydsl, absent from the base -> DEP8 is out of scope for this route). +set -uo pipefail +AITER_SHA="97d0c6e4cb7a0919c12291c7c7d560ad412f15c1" +AITER_REPO="https://github.com/ROCm/aiter" +VLLM_REPO="https://github.com/vllm-project/vllm" +ROOT="$(python -c 'import importlib.util as u, os; print(os.path.dirname(os.path.dirname(u.find_spec("vllm").origin)))')" +[ -d "$ROOT/vllm" ] && [ -d "$ROOT/aiter" ] || { echo "ERROR ROOT=$ROOT"; exit 1; } +echo "[add] ROOT=$ROOT" +WS=/tmp/dsv4_add; mkdir -p "$WS" + +# --- gluon kernels only (additive; base lacks _gluon_kernels + these variants) -- +GLUON_PATHS=( + aiter/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py + aiter/ops/triton/_gluon_kernels/gfx1250/attention/pa_decode_sparse.py + aiter/ops/triton/_triton_kernels/attention/pa_decode_sparse.py + aiter/ops/triton/attention/pa_decode_sparse.py +) +SRC="$WS/aiter_src" +if [ ! -d "$SRC/.git" ]; then git clone --filter=blob:none --no-checkout "$AITER_REPO" "$SRC" 2>&1 | tail -1; fi +( cd "$SRC" && git fetch --depth 1 origin "$AITER_SHA" 2>&1 | tail -1 && git checkout -q "$AITER_SHA" -- "${GLUON_PATHS[@]}" ) +for p in "${GLUON_PATHS[@]}"; do + [ -e "$SRC/$p" ] || { echo " MISSING in src: $p"; continue; } + mkdir -p "$(dirname "$ROOT/$p")"; cp -a "$SRC/$p" "$ROOT/$p"; echo " added $p" +done + +# --- aiter #4417: large-token FlyDSL MoE launch/output limits ------------------- +# The base's vendored aiter predates #4417 (merged 2026-07-30) even though the +# nightly itself is 08-12 -- the vendored revision is pinned, it does not track +# aiter main. #4417 guards two large-token limits: stage2 buffer atomics address +# the output with 32-bit byte offsets (>4 GiB walks off the end), and HIP caps +# grid.y at 65535. Neither fires at the DSv4-Pro TP8 shape we measure +# (requires_flydsl_stage2_reduce(65536, 7168, 2) is False -- ~939 MB), so this is +# NOT the fix for the inter_dim=384 profile-run memfault; it is carried because +# it is a real gap in the base that any larger-token sweep row would hit. +# The upstream .diff will NOT apply here (its context postdates aiter's typing +# modernization), so graft the hunks by anchor. +python "$(dirname "$0")/graft_aiter_4417.py" "$ROOT/aiter/ops/flydsl/moe_kernels.py" || \ + { echo " #4417: GRAFT FAILED"; exit 1; } + +# --- vllm wiring PRs (dormant unless the env/backend selects them) -------------- +apply_pr(){ local pr="$1" mf="$ROOT/$2" mk="$3" d="$WS/vllm_$1.diff" + if grep -qF "$mk" "$mf" 2>/dev/null; then echo " #$pr: already present"; return; fi + curl -ksSL -o "$d" "$VLLM_REPO/pull/$pr.diff" || { echo " #$pr: FETCH FAIL"; return 1; } + if ( cd "$ROOT" && git apply -p1 --3way "$d" ) 2>/dev/null || ( cd "$ROOT" && git apply -p1 "$d" ) 2>/dev/null + then echo " #$pr: APPLIED"; else echo " #$pr: FAILED"; return 1; fi +} +apply_pr 51714 "vllm/v1/attention/ops/rocm_aiter_mla_sparse.py" "_DSV4_SPARSE_GLUON" || true +apply_pr 51918 "vllm/config/kernel.py" "flydsl_mega_moe" || true + +# --- verify: py_compile + the ABI-sensitive imports (needs GPU) ------------------ +echo "chk gluon kernel = $([ -f "$ROOT/aiter/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py" ] && echo present || echo MISSING)" +echo "chk aiter #4417 guards = $(grep -c 'requires_flydsl_stage2_reduce\|resolve_flydsl_grid_y_persist_m' "$ROOT/aiter/ops/flydsl/moe_kernels.py" 2>/dev/null) (expect 5)" +echo "chk vllm gluon wiring = $(grep -c '_DSV4_SPARSE_GLUON' "$ROOT/vllm/v1/attention/ops/rocm_aiter_mla_sparse.py" 2>/dev/null)" +python -m py_compile "$ROOT/aiter/ops/triton/attention/pa_decode_sparse.py" \ + "$ROOT/vllm/v1/attention/ops/rocm_aiter_mla_sparse.py" && echo PY_COMPILE_OK || { echo PY_COMPILE_FAIL; exit 1; } +python - <<'PYEOF' +import importlib +for m in ["aiter.jit.core","aiter.ops.triton.attention.pa_decode_sparse", + "vllm.v1.attention.ops.rocm_aiter_mla_sparse","vllm._aiter_ops"]: + try: importlib.import_module(m); print("IMPORT_OK",m) + except Exception as e: print("IMPORT_ERR",m,type(e).__name__,(str(e).splitlines() or [''])[-1]) +# the exact symbol the wholesale overlay dropped: +try: + import aiter.jit.core as c + print("has _set_current_hip_stream in core:", hasattr(c,"_set_current_hip_stream")) +except Exception as e: print("core probe err",e) +PYEOF +echo "[add] DONE" diff --git a/dsv4_serve_dep8.sh b/dsv4_serve_dep8.sh new file mode 100644 index 000000000..d8ed26a1f --- /dev/null +++ b/dsv4_serve_dep8.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +# DEP8 winner-arm vLLM serve for the patch-route validation on the STOCK nightly +# base (3ee2df30) patched in-place by apply_dsv4_container_patches.sh. This is the +# config the campaign measured as the winner: DP8 + expert-parallel + MegaMoE +# (flydsl_mega_moe, patched) + gluon sparse (VLLM_ROCM_DSV4_SPARSE_GLUON=1, +# patched), MTP num_spec=3 synthetic accept 2.49, kv fp8, gmu 0.8. FSE and the +# 384-shard oracle are TP8-arm only, so they stay OFF here. +set -uo pipefail + +# The campaign served /models/DeepSeek-V4-Pro (deepseek_v4_fp8 quant: fp8 +# weight-block + fp4 experts). That is /it-shared/models/DeepSeek-V4-Pro here. +# NOT the amd--DeepSeek-V4-Pro-MXFP4 HF snapshot (Quark MXFP4 -- different +# merged-column packing, fails the load_merged_column_weight shape assert). +MODEL_PATH=/it-shared/models/DeepSeek-V4-Pro +SERVED=deepseek-ai/DeepSeek-V4-Pro +PORT=8000 +LOG=/home/jiacao/InferenceX/dsv4-serve.log + +export VLLM_ROCM_USE_AITER=1 +export VLLM_ROCM_USE_AITER_MOE=1 +export VLLM_ROCM_DSV4_SPARSE_GLUON=1 +export VLLM_ENGINE_READY_TIMEOUT_S=10800 + +exec vllm serve "$MODEL_PATH" --served-model-name "$SERVED" \ + --host 0.0.0.0 --port "$PORT" --trust-remote-code \ + --async-scheduling --distributed-executor-backend mp --kv-cache-dtype fp8 \ + --tensor-parallel-size 1 --data-parallel-size 8 --enable-expert-parallel \ + --gpu-memory-utilization 0.8 --moe-backend flydsl_mega_moe \ + --compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}' \ + --speculative-config '{"method":"mtp","num_speculative_tokens":3,"rejection_sample_method":"synthetic","synthetic_acceptance_length":2.49}' \ + --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --reasoning-parser deepseek_v4 \ + --enable-auto-tool-choice --enable-prefix-caching --no-disable-hybrid-kv-cache-manager \ + --max-num-seqs 64 > "$LOG" 2>&1 diff --git a/dsv4_serve_stock.sh b/dsv4_serve_stock.sh new file mode 100644 index 000000000..a875b7744 --- /dev/null +++ b/dsv4_serve_stock.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +# CONTROL ARM: serve DSv4-Pro FP4 TP8 on the *pristine* 08-12 nightly, with no +# patches applied at all. Same model, same parallelism, same MTP/compile config +# as dsv4_serve_tp8.sh -- the only difference is the container has an untouched +# aiter/vllm tree. +# +# Purpose: the profile-run memfault has only ever been observed on the PATCHED +# container. Nothing established that the patch is innocent. In particular +# apply/dsv4_patch_additive.sh copies four gluon kernel files out of +# aiter@97d0c6e4 into the nightly's much older aiter tree, and those two aiter +# revisions differ semantically in 418 files -- exactly the setup that imports +# cleanly and then faults at runtime. The earlier VLLM_ROCM_DSV4_SPARSE_GLUON=0 +# A/B did NOT clear it: that env only avoids one call path, the foreign files +# were still on sys.path. +# +# If this arm serves, the fault is ours and the patch is the bug. +# If this arm faults too, the fault is in the stock base and the patch is clean. +set -uo pipefail + +MODEL_PATH=/it-shared/models/DeepSeek-V4-Pro +SERVED=deepseek-ai/DeepSeek-V4-Pro +PORT=8000 +LOG=/home/jiacao/InferenceX/dsv4-serve-stock.log + +export VLLM_ROCM_USE_AITER=1 +export VLLM_ROCM_USE_AITER_MOE=1 +# No gluon knob here: VLLM_ROCM_DSV4_SPARSE_GLUON is introduced by vllm PR #51714, +# which this container does not have. Stock takes its default sparse-MLA path. +export VLLM_ENGINE_READY_TIMEOUT_S=10800 + +exec vllm serve "$MODEL_PATH" --served-model-name "$SERVED" \ + --host 0.0.0.0 --port "$PORT" --trust-remote-code \ + --async-scheduling --distributed-executor-backend mp --kv-cache-dtype fp8 \ + --tensor-parallel-size 8 --data-parallel-size 1 \ + --gpu-memory-utilization 0.8 --moe-backend aiter \ + --compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}' \ + --speculative-config '{"method":"mtp","num_speculative_tokens":3,"rejection_sample_method":"synthetic","synthetic_acceptance_length":2.49}' \ + --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --reasoning-parser deepseek_v4 \ + --enable-auto-tool-choice --enable-prefix-caching --no-disable-hybrid-kv-cache-manager \ + --max-num-seqs 64 > "$LOG" 2>&1 diff --git a/dsv4_serve_tp8.sh b/dsv4_serve_tp8.sh new file mode 100644 index 000000000..6e5d5b585 --- /dev/null +++ b/dsv4_serve_tp8.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# TP8 arm vLLM serve on the 08-12 nightly + as many campaign PRs as can be +# carried safely. Goal here is RUNNABLE FIRST, maximum PR coverage second. +# +# Carried: aiter #4417 (in-file guards), vllm #51714 (gluon sparse wiring), +# vllm #51918 (flydsl_mega_moe kernel config). +# Also out: MegaMoE/DEP8 (needs mori.ir.flydsl, absent from the base) and FSE +# (#4269 needs aiter fhmoe.py, also absent). +# +# CORRECTION -- do not repeat this mistake. An earlier revision of this header +# claimed the container carried "four aiter@97d0c6e4 gluon kernel FILES" that +# had to be deleted as a cross-version transplant. That was wrong: +# aiter/ops/triton/attention/pa_decode_sparse.py SHIPS IN THE STOCK NIGHTLY. +# Diffing the attention dir between a pristine `dsv4stock` container and the +# working one shows dsv4v is MISSING that base file and has nothing extra -- +# the deletion damaged the base rather than removing foreign code. Any container +# built from this script must keep the base file; VLLM_ROCM_DSV4_SPARSE_GLUON=0 +# below is enough to keep the gluon path dormant, since #51714's call site is a +# lazy in-function import (rocm_aiter_mla_sparse.py:2211) behind that env. +set -uo pipefail + +MODEL_PATH=/it-shared/models/DeepSeek-V4-Pro +SERVED=deepseek-ai/DeepSeek-V4-Pro +PORT=8000 +LOG=/home/jiacao/InferenceX/dsv4-serve.log + +export VLLM_ROCM_USE_AITER=1 +export VLLM_ROCM_USE_AITER_MOE=1 +# Default off: #51714's gluon sparse-MLA path is not part of the runnable-first +# baseline. The kernel it dispatches to (pa_decode_sparse.py) is present in the +# stock nightly, so this can be flipped to 1 to A/B the gluon arm once the +# baseline serves. +export VLLM_ROCM_DSV4_SPARSE_GLUON=${VLLM_ROCM_DSV4_SPARSE_GLUON:-0} +# Probe knobs for the profile-run memfault, NOT a fix. Both default to aiter's +# own values so a plain run reproduces the campaign config. +# +# The MoE at inter_dim=384 is NOT the fault, despite being the loudest thing in +# the log. Two dead theories, recorded so nobody re-derives them: +# +# 1. "the tuned row names an illegal opus stage2 tile". Wrong. opus's 256 is a +# *logical fp4* K step; packed it is K_STEP_PACKED=128, the kernel validates +# effective_inter_dim % 128, and 384 % 128 == 0. 384 is a first-class opus +# codegen seed (OPUS_A8W4_CODEGEN_SEED_EFFECTIVE_INTER_DIMS). FlyDSL's +# tile_n/tile_k must divide inter_dim; opus's step need only divide by 128. +# The two families collide only in naming. +# 2. "stage1 silently drops to tile_n=128 and opus stage2 still reads a 256 +# layout". Also wrong: stage1's output buffer is (token, topk, inter_dim), +# independent of tile_n, and .repro/moe384.py drives this exact shape +# standalone -- preshuffled weights, same kernel pair, same tile_n downgrade +# warning -- and returns finite results. The kernel is fine. +# +# What the log actually shows: TP6 finishes MoE, loads bf16_tuned_gemm.csv, and +# faults there; the other seven workers are parked on that CSV's baton lock and +# survive. Immediately after come M:65536/M:16384 x N:7168 x K:7168 bf16 GEMMs +# with no tuned entry, running default configs. The fault is in the post-MoE +# bf16 GEMM, not the MoE. +export AITER_BYPASS_TUNE_CONFIG=${AITER_BYPASS_TUNE_CONFIG:-0} +export AITER_FLYDSL_FORCE=${AITER_FLYDSL_FORCE:-1} +export VLLM_ENGINE_READY_TIMEOUT_S=10800 + +# Overridable so the profile-run peak can be walked down without editing this +# file. The profile run builds a dummy batch of max_num_batched_tokens (16384) +# x the MTP fan-out (num_speculative_tokens=3 -> 4), hence the M:65536 GEMMs in +# the log; that transient is the largest allocation the server ever makes. +GPU_MEM_UTIL=${GPU_MEM_UTIL:-0.8} + +exec vllm serve "$MODEL_PATH" --served-model-name "$SERVED" \ + --host 0.0.0.0 --port "$PORT" --trust-remote-code \ + --async-scheduling --distributed-executor-backend mp --kv-cache-dtype fp8 \ + --tensor-parallel-size 8 --data-parallel-size 1 \ + --gpu-memory-utilization "$GPU_MEM_UTIL" --moe-backend aiter \ + --compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}' \ + --speculative-config '{"method":"mtp","num_speculative_tokens":3,"rejection_sample_method":"synthetic","synthetic_acceptance_length":2.49}' \ + --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --reasoning-parser deepseek_v4 \ + --enable-auto-tool-choice --enable-prefix-caching --no-disable-hybrid-kv-cache-manager \ + --max-num-seqs 64 > "$LOG" 2>&1 diff --git a/dsv4_serve_tp8_forcereduce.sh b/dsv4_serve_tp8_forcereduce.sh new file mode 100755 index 000000000..5b08e4df2 --- /dev/null +++ b/dsv4_serve_tp8_forcereduce.sh @@ -0,0 +1,6 @@ +#!/bin/bash +# TP8 arm with AITER_FLYDSL_FORCE_REDUCE=1 — diagnostic for the missing aiter #4417 +# guards in the 08-12 nightly (stage2 buffer atomics overflow 32-bit offsets). +set -euo pipefail +export AITER_FLYDSL_FORCE_REDUCE=1 +exec bash "$(dirname "$0")/dsv4_serve_tp8.sh" diff --git a/dsv4_serve_tp8_notiter.sh b/dsv4_serve_tp8_notiter.sh new file mode 100755 index 000000000..7782c095a --- /dev/null +++ b/dsv4_serve_tp8_notiter.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# TP8 arm vLLM serve -- the config the patch route CAN fully reproduce on a stock +# nightly (moe-backend aiter, so no MegaMoE/flydsl -> no mori.ir.flydsl dep), +# gluon sparse on (VLLM_ROCM_DSV4_SPARSE_GLUON=1, patched), base #4269 FSE on +# (VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1), 384-shard native in the 08-12 +# base. Used to validate the patched container actually serves DSv4-Pro FP4 at +# 8k/1k c32. (DEP8/MegaMoE is blocked by a mori version gap -- see notes.) +set -uo pipefail + +MODEL_PATH=/it-shared/models/DeepSeek-V4-Pro +SERVED=deepseek-ai/DeepSeek-V4-Pro +PORT=8000 +LOG=/home/jiacao/InferenceX/dsv4-serve-triton.log + +export VLLM_ROCM_USE_AITER=1 +export VLLM_ROCM_USE_AITER_MOE=0 +# FSE (#4269) intentionally OFF: the 08-12 base aiter has no fhmoe.py, and the +# additive route does not restore it. +# gluon: the additively-injected gluon sparse-MLA kernel (from aiter@97d0c6e4) +# GPU-faults during memory profiling on the 08-12 base aiter runtime +# (hc_head_fuse_tilelang path, VllmWorker GPU coredump). Overridable so we can +# validate a servable baseline with gluon OFF, then flip it back on to reproduce. +export VLLM_ROCM_DSV4_SPARSE_GLUON=${VLLM_ROCM_DSV4_SPARSE_GLUON:-1} +export VLLM_ENGINE_READY_TIMEOUT_S=10800 + +exec vllm serve "$MODEL_PATH" --served-model-name "$SERVED" \ + --host 0.0.0.0 --port "$PORT" --trust-remote-code \ + --async-scheduling --distributed-executor-backend mp --kv-cache-dtype fp8 \ + --tensor-parallel-size 8 --data-parallel-size 1 \ + --gpu-memory-utilization 0.8 --moe-backend triton \ + --compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}' \ + --speculative-config '{"method":"mtp","num_speculative_tokens":3,"rejection_sample_method":"synthetic","synthetic_acceptance_length":2.49}' \ + --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --reasoning-parser deepseek_v4 \ + --enable-auto-tool-choice --enable-prefix-caching --no-disable-hybrid-kv-cache-manager \ + --max-num-seqs 64 > "$LOG" 2>&1 diff --git a/dsv4_serve_tp8_triton.sh b/dsv4_serve_tp8_triton.sh new file mode 100755 index 000000000..05b4aa8ea --- /dev/null +++ b/dsv4_serve_tp8_triton.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# TP8 arm vLLM serve -- the config the patch route CAN fully reproduce on a stock +# nightly (moe-backend aiter, so no MegaMoE/flydsl -> no mori.ir.flydsl dep), +# gluon sparse on (VLLM_ROCM_DSV4_SPARSE_GLUON=1, patched), base #4269 FSE on +# (VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1), 384-shard native in the 08-12 +# base. Used to validate the patched container actually serves DSv4-Pro FP4 at +# 8k/1k c32. (DEP8/MegaMoE is blocked by a mori version gap -- see notes.) +set -uo pipefail + +MODEL_PATH=/it-shared/models/DeepSeek-V4-Pro +SERVED=deepseek-ai/DeepSeek-V4-Pro +PORT=8000 +LOG=/home/jiacao/InferenceX/dsv4-serve.log + +export VLLM_ROCM_USE_AITER=1 +export VLLM_ROCM_USE_AITER_MOE=1 +# FSE (#4269) intentionally OFF: the 08-12 base aiter has no fhmoe.py, and the +# additive route does not restore it. +# gluon: the additively-injected gluon sparse-MLA kernel (from aiter@97d0c6e4) +# GPU-faults during memory profiling on the 08-12 base aiter runtime +# (hc_head_fuse_tilelang path, VllmWorker GPU coredump). Overridable so we can +# validate a servable baseline with gluon OFF, then flip it back on to reproduce. +export VLLM_ROCM_DSV4_SPARSE_GLUON=${VLLM_ROCM_DSV4_SPARSE_GLUON:-1} +export VLLM_ENGINE_READY_TIMEOUT_S=10800 + +exec vllm serve "$MODEL_PATH" --served-model-name "$SERVED" \ + --host 0.0.0.0 --port "$PORT" --trust-remote-code \ + --async-scheduling --distributed-executor-backend mp --kv-cache-dtype fp8 \ + --tensor-parallel-size 8 --data-parallel-size 1 \ + --gpu-memory-utilization 0.8 --moe-backend triton \ + --compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}' \ + --speculative-config '{"method":"mtp","num_speculative_tokens":3,"rejection_sample_method":"synthetic","synthetic_acceptance_length":2.49}' \ + --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --reasoning-parser deepseek_v4 \ + --enable-auto-tool-choice --enable-prefix-caching --no-disable-hybrid-kv-cache-manager \ + --max-num-seqs 64 > "$LOG" 2>&1 diff --git a/dsv4_wait_then_bench.sh b/dsv4_wait_then_bench.sh new file mode 100644 index 000000000..e36b86838 --- /dev/null +++ b/dsv4_wait_then_bench.sh @@ -0,0 +1,34 @@ +#!/usr/bin/env bash +# Poll the DEP8 server /health until ready (cold 805 GiB NFS load ~2h + compile + +# cudagraph capture), then fire the 8k/1k c32 benchmark. Runs detached in the +# container; watch /home/jiacao/InferenceX/dsv4-waitbench.log. +set -uo pipefail +STAMP(){ date -u +%FT%TZ; } +WLOG=/home/jiacao/InferenceX/dsv4-waitbench.log +SLOG=/home/jiacao/InferenceX/dsv4-serve.log + +echo "$(STAMP) waiting for http://localhost:8000/health ..." >> "$WLOG" +for i in $(seq 1 720); do # 720 * 20s = 4h ceiling + code=$(curl -s -o /dev/null -w '%{http_code}' http://localhost:8000/health 2>/dev/null || echo 000) + if [ "$code" = "200" ]; then + echo "$(STAMP) server READY (health 200) after $((i*20))s" >> "$WLOG" + break + fi + # bail early if the server process died + if ! pgrep -f "vllm serve" >/dev/null 2>&1; then + echo "$(STAMP) FATAL: vllm serve process gone; tail serve log:" >> "$WLOG" + tail -40 "$SLOG" >> "$WLOG" 2>&1 + exit 1 + fi + sleep 20 +done + +code=$(curl -s -o /dev/null -w '%{http_code}' http://localhost:8000/health 2>/dev/null || echo 000) +if [ "$code" != "200" ]; then + echo "$(STAMP) FATAL: health never reached 200 (last=$code)" >> "$WLOG" + exit 1 +fi + +echo "$(STAMP) launching 8k/1k c32 benchmark ..." >> "$WLOG" +bash /home/jiacao/InferenceX/.claude/worktrees/dsv4-ci-sweep-pr/dsv4_bench_8k1k_c32.sh +echo "$(STAMP) benchmark finished rc=$? ; results in dsv4-bench-8k1k-c32.json / .log" >> "$WLOG" diff --git a/graft_aiter_4417.py b/graft_aiter_4417.py new file mode 100644 index 000000000..f347870ad --- /dev/null +++ b/graft_aiter_4417.py @@ -0,0 +1,125 @@ +#!/usr/bin/env python3 +"""Graft ROCm/aiter #4417 into an aiter that predates it. + +#4417 "Fix large-token FlyDSL MoE launch and output limits" (merged 2026-07-30, +single file, +32/-2) adds two guards to aiter/ops/flydsl/moe_kernels.py: + + * requires_flydsl_stage2_reduce() -- stage2 buffer atomics address the output + with 32-bit byte offsets, so a >4 GiB output silently walks off the end. + The guard flips mode to "reduce" before that happens. + * resolve_flydsl_grid_y_persist_m() -- HIP caps grid.y at 65535; the guard + raises persist_m enough to keep the launch legal. + +Neither guard fires at the DSv4-Pro TP8 shape this campaign measures -- +requires_flydsl_stage2_reduce(65536, 7168, 2) is False (~939 MB, well under the +4 GiB threshold) -- so #4417 does NOT explain the inter_dim=384 profile-run +memfault. That fault is a stage2 tile_k mismatch: the tuned CSV names +opus_moe2_*_t64x256x256_* for inter_dim=384, and resolve_flydsl_stage2_tile_k +only guards flydsl_* kernel names, so tile_k=256 runs against K=384. #4417 is +grafted anyway because it is a genuine gap in the base that any larger-token +sweep row would hit. + +The upstream .diff does NOT apply to the 08-12 nightly base: that aiter predates +both #4417 and aiter's typing modernization, so every context line still reads +`Dict[str, Dict]` / `Optional[X]` where the diff expects `dict[str, dict]` / +`X | None`. The three hunks are grafted here by anchor instead of by context. +Idempotent: re-running on an already-grafted (or already-new) file is a no-op. +""" + +import sys + +HELPERS = ''' + +def resolve_flydsl_grid_y_persist_m( + num_m_blocks: int, requested_persist_m: int = 0 +) -> int: + """Increase persist_m as needed to keep grid.y within HIP's limit.""" + num_m_blocks = max(int(num_m_blocks), 0) + requested_persist_m = max(int(requested_persist_m), 1) + required_persist_m = max( + 1, (num_m_blocks + _HIP_MAX_GRID_DIM_Y - 1) // _HIP_MAX_GRID_DIM_Y + ) + return max(requested_persist_m, required_persist_m) + + +def requires_flydsl_stage2_reduce( + token_num: int, model_dim: int, element_size: int +) -> bool: + """Return whether stage2 atomic output exceeds 32-bit byte offsets.""" + return int(token_num) * int(model_dim) * int(element_size) > 0xFFFFFFFF + +''' + + +def graft(path): + src = open(path).read() + if "requires_flydsl_stage2_reduce" in src and "_HIP_MAX_GRID_DIM_Y" in src: + print("SKIP already grafted/present:", path) + return 0 + orig = src + done = [] + + # hunk 1a: the grid.y constant, right after the _KERNEL_PARAMS declaration. + a = "_KERNEL_PARAMS: Dict[str, Dict] = {}\n" + if a not in src: + a = "_KERNEL_PARAMS: dict[str, dict] = {}\n" + if src.count(a) != 1: + print("FAIL anchor _KERNEL_PARAMS count=", src.count(a)) + return 1 + src = src.replace(a, a + "\n# HIP limits grid.y/grid.z to 65535.\n" + "_HIP_MAX_GRID_DIM_Y = 65535\n", 1) + done.append("const") + + # hunk 1b: the two helpers, ahead of resolve_flydsl_stage2_tile_k. + b = "\ndef resolve_flydsl_stage2_tile_k(" + if src.count(b) != 1: + print("FAIL anchor stage2_tile_k count=", src.count(b)) + return 1 + src = src.replace(b, HELPERS + "\ndef resolve_flydsl_stage2_tile_k(", 1) + done.append("helpers") + + # hunk 2: stage1 caps grid.y through persist_m. + c = " _persist_m = persist_m if persist_m > 0 else 1\n" + if src.count(c) != 1: + print("FAIL anchor stage1 persist_m count=", src.count(c)) + return 1 + src = src.replace(c, " _persist_m = resolve_flydsl_grid_y_persist_m(_grid_y, persist_m)\n", 1) + done.append("stage1_persist") + + # hunk 3a: stage2 falls back to reduce when the atomic output exceeds 4 GiB. + d = (' if os.environ.get("AITER_FLYDSL_FORCE_REDUCE", "0") == "1":\n' + ' mode = "reduce"\n') + if src.count(d) != 1: + print("FAIL anchor FORCE_REDUCE count=", src.count(d)) + return 1 + src = src.replace(d, d + ( + " elif (\n" + " mode != \"reduce\"\n" + " and not return_per_slot\n" + " and requires_flydsl_stage2_reduce(token_num, model_dim, 2)\n" + " ):\n" + " # Buffer atomics use 32-bit offsets; reduce outputs larger than 4 GiB.\n" + " mode = \"reduce\"\n"), 1) + done.append("stage2_reduce") + + # hunk 3b: fp8 stage2 is non-persistent, so cap grid.y the same way. + e = ' if a_dtype == "fp8":\n _persist_m = 1\n' + if src.count(e) != 1: + print("FAIL anchor fp8 persist_m count=", src.count(e)) + return 1 + src = src.replace(e, ' if a_dtype == "fp8":\n' + ' # FP8 uses non-persistent scheduling, so cap grid.y via persist_m.\n' + ' _persist_m = resolve_flydsl_grid_y_persist_m(m_blocks)\n', 1) + done.append("stage2_fp8_persist") + + if src == orig: + print("FAIL no change") + return 1 + open(path + ".pre4417", "w").write(orig) + open(path, "w").write(src) + print("GRAFTED", path, "hunks:", ",".join(done)) + return 0 + + +if __name__ == "__main__": + sys.exit(graft(sys.argv[1])) diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 6d484a53e..ef4ef8234 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -5929,4 +5929,4 @@ - "Patch stack is strictly additive and idempotent: vllm #51473 (native MXFP4 TP8 shard allocation, merged 2026-08-11 i.e. after this base), vllm #51714 (opt-in AITER gluon sparse-MLA decode for gfx950), vllm #51918 config/kernel.py only (registers the flydsl_mega_moe backend name), aiter #4417 (large-token FlyDSL MoE launch/output limits, grafted by anchor), plus the one gluon kernel file the base lacks" - "Nothing that ships in the base is overwritten: three of the four pa_decode_sparse variants already ship and are left untouched, aiter core is not modified, and the MegaMoE model-side hunks of #51918 are deliberately not applied because the base has no mori.ir.flydsl" - "Enable VLLM_ROCM_DSV4_SPARSE_GLUON=1 by default (overridable); the knob is read via os.environ rather than envs.py, so vLLM logs it as an unknown environment variable" - pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/PRNUM + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2589 diff --git a/sbatch_3way_patchset.sh b/sbatch_3way_patchset.sh new file mode 100644 index 000000000..cbae44722 --- /dev/null +++ b/sbatch_3way_patchset.sh @@ -0,0 +1,256 @@ +#!/usr/bin/env bash +#SBATCH --job-name=3way-patch +#SBATCH --account=amd-aifw-aim +#SBATCH --qos=amd-aifw-aim-qos +#SBATCH --partition=amd-spur +#SBATCH --nodes=1 +#SBATCH --gpus-per-node=0 +#SBATCH --time=02:00:00 +#SBATCH --output=/home/jiacao/InferenceX/3way-%j.out +#SBATCH --exclude=crsuse2-m2m-203,crsuse2-m2m-074,crsuse2-m2m-240,crsuse2-m2m-029,crsuse2-m2m-061,crsuse2-m2m-071 +# +# Extract the *vendor patch set* out of the DSv4 measurement image and port it +# onto the intended CI baseline. +# +# A straight VENDOR-vs-TARGET diff (job 6316) is useless for this: it returned +# 769 vllm files / 4.8 MB, because TARGET is 454 upstream commits AHEAD of the +# vendor image's base, so the diff is dominated by upstream drift running the +# wrong direction. The vendor's own changes are buried in it. +# +# Three-way subtraction fixes that. REF is a public ROCm nightly built from +# 6f91edf9 (2026-07-29), only 11 commits behind the vendor image's own base +# 02e63f2e4 (2026-07-30). So: +# +# REF ---- 11 commits ----> vendor base 02e63f2e4 --(vendor patches)--> VENDOR +# REF ---- 465 commits ---> TARGET b22afe45 +# +# diff(REF, VENDOR) = 11 commits of drift + THE VENDOR PATCH SET <-- small +# diff(REF, TARGET) = 465 commits of upstream <-- large +# +# The first is the deliverable. Anything it touches that the second also +# touches is a conflict that needs a human decision; anything it touches that +# the second does not is a clean port onto TARGET. +# +# Deliverables under $OUT: +# vendor_patchset.{vllm,aiter}.diff - REF -> VENDOR, the thing to port +# upstream_drift.vllm.diff - REF -> TARGET, for conflict detection +# CONFLICTS.txt - files both sides touch +# CLEAN.txt - files only the vendor touches +# applied/ - TARGET tree with the patch set applied +# APPLY_LOG.txt - per-file apply result +set -uo pipefail + +REF="${REF:-vllm/vllm-openai-rocm:nightly-6f91edf96d3f3272945809c04702380053bff4de}" +VENDOR="${VENDOR:-jiahcao/vllm-dsv4:dsv4-pr51473-20260812}" +TARGET="${TARGET:-vllm/vllm-openai-rocm:nightly-b22afe45ac797ae58e67a7a3ad79ee5714024420}" +OUT="${OUT:-/home/jiacao/3way-$(date -u +%Y%m%d-%H%M)}" + +hostname; date -u +echo "REF : $REF (6f91edf9, 2026-07-29)" +echo "VENDOR : $VENDOR (base 02e63f2e4, 2026-07-30)" +echo "TARGET : $TARGET (b22afe45, 2026-08-10)" +echo "OUT : $OUT" +echo + +fail() { echo "FATAL: $*" >&2; exit 1; } +docker info >/dev/null 2>&1 || fail "no docker access on $(hostname)" +rm -rf "$OUT"; mkdir -p "$OUT" || fail "cannot create $OUT" + +# ---------------------------------------------------------------- pull +for img in "$REF" "$VENDOR" "$TARGET"; do + for _try in 1 2 3; do + docker image inspect "$img" >/dev/null 2>&1 && break + echo "pulling $img (attempt $_try/3) ... $(date -u +%H:%M:%S)Z" + docker pull "$img" || true + done + docker image inspect "$img" >/dev/null 2>&1 || fail "pull failed: $img" +done +echo "PULL OK $(date -u +%H:%M:%S)Z" + +# ---------------------------------------------------------------- locate +# Layouts differ between images: the nightlies install into dist-packages, the +# vendor image carries an editable /src/vllm checkout. Ask each image where it +# imports from rather than assuming (job 6309 extracted 0 bytes by assuming). +locate_in() { # $1=image $2=pkg + docker run --rm --entrypoint /bin/bash "$1" -c \ + "python3 -c 'import importlib.util as u,os +s=u.find_spec(\"$2\") +print(os.path.dirname(s.origin) if s and s.origin else \"\")' 2>/dev/null" 2>/dev/null | tr -d '\r' | tail -1 +} + +declare -A P +for side in ref vendor target; do + case $side in ref) img="$REF";; vendor) img="$VENDOR";; target) img="$TARGET";; esac + for pkg in vllm aiter; do + P[$side:$pkg]=$(locate_in "$img" "$pkg") + echo " $side/$pkg -> ${P[$side:$pkg]:-}" + done +done +echo + +# ---------------------------------------------------------------- extract +extract() { # $1=image $2=destdir $3..=in-container dirs + local img="$1" dst="$2"; shift 2 + local cid pats=() + mkdir -p "$dst" + for p in "$@"; do [ -n "$p" ] && pats+=("${p#/}/*"); done + [ ${#pats[@]} -eq 0 ] && return 1 + cid=$(docker create "$img") || fail "docker create failed: $img" + docker export "$cid" | tar -x -C "$dst" --wildcards "${pats[@]}" 2>/dev/null + docker rm -f "$cid" >/dev/null 2>&1 +} + +for side in ref vendor target; do + case $side in ref) img="$REF";; vendor) img="$VENDOR";; target) img="$TARGET";; esac + echo "extracting $side ... $(date -u +%H:%M:%S)Z" + extract "$img" "$OUT/$side" "${P[$side:vllm]}" "${P[$side:aiter]}" + echo " $(du -sh "$OUT/$side" 2>/dev/null | cut -f1)" +done + +root() { # $1=side $2=pkg -> on-disk package root + local d="$OUT/$1/${P[$1:$2]#/}" + [ -d "$d" ] && { echo "$d"; return; } + find "$OUT/$1" -type f -path "*/$2/__init__.py" 2>/dev/null \ + | awk '{print length"\t"$0}' | sort -n | head -1 | cut -f2- | xargs -r dirname +} + +# ---------------------------------------------------------------- diff +# Normalise away the layout difference: diff the package dirs directly and +# rewrite the labels to bare a/ and b/ so the resulting patch applies +# with -p1 regardless of where each image kept the package. +DIFFOPTS=(-ruN -x '__pycache__' -x '*.pyc' -x '*.so' -x '*.pyd' -x '_version.py') + +echo +for pkg in vllm aiter; do + r=$(root ref "$pkg"); v=$(root vendor "$pkg"); t=$(root target "$pkg") + echo "=============== $pkg ===============" + echo " ref : ${r:-}" + echo " vendor: ${v:-}" + echo " target: ${t:-}" + if [ -z "$r" ] || [ -z "$v" ] || [ -z "$t" ]; then + echo " SKIP: missing on one side"; continue + fi + + # (1) THE DELIVERABLE: ref -> vendor. 11 commits of drift + vendor patches. + ( cd "$(dirname "$r")" && diff "${DIFFOPTS[@]}" \ + --label "a/$pkg" --label "b/$pkg" "$pkg" "$v" ) \ + > "$OUT/vendor_patchset.$pkg.diff" 2>/dev/null + # (2) ref -> target: 465 upstream commits, used only to find conflicts. + ( cd "$(dirname "$r")" && diff "${DIFFOPTS[@]}" \ + --label "a/$pkg" --label "b/$pkg" "$pkg" "$t" ) \ + > "$OUT/upstream_drift.$pkg.diff" 2>/dev/null + + vs=$(stat -c%s "$OUT/vendor_patchset.$pkg.diff" 2>/dev/null || echo 0) + us=$(stat -c%s "$OUT/upstream_drift.$pkg.diff" 2>/dev/null || echo 0) + vf=$(grep -c '^diff ' "$OUT/vendor_patchset.$pkg.diff" 2>/dev/null || echo 0) + uf=$(grep -c '^diff ' "$OUT/upstream_drift.$pkg.diff" 2>/dev/null || echo 0) + echo " vendor patch set : $vf files, $vs bytes <-- to port" + echo " upstream drift : $uf files, $us bytes" + + # Real line counts. (Job 6316's per-file awk under-reported removals; total + # +/- over the whole file is the number to trust.) + printf " vendor patch set : +%s / -%s lines\n" \ + "$(grep -c '^+[^+]' "$OUT/vendor_patchset.$pkg.diff" 2>/dev/null || echo 0)" \ + "$(grep -c '^-[^-]' "$OUT/vendor_patchset.$pkg.diff" 2>/dev/null || echo 0)" + + # ---- conflict analysis ------------------------------------------------- + grep '^diff ' "$OUT/vendor_patchset.$pkg.diff" 2>/dev/null \ + | awk '{print $NF}' | sed "s|^$v/||;s|^$pkg/||" | sort -u > "$OUT/.v.$pkg" + grep '^diff ' "$OUT/upstream_drift.$pkg.diff" 2>/dev/null \ + | awk '{print $NF}' | sed "s|^$t/||;s|^$pkg/||" | sort -u > "$OUT/.u.$pkg" + comm -12 "$OUT/.v.$pkg" "$OUT/.u.$pkg" | sed "s|^|$pkg/|" >> "$OUT/CONFLICTS.txt" + comm -23 "$OUT/.v.$pkg" "$OUT/.u.$pkg" | sed "s|^|$pkg/|" >> "$OUT/CLEAN.txt" + echo " files vendor-only (clean port): $(comm -23 "$OUT/.v.$pkg" "$OUT/.u.$pkg" | wc -l)" + echo " files touched by BOTH (conflict): $(comm -12 "$OUT/.v.$pkg" "$OUT/.u.$pkg" | wc -l)" +done + +# ---------------------------------------------------------------- apply +# Apply the vendor patch set onto a copy of TARGET, file by file, so a failure +# names the file instead of aborting the whole port. git apply is tried first +# (exact), then patch --fuzz=3 (tolerant of the 11-commit drift), and finally +# a whole-file copy for files that exist only in the vendor image -- those are +# additions, not modifications, and are the bulk of the MegaMoE kernel tree. +echo +echo "=============== APPLY onto TARGET ===============" +: > "$OUT/APPLY_LOG.txt" +for pkg in vllm aiter; do + v=$(root vendor "$pkg"); t=$(root target "$pkg") + [ -z "$v" ] || [ -z "$t" ] && continue + dst="$OUT/applied/$pkg" + mkdir -p "$(dirname "$dst")" + cp -a "$t" "$dst" || fail "cannot stage TARGET copy for $pkg" + + ok=0; fuzz=0; added=0; bad=0 + # Walk the vendor tree: every .py that differs from the staged target. + while IFS= read -r f; do + rel="${f#$v/}" + if [ ! -e "$dst/$rel" ]; then + mkdir -p "$(dirname "$dst/$rel")" + cp -a "$f" "$dst/$rel" && { added=$((added+1)); \ + echo "ADD $pkg/$rel" >> "$OUT/APPLY_LOG.txt"; } \ + || { bad=$((bad+1)); echo "FAIL-ADD $pkg/$rel" >> "$OUT/APPLY_LOG.txt"; } + continue + fi + cmp -s "$f" "$dst/$rel" && continue + # Modified on both sides: produce a patch from ref->vendor for just this + # file and try to apply it to the target copy. + r=$(root ref "$pkg") + if [ -e "$r/$rel" ]; then + diff -u --label "a/$rel" --label "b/$rel" "$r/$rel" "$f" > "$OUT/.one.diff" 2>/dev/null + if patch -s -p1 -d "$dst" --fuzz=3 --forward --no-backup-if-mismatch \ + -i "$OUT/.one.diff" >/dev/null 2>&1; then + fuzz=$((fuzz+1)); echo "PATCH $pkg/$rel" >> "$OUT/APPLY_LOG.txt" + else + bad=$((bad+1)); echo "CONFLICT $pkg/$rel" >> "$OUT/APPLY_LOG.txt" + fi + else + # No ref version: vendor rewrote a file that upstream also changed. + bad=$((bad+1)); echo "CONFLICT-NOREF $pkg/$rel" >> "$OUT/APPLY_LOG.txt" + fi + done < <(find "$v" -type f -name '*.py' ! -path '*__pycache__*' 2>/dev/null) + + echo " $pkg: added=$added patched=$fuzz conflicts=$bad" +done +rm -f "$OUT/.one.diff" "$OUT"/.v.* "$OUT"/.u.* + +echo +echo "--- conflicts needing a decision ---" +grep -E '^(CONFLICT|FAIL)' "$OUT/APPLY_LOG.txt" 2>/dev/null | head -60 +echo " total: $(grep -cE '^(CONFLICT|FAIL)' "$OUT/APPLY_LOG.txt" 2>/dev/null || echo 0)" + +# ---------------------------------------------------------------- verify +echo +echo "=============== feature surface after port ===============" +probe() { # $1=label $2=pattern + printf "%-45s" "$1:" + for side in "$OUT/target" "$OUT/applied"; do + n=$(grep -rl -- "$2" "$side" --include='*.py' 2>/dev/null | wc -l) + printf " %-4s" "$n" + done + echo " (target -> applied)" +} +probe "VLLM_ROCM_DSV4_SPARSE_GLUON" "VLLM_ROCM_DSV4_SPARSE_GLUON" +probe "flydsl_mega_moe" "flydsl_mega_moe" +probe "mega_moe (any)" "mega_moe" +probe "#51473 marker" "AITER_MXFP4_BF16 and activation == MoEActivation.SILU" + +echo +echo "=============== python syntax check on applied tree ===============" +# A port that lands syntactically-broken files would only surface 20 minutes +# into a measured cell. Parse every file the port touched. +bad=0 +while IFS= read -r line; do + rel=$(echo "$line" | awk '{print $2}') + f="$OUT/applied/$rel" + [ -f "$f" ] || continue + python3 -c "import ast,sys; ast.parse(open(sys.argv[1],encoding='utf-8').read())" "$f" 2>/dev/null \ + || { echo " SYNTAX FAIL: $rel"; bad=$((bad+1)); } +done < <(grep -E '^(ADD|PATCH)' "$OUT/APPLY_LOG.txt" 2>/dev/null) +echo " syntax failures: $bad" + +echo +echo "DONE $(date -u +%H:%M:%S)Z" +echo "Artifacts: $OUT" +echo " vendor_patchset.{vllm,aiter}.diff <- the port" +echo " CONFLICTS.txt / CLEAN.txt / APPLY_LOG.txt" +echo " applied/ <- TARGET + vendor patch set" diff --git a/sbatch_diff_images.sh b/sbatch_diff_images.sh new file mode 100755 index 000000000..5222cb24d --- /dev/null +++ b/sbatch_diff_images.sh @@ -0,0 +1,183 @@ +#!/usr/bin/env bash +#SBATCH --job-name=img-diff +#SBATCH --account=amd-aifw-aim +#SBATCH --qos=amd-aifw-aim-qos +#SBATCH --partition=amd-spur +#SBATCH --nodes=1 +#SBATCH --gpus-per-node=0 +#SBATCH --time=01:30:00 +#SBATCH --output=/home/jiacao/InferenceX/imgdiff-%j.out +#SBATCH --exclude=crsuse2-m2m-203,crsuse2-m2m-074,crsuse2-m2m-240,crsuse2-m2m-029,crsuse2-m2m-061,crsuse2-m2m-071 +# +# Establish the provenance of the DSv4 measurement image against the intended +# CI base, so every delta can be attributed to an upstream PR (or flagged as +# having none) in a docs/waiver/.md table. +# +# BASE is the image the CI config would pin. NEW is the image every measured +# number in this campaign was produced on (see sbatch_bake_pr51473_image.sh: +# it is vendor base f2fbead + the two-file #51473 back-port). +# +# No GPU is requested: this only untars layers and runs diff. That also lets +# the job schedule immediately alongside the measurement jobs rather than +# queueing behind them. +# +# Run 6309 extracted 0 bytes from BASE: its tar wildcards assumed a layout +# ('*/site-packages/vllm/*.py', '*/vllm/vllm/*.py') that matches the vendor +# image's editable /src/vllm checkout but not the nightly's. Rather than guess +# again, this version extracts the whole filesystem and then locates each +# package by finding its __init__.py. Costs disk and a few minutes; removes the +# failure mode entirely. +set -uo pipefail + +BASE="${BASE:-vllm/vllm-openai-rocm:nightly-b22afe45ac797ae58e67a7a3ad79ee5714024420}" +NEW="${NEW:-jiahcao/vllm-dsv4@sha256:5e44cbd690811bdf9129bd2f552f22241177c6ad0c51bd057afb817cca35e1e9}" +OUT="${OUT:-/home/jiacao/imgdiff-$(date -u +%Y%m%d)-full}" + +hostname; date -u +echo "BASE : $BASE" +echo "NEW : $NEW" +echo "OUT : $OUT" +echo + +fail() { echo "FATAL: $*" >&2; exit 1; } +docker info >/dev/null 2>&1 || fail "no docker access on $(hostname)" +rm -rf "$OUT"; mkdir -p "$OUT" || fail "cannot create $OUT" + +# ---------------------------------------------------------------- pull +for img in "$BASE" "$NEW"; do + for _try in 1 2 3; do + docker image inspect "$img" >/dev/null 2>&1 && break + echo "pulling $img (attempt $_try/3) ... $(date -u +%H:%M:%S)Z" + docker pull "$img" || true + done + docker image inspect "$img" >/dev/null 2>&1 || fail "pull failed: $img" +done +echo "PULL OK $(date -u +%H:%M:%S)Z" + +# ---------------------------------------------------------------- locate +# Ask each image where it actually imports vllm/aiter from, instead of +# assuming. --entrypoint /bin/bash is safe here because nothing imports the +# package (importing aiter runs rocminfo, which aborts without a GPU); this +# only reads importlib's spec origin. +locate_in() { # $1=image $2=pkg -> prints the in-container package dir + docker run --rm --entrypoint /bin/bash "$1" -c \ + "python3 -c 'import importlib.util as u,os,sys +s=u.find_spec(\"$2\") +print(os.path.dirname(s.origin) if s and s.origin else \"\")' 2>/dev/null" 2>/dev/null | tr -d '\r' | tail -1 +} + +declare -A PKGDIR +for side in base new; do + img="$BASE"; [ "$side" = "new" ] && img="$NEW" + for pkg in vllm aiter; do + d=$(locate_in "$img" "$pkg") + PKGDIR[$side:$pkg]="$d" + echo " $side/$pkg -> ${d:-}" + done +done +echo + +# ---------------------------------------------------------------- export +# Extract the located package dirs only. `docker export` streams the flattened +# container filesystem; the tar member paths are relative (no leading /), so +# strip the leading slash off the located path to build the match pattern. +extract() { # $1=image $2=destdir $3..=in-container abs dirs + local img="$1" dst="$2"; shift 2 + local cid pats=() + mkdir -p "$dst" + for p in "$@"; do + [ -n "$p" ] && pats+=("${p#/}/*") && pats+=("${p#/}") + done + [ ${#pats[@]} -eq 0 ] && { echo " no patterns for $img"; return 1; } + cid=$(docker create "$img") || fail "docker create failed for $img" + docker export "$cid" | tar -x -C "$dst" --wildcards "${pats[@]}" 2>/dev/null + docker rm -f "$cid" >/dev/null 2>&1 + return 0 +} + +echo "extracting BASE ... $(date -u +%H:%M:%S)Z" +extract "$BASE" "$OUT/base" "${PKGDIR[base:vllm]}" "${PKGDIR[base:aiter]}" +echo " base tree: $(du -sh "$OUT/base" 2>/dev/null | cut -f1)" +echo "extracting NEW ... $(date -u +%H:%M:%S)Z" +extract "$NEW" "$OUT/new" "${PKGDIR[new:vllm]}" "${PKGDIR[new:aiter]}" +echo " new tree: $(du -sh "$OUT/new" 2>/dev/null | cut -f1)" + +# Resolve back to the extracted-on-disk package root. The in-container path is +# reproduced verbatim under $dst, so join them -- but fall back to a search in +# case the layout surprises us again. +onbox() { # $1=side $2=pkg + local d="$OUT/$1/${PKGDIR[$1:$2]#/}" + [ -d "$d" ] && { echo "$d"; return; } + find "$OUT/$1" -type f -path "*/$2/__init__.py" 2>/dev/null \ + | awk '{print length"\t"$0}' | sort -n | head -1 | cut -f2- | xargs -r dirname +} + +# ---------------------------------------------------------------- diff +for pkg in vllm aiter; do + b=$(onbox base "$pkg"); n=$(onbox new "$pkg") + echo + echo "=============== $pkg ===============" + echo "base: ${b:-}" + echo "new : ${n:-}" + if [ -z "$b" ] || [ -z "$n" ] || [ ! -d "$b" ] || [ ! -d "$n" ]; then + echo "SKIP: $pkg missing on one side" + continue + fi + # Compare .py only: the compiled extensions and .co kernels are opaque to + # diff, and __pycache__ differs on mtime alone. + diff -ruN -x '__pycache__' -x '*.pyc' -x '*.so' "$b" "$n" > "$OUT/$pkg.diff" 2>/dev/null + nfiles=$(grep -c '^diff -ruN' "$OUT/$pkg.diff" 2>/dev/null || echo 0) + echo "changed files: $nfiles" + echo "diff bytes : $(stat -c%s "$OUT/$pkg.diff" 2>/dev/null || echo 0)" + echo + echo "--- per-file churn (+added -removed) ---" + awk -v base="$b" -v new="$n" ' + /^diff -ruN/ { + if (f != "") printf " %+7d %-7d %s\n", a, -r, f + f=$NF; sub(new"/","",f); a=0; r=0; next + } + /^\+/ && !/^\+\+\+/ { a++ } + /^-/ && !/^---/ { r++ } + END { if (f != "") printf " %+7d %-7d %s\n", a, -r, f } + ' "$OUT/$pkg.diff" | sort -k3 +done + +# ---------------------------------------------------------------- surfaces +# The knobs the CI cannot reach are engine-side features, not recipe flags. +# Whether BASE honours them at all is the question the grid depends on. +echo +echo "=============== feature surface: BASE vs NEW ===============" +probe() { # $1=label $2=pattern + echo "$1:" + for side in base new; do + hits=$(grep -rl -- "$2" "$OUT/$side" --include='*.py' 2>/dev/null | wc -l) + printf " %-5s %s file(s)\n" "$side" "$hits" + done +} +probe "VLLM_ROCM_DSV4_SPARSE_GLUON" "VLLM_ROCM_DSV4_SPARSE_GLUON" +probe "VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS" "VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS" +probe "flydsl_mega_moe (literal)" "flydsl_mega_moe" +probe "mega_moe (any)" "mega_moe" + +echo +echo "=============== accepted --moe-backend literals ===============" +for side in base new; do + echo "-- $side" + grep -rhoE '"[a-z0-9_]*mega_moe[a-z0-9_]*"' "$OUT/$side" --include='*.py' 2>/dev/null \ + | sort -u | sed 's/^/ /' +done + +# The #51473 marker: the one line the whole back-port reduces to. +echo +echo "=============== vLLM #51473 marker (384-wide TP8 shard) ===============" +for side in base new; do + if grep -rq 'AITER_MXFP4_BF16 and activation == MoEActivation.SILU' "$OUT/$side" 2>/dev/null; then + echo " $side: PRESENT" + else + echo " $side: ABSENT" + fi +done + +echo +echo "DONE $(date -u +%H:%M:%S)Z" +echo "Diffs written under: $OUT" diff --git a/spur-6309.out b/spur-6309.out new file mode 100644 index 000000000..e69de29bb diff --git a/spur-6316.out b/spur-6316.out new file mode 100644 index 000000000..e69de29bb diff --git a/spur-6345.out b/spur-6345.out new file mode 100644 index 000000000..e69de29bb From e6b474a69dd3f5b1b8661c2893d1b52d7c47e634 Mon Sep 17 00:00:00 2001 From: Jiahui Cao Date: Thu, 13 Aug 2026 05:09:27 +0000 Subject: [PATCH 3/4] chore(dsv4): drop debug scratch files swept into the branch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An earlier `git add -A` in this worktree picked up ~70 local bring-up scratch files (dotted probe scripts, alternative serve variants, spur job stdout, raw PR diffs) that have no business in the repo. Only the six intended files remain: the patch script and its aiter graft helper, the recipe, the master config, the waiver, and the changelog entry. 清理早前 `git add -A` 误扫入分支的约 70 个本地调试脚本，仅保留应提交的 6 个文件。 Co-Authored-By: Claude Opus 5 --- .aiter_fp.sh | 4 - .aiter_ver.sh | 6 - .build_dsv4_image.sh | 98 ------------- .cmp_fmoe.sh | 15 -- .commitmsg_51714.txt | 22 --- .dsv4tbl.sh | 6 - .fetch_pr_diffs.sh | 14 -- .find384.sh | 10 -- .find384b.sh | 10 -- .flydsl_fp.sh | 9 -- .opusfp.sh | 5 - .prbody_51714.md | 71 ---------- .probe_aiter_features.sh | 32 ----- .probe_aiter_ident.sh | 61 -------- .probe_aiter_provenance.sh | 47 ------- .probe_aiter_sha_copy.sh | 39 ------ .probe_apply_aiter.sh | 30 ---- .probe_apply_vllm.sh | 29 ---- .probe_apply_vllm2.sh | 41 ------ .probe_attr2.sh | 30 ---- .probe_breaking_callers.sh | 76 ---------- .probe_c96_span.sh | 14 -- .probe_cache_bytes.sh | 34 ----- .probe_csv_delta.sh | 43 ------ .probe_defaults_equiv.sh | 33 ----- .probe_drift_direction.sh | 116 --------------- .probe_dsv4_csv.sh | 27 ---- .probe_final_markers.sh | 53 ------- .probe_gluon_build.sh | 75 ---------- .probe_gluon_drift.sh | 37 ----- .probe_gluon_only.sh | 80 ----------- .probe_gluon_seed_fix.sh | 42 ------ .probe_import_closure.sh | 52 ------- .probe_megamoe_closure.sh | 70 ---------- .probe_megamoe_closure2.sh | 71 ---------- .probe_megamoe_closure3.sh | 101 -------------- .probe_megamoe_match.sh | 28 ---- .probe_mori_flydsl.sh | 30 ---- .probe_post2.sh | 40 ------ .probe_post2_base.sh | 117 ---------------- .probe_post2_features.sh | 56 -------- .probe_so_delta.sh | 19 --- .probe_target_aiter.sh | 36 ----- .probe_tgemm.sh | 29 ---- .probe_vendor_over_post2.sh | 60 -------- .probe_vllm_classify.sh | 37 ----- .probe_vllm_delta.sh | 38 ----- .probe_vllm_markers.sh | 49 ------- .probe_vllm_markers2.sh | 35 ----- .verify_dsv4_image.sh | 89 ------------ aiter-4417.diff | 73 ---------- apply_aiter_4417.py | 112 --------------- dsv4_bench_8k1k_c32.sh | 19 --- dsv4_patch_additive.sh | 76 ---------- dsv4_serve_dep8.sh | 33 ----- dsv4_serve_stock.sh | 40 ------ dsv4_serve_tp8.sh | 76 ---------- dsv4_serve_tp8_forcereduce.sh | 6 - dsv4_serve_tp8_notiter.sh | 35 ----- dsv4_serve_tp8_triton.sh | 35 ----- dsv4_wait_then_bench.sh | 34 ----- graft_aiter_4417.py | 125 ----------------- sbatch_3way_patchset.sh | 256 ---------------------------------- sbatch_diff_images.sh | 183 ------------------------ spur-6309.out | 0 spur-6316.out | 0 spur-6345.out | 0 67 files changed, 3269 deletions(-) delete mode 100644 .aiter_fp.sh delete mode 100644 .aiter_ver.sh delete mode 100644 .build_dsv4_image.sh delete mode 100755 .cmp_fmoe.sh delete mode 100644 .commitmsg_51714.txt delete mode 100644 .dsv4tbl.sh delete mode 100644 .fetch_pr_diffs.sh delete mode 100644 .find384.sh delete mode 100644 .find384b.sh delete mode 100644 .flydsl_fp.sh delete mode 100644 .opusfp.sh delete mode 100644 .prbody_51714.md delete mode 100644 .probe_aiter_features.sh delete mode 100644 .probe_aiter_ident.sh delete mode 100644 .probe_aiter_provenance.sh delete mode 100644 .probe_aiter_sha_copy.sh delete mode 100644 .probe_apply_aiter.sh delete mode 100644 .probe_apply_vllm.sh delete mode 100644 .probe_apply_vllm2.sh delete mode 100644 .probe_attr2.sh delete mode 100644 .probe_breaking_callers.sh delete mode 100644 .probe_c96_span.sh delete mode 100644 .probe_cache_bytes.sh delete mode 100644 .probe_csv_delta.sh delete mode 100644 .probe_defaults_equiv.sh delete mode 100644 .probe_drift_direction.sh delete mode 100644 .probe_dsv4_csv.sh delete mode 100644 .probe_final_markers.sh delete mode 100644 .probe_gluon_build.sh delete mode 100644 .probe_gluon_drift.sh delete mode 100644 .probe_gluon_only.sh delete mode 100644 .probe_gluon_seed_fix.sh delete mode 100644 .probe_import_closure.sh delete mode 100644 .probe_megamoe_closure.sh delete mode 100644 .probe_megamoe_closure2.sh delete mode 100644 .probe_megamoe_closure3.sh delete mode 100644 .probe_megamoe_match.sh delete mode 100644 .probe_mori_flydsl.sh delete mode 100644 .probe_post2.sh delete mode 100644 .probe_post2_base.sh delete mode 100644 .probe_post2_features.sh delete mode 100644 .probe_so_delta.sh delete mode 100644 .probe_target_aiter.sh delete mode 100644 .probe_tgemm.sh delete mode 100644 .probe_vendor_over_post2.sh delete mode 100644 .probe_vllm_classify.sh delete mode 100644 .probe_vllm_delta.sh delete mode 100644 .probe_vllm_markers.sh delete mode 100644 .probe_vllm_markers2.sh delete mode 100644 .verify_dsv4_image.sh delete mode 100644 aiter-4417.diff delete mode 100644 apply_aiter_4417.py delete mode 100644 dsv4_bench_8k1k_c32.sh delete mode 100644 dsv4_patch_additive.sh delete mode 100644 dsv4_serve_dep8.sh delete mode 100644 dsv4_serve_stock.sh delete mode 100644 dsv4_serve_tp8.sh delete mode 100755 dsv4_serve_tp8_forcereduce.sh delete mode 100755 dsv4_serve_tp8_notiter.sh delete mode 100755 dsv4_serve_tp8_triton.sh delete mode 100644 dsv4_wait_then_bench.sh delete mode 100644 graft_aiter_4417.py delete mode 100644 sbatch_3way_patchset.sh delete mode 100755 sbatch_diff_images.sh delete mode 100644 spur-6309.out delete mode 100644 spur-6316.out delete mode 100644 spur-6345.out diff --git a/.aiter_fp.sh b/.aiter_fp.sh deleted file mode 100644 index 5b03da2c1..000000000 --- a/.aiter_fp.sh +++ /dev/null @@ -1,4 +0,0 @@ -#!/bin/bash -# Fingerprint the whole aiter python tree so we can size the nightly-vs-baked gap. -cd /usr/local/lib/python3.12/dist-packages/aiter || exit 1 -find . -name '*.py' -not -path './jit/*' | sort | xargs md5sum 2>/dev/null diff --git a/.aiter_ver.sh b/.aiter_ver.sh deleted file mode 100644 index 8b8e8ef13..000000000 --- a/.aiter_ver.sh +++ /dev/null @@ -1,6 +0,0 @@ -#!/bin/bash -D=/usr/local/lib/python3.12/dist-packages -for f in "$D"/aiter*.dist-info/METADATA "$D"/aiter/__init__.py; do - [ -e "$f" ] && echo "== $f" && grep -m3 -iE '^version:|__version__|commit' "$f" -done -ls -d "$D"/aiter*.dist-info 2>/dev/null diff --git a/.build_dsv4_image.sh b/.build_dsv4_image.sh deleted file mode 100644 index 83bac45ba..000000000 --- a/.build_dsv4_image.sh +++ /dev/null @@ -1,98 +0,0 @@ -#!/usr/bin/env bash -#SBATCH --job-name=dsv4-build -#SBATCH --account=amd-aifw-aim -#SBATCH --qos=amd-aifw-aim-qos -#SBATCH --partition=amd-spur -#SBATCH --nodes=1 -#SBATCH --gpus-per-node=8 -#SBATCH --exclusive -#SBATCH --time=01:10:00 -#SBATCH --output=/home/jiacao/InferenceX/dsv4-build-%j.out -#SBATCH --exclude=crsuse2-m2m-203,crsuse2-m2m-074,crsuse2-m2m-240,crsuse2-m2m-029,crsuse2-m2m-061,crsuse2-m2m-071 -# -# Build the DSv4-Pro FP4 measurement image the way CI would: take the pinned -# stock nightly base, run apply_dsv4_container_patches.sh *inside* it (the exact -# runtime-patch mechanism PR #2508 uses -- no aiter rebuild, no private vendor -# blobs), and `docker commit` the result to a local tag. The GPU is present so -# the script's import-verify actually loads aiter (catches a post2-python-over- -# v0.1.19-.so ABI break; a login node cannot, aiter probes rocminfo at import). -# -# The commit lives only on this node's docker daemon; we also `docker save` it to -# the shared FS so a later confirmation run (or a push you approve) can load it. -# We do NOT `docker push` -- publishing is your call. -set -uo pipefail - -hostname -date -u - -BASE="vllm/vllm-openai-rocm:nightly-3ee2df30337a301164c46ae444b76ee67e71c106" -OUT="dsv4-pro-fp4-mi355x:3ee2df30-patched" -PATCH="/home/jiacao/InferenceX/.claude/worktrees/dsv4-ci-sweep-pr/benchmarks/single_node/agentic/apply_dsv4_container_patches.sh" -SAVE_TAR="/home/jiacao/InferenceX/dsv4-pro-fp4-mi355x-f8d03e77-patched.${SLURM_JOB_ID:-manual}.tar" -CTR="dsv4_build_${SLURM_JOB_ID:-manual}" - -[ -f "$PATCH" ] || { echo "FATAL: patch script not found: $PATCH" >&2; exit 1; } - -# --- pull the pinned base ----------------------------------------------------- -for _try in 1 2 3; do - docker image inspect "$BASE" >/dev/null 2>&1 && break - echo "pulling $BASE (attempt $_try/3) ..." - docker pull "$BASE" || true -done -docker image inspect "$BASE" >/dev/null 2>&1 \ - || { echo "FATAL: base pull failed after 3 attempts" >&2; exit 1; } - -echo "=== base identity ===" -docker image inspect "$BASE" \ - --format 'id={{.Id}} -created={{.Created}} -digest={{index .RepoDigests 0}}' 2>/dev/null - -# --- start an idle GPU container --------------------------------------------- -# network=host so the patch script can git-clone aiter and curl the PR diffs; -# ROCm device flags so import-verify can load the JIT kernels. -docker rm -f "$CTR" >/dev/null 2>&1 || true -docker run -d --name "$CTR" \ - --device=/dev/kfd --device=/dev/dri --group-add video \ - --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \ - --ipc=host --network=host \ - --entrypoint sleep "$BASE" infinity \ - || { echo "FATAL: could not start container" >&2; exit 1; } - -# --- run the patch script inside --------------------------------------------- -echo -echo "=== applying apply_dsv4_container_patches.sh in $CTR ===" -docker exec -i "$CTR" bash < "$PATCH" -rc=$? -echo "patch script rc=$rc" - -# --- commit only on a clean apply + verify ----------------------------------- -if [ "$rc" -eq 0 ]; then - docker commit \ - --change 'LABEL dsv4.base=nightly-3ee2df30337a301164c46ae444b76ee67e71c106' \ - --change 'LABEL dsv4.aiter=v0.1.19.post2+main@97d0c6e4+pr4673' \ - --change 'LABEL dsv4.vllm=pr51473(native)+pr51714+pr51918;pr48728=DROPPED(base-drift)' \ - "$CTR" "$OUT" - echo "committed local image: $OUT" - docker images "$OUT" - - # docker save is OFF by default: the shared /home volume is at 100% and an - # 11GB tar hits EDQUOT. The image lives on this node's daemon; verify pins - # here, and distribution is a registry push (your call). DO_SAVE=1 to force. - if [ "${DO_SAVE:-0}" = "1" ]; then - echo "=== docker save -> $SAVE_TAR ===" - if docker save "$OUT" -o "$SAVE_TAR"; then ls -lh "$SAVE_TAR"; echo "SAVED_OK $SAVE_TAR"; - else echo "SAVE_FAILED (image still present locally as $OUT)"; fi - else - echo "docker save skipped (DO_SAVE=0); image is on $(hostname)'s daemon as $OUT" - fi -else - echo "patch failed (rc=$rc) -- NOT committing, NOT saving." -fi - -# --- cleanup ------------------------------------------------------------------ -docker rm -f "$CTR" >/dev/null 2>&1 || true -echo -echo "BUILD_RESULT rc=$rc image=$OUT" -date -u -exit "$rc" diff --git a/.cmp_fmoe.sh b/.cmp_fmoe.sh deleted file mode 100755 index 9ba56ff92..000000000 --- a/.cmp_fmoe.sh +++ /dev/null @@ -1,15 +0,0 @@ -#!/bin/bash -# Inspect aiter fused_moe tuned tables for the inter_dim=384 / tile_n=256 mismatch. -for f in /usr/local/lib/python3.12/dist-packages/aiter/configs/tuned_fmoe.csv \ - /usr/local/lib/python3.12/dist-packages/aiter/configs/model_configs/dsv4_tuned_fmoe.csv \ - /tmp/aiter_configs/tuned_fmoe.csv; do - if [ -f "$f" ]; then - n384=$(awk -F, '$4==384 && $5==384' "$f" | wc -l) - n256=$(awk -F, '$4==384 && $5==384' "$f" | grep -c 'x256x256') - echo "$f rows384=$n384 naming_tile256=$n256" - else - echo "$f MISSING" - fi -done -echo "=== aiter version ===" -python -c "import aiter; print(getattr(aiter,'__version__','?'), aiter.__file__)" 2>&1 | tail -1 diff --git a/.commitmsg_51714.txt b/.commitmsg_51714.txt deleted file mode 100644 index 7706363d0..000000000 --- a/.commitmsg_51714.txt +++ /dev/null @@ -1,22 +0,0 @@ -Drop the DP-attention guard now that aiter#4673 fixes the root cause - -The guard was added when DP-attention + gluon degenerated generation -(gsm8k 1.00 -> 0.30 on EP8) and the cause was still unknown. It is now -known: the gluon driver decided `use_buffer_load` from -`nelement() * element_size()`, which undercounts a strided KV pool whose -addressable span exceeds a 32-bit offset. Past 2 GiB the buffer_load -offsets wrap and the gather returns zeros. DP-attention only mattered -because it is what grows the pool far enough to cross the boundary. - -ROCm/aiter#4673 sizes the decision from the real strided span, so the -condition the guard was standing in for no longer exists, and gating on -data_parallel_size just disables the kernel on the configuration it was -written for. Replace it with a comment naming aiter#4673 as the -prerequisite; the existing try/except still falls back to Triton for the -rest of the process if the kernel raises. - -Measured on MI355X (gfx950) with the fix in place: DEP8 (DP-attention, -EP8) + MegaMoE + gluon at concurrency 96 reaches 12,244 tok/s/GPU, and -all eight DP ranks log the gluon path engaging. - -Co-Authored-By: Claude Opus 5 diff --git a/.dsv4tbl.sh b/.dsv4tbl.sh deleted file mode 100644 index 7b2e1c3fc..000000000 --- a/.dsv4tbl.sh +++ /dev/null @@ -1,6 +0,0 @@ -#!/bin/bash -f=/usr/local/lib/python3.12/dist-packages/aiter/configs/model_configs/dsv4_fp8fp4_tuned_fmoe.csv -echo "lines=$(wc -l < "$f") md5=$(md5sum "$f" | cut -d' ' -f1)" -echo "--- header ---"; head -1 "$f" -echo "--- rows naming x256x256 ---"; grep -c 'x256x256' "$f" -echo "--- sample rows ---"; grep 'x256x256' "$f" | head -3 diff --git a/.fetch_pr_diffs.sh b/.fetch_pr_diffs.sh deleted file mode 100644 index cb7712cae..000000000 --- a/.fetch_pr_diffs.sh +++ /dev/null @@ -1,14 +0,0 @@ -#!/usr/bin/env bash -# Pull every upstream PR diff we intend to embed, so the sizes are known before -# the container-patch script is written. -set -u -mkdir -p /tmp/dsv4patch -cd /tmp/dsv4patch || exit 1 -for n in 4269 4382 4439 4664 4673; do - curl -sSL -o "aiter-$n.diff" "https://github.com/ROCm/aiter/pull/$n.diff" - echo "aiter#$n: $(wc -l < "aiter-$n.diff") lines, $(wc -c < "aiter-$n.diff") bytes" -done -for n in 48728 51473 51713 51714 51918; do - curl -sSL -o "vllm-$n.diff" "https://github.com/vllm-project/vllm/pull/$n.diff" - echo "vllm#$n: $(wc -l < "vllm-$n.diff") lines, $(wc -c < "vllm-$n.diff") bytes" -done diff --git a/.find384.sh b/.find384.sh deleted file mode 100644 index dc3e0043e..000000000 --- a/.find384.sh +++ /dev/null @@ -1,10 +0,0 @@ -#!/bin/bash -# Find which shipped aiter tuned-fmoe table contributes the inter_dim=384 rows. -echo "--- shipped csvs mentioning fmoe ---" -find /usr/local/lib/python3.12/dist-packages/aiter/configs -name '*fmoe*' | while read -r f; do - n=$(awk -F, '$4==384 && $5==384' "$f" | wc -l) - t=$(awk -F, '$4==384 && $5==384' "$f" | grep -c 'x256x256') - echo "$(basename "$f") rows384=$n tile256=$t total=$(wc -l < "$f")" -done -echo "--- /tmp/aiter_configs contents ---" -ls -la /tmp/aiter_configs 2>/dev/null | head -20 diff --git a/.find384b.sh b/.find384b.sh deleted file mode 100644 index d5a7b2ee8..000000000 --- a/.find384b.sh +++ /dev/null @@ -1,10 +0,0 @@ -#!/bin/bash -# Report only the tuned-fmoe tables that actually carry inter_dim=384 rows. -find /usr/local/lib/python3.12/dist-packages/aiter/configs -name '*fmoe*' | sort | while read -r f; do - n=$(awk -F, '$4==384 && $5==384' "$f" | wc -l) - [ "$n" -gt 0 ] || continue - t=$(awk -F, '$4==384 && $5==384' "$f" | grep -c 'x256x256') - echo "HIT $f rows384=$n tile256=$t" -done -echo "--- dsv4-named tables ---" -find /usr/local/lib/python3.12/dist-packages/aiter/configs -name '*dsv4*' | sort diff --git a/.flydsl_fp.sh b/.flydsl_fp.sh deleted file mode 100644 index c5dc58445..000000000 --- a/.flydsl_fp.sh +++ /dev/null @@ -1,9 +0,0 @@ -#!/bin/bash -D=/usr/local/lib/python3.12/dist-packages/aiter/ops/flydsl -echo "--- flydsl file md5s ---" -find "$D" -name '*.py' | sort | xargs md5sum 2>/dev/null | sed "s#$D/##" -echo "--- dsv4 tuned table, correct columns (gfx,cu,token,model_dim,inter_dim,expert) ---" -T=/usr/local/lib/python3.12/dist-packages/aiter/configs/model_configs/dsv4_fp8fp4_tuned_fmoe.csv -awk -F, 'NR>1{print $5}' "$T" | sort -u | tr '\n' ' '; echo -echo "rows inter_dim=384: $(awk -F, 'NR>1 && $5==384' "$T" | wc -l)" -echo " of which naming x256x256: $(awk -F, 'NR>1 && $5==384' "$T" | grep -c 'x256x256')" diff --git a/.opusfp.sh b/.opusfp.sh deleted file mode 100644 index 0ffc33d63..000000000 --- a/.opusfp.sh +++ /dev/null @@ -1,5 +0,0 @@ -#!/bin/bash -D=/usr/local/lib/python3.12/dist-packages/aiter/ops/opus -find "$D" -name '*.py' | sort | xargs md5sum 2>/dev/null | sed "s#$D/##" -echo "--- fused_moe.py ---" -md5sum /usr/local/lib/python3.12/dist-packages/aiter/fused_moe.py diff --git a/.prbody_51714.md b/.prbody_51714.md deleted file mode 100644 index 23159129e..000000000 --- a/.prbody_51714.md +++ /dev/null @@ -1,71 +0,0 @@ -## Purpose - -Adds an opt-in gfx950 gluon replacement for the Triton sparse-MLA decode pair -(`_sparse_attn_decode_partial_kernel` + `_sparse_attn_decode_reduce_kernel`), -provided by AITER ([ROCm/aiter#4382](https://github.com/ROCm/aiter/pull/4382), merged). - -## Changes - -Single file, `v1/attention/ops/rocm_aiter_mla_sparse.py`, additive only (+78 / -0): - -- `VLLM_ROCM_DSV4_SPARSE_GLUON` (**default off**). Read once at import — this sits on - the per-step decode path, so re-reading the environment per call would be pure - overhead. With the flag unset the decode path is byte-for-byte unchanged. -- The gluon entrypoint is resolved lazily behind `functools.cache`, so builds whose - AITER lacks the kernel are unaffected. -- A failed launch latches a process-wide flag and degrades to Triton for the rest of - the process, rather than taking the server down mid-request. -- The first successful dispatch logs once, so enabling the flag is observable rather - than silent. -- Additionally gated on `_ON_GFX950`. - -## Requires ROCm/aiter#4673 - -The kernel needs [ROCm/aiter#4673](https://github.com/ROCm/aiter/pull/4673) to be -correct on large KV pools. Without it the gluon driver decides `use_buffer_load` from -`nelement() * element_size()`, which undercounts a strided pool whose addressable span -exceeds a 32-bit offset; past 2 GiB the `buffer_load` offsets wrap and the gather -silently returns zeros. - -An earlier revision of this PR refused to run under DP-attention, citing -[ROCm/aiter#4667](https://github.com/ROCm/aiter/issues/4667) (gsm8k 1.00 → 0.30 on -EP8 + DP-attention). That guard has been removed: #4667 is the overflow above, and -DP-attention was only ever the thing that grew the pool far enough to cross the -boundary — not a property of the kernel. Gating on `data_parallel_size` therefore -disabled the kernel on exactly the configuration it was written for. With #4673 -applied, DP-attention runs clean. - -## Test Plan - -- DeepSeek-V4 on MI355X (gfx950) with `VLLM_ROCM_DSV4_SPARSE_GLUON=0` and `=1`, - comparing decode outputs for parity and measuring decode throughput. -- Cover both `swa_only` (single-segment) and two-loop layers. -- Confirm the fallback latch by forcing a launch failure. -- Cover DP-attention (DEP8) with aiter#4673 applied. - -## Test Result - -| Concurrency | Output tok/s (base → +PR) | Δ | Median TPOT (base → +PR) | Δ | -|---|---|---|---|---| -| 8 | 284.79 → 291.33 | **+2.30%** | 24.67 → 24.16 ms | −2.04% | -| 16 | 483.69 → 488.59 | **+1.01%** | 29.32 → 29.02 ms | −1.03% | -| 32 | 708.51 → 712.89 | **+0.62%** | 40.78 → 40.52 ms | −0.65% | -| 64 | 955.23 → 958.04 | **+0.29%** | 62.55 → 62.36 ms | −0.30% | - -### AgentX agentic replay (fast mode, TP=8, concurrency 32, MTP on) - -| Metric | base (Triton) | +PR (gluon) | Δ | -|---|---|---|---| -| Output tok/s | 430.76 | 453.23 | **+5.22%** | -| Output tok/s/GPU | 53.84 | 56.65 | **+5.22%** | -| Per-user tok/s | 32.72 | 39.96 | **+22.12%** | -| ITL avg | 39.83 ms | 34.60 ms | **−13.11%** | -| ITL p50 | 35.87 ms | 26.44 ms | **−26.30%** | -| Time-to-2nd-token p50 | 38.19 ms | 21.79 ms | **−42.95%** | -| TTFT p50 | 656.7 ms | 700.6 ms | +6.68% (regression) | - -### DP-attention, with aiter#4673 applied - -DEP8 (DP-attention + EP8) + MegaMoE + gluon, MI355X, agentic replay at concurrency 96: -**12,244 tok/s/GPU**. All eight DP ranks log the gluon path engaging. This is the -configuration the removed guard used to refuse. diff --git a/.probe_aiter_features.sh b/.probe_aiter_features.sh deleted file mode 100644 index 1935055b5..000000000 --- a/.probe_aiter_features.sh +++ /dev/null @@ -1,32 +0,0 @@ -#!/usr/bin/env bash -# Which of the four DSv4 features are already in each image's aiter tree? -# -# The question this answers: aiter ships as its own wheel inside each image, so -# "the upstream PR is merged" does not imply "this image's aiter has it". Check -# the trees, not the PR states. -set -u -D="${1:-/home/jiacao/3way-20260812-2214}" - -for side in ref vendor target; do - A=$(find "$D/$side" -maxdepth 9 -type d -name aiter -path '*dist-packages*' 2>/dev/null | head -1) - echo "=== $side" - echo " root: ${A:-}" - [ -z "$A" ] && continue - - n=$(ls "$A/ops/flydsl/kernels/mega_moe"/*.py 2>/dev/null | wc -l) - echo " MegaMoE kernels (ops/flydsl/kernels/mega_moe/*.py) : ${n}" - - for f in \ - "ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py" \ - "fhmoe.py" \ - "ops/flydsl/fhmoe.py" \ - "aot/flydsl/fhmoe.py" \ - "ops/flydsl/kernels/mixed_moe_gemm_2stage_common.py" \ - ; do - [ -f "$A/$f" ] && echo " PRESENT $f" || echo " absent $f" - done - - echo -n " tgemm: " - ls "$A"/ops/*tgemm* "$A"/*tgemm* 2>/dev/null | head -3 | tr '\n' ' ' - grep -rl "tgemm" "$A" --include='*.py' 2>/dev/null | wc -l | xargs echo "files mentioning tgemm:" -done diff --git a/.probe_aiter_ident.sh b/.probe_aiter_ident.sh deleted file mode 100644 index bde8935d3..000000000 --- a/.probe_aiter_ident.sh +++ /dev/null @@ -1,61 +0,0 @@ -#!/usr/bin/env bash -# What aiter is actually installed in each image? _version.py alone lies for -# source builds (it reads 0.1.0), so also fingerprint the tree: how many .py -# files match the v0.1.19 tag vs the MegaMoE-merge SHA on main. -set -u -D=/home/jiacao/3way-20260812-2214 -W=/tmp/aitermain # clone of ROCm/aiter, currently at 97d0c6e4 - -echo "=== declared _version.py ===" -for i in ref target vendor; do - f="$D/$i/usr/local/lib/python3.12/dist-packages/aiter/_version.py" - [ -f "$f" ] && echo " $i: $(head -1 "$f")" || echo " $i: (absent)" -done - -echo -echo "=== fingerprint: fraction of .py files byte-identical to each candidate ===" -python3 - <<'PY' -import os, subprocess, filecmp -D = "/home/jiacao/3way-20260812-2214" -W = "/tmp/aitermain" - -def snapshot(ref): - subprocess.run(["git", "-C", W, "checkout", "-q", ref], check=True) - out = {} - root = os.path.join(W, "aiter") - for dp, _, fns in os.walk(root): - if "__pycache__" in dp: continue - for fn in fns: - if fn.endswith(".py"): - p = os.path.join(dp, fn) - out[os.path.relpath(p, root)] = open(p, "rb").read() - return out - -def image(name): - root = os.path.join(D, name, "usr/local/lib/python3.12/dist-packages/aiter") - out = {} - for dp, _, fns in os.walk(root): - if "__pycache__" in dp: continue - for fn in fns: - if fn.endswith(".py"): - p = os.path.join(dp, fn) - out[os.path.relpath(p, root)] = open(p, "rb").read() - return out - -cands = { - "v0.1.19": snapshot("v0.1.19"), - "v0.1.19.post2": snapshot("v0.1.19.post2"), - "main@97d0c6e4(#4439)": snapshot("97d0c6e4cb7a0919c12291c7c7d560ad412f15c1"), -} -for img in ("target", "vendor"): - files = image(img) - print(f" --- {img} ({len(files)} .py files)") - for label, snap in cands.items(): - common = set(files) & set(snap) - same = sum(1 for k in common if files[k] == snap[k]) - only_img = len(set(files) - set(snap)) - only_snap = len(set(snap) - set(files)) - pct = 100.0 * same / len(common) if common else 0 - print(f" vs {label:22} identical {same}/{len(common)} ({pct:.1f}%)" - f" img-only={only_img} snap-only={only_snap}") -PY diff --git a/.probe_aiter_provenance.sh b/.probe_aiter_provenance.sh deleted file mode 100644 index 508e2258a..000000000 --- a/.probe_aiter_provenance.sh +++ /dev/null @@ -1,47 +0,0 @@ -#!/usr/bin/env bash -# Why does the target image's aiter lack PR #4269 when #4269 merged 2026-08-01, -# nine days before the image's vLLM commit (b22afe45, 2026-08-10)? -# -# Candidate explanations, in order of likelihood: -# (a) the image installs aiter from a RELEASE TAG / pinned SHA, not main -# (b) the image installs from a ROCm-vendored branch that lags main -# (c) the wheel was built earlier and cached -# Distinguish them by reading the installed distribution's own metadata: -# dist-info/METADATA and direct_url.json record where pip got it, and -# aiter/_version.py records what the build tree called itself. -set -u -D="${1:-/home/jiacao/3way-20260812-2214}" - -for side in ref target vendor; do - R=$(find "$D/$side" -maxdepth 9 -type d -name aiter -path '*dist-packages*' 2>/dev/null | head -1) - [ -z "$R" ] && { echo "=== $side "; continue; } - P=$(dirname "$R") - echo "=========================== $side" - - echo "--- _version.py ---" - grep -hE "version|commit" "$R/_version.py" 2>/dev/null | grep -E "^\s*(__version__|version|__commit|commit)" | head -4 | sed 's/^/ /' - - echo "--- dist-info ---" - for di in "$P"/aiter*.dist-info; do - [ -d "$di" ] || continue - echo " $(basename "$di")" - # direct_url.json is written by pip when installing from a VCS/URL and - # carries the exact requested ref and resolved commit. - if [ -f "$di/direct_url.json" ]; then - echo " direct_url.json:" - cat "$di/direct_url.json" 2>/dev/null | sed 's/^/ /' - echo - else - echo " (no direct_url.json -- not a VCS install)" - fi - grep -iE "^(Version|Home-page|Download-URL|Project-URL)" "$di/METADATA" 2>/dev/null | head -6 | sed 's/^/ /' - done - - echo "--- git metadata left in the tree? ---" - for g in "$R/../.git" "$R/.git" "$R/../aiter/.git"; do - [ -e "$g" ] && echo " found: $g" - done - # Some builds stamp the SHA into a hidden file or the jit config. - grep -rhoE "[0-9a-f]{40}" "$R/jit/optCompilerConfig.json" 2>/dev/null | head -2 | sed 's/^/ optCompilerConfig sha: /' - echo -done diff --git a/.probe_aiter_sha_copy.sh b/.probe_aiter_sha_copy.sh deleted file mode 100644 index b73ec9f26..000000000 --- a/.probe_aiter_sha_copy.sh +++ /dev/null @@ -1,39 +0,0 @@ -#!/usr/bin/env bash -# The published PR diffs don't apply to the base image's aiter v0.1.19 -- the -# tag is 107 commits behind the MegaMoE merge. So the question becomes: is the -# feature set carried by NEW files (copyable from a pinned main SHA) or by edits -# to files that have drifted? -# -# Clone aiter main at the MegaMoE merge (97d0c6e4, which is a descendant of both -# #4382 and #4269) and diff the feature-relevant files against the base image. -set -u -W=/tmp/aitermain -if [ ! -d "$W/.git" ]; then - git clone -q --filter=blob:none https://github.com/ROCm/aiter "$W" || exit 1 -fi -cd "$W" || exit 1 -git fetch -q origin 97d0c6e4cb7a0919c12291c7c7d560ad412f15c1 2>/dev/null -git checkout -q 97d0c6e4cb7a0919c12291c7c7d560ad412f15c1 || exit 1 -echo "aiter main @ $(git rev-parse --short HEAD) version=$(cat aiter/_version.py 2>/dev/null)" - -T=/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter -echo -echo "=== how many aiter/*.py differ between v0.1.19 (base image) and this SHA?" -cd "$W/aiter" || exit 1 -find . -name '*.py' | sort > /tmp/main_py.txt -cd "$T" || exit 1 -find . -name '*.py' | sort > /tmp/base_py.txt -echo " main-only files: $(comm -13 /tmp/base_py.txt /tmp/main_py.txt | wc -l)" -echo " base-only files: $(comm -23 /tmp/base_py.txt /tmp/main_py.txt | wc -l)" -common=$(comm -12 /tmp/base_py.txt /tmp/main_py.txt | wc -l) -echo " common files: $common" -changed=0 -while IFS= read -r f; do - cmp -s "$T/$f" "$W/aiter/$f" || changed=$((changed+1)) -done < <(comm -12 /tmp/base_py.txt /tmp/main_py.txt) -echo " common but differing: $changed" - -echo -echo "=== non-Python that would need a rebuild (csrc / hsa / .cu / .cpp) ===" -cd "$W" || exit 1 -git diff --stat v0.1.19..HEAD -- csrc hsa '*.cu' '*.cpp' '*.hpp' '*.cuh' 2>/dev/null | tail -3 diff --git a/.probe_apply_aiter.sh b/.probe_apply_aiter.sh deleted file mode 100644 index 70d0834c3..000000000 --- a/.probe_apply_aiter.sh +++ /dev/null @@ -1,30 +0,0 @@ -#!/usr/bin/env bash -# Decisive experiment: can the five upstream aiter PRs be applied, as published -# diffs, straight onto the BASE image's installed aiter (v0.1.19)? The installed -# layout has no op_tests/ or .github/, so restrict to the aiter/ package. -# If this works the container patch can fetch-and-apply instead of embedding -# ~1.5 MB of heredoc. -set -u -SRC=/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter -W=/tmp/aitertest -rm -rf "$W" -mkdir -p "$W" -cp -a "$SRC" "$W/aiter" -cd "$W" || exit 1 -git init -q . -git add -A -f >/dev/null 2>&1 -git -c user.email=x@y -c user.name=x commit -qm base >/dev/null 2>&1 - -for n in 4269 4382 4439 4664 4673; do - d=/tmp/dsv4patch/aiter-$n.diff - echo "=== aiter#$n" - if git apply --include='aiter/*' -p1 --check "$d" 2>&1 | head -20; then - echo " CHECK CLEAN" - git apply --include='aiter/*' -p1 "$d" && echo " APPLIED" - git add -A >/dev/null 2>&1 - git -c user.email=x@y -c user.name=x commit -qm "pr$n" >/dev/null 2>&1 - else - echo " CHECK FAILED -> trying 3-way / fuzz" - patch -p1 --dry-run --fuzz=3 --forward < "$d" 2>&1 | grep -iv '^patching file aiter/' | head -30 - fi -done diff --git a/.probe_apply_vllm.sh b/.probe_apply_vllm.sh deleted file mode 100644 index abb144f63..000000000 --- a/.probe_apply_vllm.sh +++ /dev/null @@ -1,29 +0,0 @@ -#!/usr/bin/env bash -# vLLM side: do the five PRs apply, as published diffs, onto the base image's -# installed vllm package tree (b22afe45)? Restrict to vllm/ -- the installed -# layout has no tests/ or docs/. -set -u -SRC=/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/vllm -W=/tmp/vllmtest -rm -rf "$W"; mkdir -p "$W" -cp -a "$SRC" "$W/vllm" -cd "$W" || exit 1 -git init -q . -git add -A -f >/dev/null 2>&1 -git -c user.email=x@y -c user.name=x commit -qm base >/dev/null 2>&1 - -for n in 51473 51714 51713 51918 48728; do - d=/tmp/dsv4patch/vllm-$n.diff - echo "=== vllm#$n" - out=$(git apply --include='vllm/*' -p1 --check "$d" 2>&1) - if [ -z "$out" ]; then - echo " CLEAN" - git apply --include='vllm/*' -p1 "$d" - git add -A >/dev/null 2>&1 - git -c user.email=x@y -c user.name=x commit -qm "pr$n" >/dev/null 2>&1 - else - echo "$out" | sed 's/^/ /' | head -20 - echo " -- retry with 3-way + fuzz --" - patch -p1 --dry-run --fuzz=3 --forward -s < "$d" 2>&1 | head -20 - fi -done diff --git a/.probe_apply_vllm2.sh b/.probe_apply_vllm2.sh deleted file mode 100644 index a6a92ef13..000000000 --- a/.probe_apply_vllm2.sh +++ /dev/null @@ -1,41 +0,0 @@ -#!/usr/bin/env bash -# The sequential run confounds PR-vs-base conflicts with PR-vs-PR conflicts. -# Re-test each PR ALONE on a pristine base, then test the two amd/model.py -# PRs (#51918, #48728) stacked in each order to see which way round works. -set -u -SRC=/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/vllm - -fresh() { - rm -rf /tmp/vt; mkdir -p /tmp/vt - cp -a "$SRC" /tmp/vt/vllm - cd /tmp/vt || exit 1 - git init -q .; git add -A -f >/dev/null 2>&1 - git -c user.email=x@y -c user.name=x commit -qm base >/dev/null 2>&1 -} - -try() { # $1=pr - local out - out=$(git apply --include='vllm/*' -p1 --check "/tmp/dsv4patch/vllm-$1.diff" 2>&1) - if [ -z "$out" ]; then - echo " #$1 CLEAN" - git apply --include='vllm/*' -p1 "/tmp/dsv4patch/vllm-$1.diff" - return 0 - fi - echo " #$1 CONFLICT:"; echo "$out" | sed 's/^/ /' - return 1 -} - -echo "=== each PR alone on pristine b22afe45 ===" -for n in 51473 51714 51713 51918 48728; do - fresh - out=$(git apply --include='vllm/*' -p1 --check "/tmp/dsv4patch/vllm-$n.diff" 2>&1) - if [ -z "$out" ]; then echo " #$n CLEAN"; else echo " #$n CONFLICT:"; echo "$out" | sed 's/^/ /'; fi -done - -echo -echo "=== #48728 then #51918 ===" -fresh; try 48728; try 51918 - -echo -echo "=== #51918 then #48728 ===" -fresh; try 51918; try 48728 diff --git a/.probe_attr2.sh b/.probe_attr2.sh deleted file mode 100644 index 35f529f66..000000000 --- a/.probe_attr2.sh +++ /dev/null @@ -1,30 +0,0 @@ -#!/usr/bin/env bash -# Attribute the still-unattributed vendor hunks: cudagraph_warmup_context, -# the tgemm call sites (#51713), and the #51473 MXFP4 TP8 shard. For each, -# print which vendor files carry it and whether the base image (target) has it. -set -u -D=/home/jiacao/3way-20260812-2214 -R="$D/ref/usr/local/lib/python3.12/dist-packages/vllm" -V="$D/vendor/src/vllm/vllm" -T="$D/target/usr/local/lib/python3.12/dist-packages/vllm" - -show() { # $1=marker - echo "### $1" - echo " vendor files:" - grep -rl -- "$1" "$V" --include='*.py' 2>/dev/null | sed "s|$V/| |" - echo " target files:" - grep -rl -- "$1" "$T" --include='*.py' 2>/dev/null | sed "s|$T/| |" - echo " ref files:" - grep -rl -- "$1" "$R" --include='*.py' 2>/dev/null | sed "s|$R/| |" - echo -} - -show "cudagraph_warmup_context" -show "tgemm.mm" -show "AITER_MXFP4_BF16" - -echo "### #51473: which MoEActivation branch guards AITER_MXFP4_BF16" -for t in "$T" "$V"; do - echo "--- ${t##*/dist-packages/}${t##*/src/vllm/}" - grep -rn -B4 -A4 "AITER_MXFP4_BF16" "$t" --include='*.py' 2>/dev/null | head -40 -done diff --git a/.probe_breaking_callers.sh b/.probe_breaking_callers.sh deleted file mode 100644 index 9c64a5e08..000000000 --- a/.probe_breaking_callers.sh +++ /dev/null @@ -1,76 +0,0 @@ -#!/usr/bin/env bash -# The 3 breaking signature changes are only safe if every caller is itself in -# the patch set. Find the real callers of each resigned function in post2, and -# check whether the vendor's copy of that caller passes the new argument. -set -u -python3 - <<'PY' -import os, re, subprocess -W = "/tmp/aitermain" -subprocess.run(["git", "-C", W, "checkout", "-q", "v0.1.19.post2"], check=True) -P = os.path.join(W, "aiter") -V = "/home/jiacao/3way-20260812-2214/vendor/usr/local/lib/python3.12/dist-packages/aiter" - -PATCHSET = { - "ops/flydsl/kernels/flydsl_dispatch_combine_intranode_op.py", - "ops/flydsl/kernels/flydsl_dispatch_combine_intranode_kernel.py", - "ops/flydsl/kernels/communication_ops_utils.py", - "ops/flydsl/kernels/mxfp4_gemm_common.py", - "ops/flydsl/kernels/vector.py", -} -PATCHSET |= {"ops/flydsl/kernels/mega_moe/" + f for f in os.listdir(os.path.join(V, "ops/flydsl/kernels/mega_moe")) - if f.endswith(".py")} - -FUNCS = { - "atomic_add_global_at": "syncscope", - "make_combine_jit": "blockwise_fp8_transport", - "make_combine_kernel": "blockwise_fp8_transport", - "_lds_swizzle_mask": "row_bytes", -} - -for fn, newarg in FUNCS.items(): - print(f"=== {fn}() new arg: {newarg}") - callers = [] - for dp, _, fns in os.walk(P): - if "__pycache__" in dp: continue - for f in fns: - if not f.endswith(".py"): continue - p = os.path.join(dp, f) - rel = os.path.relpath(p, P) - src = open(p).read() - # a call site, not the definition - if re.search(r"(? /tmp/csv_target.txt -ls "$V/configs/model_configs/" > /tmp/csv_vendor.txt - -echo "target csv count: $(wc -l < /tmp/csv_target.txt)" -echo "vendor csv count: $(wc -l < /tmp/csv_vendor.txt)" -echo -echo "=== in VENDOR but not in TARGET (must be added) ===" -comm -13 /tmp/csv_target.txt /tmp/csv_vendor.txt | sed 's/^/ /' -echo -echo "=== in TARGET but not in VENDOR (target is newer here) ===" -comm -23 /tmp/csv_target.txt /tmp/csv_vendor.txt | sed 's/^/ /' -echo -echo "=== upstream availability of the vendor-only ones (aiter/main) ===" -comm -13 /tmp/csv_target.txt /tmp/csv_vendor.txt | while read -r f; do - [ -z "$f" ] && continue - s=$(gh api "/repos/ROCm/aiter/contents/aiter/configs/model_configs/$f?ref=main" --jq '.size' 2>/dev/null) - if [[ "$s" =~ ^[0-9]+$ ]]; then printf " %-9s %-9s %s\n" PRESENT "${s}B" "$f" - else printf " %-9s %-9s %s\n" ABSENT "-" "$f"; fi -done -echo -echo "=== dsv4 csvs: vendor line counts vs target ===" -for f in $(ls "$V/configs/model_configs/" | grep -i dsv4); do - lv=$(wc -l < "$V/configs/model_configs/$f") - if [ -f "$T/configs/model_configs/$f" ]; then - lt=$(wc -l < "$T/configs/model_configs/$f") - else - lt="" - fi - printf " %-52s vendor=%-6s target=%s\n" "$f" "$lv" "$lt" -done diff --git a/.probe_defaults_equiv.sh b/.probe_defaults_equiv.sh deleted file mode 100644 index 505259472..000000000 --- a/.probe_defaults_equiv.sh +++ /dev/null @@ -1,33 +0,0 @@ -#!/usr/bin/env bash -# A default only makes the change safe if the default RESTORES the old behavior. -# Print post2's body next to the vendor's body for each of the four functions, -# so the equivalence at the default value can be read directly. -set -u -python3 - <<'PY' -import ast, os, subprocess -W = "/tmp/aitermain" -subprocess.run(["git", "-C", W, "checkout", "-q", "v0.1.19.post2"], check=True) -P = os.path.join(W, "aiter") -V = "/home/jiacao/3way-20260812-2214/vendor/usr/local/lib/python3.12/dist-packages/aiter" - -TARGETS = [ - ("ops/flydsl/kernels/communication_ops_utils.py", "atomic_add_global_at"), - ("ops/flydsl/kernels/mxfp4_gemm_common.py", "_lds_swizzle_mask"), -] - -def body(path, name): - src = open(path).read() - for n in ast.walk(ast.parse(src)): - if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef)) and n.name == name: - return ast.get_source_segment(src, n) - return None - -for rel, fn in TARGETS: - print("#" * 70) - print(f"### {fn} ({rel})") - print("--- post2 (base image would have this)") - print(body(os.path.join(P, rel), fn)) - print("--- vendor (measurement image)") - print(body(os.path.join(V, rel), fn)) - print() -PY diff --git a/.probe_drift_direction.sh b/.probe_drift_direction.sh deleted file mode 100644 index a43126a72..000000000 --- a/.probe_drift_direction.sh +++ /dev/null @@ -1,116 +0,0 @@ -#!/usr/bin/env bash -# Two questions before committing to a patch shape: -# (a) the 5 drifted flydsl files MegaMoE needs -- is the base->main drift purely -# additive? tensor_shim has 35 importers in the base image, so replacing it -# wholesale is only safe if nothing is removed or re-signatured. -# (b) FSE (#4269): what is its transitive file closure, same method as MegaMoE? -set -u -W=/tmp/aitermain/aiter -T=/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter - -echo "=== (a) drift direction on the 5 shared flydsl files ===" -python3 - <<'PY' -import ast, os -W = "/tmp/aitermain/aiter" -T = "/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter" -FILES = ["ops/flydsl/kernels/flydsl_dispatch_combine_intranode_op.py", - "ops/flydsl/kernels/flydsl_dispatch_combine_intranode_kernel.py", - "ops/flydsl/kernels/tensor_shim.py", - "ops/flydsl/kernels/mxfp4_gemm_common.py", - "ops/flydsl/kernels/communication_ops_utils.py"] - -def api(path): - """Top-level public names and, for functions, their arg lists.""" - out = {} - for n in ast.parse(open(path).read()).body: - if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef)): - a = n.args - sig = [x.arg for x in a.posonlyargs + a.args] + \ - (["*" + a.vararg.arg] if a.vararg else []) + \ - [x.arg for x in a.kwonlyargs] + \ - (["**" + a.kwarg.arg] if a.kwarg else []) - out[n.name] = tuple(sig) - elif isinstance(n, ast.ClassDef): - out[n.name] = ("",) - elif isinstance(n, ast.Assign): - for t in n.targets: - if isinstance(t, ast.Name): - out[t.id] = ("",) - return out - -for f in FILES: - b, m = api(os.path.join(T, f)), api(os.path.join(W, f)) - removed = sorted(set(b) - set(m)) - changed = sorted(k for k in set(b) & set(m) if b[k] != m[k]) - added = sorted(set(m) - set(b)) - verdict = "ADDITIVE-SAFE" if not removed and not changed else "BREAKING" - print(f" {verdict:14} {os.path.basename(f)} +{len(added)} -{len(removed)} ~{len(changed)}") - for k in removed: print(f" removed: {k}") - for k in changed: print(f" resigned: {k} {b[k]} -> {m[k]}") -PY - -echo -echo "=== (b) FSE transitive closure (#4269) ===" -python3 - <<'PY' -import ast, os, collections, difflib -W = "/tmp/aitermain/aiter" -T = "/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter" - -def find(root, rel): - for s in (".py", "/__init__.py"): - if os.path.exists(os.path.join(root, rel + s)): - return rel + s - return None - -def imports_of(path, self_rel): - pkg = os.path.dirname(self_rel); out = set() - for n in ast.walk(ast.parse(open(path).read())): - if isinstance(n, ast.ImportFrom): - if n.level: - base = pkg - for _ in range(n.level - 1): base = os.path.dirname(base) - mod = os.path.join(base, (n.module or "").replace(".", "/")).rstrip("/") - elif n.module and n.module.startswith("aiter"): - mod = n.module[len("aiter"):].lstrip(".").replace(".", "/") - else: - continue - out.add(mod) - for a in n.names: out.add(os.path.join(mod, a.name)) - elif isinstance(n, ast.Import): - for a in n.names: - if a.name.startswith("aiter"): - out.add(a.name[len("aiter"):].lstrip(".").replace(".", "/")) - return {m for m in out if m} - -seed = ["fhmoe.py", "ops/flydsl/fhmoe.py", "ops/flydsl/kernels/fhmoe.py", - "aot/flydsl/fhmoe.py"] -seen, q = set(seed), collections.deque(seed) -new, drift = set(), {} -while q: - f = q.popleft() - p = os.path.join(W, f) - if not os.path.exists(p): continue - if find(T, f[:-3] if f.endswith(".py") else f) is None: - new.add(f) - for m in imports_of(p, f): - rm = find(W, m) - if rm is None or rm in seen: continue - seen.add(rm) - rb = find(T, m) - if rb is None: - new.add(rm); q.append(rm) - else: - a, b = open(os.path.join(T, rb)).readlines(), open(os.path.join(W, rm)).readlines() - if a != b: - drift[rm] = sum(1 for l in difflib.unified_diff(a, b, n=0) - if l[:1] in "+-" and l[:3] not in ("---", "+++")) - q.append(rm) - -print(f" NEW files: {len(new)}") -for f in sorted(new): - n = sum(1 for _ in open(os.path.join(W, f))) if os.path.exists(os.path.join(W, f)) else 0 - print(f" {n:6} {f}") -print(f" DRIFTED existing files: {len(drift)}") -for f, d in sorted(drift.items(), key=lambda kv: -kv[1])[:15]: - print(f" {d:6} lines {f}") -PY diff --git a/.probe_dsv4_csv.sh b/.probe_dsv4_csv.sh deleted file mode 100644 index 19d6a58ae..000000000 --- a/.probe_dsv4_csv.sh +++ /dev/null @@ -1,27 +0,0 @@ -#!/usr/bin/env bash -# The imgdiff run flagged 'configs/model_configs/a8w8_blockscale_tuned_gemm_dsv4.csv' -# as +46 lines in the vendor tree. Tuned-GEMM configs are data, not code: if the -# dsv4 csv is vendor-only, the tuned shapes for this model simply do not exist -# in the nightly line and the GEMMs fall back to untuned dispatch. -set -u -D="${1:-/home/jiacao/3way-20260812-2214}" - -for side in ref vendor target; do - A=$(find "$D/$side" -maxdepth 9 -type d -name aiter -path '*dist-packages*' 2>/dev/null | head -1) - echo "=== $side" - [ -z "$A" ] && { echo " "; continue; } - echo " -- dsv4 csv files --" - ls "$A/configs/model_configs/" 2>/dev/null | grep -i dsv4 | sed 's/^/ /' || true - n=$(ls "$A/configs/model_configs/" 2>/dev/null | grep -ci dsv4) - echo " count: $n" - f="$A/configs/model_configs/a8w8_blockscale_tuned_gemm_dsv4.csv" - if [ -f "$f" ]; then - echo " a8w8_blockscale_tuned_gemm_dsv4.csv: $(wc -l < "$f") lines" - fi - # gluon / sparse attention entry points, by name rather than by path - echo " -- gluon sparse attention --" - for pat in "pa_decode_sparse" "_gluon_kernels" "mla_gluon"; do - c=$(grep -rl -- "$pat" "$A" --include='*.py' 2>/dev/null | wc -l) - printf " %-20s %s file(s)\n" "$pat" "$c" - done -done diff --git a/.probe_final_markers.sh b/.probe_final_markers.sh deleted file mode 100644 index eb4c3f5de..000000000 --- a/.probe_final_markers.sh +++ /dev/null @@ -1,53 +0,0 @@ -#!/usr/bin/env bash -# Final marker sweep: the precise identifiers each vendor hunk introduces, -# checked against the target tree. Anything ABSENT in target is a real addition -# we must carry; anything PRESENT means upstream already landed it in the 465 -# commits between ref and target and the hunk should be dropped, not forced. -set -u -D=/home/jiacao/3way-20260812-2214 -R="$D/ref/usr/local/lib/python3.12/dist-packages/vllm" -V="$D/vendor/src/vllm/vllm" -T="$D/target/usr/local/lib/python3.12/dist-packages/vllm" - -chk() { # $1=marker $2=feature-label - local m="$1" lbl="$2" - local r v t - r=$(grep -rl -- "$m" "$R" --include='*.py' 2>/dev/null | wc -l) - v=$(grep -rl -- "$m" "$V" --include='*.py' 2>/dev/null | wc -l) - t=$(grep -rl -- "$m" "$T" --include='*.py' 2>/dev/null | wc -l) - local verdict - if [ "$v" -eq 0 ]; then verdict="n/a" - elif [ "$t" -gt 0 ] && [ "$r" -gt 0 ]; then verdict="pre-existing" - elif [ "$t" -gt 0 ]; then verdict="UPSTREAM-LANDED" - else verdict="MUST-ADD" - fi - printf " r=%-3s v=%-3s t=%-3s %-16s %-46s %s\n" "$r" "$v" "$t" "$verdict" "$m" "$lbl" -} - -echo "=== MegaMoE ===" -chk "flydsl_mega_moe" "kernel-backend enum value" -chk "use_mega_moe" "layer flag" -chk "mega_moe_experts" "new module" -chk "mega_moe_runtime" "new module" -chk "finalize_mega_moe_layers" "post-load hook" -chk "make_deepseek_v4_mega_expert_params_mapping" "weight mapping" - -echo "=== Gluon sparse attention ===" -chk "VLLM_ROCM_DSV4_SPARSE_GLUON" "env knob" -chk "pa_decode_sparse" "aiter entry point" - -echo "=== FSE / heterogeneous shared expert ===" -chk "shared_expert_id" "fused_moe kwarg" -chk "fused_moe_supports_heterogeneous_shared_expert" "capability probe" - -echo "=== tuned GEMM (DSv4 attention projections) ===" -chk "from aiter.tuned_gemm import tgemm" "tgemm import" - -echo "=== misc plumbing the above depend on ===" -chk "cudagraph_warmup_context" "platform hook" -chk "gemm_a8w8_blockscale_bpreshuffle" "bpreshuffle GEMM op" -chk "aiter_per1x128_quant" "per-1x128 quant" -chk "transpose_scale" "quant kwarg" -chk "process_weights_after_loading()" "model-level post-load hook" -chk "_fused_wqa_wkv_gemm" "attention refactor hook" -chk "swiglu_limit" "swiglu limit passthrough" diff --git a/.probe_gluon_build.sh b/.probe_gluon_build.sh deleted file mode 100644 index b43f4975d..000000000 --- a/.probe_gluon_build.sh +++ /dev/null @@ -1,75 +0,0 @@ -#!/usr/bin/env bash -# Build the candidate gluon patch for real: take the two files from aiter main -# (#4382's descendant, byte-identical to the measurement image), apply #4673 on -# top, drop them onto a copy of the base image's aiter, and check that every -# name the routing file imports actually resolves. This is the go/no-go for the -# gluon arm being deliverable as a container patch. -set -u -W=/tmp/aitermain/aiter -T=/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter -S=/tmp/gluonstage -rm -rf "$S"; mkdir -p "$S" -cp -a "$T" "$S/aiter" - -mkdir -p "$S/aiter/ops/triton/_gluon_kernels/gfx950/attention" -cp "$W/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py" \ - "$S/aiter/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py" -[ -f "$W/ops/triton/_gluon_kernels/gfx950/attention/__init__.py" ] && \ - cp "$W/ops/triton/_gluon_kernels/gfx950/attention/__init__.py" \ - "$S/aiter/ops/triton/_gluon_kernels/gfx950/attention/__init__.py" -cp "$W/ops/triton/attention/pa_decode_sparse.py" "$S/aiter/ops/triton/attention/pa_decode_sparse.py" - -echo "=== apply #4673 on top (aiter/ only) ===" -cd "$S" || exit 1 -git init -q .; git add -A -f >/dev/null 2>&1 -git -c user.email=x@y -c user.name=x commit -qm stage >/dev/null 2>&1 -out=$(git apply --include='aiter/*' -p1 --check /tmp/dsv4patch/aiter-4673.diff 2>&1) -if [ -z "$out" ]; then - git apply --include='aiter/*' -p1 /tmp/dsv4patch/aiter-4673.diff && echo " #4673 APPLIED CLEAN" -else - echo " #4673 CONFLICT:"; echo "$out" | sed 's/^/ /' -fi - -echo -echo "=== does gfx950/attention have an __init__.py in main? ===" -ls "$W/ops/triton/_gluon_kernels/gfx950/attention/" | sed 's/^/ /' -echo " gfx950/ dir:"; ls "$W/ops/triton/_gluon_kernels/gfx950/" | sed 's/^/ /' - -echo -echo "=== resolve every name the staged routing file imports ===" -python3 - <<'PY' -import ast, os -S = "/tmp/gluonstage/aiter" -FILES = ["ops/triton/attention/pa_decode_sparse.py", - "ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py", - "ops/triton/utils/common_utils.py"] -def find(rel): - for s in (".py", "/__init__.py"): - if os.path.exists(os.path.join(S, rel + s)): return rel + s - return None -bad = 0 -for f in FILES: - for n in ast.walk(ast.parse(open(os.path.join(S, f)).read())): - if isinstance(n, ast.ImportFrom) and n.module and n.module.startswith("aiter"): - rel = n.module[len("aiter"):].lstrip(".").replace(".", "/") - tgt = find(rel) - if tgt is None: - print(f" MODULE-ABSENT {n.module} (from {f})"); bad += 1; continue - text = open(os.path.join(S, tgt)).read() - for a in n.names: - if a.name != "*" and a.name not in text: - print(f" SYMBOL-ABSENT {n.module}.{a.name} (from {f})"); bad += 1 -if not bad: - print(" all aiter imports resolve") -PY - -echo -echo "=== syntax check ===" -python3 -m py_compile \ - "$S/aiter/ops/triton/attention/pa_decode_sparse.py" \ - "$S/aiter/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py" \ - "$S/aiter/ops/triton/utils/common_utils.py" && echo " OK" - -echo -echo "=== is max_addressable_bytes now wired into the decision? ===" -grep -n "max_addressable_bytes\|use_buffer_load" "$S/aiter/ops/triton/attention/pa_decode_sparse.py" | head diff --git a/.probe_gluon_drift.sh b/.probe_gluon_drift.sh deleted file mode 100644 index c3cef5f8f..000000000 --- a/.probe_gluon_drift.sh +++ /dev/null @@ -1,37 +0,0 @@ -#!/usr/bin/env bash -# arch_info is a submodule, not a symbol -- the earlier MODULE-ABSENT was a -# false positive. So the gluon surface resolves. Remaining question: how much of -# the base->main delta on the two shared files is #4382/#4673, and how much is -# unrelated drift we'd be dragging in by copying main's version wholesale? -set -u -W=/tmp/aitermain/aiter -T=/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter - -echo "=== base(v0.1.19) -> main(97d0c6e4) : ops/triton/attention/pa_decode_sparse.py" -diff -u "$T/ops/triton/attention/pa_decode_sparse.py" "$W/ops/triton/attention/pa_decode_sparse.py" | diffstat 2>/dev/null \ - || diff -u "$T/ops/triton/attention/pa_decode_sparse.py" "$W/ops/triton/attention/pa_decode_sparse.py" | grep -c '^[+-]' - -echo -echo "=== the same file, but versus the vendor (measurement) image ===" -V=/home/jiacao/3way-20260812-2214/vendor/usr/local/lib/python3.12/dist-packages/aiter -if cmp -s "$V/ops/triton/attention/pa_decode_sparse.py" "$W/ops/triton/attention/pa_decode_sparse.py"; then - echo " vendor == main : the measurement image runs stock upstream here" -else - echo " vendor != main, delta:" - diff -u "$V/ops/triton/attention/pa_decode_sparse.py" "$W/ops/triton/attention/pa_decode_sparse.py" | head -40 -fi - -echo -echo "=== gluon kernel file: vendor vs main ===" -if cmp -s "$V/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py" \ - "$W/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py"; then - echo " vendor == main" -else - diff -u "$V/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py" \ - "$W/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py" | grep -c '^[+-]' -fi - -echo -echo "=== common_utils.py: does main already carry #4673? ===" -grep -n "max_addressable_bytes" "$W/ops/triton/utils/common_utils.py" | head -echo " (empty above = #4673 not merged yet, must come from the PR)" diff --git a/.probe_gluon_only.sh b/.probe_gluon_only.sh deleted file mode 100644 index fbdbfd9bc..000000000 --- a/.probe_gluon_only.sh +++ /dev/null @@ -1,80 +0,0 @@ -#!/usr/bin/env bash -# Narrow the question to the highest-value, smallest patch: the gluon sparse -# decode path (#4382 + #4673) alone. Its aiter surface is 3 files. Check every -# symbol it pulls from the DRIFTED modules and confirm the base image's copies -# already define them -- if so, the gluon patch is a pure 3-file drop-in and -# needs no aiter sync at all. -set -u -W=/tmp/aitermain/aiter -T=/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter - -FILES="ops/triton/attention/pa_decode_sparse.py ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py ops/triton/utils/common_utils.py" - -echo "=== imported symbols and where they resolve ===" -for f in $FILES; do - echo "--- $f" - grep -nE '^\s*(from|import)\s+' "$W/$f" | sed 's/^/ /' -done - -echo -echo "=== symbol-by-symbol availability in the base image ===" -python3 - <<'PY' -import ast, os, sys -W = "/tmp/aitermain/aiter" -T = "/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter" -FILES = [ - "ops/triton/attention/pa_decode_sparse.py", - "ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py", - "ops/triton/utils/common_utils.py", -] -new_files = set(FILES) - -def defined_names(path): - try: - tree = ast.parse(open(path).read()) - except Exception: - return None - names = set() - for n in tree.body: - if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)): - names.add(n.name) - elif isinstance(n, ast.Assign): - for t in n.targets: - if isinstance(t, ast.Name): - names.add(t.id) - elif isinstance(n, ast.AnnAssign) and isinstance(n.target, ast.Name): - names.add(n.target.id) - return names - -missing = [] -for f in FILES: - tree = ast.parse(open(os.path.join(W, f)).read()) - for n in ast.walk(tree): - if not isinstance(n, ast.ImportFrom) or not n.module: - continue - if not n.module.startswith("aiter"): - continue - rel = n.module[len("aiter"):].lstrip(".").replace(".", "/") - if rel + ".py" in new_files: - continue # provided by the patch itself - cand = os.path.join(T, rel + ".py") - if not os.path.exists(cand): - cand = os.path.join(T, rel, "__init__.py") - if not os.path.exists(cand): - for a in n.names: - missing.append((f, n.module, a.name, "MODULE-ABSENT")) - continue - have = defined_names(cand) or set() - # names re-exported through __init__ are hard to see statically; also - # accept a textual hit anywhere in the file. - text = open(cand).read() - for a in n.names: - if a.name not in have and a.name not in text: - missing.append((f, n.module, a.name, "SYMBOL-ABSENT")) - -if missing: - for f, m, s, why in missing: - print(f" {why:14} {m}.{s} (needed by {f})") -else: - print(" all imported aiter symbols resolve against the base image") -PY diff --git a/.probe_gluon_seed_fix.sh b/.probe_gluon_seed_fix.sh deleted file mode 100644 index c031244e2..000000000 --- a/.probe_gluon_seed_fix.sh +++ /dev/null @@ -1,42 +0,0 @@ -#!/usr/bin/env bash -# My closure script only measured drift on IMPORTED modules, never on the seed -# files themselves -- so the "gluon needs 0 existing changes" line understates -# it. Measure the seed files directly, plus common_utils.py which #4673 touches. -set -u -python3 - <<'PY' -import os, subprocess, difflib -W = "/tmp/aitermain" -subprocess.run(["git", "-C", W, "checkout", "-q", "v0.1.19.post2"], check=True) -P = os.path.join(W, "aiter") -V = "/home/jiacao/3way-20260812-2214/vendor/usr/local/lib/python3.12/dist-packages/aiter" - -FILES = [ - "ops/triton/attention/pa_decode_sparse.py", # routing (#4382) - "ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py", # kernel (#4382) - "ops/triton/utils/common_utils.py", # #4673 -] -for f in FILES: - pp, vp = os.path.join(P, f), os.path.join(V, f) - if not os.path.exists(pp): - n = sum(1 for _ in open(vp)) - print(f" NEW FILE {n:5} lines {f}") - continue - a, b = open(pp).readlines(), open(vp).readlines() - if a == b: - print(f" IDENTICAL {f}") - continue - d = sum(1 for l in difflib.unified_diff(a, b, n=0) - if l[:1] in "+-" and l[:3] not in ("---", "+++")) - print(f" MODIFIED {d:5} lines {f} (post2 {len(a)} -> vendor {len(b)})") - -print() -print("=== does the vendor's common_utils.py carry #4673's max_addressable_bytes? ===") -src = open(os.path.join(V, "ops/triton/utils/common_utils.py")).read() -print(" max_addressable_bytes in vendor:", "YES" if "max_addressable_bytes" in src else "NO") -src2 = open(os.path.join(V, "ops/triton/attention/pa_decode_sparse.py")).read() -print(" max_addressable_bytes used by vendor routing:", - "YES" if "max_addressable_bytes" in src2 else "NO") -import re -m = re.search(r"use_buffer_load\s*=.*", src2) -print(" vendor's use_buffer_load decision:", m.group(0).strip() if m else "(not found)") -PY diff --git a/.probe_import_closure.sh b/.probe_import_closure.sh deleted file mode 100644 index 3da9db840..000000000 --- a/.probe_import_closure.sh +++ /dev/null @@ -1,52 +0,0 @@ -#!/usr/bin/env bash -# 444/538 shared aiter files drift between v0.1.19 and the MegaMoE merge, so a -# wholesale sync is out. The narrow question instead: if we copy ONLY the -# feature-carrying files from main onto the base image, what do they import that -# the base image does not have? That set is the true blast radius. -set -u -W=/tmp/aitermain/aiter -T=/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter - -FEATURE_FILES=" -ops/flydsl/kernels/mega_moe/__init__.py -ops/flydsl/kernels/mega_moe/dispatch.py -ops/flydsl/kernels/mega_moe/gemm1.py -ops/flydsl/kernels/mega_moe/gemm2.py -ops/flydsl/kernels/mega_moe/gemm_util.py -ops/flydsl/kernels/mega_moe/mega_moe_config.py -ops/flydsl/kernels/mega_moe/mega_moe_stage1.py -ops/flydsl/kernels/mega_moe/mega_moe_stage2.py -ops/flydsl/kernels/mega_moe/mega_moe_v2.py -ops/flydsl/kernels/mega_moe/quant.py -ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py -ops/triton/attention/pa_decode_sparse.py -ops/triton/utils/common_utils.py -fhmoe.py -ops/flydsl/fhmoe.py -ops/flydsl/kernels/fhmoe.py -" - -echo "=== what these files import from aiter, and whether the base image has it ===" -for f in $FEATURE_FILES; do - [ -f "$W/$f" ] || { echo " !! not in main: $f"; continue; } - grep -hoE '^\s*(from|import)\s+aiter[.a-zA-Z0-9_]*' "$W/$f" -done | sed -E 's/^\s*(from|import)\s+//' | sort -u > /tmp/imports.txt - -while IFS= read -r mod; do - rel="${mod#aiter}"; rel="${rel#.}"; rel="${rel//./\/}" - if [ -z "$rel" ]; then continue; fi - if [ -f "$T/$rel.py" ] || [ -d "$T/$rel" ]; then - # exists -- but does it differ from main? - if [ -f "$T/$rel.py" ] && [ -f "$W/$rel.py" ] && ! cmp -s "$T/$rel.py" "$W/$rel.py"; then - echo " DRIFTED aiter.$(echo "$rel" | tr '/' '.')" - else - echo " ok aiter.$(echo "$rel" | tr '/' '.')" - fi - else - echo " MISSING aiter.$(echo "$rel" | tr '/' '.')" - fi -done < /tmp/imports.txt - -echo -echo "=== do the FlyDSL kernels depend on the compiled csrc that changed? ===" -grep -l "import aiter.jit\|from aiter.jit\|compile_ops\|get_module" $(for f in $FEATURE_FILES; do [ -f "$W/$f" ] && echo "$W/$f"; done) 2>/dev/null | sed "s|$W/| |" diff --git a/.probe_megamoe_closure.sh b/.probe_megamoe_closure.sh deleted file mode 100644 index 6adb8d3c8..000000000 --- a/.probe_megamoe_closure.sh +++ /dev/null @@ -1,70 +0,0 @@ -#!/usr/bin/env bash -# Gluon is a clean drop-in (vendor == main on both files). Now the same test for -# MegaMoE: resolve every symbol the 10 mega_moe files import from aiter against -# the BASE image. Anything absent means MegaMoE cannot be delivered as a file -# copy and needs an aiter sync instead. -set -u -python3 - <<'PY' -import ast, os -W = "/tmp/aitermain/aiter" -T = "/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter" -V = "/home/jiacao/3way-20260812-2214/vendor/usr/local/lib/python3.12/dist-packages/aiter" - -MEGA = ["ops/flydsl/kernels/mega_moe/" + f for f in - ["__init__.py","dispatch.py","gemm1.py","gemm2.py","gemm_util.py", - "mega_moe_config.py","mega_moe_stage1.py","mega_moe_stage2.py", - "mega_moe_v2.py","quant.py"]] -new = set(MEGA) - -def resolve(root, mod): - rel = mod[len("aiter"):].lstrip(".").replace(".", "/") - for c in (os.path.join(root, rel + ".py"), os.path.join(root, rel, "__init__.py")): - if os.path.exists(c): - return c - return None - -problems, ok = [], 0 -for f in MEGA: - p = os.path.join(W, f) - if not os.path.exists(p): - problems.append(("FILE-ABSENT-IN-MAIN", f, "")); continue - for n in ast.walk(ast.parse(open(p).read())): - if isinstance(n, ast.ImportFrom) and n.module and n.module.startswith("aiter"): - rel = n.module[len("aiter"):].lstrip(".").replace(".", "/") - if rel + ".py" in new or rel + "/__init__.py" in new: - continue - tgt = resolve(T, n.module) - if tgt is None: - problems.append(("MODULE-ABSENT", n.module, f)); continue - text = open(tgt).read() - for a in n.names: - if a.name == "*": - continue - if a.name not in text: - problems.append(("SYMBOL-ABSENT", f"{n.module}.{a.name}", f)) - else: - ok += 1 - elif isinstance(n, ast.Import): - for a in n.names: - if a.name.startswith("aiter") and resolve(T, a.name) is None: - problems.append(("MODULE-ABSENT", a.name, f)) - -seen = set() -for why, what, where in problems: - k = (why, what) - if k in seen: continue - seen.add(k) - print(f" {why:16} {what}") -print(f" ({ok} imported symbols resolved cleanly)") - -print() -print("=== mega_moe: vendor vs main, file by file ===") -import filecmp -for f in MEGA: - v, w = os.path.join(V, f), os.path.join(W, f) - if not os.path.exists(v): print(f" vendor-absent {os.path.basename(f)}"); continue - if not os.path.exists(w): print(f" main-absent {os.path.basename(f)}"); continue - same = filecmp.cmp(v, w, shallow=False) - lv, lw = sum(1 for _ in open(v)), sum(1 for _ in open(w)) - print(f" {'identical' if same else 'DIFFERS ':14} {os.path.basename(f):24} vendor={lv:5} main={lw:5}") -PY diff --git a/.probe_megamoe_closure2.sh b/.probe_megamoe_closure2.sh deleted file mode 100644 index 76aa92e14..000000000 --- a/.probe_megamoe_closure2.sh +++ /dev/null @@ -1,71 +0,0 @@ -#!/usr/bin/env bash -# Transitive closure: start from the 10 mega_moe files, add buffer_ops.py, and -# keep pulling in every aiter module they import that the base image lacks, -# until the set stops growing. That closure IS the MegaMoE file-copy manifest. -set -u -python3 - <<'PY' -import ast, os, collections -W = "/tmp/aitermain/aiter" -T = "/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter" - -def relpath_for(mod): - return mod[len("aiter"):].lstrip(".").replace(".", "/") - -def find(root, mod): - rel = relpath_for(mod) - for suffix in (".py", "/__init__.py"): - c = os.path.join(root, rel + suffix) - if os.path.exists(c): - return rel + suffix - return None - -seed = ["ops/flydsl/kernels/mega_moe/" + f for f in - ["__init__.py","dispatch.py","gemm1.py","gemm2.py","gemm_util.py", - "mega_moe_config.py","mega_moe_stage1.py","mega_moe_stage2.py", - "mega_moe_v2.py","quant.py"]] - -closure = set(seed) -queue = collections.deque(seed) -drifted = set() -while queue: - f = queue.popleft() - p = os.path.join(W, f) - if not os.path.exists(p): - continue - mods = set() - for n in ast.walk(ast.parse(open(p).read())): - if isinstance(n, ast.ImportFrom) and n.module and n.module.startswith("aiter"): - mods.add(n.module) - for a in n.names: # `from pkg import submodule` form - mods.add(n.module + "." + a.name) - elif isinstance(n, ast.Import): - for a in n.names: - if a.name.startswith("aiter"): - mods.add(a.name) - for m in mods: - rel_main = find(W, m) - if rel_main is None: - continue - rel_base = find(T, m) - if rel_base is None: - if rel_main not in closure: - closure.add(rel_main); queue.append(rel_main) - else: - bp, wp = os.path.join(T, rel_base), os.path.join(W, rel_main) - if os.path.exists(bp) and os.path.exists(wp) and open(bp,'rb').read() != open(wp,'rb').read(): - drifted.add(rel_main) - -new = sorted(f for f in closure if find(T, "aiter." + f[:-3].replace("/", ".").removesuffix(".__init__")) is None or not os.path.exists(os.path.join(T, f))) -print(f"=== files to COPY from main (absent in base image): {len(new)}") -for f in new: - n = sum(1 for _ in open(os.path.join(W, f))) if os.path.exists(os.path.join(W, f)) else 0 - print(f" {n:5} {f}") -print() -print(f"=== modules the closure touches that EXIST in base but DRIFTED: {len(drifted)}") -for f in sorted(drifted): - bp, wp = os.path.join(T, f), os.path.join(W, f) - import difflib - a = open(bp).readlines(); b = open(wp).readlines() - d = sum(1 for l in difflib.unified_diff(a, b, n=0) if l[:1] in "+-" and l[:3] not in ("---","+++")) - print(f" {d:5} changed lines {f}") -PY diff --git a/.probe_megamoe_closure3.sh b/.probe_megamoe_closure3.sh deleted file mode 100644 index df46526f7..000000000 --- a/.probe_megamoe_closure3.sh +++ /dev/null @@ -1,101 +0,0 @@ -#!/usr/bin/env bash -# The previous closure missed RELATIVE imports (`from .. import x`), which is -# exactly how mega_moe reaches communication_ops_utils and -# flydsl_dispatch_combine_intranode_op -- two files #4439 modifies in place. -# Redo the closure with relative-import resolution, and also report the external -# (non-aiter) package requirements: flydsl, mori. -set -u -python3 - <<'PY' -import ast, os, collections, difflib -W = "/tmp/aitermain/aiter" -T = "/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter" -V = "/home/jiacao/3way-20260812-2214/vendor/usr/local/lib/python3.12/dist-packages/aiter" - -def find(root, rel): - for s in (".py", "/__init__.py"): - if os.path.exists(os.path.join(root, rel + s)): - return rel + s - return None - -def imports_of(path, self_rel): - """Yield aiter-relative module paths (no extension) imported by `path`.""" - pkg = os.path.dirname(self_rel) - out = set() - for n in ast.walk(ast.parse(open(path).read())): - if isinstance(n, ast.ImportFrom): - if n.level: # relative - base = pkg - for _ in range(n.level - 1): - base = os.path.dirname(base) - mod = os.path.join(base, (n.module or "").replace(".", "/")).rstrip("/") - out.add(mod) - for a in n.names: - out.add(os.path.join(mod, a.name)) - elif n.module and n.module.startswith("aiter"): - mod = n.module[len("aiter"):].lstrip(".").replace(".", "/") - out.add(mod) - for a in n.names: - out.add(os.path.join(mod, a.name)) - elif isinstance(n, ast.Import): - for a in n.names: - if a.name.startswith("aiter"): - out.add(a.name[len("aiter"):].lstrip(".").replace(".", "/")) - return {m for m in out if m} - -seed = ["ops/flydsl/kernels/mega_moe/" + f for f in - ["__init__.py","dispatch.py","gemm1.py","gemm2.py","gemm_util.py", - "mega_moe_config.py","mega_moe_stage1.py","mega_moe_stage2.py", - "mega_moe_v2.py","quant.py"]] -seen = set(seed) -q = collections.deque(seed) -to_copy, drifted = set(seed), {} -while q: - f = q.popleft() - p = os.path.join(W, f) - if not os.path.exists(p): - continue - for m in imports_of(p, f): - rm = find(W, m) - if rm is None or rm in seen: - continue - seen.add(rm) - rb = find(T, m) - if rb is None: - to_copy.add(rm); q.append(rm) - else: - a, b = open(os.path.join(T, rb)).readlines(), open(os.path.join(W, rm)).readlines() - if a != b: - d = sum(1 for l in difflib.unified_diff(a, b, n=0) - if l[:1] in "+-" and l[:3] not in ("---", "+++")) - drifted[rm] = d - q.append(rm) - -print(f"=== NEW files to copy: {len(to_copy)}") -for f in sorted(to_copy): - n = sum(1 for _ in open(os.path.join(W, f))) if os.path.exists(os.path.join(W, f)) else 0 - print(f" {n:5} {f}") - -print() -print(f"=== EXISTING files the closure needs but which DRIFTED base->main: {len(drifted)}") -for f, d in sorted(drifted.items(), key=lambda kv: -kv[1]): - same_as_vendor = "" - vp, wp = os.path.join(V, f), os.path.join(W, f) - if os.path.exists(vp): - same_as_vendor = " (vendor==main)" if open(vp,'rb').read() == open(wp,'rb').read() else " (vendor!=main)" - print(f" {d:5} lines {f}{same_as_vendor}") - -print() -print("=== external (non-aiter) packages the closure requires ===") -ext = set() -for f in sorted(seen | to_copy): - p = os.path.join(W, f) - if not os.path.exists(p): continue - for n in ast.walk(ast.parse(open(p).read())): - if isinstance(n, ast.Import): - for a in n.names: - ext.add(a.name.split(".")[0]) - elif isinstance(n, ast.ImportFrom) and not n.level and n.module: - ext.add(n.module.split(".")[0]) -for e in sorted(ext - {"aiter"}): - print(f" {e}") -PY diff --git a/.probe_megamoe_match.sh b/.probe_megamoe_match.sh deleted file mode 100644 index 3879081aa..000000000 --- a/.probe_megamoe_match.sh +++ /dev/null @@ -1,28 +0,0 @@ -#!/usr/bin/env bash -# Does the vendor image's MegaMoE tree match what aiter PR #4439 merged, or is -# the vendor carrying a private variant? Sizes differ enough between vendor and -# aiter/main that this needs checking at the merge commit itself (97d0c6e4cb7a), -# not at main -- main has 87 commits of drift past post2. -# -# If sizes match at 97d0c6e4, the port is a clean cherry-pick from upstream. -# If they don't, the vendor delta is real and needs its own PR. -set -u -REF="${1:-97d0c6e4cb7a}" -V=$(find /home/jiacao/3way-20260812-2214/vendor -maxdepth 9 -type d -name aiter -path '*dist-packages*' | head -1) - -printf "%-24s %10s %10s %10s %s\n" FILE VENDOR "@$REF" MAIN VERDICT -for f in __init__ dispatch gemm1 gemm2 gemm_util mega_moe_config \ - mega_moe_stage1 mega_moe_stage2 mega_moe_v2 quant; do - p="aiter/ops/flydsl/kernels/mega_moe/$f.py" - lv=$(stat -c %s "$V/ops/flydsl/kernels/mega_moe/$f.py" 2>/dev/null || echo -) - sr=$(gh api "/repos/ROCm/aiter/contents/$p?ref=$REF" --jq '.size' 2>/dev/null) - [[ "$sr" =~ ^[0-9]+$ ]] || sr=- - sm=$(gh api "/repos/ROCm/aiter/contents/$p?ref=main" --jq '.size' 2>/dev/null) - [[ "$sm" =~ ^[0-9]+$ ]] || sm=- - if [ "$lv" = "$sr" ]; then v="EXACT@ref" - elif [ "$lv" = "$sm" ]; then v="EXACT@main" - elif [ "$sr" = "-" ]; then v="UPSTREAM-MISSING" - else v="DIFFERS" - fi - printf "%-24s %10s %10s %10s %s\n" "$f.py" "$lv" "$sr" "$sm" "$v" -done diff --git a/.probe_mori_flydsl.sh b/.probe_mori_flydsl.sh deleted file mode 100644 index 902e9f19f..000000000 --- a/.probe_mori_flydsl.sh +++ /dev/null @@ -1,30 +0,0 @@ -#!/usr/bin/env bash -# MegaMoE needs `flydsl` and `mori` as top-level packages. Are they in the base -# image at all, and at what version relative to the measurement image? If they -# differ these are compiled deps and no Python patch can bridge them. -set -u -D=/home/jiacao/3way-20260812-2214 -for img in target vendor; do - echo "=== $img" - for pkg in flydsl mori; do - found=$(find "$D/$img" -maxdepth 8 -type d -name "$pkg" -not -path '*/__pycache__/*' 2>/dev/null | head -3) - if [ -z "$found" ]; then echo " $pkg: ABSENT"; continue; fi - echo " $pkg: $found" - for d in $found; do - v=$(find "$(dirname "$d")" -maxdepth 1 -name "$pkg*dist-info" -o -maxdepth 1 -name "$pkg*egg-info" 2>/dev/null | head -1) - [ -n "$v" ] && echo " dist: $(basename "$v")" - [ -f "$d/_version.py" ] && echo " _version: $(head -3 "$d/_version.py" | tr '\n' ' ')" - [ -f "$d/version.py" ] && echo " version: $(head -3 "$d/version.py" | tr '\n' ' ')" - echo " .so count: $(find "$d" -name '*.so' | wc -l)" - done - done -done - -echo -echo "=== which files in the BASE image import the 5 drifted flydsl kernels? ===" -T="$D/target/usr/local/lib/python3.12/dist-packages/aiter" -for m in flydsl_dispatch_combine_intranode_op flydsl_dispatch_combine_intranode_kernel tensor_shim mxfp4_gemm_common communication_ops_utils; do - n=$(grep -rl "$m" "$T" --include='*.py' 2>/dev/null | wc -l) - echo " $m: $n importer(s)" - grep -rl "$m" "$T" --include='*.py' 2>/dev/null | sed "s|$T/| |" -done diff --git a/.probe_post2.sh b/.probe_post2.sh deleted file mode 100644 index a7f49106b..000000000 --- a/.probe_post2.sh +++ /dev/null @@ -1,40 +0,0 @@ -#!/usr/bin/env bash -# Which of the vendor-only aiter files exist at tag v0.1.19.post2 (a63ede724b15)? -# -# post2 already contains the PR #4269 merge (compare says ahead=0/behind=24), -# so anything #4269 shipped should be there. Anything still missing came from a -# different source -- either a later upstream PR, or nothing upstream at all. -# That split is exactly what decides "bump the pin" vs "we must open a PR". -set -u -REF="${1:-a63ede724b15}" # v0.1.19.post2 -TAG_NAME="${2:-v0.1.19.post2}" - -check() { # $1 = repo path - # `gh api --jq .size` prints the 404 JSON body on a miss, which is a - # non-empty string -- test for an all-digits size instead of non-emptiness. - local p="$1" s - s=$(gh api "/repos/ROCm/aiter/contents/$p?ref=$REF" --jq '.size' 2>/dev/null) - if [[ "$s" =~ ^[0-9]+$ ]]; then printf " %-8s %-9s %s\n" "PRESENT" "${s}B" "$p" - else printf " %-8s %-9s %s\n" "ABSENT" "-" "$p"; fi -} - -echo "=== $TAG_NAME ($REF) ===" -echo "-- FHMoE / FSE (PR #4269) --" -check aiter/fhmoe.py -check aiter/ops/flydsl/fhmoe.py -check aiter/aot/flydsl/fhmoe.py -check aiter/ops/flydsl/kernels/fhmoe.py -check aiter/ops/flydsl/kernels/mixed_moe_gemm_2stage.py -check aiter/ops/flydsl/kernels/mixed_moe_gemm_2stage_common.py - -echo "-- MegaMoE kernel tree --" -for f in __init__ dispatch gemm1 gemm2 gemm_util mega_moe_config \ - mega_moe_stage1 mega_moe_stage2 mega_moe_v2 quant; do - check "aiter/ops/flydsl/kernels/mega_moe/$f.py" -done - -echo "-- tuned-GEMM data --" -check aiter/configs/model_configs/a8w8_blockscale_tuned_gemm_dsv4.csv - -echo "-- Gluon sparse attention (PR #4382) --" -check aiter/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py diff --git a/.probe_post2_base.sh b/.probe_post2_base.sh deleted file mode 100644 index b6405c9a7..000000000 --- a/.probe_post2_base.sh +++ /dev/null @@ -1,117 +0,0 @@ -#!/usr/bin/env bash -# If v0.1.19.post2 (a real tag, 59 commits past v0.1.19) is the base instead of -# v0.1.19, how big is the gluon + MegaMoE patch? Redo the closure/drift analysis -# against post2, and check the 5 shared flydsl files for breaking signature -# changes -- those were the blocker when measuring against v0.1.19. -set -u -python3 - <<'PY' -import ast, os, subprocess, collections, difflib -W = "/tmp/aitermain" -subprocess.run(["git", "-C", W, "checkout", "-q", "v0.1.19.post2"], check=True) -P = os.path.join(W, "aiter") # post2 tree -V = "/home/jiacao/3way-20260812-2214/vendor/usr/local/lib/python3.12/dist-packages/aiter" - -# --- the vendor's own gluon + megamoe files are the ground truth: they are what -# --- actually produced 12,244. Use the vendor tree as the source, post2 as base. -GLUON = ["ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py", - "ops/triton/attention/pa_decode_sparse.py"] -MEGA = ["ops/flydsl/kernels/mega_moe/" + f for f in - ["__init__.py","dispatch.py","gemm1.py","gemm2.py","gemm_util.py", - "mega_moe_config.py","mega_moe_stage1.py","mega_moe_stage2.py", - "mega_moe_v2.py","quant.py"]] - -def find(root, rel): - for s in (".py", "/__init__.py"): - if os.path.exists(os.path.join(root, rel + s)): return rel + s - return None - -def imports_of(path, self_rel): - pkg = os.path.dirname(self_rel); out = set() - for n in ast.walk(ast.parse(open(path).read())): - if isinstance(n, ast.ImportFrom): - if n.level: - base = pkg - for _ in range(n.level - 1): base = os.path.dirname(base) - mod = os.path.join(base, (n.module or "").replace(".", "/")).rstrip("/") - elif n.module and n.module.startswith("aiter"): - mod = n.module[len("aiter"):].lstrip(".").replace(".", "/") - else: continue - out.add(mod) - for a in n.names: out.add(os.path.join(mod, a.name)) - elif isinstance(n, ast.Import): - for a in n.names: - if a.name.startswith("aiter"): - out.add(a.name[len("aiter"):].lstrip(".").replace(".", "/")) - return {m for m in out if m} - -def closure(seed, label): - seen, q = set(seed), collections.deque(seed) - new, drift = set(), {} - while q: - f = q.popleft() - p = os.path.join(V, f) - if not os.path.exists(p): continue - if find(P, f[:-3]) is None: new.add(f) - for m in imports_of(p, f): - rv = find(V, m) - if rv is None or rv in seen: continue - seen.add(rv) - rp = find(P, m) - if rp is None: - new.add(rv); q.append(rv) - else: - a = open(os.path.join(P, rp)).readlines() - b = open(os.path.join(V, rv)).readlines() - if a != b: - drift[rv] = sum(1 for l in difflib.unified_diff(a, b, n=0) - if l[:1] in "+-" and l[:3] not in ("---","+++")) - q.append(rv) - print(f"=== {label}: onto v0.1.19.post2") - print(f" NEW files to add: {len(new)}") - for f in sorted(new): - n = sum(1 for _ in open(os.path.join(V, f))) - print(f" {n:5} {f}") - print(f" EXISTING files that must change: {len(drift)}") - for f, d in sorted(drift.items(), key=lambda kv: -kv[1]): - print(f" {d:5} lines {f}") - return new, drift - -gnew, gdrift = closure(GLUON, "GLUON (#4382+#4673)") -print() -mnew, mdrift = closure(MEGA, "MEGAMOE (#4439)") - -# --- signature safety for every file that must change --- -print() -print("=== signature check on the files that must change (post2 -> vendor) ===") -def api(path): - out = {} - for n in ast.parse(open(path).read()).body: - if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef)): - a = n.args - out[n.name] = tuple([x.arg for x in a.posonlyargs + a.args] + - (["*"+a.vararg.arg] if a.vararg else []) + - [x.arg for x in a.kwonlyargs] + - (["**"+a.kwarg.arg] if a.kwarg else [])) - elif isinstance(n, ast.ClassDef): out[n.name] = ("",) - return out - -for f in sorted(set(gdrift) | set(mdrift)): - if not f.endswith(".py"): continue - try: - b, v = api(os.path.join(P, f)), api(os.path.join(V, f)) - except Exception as e: - print(f" ?? {f}: {e}"); continue - removed = sorted(set(b) - set(v)) - resigned = sorted(k for k in set(b) & set(v) if b[k] != v[k]) - verdict = "ADDITIVE-SAFE" if not removed and not resigned else "BREAKING" - importers = 0 - for dp, _, fns in os.walk(P): - if "__pycache__" in dp: continue - for fn in fns: - if fn.endswith(".py"): - if os.path.basename(f)[:-3] in open(os.path.join(dp, fn)).read(): - importers += 1 - print(f" {verdict:14} {f} ({importers} importers in post2)") - for k in removed: print(f" removed: {k}") - for k in resigned: print(f" resigned: {k}") -PY diff --git a/.probe_post2_features.sh b/.probe_post2_features.sh deleted file mode 100644 index db5aaf84f..000000000 --- a/.probe_post2_features.sh +++ /dev/null @@ -1,56 +0,0 @@ -#!/usr/bin/env bash -# The measurement image is 85.8% v0.1.19.post2, not main. If post2 already -# carries gluon / MegaMoE / FSE, then the whole "aiter is 107 commits behind" -# framing is wrong: the fix is a TAG BUMP in Dockerfile.rocm_base, not a patch. -set -u -W=/tmp/aitermain -cd "$W" || exit 1 -git checkout -q v0.1.19.post2 - -echo "=== v0.1.19.post2 = $(git rev-parse --short HEAD) ===" -echo " date: $(git log -1 --format=%ci)" -echo " is v0.1.19 an ancestor? $(git merge-base --is-ancestor v0.1.19 HEAD && echo yes || echo no)" -echo " commits v0.1.19..post2: $(git rev-list --count v0.1.19..HEAD)" - -echo -echo "=== feature presence in v0.1.19.post2 ===" -p() { if [ -e "$W/$1" ]; then echo " PRESENT $2"; else echo " MISSING $2"; fi; } -m() { if grep -qF -- "$1" "$W/$2" 2>/dev/null; then echo " PRESENT $3"; else echo " MISSING $3"; fi; } - -p "aiter/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py" "#4382 gluon kernel" -m "_pa_decode_sparse_gfx950_gluon" "aiter/ops/triton/attention/pa_decode_sparse.py" "#4382 gfx950 routing" -m "max_addressable_bytes" "aiter/ops/triton/utils/common_utils.py" "#4673 overflow fix" -p "aiter/ops/flydsl/kernels/mega_moe/mega_moe_v2.py" "#4439 MegaMoE" -p "aiter/fhmoe.py" "#4269 FSE / FHMoE" -m "shared_expert_id" "aiter/fused_moe.py" "#4269 fused_moe shared_expert_id" -p "aiter/configs/model_configs/dsv4_a8w8_blockscale_tuned_gemm.csv" "tuned-gemm csv" - -echo -echo "=== which merged PRs are in post2 but not v0.1.19? ===" -for sha in b3c13c932207bef03aa3a8123bf34acad02f40d3 97d0c6e4cb7a0919c12291c7c7d560ad412f15c1 00cbe979f20cf380548d7d1d9d73136aa359c276; do - case "$sha" in - b3c13c*) n="#4382 gluon" ;; - 97d0c6*) n="#4439 MegaMoE" ;; - 00cbe9*) n="#4269 FSE" ;; - esac - if git merge-base --is-ancestor "$sha" HEAD 2>/dev/null; then - echo " IN post2 $n" - else - echo " NOT in post2 $n" - fi -done - -echo -echo "=== and in v0.1.19 (the base image's pin)? ===" -for sha in b3c13c932207bef03aa3a8123bf34acad02f40d3 97d0c6e4cb7a0919c12291c7c7d560ad412f15c1 00cbe979f20cf380548d7d1d9d73136aa359c276; do - case "$sha" in - b3c13c*) n="#4382 gluon" ;; - 97d0c6*) n="#4439 MegaMoE" ;; - 00cbe9*) n="#4269 FSE" ;; - esac - if git merge-base --is-ancestor "$sha" v0.1.19 2>/dev/null; then - echo " IN v0.1.19 $n" - else - echo " NOT in v0.1.19 $n" - fi -done diff --git a/.probe_so_delta.sh b/.probe_so_delta.sh deleted file mode 100644 index 0a8fd9ea9..000000000 --- a/.probe_so_delta.sh +++ /dev/null @@ -1,19 +0,0 @@ -#!/usr/bin/env bash -# Do the vendor and base images ship the same set of prebuilt aiter modules? -# If the vendor has extra .so files, the container patch cannot be pure-Python. -set -u -D=/home/jiacao/3way-20260812-2214/ -T="$D/target/usr/local/lib/python3.12/dist-packages/aiter" -V="$D/vendor/usr/local/lib/python3.12/dist-packages/aiter" -cd "$T" && find . -name '*.so' | sort > /tmp/so_target.txt -cd "$V" && find . -name '*.so' | sort > /tmp/so_vendor.txt -echo "vendor-only .so:" -comm -13 /tmp/so_target.txt /tmp/so_vendor.txt | sed 's/^/ /' -echo "target-only .so:" -comm -23 /tmp/so_target.txt /tmp/so_vendor.txt | sed 's/^/ /' -echo -echo "hsaco/co asm blobs:" -cd "$T" && find . \( -name '*.co' -o -name '*.hsaco' \) | sort > /tmp/co_target.txt -cd "$V" && find . \( -name '*.co' -o -name '*.hsaco' \) | sort > /tmp/co_vendor.txt -echo " vendor-only: $(comm -13 /tmp/co_target.txt /tmp/co_vendor.txt | wc -l)" -comm -13 /tmp/co_target.txt /tmp/co_vendor.txt | head -20 | sed 's/^/ /' diff --git a/.probe_target_aiter.sh b/.probe_target_aiter.sh deleted file mode 100644 index e53c28062..000000000 --- a/.probe_target_aiter.sh +++ /dev/null @@ -1,36 +0,0 @@ -#!/usr/bin/env bash -# What does the BASE image's aiter (v0.1.19, target/) already carry, and what -# must the container patch add? Anything MISSING here is a real gap for the -# corresponding upstream aiter PR. -set -u -T=/home/jiacao/3way-20260812-2214/target/usr/local/lib/python3.12/dist-packages/aiter - -p() { # $1=path-relative-to-aiter $2=label - if [ -e "$T/$1" ]; then echo " PRESENT $2 ($1)"; else echo " MISSING $2 ($1)"; fi -} -m() { # $1=marker $2=file $3=label - if grep -qF -- "$1" "$T/$2" 2>/dev/null; then echo " PRESENT $3"; else echo " MISSING $3"; fi -} - -echo "== #4382 gfx950 gluon sparse decode ==" -p "ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py" "gluon kernel" -m "_pa_decode_sparse_gfx950_gluon" "ops/triton/attention/pa_decode_sparse.py" "gfx950 routing entry" - -echo "== #4673 buffer_load span fix ==" -m "max_addressable_bytes" "ops/triton/utils/common_utils.py" "max_addressable_bytes helper" - -echo "== #4439 MegaMoE ==" -p "ops/flydsl/kernels/mega_moe/mega_moe_v2.py" "mega_moe package" -p "ops/flydsl/kernels/mega_moe/__init__.py" "mega_moe __init__" - -echo "== #4269 FSE / heterogeneous shared expert (FHMoE) ==" -grep -rlF "shared_expert_id" "$T" --include='*.py' 2>/dev/null | sed 's|^| hit: |' | head -grep -rlF "fhmoe" "$T" --include='*.py' 2>/dev/null | sed 's|^| hit: |' | head - -echo "== #4664 tuned GEMM CSV rows ==" -p "configs/model_configs/dsv4_a8w8_blockscale_tuned_gemm.csv" "dsv4 tuned-gemm csv" -if [ -e "$T/configs/model_configs/dsv4_a8w8_blockscale_tuned_gemm.csv" ]; then - echo " lines: $(wc -l < "$T/configs/model_configs/dsv4_a8w8_blockscale_tuned_gemm.csv")" - grep -c '^6144,7168\|^65536,1536\|^7168,3072' "$T/configs/model_configs/dsv4_a8w8_blockscale_tuned_gemm.csv" \ - | sed 's|^| dsv4 shapes present: |' -fi diff --git a/.probe_tgemm.sh b/.probe_tgemm.sh deleted file mode 100644 index 30825452c..000000000 --- a/.probe_tgemm.sh +++ /dev/null @@ -1,29 +0,0 @@ -#!/usr/bin/env bash -# 'tgemm' as a bare string appears in exactly 1 file on all three sides, which -# means it is not the identifier the DSv4 arm actually uses. Widen the search: -# check aiter's own version, the vendor's aiter provenance, and the plausible -# tuned-GEMM entry points by name. -set -u -D="${1:-/home/jiacao/3way-20260812-2214}" - -for side in ref vendor target; do - A=$(find "$D/$side" -maxdepth 9 -type d -name aiter -path '*dist-packages*' 2>/dev/null | head -1) - echo "=== $side ${A:-}" - [ -z "$A" ] && continue - - echo -n " aiter version: " - grep -hE "^__version__|^version" "$A/_version.py" 2>/dev/null | head -2 | tr '\n' ' '; echo - - echo " -- tgemm hits (any case, with context) --" - grep -rn "tgemm" "$A" --include='*.py' 2>/dev/null | head -5 | sed 's|'"$A"'|.|' | sed 's/^/ /' - - echo " -- tuned-GEMM surface --" - for pat in "tuned_gemm" "TunedGemm" "tuned_gemm_dsv4" "gemm_a8w8_blockscale" "gemm_tune"; do - n=$(grep -rl -- "$pat" "$A" --include='*.py' 2>/dev/null | wc -l) - printf " %-28s %s file(s)\n" "$pat" "$n" - done - - echo " -- tuned-GEMM csv configs --" - ls "$A/configs/model_configs/" 2>/dev/null | grep -iE "gemm|dsv4" | head -6 | sed 's/^/ /' - echo -n " total csv: "; ls "$A/configs/model_configs/"*.csv 2>/dev/null | wc -l -done diff --git a/.probe_vendor_over_post2.sh b/.probe_vendor_over_post2.sh deleted file mode 100644 index 87d6d0a90..000000000 --- a/.probe_vendor_over_post2.sh +++ /dev/null @@ -1,60 +0,0 @@ -#!/usr/bin/env bash -# The measurement image = v0.1.19.post2 + something. Enumerate exactly what that -# something is: which files differ, and does each one trace to #4382 / #4439 / -# #4673 or to an unattributed vendor edit? -set -u -python3 - <<'PY' -import os, subprocess -D = "/home/jiacao/3way-20260812-2214" -W = "/tmp/aitermain" -V = os.path.join(D, "vendor/usr/local/lib/python3.12/dist-packages/aiter") - -def snap(ref): - subprocess.run(["git", "-C", W, "checkout", "-q", ref], check=True) - root = os.path.join(W, "aiter"); out = {} - for dp, _, fns in os.walk(root): - if "__pycache__" in dp: continue - for fn in fns: - if fn.endswith(".py") or fn.endswith(".csv"): - p = os.path.join(dp, fn) - out[os.path.relpath(p, root)] = open(p, "rb").read() - return out - -def img(root): - out = {} - for dp, _, fns in os.walk(root): - if "__pycache__" in dp: continue - for fn in fns: - if fn.endswith(".py") or fn.endswith(".csv"): - p = os.path.join(dp, fn) - out[os.path.relpath(p, root)] = open(p, "rb").read() - return out - -post2 = snap("v0.1.19.post2") -main = snap("97d0c6e4cb7a0919c12291c7c7d560ad412f15c1") -vend = img(V) - -added = sorted(set(vend) - set(post2)) -removed = sorted(set(post2) - set(vend)) -changed = sorted(k for k in set(vend) & set(post2) if vend[k] != post2[k]) - -def origin(k): - """Does the vendor's copy match main's copy? -> upstream cherry-pick.""" - if k in main and vend[k] == main[k]: - return "== main@97d0c6e4" - if k in main: - return "!= main (vendor-modified or older main)" - return "not in main either -> VENDOR-ONLY" - -print(f"=== vendor ADDS {len(added)} files over v0.1.19.post2") -for k in added: - print(f" {origin(k):38} {k}") -print() -print(f"=== vendor CHANGES {len(changed)} files") -for k in changed: - print(f" {origin(k):38} {k}") -print() -print(f"=== vendor REMOVES {len(removed)} files") -for k in removed: - print(f" {k}") -PY diff --git a/.probe_vllm_classify.sh b/.probe_vllm_classify.sh deleted file mode 100644 index 3aca94374..000000000 --- a/.probe_vllm_classify.sh +++ /dev/null @@ -1,37 +0,0 @@ -#!/usr/bin/env bash -# Classify each of the 19 vendor-changed vLLM files by which feature it carries. -# Print the ref->vendor diff for the small ones in full and a keyword census for -# the large ones, so every hunk can be attributed to gluon / MegaMoE / FSE / -# #51473 / other before it goes in a waiver table. -set -u -D=/home/jiacao/3way-20260812-2214 -R="$D/ref/usr/local/lib/python3.12/dist-packages/vllm" -V="$D/vendor/src/vllm/vllm" - -FILES=" -config/kernel.py -model_executor/layers/fused_moe/experts/rocm_aiter_moe.py -model_executor/model_loader/utils.py -models/deepseek_v4/attention.py -platforms/interface.py -platforms/rocm.py -v1/worker/gpu/cudagraph_utils.py -v1/worker/gpu_ubatch_wrapper.py -models/deepseek_v4/amd/dspark.py -models/deepseek_v4/amd/mtp.py -v1/worker/gpu_model_runner.py -" -for f in $FILES; do - echo "################ $f" - diff -u "$R/$f" "$V/$f" | tail -n +3 -done - -echo -echo "############ keyword census for the large files ############" -for f in _aiter_ops.py models/deepseek_v4/amd/model.py models/deepseek_v4/amd/rocm.py \ - v1/attention/ops/rocm_aiter_mla_sparse.py; do - echo "--- $f" - diff -u "$R/$f" "$V/$f" | grep '^+' | grep -oiE \ - "gluon|mega_?moe|flydsl|fhmoe|fusion_shared|tgemm|tuned_gemm|sparse|mxfp4|dspark" \ - | tr 'A-Z' 'a-z' | sort | uniq -c | sort -rn | sed 's/^/ /' -done diff --git a/.probe_vllm_delta.sh b/.probe_vllm_delta.sh deleted file mode 100644 index 35b9782c4..000000000 --- a/.probe_vllm_delta.sh +++ /dev/null @@ -1,38 +0,0 @@ -#!/usr/bin/env bash -# The vendor_patchset.vllm.diff was generated per-file with a fixed --label, so -# the diff headers all read "a/vllm b/vllm" and the filenames are lost. Recover -# them by re-diffing the ref and vendor trees directly, and for each changed -# file report its churn plus whether the target already has the same change. -# -# ref = the vendor's upstream base (vLLM 02e63f2e4, 2026-07-30) -# vendor = the measurement image (ref + the vendor patch set) -# target = the nightly we want to land on (b22afe45, 465 commits past ref) -set -u -D=/home/jiacao/3way-20260812-2214 -R=$(find "$D/ref" -maxdepth 9 -type d -name vllm -path '*packages*' | head -1) -# The vendor image installs vLLM editable from /src/vllm, so its package root is -# /src/vllm/vllm rather than a dist-packages path like the other two sides. -V="$D/vendor/src/vllm/vllm" -T=$(find "$D/target" -maxdepth 9 -type d -name vllm -path '*packages*' | head -1) -echo "ref=$R"; echo "vendor=$V"; echo "target=$T"; echo - -printf "%-6s %6s %6s %-9s %s\n" STATUS +LINES -LINES "IN-TARGET" FILE -( cd "$V" && find . -name '*.py' -not -path '*__pycache__*' | sort ) | while read -r rel; do - rel=${rel#./} - [ "$rel" = "_version.py" ] && continue - vf="$V/$rel"; rf="$R/$rel"; tf="$T/$rel" - if [ ! -f "$rf" ]; then - st=NEW; add=$(wc -l < "$vf"); del=0 - else - cmp -s "$rf" "$vf" && continue - st=MOD - add=$(diff "$rf" "$vf" | grep -c '^>') - del=$(diff "$rf" "$vf" | grep -c '^<') - fi - # Does the target already carry this file, and does it already match vendor? - if [ ! -f "$tf" ]; then int="MISSING" - elif cmp -s "$tf" "$vf"; then int="SAME" - else int="DIFFERS" - fi - printf "%-6s %6s %6s %-9s %s\n" "$st" "$add" "$del" "$int" "$rel" -done diff --git a/.probe_vllm_markers.sh b/.probe_vllm_markers.sh deleted file mode 100644 index f5ace13e4..000000000 --- a/.probe_vllm_markers.sh +++ /dev/null @@ -1,49 +0,0 @@ -#!/usr/bin/env bash -# For each vendor-only vLLM change, pick a marker string that only exists if the -# feature is present, and check it in ref / target / vendor. -# -# "DIFFERS" from the file-level probe is not evidence of a missing feature -- -# 465 upstream commits touch these files anyway. Markers are. -# ref absent, vendor present, target absent -> we must add it -# ref absent, vendor present, target present -> upstream already landed it -set -u -D=/home/jiacao/3way-20260812-2214 -R="$D/ref/usr/local/lib/python3.12/dist-packages/vllm" -V="$D/vendor/src/vllm/vllm" -T="$D/target/usr/local/lib/python3.12/dist-packages/vllm" - -row() { # $1=marker $2=relpath-or-TREE $3=label - local m="$1" p="$2" lbl="$3" out="" - for side in R V T; do - local base; base=$(eval echo "\$$side") - local n - if [ "$p" = "TREE" ]; then - n=$(grep -rl -- "$m" "$base" --include='*.py' 2>/dev/null | wc -l) - elif [ -f "$base/$p" ]; then - n=$(grep -c -- "$m" "$base/$p" 2>/dev/null || echo 0) - else - n="-" - fi - out="$out$(printf '%6s' "$n")" - done - printf "%s %-34s %s\n" "$out" "$lbl" "$m" -} - -echo " ref vendor target FEATURE MARKER" -row "VLLM_ROCM_DSV4_SPARSE_GLUON" TREE "gluon env knob" -row "pa_decode_sparse" TREE "gluon sparse decode call" -row "mla_gluon" TREE "gluon MLA" -row "flydsl_mega_moe" TREE "MegaMoE backend name" -row "mega_moe" TREE "MegaMoE (any)" -row "MegaMoE" TREE "MegaMoE (class)" -row "FUSION_SHARED_EXPERTS" TREE "FSE env knob" -row "fhmoe" TREE "FHMoE" -row "AITER_MXFP4_BF16" TREE "PR #51473 marker" -row "tuned_gemm" TREE "tuned GEMM" -echo -echo "-- new files --" -for f in models/deepseek_v4/amd/mega_moe_experts.py models/deepseek_v4/amd/mega_moe_runtime.py; do - printf " ref=%-9s target=%-9s %s\n" \ - "$([ -f "$R/$f" ] && echo yes || echo no)" \ - "$([ -f "$T/$f" ] && echo yes || echo no)" "$f" -done diff --git a/.probe_vllm_markers2.sh b/.probe_vllm_markers2.sh deleted file mode 100644 index e563ea66d..000000000 --- a/.probe_vllm_markers2.sh +++ /dev/null @@ -1,35 +0,0 @@ -#!/usr/bin/env bash -# Second pass: the coarse markers were too coarse. FUSION_SHARED_EXPERTS and -# AITER_MXFP4_BF16 appear on all three sides, so their mere presence proves -# nothing -- the vendor delta is in the specific condition / call site. -set -u -D=/home/jiacao/3way-20260812-2214 -R="$D/ref/usr/local/lib/python3.12/dist-packages/vllm" -V="$D/vendor/src/vllm/vllm" -T="$D/target/usr/local/lib/python3.12/dist-packages/vllm" - -echo "=== PR #51473 exact condition (oracle/mxfp4.py) ===" -for s in R V T; do b=$(eval echo "\$$s") - printf " %-7s " "$s" - grep -n "AITER_MXFP4_BF16" "$b/model_executor/layers/fused_moe/oracle/mxfp4.py" 2>/dev/null \ - | head -4 | tr '\n' '|' ; echo -done - -echo -echo "=== FSE: where do the extra 7 vendor hits live? ===" -echo "--- vendor files mentioning FUSION_SHARED_EXPERTS ---" -grep -rl "FUSION_SHARED_EXPERTS" "$V" --include='*.py' 2>/dev/null | sed "s|$V/| |" -echo "--- target files mentioning FUSION_SHARED_EXPERTS ---" -grep -rl "FUSION_SHARED_EXPERTS" "$T" --include='*.py' 2>/dev/null | sed "s|$T/| |" - -echo -echo "=== MegaMoE: which files, each side ===" -for s in R V T; do b=$(eval echo "\$$s"); echo "--- $s ---" - grep -rl "mega_moe\|MegaMoE" "$b" --include='*.py' 2>/dev/null | sed "s|$b/| |" -done - -echo -echo "=== tuned_gemm call sites ===" -for s in R V T; do b=$(eval echo "\$$s"); echo "--- $s ---" - grep -rn "tuned_gemm" "$b" --include='*.py' 2>/dev/null | sed "s|$b/| |" | head -6 -done diff --git a/.verify_dsv4_image.sh b/.verify_dsv4_image.sh deleted file mode 100644 index d0c3373b2..000000000 --- a/.verify_dsv4_image.sh +++ /dev/null @@ -1,89 +0,0 @@ -#!/usr/bin/env bash -#SBATCH --job-name=dsv4-verify -#SBATCH --account=amd-aifw-aim -#SBATCH --qos=amd-aifw-aim-qos -#SBATCH --partition=amd-spur -#SBATCH --nodes=1 -#SBATCH --cpus-per-task=4 -#SBATCH --mem=16G -#SBATCH --time=00:15:00 -#SBATCH --nodelist=crsuse2-m2m-006 -#SBATCH --output=/home/jiacao/InferenceX/dsv4-verify-%j.out -# -# Ground-truth the committed image dsv4-pro-fp4-mi355x:f8d03e77-patched. The build -# job's own verify greps were mis-targeted (looked under fused_moe/ for the -# flydsl_mega_moe enum that actually lives in config/kernel.py). These probes call -# the real code paths -- no GPU needed, all pure-Python config/oracle/env reads. -set -uo pipefail -hostname; date -u - -IMG="dsv4-pro-fp4-mi355x:3ee2df30-patched" -docker image inspect "$IMG" >/dev/null 2>&1 \ - || { echo "FATAL: image $IMG not on this node; run on the build node (006)"; exit 1; } -run(){ docker run --rm --entrypoint /bin/bash "$IMG" -c "$1"; } - -echo "=== [DEP8] --moe-backend flydsl_mega_moe accepted by the enum? (#51918) ===" -run 'python3 - </dev/null | head' - -echo -echo "=== [TP8] 384-wide MXFP4 shard oracle (#51473, expect 384) ===" -run 'python3 - <",i) - print("VERDICT:", "384 shard PRESERVED" if i==384 else "ROUNDED to %d"%i) -except Exception: - import sys - traceback.print_exc(file=sys.stdout) - print("VERDICT: PROBE ERROR") -PY' - -echo -echo "=== [TP8] FSE env + wiring (base #4269 present; #48728 DROPPED) ===" -run 'python3 -c "import vllm.envs as e; print(\"VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS =\", e.VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS)"' -run 'V=$(python3 -c "import vllm,os;print(os.path.dirname(vllm.__file__))"); - M="$V/models/deepseek_v4/amd/model.py"; - echo "base OLD FSE (_fuse_shared_experts_enabled), expect >0:"; - grep -c "_fuse_shared_experts_enabled" "$M" 2>/dev/null; - echo "#48728 hetero FSE (_should_fuse_shared_expert), expect 0 (dropped):"; - grep -c "_should_fuse_shared_expert" "$M" 2>/dev/null' - -echo -echo "=== [gluon] env + vllm wiring (#51714) ===" -run 'python3 -c "import vllm.envs as e; print(\"VLLM_ROCM_DSV4_SPARSE_GLUON =\", e.VLLM_ROCM_DSV4_SPARSE_GLUON)"' -run 'V=$(python3 -c "import vllm,os;print(os.path.dirname(vllm.__file__))"); - grep -c "_DSV4_SPARSE_GLUON" "$V/v1/attention/ops/rocm_aiter_mla_sparse.py"' - -echo -echo "=== [#4673] aiter gluon buffer_load int64 span fix present? ===" -run 'V=$(python3 -c "import aiter,os;print(os.path.dirname(aiter.__file__))"); - echo "max_addressable_bytes in common_utils.py:"; - grep -c "max_addressable_bytes" "$V/ops/triton/utils/common_utils.py" 2>/dev/null || echo 0; - echo "max_addressable_bytes in routing pa_decode_sparse.py:"; - grep -c "max_addressable_bytes" "$V/ops/triton/attention/pa_decode_sparse.py" 2>/dev/null || echo 0' - -date -u diff --git a/aiter-4417.diff b/aiter-4417.diff deleted file mode 100644 index f6d4432fc..000000000 --- a/aiter-4417.diff +++ /dev/null @@ -1,73 +0,0 @@ -diff --git a/aiter/ops/flydsl/moe_kernels.py b/aiter/ops/flydsl/moe_kernels.py -index 4b3253b7c0..454a58b523 100644 ---- a/aiter/ops/flydsl/moe_kernels.py -+++ b/aiter/ops/flydsl/moe_kernels.py -@@ -13,6 +13,9 @@ - - _KERNEL_PARAMS: dict[str, dict] = {} - -+# HIP limits grid.y/grid.z to 65535. -+_HIP_MAX_GRID_DIM_Y = 65535 -+ - - def _get_dtypes(): - from aiter.utility import dtypes -@@ -94,6 +97,25 @@ def pick_flydsl_stage1_tile_n(inter_dim: int) -> int: - return 256 if (inter_dim % 256 == 0) else 128 - - -+def resolve_flydsl_grid_y_persist_m( -+ num_m_blocks: int, requested_persist_m: int = 0 -+) -> int: -+ """Increase persist_m as needed to keep grid.y within HIP's limit.""" -+ num_m_blocks = max(int(num_m_blocks), 0) -+ requested_persist_m = max(int(requested_persist_m), 1) -+ required_persist_m = max( -+ 1, (num_m_blocks + _HIP_MAX_GRID_DIM_Y - 1) // _HIP_MAX_GRID_DIM_Y -+ ) -+ return max(requested_persist_m, required_persist_m) -+ -+ -+def requires_flydsl_stage2_reduce( -+ token_num: int, model_dim: int, element_size: int -+) -> bool: -+ """Return whether stage2 atomic output exceeds 32-bit byte offsets.""" -+ return int(token_num) * int(model_dim) * int(element_size) > 0xFFFFFFFF -+ -+ - def resolve_flydsl_stage2_tile_k(inter_dim: int, tile_k: int) -> int: - """Return a ``tile_k`` that divides ``inter_dim``, preferring the caller value. - -@@ -1403,7 +1425,7 @@ def flydsl_moe_stage1( - ) - _grid_y = min(_dense_blks, _all_blks) - -- _persist_m = persist_m if persist_m > 0 else 1 -+ _persist_m = resolve_flydsl_grid_y_persist_m(_grid_y, persist_m) - - # Allocate sorted-scale buffer with padding for tiled layout - scale_cols = inter_dim // 32 -@@ -1706,6 +1728,13 @@ def flydsl_moe_stage2( - # accumulate. Enabled by default; set AITER_FLYDSL_FORCE_REDUCE=0 to opt out. - if os.environ.get("AITER_FLYDSL_FORCE_REDUCE", "0") == "1": - mode = "reduce" -+ elif ( -+ mode != "reduce" -+ and not return_per_slot -+ and requires_flydsl_stage2_reduce(token_num, model_dim, 2) -+ ): -+ # Buffer atomics use 32-bit offsets; reduce outputs larger than 4 GiB. -+ mode = "reduce" - - accumulate = mode != "reduce" and not return_per_slot - -@@ -1764,7 +1793,8 @@ def flydsl_moe_stage2( - _persist_m = -1 if m_blocks > 256 else 1 - - if a_dtype == "fp8": -- _persist_m = 1 -+ # FP8 uses non-persistent scheduling, so cap grid.y via persist_m. -+ _persist_m = resolve_flydsl_grid_y_persist_m(m_blocks) - - if bias is not None and bias.dtype != torch.float32: - bias = bias.to(torch.float32) diff --git a/apply_aiter_4417.py b/apply_aiter_4417.py deleted file mode 100644 index b0fe7288a..000000000 --- a/apply_aiter_4417.py +++ /dev/null @@ -1,112 +0,0 @@ -#!/usr/bin/env python3 -"""Graft ROCm/aiter#4417 ("Fix large-token FlyDSL MoE launch and output limits") -onto the aiter shipped in the 2026-08-12 vLLM ROCm nightly. - -Why not `git apply` the upstream diff: the nightly's aiter predates the -`dict[str, dict]` typing modernization, so every hunk's context is off by the -`Dict[str, Dict]` spelling. The substance of #4417 is five small anchored edits, -so we apply those directly. Idempotent — re-running is a no-op. - -Without this patch, DSv4-Pro FP8/FP4 stage2 uses buffer atomics whose 32-bit -byte offsets overflow once token_num*model_dim*2 exceeds 4 GiB, which shows up -as `Memory access fault by GPU node-N ... Reason: Unknown` during the vLLM -profile run (determine_available_memory), killing the engine core. -""" - -import sys - -HELPERS = ''' -# HIP limits grid.y/grid.z to 65535. -_HIP_MAX_GRID_DIM_Y = 65535 - - -def resolve_flydsl_grid_y_persist_m( - num_m_blocks: int, requested_persist_m: int = 0 -) -> int: - """Increase persist_m as needed to keep grid.y within HIP's limit.""" - num_m_blocks = max(int(num_m_blocks), 0) - requested_persist_m = max(int(requested_persist_m), 1) - required_persist_m = max( - 1, (num_m_blocks + _HIP_MAX_GRID_DIM_Y - 1) // _HIP_MAX_GRID_DIM_Y - ) - return max(requested_persist_m, required_persist_m) - - -def requires_flydsl_stage2_reduce( - token_num: int, model_dim: int, element_size: int -) -> bool: - """Return whether stage2 atomic output exceeds 32-bit byte offsets.""" - return int(token_num) * int(model_dim) * int(element_size) > 0xFFFFFFFF - - -''' - -# (description, exact old text, new text) — each old text must be unique. -EDITS = [ - ( - "add _HIP_MAX_GRID_DIM_Y + the two guard helpers", - "def resolve_flydsl_stage2_tile_k(inter_dim: int, tile_k: int) -> int:", - HELPERS.lstrip("\n") - + "def resolve_flydsl_stage2_tile_k(inter_dim: int, tile_k: int) -> int:", - ), - ( - "stage1: cap grid.y via persist_m", - " _persist_m = persist_m if persist_m > 0 else 1", - " _persist_m = resolve_flydsl_grid_y_persist_m(_grid_y, persist_m)", - ), - ( - "stage2: switch to reduce when atomic output exceeds 4 GiB", - ' if os.environ.get("AITER_FLYDSL_FORCE_REDUCE", "0") == "1":\n' - " mode = \"reduce\"\n", - ' if os.environ.get("AITER_FLYDSL_FORCE_REDUCE", "0") == "1":\n' - ' mode = "reduce"\n' - " elif (\n" - ' mode != "reduce"\n' - " and not return_per_slot\n" - " and requires_flydsl_stage2_reduce(token_num, model_dim, 2)\n" - " ):\n" - " # Buffer atomics use 32-bit offsets; reduce outputs larger than 4 GiB.\n" - ' mode = "reduce"\n', - ), - ( - "stage2 fp8: cap grid.y via persist_m", - ' if a_dtype == "fp8":\n _persist_m = 1\n', - ' if a_dtype == "fp8":\n' - " # FP8 uses non-persistent scheduling, so cap grid.y via persist_m.\n" - " _persist_m = resolve_flydsl_grid_y_persist_m(m_blocks)\n", - ), -] - - -def main() -> int: - path = sys.argv[1] if len(sys.argv) > 1 else None - if path is None: - import importlib.util as u - - spec = u.find_spec("aiter.ops.flydsl.moe_kernels") - if spec is None or spec.origin is None: - print("FAIL: cannot locate aiter.ops.flydsl.moe_kernels", file=sys.stderr) - return 1 - path = spec.origin - - src = open(path).read() - - if "requires_flydsl_stage2_reduce" in src and "resolve_flydsl_grid_y_persist_m" in src: - print(f"SKIP: aiter#4417 already present in {path}") - return 0 - - for desc, old, new in EDITS: - n = src.count(old) - if n != 1: - print(f"FAIL: anchor for '{desc}' matched {n} times (want 1)", file=sys.stderr) - return 1 - src = src.replace(old, new, 1) - print(f" ok: {desc}") - - open(path, "w").write(src) - print(f"OK: grafted aiter#4417 into {path}") - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/dsv4_bench_8k1k_c32.sh b/dsv4_bench_8k1k_c32.sh deleted file mode 100644 index 9fe55c4ad..000000000 --- a/dsv4_bench_8k1k_c32.sh +++ /dev/null @@ -1,19 +0,0 @@ -#!/usr/bin/env bash -# 8k-in / 1k-out random benchmark at concurrency 32 against the patched DEP8 -# server. Standalone `vllm bench serve` (NOT the agentic aiperf replay) -- the -# quick throughput smoke the user asked for first. -set -uo pipefail - -BASE=http://localhost:8000 -SERVED=deepseek-ai/DeepSeek-V4-Pro -LOG=/home/jiacao/InferenceX/dsv4-bench-8k1k-c32.log - -exec vllm bench serve \ - --backend openai-chat --endpoint /v1/chat/completions \ - --base-url "$BASE" --model "$SERVED" \ - --dataset-name random \ - --random-input-len 8000 --random-output-len 1000 \ - --max-concurrency 32 --num-prompts 128 \ - --ignore-eos --percentile-metrics ttft,tpot,itl,e2el \ - --save-result --result-dir /home/jiacao/InferenceX \ - --result-filename dsv4-bench-8k1k-c32.json > "$LOG" 2>&1 diff --git a/dsv4_patch_additive.sh b/dsv4_patch_additive.sh deleted file mode 100644 index dec899ae8..000000000 --- a/dsv4_patch_additive.sh +++ /dev/null @@ -1,76 +0,0 @@ -#!/usr/bin/env bash -# ADDITIVE patch for the 08-12 nightly base (3ee2df30). The 08-12 base aiter is -# NEWER than v0.1.19.post2 and already exposes _set_current_hip_stream (which the -# nightly vllm calls for module_rmsnorm_quant) plus the tuned-gemm CSVs. The old -# wholesale post2-python overlay REGRESSED aiter/jit/core.py and dropped that -# symbol -> rmsnorm_quant warmup crash. So here we ONLY ADD the gluon sparse-MLA -# kernels that the base genuinely lacks, and apply the two vllm wiring PRs. We do -# NOT touch aiter core, and we do NOT add MegaMoE (its intranode kernel needs -# mori.ir.flydsl, absent from the base -> DEP8 is out of scope for this route). -set -uo pipefail -AITER_SHA="97d0c6e4cb7a0919c12291c7c7d560ad412f15c1" -AITER_REPO="https://github.com/ROCm/aiter" -VLLM_REPO="https://github.com/vllm-project/vllm" -ROOT="$(python -c 'import importlib.util as u, os; print(os.path.dirname(os.path.dirname(u.find_spec("vllm").origin)))')" -[ -d "$ROOT/vllm" ] && [ -d "$ROOT/aiter" ] || { echo "ERROR ROOT=$ROOT"; exit 1; } -echo "[add] ROOT=$ROOT" -WS=/tmp/dsv4_add; mkdir -p "$WS" - -# --- gluon kernels only (additive; base lacks _gluon_kernels + these variants) -- -GLUON_PATHS=( - aiter/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py - aiter/ops/triton/_gluon_kernels/gfx1250/attention/pa_decode_sparse.py - aiter/ops/triton/_triton_kernels/attention/pa_decode_sparse.py - aiter/ops/triton/attention/pa_decode_sparse.py -) -SRC="$WS/aiter_src" -if [ ! -d "$SRC/.git" ]; then git clone --filter=blob:none --no-checkout "$AITER_REPO" "$SRC" 2>&1 | tail -1; fi -( cd "$SRC" && git fetch --depth 1 origin "$AITER_SHA" 2>&1 | tail -1 && git checkout -q "$AITER_SHA" -- "${GLUON_PATHS[@]}" ) -for p in "${GLUON_PATHS[@]}"; do - [ -e "$SRC/$p" ] || { echo " MISSING in src: $p"; continue; } - mkdir -p "$(dirname "$ROOT/$p")"; cp -a "$SRC/$p" "$ROOT/$p"; echo " added $p" -done - -# --- aiter #4417: large-token FlyDSL MoE launch/output limits ------------------- -# The base's vendored aiter predates #4417 (merged 2026-07-30) even though the -# nightly itself is 08-12 -- the vendored revision is pinned, it does not track -# aiter main. #4417 guards two large-token limits: stage2 buffer atomics address -# the output with 32-bit byte offsets (>4 GiB walks off the end), and HIP caps -# grid.y at 65535. Neither fires at the DSv4-Pro TP8 shape we measure -# (requires_flydsl_stage2_reduce(65536, 7168, 2) is False -- ~939 MB), so this is -# NOT the fix for the inter_dim=384 profile-run memfault; it is carried because -# it is a real gap in the base that any larger-token sweep row would hit. -# The upstream .diff will NOT apply here (its context postdates aiter's typing -# modernization), so graft the hunks by anchor. -python "$(dirname "$0")/graft_aiter_4417.py" "$ROOT/aiter/ops/flydsl/moe_kernels.py" || \ - { echo " #4417: GRAFT FAILED"; exit 1; } - -# --- vllm wiring PRs (dormant unless the env/backend selects them) -------------- -apply_pr(){ local pr="$1" mf="$ROOT/$2" mk="$3" d="$WS/vllm_$1.diff" - if grep -qF "$mk" "$mf" 2>/dev/null; then echo " #$pr: already present"; return; fi - curl -ksSL -o "$d" "$VLLM_REPO/pull/$pr.diff" || { echo " #$pr: FETCH FAIL"; return 1; } - if ( cd "$ROOT" && git apply -p1 --3way "$d" ) 2>/dev/null || ( cd "$ROOT" && git apply -p1 "$d" ) 2>/dev/null - then echo " #$pr: APPLIED"; else echo " #$pr: FAILED"; return 1; fi -} -apply_pr 51714 "vllm/v1/attention/ops/rocm_aiter_mla_sparse.py" "_DSV4_SPARSE_GLUON" || true -apply_pr 51918 "vllm/config/kernel.py" "flydsl_mega_moe" || true - -# --- verify: py_compile + the ABI-sensitive imports (needs GPU) ------------------ -echo "chk gluon kernel = $([ -f "$ROOT/aiter/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py" ] && echo present || echo MISSING)" -echo "chk aiter #4417 guards = $(grep -c 'requires_flydsl_stage2_reduce\|resolve_flydsl_grid_y_persist_m' "$ROOT/aiter/ops/flydsl/moe_kernels.py" 2>/dev/null) (expect 5)" -echo "chk vllm gluon wiring = $(grep -c '_DSV4_SPARSE_GLUON' "$ROOT/vllm/v1/attention/ops/rocm_aiter_mla_sparse.py" 2>/dev/null)" -python -m py_compile "$ROOT/aiter/ops/triton/attention/pa_decode_sparse.py" \ - "$ROOT/vllm/v1/attention/ops/rocm_aiter_mla_sparse.py" && echo PY_COMPILE_OK || { echo PY_COMPILE_FAIL; exit 1; } -python - <<'PYEOF' -import importlib -for m in ["aiter.jit.core","aiter.ops.triton.attention.pa_decode_sparse", - "vllm.v1.attention.ops.rocm_aiter_mla_sparse","vllm._aiter_ops"]: - try: importlib.import_module(m); print("IMPORT_OK",m) - except Exception as e: print("IMPORT_ERR",m,type(e).__name__,(str(e).splitlines() or [''])[-1]) -# the exact symbol the wholesale overlay dropped: -try: - import aiter.jit.core as c - print("has _set_current_hip_stream in core:", hasattr(c,"_set_current_hip_stream")) -except Exception as e: print("core probe err",e) -PYEOF -echo "[add] DONE" diff --git a/dsv4_serve_dep8.sh b/dsv4_serve_dep8.sh deleted file mode 100644 index d8ed26a1f..000000000 --- a/dsv4_serve_dep8.sh +++ /dev/null @@ -1,33 +0,0 @@ -#!/usr/bin/env bash -# DEP8 winner-arm vLLM serve for the patch-route validation on the STOCK nightly -# base (3ee2df30) patched in-place by apply_dsv4_container_patches.sh. This is the -# config the campaign measured as the winner: DP8 + expert-parallel + MegaMoE -# (flydsl_mega_moe, patched) + gluon sparse (VLLM_ROCM_DSV4_SPARSE_GLUON=1, -# patched), MTP num_spec=3 synthetic accept 2.49, kv fp8, gmu 0.8. FSE and the -# 384-shard oracle are TP8-arm only, so they stay OFF here. -set -uo pipefail - -# The campaign served /models/DeepSeek-V4-Pro (deepseek_v4_fp8 quant: fp8 -# weight-block + fp4 experts). That is /it-shared/models/DeepSeek-V4-Pro here. -# NOT the amd--DeepSeek-V4-Pro-MXFP4 HF snapshot (Quark MXFP4 -- different -# merged-column packing, fails the load_merged_column_weight shape assert). -MODEL_PATH=/it-shared/models/DeepSeek-V4-Pro -SERVED=deepseek-ai/DeepSeek-V4-Pro -PORT=8000 -LOG=/home/jiacao/InferenceX/dsv4-serve.log - -export VLLM_ROCM_USE_AITER=1 -export VLLM_ROCM_USE_AITER_MOE=1 -export VLLM_ROCM_DSV4_SPARSE_GLUON=1 -export VLLM_ENGINE_READY_TIMEOUT_S=10800 - -exec vllm serve "$MODEL_PATH" --served-model-name "$SERVED" \ - --host 0.0.0.0 --port "$PORT" --trust-remote-code \ - --async-scheduling --distributed-executor-backend mp --kv-cache-dtype fp8 \ - --tensor-parallel-size 1 --data-parallel-size 8 --enable-expert-parallel \ - --gpu-memory-utilization 0.8 --moe-backend flydsl_mega_moe \ - --compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}' \ - --speculative-config '{"method":"mtp","num_speculative_tokens":3,"rejection_sample_method":"synthetic","synthetic_acceptance_length":2.49}' \ - --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --reasoning-parser deepseek_v4 \ - --enable-auto-tool-choice --enable-prefix-caching --no-disable-hybrid-kv-cache-manager \ - --max-num-seqs 64 > "$LOG" 2>&1 diff --git a/dsv4_serve_stock.sh b/dsv4_serve_stock.sh deleted file mode 100644 index a875b7744..000000000 --- a/dsv4_serve_stock.sh +++ /dev/null @@ -1,40 +0,0 @@ -#!/usr/bin/env bash -# CONTROL ARM: serve DSv4-Pro FP4 TP8 on the *pristine* 08-12 nightly, with no -# patches applied at all. Same model, same parallelism, same MTP/compile config -# as dsv4_serve_tp8.sh -- the only difference is the container has an untouched -# aiter/vllm tree. -# -# Purpose: the profile-run memfault has only ever been observed on the PATCHED -# container. Nothing established that the patch is innocent. In particular -# apply/dsv4_patch_additive.sh copies four gluon kernel files out of -# aiter@97d0c6e4 into the nightly's much older aiter tree, and those two aiter -# revisions differ semantically in 418 files -- exactly the setup that imports -# cleanly and then faults at runtime. The earlier VLLM_ROCM_DSV4_SPARSE_GLUON=0 -# A/B did NOT clear it: that env only avoids one call path, the foreign files -# were still on sys.path. -# -# If this arm serves, the fault is ours and the patch is the bug. -# If this arm faults too, the fault is in the stock base and the patch is clean. -set -uo pipefail - -MODEL_PATH=/it-shared/models/DeepSeek-V4-Pro -SERVED=deepseek-ai/DeepSeek-V4-Pro -PORT=8000 -LOG=/home/jiacao/InferenceX/dsv4-serve-stock.log - -export VLLM_ROCM_USE_AITER=1 -export VLLM_ROCM_USE_AITER_MOE=1 -# No gluon knob here: VLLM_ROCM_DSV4_SPARSE_GLUON is introduced by vllm PR #51714, -# which this container does not have. Stock takes its default sparse-MLA path. -export VLLM_ENGINE_READY_TIMEOUT_S=10800 - -exec vllm serve "$MODEL_PATH" --served-model-name "$SERVED" \ - --host 0.0.0.0 --port "$PORT" --trust-remote-code \ - --async-scheduling --distributed-executor-backend mp --kv-cache-dtype fp8 \ - --tensor-parallel-size 8 --data-parallel-size 1 \ - --gpu-memory-utilization 0.8 --moe-backend aiter \ - --compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}' \ - --speculative-config '{"method":"mtp","num_speculative_tokens":3,"rejection_sample_method":"synthetic","synthetic_acceptance_length":2.49}' \ - --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --reasoning-parser deepseek_v4 \ - --enable-auto-tool-choice --enable-prefix-caching --no-disable-hybrid-kv-cache-manager \ - --max-num-seqs 64 > "$LOG" 2>&1 diff --git a/dsv4_serve_tp8.sh b/dsv4_serve_tp8.sh deleted file mode 100644 index 6e5d5b585..000000000 --- a/dsv4_serve_tp8.sh +++ /dev/null @@ -1,76 +0,0 @@ -#!/usr/bin/env bash -# TP8 arm vLLM serve on the 08-12 nightly + as many campaign PRs as can be -# carried safely. Goal here is RUNNABLE FIRST, maximum PR coverage second. -# -# Carried: aiter #4417 (in-file guards), vllm #51714 (gluon sparse wiring), -# vllm #51918 (flydsl_mega_moe kernel config). -# Also out: MegaMoE/DEP8 (needs mori.ir.flydsl, absent from the base) and FSE -# (#4269 needs aiter fhmoe.py, also absent). -# -# CORRECTION -- do not repeat this mistake. An earlier revision of this header -# claimed the container carried "four aiter@97d0c6e4 gluon kernel FILES" that -# had to be deleted as a cross-version transplant. That was wrong: -# aiter/ops/triton/attention/pa_decode_sparse.py SHIPS IN THE STOCK NIGHTLY. -# Diffing the attention dir between a pristine `dsv4stock` container and the -# working one shows dsv4v is MISSING that base file and has nothing extra -- -# the deletion damaged the base rather than removing foreign code. Any container -# built from this script must keep the base file; VLLM_ROCM_DSV4_SPARSE_GLUON=0 -# below is enough to keep the gluon path dormant, since #51714's call site is a -# lazy in-function import (rocm_aiter_mla_sparse.py:2211) behind that env. -set -uo pipefail - -MODEL_PATH=/it-shared/models/DeepSeek-V4-Pro -SERVED=deepseek-ai/DeepSeek-V4-Pro -PORT=8000 -LOG=/home/jiacao/InferenceX/dsv4-serve.log - -export VLLM_ROCM_USE_AITER=1 -export VLLM_ROCM_USE_AITER_MOE=1 -# Default off: #51714's gluon sparse-MLA path is not part of the runnable-first -# baseline. The kernel it dispatches to (pa_decode_sparse.py) is present in the -# stock nightly, so this can be flipped to 1 to A/B the gluon arm once the -# baseline serves. -export VLLM_ROCM_DSV4_SPARSE_GLUON=${VLLM_ROCM_DSV4_SPARSE_GLUON:-0} -# Probe knobs for the profile-run memfault, NOT a fix. Both default to aiter's -# own values so a plain run reproduces the campaign config. -# -# The MoE at inter_dim=384 is NOT the fault, despite being the loudest thing in -# the log. Two dead theories, recorded so nobody re-derives them: -# -# 1. "the tuned row names an illegal opus stage2 tile". Wrong. opus's 256 is a -# *logical fp4* K step; packed it is K_STEP_PACKED=128, the kernel validates -# effective_inter_dim % 128, and 384 % 128 == 0. 384 is a first-class opus -# codegen seed (OPUS_A8W4_CODEGEN_SEED_EFFECTIVE_INTER_DIMS). FlyDSL's -# tile_n/tile_k must divide inter_dim; opus's step need only divide by 128. -# The two families collide only in naming. -# 2. "stage1 silently drops to tile_n=128 and opus stage2 still reads a 256 -# layout". Also wrong: stage1's output buffer is (token, topk, inter_dim), -# independent of tile_n, and .repro/moe384.py drives this exact shape -# standalone -- preshuffled weights, same kernel pair, same tile_n downgrade -# warning -- and returns finite results. The kernel is fine. -# -# What the log actually shows: TP6 finishes MoE, loads bf16_tuned_gemm.csv, and -# faults there; the other seven workers are parked on that CSV's baton lock and -# survive. Immediately after come M:65536/M:16384 x N:7168 x K:7168 bf16 GEMMs -# with no tuned entry, running default configs. The fault is in the post-MoE -# bf16 GEMM, not the MoE. -export AITER_BYPASS_TUNE_CONFIG=${AITER_BYPASS_TUNE_CONFIG:-0} -export AITER_FLYDSL_FORCE=${AITER_FLYDSL_FORCE:-1} -export VLLM_ENGINE_READY_TIMEOUT_S=10800 - -# Overridable so the profile-run peak can be walked down without editing this -# file. The profile run builds a dummy batch of max_num_batched_tokens (16384) -# x the MTP fan-out (num_speculative_tokens=3 -> 4), hence the M:65536 GEMMs in -# the log; that transient is the largest allocation the server ever makes. -GPU_MEM_UTIL=${GPU_MEM_UTIL:-0.8} - -exec vllm serve "$MODEL_PATH" --served-model-name "$SERVED" \ - --host 0.0.0.0 --port "$PORT" --trust-remote-code \ - --async-scheduling --distributed-executor-backend mp --kv-cache-dtype fp8 \ - --tensor-parallel-size 8 --data-parallel-size 1 \ - --gpu-memory-utilization "$GPU_MEM_UTIL" --moe-backend aiter \ - --compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}' \ - --speculative-config '{"method":"mtp","num_speculative_tokens":3,"rejection_sample_method":"synthetic","synthetic_acceptance_length":2.49}' \ - --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --reasoning-parser deepseek_v4 \ - --enable-auto-tool-choice --enable-prefix-caching --no-disable-hybrid-kv-cache-manager \ - --max-num-seqs 64 > "$LOG" 2>&1 diff --git a/dsv4_serve_tp8_forcereduce.sh b/dsv4_serve_tp8_forcereduce.sh deleted file mode 100755 index 5b08e4df2..000000000 --- a/dsv4_serve_tp8_forcereduce.sh +++ /dev/null @@ -1,6 +0,0 @@ -#!/bin/bash -# TP8 arm with AITER_FLYDSL_FORCE_REDUCE=1 — diagnostic for the missing aiter #4417 -# guards in the 08-12 nightly (stage2 buffer atomics overflow 32-bit offsets). -set -euo pipefail -export AITER_FLYDSL_FORCE_REDUCE=1 -exec bash "$(dirname "$0")/dsv4_serve_tp8.sh" diff --git a/dsv4_serve_tp8_notiter.sh b/dsv4_serve_tp8_notiter.sh deleted file mode 100755 index 7782c095a..000000000 --- a/dsv4_serve_tp8_notiter.sh +++ /dev/null @@ -1,35 +0,0 @@ -#!/usr/bin/env bash -# TP8 arm vLLM serve -- the config the patch route CAN fully reproduce on a stock -# nightly (moe-backend aiter, so no MegaMoE/flydsl -> no mori.ir.flydsl dep), -# gluon sparse on (VLLM_ROCM_DSV4_SPARSE_GLUON=1, patched), base #4269 FSE on -# (VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1), 384-shard native in the 08-12 -# base. Used to validate the patched container actually serves DSv4-Pro FP4 at -# 8k/1k c32. (DEP8/MegaMoE is blocked by a mori version gap -- see notes.) -set -uo pipefail - -MODEL_PATH=/it-shared/models/DeepSeek-V4-Pro -SERVED=deepseek-ai/DeepSeek-V4-Pro -PORT=8000 -LOG=/home/jiacao/InferenceX/dsv4-serve-triton.log - -export VLLM_ROCM_USE_AITER=1 -export VLLM_ROCM_USE_AITER_MOE=0 -# FSE (#4269) intentionally OFF: the 08-12 base aiter has no fhmoe.py, and the -# additive route does not restore it. -# gluon: the additively-injected gluon sparse-MLA kernel (from aiter@97d0c6e4) -# GPU-faults during memory profiling on the 08-12 base aiter runtime -# (hc_head_fuse_tilelang path, VllmWorker GPU coredump). Overridable so we can -# validate a servable baseline with gluon OFF, then flip it back on to reproduce. -export VLLM_ROCM_DSV4_SPARSE_GLUON=${VLLM_ROCM_DSV4_SPARSE_GLUON:-1} -export VLLM_ENGINE_READY_TIMEOUT_S=10800 - -exec vllm serve "$MODEL_PATH" --served-model-name "$SERVED" \ - --host 0.0.0.0 --port "$PORT" --trust-remote-code \ - --async-scheduling --distributed-executor-backend mp --kv-cache-dtype fp8 \ - --tensor-parallel-size 8 --data-parallel-size 1 \ - --gpu-memory-utilization 0.8 --moe-backend triton \ - --compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}' \ - --speculative-config '{"method":"mtp","num_speculative_tokens":3,"rejection_sample_method":"synthetic","synthetic_acceptance_length":2.49}' \ - --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --reasoning-parser deepseek_v4 \ - --enable-auto-tool-choice --enable-prefix-caching --no-disable-hybrid-kv-cache-manager \ - --max-num-seqs 64 > "$LOG" 2>&1 diff --git a/dsv4_serve_tp8_triton.sh b/dsv4_serve_tp8_triton.sh deleted file mode 100755 index 05b4aa8ea..000000000 --- a/dsv4_serve_tp8_triton.sh +++ /dev/null @@ -1,35 +0,0 @@ -#!/usr/bin/env bash -# TP8 arm vLLM serve -- the config the patch route CAN fully reproduce on a stock -# nightly (moe-backend aiter, so no MegaMoE/flydsl -> no mori.ir.flydsl dep), -# gluon sparse on (VLLM_ROCM_DSV4_SPARSE_GLUON=1, patched), base #4269 FSE on -# (VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1), 384-shard native in the 08-12 -# base. Used to validate the patched container actually serves DSv4-Pro FP4 at -# 8k/1k c32. (DEP8/MegaMoE is blocked by a mori version gap -- see notes.) -set -uo pipefail - -MODEL_PATH=/it-shared/models/DeepSeek-V4-Pro -SERVED=deepseek-ai/DeepSeek-V4-Pro -PORT=8000 -LOG=/home/jiacao/InferenceX/dsv4-serve.log - -export VLLM_ROCM_USE_AITER=1 -export VLLM_ROCM_USE_AITER_MOE=1 -# FSE (#4269) intentionally OFF: the 08-12 base aiter has no fhmoe.py, and the -# additive route does not restore it. -# gluon: the additively-injected gluon sparse-MLA kernel (from aiter@97d0c6e4) -# GPU-faults during memory profiling on the 08-12 base aiter runtime -# (hc_head_fuse_tilelang path, VllmWorker GPU coredump). Overridable so we can -# validate a servable baseline with gluon OFF, then flip it back on to reproduce. -export VLLM_ROCM_DSV4_SPARSE_GLUON=${VLLM_ROCM_DSV4_SPARSE_GLUON:-1} -export VLLM_ENGINE_READY_TIMEOUT_S=10800 - -exec vllm serve "$MODEL_PATH" --served-model-name "$SERVED" \ - --host 0.0.0.0 --port "$PORT" --trust-remote-code \ - --async-scheduling --distributed-executor-backend mp --kv-cache-dtype fp8 \ - --tensor-parallel-size 8 --data-parallel-size 1 \ - --gpu-memory-utilization 0.8 --moe-backend triton \ - --compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}' \ - --speculative-config '{"method":"mtp","num_speculative_tokens":3,"rejection_sample_method":"synthetic","synthetic_acceptance_length":2.49}' \ - --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --reasoning-parser deepseek_v4 \ - --enable-auto-tool-choice --enable-prefix-caching --no-disable-hybrid-kv-cache-manager \ - --max-num-seqs 64 > "$LOG" 2>&1 diff --git a/dsv4_wait_then_bench.sh b/dsv4_wait_then_bench.sh deleted file mode 100644 index e36b86838..000000000 --- a/dsv4_wait_then_bench.sh +++ /dev/null @@ -1,34 +0,0 @@ -#!/usr/bin/env bash -# Poll the DEP8 server /health until ready (cold 805 GiB NFS load ~2h + compile + -# cudagraph capture), then fire the 8k/1k c32 benchmark. Runs detached in the -# container; watch /home/jiacao/InferenceX/dsv4-waitbench.log. -set -uo pipefail -STAMP(){ date -u +%FT%TZ; } -WLOG=/home/jiacao/InferenceX/dsv4-waitbench.log -SLOG=/home/jiacao/InferenceX/dsv4-serve.log - -echo "$(STAMP) waiting for http://localhost:8000/health ..." >> "$WLOG" -for i in $(seq 1 720); do # 720 * 20s = 4h ceiling - code=$(curl -s -o /dev/null -w '%{http_code}' http://localhost:8000/health 2>/dev/null || echo 000) - if [ "$code" = "200" ]; then - echo "$(STAMP) server READY (health 200) after $((i*20))s" >> "$WLOG" - break - fi - # bail early if the server process died - if ! pgrep -f "vllm serve" >/dev/null 2>&1; then - echo "$(STAMP) FATAL: vllm serve process gone; tail serve log:" >> "$WLOG" - tail -40 "$SLOG" >> "$WLOG" 2>&1 - exit 1 - fi - sleep 20 -done - -code=$(curl -s -o /dev/null -w '%{http_code}' http://localhost:8000/health 2>/dev/null || echo 000) -if [ "$code" != "200" ]; then - echo "$(STAMP) FATAL: health never reached 200 (last=$code)" >> "$WLOG" - exit 1 -fi - -echo "$(STAMP) launching 8k/1k c32 benchmark ..." >> "$WLOG" -bash /home/jiacao/InferenceX/.claude/worktrees/dsv4-ci-sweep-pr/dsv4_bench_8k1k_c32.sh -echo "$(STAMP) benchmark finished rc=$? ; results in dsv4-bench-8k1k-c32.json / .log" >> "$WLOG" diff --git a/graft_aiter_4417.py b/graft_aiter_4417.py deleted file mode 100644 index f347870ad..000000000 --- a/graft_aiter_4417.py +++ /dev/null @@ -1,125 +0,0 @@ -#!/usr/bin/env python3 -"""Graft ROCm/aiter #4417 into an aiter that predates it. - -#4417 "Fix large-token FlyDSL MoE launch and output limits" (merged 2026-07-30, -single file, +32/-2) adds two guards to aiter/ops/flydsl/moe_kernels.py: - - * requires_flydsl_stage2_reduce() -- stage2 buffer atomics address the output - with 32-bit byte offsets, so a >4 GiB output silently walks off the end. - The guard flips mode to "reduce" before that happens. - * resolve_flydsl_grid_y_persist_m() -- HIP caps grid.y at 65535; the guard - raises persist_m enough to keep the launch legal. - -Neither guard fires at the DSv4-Pro TP8 shape this campaign measures -- -requires_flydsl_stage2_reduce(65536, 7168, 2) is False (~939 MB, well under the -4 GiB threshold) -- so #4417 does NOT explain the inter_dim=384 profile-run -memfault. That fault is a stage2 tile_k mismatch: the tuned CSV names -opus_moe2_*_t64x256x256_* for inter_dim=384, and resolve_flydsl_stage2_tile_k -only guards flydsl_* kernel names, so tile_k=256 runs against K=384. #4417 is -grafted anyway because it is a genuine gap in the base that any larger-token -sweep row would hit. - -The upstream .diff does NOT apply to the 08-12 nightly base: that aiter predates -both #4417 and aiter's typing modernization, so every context line still reads -`Dict[str, Dict]` / `Optional[X]` where the diff expects `dict[str, dict]` / -`X | None`. The three hunks are grafted here by anchor instead of by context. -Idempotent: re-running on an already-grafted (or already-new) file is a no-op. -""" - -import sys - -HELPERS = ''' - -def resolve_flydsl_grid_y_persist_m( - num_m_blocks: int, requested_persist_m: int = 0 -) -> int: - """Increase persist_m as needed to keep grid.y within HIP's limit.""" - num_m_blocks = max(int(num_m_blocks), 0) - requested_persist_m = max(int(requested_persist_m), 1) - required_persist_m = max( - 1, (num_m_blocks + _HIP_MAX_GRID_DIM_Y - 1) // _HIP_MAX_GRID_DIM_Y - ) - return max(requested_persist_m, required_persist_m) - - -def requires_flydsl_stage2_reduce( - token_num: int, model_dim: int, element_size: int -) -> bool: - """Return whether stage2 atomic output exceeds 32-bit byte offsets.""" - return int(token_num) * int(model_dim) * int(element_size) > 0xFFFFFFFF - -''' - - -def graft(path): - src = open(path).read() - if "requires_flydsl_stage2_reduce" in src and "_HIP_MAX_GRID_DIM_Y" in src: - print("SKIP already grafted/present:", path) - return 0 - orig = src - done = [] - - # hunk 1a: the grid.y constant, right after the _KERNEL_PARAMS declaration. - a = "_KERNEL_PARAMS: Dict[str, Dict] = {}\n" - if a not in src: - a = "_KERNEL_PARAMS: dict[str, dict] = {}\n" - if src.count(a) != 1: - print("FAIL anchor _KERNEL_PARAMS count=", src.count(a)) - return 1 - src = src.replace(a, a + "\n# HIP limits grid.y/grid.z to 65535.\n" - "_HIP_MAX_GRID_DIM_Y = 65535\n", 1) - done.append("const") - - # hunk 1b: the two helpers, ahead of resolve_flydsl_stage2_tile_k. - b = "\ndef resolve_flydsl_stage2_tile_k(" - if src.count(b) != 1: - print("FAIL anchor stage2_tile_k count=", src.count(b)) - return 1 - src = src.replace(b, HELPERS + "\ndef resolve_flydsl_stage2_tile_k(", 1) - done.append("helpers") - - # hunk 2: stage1 caps grid.y through persist_m. - c = " _persist_m = persist_m if persist_m > 0 else 1\n" - if src.count(c) != 1: - print("FAIL anchor stage1 persist_m count=", src.count(c)) - return 1 - src = src.replace(c, " _persist_m = resolve_flydsl_grid_y_persist_m(_grid_y, persist_m)\n", 1) - done.append("stage1_persist") - - # hunk 3a: stage2 falls back to reduce when the atomic output exceeds 4 GiB. - d = (' if os.environ.get("AITER_FLYDSL_FORCE_REDUCE", "0") == "1":\n' - ' mode = "reduce"\n') - if src.count(d) != 1: - print("FAIL anchor FORCE_REDUCE count=", src.count(d)) - return 1 - src = src.replace(d, d + ( - " elif (\n" - " mode != \"reduce\"\n" - " and not return_per_slot\n" - " and requires_flydsl_stage2_reduce(token_num, model_dim, 2)\n" - " ):\n" - " # Buffer atomics use 32-bit offsets; reduce outputs larger than 4 GiB.\n" - " mode = \"reduce\"\n"), 1) - done.append("stage2_reduce") - - # hunk 3b: fp8 stage2 is non-persistent, so cap grid.y the same way. - e = ' if a_dtype == "fp8":\n _persist_m = 1\n' - if src.count(e) != 1: - print("FAIL anchor fp8 persist_m count=", src.count(e)) - return 1 - src = src.replace(e, ' if a_dtype == "fp8":\n' - ' # FP8 uses non-persistent scheduling, so cap grid.y via persist_m.\n' - ' _persist_m = resolve_flydsl_grid_y_persist_m(m_blocks)\n', 1) - done.append("stage2_fp8_persist") - - if src == orig: - print("FAIL no change") - return 1 - open(path + ".pre4417", "w").write(orig) - open(path, "w").write(src) - print("GRAFTED", path, "hunks:", ",".join(done)) - return 0 - - -if __name__ == "__main__": - sys.exit(graft(sys.argv[1])) diff --git a/sbatch_3way_patchset.sh b/sbatch_3way_patchset.sh deleted file mode 100644 index cbae44722..000000000 --- a/sbatch_3way_patchset.sh +++ /dev/null @@ -1,256 +0,0 @@ -#!/usr/bin/env bash -#SBATCH --job-name=3way-patch -#SBATCH --account=amd-aifw-aim -#SBATCH --qos=amd-aifw-aim-qos -#SBATCH --partition=amd-spur -#SBATCH --nodes=1 -#SBATCH --gpus-per-node=0 -#SBATCH --time=02:00:00 -#SBATCH --output=/home/jiacao/InferenceX/3way-%j.out -#SBATCH --exclude=crsuse2-m2m-203,crsuse2-m2m-074,crsuse2-m2m-240,crsuse2-m2m-029,crsuse2-m2m-061,crsuse2-m2m-071 -# -# Extract the *vendor patch set* out of the DSv4 measurement image and port it -# onto the intended CI baseline. -# -# A straight VENDOR-vs-TARGET diff (job 6316) is useless for this: it returned -# 769 vllm files / 4.8 MB, because TARGET is 454 upstream commits AHEAD of the -# vendor image's base, so the diff is dominated by upstream drift running the -# wrong direction. The vendor's own changes are buried in it. -# -# Three-way subtraction fixes that. REF is a public ROCm nightly built from -# 6f91edf9 (2026-07-29), only 11 commits behind the vendor image's own base -# 02e63f2e4 (2026-07-30). So: -# -# REF ---- 11 commits ----> vendor base 02e63f2e4 --(vendor patches)--> VENDOR -# REF ---- 465 commits ---> TARGET b22afe45 -# -# diff(REF, VENDOR) = 11 commits of drift + THE VENDOR PATCH SET <-- small -# diff(REF, TARGET) = 465 commits of upstream <-- large -# -# The first is the deliverable. Anything it touches that the second also -# touches is a conflict that needs a human decision; anything it touches that -# the second does not is a clean port onto TARGET. -# -# Deliverables under $OUT: -# vendor_patchset.{vllm,aiter}.diff - REF -> VENDOR, the thing to port -# upstream_drift.vllm.diff - REF -> TARGET, for conflict detection -# CONFLICTS.txt - files both sides touch -# CLEAN.txt - files only the vendor touches -# applied/ - TARGET tree with the patch set applied -# APPLY_LOG.txt - per-file apply result -set -uo pipefail - -REF="${REF:-vllm/vllm-openai-rocm:nightly-6f91edf96d3f3272945809c04702380053bff4de}" -VENDOR="${VENDOR:-jiahcao/vllm-dsv4:dsv4-pr51473-20260812}" -TARGET="${TARGET:-vllm/vllm-openai-rocm:nightly-b22afe45ac797ae58e67a7a3ad79ee5714024420}" -OUT="${OUT:-/home/jiacao/3way-$(date -u +%Y%m%d-%H%M)}" - -hostname; date -u -echo "REF : $REF (6f91edf9, 2026-07-29)" -echo "VENDOR : $VENDOR (base 02e63f2e4, 2026-07-30)" -echo "TARGET : $TARGET (b22afe45, 2026-08-10)" -echo "OUT : $OUT" -echo - -fail() { echo "FATAL: $*" >&2; exit 1; } -docker info >/dev/null 2>&1 || fail "no docker access on $(hostname)" -rm -rf "$OUT"; mkdir -p "$OUT" || fail "cannot create $OUT" - -# ---------------------------------------------------------------- pull -for img in "$REF" "$VENDOR" "$TARGET"; do - for _try in 1 2 3; do - docker image inspect "$img" >/dev/null 2>&1 && break - echo "pulling $img (attempt $_try/3) ... $(date -u +%H:%M:%S)Z" - docker pull "$img" || true - done - docker image inspect "$img" >/dev/null 2>&1 || fail "pull failed: $img" -done -echo "PULL OK $(date -u +%H:%M:%S)Z" - -# ---------------------------------------------------------------- locate -# Layouts differ between images: the nightlies install into dist-packages, the -# vendor image carries an editable /src/vllm checkout. Ask each image where it -# imports from rather than assuming (job 6309 extracted 0 bytes by assuming). -locate_in() { # $1=image $2=pkg - docker run --rm --entrypoint /bin/bash "$1" -c \ - "python3 -c 'import importlib.util as u,os -s=u.find_spec(\"$2\") -print(os.path.dirname(s.origin) if s and s.origin else \"\")' 2>/dev/null" 2>/dev/null | tr -d '\r' | tail -1 -} - -declare -A P -for side in ref vendor target; do - case $side in ref) img="$REF";; vendor) img="$VENDOR";; target) img="$TARGET";; esac - for pkg in vllm aiter; do - P[$side:$pkg]=$(locate_in "$img" "$pkg") - echo " $side/$pkg -> ${P[$side:$pkg]:-}" - done -done -echo - -# ---------------------------------------------------------------- extract -extract() { # $1=image $2=destdir $3..=in-container dirs - local img="$1" dst="$2"; shift 2 - local cid pats=() - mkdir -p "$dst" - for p in "$@"; do [ -n "$p" ] && pats+=("${p#/}/*"); done - [ ${#pats[@]} -eq 0 ] && return 1 - cid=$(docker create "$img") || fail "docker create failed: $img" - docker export "$cid" | tar -x -C "$dst" --wildcards "${pats[@]}" 2>/dev/null - docker rm -f "$cid" >/dev/null 2>&1 -} - -for side in ref vendor target; do - case $side in ref) img="$REF";; vendor) img="$VENDOR";; target) img="$TARGET";; esac - echo "extracting $side ... $(date -u +%H:%M:%S)Z" - extract "$img" "$OUT/$side" "${P[$side:vllm]}" "${P[$side:aiter]}" - echo " $(du -sh "$OUT/$side" 2>/dev/null | cut -f1)" -done - -root() { # $1=side $2=pkg -> on-disk package root - local d="$OUT/$1/${P[$1:$2]#/}" - [ -d "$d" ] && { echo "$d"; return; } - find "$OUT/$1" -type f -path "*/$2/__init__.py" 2>/dev/null \ - | awk '{print length"\t"$0}' | sort -n | head -1 | cut -f2- | xargs -r dirname -} - -# ---------------------------------------------------------------- diff -# Normalise away the layout difference: diff the package dirs directly and -# rewrite the labels to bare a/ and b/ so the resulting patch applies -# with -p1 regardless of where each image kept the package. -DIFFOPTS=(-ruN -x '__pycache__' -x '*.pyc' -x '*.so' -x '*.pyd' -x '_version.py') - -echo -for pkg in vllm aiter; do - r=$(root ref "$pkg"); v=$(root vendor "$pkg"); t=$(root target "$pkg") - echo "=============== $pkg ===============" - echo " ref : ${r:-}" - echo " vendor: ${v:-}" - echo " target: ${t:-}" - if [ -z "$r" ] || [ -z "$v" ] || [ -z "$t" ]; then - echo " SKIP: missing on one side"; continue - fi - - # (1) THE DELIVERABLE: ref -> vendor. 11 commits of drift + vendor patches. - ( cd "$(dirname "$r")" && diff "${DIFFOPTS[@]}" \ - --label "a/$pkg" --label "b/$pkg" "$pkg" "$v" ) \ - > "$OUT/vendor_patchset.$pkg.diff" 2>/dev/null - # (2) ref -> target: 465 upstream commits, used only to find conflicts. - ( cd "$(dirname "$r")" && diff "${DIFFOPTS[@]}" \ - --label "a/$pkg" --label "b/$pkg" "$pkg" "$t" ) \ - > "$OUT/upstream_drift.$pkg.diff" 2>/dev/null - - vs=$(stat -c%s "$OUT/vendor_patchset.$pkg.diff" 2>/dev/null || echo 0) - us=$(stat -c%s "$OUT/upstream_drift.$pkg.diff" 2>/dev/null || echo 0) - vf=$(grep -c '^diff ' "$OUT/vendor_patchset.$pkg.diff" 2>/dev/null || echo 0) - uf=$(grep -c '^diff ' "$OUT/upstream_drift.$pkg.diff" 2>/dev/null || echo 0) - echo " vendor patch set : $vf files, $vs bytes <-- to port" - echo " upstream drift : $uf files, $us bytes" - - # Real line counts. (Job 6316's per-file awk under-reported removals; total - # +/- over the whole file is the number to trust.) - printf " vendor patch set : +%s / -%s lines\n" \ - "$(grep -c '^+[^+]' "$OUT/vendor_patchset.$pkg.diff" 2>/dev/null || echo 0)" \ - "$(grep -c '^-[^-]' "$OUT/vendor_patchset.$pkg.diff" 2>/dev/null || echo 0)" - - # ---- conflict analysis ------------------------------------------------- - grep '^diff ' "$OUT/vendor_patchset.$pkg.diff" 2>/dev/null \ - | awk '{print $NF}' | sed "s|^$v/||;s|^$pkg/||" | sort -u > "$OUT/.v.$pkg" - grep '^diff ' "$OUT/upstream_drift.$pkg.diff" 2>/dev/null \ - | awk '{print $NF}' | sed "s|^$t/||;s|^$pkg/||" | sort -u > "$OUT/.u.$pkg" - comm -12 "$OUT/.v.$pkg" "$OUT/.u.$pkg" | sed "s|^|$pkg/|" >> "$OUT/CONFLICTS.txt" - comm -23 "$OUT/.v.$pkg" "$OUT/.u.$pkg" | sed "s|^|$pkg/|" >> "$OUT/CLEAN.txt" - echo " files vendor-only (clean port): $(comm -23 "$OUT/.v.$pkg" "$OUT/.u.$pkg" | wc -l)" - echo " files touched by BOTH (conflict): $(comm -12 "$OUT/.v.$pkg" "$OUT/.u.$pkg" | wc -l)" -done - -# ---------------------------------------------------------------- apply -# Apply the vendor patch set onto a copy of TARGET, file by file, so a failure -# names the file instead of aborting the whole port. git apply is tried first -# (exact), then patch --fuzz=3 (tolerant of the 11-commit drift), and finally -# a whole-file copy for files that exist only in the vendor image -- those are -# additions, not modifications, and are the bulk of the MegaMoE kernel tree. -echo -echo "=============== APPLY onto TARGET ===============" -: > "$OUT/APPLY_LOG.txt" -for pkg in vllm aiter; do - v=$(root vendor "$pkg"); t=$(root target "$pkg") - [ -z "$v" ] || [ -z "$t" ] && continue - dst="$OUT/applied/$pkg" - mkdir -p "$(dirname "$dst")" - cp -a "$t" "$dst" || fail "cannot stage TARGET copy for $pkg" - - ok=0; fuzz=0; added=0; bad=0 - # Walk the vendor tree: every .py that differs from the staged target. - while IFS= read -r f; do - rel="${f#$v/}" - if [ ! -e "$dst/$rel" ]; then - mkdir -p "$(dirname "$dst/$rel")" - cp -a "$f" "$dst/$rel" && { added=$((added+1)); \ - echo "ADD $pkg/$rel" >> "$OUT/APPLY_LOG.txt"; } \ - || { bad=$((bad+1)); echo "FAIL-ADD $pkg/$rel" >> "$OUT/APPLY_LOG.txt"; } - continue - fi - cmp -s "$f" "$dst/$rel" && continue - # Modified on both sides: produce a patch from ref->vendor for just this - # file and try to apply it to the target copy. - r=$(root ref "$pkg") - if [ -e "$r/$rel" ]; then - diff -u --label "a/$rel" --label "b/$rel" "$r/$rel" "$f" > "$OUT/.one.diff" 2>/dev/null - if patch -s -p1 -d "$dst" --fuzz=3 --forward --no-backup-if-mismatch \ - -i "$OUT/.one.diff" >/dev/null 2>&1; then - fuzz=$((fuzz+1)); echo "PATCH $pkg/$rel" >> "$OUT/APPLY_LOG.txt" - else - bad=$((bad+1)); echo "CONFLICT $pkg/$rel" >> "$OUT/APPLY_LOG.txt" - fi - else - # No ref version: vendor rewrote a file that upstream also changed. - bad=$((bad+1)); echo "CONFLICT-NOREF $pkg/$rel" >> "$OUT/APPLY_LOG.txt" - fi - done < <(find "$v" -type f -name '*.py' ! -path '*__pycache__*' 2>/dev/null) - - echo " $pkg: added=$added patched=$fuzz conflicts=$bad" -done -rm -f "$OUT/.one.diff" "$OUT"/.v.* "$OUT"/.u.* - -echo -echo "--- conflicts needing a decision ---" -grep -E '^(CONFLICT|FAIL)' "$OUT/APPLY_LOG.txt" 2>/dev/null | head -60 -echo " total: $(grep -cE '^(CONFLICT|FAIL)' "$OUT/APPLY_LOG.txt" 2>/dev/null || echo 0)" - -# ---------------------------------------------------------------- verify -echo -echo "=============== feature surface after port ===============" -probe() { # $1=label $2=pattern - printf "%-45s" "$1:" - for side in "$OUT/target" "$OUT/applied"; do - n=$(grep -rl -- "$2" "$side" --include='*.py' 2>/dev/null | wc -l) - printf " %-4s" "$n" - done - echo " (target -> applied)" -} -probe "VLLM_ROCM_DSV4_SPARSE_GLUON" "VLLM_ROCM_DSV4_SPARSE_GLUON" -probe "flydsl_mega_moe" "flydsl_mega_moe" -probe "mega_moe (any)" "mega_moe" -probe "#51473 marker" "AITER_MXFP4_BF16 and activation == MoEActivation.SILU" - -echo -echo "=============== python syntax check on applied tree ===============" -# A port that lands syntactically-broken files would only surface 20 minutes -# into a measured cell. Parse every file the port touched. -bad=0 -while IFS= read -r line; do - rel=$(echo "$line" | awk '{print $2}') - f="$OUT/applied/$rel" - [ -f "$f" ] || continue - python3 -c "import ast,sys; ast.parse(open(sys.argv[1],encoding='utf-8').read())" "$f" 2>/dev/null \ - || { echo " SYNTAX FAIL: $rel"; bad=$((bad+1)); } -done < <(grep -E '^(ADD|PATCH)' "$OUT/APPLY_LOG.txt" 2>/dev/null) -echo " syntax failures: $bad" - -echo -echo "DONE $(date -u +%H:%M:%S)Z" -echo "Artifacts: $OUT" -echo " vendor_patchset.{vllm,aiter}.diff <- the port" -echo " CONFLICTS.txt / CLEAN.txt / APPLY_LOG.txt" -echo " applied/ <- TARGET + vendor patch set" diff --git a/sbatch_diff_images.sh b/sbatch_diff_images.sh deleted file mode 100755 index 5222cb24d..000000000 --- a/sbatch_diff_images.sh +++ /dev/null @@ -1,183 +0,0 @@ -#!/usr/bin/env bash -#SBATCH --job-name=img-diff -#SBATCH --account=amd-aifw-aim -#SBATCH --qos=amd-aifw-aim-qos -#SBATCH --partition=amd-spur -#SBATCH --nodes=1 -#SBATCH --gpus-per-node=0 -#SBATCH --time=01:30:00 -#SBATCH --output=/home/jiacao/InferenceX/imgdiff-%j.out -#SBATCH --exclude=crsuse2-m2m-203,crsuse2-m2m-074,crsuse2-m2m-240,crsuse2-m2m-029,crsuse2-m2m-061,crsuse2-m2m-071 -# -# Establish the provenance of the DSv4 measurement image against the intended -# CI base, so every delta can be attributed to an upstream PR (or flagged as -# having none) in a docs/waiver/.md table. -# -# BASE is the image the CI config would pin. NEW is the image every measured -# number in this campaign was produced on (see sbatch_bake_pr51473_image.sh: -# it is vendor base f2fbead + the two-file #51473 back-port). -# -# No GPU is requested: this only untars layers and runs diff. That also lets -# the job schedule immediately alongside the measurement jobs rather than -# queueing behind them. -# -# Run 6309 extracted 0 bytes from BASE: its tar wildcards assumed a layout -# ('*/site-packages/vllm/*.py', '*/vllm/vllm/*.py') that matches the vendor -# image's editable /src/vllm checkout but not the nightly's. Rather than guess -# again, this version extracts the whole filesystem and then locates each -# package by finding its __init__.py. Costs disk and a few minutes; removes the -# failure mode entirely. -set -uo pipefail - -BASE="${BASE:-vllm/vllm-openai-rocm:nightly-b22afe45ac797ae58e67a7a3ad79ee5714024420}" -NEW="${NEW:-jiahcao/vllm-dsv4@sha256:5e44cbd690811bdf9129bd2f552f22241177c6ad0c51bd057afb817cca35e1e9}" -OUT="${OUT:-/home/jiacao/imgdiff-$(date -u +%Y%m%d)-full}" - -hostname; date -u -echo "BASE : $BASE" -echo "NEW : $NEW" -echo "OUT : $OUT" -echo - -fail() { echo "FATAL: $*" >&2; exit 1; } -docker info >/dev/null 2>&1 || fail "no docker access on $(hostname)" -rm -rf "$OUT"; mkdir -p "$OUT" || fail "cannot create $OUT" - -# ---------------------------------------------------------------- pull -for img in "$BASE" "$NEW"; do - for _try in 1 2 3; do - docker image inspect "$img" >/dev/null 2>&1 && break - echo "pulling $img (attempt $_try/3) ... $(date -u +%H:%M:%S)Z" - docker pull "$img" || true - done - docker image inspect "$img" >/dev/null 2>&1 || fail "pull failed: $img" -done -echo "PULL OK $(date -u +%H:%M:%S)Z" - -# ---------------------------------------------------------------- locate -# Ask each image where it actually imports vllm/aiter from, instead of -# assuming. --entrypoint /bin/bash is safe here because nothing imports the -# package (importing aiter runs rocminfo, which aborts without a GPU); this -# only reads importlib's spec origin. -locate_in() { # $1=image $2=pkg -> prints the in-container package dir - docker run --rm --entrypoint /bin/bash "$1" -c \ - "python3 -c 'import importlib.util as u,os,sys -s=u.find_spec(\"$2\") -print(os.path.dirname(s.origin) if s and s.origin else \"\")' 2>/dev/null" 2>/dev/null | tr -d '\r' | tail -1 -} - -declare -A PKGDIR -for side in base new; do - img="$BASE"; [ "$side" = "new" ] && img="$NEW" - for pkg in vllm aiter; do - d=$(locate_in "$img" "$pkg") - PKGDIR[$side:$pkg]="$d" - echo " $side/$pkg -> ${d:-}" - done -done -echo - -# ---------------------------------------------------------------- export -# Extract the located package dirs only. `docker export` streams the flattened -# container filesystem; the tar member paths are relative (no leading /), so -# strip the leading slash off the located path to build the match pattern. -extract() { # $1=image $2=destdir $3..=in-container abs dirs - local img="$1" dst="$2"; shift 2 - local cid pats=() - mkdir -p "$dst" - for p in "$@"; do - [ -n "$p" ] && pats+=("${p#/}/*") && pats+=("${p#/}") - done - [ ${#pats[@]} -eq 0 ] && { echo " no patterns for $img"; return 1; } - cid=$(docker create "$img") || fail "docker create failed for $img" - docker export "$cid" | tar -x -C "$dst" --wildcards "${pats[@]}" 2>/dev/null - docker rm -f "$cid" >/dev/null 2>&1 - return 0 -} - -echo "extracting BASE ... $(date -u +%H:%M:%S)Z" -extract "$BASE" "$OUT/base" "${PKGDIR[base:vllm]}" "${PKGDIR[base:aiter]}" -echo " base tree: $(du -sh "$OUT/base" 2>/dev/null | cut -f1)" -echo "extracting NEW ... $(date -u +%H:%M:%S)Z" -extract "$NEW" "$OUT/new" "${PKGDIR[new:vllm]}" "${PKGDIR[new:aiter]}" -echo " new tree: $(du -sh "$OUT/new" 2>/dev/null | cut -f1)" - -# Resolve back to the extracted-on-disk package root. The in-container path is -# reproduced verbatim under $dst, so join them -- but fall back to a search in -# case the layout surprises us again. -onbox() { # $1=side $2=pkg - local d="$OUT/$1/${PKGDIR[$1:$2]#/}" - [ -d "$d" ] && { echo "$d"; return; } - find "$OUT/$1" -type f -path "*/$2/__init__.py" 2>/dev/null \ - | awk '{print length"\t"$0}' | sort -n | head -1 | cut -f2- | xargs -r dirname -} - -# ---------------------------------------------------------------- diff -for pkg in vllm aiter; do - b=$(onbox base "$pkg"); n=$(onbox new "$pkg") - echo - echo "=============== $pkg ===============" - echo "base: ${b:-}" - echo "new : ${n:-}" - if [ -z "$b" ] || [ -z "$n" ] || [ ! -d "$b" ] || [ ! -d "$n" ]; then - echo "SKIP: $pkg missing on one side" - continue - fi - # Compare .py only: the compiled extensions and .co kernels are opaque to - # diff, and __pycache__ differs on mtime alone. - diff -ruN -x '__pycache__' -x '*.pyc' -x '*.so' "$b" "$n" > "$OUT/$pkg.diff" 2>/dev/null - nfiles=$(grep -c '^diff -ruN' "$OUT/$pkg.diff" 2>/dev/null || echo 0) - echo "changed files: $nfiles" - echo "diff bytes : $(stat -c%s "$OUT/$pkg.diff" 2>/dev/null || echo 0)" - echo - echo "--- per-file churn (+added -removed) ---" - awk -v base="$b" -v new="$n" ' - /^diff -ruN/ { - if (f != "") printf " %+7d %-7d %s\n", a, -r, f - f=$NF; sub(new"/","",f); a=0; r=0; next - } - /^\+/ && !/^\+\+\+/ { a++ } - /^-/ && !/^---/ { r++ } - END { if (f != "") printf " %+7d %-7d %s\n", a, -r, f } - ' "$OUT/$pkg.diff" | sort -k3 -done - -# ---------------------------------------------------------------- surfaces -# The knobs the CI cannot reach are engine-side features, not recipe flags. -# Whether BASE honours them at all is the question the grid depends on. -echo -echo "=============== feature surface: BASE vs NEW ===============" -probe() { # $1=label $2=pattern - echo "$1:" - for side in base new; do - hits=$(grep -rl -- "$2" "$OUT/$side" --include='*.py' 2>/dev/null | wc -l) - printf " %-5s %s file(s)\n" "$side" "$hits" - done -} -probe "VLLM_ROCM_DSV4_SPARSE_GLUON" "VLLM_ROCM_DSV4_SPARSE_GLUON" -probe "VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS" "VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS" -probe "flydsl_mega_moe (literal)" "flydsl_mega_moe" -probe "mega_moe (any)" "mega_moe" - -echo -echo "=============== accepted --moe-backend literals ===============" -for side in base new; do - echo "-- $side" - grep -rhoE '"[a-z0-9_]*mega_moe[a-z0-9_]*"' "$OUT/$side" --include='*.py' 2>/dev/null \ - | sort -u | sed 's/^/ /' -done - -# The #51473 marker: the one line the whole back-port reduces to. -echo -echo "=============== vLLM #51473 marker (384-wide TP8 shard) ===============" -for side in base new; do - if grep -rq 'AITER_MXFP4_BF16 and activation == MoEActivation.SILU' "$OUT/$side" 2>/dev/null; then - echo " $side: PRESENT" - else - echo " $side: ABSENT" - fi -done - -echo -echo "DONE $(date -u +%H:%M:%S)Z" -echo "Diffs written under: $OUT" diff --git a/spur-6309.out b/spur-6309.out deleted file mode 100644 index e69de29bb..000000000 diff --git a/spur-6316.out b/spur-6316.out deleted file mode 100644 index e69de29bb..000000000 diff --git a/spur-6345.out b/spur-6345.out deleted file mode 100644 index e69de29bb..000000000 From 1658f3b4dc02b688304759428bbd75b3d6aacba1 Mon Sep 17 00:00:00 2001 From: Jiahui Cao Date: Thu, 13 Aug 2026 05:54:17 +0000 Subject: [PATCH 4/4] feat(dsv4): enable gluon, MegaMoE and FSE from upstream refs; drop the #4417 graft The patch stack went in as "stock base + upstream PRs", but three knobs it claimed or implied were reachable were not. sparse gluon was silently dead. An earlier revision copied only the gfx950 gluon kernel and deliberately left the three pa_decode_sparse variants that ship in the base untouched, to avoid a cross-version transplant. That was the right instinct and the wrong call for the facade: the base's aiter/ops/triton/attention/pa_decode_sparse.py predates the extra_cache / extra_indices / extra_indptr keywords that vllm #51714's call site passes, so with VLLM_ROCM_DSV4_SPARSE_GLUON=1 all eight workers raised TypeError: pa_decode_sparse() got an unexpected keyword argument 'extra_cache' on first decode and latched onto the Triton fallback for the life of the process. The knob was accepted, logged as on, and inert. The facade and the kernel it dispatches to now move together, and the verify step asserts the accepted keywords by signature introspection -- file presence is exactly what gave the false positive. MegaMoE and FSE are now reachable. mori/ir/flydsl is 3 plain .py files, not a compiled artifact, but they drag in the cov cascade: runtime.py calls find_bitcode(cov=6) and cov threads through ir/bitcode.py, jit/cache.py and jit/core.py (37 upstream call sites, 0 in the base) plus jit/config.py for is_debuginfo_enabled. Partial installs fail in that exact order. FSE needs aiter fhmoe plus _flydsl_moe_stage1_impl/_flydsl_moe_stage2_impl, which live in a moe_kernels.py the base does not have; that file is the one wholesale replacement, AST symbol-diffed first (58 -> 66 symbols, 0 lost, 8 gained). With mori.ir.flydsl present, #51918's model-side hunks are no longer pointless, so it now applies in full across all 6 files. graft_aiter_4417.py is deleted. #4417 ships in the aiter@97d0c6e4 moe_kernels.py installed above (grep -c = 5), and neither guard can fire here regardless: requires_flydsl_stage2_reduce first fires at 299594 tokens against this recipe's 65536 ceiling, and resolve_flydsl_grid_y_persist_m returns persist_m=1 with 8x of headroom under the grid.y cap. The sweep grid varies concurrency only. Everything is sourced from aiter@97d0c6e4 or mori@84a33cc -- no private image, no binary payloads (libmori_shmem_device.bc is JIT-compiled in-container with hipcc on first import). Verified from scratch on a container created from the pinned image: 21 steps, VERIFY: ALL_OK, and a second run reports same / already present for all of them. Co-Authored-By: Claude Opus 5 --- .../agentic/apply_dsv4_container_patches.sh | 370 ++++++++++++------ .../single_node/agentic/graft_aiter_4417.py | 125 ------ docs/waiver/2589.md | 121 ++++-- perf-changelog.yaml | 7 +- 4 files changed, 337 insertions(+), 286 deletions(-) delete mode 100644 benchmarks/single_node/agentic/graft_aiter_4417.py diff --git a/benchmarks/single_node/agentic/apply_dsv4_container_patches.sh b/benchmarks/single_node/agentic/apply_dsv4_container_patches.sh index 85a25663d..ef9f5c1b2 100755 --- a/benchmarks/single_node/agentic/apply_dsv4_container_patches.sh +++ b/benchmarks/single_node/agentic/apply_dsv4_container_patches.sh @@ -1,165 +1,279 @@ #!/usr/bin/env bash # ============================================================================= -# ADDITIVE patch stack for the 08-09 nightly base +# Container patch stack for the 08-09 nightly base # vllm/vllm-openai-rocm:nightly-f8d03e77416bf90c49acbe50e233275722f02c4b # (vllm 0.26.1rc1.dev528+gf8d03e774) # -# STRICTLY ADDITIVE. Nothing that ships in the base is overwritten: -# * only the ONE gluon kernel file the base genuinely lacks is copied in -# (gfx950/attention/pa_decode_sparse.py). The other three pa_decode_sparse -# variants SHIP IN THE BASE and are deliberately NOT touched -- an earlier -# revision of this script copied all four out of aiter@97d0c6e4, silently -# replacing three base files with a cross-version transplant. Do not -# reintroduce that. -# * aiter core (jit/core.py, fused_moe.py) is not modified. The base already -# exposes _set_current_hip_stream, which the nightly vllm calls for -# module_rmsnorm_quant; the old wholesale post2-python overlay regressed it -# and crashed rmsnorm_quant warmup. -# * MegaMoE/DEP8 is out of scope on this route: its intranode kernel needs -# mori.ir.flydsl, which is absent from the base (verified: ModuleNotFoundError). -# FSE is likewise out (aiter #4269 needs aiter/fhmoe.py, also absent). -# -# Carried: -# vllm #51473 native MXFP4 TP8 shard allocation (MERGED 2026-08-11 -- i.e. -# AFTER this 08-09 base, so unlike the 08-12 base it must be -# applied here). One hunk on -# vllm/model_executor/layers/fused_moe/oracle/mxfp4.py. -# vllm #51714 opt-in AITER gluon kernel for sparse-MLA decode on gfx950 -# (open). Dormant unless VLLM_ROCM_DSV4_SPARSE_GLUON=1. -# vllm #51918 FlyDSL fused mega-MoE backend (open). Dormant unless -# --moe-backend flydsl_mega_moe is selected. -# aiter #4417 large-token FlyDSL MoE launch/output limits (merged -# 2026-07-30; the base's VENDORED aiter predates it -- the -# vendored revision is pinned and does not track aiter main). +# Everything here comes from an UPSTREAM ref -- either a vllm PR diff or a +# pinned aiter/mori commit. Nothing is sourced from a private measurement +# image. Idempotent: every step is marker-gated, re-running is a no-op. # # Run INSIDE a fresh container of the pinned image: -# docker exec -i bash /path/to/dsv4_patch_0809.sh -# Idempotent: every step is marker-gated, re-running is a no-op. +# docker exec -i bash /path/to/apply_dsv4_container_patches.sh +# +# WHAT THIS ENABLES +# Three knobs that the base cannot run unpatched, all three verified to +# import cleanly afterwards (see the checks at the end): +# * sparse gluon decode (VLLM_ROCM_DSV4_SPARSE_GLUON=1) +# * FSE (aiter fhmoe) +# * MegaMoE / DEP8 (--moe-backend flydsl_mega_moe) +# +# The gluon knob in particular was silently dead before this: the base's +# aiter facade pa_decode_sparse() predates the extra_cache/extra_indices/ +# extra_indptr keywords that vllm #51714's call site passes, so every worker +# raised +# TypeError: pa_decode_sparse() got an unexpected keyword argument +# 'extra_cache' +# on its first decode and latched onto the Triton fallback for the rest of +# the process. Copying only the gfx950 gluon kernel is NOT enough; the facade +# that dispatches to it has to come along. +# +# WHAT IS NOT DONE +# * aiter core (jit/core.py) is not modified. The base already exposes +# _set_current_hip_stream, which the nightly vllm calls for +# module_rmsnorm_quant; an earlier wholesale post2-python overlay regressed +# it and crashed rmsnorm_quant warmup. +# * aiter #4417 is NOT grafted separately. It is already contained in the +# aiter@$AITER_SHA moe_kernels.py this script installs, so the old +# anchor-graft helper is gone. # ============================================================================= set -uo pipefail AITER_SHA="97d0c6e4cb7a0919c12291c7c7d560ad412f15c1" AITER_REPO="https://github.com/ROCm/aiter" +MORI_SHA="84a33cc0f15f019c78c995728973b70ea3d10bb7" +MORI_REPO="https://github.com/ROCm/mori" VLLM_REPO="https://github.com/vllm-project/vllm" ROOT="$(python -c 'import importlib.util as u, os; print(os.path.dirname(os.path.dirname(u.find_spec("vllm").origin)))')" [ -d "$ROOT/vllm" ] && [ -d "$ROOT/aiter" ] || { echo "ERROR ROOT=$ROOT"; exit 1; } -echo "[add] ROOT=$ROOT" -echo "[add] vllm = $(python -c 'import vllm;print(vllm.__version__)' 2>/dev/null)" -WS=/tmp/dsv4_add; mkdir -p "$WS" - -# --- 1/4 gluon kernel: ONLY the file the base lacks --------------------------- -# Verified against this base: gfx1250/, _triton_kernels/ and the ops/triton/ -# facade all ship. Copying the aiter@97d0c6e4 versions over them is the -# cross-version transplant that has to be avoided, so the loop SKIPS anything -# already present rather than overwriting it. -GLUON_PATHS=( - aiter/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py - aiter/ops/triton/_gluon_kernels/gfx1250/attention/pa_decode_sparse.py - aiter/ops/triton/_triton_kernels/attention/pa_decode_sparse.py +echo "[patch] ROOT=$ROOT" +echo "[patch] vllm = $(python -c 'import vllm;print(vllm.__version__)' 2>/dev/null)" +WS=/tmp/dsv4_patch; mkdir -p "$WS" + +# --- 1/4 aiter files from the pinned SHA -------------------------------------- +# Sourced from aiter@$AITER_SHA, not from any prebuilt image. +# +# A note on why the facade is in this list. An earlier revision of this script +# copied only the ONE file the base lacked (the gfx950 gluon kernel) and +# deliberately skipped the three pa_decode_sparse variants that ship in the +# base, to avoid a cross-version transplant. That was the right instinct but +# the wrong call for the facade specifically: keeping the base's older facade +# is what produced the extra_cache TypeError above. The facade and the kernel +# it dispatches to are one unit and have to move together. +# +# moe_kernels.py is replaced rather than patched. Symbol-diffed base vs +# upstream before doing so: 58 base symbols, 66 upstream, ZERO lost, +8 gained +# (including _flydsl_moe_stage1_impl / _flydsl_moe_stage2_impl, which FSE's +# aiter/ops/flydsl/fhmoe.py imports and the base simply does not define). +AITER_FILES=( + # sparse gluon decode: the kernel AND the facade that dispatches to it aiter/ops/triton/attention/pa_decode_sparse.py + aiter/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py + # FSE (fused heterogeneous MoE) + aiter/fhmoe.py + aiter/ops/flydsl/fhmoe.py + aiter/ops/flydsl/kernels/fhmoe.py + aiter/aot/flydsl/fhmoe.py + # carries FSE's stage1/stage2 impls, and aiter #4417's two guards + aiter/ops/flydsl/moe_kernels.py ) -NEED=() -for p in "${GLUON_PATHS[@]}"; do - if [ -f "$ROOT/$p" ]; then echo " keep base $p"; else NEED+=("$p"); fi +ASRC="$WS/aiter_src" +if [ ! -d "$ASRC/.git" ]; then git clone -q --filter=blob:none --no-checkout "$AITER_REPO" "$ASRC" 2>&1 | tail -1; fi +( cd "$ASRC" && git fetch -q --depth 1 origin "$AITER_SHA" 2>&1 | tail -1 && git checkout -q "$AITER_SHA" -- "${AITER_FILES[@]}" ) \ + || { echo " aiter: CHECKOUT FAILED"; exit 1; } +for p in "${AITER_FILES[@]}"; do + [ -e "$ASRC/$p" ] || { echo " MISSING in src: $p"; continue; } + if [ -f "$ROOT/$p" ] && cmp -s "$ASRC/$p" "$ROOT/$p"; then echo " same $p"; continue; fi + mkdir -p "$(dirname "$ROOT/$p")"; cp -a "$ASRC/$p" "$ROOT/$p"; echo " installed $p" done -if [ ${#NEED[@]} -gt 0 ]; then - SRC="$WS/aiter_src" - if [ ! -d "$SRC/.git" ]; then git clone --filter=blob:none --no-checkout "$AITER_REPO" "$SRC" 2>&1 | tail -1; fi - ( cd "$SRC" && git fetch --depth 1 origin "$AITER_SHA" 2>&1 | tail -1 && git checkout -q "$AITER_SHA" -- "${NEED[@]}" ) - for p in "${NEED[@]}"; do - [ -e "$SRC/$p" ] || { echo " MISSING in src: $p"; continue; } - mkdir -p "$(dirname "$ROOT/$p")"; cp -a "$SRC/$p" "$ROOT/$p"; echo " added $p" + +# --- 2/4 mori: mori.ir.flydsl + the cov cascade ------------------------------- +# MegaMoE's intranode dispatch/combine kernel does `import mori.ir.flydsl`, and +# the base ships mori WITHOUT that subpackage. Three .py files -- no compiled +# artifact, contrary to a first reading of the ModuleNotFoundError. +# +# They cannot be dropped in alone. mori/ir/flydsl/runtime.py calls +# find_bitcode(cov=6) (FlyDSL needs ABI 600), and `cov` is a parameter that +# cascades through three more files the base predates: ir/bitcode.py, +# jit/cache.py, jit/core.py (37 call sites upstream, 0 in the base) plus +# jit/config.py for is_debuginfo_enabled. Installing a subset yields, in order, +# TypeError: find_bitcode() got an unexpected keyword argument 'cov' +# ImportError: cannot import name 'is_debuginfo_enabled' +# so the whole cascade goes or none of it does. +# +# libmori_shmem_device.bc is NOT shipped here. It is a 485 KB LLVM IR blob and +# a build artifact; find_bitcode() falls back to mori.jit.core.ensure_bitcode(), +# which compiles it in-container with hipcc on first import (~1 min). That +# keeps this script free of binary payloads. +MORI_FILES=( + mori/ir/flydsl/__init__.py + mori/ir/flydsl/ops.py + mori/ir/flydsl/runtime.py + mori/ir/bitcode.py + mori/jit/cache.py + mori/jit/config.py + mori/jit/core.py +) +if [ -d "$ROOT/mori" ]; then + MSRC="$WS/mori_src" + if [ ! -d "$MSRC/.git" ]; then git clone -q --filter=blob:none --no-checkout "$MORI_REPO" "$MSRC" 2>&1 | tail -1; fi + ( cd "$MSRC" && git fetch -q --depth 1 origin "$MORI_SHA" 2>&1 | tail -1 && git checkout -q "$MORI_SHA" -- python/mori ) \ + || { echo " mori: CHECKOUT FAILED"; } + for p in "${MORI_FILES[@]}"; do + src="$MSRC/python/$p" + [ -e "$src" ] || { echo " MISSING in src: $p"; continue; } + if [ -f "$ROOT/$p" ] && cmp -s "$src" "$ROOT/$p"; then echo " same $p"; continue; fi + mkdir -p "$(dirname "$ROOT/$p")"; cp -a "$src" "$ROOT/$p"; echo " installed $p" done + find "$ROOT/mori" -name '__pycache__' -type d -exec rm -rf {} + 2>/dev/null else - echo " (nothing to add)" + echo " mori: not installed in this image, skipping MegaMoE deps" fi - -# --- 2/4 aiter #4417: large-token FlyDSL MoE launch/output limits -------------- -# Two guards: stage2 buffer atomics address the output with 32-bit byte offsets -# (>4 GiB walks off the end), and HIP caps grid.y at 65535. Neither fires at the -# DSv4-Pro TP8 shape measured here -- requires_flydsl_stage2_reduce(65536, 7168, -# 2) is False (~939 MB) -- so this is NOT a fix for the profile-run memfault; it -# is carried because it is a real gap any larger-token sweep row would hit. -# The upstream .diff will NOT apply: the base's aiter predates aiter's typing -# modernization, so every context line still reads Dict[str, Dict] where the -# diff expects dict[str, dict]. The hunks are grafted by anchor instead. -python "$(dirname "$0")/graft_aiter_4417.py" "$ROOT/aiter/ops/flydsl/moe_kernels.py" || \ - { echo " #4417: GRAFT FAILED"; exit 1; } +find "$ROOT/aiter" -name '__pycache__' -type d -exec rm -rf {} + 2>/dev/null # --- 3/4 vllm PRs ------------------------------------------------------------- -apply_pr(){ local pr="$1" mf="$ROOT/$2" mk="$3" d="$WS/vllm_$1.diff" - if grep -qF "$mk" "$mf" 2>/dev/null; then echo " #$pr: already present"; return 0; fi - curl -ksSL -o "$d" "$VLLM_REPO/pull/$pr.diff" || { echo " #$pr: FETCH FAIL"; return 1; } - if ( cd "$ROOT" && git apply -p1 --3way "$d" ) 2>/dev/null || ( cd "$ROOT" && git apply -p1 "$d" ) 2>/dev/null - then echo " #$pr: APPLIED"; else echo " #$pr: FAILED"; return 1; fi +fetch_diff(){ local pr="$1"; local d="$WS/vllm_$pr.diff" + [ -s "$d" ] || curl -ksSL -o "$d" "$VLLM_REPO/pull/$pr.diff" || return 1 + [ -s "$d" ] || return 1; echo "$d" +} + +# Split a PR diff down to the hunks for ONE file. `patch -i ` +# does NOT do this -- it applies every hunk in the diff to the named file. Doing +# that once wrote dspark.py's and mtp.py's hunks into model.py and left it with +# a SyntaxError (an import landed mid-docstring). Always split first. +split_diff(){ local d="$1" f="$2" out="$3" + python - "$d" "$f" "$out" <<'PYEOF' +import re, sys +src, target, out = sys.argv[1], sys.argv[2], sys.argv[3] +parts = re.split(r"(?m)^(?=diff --git )", open(src).read()) +keep = [p for p in parts if p.startswith("diff --git a/%s " % target)] +open(out, "w").write("".join(keep)) +sys.exit(0 if keep else 1) +PYEOF } -# #51473 carries a tests/ hunk that has no counterpart in an installed wheel, so -# apply only the runtime file rather than the whole PR diff. -apply_51473(){ - local mf="$ROOT/vllm/model_executor/layers/fused_moe/oracle/mxfp4.py" - local mk="AITER_MXFP4_BF16 and activation == MoEActivation.SILU" - if grep -qF "$mk" "$mf" 2>/dev/null; then echo " #51473: already present"; return 0; fi - curl -ksSL -o "$WS/vllm_51473.diff" "$VLLM_REPO/pull/51473.diff" || { echo " #51473: FETCH FAIL"; return 1; } - ( cd "$ROOT" && git apply -p1 --include='vllm/*' --3way "$WS/vllm_51473.diff" ) 2>/dev/null \ - || ( cd "$ROOT" && git apply -p1 --include='vllm/*' "$WS/vllm_51473.diff" ) 2>/dev/null \ - && echo " #51473: APPLIED" || { echo " #51473: FAILED"; return 1; } + +# Apply one file's hunks. git apply first (exact); fall back to patch --fuzz, +# which tolerates the context drift from #51918 being written against a tree a +# few days newer than this base. +apply_file(){ local pr="$1" f="$2" mk="$3" fuzz="${4:-0}" + if [ -n "$mk" ] && grep -qF "$mk" "$ROOT/$f" 2>/dev/null; then echo " #$pr $f: already present"; return 0; fi + local d; d="$(fetch_diff "$pr")" || { echo " #$pr: FETCH FAIL"; return 1; } + local one="$WS/${pr}_$(echo "$f" | tr / _).diff" + split_diff "$d" "$f" "$one" || { echo " #$pr $f: no hunks in diff"; return 1; } + if ( cd "$ROOT" && git apply -p1 "$one" ) 2>/dev/null; then + echo " #$pr $f: APPLIED"; return 0 + fi + if [ "$fuzz" != "0" ]; then + if ( cd "$ROOT" && patch -p1 --forward --fuzz="$fuzz" --no-backup-if-mismatch -s -i "$one" ) 2>/dev/null; then + echo " #$pr $f: APPLIED (fuzz=$fuzz)" + ( cd "$ROOT" && rm -f "$f.rej" "$f.orig" ) + return 0 + fi + ( cd "$ROOT" && rm -f "$f.rej" "$f.orig" ) + fi + echo " #$pr $f: FAILED"; return 1 } -apply_51473 || true -apply_pr 51714 "vllm/v1/attention/ops/rocm_aiter_mla_sparse.py" "_DSV4_SPARSE_GLUON" || true -# #51918 is taken PARTIALLY, on purpose: only vllm/config/kernel.py, which -# registers "flydsl_mega_moe" as an accepted --moe-backend value. The model-side -# hunks are NOT applied. -# -# Two independent reasons, either of which is sufficient: -# 1. The backend cannot run on this base at all. MegaMoE's intranode kernel -# imports mori.ir.flydsl, and this image has no mori.ir.flydsl (verified: -# ModuleNotFoundError). Applying the model hunks would buy a backend that -# raises on first use. -# 2. The model.py hunks do not apply cleanly here. #51918 is written against a -# tree ~3 days newer than this 08-09 base; the two earlier hunks shift the -# line numbering enough that the third fails at model.py:300. `git apply -# --3way` cannot rescue it because site-packages is not a git repo, so -# there are no blobs to 3-way against. Force-grafting a DEP8 code path that -# cannot execute anyway is not worth the transplant risk -- that is exactly -# how the earlier pa_decode_sparse damage happened. +# #51473 -- native MXFP4 TP8 shard allocation. MERGED 2026-08-11, i.e. AFTER +# this 08-09 base, so unlike the 08-12 line it must be applied here. Its tests/ +# hunk has no counterpart in an installed wheel, hence per-file application. +apply_file 51473 "vllm/model_executor/layers/fused_moe/oracle/mxfp4.py" \ + "AITER_MXFP4_BF16 and activation == MoEActivation.SILU" || true + +# #51714 -- opt-in gluon sparse-MLA decode for gfx950. Dormant unless +# VLLM_ROCM_DSV4_SPARSE_GLUON=1. Needs the aiter facade from step 1 to work. +apply_file 51714 "vllm/v1/attention/ops/rocm_aiter_mla_sparse.py" "_DSV4_SPARSE_GLUON" || true + +# #51918 -- FlyDSL mega-MoE backend, now applied IN FULL. +# A previous revision took only config/kernel.py (the backend name) and skipped +# the model side, on the grounds that mori.ir.flydsl was missing so the backend +# could never run. Step 2 removes that blocker, so the model hunks go in too. +# model.py needs fuzz=3: #51918 targets a tree a few days newer and the earlier +# hunks shift line numbering. All 6 of its hunks land; the result is +# py_compile-clean and imports (checked below). # -# Net effect: the TP8 arm is unaffected (it never selects this backend), and a -# DEP8 row would be rejected at config time with a clear error instead of -# failing deep inside a kernel. DEP8/MegaMoE stays out of scope on this route. -apply_pr_files(){ local pr="$1" inc="$2" mf="$ROOT/$3" mk="$4" d="$WS/vllm_$1.diff" - if grep -qF "$mk" "$mf" 2>/dev/null; then echo " #$pr ($inc): already present"; return 0; fi - curl -ksSL -o "$d" "$VLLM_REPO/pull/$pr.diff" || { echo " #$pr: FETCH FAIL"; return 1; } - ( cd "$ROOT" && git apply -p1 --include="$inc" "$d" ) 2>/dev/null \ - && echo " #$pr ($inc): APPLIED" || { echo " #$pr ($inc): FAILED"; return 1; } -} -apply_pr_files 51918 "vllm/config/kernel.py" "vllm/config/kernel.py" "flydsl_mega_moe" || true +# Each entry carries its own presence marker. Without one the step re-runs on an +# already-patched container, git apply correctly refuses, and the log says +# FAILED -- functionally a no-op, but it reads like a real failure in CI output. +apply_file 51918 "vllm/config/kernel.py" "flydsl_mega_moe" || true +apply_file 51918 "vllm/models/deepseek_v4/amd/dspark.py" "finalize_mega_moe_layers" || true +apply_file 51918 "vllm/models/deepseek_v4/amd/mega_moe_experts.py" "MegaMoE expert layer" || true +apply_file 51918 "vllm/models/deepseek_v4/amd/mega_moe_runtime.py" "MegaMoEV2 runtime" || true +apply_file 51918 "vllm/models/deepseek_v4/amd/mtp.py" "finalize_mega_moe_layers" || true +apply_file 51918 "vllm/models/deepseek_v4/amd/model.py" "use_mega_moe" 3 || true # --- 4/4 verify --------------------------------------------------------------- -echo "chk gluon gfx950 = $([ -f "$ROOT/aiter/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py" ] && echo present || echo MISSING)" -echo "chk aiter #4417 = $(grep -c 'requires_flydsl_stage2_reduce\|resolve_flydsl_grid_y_persist_m' "$ROOT/aiter/ops/flydsl/moe_kernels.py" 2>/dev/null) (expect 5)" -echo "chk vllm #51473 = $(grep -c 'AITER_MXFP4_BF16 and activation == MoEActivation.SILU' "$ROOT/vllm/model_executor/layers/fused_moe/oracle/mxfp4.py" 2>/dev/null) (expect 1)" -echo "chk vllm #51714 = $(grep -c '_DSV4_SPARSE_GLUON' "$ROOT/vllm/v1/attention/ops/rocm_aiter_mla_sparse.py" 2>/dev/null)" -echo "chk vllm #51918 = $(grep -c 'flydsl_mega_moe' "$ROOT/vllm/config/kernel.py" 2>/dev/null)" -python -m py_compile "$ROOT/aiter/ops/triton/attention/pa_decode_sparse.py" \ +echo "chk gluon gfx950 kernel = $([ -f "$ROOT/aiter/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py" ] && echo present || echo MISSING)" +echo "chk aiter #4417 guards = $(grep -c 'requires_flydsl_stage2_reduce\|resolve_flydsl_grid_y_persist_m' "$ROOT/aiter/ops/flydsl/moe_kernels.py" 2>/dev/null) (expect 5)" +echo "chk vllm #51473 = $(grep -c 'AITER_MXFP4_BF16 and activation == MoEActivation.SILU' "$ROOT/vllm/model_executor/layers/fused_moe/oracle/mxfp4.py" 2>/dev/null) (expect 1)" +echo "chk vllm #51714 = $(grep -c '_DSV4_SPARSE_GLUON' "$ROOT/vllm/v1/attention/ops/rocm_aiter_mla_sparse.py" 2>/dev/null)" +echo "chk vllm #51918 name = $(grep -c 'flydsl_mega_moe' "$ROOT/vllm/config/kernel.py" 2>/dev/null)" +echo "chk vllm #51918 model = $(grep -c 'use_mega_moe' "$ROOT/vllm/models/deepseek_v4/amd/model.py" 2>/dev/null) (expect >0)" + +python -m py_compile \ + "$ROOT/aiter/ops/triton/attention/pa_decode_sparse.py" \ + "$ROOT/aiter/ops/flydsl/moe_kernels.py" \ "$ROOT/vllm/model_executor/layers/fused_moe/oracle/mxfp4.py" \ - "$ROOT/vllm/v1/attention/ops/rocm_aiter_mla_sparse.py" && echo PY_COMPILE_OK || { echo PY_COMPILE_FAIL; exit 1; } + "$ROOT/vllm/v1/attention/ops/rocm_aiter_mla_sparse.py" \ + "$ROOT/vllm/models/deepseek_v4/amd/model.py" \ + "$ROOT/vllm/models/deepseek_v4/amd/mtp.py" \ + "$ROOT/vllm/models/deepseek_v4/amd/dspark.py" \ + "$ROOT/vllm/models/deepseek_v4/amd/mega_moe_experts.py" \ + "$ROOT/vllm/models/deepseek_v4/amd/mega_moe_runtime.py" \ + "$ROOT/vllm/config/kernel.py" && echo PY_COMPILE_OK || { echo PY_COMPILE_FAIL; exit 1; } + python - <<'PYEOF' -import importlib -for m in ["aiter.jit.core","aiter.ops.triton.attention.pa_decode_sparse", +import importlib, inspect, re, sys + +fail = False + +# The facade signature is the check that actually matters for the gluon knob: +# the kernel file being present says nothing about whether the call site's +# keywords are accepted. +try: + from aiter.ops.triton.attention.pa_decode_sparse import pa_decode_sparse as f + p = inspect.signature(f).parameters + missing = [k for k in ("extra_cache", "extra_indices", "extra_indptr") if k not in p] + if missing: + fail = True + print("FACADE_MISSING", missing, "-- gluon would fall back to Triton at runtime") + else: + print("FACADE_OK extra_cache/extra_indices/extra_indptr accepted") +except Exception as e: + fail = True + print("FACADE_ERR", type(e).__name__, e) + +for m in ["aiter.jit.core", + "aiter.ops.triton.attention.pa_decode_sparse", + "aiter.fhmoe", # FSE + "aiter.ops.flydsl.fhmoe", # FSE + "mori.ir.flydsl", # MegaMoE + "aiter.ops.flydsl.kernels.flydsl_dispatch_combine_intranode_kernel", "vllm.model_executor.layers.fused_moe.oracle.mxfp4", - "vllm.v1.attention.ops.rocm_aiter_mla_sparse","vllm._aiter_ops"]: - try: importlib.import_module(m); print("IMPORT_OK",m) - except Exception as e: print("IMPORT_ERR",m,type(e).__name__,(str(e).splitlines() or [''])[-1]) -# _set_current_hip_stream is NOT a top-level attribute of aiter.jit.core -- it is -# called there as `module._set_current_hip_stream(...)`, i.e. it lives on the + "vllm.v1.attention.ops.rocm_aiter_mla_sparse", + "vllm.models.deepseek_v4.amd.model", + "vllm._aiter_ops"]: + try: + importlib.import_module(m) + print("IMPORT_OK", m) + except Exception as e: + fail = True + print("IMPORT_ERR", m, type(e).__name__, (str(e).splitlines() or [""])[-1]) + +# _set_current_hip_stream is NOT a top-level attribute of aiter.jit.core -- it +# is called there as `module._set_current_hip_stream(...)`, i.e. it lives on the # compiled .so that core.py loads. hasattr(core, ...) therefore returns False on # a perfectly healthy tree; an earlier version of this probe read that False as -# a missing symbol. Grep the call site instead: what the old wholesale -# post2-python overlay actually did was regress core.py so the call vanished. +# a missing symbol. Grep the call site instead. try: - import aiter.jit.core as c, inspect, re + import aiter.jit.core as c n = len(re.findall(r"_set_current_hip_stream", inspect.getsource(c))) print(f"core.py _set_current_hip_stream call sites: {n} (expect >=1)") -except Exception as e: print("core probe err",e) +except Exception as e: + print("core probe err", e) + +print("VERIFY:", "FAILURES_PRESENT" if fail else "ALL_OK") +sys.exit(1 if fail else 0) PYEOF -echo "[add] DONE" +rc=$? +echo "[patch] DONE (verify rc=$rc)" +exit $rc diff --git a/benchmarks/single_node/agentic/graft_aiter_4417.py b/benchmarks/single_node/agentic/graft_aiter_4417.py deleted file mode 100644 index f347870ad..000000000 --- a/benchmarks/single_node/agentic/graft_aiter_4417.py +++ /dev/null @@ -1,125 +0,0 @@ -#!/usr/bin/env python3 -"""Graft ROCm/aiter #4417 into an aiter that predates it. - -#4417 "Fix large-token FlyDSL MoE launch and output limits" (merged 2026-07-30, -single file, +32/-2) adds two guards to aiter/ops/flydsl/moe_kernels.py: - - * requires_flydsl_stage2_reduce() -- stage2 buffer atomics address the output - with 32-bit byte offsets, so a >4 GiB output silently walks off the end. - The guard flips mode to "reduce" before that happens. - * resolve_flydsl_grid_y_persist_m() -- HIP caps grid.y at 65535; the guard - raises persist_m enough to keep the launch legal. - -Neither guard fires at the DSv4-Pro TP8 shape this campaign measures -- -requires_flydsl_stage2_reduce(65536, 7168, 2) is False (~939 MB, well under the -4 GiB threshold) -- so #4417 does NOT explain the inter_dim=384 profile-run -memfault. That fault is a stage2 tile_k mismatch: the tuned CSV names -opus_moe2_*_t64x256x256_* for inter_dim=384, and resolve_flydsl_stage2_tile_k -only guards flydsl_* kernel names, so tile_k=256 runs against K=384. #4417 is -grafted anyway because it is a genuine gap in the base that any larger-token -sweep row would hit. - -The upstream .diff does NOT apply to the 08-12 nightly base: that aiter predates -both #4417 and aiter's typing modernization, so every context line still reads -`Dict[str, Dict]` / `Optional[X]` where the diff expects `dict[str, dict]` / -`X | None`. The three hunks are grafted here by anchor instead of by context. -Idempotent: re-running on an already-grafted (or already-new) file is a no-op. -""" - -import sys - -HELPERS = ''' - -def resolve_flydsl_grid_y_persist_m( - num_m_blocks: int, requested_persist_m: int = 0 -) -> int: - """Increase persist_m as needed to keep grid.y within HIP's limit.""" - num_m_blocks = max(int(num_m_blocks), 0) - requested_persist_m = max(int(requested_persist_m), 1) - required_persist_m = max( - 1, (num_m_blocks + _HIP_MAX_GRID_DIM_Y - 1) // _HIP_MAX_GRID_DIM_Y - ) - return max(requested_persist_m, required_persist_m) - - -def requires_flydsl_stage2_reduce( - token_num: int, model_dim: int, element_size: int -) -> bool: - """Return whether stage2 atomic output exceeds 32-bit byte offsets.""" - return int(token_num) * int(model_dim) * int(element_size) > 0xFFFFFFFF - -''' - - -def graft(path): - src = open(path).read() - if "requires_flydsl_stage2_reduce" in src and "_HIP_MAX_GRID_DIM_Y" in src: - print("SKIP already grafted/present:", path) - return 0 - orig = src - done = [] - - # hunk 1a: the grid.y constant, right after the _KERNEL_PARAMS declaration. - a = "_KERNEL_PARAMS: Dict[str, Dict] = {}\n" - if a not in src: - a = "_KERNEL_PARAMS: dict[str, dict] = {}\n" - if src.count(a) != 1: - print("FAIL anchor _KERNEL_PARAMS count=", src.count(a)) - return 1 - src = src.replace(a, a + "\n# HIP limits grid.y/grid.z to 65535.\n" - "_HIP_MAX_GRID_DIM_Y = 65535\n", 1) - done.append("const") - - # hunk 1b: the two helpers, ahead of resolve_flydsl_stage2_tile_k. - b = "\ndef resolve_flydsl_stage2_tile_k(" - if src.count(b) != 1: - print("FAIL anchor stage2_tile_k count=", src.count(b)) - return 1 - src = src.replace(b, HELPERS + "\ndef resolve_flydsl_stage2_tile_k(", 1) - done.append("helpers") - - # hunk 2: stage1 caps grid.y through persist_m. - c = " _persist_m = persist_m if persist_m > 0 else 1\n" - if src.count(c) != 1: - print("FAIL anchor stage1 persist_m count=", src.count(c)) - return 1 - src = src.replace(c, " _persist_m = resolve_flydsl_grid_y_persist_m(_grid_y, persist_m)\n", 1) - done.append("stage1_persist") - - # hunk 3a: stage2 falls back to reduce when the atomic output exceeds 4 GiB. - d = (' if os.environ.get("AITER_FLYDSL_FORCE_REDUCE", "0") == "1":\n' - ' mode = "reduce"\n') - if src.count(d) != 1: - print("FAIL anchor FORCE_REDUCE count=", src.count(d)) - return 1 - src = src.replace(d, d + ( - " elif (\n" - " mode != \"reduce\"\n" - " and not return_per_slot\n" - " and requires_flydsl_stage2_reduce(token_num, model_dim, 2)\n" - " ):\n" - " # Buffer atomics use 32-bit offsets; reduce outputs larger than 4 GiB.\n" - " mode = \"reduce\"\n"), 1) - done.append("stage2_reduce") - - # hunk 3b: fp8 stage2 is non-persistent, so cap grid.y the same way. - e = ' if a_dtype == "fp8":\n _persist_m = 1\n' - if src.count(e) != 1: - print("FAIL anchor fp8 persist_m count=", src.count(e)) - return 1 - src = src.replace(e, ' if a_dtype == "fp8":\n' - ' # FP8 uses non-persistent scheduling, so cap grid.y via persist_m.\n' - ' _persist_m = resolve_flydsl_grid_y_persist_m(m_blocks)\n', 1) - done.append("stage2_fp8_persist") - - if src == orig: - print("FAIL no change") - return 1 - open(path + ".pre4417", "w").write(orig) - open(path, "w").write(src) - print("GRAFTED", path, "hunks:", ",".join(done)) - return 0 - - -if __name__ == "__main__": - sys.exit(graft(sys.argv[1])) diff --git a/docs/waiver/2589.md b/docs/waiver/2589.md index 444915fdb..df9892a20 100644 --- a/docs/waiver/2589.md +++ b/docs/waiver/2589.md @@ -12,37 +12,92 @@ prohibits unless covered by a filled-out waiver. - **Patch entrypoint:** `bash "$(dirname "$0")/apply_dsv4_container_patches.sh"` invoked from [`benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh`](../../benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm_mtp.sh) - **Patch script:** [`benchmarks/single_node/agentic/apply_dsv4_container_patches.sh`](../../benchmarks/single_node/agentic/apply_dsv4_container_patches.sh) - (helper: [`graft_aiter_4417.py`](../../benchmarks/single_node/agentic/graft_aiter_4417.py)) ## What is patched -`apply_dsv4_container_patches.sh` is **strictly additive and idempotent**: every step is marker-gated, and -nothing that ships in the base image is overwritten. Re-running it on an already-patched container is a -verified no-op. +Every change is sourced from an **upstream ref** — a vLLM PR `.diff`, or a pinned `ROCm/aiter` / +`ROCm/mori` commit. Nothing comes from a private or prebuilt measurement image. -| Upstream PR | What it changes | Notes | +The script is **idempotent**: every step is marker-gated (file-content markers for the PR diffs, byte +comparison for the file syncs), and re-running it on an already-patched container is a verified no-op — +confirmed by running it twice in a row on a container created from the pinned image, with the second run +reporting `same` / `already present` for all 21 steps and `VERIFY: ALL_OK`. + +| Upstream ref | What it changes | Notes | |---|---|---| -| vllm #51473 | native MXFP4 TP8 shard allocation (`vllm/model_executor/layers/fused_moe/oracle/mxfp4.py`) | **merged 2026-08-11, i.e. after this base.** Applied with `--include='vllm/*'`: the PR carries a `tests/` hunk with no counterpart in an installed wheel | -| vllm #51714 | opt-in AITER gluon sparse-MLA decode kernel for gfx950 (`vllm/v1/attention/ops/rocm_aiter_mla_sparse.py`) | open upstream. Dormant unless `VLLM_ROCM_DSV4_SPARSE_GLUON=1`; the call site is a lazy in-function import behind that env | -| vllm #51918 (**PARTIAL**) | `vllm/config/kernel.py` **only** — registers `flydsl_mega_moe` as an accepted `--moe-backend` value | model-side hunks deliberately NOT taken; see below | -| aiter #4417 | large-token FlyDSL MoE launch/output limits (`aiter/ops/flydsl/moe_kernels.py`) | merged upstream 2026-07-30, but the base's **vendored** aiter predates it and does not track aiter main. The upstream `.diff` does not apply — the base still reads `Dict[str, Dict]` where the diff's context expects `dict[str, dict]` (aiter's typing modernization postdates the vendored revision) — so the hunks are grafted by anchor via `graft_aiter_4417.py` | -| aiter gluon kernel | adds `aiter/ops/triton/_gluon_kernels/gfx950/attention/pa_decode_sparse.py` from `aiter@97d0c6e4` | the single file the base genuinely lacks. The other three `pa_decode_sparse` variants (`gfx1250/`, `_triton_kernels/`, and the `ops/triton/` facade) **ship in the base and are left untouched** — an earlier revision of this script copied all four, silently replacing base files with a cross-version transplant. The script now skips anything already present | +| vllm #51473 | native MXFP4 TP8 shard allocation (`vllm/model_executor/layers/fused_moe/oracle/mxfp4.py`) | **merged 2026-08-11, i.e. after this base.** Applied per-file: the PR carries a `tests/` hunk with no counterpart in an installed wheel | +| vllm #51714 | opt-in AITER gluon sparse-MLA decode for gfx950 (`vllm/v1/attention/ops/rocm_aiter_mla_sparse.py`) | open upstream. Dormant unless `VLLM_ROCM_DSV4_SPARSE_GLUON=1`. Requires the aiter facade below to be functional — see "The gluon facade" | +| vllm #51918 (**in full**, 6 files) | FlyDSL mega-MoE backend: `config/kernel.py` (backend name) plus `models/deepseek_v4/amd/{model,mtp,dspark,mega_moe_experts,mega_moe_runtime}.py` | open upstream. `model.py` needs `patch --fuzz=3`; the diff targets a tree a few days newer and the earlier hunks shift its line numbering. All 6 of its hunks land | +| `aiter@97d0c6e4` (7 files) | sparse gluon decode kernel **and its facade**; FSE (`fhmoe`); `ops/flydsl/moe_kernels.py` | see below | +| `mori@84a33cc` (7 files) | `mori/ir/flydsl/` (3 files) plus the `cov` cascade `ir/bitcode.py`, `jit/{cache,config,core}.py` | see below | + +### The gluon facade + +Copying only the gfx950 gluon kernel is **not** enough, and an earlier revision of this script made exactly +that mistake. The base's `aiter/ops/triton/attention/pa_decode_sparse.py` facade predates the +`extra_cache` / `extra_indices` / `extra_indptr` keywords that #51714's call site passes, so with +`VLLM_ROCM_DSV4_SPARSE_GLUON=1` every worker raised + +``` +TypeError: pa_decode_sparse() got an unexpected keyword argument 'extra_cache' +VLLM_ROCM_DSV4_SPARSE_GLUON: aiter gluon sparse decode failed; falling back to +the Triton path for the rest of this process. +``` + +on its first decode and latched onto the Triton fallback for the life of the process — i.e. the knob was +accepted, logged as on, and silently inert. The facade and the kernel it dispatches to are one unit and +move together. The script now asserts the accepted keywords by signature introspection, not by file +presence, precisely because file presence is what gave the false positive. + +### `mori.ir.flydsl` and the `cov` cascade + +MegaMoE's intranode dispatch/combine kernel does `import mori.ir.flydsl`, and the base ships mori without +that subpackage. It is three plain `.py` files — no compiled artifact. + +They cannot be dropped in alone. `mori/ir/flydsl/runtime.py` calls `find_bitcode(cov=6)` (FlyDSL needs +AMDGPU code-object ABI 600), and `cov` cascades through three more files the base predates — +`ir/bitcode.py`, `jit/cache.py`, `jit/core.py` (37 call sites upstream, **0** in the base) — plus +`jit/config.py` for `is_debuginfo_enabled`. Installing a subset yields, in order, +`TypeError: find_bitcode() got an unexpected keyword argument 'cov'` and then +`ImportError: cannot import name 'is_debuginfo_enabled'`. The whole cascade goes, or none of it does. + +`libmori_shmem_device.bc` is deliberately **not** shipped: it is a 485 KB LLVM IR build artifact. +`find_bitcode()` falls back to `mori.jit.core.ensure_bitcode()`, which compiles it in-container with hipcc +on first import (~1 min, logged as `[mori-jit] Compiling shmem device bitcode for gfx950 (nic=ionic, +cov=6)`). This keeps the patch stack free of binary payloads. + +### `moe_kernels.py` is replaced, not patched + +`aiter/ops/flydsl/moe_kernels.py` is the one file swapped wholesale (2409 → 3003 lines). FSE's +`aiter/ops/flydsl/fhmoe.py` imports `_flydsl_moe_stage1_impl` / `_flydsl_moe_stage2_impl`, which the base +simply does not define. Before replacing, base and upstream were symbol-diffed at the AST level: + +``` +base symbols: 58 upstream symbols: 66 +LOST if replaced: 0 +GAINED: 8 (incl. _flydsl_moe_stage1_impl, _flydsl_moe_stage2_impl) +``` + +Zero symbols lost, so no in-tree caller can break on the swap. + +### aiter #4417 is *not* carried separately -### Why #51918 is taken only partially +An earlier revision grafted aiter #4417 (large-token FlyDSL MoE launch/output limits) by anchor, via a +`graft_aiter_4417.py` helper. That helper is **gone** — both for a correctness reason and a scope reason: -Two independent reasons, either sufficient on its own: +- Its two guards cannot fire on this config. `requires_flydsl_stage2_reduce` first fires at 299594 tokens; + this recipe's ceiling is 65536 (`max_num_batched_tokens` 16384 × the MTP fan-out of 4), 4.6× below the + threshold. `resolve_flydsl_grid_y_persist_m` guards `grid.y > 65535`; at 65536 tokens `num_m_blocks` is + 8192 and it returns `persist_m=1`, i.e. inert, with 8× of headroom. The sweep grid varies concurrency + only, never `max_num_batched_tokens`. +- It is redundant anyway: the `aiter@97d0c6e4` `moe_kernels.py` installed above already contains both + guards (verified, `grep -c` = 5). -1. **The backend cannot run on this base at all.** MegaMoE's intranode kernel imports `mori.ir.flydsl`, - which this image does not have (verified: `ModuleNotFoundError`). Applying the model hunks would buy a - backend that raises on first use. -2. **The model hunks do not apply cleanly.** #51918 is written against a tree ~3 days newer than this - base; the two earlier hunks shift line numbering enough that the third fails at - `vllm/models/deepseek_v4/amd/model.py:300`. `git apply --3way` cannot rescue it because `site-packages` - is not a git repo, so there are no blobs to three-way against. +### What is *not* touched -Net effect: the TP8 arm is unaffected (it never selects this backend), and a DEP8 row is rejected at -config time with a clear error rather than failing deep inside a kernel. **DEP8/MegaMoE is explicitly out -of scope on this route**, as is FSE (aiter #4269 needs `aiter/fhmoe.py`, also absent from the base). +`aiter/jit/core.py` is left alone. The base already exposes `_set_current_hip_stream`, which the nightly +vLLM calls for `module_rmsnorm_quant`; an earlier wholesale post2-python overlay regressed it and crashed +rmsnorm_quant warmup. ## Why the unmodified upstream image cannot run this benchmark @@ -77,16 +132,20 @@ matching the pinned `synthetic_acceptance_length` of 2.49. - vLLM: https://github.com/vllm-project/vllm/pull/51473 (merged 2026-08-11), https://github.com/vllm-project/vllm/pull/51714 (open), - https://github.com/vllm-project/vllm/pull/51918 (open, partial) -- AITER: https://github.com/ROCm/aiter/pull/4417 (merged 2026-07-30) -- Gluon sparse-MLA decode kernel: `aiter@97d0c6e4cb7a0919c12291c7c7d560ad412f15c1` + https://github.com/vllm-project/vllm/pull/51918 (open) +- AITER: `aiter@97d0c6e4cb7a0919c12291c7c7d560ad412f15c1` — + https://github.com/ROCm/aiter/commit/97d0c6e4cb7a0919c12291c7c7d560ad412f15c1 + (includes https://github.com/ROCm/aiter/pull/4417, merged 2026-07-30) +- mori: `mori@84a33cc0f15f019c78c995728973b70ea3d10bb7` — + https://github.com/ROCm/mori/commit/84a33cc0f15f019c78c995728973b70ea3d10bb7 ## Removal plan Retire `apply_dsv4_container_patches.sh` and its invocation from `dsv4_fp4_mi355x_vllm_mtp.sh` once a ROCm -vLLM nightly satisfies both halves at once: it must carry the merged vllm #51473 and aiter #4417 fixes, -ship the gfx950 `pa_decode_sparse` gluon kernel, and **not** carry the profile-run bf16-GEMM memory fault -described above. At that point bump the pinned `image:` for `dsv4-fp4-mi355x-vllm-agentic-mtp` to that -nightly, drop the patch script, `graft_aiter_4417.py` and the `bash …/apply_dsv4_container_patches.sh` -call, and delete this waiver in the same PR. The vllm #51714 gluon knob and the #51918 backend -registration can be dropped as soon as those PRs land upstream, independently of the fault. +vLLM nightly satisfies all of: it carries merged vllm #51473, it ships the gfx950 `pa_decode_sparse` gluon +kernel **together with a facade that accepts the `extra_*` keywords**, its vendored aiter is at or past +`97d0c6e4` and its mori ships `mori/ir/flydsl/` with the `cov` cascade, and it does **not** carry the +profile-run bf16-GEMM memory fault described above. At that point bump the pinned `image:` for +`dsv4-fp4-mi355x-vllm-agentic-mtp` to that nightly, drop the patch script and the +`bash …/apply_dsv4_container_patches.sh` call, and delete this waiver in the same PR. The vllm #51714 and +#51918 hunks can be dropped as soon as those PRs land upstream, independently of the fault. diff --git a/perf-changelog.yaml b/perf-changelog.yaml index ef4ef8234..9f960569f 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -5926,7 +5926,10 @@ description: - "Pin the image to the 08-09 nightly (f8d03e77) and patch it before serving via benchmarks/single_node/agentic/apply_dsv4_container_patches.sh, so the config can actually start: the previously pinned nightly memory-faults during the profile run on this recipe" - "The fault is in the base, not in the patches -- it reproduces identically on a pristine 08-12 nightly (3ee2df30) with no patches applied at all: eight TP workers serialize on /tmp/aiter_configs/bf16_tuned_gemm.csv.lock and the worker holding the baton dies in the post-MoE bf16 GEMM" - - "Patch stack is strictly additive and idempotent: vllm #51473 (native MXFP4 TP8 shard allocation, merged 2026-08-11 i.e. after this base), vllm #51714 (opt-in AITER gluon sparse-MLA decode for gfx950), vllm #51918 config/kernel.py only (registers the flydsl_mega_moe backend name), aiter #4417 (large-token FlyDSL MoE launch/output limits, grafted by anchor), plus the one gluon kernel file the base lacks" - - "Nothing that ships in the base is overwritten: three of the four pa_decode_sparse variants already ship and are left untouched, aiter core is not modified, and the MegaMoE model-side hunks of #51918 are deliberately not applied because the base has no mori.ir.flydsl" + - "Patch stack is idempotent and entirely upstream-sourced: vllm #51473 (native MXFP4 TP8 shard allocation, merged 2026-08-11 i.e. after this base), vllm #51714 (opt-in AITER gluon sparse-MLA decode for gfx950), vllm #51918 in full across 6 files (the flydsl_mega_moe backend name plus the MegaMoE model side), 7 files from aiter@97d0c6e4 and 7 from mori@84a33cc" + - "The gluon knob was silently dead before this: copying only the gfx950 kernel left the base's older aiter pa_decode_sparse facade in place, which rejects the extra_cache/extra_indices/extra_indptr keywords #51714's call site passes, so every worker hit a TypeError on first decode and latched onto the Triton fallback for the life of the process. The facade now moves with the kernel, and the script asserts the accepted keywords by signature introspection rather than by file presence" + - "MegaMoE and FSE are now reachable on this base: mori/ir/flydsl (3 .py files) plus the cov cascade it needs (ir/bitcode.py, jit/{cache,config,core}.py -- 37 cov call sites upstream, 0 in the base), and aiter fhmoe plus the moe_kernels.py that defines its stage1/stage2 impls. moe_kernels.py is the one wholesale replacement, AST symbol-diffed first: 0 symbols lost, 8 gained" + - "aiter #4417 is no longer grafted separately -- it is already contained in the aiter@97d0c6e4 moe_kernels.py, and neither of its guards can fire on this recipe anyway (stage2 reduce first fires at 299594 tokens vs a 65536 ceiling; grid.y persist_m is inert with 8x headroom)" + - "aiter core is not modified: the base already exposes _set_current_hip_stream, which the nightly vLLM needs for module_rmsnorm_quant. No binary payloads are shipped -- libmori_shmem_device.bc is JIT-compiled in-container with hipcc on first import" - "Enable VLLM_ROCM_DSV4_SPARSE_GLUON=1 by default (overridable); the knob is read via os.environ rather than envs.py, so vLLM logs it as an unknown environment variable" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2589