Conversation
…DEP8) Ports the agentX (llm-manifesto/agentx-mvp) DeepSeek-V4-Pro 1P-EP8/1D-EP8 K8s deployment to InferenceX's SLURM-based llmd-vllm framework on B200 (8 GPUs/node, so each DEP8 role fits on one node vs GB200's 4 GPUs/node needing two). New master-config key dsv4-fp4-b200-llmd-vllm-agentx and recipe carry the agentX EPP scheduling config (disagg-profile-handler's `deciders:` shape, prefix-cache-affinity-filter, token-load-scorer, etc., ported verbatim from a real agentx-mvp results snapshot) and per-role vLLM tuning, dropping the Kubernetes/manifesto-specific Mooncake KV chain in favor of server.sh's existing plain NixlConnector. Wires a self-contained FRAMEWORK==llmd-vllm branch into runners/launch_b200-dgxc.sh (ahead of the srtctl/srt-slurm path used by the other B200 multinode frameworks, which llm-d bypasses entirely), mirroring launch_gb200-nv.sh's pyxis/enroot + slurm_utils.sh helpers pattern. Pins a B200-only EPP/pd-sidecar v0.10.0 binary override (LLMD_BIN_DIR) for the newer EPP plugin shape, isolated from GB200's shared v0.9.0 binaries/path. Co-Authored-By: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
- dsv4-fp4-b200-llmd-vllm-agentx: drop c256 from the 1P-DEP8/1D-DEP8 conc-list per request. - extract-binaries.sh: support BINARIES_ENV_FILE to source an alternate pins file, so a cluster-scoped router-version bump doesn't require editing the shared binaries.env (whose EPP_FROM_IMAGE/ ROUTING_SIDECAR_IMAGE are plain assignments and can't be overridden via pre-exported env vars). - Add binaries-b200-v0.10.0.env: B200-only v0.10.0 EPP/pd-sidecar pins, linux/amd64 platform (B200 is x86_64, unlike GB200's arm64 default), dedicated LLMD_BIN_DIR. - launch_b200-dgxc.sh: fix the extraction command documented in the LLMD_BIN_DIR comment, which previously relied on the non-overridable env vars above and would have silently extracted v0.9.0 binaries. Co-authored-by: Cursor <cursoragent@cursor.com>
Sibling of the P/D disagg benchmark: one engine per topology does both
prefill and decode (1 node / 8 GPUs each), no P/D KV handoff.
- server.sh/job.slurm: add an aggregated mode (DECODE_NODES=0) - no
decode role, no pd-sidecar, no --kv_transfer_config (there is nothing
to hand KV off to), and the sole engine's rank 0 takes over the
EPP/Envoy/bench coordinator duties normally owned by the decode
leader. Also fixes the underlying DECODE_NODES=0 bugs: job.slurm's
out-of-bounds IPS[] read for DECODE_LEADER_IP, and server.sh emitting
a bogus empty-address decode-0 endpoint in endpoints.yaml.
- New recipes dsv4-fp4-b200-agentx-agg-{tp8,dep8}.yaml: per-role vLLM
tuning ported from agentX's ix-agg-base/agg-tp8 and
ix-agg-dep-base/agg-dep8, plus a minimal single-schedulingProfile EPP
config (no disagg-profile-handler needed - llm-d auto-selects
single-profile-handler for one profile). Needs no EPP router bump
since every plugin used is already in the default v0.9.0 binaries.
- New dsv4_fp4_b200_llmd-vllm-agg.sh wrapper + launch_b200-dgxc.sh now
branches its llmd-vllm wrapper-script choice on $DISAGG instead of
hardcoding the disagg wrapper.
- New dsv4-fp4-b200-llmd-vllm-agentx-agg master-config key (disagg:
false, prefill num-worker=1 / decode num-worker=0 per the
process_result.py / dynamo-vllm aggregated convention). conc-lists
mirror the agentX dspark historical runs (results_ilmarkov-2-ix-agg-
tp8-dspark-v3-long, results_ilmarkov-ix-agg-dep8-dspark-1308-long).
- README.md + perf-changelog.yaml updated.
Co-authored-by: Cursor <cursoragent@cursor.com>
|
Claude finished @ilmarkov's task in 13m 58s —— View job Review of PR #2719 — DeepSeek-V4-Pro llm-d B200 (AgentX)
SummaryFound 3 blocking and 2 warning issues (posted as inline comments):
Doc-consistency nits (no inline comments): many comments reference Positive notes: server.sh's aggregated-mode gating ( |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| if [[ "$registry" == "registry-1.docker.io" && "$repository" != */* ]]; then | ||
| repository="library/$repository" | ||
| fi | ||
| printf 'docker://%s#%s:%s\n' "$registry" "$repository" "$digest" |
There was a problem hiding this comment.
🟡 WARNING: The digest branch keeps the tag inside repository, producing a malformed enroot URI for repo:tag@sha256:... images.
Why it matters: For an image like the sibling GB200 llm-d pin quay.io/rh-ee-imarkov/llm-d-nokube-vllm:vllm0.26@sha256:a909..., image_without_digest still contains :vllm0.26, so this prints docker://quay.io#rh-ee-imarkov/llm-d-nokube-vllm:vllm0.26:sha256:a909... — exactly the tag@digest ambiguity this function exists to avoid (per its own header comment). It's latent today only because the new B200 image is tag-only; the moment this image gets digest-pinned like its GB200 counterpart, the import breaks.
Fix:
| if [[ "$registry" == "registry-1.docker.io" && "$repository" != */* ]]; then | |
| repository="library/$repository" | |
| fi | |
| printf 'docker://%s#%s:%s\n' "$registry" "$repository" "$digest" | |
| if [[ "$registry" == "registry-1.docker.io" && "$repository" != */* ]]; then | |
| repository="library/$repository" | |
| fi | |
| # tag@digest form: drop the tag; enroot wants registry#repo:digest. | |
| repository="${repository%%:*}" | |
| printf 'docker://%s#%s:%s\n' "$registry" "$repository" "$digest" |
| # benchmarks/llm-d/extract-binaries.sh | ||
| # before this recipe can actually run; job.slurm's pyxis mount loop is | ||
| # a no-op (falls back to the image's baked-in v0.9.0) until then. | ||
| export LLMD_BIN_DIR="${LLMD_BIN_DIR:-/home/sa-shared/llm-d-bins-v0.10.0}" |
There was a problem hiding this comment.
🟡 WARNING: The v0.10.0 LLMD_BIN_DIR override is applied to every B200 llmd-vllm run, including the aggregated configs that declare router: { version: "0.9.0" }.
Why it matters: This export happens before the DISAGG branch, and job.slurm's mount loop mounts whatever executables it finds there over the image's baked-in binaries. Once /home/sa-shared/llm-d-bins-v0.10.0 is populated (a prerequisite for the disagg recipe to work at all), the dsv4-fp4-b200-llmd-vllm-agentx-agg runs will silently execute the v0.10.0 EPP/pd-sidecar while their master-config metadata, the recipe comments, and the perf-changelog all state they use the default v0.9.0 binaries — the published router version won't match what actually ran.
Fix: Either gate the override on the disagg path (move the export LLMD_BIN_DIR=... inside the DISAGG == true branch, added after the wrapper-selection if), or update the agg key's router: version/comments to 0.10.0 so the metadata matches reality.
There was a problem hiding this comment.
Beyond the inline findings, I checked the ruled-out changelog entry more closely: the Mooncake entry's config-keys (dsv4-fp4-b200-llmd-vllm-agentx) does match its "disagg DEP8 arm" description, since that key names the disagg config (not the -agg aggregated one), so there's no additional config-keys mismatch there.
Extended reasoning...
This run's bug hunt already surfaced five confirmed, non-trivial issues (env-var export bug breaking SLURM submission, append-only violation and invalid PLACEHOLDER pr-links in the byte-sensitive perf-changelog.yaml, a misnamed gb200 prefix on B200-only recipe files, and a malformed enroot URI for tag@ digest images), which on their own are sufficient to require human attention and block approval. I independently re-verified the one ruled-out candidate (the Mooncake entry's config-keys pointing at the disagg key dsv4-fp4-b200-llmd-vllm-agentx versus its "disagg DEP8 arm" description) by reading the surrounding perf-changelog.yaml entries directly, and confirmed the key correctly refers to the disagg config (the aggregated variant is the separate -agg key), so there is no mismatch there. Given the confirmed findings already speak for themselves as inline comments, this defer note is limited to recording that one additional, distinct concern was checked and ruled out rather than restating the existing findings.
Findings marked 🟡 are optional suggestions and need no follow-up push.
| if [[ "$FRAMEWORK" == "llmd-vllm" ]]; then | ||
| # Self-contained: bypasses the srtctl/srt-slurm path entirely (the | ||
| # "Validate framework" guard and the dsv4-only-dynamo-vllm guard | ||
| # further below are srtctl-specific and are never reached from here). | ||
| # MODEL_PATH is already resolved above (the dsv4/fp4 branch); llm-d | ||
| # additionally needs MODEL_NAME (served-model-name), which this file | ||
| # doesn't otherwise set outside the srtctl block - reuse $MODEL (the | ||
| # master-config `model:` field), matching this file's own | ||
| # `export SERVED_MODEL_NAME=$MODEL` convention below. | ||
| if [[ ! ( "$MODEL_PREFIX" == "dsv4" && "$PRECISION" == "fp4" ) ]]; then | ||
| echo "Unsupported MODEL_PREFIX/PRECISION for llmd-vllm on B200: $MODEL_PREFIX/$PRECISION" >&2 | ||
| exit 1 | ||
| fi | ||
| export MODEL_NAME="$MODEL" |
There was a problem hiding this comment.
🔴 The new llmd-vllm block never exports SLURM_PARTITION/SLURM_ACCOUNT before handing off to bash "$BENCH_SCRIPT", which forks submit.sh (a separate process) whose check_env SLURM_ACCOUNT/check_env SLURM_PARTITION require them in the environment; every B200 llmd-vllm job submission fails immediately with "Error: SLURM_ACCOUNT not set". Fix: add export SLURM_PARTITION SLURM_ACCOUNT in the llmd-vllm block, mirroring the sibling tilert branch a few lines below (line 249) which does exactly this before its own subprocess handoff.
Extended reasoning...
SLURM_PARTITION/SLURM_ACCOUNT are set as plain (non-exported) shell vars at the top of the file (lines 5-6) and no workflow sets them as job-level env, so they exist only in this script's local scope. The llmd-vllm branch runs JOB_ID=$(bash "$BENCH_SCRIPT") which execs dsv4_fp4_b200_llmd-vllm-agg.sh/disagg.sh, which runs bash ./submit.sh ... - a brand-new process that only inherits exported vars. submit.sh's check_env SLURM_ACCOUNT / check_env SLURM_PARTITION (both required first) then exit 1 before sbatch ever runs. Contrast with the tilert branch immediately below, which does export SLURM_PARTITION SLURM_ACCOUNT right before its own exec bash "$TILERT_DISAGG" handoff - the same pattern the new block omits.
Verification: normal — the new llmd-vllm block forks a subprocess without exporting the SLURM credentials that subprocess requires. runners/launch_b200-nscale-compat.sh:5-6 set SLURM_PARTITION="${SLURM_PARTITION:-batch_1}" / SLURM_ACCOUNT="${SLURM_ACCOUNT:-benchmark}" as plain (non-exported) vars. The new llmd-vllm branch runs JOB_ID=$(bash "$BENCH_SCRIPT") which execs… | normal — new code…
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33643687916 |
Resolve perf-changelog.yaml by keeping upstream entries and the 2P/3P multiprefill AgentX sweep changelog. 将 upstream/main 合并入 imarkov/dsv4-pro-agentic，解决 perf-changelog.yaml 冲突：保留上游条目及 2P/3P multiprefill AgentX 扫描变更日志。 Co-authored-by: Cursor <cursoragent@cursor.com>
|
/stage-results 35633554748 |
|
@ilmarkov staged run 35633554748: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-09-21~r35633554748 This run remains available across future |
Signed-off-by: Markov Ilya <markovilya197@gmail.com>
Resolve server.sh and perf-changelog.yaml conflicts while keeping AgentX agentic.sh routing and upstream main changelog entries. 将 upstream/main 合并到 imarkov/dsv4-pro-agentic，并解决 server.sh 与 perf-changelog.yaml 冲突，同时保留 AgentX agentic.sh 路由与上游 changelog。
…erification Give multi-prefill 2P1D jobs a 10-minute coordinator prefill /health poll after decode bring-up, and turn on DSpark adaptive verification for aggregated DEP8 AgentX recipes. 将多预填充 2P1D 在 decode 就绪后的 prefill /health 轮询延长至 10 分钟，并为聚合 DEP8 AgentX 配方启用 DSpark 自适应验证。
Signed-off-by: Markov Ilya <markovilya197@gmail.com>
|
Sorry, over the weekend, there was 2 major refactors to clean up the technical debt accumalated over the past 11 months of moving at the speed of light. We don't see any major refactors in the forthseeable future besides cleaning up AMD multinode AgentX pile of bash. As much, due to the refactors, u would need to ask your agent to rebase from remote main@latest. Thank you in advance for ur understanding |
合并 upstream/main 到 imarkov/dsv4-pro-agentic；随 inferencex-e2e 重组迁移 GB200 llm-d AgentX 文件并解决 perf-changelog 与 server.sh 冲突。
Signed-off-by: Markov Ilya <markovilya197@gmail.com>
Signed-off-by: Markov Ilya <markovilya197@gmail.com>
Lower Mooncake gpu-memory-utilization to 0.87 after c52/c72 OOM, match aggregated DEP8 speculative decoding to disagg decode (3 tokens with adaptive verification), and extend 2P1D conc-list with 576. 将 Mooncake gpu-memory-utilization 降至 0.87 以修复 c52/c72 OOM， 聚合 DEP8 DSpark 与分离式 decode 对齐（3 token + 自适应验证）， 并为 2P1D 新增并发 576。
Signed-off-by: Markov Ilya <markovilya197@gmail.com>
…entic Signed-off-by: Markov Ilya <markovilya197@gmail.com>
|
Heads-up: #3576 (merged) replaced the bash launchers with a Python launcher, so this PR will conflict when you merge |
This branch has not been deployed
4 participants
Changes
cluster:gb200-nv: aggregated TP8/DEP8 on 2 nodes (8 GPUs), or 1P-DEP8/1D-DEP8 on 4 nodes (16 GPUs).Validation
Generated GB200 matrices have 2-node aggregate and 4-node disaggregated allocations. Live sweep validation is in progress; no successful GPU result is claimed yet.
Note
Medium Risk
Touches multi-node serving (Mooncake master, KV connectors, SLURM env contract) and benchmark invariants (golden AL, offload metadata); mostly benchmark infra but misconfiguration could fail jobs or skew metrics.
Overview
Adds DeepSeek-V4-Pro-0813 AgentX on GB200 via llm-d (
cluster:gb200-nv): master configs for aggregated TP8/DEP8 (2 nodes) and 1P-DEP8/1D-DEP8 disagg (4 nodes), plus GB200 DSpark recipes (prefix/token-load EPP, optional Mooncake DRAM offload).Runtime/orchestration changes:
DECODE_NODES=0aggregated mode (rank 0 runs EPP/Envoy/bench, no pd-sidecar).recipe.pyenforces AgentX offload metadata, injects DSpark golden AL for throughput, and renders Mooncake store config.server.shwires MultiConnector (NIXL ± Mooncake), longer KV lease (1800s) for Mooncake P/D, and routes AgentX throughagentic.sh.agentic.shscrapes per-node vLLM/metricsfrom discovery; Envoy now 404s/metricson the frontend to avoid duplicated counters. Thin GB200 wrappers,submit.shhardware→pyxis/docker mapping, andlaunch_gb200-nv.sh(0813 checkpoint, agg vs disagg script, AgentX log copy). AgentX uses image-bundled router v0.10.0 (no legacy binary mounts on pyxis).Reviewed by Cursor Bugbot for commit 36c6f4d. Bugbot is set up for automated code reviews on this repo. Configure here.