[https://nvbugs/6422339][test] Unwaive 3 disagg-perf-sanity GB200 tests after #15961 revert#16220
[https://nvbugs/6422339][test] Unwaive 3 disagg-perf-sanity GB200 tests after #15961 revert#16220chenfeiz0326 wants to merge 1 commit into
Conversation
Remove waives for the 3 disagg_upload-gen_only-* GB200 tests that failed in L0_PostMerge NVIDIA#2824 with the "ctx/gen never become ready" hang: - disagg_upload-gen_only-gb200_deepseek-r1-fp4_128k8k_con64_ctx1_pp8_gen1_dep32_eplb0_mtp3_ccb-NIXL - disagg_upload-gen_only-gb200_deepseek-r1-fp4_1k1k_con1024_ctx1_dep4_gen1_dep8_eplb0_mtp0_ccb-NIXL - disagg_upload-gen_only-gb200_deepseek-v32-fp4_8k1k_con4096_ctx1_dep4_gen1_dep32_eplb256_mtp0_ccb-NIXL Root cause was PR NVIDIA#15654 (IPC HMAC key via file descriptor, commit 48fc753, merged 2026-07-03) re-landing the same fd-passing scheme that had already been reverted once. Under the disagg-perf-sanity launch chain (bash trtllm-llmapi-launch -> pytest -> subprocess.Popen(trtllm-serve, close_fds=True)), the numeric fd advertised via TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY_FD is not preserved, so os.read(fd) blocks forever instead of raising -- CTX/GEN workers silently hang after "TensorRT LLM inited" and the disagg server's /health polls never succeed, timing out at pytest's 3600s outer timeout. Details in nvbugs/6435109. PR NVIDIA#15961 (commit 0d97e9c, merged 2026-07-06 22:22 CST) reverted NVIDIA#15654. Build 2824's commit 4c5f3b0 (2026-07-06 13:16 CST) sits in the 3.5-day re-land window; every post-merge build after the revert should no longer hit this. This PR re-enables the three affected tests so that verification actually happens.
|
/bot run --extra-stage "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge, GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge, GB200-40_GPUs-10_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE8-GPU32-Post-Merge" |
📝 WalkthroughWalkthroughThe integration-test waiver list for GB200 disaggregated upload performance tests now skips fewer DeepSeek R1/V32 FP4 parameter combinations. ChangesGB200 waiver update
Estimated code review effort: 1 (Trivial) | ~3 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
PR_Github #58617 [ run ] triggered by Bot. Commit: |
|
PR_Github #58617 [ run ] completed with state
|
|
/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-10,GB200-40_GPUs-10_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE8-GPU32-Post-Merge-1" |
|
PR_Github #58644 [ run ] triggered by Bot. Commit: |
|
PR_Github #58644 [ run ] completed with state |
Summary
Unwaive the 3
disagg_upload-gen_only-*GB200 tests that failed inLLM/main/L0_PostMerge #2824under nvbugs/6422339 (also tracked in nvbugs/6435109 triage).Root cause: PR #15654 (
Pass IPC HMAC key through file descriptor, commit48fc7537ba, merged 2026-07-03) re-landed a scheme that had already been reverted once. Under the disagg-perf-sanity launch chain (bashtrtllm-llmapi-launch→ pytest →subprocess.Popen(trtllm-serve, close_fds=True)) the numeric fd advertised viaTLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY_FDis not preserved into the child, so_read_spawn_proxy_process_ipc_hmac_key_fd'sos.read(fd, 4096)blocks forever instead of raising — CTX/GEN workers silently hang afterTensorRT LLM initedand the disagg server's/healthpolls never succeed, timing out at pytest's 3600 s outer timeout.PR #15961 (commit
0d97e9c76f, merged 2026-07-06 22:22 CST) reverted #15654. Build 2824's baseline commit4c5f3b06d2(2026-07-06 13:16 CST) sat inside the 3.5-day re-land window, which is why these three stages failed there. Every post-merge build after the revert should no longer hit this, but the tests have been silently skipped since the auto-waive on 2026-07-07, so we have no verifying signal yet. This PR re-enables them.Related: nvbugs/6208457 (original), nvbugs/6244695 (first revert), nvbugs/6388787 (second revert #15961), nvbugs/6435109 (this-build triage), nvbugs/6422339 (auto-generated CI waive umbrella).
Test plan
/bot run --extra-stage "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge, GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge, GB200-40_GPUs-10_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE8-GPU32-Post-Merge"TensorRT LLM initedagain, that means the revert is not effective in ToT and the underlying HMAC-fd path needs to be re-investigated before this merges.🤖 Generated with Claude Code
Summary by CodeRabbit