Skip to content

[https://nvbugs/6422339][test] Unwaive 3 disagg-perf-sanity GB200 tests after #15961 revert#16220

Open
chenfeiz0326 wants to merge 1 commit into
NVIDIA:mainfrom
chenfeiz0326:user-chenfeiz/unwaive-6435109
Open

[https://nvbugs/6422339][test] Unwaive 3 disagg-perf-sanity GB200 tests after #15961 revert#16220
chenfeiz0326 wants to merge 1 commit into
NVIDIA:mainfrom
chenfeiz0326:user-chenfeiz/unwaive-6435109

Conversation

@chenfeiz0326

@chenfeiz0326 chenfeiz0326 commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

Unwaive the 3 disagg_upload-gen_only-* GB200 tests that failed in LLM/main/L0_PostMerge #2824 under nvbugs/6422339 (also tracked in nvbugs/6435109 triage).

Root cause: PR #15654 (Pass IPC HMAC key through file descriptor, commit 48fc7537ba, merged 2026-07-03) re-landed a scheme that had already been reverted once. Under the disagg-perf-sanity launch chain (bash trtllm-llmapi-launch → pytest → subprocess.Popen(trtllm-serve, close_fds=True)) the numeric fd advertised via TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY_FD is not preserved into the child, so _read_spawn_proxy_process_ipc_hmac_key_fd's os.read(fd, 4096) blocks forever instead of raising — CTX/GEN workers silently hang after TensorRT LLM inited and the disagg server's /health polls never succeed, timing out at pytest's 3600 s outer timeout.

PR #15961 (commit 0d97e9c76f, merged 2026-07-06 22:22 CST) reverted #15654. Build 2824's baseline commit 4c5f3b06d2 (2026-07-06 13:16 CST) sat inside the 3.5-day re-land window, which is why these three stages failed there. Every post-merge build after the revert should no longer hit this, but the tests have been silently skipped since the auto-waive on 2026-07-07, so we have no verifying signal yet. This PR re-enables them.

Related: nvbugs/6208457 (original), nvbugs/6244695 (first revert), nvbugs/6388787 (second revert #15961), nvbugs/6435109 (this-build triage), nvbugs/6422339 (auto-generated CI waive umbrella).

Test plan

  • /bot run --extra-stage "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge, GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge, GB200-40_GPUs-10_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE8-GPU32-Post-Merge"
  • All 3 stages pass; if any hangs at TensorRT LLM inited again, that means the revert is not effective in ToT and the underlying HMAC-fd path needs to be re-investigated before this merges.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Tests
    • Updated end-to-end performance test waivers for selected DeepSeek R1 and V32 FP4 variants.
    • Removed several unnecessary skips, allowing more parameter combinations to run during validation.

Remove waives for the 3 disagg_upload-gen_only-* GB200 tests that failed
in L0_PostMerge NVIDIA#2824 with the "ctx/gen never become ready" hang:

- disagg_upload-gen_only-gb200_deepseek-r1-fp4_128k8k_con64_ctx1_pp8_gen1_dep32_eplb0_mtp3_ccb-NIXL
- disagg_upload-gen_only-gb200_deepseek-r1-fp4_1k1k_con1024_ctx1_dep4_gen1_dep8_eplb0_mtp0_ccb-NIXL
- disagg_upload-gen_only-gb200_deepseek-v32-fp4_8k1k_con4096_ctx1_dep4_gen1_dep32_eplb256_mtp0_ccb-NIXL

Root cause was PR NVIDIA#15654 (IPC HMAC key via file descriptor, commit
48fc753, merged 2026-07-03) re-landing the same fd-passing scheme
that had already been reverted once. Under the disagg-perf-sanity
launch chain (bash trtllm-llmapi-launch -> pytest ->
subprocess.Popen(trtllm-serve, close_fds=True)), the numeric fd
advertised via TLLM_SPAWN_PROXY_PROCESS_IPC_HMAC_KEY_FD is not
preserved, so os.read(fd) blocks forever instead of raising -- CTX/GEN
workers silently hang after "TensorRT LLM inited" and the disagg
server's /health polls never succeed, timing out at pytest's 3600s
outer timeout. Details in nvbugs/6435109.

PR NVIDIA#15961 (commit 0d97e9c, merged 2026-07-06 22:22 CST) reverted
NVIDIA#15654. Build 2824's commit 4c5f3b0 (2026-07-06 13:16 CST) sits in
the 3.5-day re-land window; every post-merge build after the revert
should no longer hit this. This PR re-enables the three affected tests
so that verification actually happens.
@chenfeiz0326

Copy link
Copy Markdown
Collaborator Author

/bot run --extra-stage "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge, GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge, GB200-40_GPUs-10_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE8-GPU32-Post-Merge"

@coderabbitai

coderabbitai Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The integration-test waiver list for GB200 disaggregated upload performance tests now skips fewer DeepSeek R1/V32 FP4 parameter combinations.

Changes

GB200 waiver update

Layer / File(s) Summary
Narrow performance waiver entries
tests/integration/test_lists/waives.txt
The perf/test_perf_sanity.py::test_e2e waivers retain selected R1 and V32 FP4 combinations and remove several previously waived variants.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Possibly related PRs

Suggested reviewers: mzweilz, dpitman-nvda

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly states the main change and includes the NVBugs reference.
Description check ✅ Passed The PR description covers the summary and test plan, though it does not follow the full template headings or checklist.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #58617 [ run ] triggered by Bot. Commit: 04e1596 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #58617 [ run ] completed with state FAILURE. Commit: 04e1596
/LLM/main/L0_MergeRequest_PR pipeline #47207 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@chenfeiz0326

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2,GB200-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-10,GB200-40_GPUs-10_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE8-GPU32-Post-Merge-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #58644 [ run ] triggered by Bot. Commit: 04e1596 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #58644 [ run ] completed with state SUCCESS. Commit: 04e1596
/LLM/main/L0_MergeRequest_PR pipeline #47231 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants