[PowerX] use profiling-free counters for H200 Kimi-K3 / [PowerX] H200 Kimi-K3 使用无 profiling 计数器 - #3550
[PowerX] use profiling-free counters for H200 Kimi-K3 / [PowerX] H200 Kimi-K3 使用无 profiling 计数器#3550edwingao28 wants to merge 2 commits into
Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
将新增 changelog 条目关联到 PR #3550，保持历史内容不变。
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Beyond the reported pr-link finding, I also checked whether dropping DCGM_FI_PROF_* fields breaks samples-CSV parsing in multinode.py: gpu_util_pct/sm_active are only used under the optional SAMPLES_HEADER_V2 schema, with SAMPLES_HEADER (v1) as a valid fallback, so their absence is not a hard failure. I also looked at the stray trailing space before the comma in the new DCGM_FI_DEV_DEC_UTIL , line (line 15) — dcgm-exporter's CSV field loader trims whitespace around the field name, so this is a cosmetic inconsistency with the rest of the file, not a functional break.
Extended reasoning...
The diff swaps DCGM counters used for Kimi-K3 H200 power telemetry (new CSV, one command-line wiring change, a comment-only master-config touch, and an appended changelog entry); it touches no auth/crypto/permission code. I verified the confirmed pr-link:TBD finding's mechanism against validate_perf_changelog.py/prepare_perf_changelog_merge.py (PR_LINK_PLACEHOLDERS only accepts "XXX" or the XXX pull URL, so "TBD" is rejected) and independently checked two ruled-out candidates (sm_active/gpu_util_pct schema fallback in multinode.py, and dcgm-exporter's tolerance of the stray-space field name) — both check out as non-issues, leaving the pr-link problem as the one real blocker already flagged inline.
| - "Keep the stock exporter image, 100 ms watch interval, 1 s scrape interval, 2 s timeout, and strict power validation unchanged." | ||
| - "Kimi-K3 H200 latency 功耗遥测改用无 profiling 的 DCGM counters，避免同步恢复 profiling watch 导致采样卡顿；保留功耗和 GPU 利用率，省略 profiling 指标。" | ||
| - "保持原 exporter 镜像、100 ms watch 间隔、1 s 抓取间隔、2 s 超时与严格功耗校验不变。" | ||
| pr-link: TBD |
There was a problem hiding this comment.
🔴 Merging this PR via the documented reuse workflow (maintainer comments /use <run_id>) will fail because pr-link: TBD at perf-changelog.yaml:9016 is not a value the merge tooling accepts. prepare_perf_changelog_merge.py's canonicalize_appended_links() -> compare_entries() -> validate_added_pr_link() (inferencex-e2e/infx/workflows/validate_perf_changelog.py:139-148) only accepts the real PR URL or the placeholders in PR_LINK_PLACEHOLDERS = {'XXX', 'https://github.com/SemiAnalysisAI/InferenceX/pull/XXX'} (lines 22-25). 'TBD' matches neither, so validate_added_pr_link raises ChangelogValidationError and main() prints ERROR and returns 1, blocking canonicalization until a maintainer manually edits the entry. …
Why this was flagged
…Fix: use the 'XXX' placeholder (the documented convention) for pr-link, not free text like 'TBD', so newly appended entries round-trip through canonicalize_appended_links.
Trigger: a maintainer comments /use <run_id> on this PR (per its own checklist), which runs prepare_perf_changelog_merge.py main()'s 'canonicalize' subcommand. That calls canonicalize_appended_links (prepare_perf_changelog_merge.py:88) -> compare_entries -> validate_added_pr_link (validate_perf_changelog.py:139-148), requiring the appended entry's pr-link to equal the expected PR URL or be in PR_LINK_PLACEHOLDERS = {'XXX', '.../pull/XXX'} (validate_perf_changelog.py:22-25). perf-changelog.yaml:9016 sets pr-link: TBD, which is neither, so it raises ChangelogValidationError("new PR entry must use ... or an XXX placeholder; found 'TBD'"), and main() returns 1 (prepare_perf_changelog_merge.py:291-293). On the base branch this entry does not exist so the path never triggers; here it blocks the reuse-merge step until fixed by hand.
Verification: normal (blocks the documented reuse-merge path; not caught by PR CI). The appended entry at perf-changelog.yaml:9016 sets pr-link: TBD. The reuse-merge flow — which the PR's own checklist references (/use <run_id>) — calls merge_with_reuse.py:679 canonicalize_changelog(pr, repo) → :521 canonicalize_appended_links(base_raw, original, pr, repo) → prepare_perf_changelog_merge.py:88…
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=36800664055 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=36800664055 |
Drop the dcgm-counter-profile option, DCGM_COUNTER_PROFILE env plumbing, counter_profiles selector, receipts, docs and tests. Every srt-slurm recipe with a telemetry.dcgm_exporter block now sets the same command as the H200 Kimi-K3 recipe from #3550: dcgm-exporter --collect-interval=100 --address :{port} -f /configs/dcgm-counters-noprof.csv Changelog entries cover the 20 master-config keys that reference these recipes.
将新增 changelog 条目关联到 PR #3550，保持历史内容不变。
2ff047f to
dc92b73
Compare
为 H200 Kimi-K3 latency 功耗采集选择无 profiling 的 DCGM counters，保持官方 SRT pin、采样时序与严格校验不变。
0c6e2a1 to
2b6e55c
Compare
为 H200 Kimi-K3 latency AgentX 将 benchmark grace 提到 1800s，避免默认 30s 截断导致 ProfileMetricCoverage 卡在 ~94%。
This branch has not been deployed
1 participant
Description
Select profiling-free DCGM counters for H200 Kimi-K3 latency power sampling. This avoids profiling-watch repair on the exporter path; profiling metrics become unavailable. Stock image, sampling cadence and strict validation stay unchanged.
Testing: Native config/mount/launch checks passed; 7 cluster-config tests passed; changelog validation passed. Final sweep pending.
中文
H200 Kimi-K3 latency 功耗采集选择无 profiling 的 DCGM counters，避免 exporter 的 profiling-watch 修复路径。profiling 指标将不可用；stock 镜像、采样时序和严格校验保持不变。
**测试：**原生配置、挂载及启动命令检查通过；7 项 cluster-config 测试通过；changelog 校验通过。最终 sweep 待运行。
**AI 模型：**GPT-6 负责配置、验证和 PR 准备；原始实现使用的确切模型未核实。
**类型：**配置变更。关联 issue：无。
AI model disclosure
Related Issue
N/A
Type of Change
Checklist
inferencex-e2e/perf-changelog.yamland have not edited historical entriesOWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps. Remove and re-add the label to force one.