Repository navigation
ROCMAI-62: expire stale TTFT, split shared-GPU power, derive power thresholds from the device limit - #812
Draft
volen-silo wants to merge 4 commits into
Draft
ROCMAI-62: expire stale TTFT, split shared-GPU power, derive power thresholds from the device limit#812volen-silo wants to merge 4 commits into
volen-silo wants to merge 4 commits into
Conversation
…erages (ROCMAI-62) TTFT and TPOT fell back to the histogram's cumulative average whenever a scrape window saw no new requests, so an idle server kept showing its all-time latency indefinitely, with no freshness signal. Only windows with new requests now produce a value. The existing GenerationObservationTracker holds it (marked Held) for the observation validity window and then clears it. Instance gains ttft_observation and tpot_observation, serde-default and omitted when unknown, so legacy NDJSON replays still load and are never labelled Fresh. The instances table marks held latency with the shared `*` marker and legend. Signed-off-by: Eugene Volen <eugene.volen@amd.com>
Every instance was credited with the full board power of each GPU it touches, so two models on one GPU each claimed all of it: per-instance POWER overstated and tok/W understated, and the rows no longer summed to the board. attribute_power_w splits each GPU's power across its occupants in proportion to per-container VRAM (the per-process amd-smi attribution), spreads a multi-GPU instance's VRAM evenly over its GPUs, and falls back to an equal split when any occupant has no reading rather than guessing a skew. Power on a GPU nobody occupies is attributed to no instance but still counts toward node power. The function takes only snapshot data, so it adds no amd-smi dependency and works the same for any source that fills GpuMetrics and per-instance VRAM. Instance gains power_w (the attributed share, serde-default). The runner and demo use it for tokens_per_watt, and the instances table shows it in the POWER column. Signed-off-by: Eugene Volen <eugene.volen@amd.com>
…stant (ROCMAI-62)
The assistant's list_instances tool carried freshness only for gen_tps, so
it could present a held TTFT as live and had no per-instance power. Add
ttft_ms/tpot_ms with their freshness ("fresh", "held", or null when
unknown) and the attributed power_w, matching what the instances table
shows.
Signed-off-by: Eugene Volen <eugene.volen@amd.com>
…er limit (ROCMAI-62) Board-power colors compared against fixed 525/700 W constants tuned for MI355X-class parts, so the same reading looked alarming or fine depending on which GPU produced it. GpuMetrics gains power_limit_w (serde-default; None means unknown, never 0). The daemon reads each GPU's cap from `amd-smi static` (the cap currently in force, else the part's maximum) on the existing 30 s system-info cadence and stamps it onto every tick's metrics; a failed fetch keeps the previous caps. power_style now warns at 70% and goes critical at 93% of that limit, which is where the old constants sat on a 750 W part. With no known limit the value is drawn in the plain foreground color instead of being judged against some other part's TDP. Platforms with no limit source (Windows today) therefore render uncolored rather than wrongly colored. The amd-smi `limit` schema is parsed defensively and an unrecognised shape yields no limit; it has not yet been checked against a real MI-series host. Signed-off-by: Eugene Volen <eugene.volen@amd.com>
volen-silo
force-pushed
the
rocmai-62-telemetry-coverage
branch
from
October 30, 2026 14:09
2cf4c39 to
a6bbdbe
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements ROCMAI-62.
Four gaps in dashboard telemetry, in four commits that each build on their own:
*, with the existing legend) for the observation validity window and then cleared. ReusesGenerationObservationTracker::observe_direct.Instancegainsttft_observation/tpot_observation(serde-default, omitted when unknown, so old recordings load and are never labelled fresh).attribute_power_wsplits a GPU's power among its occupants by per-container VRAM, falling back to an equal split when any occupant has no reading. Power on a GPU nobody occupies is attributed to no instance but still counts toward node power.Instance.power_wcarries the share and feeds tok/W and the POWER column. Also fixes the registry view ignoringgpu_indices: instances fromrocm servealways had emptygpu_ids, so they could never show power or tok/W.list_instancesnow reportsttft_ms/tpot_mswith their freshness and the attributedpower_w, so the assistant cannot present a held TTFT as live.GpuMetrics.power_limit_w(serde-default;None= unknown, never 0) is read fromamd-smi staticon the existing 30 s system-info cadence. Warning colors are 70% / 93% of the device's own limit (where the old 525/700 W constants sat on a 750 W part). With no known limit the value is drawn uncolored instead of being judged against another part's TDP.Not covered / caveats
ttft_is_held_then_cleared_when_requests_stopandshared_gpu_power_is_split_and_limit_is_reportedincrates/rocm-dash-daemon/tests/telemetry_contract.rsdrive the real runner loop against a fake Prometheus endpoint and a fakeamd-smi. I confirmed the TTFT test fails with the old lifetime-average fallback restored.amd-smi statickey unverified. The cap is read fromlimit.socket_power, thenlimit.max_power, per my understanding of the schema, and has not been checked against a real MI-series host. Parsing is defensive; an unrecognised shape yields no limit (uncolored), not a wrong color.gpu_power_rangeis unsupported), so power is uncolored there. Windows also has no instance registry, so items 1-3 do not apply there yet.amd-smicannot run on WSL (noamdgpumodule), so there is no GPU power or limit there, and vLLM's platform detection fails onamdsmi. Not addressed here.winperf.rsand a few tests buildGpuMetricswith explicit fields; whichever PR merges second needspower_limit_w: None,added to those literals. Draft until feat(dash-tui): collect GPU/memory metrics in-process on Windows #715 lands.e2e-cucumber's twoshared_runtimesymlink tests fail on a Windows account without symlink privilege (os error 1314), independent of this change.Verification
cargo test --workspace --all-targets(Windows): all pass except the two symlink tests above.cargo clippy --workspace --all-targets -- -D warningsandpython scripts/smoke_local.pyclean.WinPerfCollectoron a Windows host with a Radeon 8060S to confirm the new field flows through (power reported, limitNone).