Skip to content

ROCMAI-62: expire stale TTFT, split shared-GPU power, derive power thresholds from the device limit - #812

Draft
volen-silo wants to merge 4 commits into
mainfrom
rocmai-62-telemetry-coverage
Draft

volen-silo wants to merge 4 commits into
mainfrom
rocmai-62-telemetry-coverage

Conversation

@volen-silo

Copy link
Copy Markdown
Collaborator

Implements ROCMAI-62.

Four gaps in dashboard telemetry, in four commits that each build on their own:

  1. TTFT/TPOT expire. The windowed average fell back to the histogram's lifetime average whenever a scrape saw no new requests, so an idle server kept showing its all-time latency forever. Only windows with new requests now produce a value; it is held (marked *, with the existing legend) for the observation validity window and then cleared. Reuses GenerationObservationTracker::observe_direct. Instance gains ttft_observation/tpot_observation (serde-default, omitted when unknown, so old recordings load and are never labelled fresh).
  2. Shared-GPU power attribution. Every instance was credited with the full board power of each GPU it touches, so two models on one GPU each claimed all of it. attribute_power_w splits a GPU's power among its occupants by per-container VRAM, falling back to an equal split when any occupant has no reading. Power on a GPU nobody occupies is attributed to no instance but still counts toward node power. Instance.power_w carries the share and feeds tok/W and the POWER column. Also fixes the registry view ignoring gpu_indices: instances from rocm serve always had empty gpu_ids, so they could never show power or tok/W.
  3. Assistant tool output. list_instances now reports ttft_ms/tpot_ms with their freshness and the attributed power_w, so the assistant cannot present a held TTFT as live.
  4. Power limit and thresholds. GpuMetrics.power_limit_w (serde-default; None = unknown, never 0) is read from amd-smi static on the existing 30 s system-info cadence. Warning colors are 70% / 93% of the device's own limit (where the old 525/700 W constants sat on a 750 W part). With no known limit the value is drawn uncolored instead of being judged against another part's TDP.

Not covered / caveats

  • No Gherkin scenario. This changes what the dashboard shows, but the dash scenarios need the Linux PTY lane; I could not run one. Coverage is at the daemon level instead: ttft_is_held_then_cleared_when_requests_stop and shared_gpu_power_is_split_and_limit_is_reported in crates/rocm-dash-daemon/tests/telemetry_contract.rs drive the real runner loop against a fake Prometheus endpoint and a fake amd-smi. I confirmed the TTFT test fails with the old lifetime-average fallback restored.
  • amd-smi static key unverified. The cap is read from limit.socket_power, then limit.max_power, per my understanding of the schema, and has not been checked against a real MI-series host. Parsing is defensive; an unrecognised shape yields no limit (uncolored), not a wrong color.
  • Windows has no limit source. ADLX exposes none for the iGPU I tried (gpu_power_range is unsupported), so power is uncolored there. Windows also has no instance registry, so items 1-3 do not apply there yet.
  • WSL. amd-smi cannot run on WSL (no amdgpu module), so there is no GPU power or limit there, and vLLM's platform detection fails on amdsmi. Not addressed here.
  • Depends on nothing, but overlaps feat(dash-tui): collect GPU/memory metrics in-process on Windows #715. Its winperf.rs and a few tests build GpuMetrics with explicit fields; whichever PR merges second needs power_limit_w: None, added to those literals. Draft until feat(dash-tui): collect GPU/memory metrics in-process on Windows #715 lands.
  • e2e-cucumber's two shared_runtime symlink tests fail on a Windows account without symlink privilege (os error 1314), independent of this change.

Verification

  • cargo test --workspace --all-targets (Windows): all pass except the two symlink tests above. cargo clippy --workspace --all-targets -- -D warnings and python scripts/smoke_local.py clean.
  • Dash crates, Linux (WSL, including the unix-only daemon tests): 301 passed.
  • Ran feat(dash-tui): collect GPU/memory metrics in-process on Windows #715's WinPerfCollector on a Windows host with a Radeon 8060S to confirm the new field flows through (power reported, limit None).

…erages (ROCMAI-62)

TTFT and TPOT fell back to the histogram's cumulative average whenever a
scrape window saw no new requests, so an idle server kept showing its
all-time latency indefinitely, with no freshness signal.

Only windows with new requests now produce a value. The existing
GenerationObservationTracker holds it (marked Held) for the observation
validity window and then clears it. Instance gains ttft_observation and
tpot_observation, serde-default and omitted when unknown, so legacy NDJSON
replays still load and are never labelled Fresh. The instances table marks
held latency with the shared `*` marker and legend.

Signed-off-by: Eugene Volen <eugene.volen@amd.com>
Every instance was credited with the full board power of each GPU it
touches, so two models on one GPU each claimed all of it: per-instance
POWER overstated and tok/W understated, and the rows no longer summed to
the board.

attribute_power_w splits each GPU's power across its occupants in
proportion to per-container VRAM (the per-process amd-smi attribution),
spreads a multi-GPU instance's VRAM evenly over its GPUs, and falls back to
an equal split when any occupant has no reading rather than guessing a
skew. Power on a GPU nobody occupies is attributed to no instance but still
counts toward node power. The function takes only snapshot data, so it adds
no amd-smi dependency and works the same for any source that fills GpuMetrics
and per-instance VRAM.

Instance gains power_w (the attributed share, serde-default). The runner and
demo use it for tokens_per_watt, and the instances table shows it in the
POWER column.

Signed-off-by: Eugene Volen <eugene.volen@amd.com>
…stant (ROCMAI-62)

The assistant's list_instances tool carried freshness only for gen_tps, so
it could present a held TTFT as live and had no per-instance power. Add
ttft_ms/tpot_ms with their freshness ("fresh", "held", or null when
unknown) and the attributed power_w, matching what the instances table
shows.

Signed-off-by: Eugene Volen <eugene.volen@amd.com>
…er limit (ROCMAI-62)

Board-power colors compared against fixed 525/700 W constants tuned for
MI355X-class parts, so the same reading looked alarming or fine depending
on which GPU produced it.

GpuMetrics gains power_limit_w (serde-default; None means unknown, never 0).
The daemon reads each GPU's cap from `amd-smi static` (the cap currently in
force, else the part's maximum) on the existing 30 s system-info cadence and
stamps it onto every tick's metrics; a failed fetch keeps the previous caps.
power_style now warns at 70% and goes critical at 93% of that limit, which
is where the old constants sat on a 750 W part. With no known limit the value
is drawn in the plain foreground color instead of being judged against some
other part's TDP. Platforms with no limit source (Windows today) therefore
render uncolored rather than wrongly colored.

The amd-smi `limit` schema is parsed defensively and an unrecognised shape
yields no limit; it has not yet been checked against a real MI-series host.

Signed-off-by: Eugene Volen <eugene.volen@amd.com>
@volen-silo
volen-silo force-pushed the rocmai-62-telemetry-coverage branch from 2cf4c39 to a6bbdbe Compare October 30, 2026 14:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant