Skip to content

feat(dash-tui): collect GPU/memory metrics in-process on Windows - #715

Open
juhovainio wants to merge 24 commits into
mainfrom
jira-ROCMAI-564
Open

juhovainio wants to merge 24 commits into
mainfrom
jira-ROCMAI-564

Conversation

@juhovainio

@juhovainio juhovainio commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Implements ROCMAI-564

Windows has no rocmdashd socket yet, so the TUI's #[cfg(windows)] client arm builds Snapshots in-process on a 1-second ticker instead of connecting anywhere: HostCollector (already cross-platform) for CPU/memory, and a new WinPerfCollector for GPU utilization/VRAM/temperature/power/clock. The unix arm is unchanged.

  • crates/rocm-dash-collectors/src/winperf.rs: new WinPerfCollector, reading the GPU Engine/GPU Adapter Memory OS performance counters via powershell.exe (the same WDDM/DXGK-backed source Task Manager's GPU tab uses — no AMD tool needed for utilization/VRAM), plus a best-effort adapter inventory (name/RAM/driver) for GpuSystemInfo. Temperature, power, and clock speed additionally use ADLX (AMD Device Library eXtra, via the adlx crate) when the AMD driver's bundled amdadlx64.dll is present, degrading to 0.0/0.0/None on any failure (no driver, non-AMD GPU, unsupported metric) so the snapshot never errors out.
  • crates/rocm-dash-collectors/Cargo.toml: adds adlx and windows-sys (Win32_System_LibraryLoader, for SetDefaultDllDirectories — see below) as Windows-only dependencies.
  • crates/rocm-dash-tui/src/client.rs: the #[cfg(windows)] connect_and_run drives HostCollector/WinPerfCollector and sends Connected like the unix arm, but then a stream of ClientMsg::HostGpuSnapshot rather than Event::Snapshot. This is deliberate, not an oversight: Event::Snapshot treats instances as authoritative and latches has_received_snapshot forever, but Windows has no instance-registry daemon, so its instances is always an empty stand-in — routing it through there would permanently blind the issue [Feature]: Stop serving a model #145 "quit while serving" confirmation prompt. See AppState::push_host_gpu_snapshot.
  • crates/rocm-dash-tui/src/ui/tabs/home.rs: fixes the hero VRAM percentage silently inflating (e.g. "80800.0%") when every GPU's vram_total_mb is 0 — guards on total > 0 like widgets.rs/dock.rs already do, instead of total.max(1). Also replaces the hero's old "Unknown GPU"/fake 0%/0°C/0W readings with "Detecting GPU…"/"—" placeholders while telemetry is genuinely pending (gpu_telemetry_pending) versus confirmed absent (gpu_data_unavailable) — this applies on every platform (not just Windows) and in --replay, since the daemon's own Linux snapshot can legitimately report zero GPUs or a disconnect/replay-end can wipe latest.
  • docs/manual-testing.md, docs/rocm-docs/install/installation.md: document the Windows dashboard's new live telemetry and the quit-while-serving prompt's Windows-specific behavior (no model name/Services-overlay hint there, since the dashboard never gets a serving-instance snapshot on Windows).
  • crates/rocm-dash-tui/Cargo.toml: adds the rocm-dash-collectors path dependency this requires.
  • xtask/src/crate_edges.rs: allowlists the new rocm-dash-tui -> rocm-dash-collectors first-party edge.

Several bugs surfaced by real-hardware testing (this machine, AMD Radeon 780M integrated GPU) and fixed in follow-up commits:

  • GPU Engine perf-counter instances are keyed per-process (pid_NNNN_luid_...) while GPU Adapter Memory instances are per-adapter (luid_..., no pid); the device-keying logic now strips the pid prefix so both aggregate onto the same real adapter instead of each process producing its own phantom near-zero device.
  • Windows can also surface multiple stale/phantom LUIDs for one physical adapter (observed: 3 LUIDs, 2 permanently zero-VRAM, for a single AMD Radeon 780M). When Win32_VideoController confirms there's exactly one real adapter, all LUIDs now collapse into one device (GPUs · 1 instead of GPUs · 3).
  • powershell.exe child-process spawn latency (not the Get-Counter query itself) was blowing the old 5s timeout on real hardware; bumped to 15s and collapsed the two independent metrics()/system_info() spawns into one snapshot() call so each tick only pays the spawn cost once.
  • WMI's Win32_VideoController.AdapterRAM is a 32-bit field that truncates (or reports 0) above ~4GiB; a WMI_ADAPTER_RAM_CAP_THRESHOLD heuristic now detects that and prefers ADLX's own total-VRAM query instead, decoupled from whether the perf-counter query itself succeeded, falling back to WMI when ADLX is unavailable.
  • Multi-engine GPUs were under-reporting utilization (max-across-engines was overwriting instead of accumulating); aggregation is now keyed per (device, engine instance): samples for the same engine instance are summed (per-process shares of that engine), and the max is taken across distinct engines, matching Task Manager.
  • SetDefaultDllDirectories's return value is now checked — ADLX's amdadlx64.dll load falls back to the default search order on failure rather than silently assuming the hardened search path took effect.
  • When WMI finds an adapter but the perf counters return nothing (locale mismatch between Get-Counter's English counter names and a non-English Windows locale is the known cause), a new warning surfaces through snapshot()'s Vec<String> return instead of silently showing zeros.

vram_total_mb is only filled in for the single-adapter case (the realistic integrated-APU target); multi-GPU hosts degrade to 0 rather than risk a wrong number, since correlating a perf-counter LUID to a specific Win32_VideoController/ADLX entry isn't reliable without vendor-specific APIs beyond index-based ordering.

Datapoints and sources

Datapoint Source Notes
GPU utilization % GPU Engine perf counter (Get-Counter via powershell.exe) sum per engine instance, max across engines, per adapter
VRAM used GPU Adapter Memory perf counter (dedicated usage) per-adapter
VRAM total ADLX total-VRAM query, falling back to Win32_VideoController.AdapterRAM (WMI) ADLX preferred (WMI's field is 32-bit and truncates/zeroes above ~4GiB); single-adapter hosts only, 0 otherwise
GPU count / identity Win32_VideoController (WMI) used to collapse phantom perf-counter LUIDs onto the real adapter(s)
Temperature ADLX PerformanceMonitoringServices::current_gpu_metrics().temperature() AMD GPUs only, via the driver's bundled amdadlx64.dll; 0.0 if unavailable
Power draw ADLX ...::current_gpu_metrics().power() same as above
Clock speed ADLX ...::current_gpu_metrics().clock_speed() same as above; None if unavailable
CPU / memory HostCollector (sysinfo, cross-platform) unchanged, not part of this PR's GPU work

Also updated README.md's platform-support table and live-dashboard description (and docs/rocm-docs/install/installation.md, see above) to reflect that Windows now gets GPU/CPU/memory telemetry (just not the serving-instance registry, which still needs the unix-socket daemon), and added a Linux e2e scenario (dash-37 in tests/e2e-cucumber/features/dash.feature) covering the no-GPU-telemetry case. The Windows collector path itself isn't covered by this repo's e2e harness — not for lack of a GPU Windows runner (the required "E2E tests (Strix Halo, Windows)" lane has one), but because the e2e-cucumber PTY driver (tui_driver.rs, portable-pty) is Linux-only for now: it compiles on Windows via ConPTY, but that path isn't yet promoted to a blocking contract, per the top-of-file comment in dash.feature.

Test plan

  • Build and run on real hardware (this machine, AMD Radeon 780M integrated GPU); confirmed the dashboard TUI shows live GPU utilization/VRAM/CPU/memory metrics without a rocmdashd daemon running, and iterated on the perf-counter keying/timeout/phantom-LUID/VRAM-total fixes above against live Get-Counter/ADLX output.
  • Verify ADLX temperature/power/clock readings against real hardware (this machine, AMD Radeon 780M; confirmed amdadlx64.dll present in the driver store and live temp/power readings in the TUI).
  • cargo test (workspace, Linux) — unit tests pass; no regressions.
  • cargo clippy --workspace --all-targets — clean.
  • cargo check/cargo clippy --target x86_64-pc-windows-gnu (cross-compiled) — validates the ADLX FFI code path compiles against the real adlx crate.

Follow-up

WinPerfCollector currently shells out to powershell.exe (Get-Counter/Get-CimInstance) per tick; process-spawn latency is why the timeout above is 15s. A native rewrite — wmi crate for the Win32_VideoController inventory query, raw Pdh* FFI for the GPU Engine/GPU Adapter Memory counters — would cut per-tick latency and drop the spawn-timeout handling entirely, at the cost of more unsafe/platform-specific code (mainly Pdh* buffer-size negotiation). Not done here since the current approach already works and is tested on real hardware; worth revisiting if tick latency becomes a problem in practice.

@juhovainio
juhovainio marked this pull request as ready for review October 21, 2026 14:09
@juhovainio
juhovainio requested a review from a team as a code owner October 21, 2026 14:09
rominf
rominf previously requested changes Oct 22, 2026

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 9bb9005

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Review — needs work

Full review of the whole change.
Implements ROCMAI-564's live GPU/memory metrics on Windows using a client-side collector. It does not cover the ticket's "Validate on Strix Halo" item, and the description's ADLX hardware check is still unticked.

All findings below are new (first round on this PR).

Blocking

  • On Windows, quitting with a model still serving no longer warns — crates/rocm-dash-tui/src/client.rs:211-219, crates/rocm-dash-tui/src/app/mod.rs:693, crates/rocm-dash-tui/src/app/mod.rs:1479

    • The new Windows arm sends a Snapshot every second with instances: Vec::new().
    • push_snapshot then sets has_received_snapshot = true and clears instances.
    • From that point has_live_instance_with stops consulting the live registry read (|| (!self.has_received_snapshot && snapshot.has_live_service)).
    • So about a second after rocm dash starts on Windows, pressing q with a Lemonade model serving exits without the prompt.
    • The issue #145 fallback was documented as authoritative "for the entire session" on Windows precisely because no snapshot ever arrived there. See the doc comment at app/mod.rs:651-657, apps/rocm/src/dash.rs:340-347 and docs/manual-testing.md:530-548.
    • The Serving view is also forced empty every tick.
    • Nothing tests the new arm, so nothing caught this.
    • Confidence 85 · logic · Fix: keep the registry fallback authoritative while the in-process source cannot enumerate instances (e.g. a snapshot or message flag that does not flip has_received_snapshot). Add a test that applies a Windows-style snapshot and asserts has_live_instance() stays true when the registry reports a live service.
  • VRAM total is capped near 4 GiB, so VRAM% can pass 100% on the target hardware — crates/rocm-dash-collectors/src/winperf.rs:45, crates/rocm-dash-collectors/src/winperf.rs:163

    • Win32_VideoController.AdapterRAM is a documented uint32, so vram_total_mb can never exceed about 4095.
    • The "Dedicated Usage" counter feeding vram_used_mb has no such cap.
    • A Strix Halo carve-out or any dGPU above 4 GiB therefore shows the wrong total and an inflated percentage.
    • The test fixtures use 17179869184 (16 GiB), a value this field cannot hold, so they hide the problem.
    • Confidence 85 · logic · Fix: take the total from ADLX (Gpu::total_vram) or the adapter's HardwareInformation.qwMemorySize registry value. Add a fixture at the 4 GiB cap.
  • ADLX temperature, power and clock are matched to devices by list position — crates/rocm-dash-collectors/src/winperf.rs:175, crates/rocm-dash-collectors/src/winperf.rs:221-229

    • adlx.get(i) pairs the i-th ADLX GPU with the i-th perf-counter LUID in first-seen order.
    • The ADLX list holds AMD GPUs only. The LUID list also holds non-AMD adapters, the Basic Render Driver and phantom LUIDs.
    • filter_map(... .ok()) drops a GPU whose metrics call fails, which shifts every later reading up one slot.
    • The result is AMD readings shown against the wrong device. That is the outcome the same file avoids for VRAM ("degrade to 0 instead of a wrong number"), and the doc comment at line 200 admits the two lists are not correlated.
    • Confidence 85 · logic · Fix: attach ADLX readings only when there is exactly one device and exactly one ADLX GPU, otherwise leave 0/None; or correlate by LUID or PCI id. Keep one slot per GPU (map, not filter_map).
  • The ADLX matching has no test, and its two helper tests pass with it removed — crates/rocm-dash-collectors/src/winperf.rs:151, crates/rocm-dash-collectors/src/winperf.rs:175, tests at crates/rocm-dash-collectors/src/winperf.rs:338-356

    • parse_metrics calls adlx_readings() internally, which is always empty off Windows. No test can supply readings, so neither adlx.first() nor adlx.get(i) is ever exercised.
    • Mutating apply_adlx(adlx.first()) to apply_adlx(None), or adlx.get(i) to get(0), left all 8 tests green (run).
    • It also makes the "pure parse" tests call real driver FFI on any Windows host that has the AMD driver.
    • Confidence 100 · logic · Fix: pass &[AdlxReading] into parse_metrics and test the matching, including multi-device cases.
  • ADLX is initialised and torn down every second, synchronously, on an async worker — crates/rocm-dash-collectors/src/winperf.rs:151, crates/rocm-dash-collectors/src/winperf.rs:213-231

    • snapshot() (async) runs parse_metrics → AdlxHelper::new() on every 1s tick. Each call loads amdadlx64.dll, initialises ADLX, lists GPUs, then terminates.
    • The crate itself marks this // TODO: This should be a singleton (adlx-0.0.0-alpha.1 src/helper.rs:14).
    • That blocking FFI runs on a tokio worker of the dashboard's multi-thread runtime.
    • Confidence 70 · architectural · Fix: create the helper once inside WinPerfCollector and query it via spawn_blocking. This combines with the injection fix in the previous finding.
  • Timed-out powershell.exe children are leaked — crates/rocm-dash-collectors/src/winperf.rs:66-74

    • timeout(RUN_TIMEOUT, Command::new(...).output()) is used without .kill_on_drop(true), and tokio's default is false.
    • When the 15s timeout fires, the future is dropped and the child keeps running.
    • The file's own comment says timeouts happened "consistently ... every tick" on real hardware, and the loop respawns immediately, so hung children can pile up.
    • Confidence 85 · logic · Fix: .kill_on_drop(true).
  • amdadlx64.dll is loaded by bare name every second (DLL search-order hijack) — crates/rocm-dash-collectors/src/winperf.rs:217

    • adlx calls libloading::Library::new("amdadlx64.dll"), which uses the default Windows search order, including PATH and the current directory.
    • On a host without the AMD driver (no copy in System32), a planted DLL in the directory rocm dash runs from would be loaded and executed.
    • Confidence 70 · architectural (security) · Fix: call ADLX only if %SystemRoot%\System32\amdadlx64.dll exists, and pre-load it with LOAD_LIBRARY_SEARCH_SYSTEM32 (keeping the handle alive) so the crate's load by name reuses that module.
  • Docs and comments still say Windows has no live dashboard (AGENTS.md §5) — README.md:34, README.md:816, docs/rocm-docs/install/installation.md:18, docs/manual-testing.md:530-548, crates/rocm-dash-tui/src/client.rs:9-10, crates/rocm-dash-tui/src/app/mod.rs:651-657, apps/rocm/src/dash.rs:340-347

    • These say, respectively: "no live dashboard"; "Live mode requires Unix domain sockets (Linux and WSL only)"; live telemetry requires Linux or WSL2; the dashboard's daemon client "never connects (Windows, for the whole session)"; "the connection is a #[cfg(unix)] stub"; and, in the last two, that Windows never receives a snapshot.
    • The code now does the opposite in each case.
    • Confidence 85 · mechanical · Fix: update them in this PR, worded to match however the first finding is resolved.
  • No scenario or stated justification for the new Windows dashboard behaviour (AGENTS.md §3) — tests/e2e-cucumber/features/dash.feature

    • rocm dash on Windows now shows live metrics instead of a disconnected view: user-observable behaviour.
    • Every dash scenario is @requires-os:linux, and no test of any kind runs the new connect_and_run Windows arm.
    • Confidence 85 · mechanical · Fix: add a scenario, or say in the PR text why there is none and which lane would run it. At minimum add a unit test of the snapshot assembly; the Windows lane runs tests.
  • The new home.rs test does not cover the sparkline fix, and its 0.0% check is vacuous — crates/rocm-dash-tui/src/ui/tabs/home.rs:374-382, crates/rocm-dash-tui/src/ui/tabs/home.rs:692-715

    • Verdict 1 (fix fully reverted): the test fails, but only on !contains("80800"). contains("0.0%") is satisfied by other text on the screen.
    • Verdict 2 (one branch broken alone): reverting only the history-sparkline hunk to .max(1) leaves the test green (run).
    • Confidence 100 · logic · Fix: assert the hero tile's VRAM value and the sparkline input directly.
  • Two winperf tests don't test what their names claim — crates/rocm-dash-collectors/src/winperf.rs:296-308, crates/rocm-dash-collectors/src/winperf.rs:329-333

    • Verdicts for the tests not already covered above:
      • multiple_processes_on_one_adapter_collapse_to_one_device still passes with the pid-prefix strip in device_key removed (run); the single-adapter collapse rescues it. Only multiple_adapters_leave_vram_total_zero catches that mutation.
      • empty_output_yields_no_metrics still passes with the order.is_empty() guard removed (run).
      • Summing utilization instead of taking the max, and taking the max of VRAM instead of the sum, both survive (run).
    • The remaining tests fail on revert of the logic they target.
    • Confidence 100 · logic · Fix: give the pid-collapse test two adapters so only device_key can merge the entries. Add a fixture with GPU lines but no counter lines, and fixtures where the max-vs-sum choice changes the result.

Non-blocking

  • CPU/memory readings stall behind the PowerShell call — crates/rocm-dash-tui/src/client.rs:205-215

    • host.tick() waits on gpu.snapshot() (up to 15s).
    • The ticker keeps the default MissedTickBehavior::Burst, though the daemon it claims to mirror sets Skip (crates/rocm-dash-daemon/src/runner.rs:300).
    • On slow-spawn hosts this means a powershell.exe running almost all the time, which skews the CPU figure being displayed.
    • Confidence 70 · logic · Fix: collect GPU metrics on their own task and use Skip/Delay.
  • English counter paths fail silently on localised Windows — crates/rocm-dash-collectors/src/winperf.rs:41-44

    • Windows PowerShell 5.1's Get-Counter expects localised counter names. With SilentlyContinue, a non-English install yields an empty GPU list with no warning.
    • Confidence 70 · logic · Fix: warn when inventory lines are present but no counter lines are.
  • Virtual or non-AMD display adapters break the single-adapter path — crates/rocm-dash-collectors/src/winperf.rs:45, crates/rocm-dash-collectors/src/winperf.rs:154, crates/rocm-dash-collectors/src/winperf.rs:264

    • Any extra Win32_VideoController entry (a remote or virtual display driver, a second GPU) switches to the multi-adapter path, which brings back the phantom LUIDs.
    • gpu_model is taken from whichever adapter is listed first.
    • Confidence 70 · logic · Fix: filter adapters to PCI devices before counting.
  • connect is silently ignored on Windows — crates/rocm-dash-tui/src/client.rs:188-200

    • The configured value is only logged, while the header keeps showing the configured unix-socket address.
    • Confidence 70 · logic · Fix: label the source as in-process, or warn when a non-default connect is set.
  • Stray marker and a wrong API name in a comment — crates/rocm-dash-collectors/src/winperf.rs:113-116

    • There is a leftover "ponytail:" marker.
    • The comment names MSPower_DeviceEnable (a WMI power-management class) as the LUID-correlation path; DXGI is the relevant one.
    • Confidence 85 · mechanical · Fix: reword.

Decisions for the author

  • In-process client-side collection vs a Windows daemon transport — tradeoff

    • In-process gets live metrics shipped without IPC work.
    • The cost is a second, divergent snapshot source with no instances. The first blocking finding is the first consequence of that, and the ticket's research left the transport question open.
  • Spawning PowerShell every second vs calling PDH natively — non-blocking-improvement

    • Querying the same counters through PDH (PdhAddEnglishCounter) in-process would remove the spawn latency, the 15s timeout, the leaked children and the localisation issue in one change.
  • Depending on adlx 0.0.0-alpha.1 — tradeoff

    • It is cheap access to ADLX.
    • But the crate's own Drop eprintln!s over the TUI when terminate fails, its GPU iterator unwrap()s, and it marks its own helper as a TODO singleton. The comment at winperf.rs:211 ("fails cleanly (Err, not a panic)") is therefore only partly true.

Positive signals

  • The fixtures are real-world captures (the 780M three-LUID case, per-process pid_ instances), which pin the actual Windows counter shapes rather than idealised ones.
  • snapshot() folds metrics and system info into one spawn instead of two per tick.

Deployment notes

  • New third-party dependencies, Windows-target only: adlx 0.0.0-alpha.1 (MIT) and libloading 0.8.9 (ISC). Both licences are accepted in about.toml, and the manifest and notices are regenerated.
  • A new first-party edge, rocm-dash-tui -> rocm-dash-collectors, is allowlisted; it is unconditional, not Windows-only. cargo xtask check-crate-edges exits 0.

What this covered

  • What was read: every file in the change, completely, against 32834b2946ee…9bb9005e60ec. Also read: callers of connect_and_run up to apps/rocm/src/dash.rs; AppState's snapshot and quit-guard logic; sibling collectors; every vram_total_mb consumer in the TUI (the other divisions are already guarded, so no unfixed twin exists); the adlx crate source.
  • Two independent workers split the code passes: collector plus dependencies, and TUI wiring plus the instruction, history and comment passes.
  • Ran:
    • cargo test -p rocm-dash-collectors winperf (8 tests) and cargo test -p rocm-dash-tui hero_vram_pct (1 test), each confirmed to have run, plus the mutations cited above.
    • A Windows-target cargo check/clippy of rocm-dash-collectors, shown to be live with an injected type error.
    • cargo xtask check-crate-edges.
  • CI at the reviewed head is all green, including windows-build-and-test.
  • Did not run:
    • A Windows-target check of rocm-dash-tui (a C build script needs mingw, which is absent). The new arm was type-checked verbatim in a scratch build inside the review checkout instead.
    • Any execution on Windows or AMD hardware.
    • The prior-changes pass: earlier review comments on these files were out of reach under this run's constraints.
  • Not reconciled against any discussion on the PR.

@juhovainio

Copy link
Copy Markdown
Collaborator Author

Verified the ADLX temp/power/clock integration against real hardware (commit 9bb9005e, this machine, AMD Radeon 780M integrated GPU). Confirmed amdadlx64.dll is present via the driver's bundled copy (C:\Windows\System32\amdadlx64.dll and 4 driver-store copies), and the dashboard TUI shows live, sane values with no rocmdashd running:

  • GPUs · 1 (phantom-LUID collapse working, not inflated to 3)
  • VRAM 86.7% (sane percentage, not the old 80800.0%-style garbage)
  • TEMP 66.0°C
  • POWER 28.0 W

All four of this PR's fixes (pid-prefix stripping, LUID collapse, VRAM% guard, ADLX wiring) confirmed working end to end on physical hardware.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved all outstanding findings, re-verified:

Blocking:

  1. ADLX-to-device matching no longer trusts index-based correlation on multi-adapter hosts — temp/power/clock now only trusted when exactly one real adapter + one ADLX reading exist (same guard vram_total_mb already used); multi-adapter hosts degrade every slot to defaults instead of risking misattribution.
  2. Strengthened the matching tests: temp/power/clock assertions added to the existing VRAM-fallback tests, plus a new test proving multi-adapter + non-empty ADLX readings still degrade to defaults.
  3. adlx_readings() now runs inside tokio::task::spawn_blocking — FFI/driver calls no longer block a tokio worker thread. Full singleton caching across ticks is a reasonable follow-up, not done here (re-init cost is one AdlxHelper::new() per 1s tick; not shown to matter in practice).
  4. run_script's Command now sets .kill_on_drop(true) — a dropped child on timeout() firing actually kills the orphaned powershell.exe instead of leaking it.
  5. Mitigated the amdadlx64.dll DLL search-order hijack (CWE-427): SetDefaultDllDirectories(LOAD_LIBRARY_SEARCH_SYSTEM32) called once process-wide (via windows-sys, already a workspace dependency) before the first AdlxHelper::new(), restricting the vendored adlx crate's unflagged libloading load to System32 only.
  6. Updated all stale "Windows has no live dashboard" docs/comments (README, installation.md, manual-testing.md, client.rs, app/mod.rs, dash.rs) to reflect that Windows now streams live GPU/CPU/memory, but the serving-instance registry view stays empty by design (no rocmdashd unix socket there).
  7. Windows e2e coverage: not added in this PR. The e2e-cucumber dash feature is explicitly Linux-only pending a ConPTY/Windows-PTY driver path in tui_driver.rs that isn't yet promoted to a blocking contract — adding a real Windows scenario is PTY-infrastructure work, not step-definition reuse, and risks the same timing flakiness already seen on this runner (see prior quit-confirm flakes). Flagging as a follow-up rather than rushing fragile coverage into this PR.
  8. Replaced the vacuous home.rs VRAM test (rendered the whole UI, string-matched output, never populated state.history) with a direct unit test of the extracted vram_used_pct helper. Strengthened winperf.rs tests: isolated pid-prefix stripping from the single-adapter collapse rescue, added an order.is_empty() guard test, added a sum-vs-max VRAM fixture with two distinct non-zero values, and fixed SAMPLE's AdapterRAM to a realistic (32-bit-representable) value.
  9. Non-blocking cleanups: ticker now sets MissedTickBehavior::Skip (matches the daemon's existing choice); fixed the stray comment's wrong API name; connect param's log-only use left as-is (documents intent, not worth restructuring). Locale-dependent Get-Counter silent failures and non-AMD/virtual-adapter edge cases remain known, lower-priority follow-ups.

Re-verified: cargo test -p rocm-dash-collectors -p rocm-dash-tui, cargo clippy --workspace --all-targets -- -D warnings, and cargo check/clippy --target x86_64-pc-windows-gnu (Windows-cfg-gated code doesn't compile-check otherwise) — all clean.

@juhovainio
juhovainio dismissed rominf’s stale review October 22, 2026 11:33

Dismissing: all 9 blocking + 5 non-blocking findings from this review are resolved and verified as of commit 87610a3 (see PR comment). Re-requesting review on current HEAD.

@juhovainio
juhovainio requested a review from rominf October 22, 2026 11:33
rominf
rominf previously requested changes Oct 22, 2026

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · fdea558

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Review — needs work

Full review of the whole change.
Implements ROCMAI-564 ("Device metrics for TUI (GPU/memory) on Windows"). It covers the GPU/memory source and its wiring into the dashboard. The ticket also asks for validation on Strix Halo / Windows HIP-SDK hosts. The PR reports testing only on a Radeon 780M, and this run's Strix Halo Windows lane failed before it ran any scenario.

Round 2. Each finding is marked new (first raised this round) or standing (raised at 9bb9005 and still holding). How the earlier round's objections were settled is under Standing objections from the previous round, at the end.

Blocking

  • [new] The hero card shows "Detecting GPU…" / "—" forever, on Linux as well as Windows — crates/rocm-dash-tui/src/ui/tabs/home.rs:76-78, :139-141, :289-296, :394-410

    • gpu_telemetry_pending is latest.is_none_or(|s| s.gpus.is_empty()), so it cannot tell "nothing has arrived yet" from "a snapshot arrived with no GPUs".
    • The Linux daemon sends gpus: Vec::new() on every tick once gpu_init_done is set and amd-smi is unavailable. It sends the same when a metrics call fails (crates/rocm-dash-daemon/src/runner.rs:467-505), alongside an "amd-smi unavailable" warning. So the hero shows a loading state that never ends, and it contradicts the warning badge. Before this change it read "Unknown GPU".
    • A model already known from gpu_system_info is also hidden.
    • On Windows, a collector error or empty counter output (gpu_inventory_without_counter_samples_yields_no_metrics) gives the same permanent "Detecting…".
    • Confidence 85 · logic · Fix: treat only latest.is_none() as pending. Once any snapshot has arrived with no GPUs, show the known model, "Unknown GPU", or an "unavailable" state.
  • [new] Windows GPU collection failures never reach the screen — crates/rocm-dash-tui/src/client.rs:224-236

    • On error the arm only calls warn! (to the log), then sends warnings: Vec::new() and gpu_system_info: Some(GpuSystemInfo::default()). That drops the adapter inventory as well.
    • The daemon is the other implementation of this tick, and the Linux dashboard reads warnings from it. It pushes "amd-smi metric: {e}" into snapshot.warnings (runner.rs:472), which feeds the header badge and the Hardware warnings box (ui/tabs/hardware.rs:230).
    • On Windows the user sees "no GPUs" / "Detecting GPU…" with no reason given. That includes the 15 s PowerShell timeout the code itself reports hitting on real hardware.
    • Confidence 85 · logic · Fix: push the error into snapshot.warnings, as the daemon does.
  • [standing, raised as non-blocking at 9bb9005] GPU counters are queried by English counter names, which silently return nothing on non-English Windows — crates/rocm-dash-collectors/src/winperf.rs:41-45

    • Get-Counter '\GPU Engine(*)\Utilization Percentage' and '\GPU Adapter Memory(*)\Dedicated Usage' must use localized counter paths on a localized OS. The script runs under $ErrorActionPreference = 'SilentlyContinue' and -ErrorAction SilentlyContinue, so the error is swallowed.
    • The test gpu_inventory_without_counter_samples_yields_no_metrics names this case ("a locale where Get-Counter's -ErrorAction silently swallows a localized-path mismatch"). The test only checks that no phantom device appears.
    • Combined with the two findings above, the result is an empty GPU panel with no warning. Meanwhile README:34 says the Windows dashboard "shows GPU/CPU/memory".
    • Argued from the code and known Get-Counter behaviour; not run on a localized host.
    • Confidence 70 · logic · Fix: use PdhAddEnglishCounterW. windows-sys 0.61 is already a dependency and has the Win32_System_Performance feature. Alternatively resolve counter indices, and at minimum surface the empty result as a warning.
  • [new] GPU utilization under-reports when more than one process uses the GPU — crates/rocm-dash-collectors/src/winperf.rs:97-100, :152-158, :110; test :386-402

    • GPU Engine instances are one per process per engine. device_key drops both the pid and the engine, and the UTIL arm then takes the max over every instance.
    • Task Manager instead sums processes per engine, then takes the max across engines. The doc comment at :110 claims this code produces "the same 'overall GPU' reading Task Manager shows". The comment is the right target and the code is what disagrees with it.
    • Two processes at 40% on the 3D engine therefore show 40%, not 80%.
    • multiple_processes_on_one_adapter_collapse_to_one_device asserts 0.682, the max of 0.682 and 0.514, so the test locks the under-count in.
    • Confidence 85 · logic · Fix: key UTIL by (LUID, engine), sum across pids, take the max over engines, and update that test's expected value.
  • [standing, narrowed] The README still says the live dashboard needs Linux/WSL, and the published docs now contradict themselves — README.md:41, README.md:205

    • Line 41: "Live dashboard telemetry requires Linux or WSL2". This is the same sentence the PR rewrote in docs/rocm-docs/install/installation.md:18.
    • Line 205: "Live mode reads telemetry over a Unix domain socket, so it requires Linux or WSL".
    • Line 205 is pulled into docs/rocm-docs/getting-started.md via {include} (start-after "## Interactive interfaces"). The new installation.md sentence links to exactly that section.
    • Both lines contradict README:34 and :816-818 as edited by this PR. AGENTS.md §5 requires these claims to be kept in sync in the same change.
    • Confidence 100 (grep) · mechanical · Fix: update both lines to match line 34.
  • [new, with standing winperf test gaps] No test covers the two new behaviours that matter — crates/rocm-dash-tui/src/app/mod.rs:1504-1511; home.rs label branches at :292-296, :394-410

    • A mutation run made push_host_gpu_snapshot set has_received_snapshot = true and clear instances. That is the issue #145 quit-confirm regression its own doc comment warns about.
    • The same run forced every hero pending label condition to false.
    • All 930 rocm-dash-tui tests still passed.
    • Test verdicts:
      • The new home tests cover only the helpers (gpu_telemetry_pending, vram_used_pct, node_throughput_title). They would fail if the change were reverted, but a single broken label branch or a broken snapshot routing survives.
      • The new winperf tests are in a new file, so a revert removes them too. A single-branch break that survives:
        • either .min(100.0) clamp (no fixture is above 100);
        • the total_vram_mb: None → WMI fallback at :190 (no fixture has a single reading with None);
        • device_key on an instance with no luid_.
    • Confidence 100 · logic · Fix: add an AppState test that push_host_gpu_snapshot leaves instances and has_received_snapshot untouched, a render test for the "—" labels, and fixtures for the three winperf branches.
  • [standing] No e2e scenario covers the user-visible changes, and the PR text does not explain why — tests/e2e-cucumber/features/dash.feature

    • The change alters what users see in two places:
      • rocm dash live mode on Windows now runs instead of erroring;
      • Linux/WSL now show the "Detecting GPU…" and "—" placeholders.
    • All 36 dash scenarios are @requires-os:linux / WSL.
    • AGENTS.md §3 requires a scenario. Where one can only run on a gated lane, or cannot run (the feature header says Windows ConPTY is not promoted), the PR text must say so and name the lane. The description does neither.
    • The Linux placeholder state can be exercised on the existing PTY lanes.
    • Confidence 85 · mechanical · Fix: add a Linux scenario for the hero's pending/unavailable state, and state the Windows gap and its lane in the PR text.
  • [new] The description and a doc comment say the Windows arm sends Event::Snapshot, and the description omits several changes — PR body; crates/rocm-dash-tui/src/client.rs:193-198

    • The PR body says the Windows arm "sends the same ClientMsg sequence (Connected then Event::Snapshot)… so the app loop doesn't need to know which platform it's on". The doc comment on connect_and_run says the same.
    • The code sends the new ClientMsg::HostGpuSnapshot precisely so the app loop treats it differently. It also adds handling in event_loop.rs:1044 and AppState::push_host_gpu_snapshot. The variant's own doc at client.rs:39-47 explains why.
    • The description also does not mention:
      • the loading-placeholder change (commit fdea558), which changes Linux behaviour;
      • that VRAM total now prefers ADLX's figure, while the datapoint table still says the source is Win32_VideoController.AdapterRAM only;
      • the process-wide SetDefaultDllDirectories change and the windows-sys dependency;
      • the README, manual-testing and installation doc edits.
    • Confidence 100 · mechanical · Fix: correct the description and the client.rs:196-198 comment to match the code (the code governs here).
  • [standing — not re-found by this round's review; settled by putting it to the reviewer, see below] VRAM total is still capped near 4 GiB on the WMI fallback path — crates/rocm-dash-collectors/src/winperf.rs:186-191, :159-160, :180; crates/rocm-dash-tui/src/ui/tabs/home.rs:37-46

    • Fixed part: with exactly one adapter and one ADLX reading carrying Some(total_vram), the ADLX total is now used.
    • Still holding: otherwise (no AMD driver/ADLX, current_gpu_metrics failing, total_vram_mb: None) the total falls back to AdapterRAM, a uint32 that caps at about 4095 MiB, while vram_used_mb sums the uncapped Dedicated Usage counter and vram_used_pct has no clamp. A single-adapter card above 4 GiB without a usable ADLX total can still show the wrong total and VRAM% above 100%.
    • The ADLX total is read only inside the Ok arm of current_gpu_metrics, so a metrics-query failure also discards a good TotalVRAM.
    • The "overflow" fixture at :446-448 uses 4294967296, a value the uint32 field cannot emit (a ≥4 GiB card reports about 0xFFF00000), and no fixture covers the None fallback.
    • Fix (unchanged from the earlier round): take the fallback total from the adapter's HardwareInformation.qwMemorySize registry value (or clamp/hide the percentage when the total is the capped WMI value), decouple the TotalVRAM read from the metrics call, and add a fixture at the real cap.

Non-blocking

  • [standing] CPU/memory refresh waits on the GPU query, and the collector inflates the CPU figure it reports — client.rs:221-232

    • host.tick() runs only after gpu.snapshot().await returns, so CPU/memory update every 5–15 s on the hardware described at winperf.rs:29-35.
    • Starting a new powershell.exe each tick lands inside the very CPU window sysinfo measures.
    • Confidence 85 · logic · Fix: tick host metrics on their own timer, or sample them before the GPU query.
  • [new] The "—" placeholder fix stops short of other values that look real but are not

    • Multi-adapter Windows hosts have vram_total_mb == 0, so the hero prints "VRAM 0%" next to non-zero used VRAM (home.rs:384, :396-400).
    • Without ADLX, TEMP and POWER print 0°C and 0W, which look like real readings.
    • Confidence 85 · logic · Fix: show "—" in these cases too.
  • [standing] Any extra display adapter disables the single-GPU path — winperf.rs:168, :334-342

    • Win32_VideoController also lists RDP, Basic Display and virtual display adapters, and the iGPU on hybrid laptops.
    • With any of these present, the code takes the multi-adapter branch: ADLX readings are dropped, VRAM total becomes 0, and phantom zero-VRAM LUIDs reappear as extra GPUs — the symptom the single-adapter collapse was written to fix.
    • gpu_model comes from the first adapter, which may be one of these.
    • Confidence 70 · logic · Fix: filter adapters by PNPDeviceID or AdapterCompatibility.
  • [new] Multi-GPU device ids can swap between ticks — winperf.rs:136-150, :210-222

    • win-gpu-{i} follows first-seen order, and UTIL lines (which exist only for processes currently holding a GPU context) come first.
    • Adapter order can therefore change between ticks, mixing per-device history.
    • Confidence 70 · logic · Fix: iterate keys in BTreeMap order.
  • [standing, was blocking — now non-blocking] ADLX is initialized and torn down every second — winperf.rs:280-304

    • AdlxHelper::new(), and the DLL load and unload with it, runs on every tick (now under spawn_blocking).
    • The adlx crate itself has // TODO: This should be a singleton.
    • Confidence 70 · architectural · Fix: keep one helper alive on a dedicated blocking thread.
  • [standing, was blocking — now non-blocking] The DLL search-path hardening applies to the whole process — winperf.rs:272-278

    • SetDefaultDllDirectories(LOAD_LIBRARY_SEARCH_SYSTEM32) affects every later LoadLibrary in rocm.exe, and also drops the application directory from the search.
    • Its return value is ignored.
    • The comment calls adlx a "vendored crate"; it is a crates.io dependency.
    • Confidence 70 · architectural · Fix: document the process-wide effect where the call is made, check the return value, and fix the wording.
  • [new] The adlx crate can panic or print to stderr mid-TUI

    • GpuIterator::next calls .unwrap(), and AdlxHelper::drop calls eprintln!.
    • A panic inside spawn_blocking is absorbed by unwrap_or_default, but the default panic hook still prints over the terminal.
    • Confidence 70 · logic.
  • [new; ponytail: marker standing] Stale or stray comments

    • winperf.rs:225-229 and :255-262 justify the design by "index-based correlation", which parse_metrics no longer does.
    • winperf.rs:124 begins with a stray ponytail: marker.
    • crates/rocm-dash-tui/src/registry_probe.rs:18 still says the Windows client "never connects at all". Its twin comments in dash.rs and app/mod.rs were updated in this PR; this one was missed.
    • crates/rocm-dash-tui/Cargo.toml:19 calls the dependency "Windows-only", but it is unconditional.
    • Confidence 85 · mechanical.
  • [new] The "overflow" test fixture uses a value WMI cannot return — winperf.rs:446-448

    • AdapterRAM is a uint32, so 4294967296 can never appear; a card with ≥4 GiB reports roughly 0xFFF00000.
    • multiple_adapters_leave_vram_total_zero is a strict subset of multiple_adapters_ignore_adlx_vram_and_leave_total_zero.
    • Confidence 85 · mechanical.
  • [new] push_host_gpu_snapshot copies the history/latest/clamp body of push_snapshot — app/mod.rs:1504-1511

    • Two copies of the same logic will drift apart. A shared helper would prevent it.
    • Confidence 85 · mechanical.

Decisions for the author

  • PowerShell spawned once per tick vs native PDH — tradeoff

    • The PR lists native PDH as a follow-up and says the current approach "already works".
    • In favour of moving now: the locale failure above, a refresh rate that is really the spawn time (1–15 s), and collection cost counted in the dashboard's own CPU reading. PDH bindings are already reachable through windows-sys.
    • In favour of deferring: less unsafe code, and PDH buffer-size negotiation.
    • Either way, the locale finding should be fixed or surfaced in this PR.
  • A second, thinner producer inside the TUI vs running the daemon's run_loop in-process on Windows — non-blocking-improvement

    • rocm_dash_daemon::runner::run_loop is not unix-gated, and apps/rocm already depends on the daemon crate.
    • A Windows maybe_spawn_embedded_daemon could run it in-process and forward its Events, with a pluggable GPU source.
    • That would give, for free: instances, Lemonade gen-tps scrapes, the bench ring, snapshot warnings, and the pending-vs-unavailable GPU state. It would also make the HostGpuSnapshot special case and the new rocm-dash-tui → rocm-dash-collectors edge unnecessary.
    • What the current approach leaves empty or misleading on Windows: the Instances, Observe and Bench tabs, and the chat agent's instance list. Lemonade is the Windows engine.
    • The current approach is honest about its scope (README:34, manual-testing) and is legal under crate_edges.rs.

Positive signals

  • Keeping the Windows tick out of Event::Snapshot (ClientMsg::HostGpuSnapshot) preserves the #145 quit-confirm semantics on a host with no instance registry.
  • kill_on_drop(true) under the timeout, so a hung powershell.exe cannot leak.
  • ADLX readings are trusted only when there is exactly one real adapter and exactly one reading. multiple_adapters_never_trust_adlx_even_when_reading_count_matches_adapter_count pins that positional coincidence is not trusted.
  • vram_used_pct fixes the old .max(1) bug, which turned a zero total into used ÷ 1.
  • The crate-edge allowlist entry was added rather than bypassed.

Deployment notes

  • New Windows-only dependencies: adlx 0.0.0-alpha.1 (a pre-release crate) and libloading 0.8.9. Manifests and notices are updated.
  • On Windows, rocm dash live mode now spawns powershell.exe once a second and loads amdadlx64.dll once a second.
  • It also changes the process's default DLL search path to System32 only.

What this covered

Every file in the change against c7a0d65f…fdea5585, plus the daemon runner's GPU tick, registry_probe.rs, the dash e2e feature and step definitions, the adlx 0.0.0-alpha.1 crate source, and the docs {include} chain.

  • The code passes ran as two coupled-group workers: collectors and manifests in one, TUI and docs in the other.
  • Two design workers looked at PDH vs PowerShell and at where Windows collection should live.
  • The TUI worker read apps/rocm/src/dash.rs, app/mod.rs and app/event_loop.rs through their hunks and the relevant cfg/snapshot regions, not in full. The diff to dash.rs is doc-comment only.

What ran:

  • cargo test -p rocm-dash-collectors: 122 passed, including 14 winperf::tests.
  • cargo test -p rocm-dash-tui: 898 lib, 20 characterization, 5 journeys and 7 wave0 tests passed.
  • The mutation run in the tests finding.
  • cargo xtask check-crate-edges: passed.
  • cargo clippy -p rocm-dash-collectors --lib --target x86_64-pc-windows-gnu -D warnings: clean, and a probe confirmed the #[cfg(windows)] code really was checked.

Did not run, and why:

  • Windows type-check of rocm-dash-tui, including the cfg(windows) connect_and_run: not done locally because aws-lc-sys needs the mingw gcc, which is not installed here. CI's windows-build-and-test and clippy jobs are green for this head.
  • Agent-instruction, history and code-comment passes: run by the driver, not as independent workers.
  • Prior-changes pass: did not run, because earlier review comments were out of bounds for this run.

CI on fdea558:

  • Hosted E2E tests: red with the same 4 unexpected failures as main's run 39631143323 (dash-07–09 and dash-33/34 on "user quits the dashboard", serve-01/02). Not caused by this PR.
  • Strix Halo Windows: failed in pre-warm on an HTTP 403 from the TheRock tarball host, before any scenario ran. Infrastructure, but it means the lane exercised nothing of this change — missing evidence, not a defect.
  • Strix Halo WSL2: failed in "Ensure native build deps" on an apt mirror-sync error. Infrastructure.
  • W7900: still queued.

Ticket ROCMAI-564 was retrieved by its ID. The review itself was not reconciled against the PR discussion; the standing objections below were.

Standing objections from the previous round (9bb9005)

The previous change request was dismissed by the author. Its objections were re-checked against this head:

  • Quit with a model serving skips the warning on Windows — resolved. This round's blind review independently credits ClientMsg::HostGpuSnapshot with keeping has_received_snapshot false (Positive signals). Its missing test is part of the tests finding above.
  • ADLX readings matched by list position — resolved (single-adapter + single-reading guard, with a test).
  • ADLX matching untested / helper tests pass with it removed — mostly resolved; the remaining None-fallback gap is in the tests finding.
  • ADLX re-initialised every second on an async worker — the async-worker part is resolved (spawn_blocking); the per-tick re-init is re-found this round as non-blocking.
  • powershell.exe children leaked on timeout — resolved (kill_on_drop(true)).
  • amdadlx64.dll loaded by bare name — mitigated by SetDefaultDllDirectories; this round rates the residue non-blocking.
  • Docs say Windows has no live dashboard — mostly fixed; README:41 and :205 still say so (blocking above).
  • No e2e scenario or stated justification — still holds (blocking above).
  • Vacuous home.rs test — replaced by a direct vram_used_pct test; new gaps are in the tests finding.
  • Winperf tests not testing what their names claim — the named cases were fixed; the remaining single-branch gaps are in the tests finding.
  • VRAM total capped near 4 GiB — not re-found by this round's blind review. Put to the reviewer afterwards, with the author's argument; its answer (anchored evidence, from a context told where to look) is that the objection still holds on the WMI fallback path. Kept as a standing blocker above.

juhovainio added a commit that referenced this pull request Oct 22, 2026
…ndows GPU metrics

winperf.rs:
- sum-then-max UTIL aggregation across engines via device_key/engine_type
  keying, fixing under-reported utilization on multi-engine devices
- WMI_ADAPTER_RAM_CAP_THRESHOLD heuristic and decoupled ADLX total-VRAM
  read from counter-query success, fixing 0/capped-near-4GiB VRAM
- counter_samples_missing_warning surfaces a locale-mismatch warning when
  WMI finds an adapter but perf counters return nothing, plumbed through
  snapshot()'s new Vec<String> return
- BTreeSet-based stable device ordering, SetDefaultDllDirectories
  return-value check, corrected test fixtures, new tests

ui/tabs/home.rs:
- split gpu_telemetry_pending (no snapshot yet) from gpu_data_unavailable
  (snapshot arrived but empty), so the hero no longer sticks on
  "Detecting GPU..." once an empty snapshot arrives
- extend "-" placeholder logic to zero vram_total/temp/power

app/mod.rs:
- extract push_history_and_latest shared helper used by push_snapshot and
  push_host_gpu_snapshot
- add coverage test for push_host_gpu_snapshot's instances/has_received
  behavior

client.rs:
- fix stale doc comment (ClientMsg::HostGpuSnapshot, not Event::Snapshot)
- plumb winperf's new warnings through to the Snapshot on both the Ok and
  Err arms
- sample host.tick() before gpu.snapshot() so the dashboard's own CPU%
  doesn't include the powershell.exe spawn/wait window

registry_probe.rs, Cargo.toml: correct stale/misleading comments

dash.feature: add dash-37 no-GPU-telemetry Linux e2e scenario

README.md: correct Windows feature-parity claims

Addresses review 5615012813 on PR #715.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
@juhovainio

Copy link
Copy Markdown
Collaborator Author

Addressed round 2 review (5615012813) in 0158a34.

Blocking items — fixed

  • Hero stuck on "Detecting GPU…" forever — home.rs: split gpu_telemetry_pending (latest.is_none()) from gpu_data_unavailable (snapshot arrived, empty). The hero now correctly falls through to the "unavailable"/"—" states once any snapshot lands, instead of treating an empty-but-real snapshot the same as "nothing yet".
  • Windows collection failures never reach the screen — client.rs: the Err arm of gpu.snapshot() now pushes the error into warnings (matching the daemon's "amd-smi metric: {e}" pattern) instead of only warn!-logging it, and both the Ok/Err arms plumb winperf's own new warnings through.
  • Locale-silent Get-Counter failures — winperf.rs: new counter_samples_missing_warning detects "WMI sees an adapter, counters produced nothing" and surfaces it as a warning instead of a silent empty panel.
  • Multi-process GPU utilization under-reporting — winperf.rs: re-keyed UTIL aggregation to (device, engine type), summing per-process samples within an engine and maxing only across engines (matches Task Manager's own method); updated the test that was locking in the under-count.
  • VRAM total still capped/wrong near 4GiB on the WMI fallback path — winperf.rs: added WMI_ADAPTER_RAM_CAP_THRESHOLD — a WMI AdapterRAM reading at/above ~4GiB is treated as unknown (0, which the UI already guards on) rather than shown as a plausible-but-wrong truncated number; ADLX's own TotalVRAM read is now also decoupled from whether the counter query itself succeeded, so a metrics-query failure no longer discards a good ADLX total. See judgment call below on why this is a sentinel-detection fix rather than the registry read you suggested.
  • README contradicts itself (:41, :205 vs :34/:816) — both corrected to match the "Windows gets GPU/CPU/memory, not the instance registry" line.
  • No test covers the two behaviours that matter — added an AppState test asserting push_host_gpu_snapshot leaves instances/has_received_snapshot untouched, render tests for the "—" label branches, and winperf fixtures for the .min(100.0) clamp, the None-fallback path, and a device_key with no luid_.
  • No e2e scenario for the user-visible change — added dash-37 (tests/e2e-cucumber/features/dash.feature), a Linux scenario for the hero's "no GPU telemetry" state with a real embedded daemon (not --demo). The transient "pending" placeholder is explicitly left uncovered with a comment explaining why (closes in well under a second, no hook to freeze the daemon mid-handshake — asserting on it would be asserting on a race).
  • PR description / doc comment say Event::Snapshot, description omits several changes — PR body rewritten (Event::Snapshot → ClientMsg::HostGpuSnapshot, with the actual reasoning), client.rs's doc comment corrected to match, and the body now covers the loading-placeholder fix, the ADLX-preferred VRAM total, the SetDefaultDllDirectories/windows-sys dependency, and the README/doc edits.

Non-blocking items — fixed (cheap ones picked up)

  • CPU/memory refresh waiting on the GPU query, and the GPU collection cost leaking into the dashboard's own CPU%: host.tick() now samples before gpu.snapshot().
  • "—" placeholder logic extended to zeroed VRAM-total/temp/power, not just the VRAM-used case.
  • Multi-GPU device ids swapping between ticks: adapter keying is now BTreeSet-ordered (stable by LUID-derived key) instead of first-seen order.
  • push_host_gpu_snapshot duplicating push_snapshot's history/latest/clamp body: extracted into a shared push_history_and_latest helper.
  • Stale/stray comments: the ponytail: marker, the "index-based correlation" comments parse_metrics no longer does, registry_probe.rs's "never connects at all" claim, and Cargo.toml's "Windows-only" wording for the now-unconditional dependency — all corrected.
  • The "overflow" test fixture used a VRAM value (4294967296) WMI's uint32 field can never actually emit — replaced with a realistic near-cap value and folded into the new cap-threshold test.
  • SetDefaultDllDirectories's return value is now checked and logged on failure.

Judgment calls — left as deliberate decisions, not oversights

  • VRAM-cap heuristic vs. the registry HardwareInformation.qwMemorySize read you suggested: went with sentinel-detection (treat a near-4GiB WMI reading as unknown) rather than the registry read. It's a smaller, self-contained fix that eliminates the "wrong number shown" and "VRAM% over 100%" failure modes this PR needs to close, without a second data source. It doesn't recover a true total for a >4GiB card without ADLX — if that gap matters in practice, the registry read is still the right follow-up; happy to do it if you'd rather have it in this PR.
  • PNPDeviceID/AdapterCompatibility filtering for extra display adapters (RDP/Basic Display/virtual adapters breaking the single-adapter fast path): not done. Left as-is because reliably distinguishing "real" physical adapters from virtual ones needs more investigation than a quick filter, and the current behavior degrades honestly (falls back to the multi-adapter path, VRAM total 0) rather than silently misattributing.
  • ADLX re-initialized every tick (the adlx crate's own TODO: should be a singleton): left as-is. Agreed it's the right shape eventually, but keeping one helper alive on a dedicated thread is a bigger structural change than this round's scope, and you already downgraded it to non-blocking.
  • adlx crate's internal .unwrap()/eprintln!: left as-is — this is inside the adlx crate itself, not this PR's code; patching around it here would mean forking or wrapping the crate, which seems like a separate piece of work if it turns out to matter in practice (the spawn_blocking panic is at least caught, just not silently).
  • PowerShell-per-tick vs. native PDH: already called out as ## Follow-up in the PR description — not changed this round. Agree PDH would fix the locale issue more fundamentally than the warning this PR adds, but it's a bigger unsafe-FFI undertaking than a review-feedback round.
  • Second in-process producer vs. running the daemon's run_loop in-process on Windows: this is the bigger architectural question from "Decisions for the author" — not attempted here. It would be a much larger change (pluggable GPU source inside the daemon, Windows-specific maybe_spawn_embedded_daemon) than this PR's scope, and the current approach is upfront about what it doesn't cover (Instances/Observe/Bench tabs, chat agent's instance list) rather than silently pretending feature parity. Worth a separate ticket if it's a priority.

Build/test/lint all green on this head: cargo test (workspace), cargo clippy --workspace --all-targets -D warnings, cargo clippy -p rocm-dash-collectors --target x86_64-pc-windows-gnu, cargo xtask check-crate-edges.

ROCMAI-564: the TUI's Windows client arm was a stub (no rocmdashd socket
there yet), so Windows users saw no GPU/memory metrics at all.

Add WinPerfCollector, reading the OS's own `GPU Engine`/`GPU Adapter
Memory` performance counters via `Get-Counter` (the same WDDM/DXGK-backed
source Task Manager's GPU tab uses) plus `Win32_VideoController` for
adapter inventory/driver version. No AMD tool required. Mirrors
amd_smi.rs's subprocess + pure-parse shape.

client.rs's Windows connect_and_run now builds Snapshots in-process each
tick from HostCollector (already cross-platform) + WinPerfCollector,
feeding the same ClientMsg sequence the Unix socket arm produces, so the
app loop doesn't need to know which platform it's on.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
Required by ROCMAI-564's in-process Windows metrics ticker in
crates/rocm-dash-tui/src/client.rs, which reuses HostCollector and
WinPerfCollector directly instead of going through the daemon socket.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
GPU Engine perf-counter instances are per-process on Windows 10/11
(pid_NNNN_luid_..._phys_0_eng_...), but GPU Adapter Memory instances
are per-adapter only (luid_..._phys_0, no pid). device_key() keyed
both verbatim, so every process using the GPU produced its own
phantom near-zero device instead of aggregating onto the real
adapter, which is why real hardware showed "Unknown GPU" with
all-zero metrics despite `examine` detecting the GPU correctly.

Confirmed against live Get-Counter output from a Strix Point laptop.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
… calls

Real hardware logs (Strix Halo) showed every tick timing out: "powershell
Get-Counter query timed out" repeating for 20+ minutes straight, even
though the identical script runs near-instantly when invoked manually.
The bottleneck is powershell.exe child-process spawn latency, not the
Get-Counter query itself, and it consistently exceeds the old 5s budget
on real hardware.

Bump RUN_TIMEOUT to 15s, and collapse the two independent metrics()/
system_info() spawns (each re-running the identical script) into one
snapshot() call so each tick only pays the spawn cost once.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
…div-by-zero

winperf.rs reported one GpuMetrics per distinct GPU Adapter Memory LUID, but
Windows can surface multiple stale/phantom LUIDs for one physical adapter
(observed: 3 LUIDs, 2 permanently zero-VRAM, for a single AMD Radeon 780M).
When Win32_VideoController confirms there's exactly one real adapter, collapse
all LUIDs into one device instead.

home.rs's hero VRAM percentage used total.max(1) to dodge division by zero,
which silently produced a nonsensical inflated percentage (used_mb * 100)
whenever every GPU's vram_total_mb was 0 instead of indicating unknown/N/A.
Guard on total > 0 like widgets.rs and dock.rs already do.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
…on Windows

winperf.rs previously hardcoded temperature_c/power_w/clock_mhz to
0.0/0.0/None since OS perf counters don't expose them. Add an ADLX
(AMD Device Library eXtra) glue path, gated to Windows only, that reads
these three metrics when the AMD driver's bundled amdadlx64.dll is
present, falling back to the prior defaults on any failure (no driver,
non-AMD GPU, unsupported metric) so the snapshot never errors out.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
…erRAM

Win32_VideoController.AdapterRAM is a 32-bit WMI field that silently
truncates/overflows above 4GiB. ADLX's own TotalVRAM query isn't subject to
that, so parse_metrics now prefers it in the single-adapter/single-ADLX-GPU
case and falls back to the WMI value otherwise (no driver, non-AMD GPU,
multi-GPU host).

parse_metrics also now takes adlx readings as a parameter instead of calling
adlx_readings() internally, so its matching/fallback logic is unit-testable
without the Windows-only ADLX FFI chain.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
current_gpu_metrics(&gpu) needs gpu borrowed, not moved, and the move has to
happen after the call since Gpu isn't Copy. Caught by actually building for
x86_64-pc-windows-gnu (this #[cfg(windows)] function was gated out of every
check/clippy run on the Linux dev workstation).

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
… GPU metrics

- ADLX-to-device matching: only trust temp/power/clock when exactly one
  real adapter and one ADLX reading exist, matching vram_total_mb's
  existing guard; multi-adapter hosts degrade every slot to defaults
  instead of index-based misattribution.
- Run adlx_readings() inside spawn_blocking so ADLX FFI/driver calls
  don't block a tokio worker thread.
- kill_on_drop on the powershell.exe Command so a timed-out child is
  actually killed instead of leaked.
- Mitigate amdadlx64.dll DLL search-order hijack via
  SetDefaultDllDirectories(LOAD_LIBRARY_SEARCH_SYSTEM32), called once
  process-wide before the first AdlxHelper::new().
- Update stale "Windows has no live dashboard" docs/comments now that
  the Windows client streams live GPU/CPU/memory (registry view still
  empty by design, no rocmdashd unix socket there).
- Strengthen winperf.rs and home.rs tests flagged as weak/vacuous:
  direct vram_used_pct unit test, ADLX temp/power/clock assertions,
  isolated pid-prefix-stripping fixture, order.is_empty() guard test,
  sum-vs-max VRAM fixture, realistic SAMPLE AdapterRAM value.
- Non-blocking cleanups: ticker MissedTickBehavior::Skip matching the
  daemon's existing choice, fixed stray comment's wrong API name.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
…U hero

Before the first GPU telemetry snapshot arrives (or a tick returns no
GPUs), the Home tab hero rendered "Unknown GPU" and 0%/0°C/0W values
that are indistinguishable from real (broken) readings. Gate a new
gpu_telemetry_pending() check on "no GPUs arrived yet" and swap in
"Detecting GPU…" and "—" placeholders only for that case, leaving the
already-tested legitimate-zero-VRAM-total behavior untouched.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
…ndows GPU metrics

winperf.rs:
- sum-then-max UTIL aggregation across engines via device_key/engine_type
  keying, fixing under-reported utilization on multi-engine devices
- WMI_ADAPTER_RAM_CAP_THRESHOLD heuristic and decoupled ADLX total-VRAM
  read from counter-query success, fixing 0/capped-near-4GiB VRAM
- counter_samples_missing_warning surfaces a locale-mismatch warning when
  WMI finds an adapter but perf counters return nothing, plumbed through
  snapshot()'s new Vec<String> return
- BTreeSet-based stable device ordering, SetDefaultDllDirectories
  return-value check, corrected test fixtures, new tests

ui/tabs/home.rs:
- split gpu_telemetry_pending (no snapshot yet) from gpu_data_unavailable
  (snapshot arrived but empty), so the hero no longer sticks on
  "Detecting GPU..." once an empty snapshot arrives
- extend "-" placeholder logic to zero vram_total/temp/power

app/mod.rs:
- extract push_history_and_latest shared helper used by push_snapshot and
  push_host_gpu_snapshot
- add coverage test for push_host_gpu_snapshot's instances/has_received
  behavior

client.rs:
- fix stale doc comment (ClientMsg::HostGpuSnapshot, not Event::Snapshot)
- plumb winperf's new warnings through to the Snapshot on both the Ok and
  Err arms
- sample host.tick() before gpu.snapshot() so the dashboard's own CPU%
  doesn't include the powershell.exe spawn/wait window

registry_probe.rs, Cargo.toml: correct stale/misleading comments

dash.feature: add dash-37 no-GPU-telemetry Linux e2e scenario

README.md: correct Windows feature-parity claims

Addresses review 5615012813 on PR #715.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
…rning

The ascii_output_contract test requires all string literals reachable by
redirectable output streams to be ASCII, since Windows consoles on
non-UTF-8 code pages (e.g. CP437) render em dashes as mojibake.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
rominf
rominf previously requested changes Oct 23, 2026

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · d6b894f

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Review — needs work

Full review of the whole change.
Implements ROCMAI-564; the ticket could not be retrieved from here, so how much of it this covers is unknown. The description claims Windows in-process GPU/CPU/memory dashboard telemetry, and the diff delivers that.

Round 3. Each finding is marked new (first raised this round) or standing (raised in an earlier round and still holding). The previous round's objections were settled against this head; see Standing objections from the previous round (fdea558) at the end. Six of its nine blockers are resolved.

Blocking

  • [new] The hero shows "Detecting GPU…" forever after a disconnect, and at the end of every --replay — crates/rocm-dash-tui/src/ui/tabs/home.rs:77, crates/rocm-dash-tui/src/app/event_loop.rs:1041, crates/rocm-dash-tui/src/replay.rs:112

    • gpu_telemetry_pending is only state.latest.is_none().
    • ClientMsg::Disconnected sets latest = None. That happens when the daemon drops on Linux, and when a replay sends "end of recording".
    • So the title claims detection is still in progress while the header says "disconnected". Before this change it read "Unknown GPU".
    • The doc comment says "True until the first snapshot has arrived at all". The code does not do that, so the comment governs.
    • Confidence 85 · logic · Fix: also gate on ConnState::Initial | Connecting, or track "ever received". Add a unit test for the disconnected case.
  • [new] On Windows, the Observe tab's empty Instances panel now tells the user to start a daemon with flags that cannot help there — crates/rocm-dash-tui/src/ui/tabs/instances.rs:379, reached via crates/rocm-dash-tui/src/client.rs:211

    • The Windows arm now sends Connected, and instances never fills on Windows (by design). So draw_empty always shows "no instances · start daemon with --enable-docker on a host with vLLM containers".
    • On Windows the daemon is unix-only and vLLM is unsupported.
    • AGENTS.md §3: remediation advice must name a command that exists and actually clears the condition.
    • Confidence 85 · logic · Fix: show a separate empty-state message for the in-process (Windows) connection, and test it.
  • [new] Tests added for the new branches stay green when those branches are broken — tests/e2e-cucumber/tests/e2e/dash_steps.rs (hero_reports_no_gpu_telemetry), crates/rocm-dash-tui/src/ui/tabs/home.rs:301, crates/rocm-dash-collectors/src/winperf.rs:265

    • e2e VRAM check: the step takes the first screen line containing "VRAM" and asserts it contains "—". The utilization gauge's "—" is drawn on that same row. With vram_label reverted to always format a percentage, the row renders │ — VRAM 0.0% │ and the check passes (render probe).
    • Utilization gauge: replacing if gpu_data_unavailable(state) with if false at home.rs:301 leaves all 29 home::/push_host_gpu unit tests green.
    • unavailable || arms in the VRAM/temp/power labels: these mutations also survive, because with no GPUs those values are already 0.
    • Multi-adapter clamp: removing .min(100.0) at winperf.rs:265 leaves all 20 winperf tests green.
    • Confidence 100 (mutation runs) · mechanical · Fix:
      • Assert on the text after the "VRAM" label rather than the whole row.
      • Add a render test with latest = Some(Snapshot::default()) asserting the GPU UTILIZATION gauge shows "—" and not "0.0%".
      • Add a multi-adapter fixture whose utilization exceeds 100.
  • [standing, Windows part] The new Windows behaviour has no scenario or test, and the description's reason for that is wrong — tests/e2e-cucumber/features/dash.feature:603-620, .github/workflows/e2e-selfhosted.yml:564

    • Windows rocm dash changes from a connect error to live GPU/CPU/memory. That is user-observable (AGENTS.md §3). Every dash scenario, including the new dash-37, is @requires-os:linux.
    • The description gives the reason as "no Windows CI runner with a GPU". The required "E2E tests (Strix Halo, Windows)" lane is a GPU Windows runner.
    • It also says this is "reasoned about explicitly in that scenario's feature-file comment". The dash-37 comment does not mention Windows. It only discusses the Linux "pending" race.
    • The code governs: the lane exists.
    • Confidence 85 · architectural · Fix: add a Windows-lane scenario. If the PTY dash harness cannot run on Windows, say that (the actual reason) in the PR text and in the feature comment.
  • [standing, narrowed] The description omits a user-visible change on every platform — crates/rocm-dash-tui/src/ui/tabs/home.rs

    • The description lists home.rs only for the VRAM-percentage fix.
    • The diff also replaces the hero's "Unknown GPU" and 0%/0°C/0W with "Detecting GPU…" and "—" placeholders on Linux and in replay too. That is the source of the first finding above.
    • The docs/manual-testing.md and docs/rocm-docs/install/installation.md edits are also unmentioned.
    • Confidence 85 · mechanical · Fix: add these to the description (the work is complete, so the description is what needs fixing).
    • Watcher note: rocm-cli squash-merges build the commit body from commit messages, not the PR description, so this item does not gate on its own; it is listed because the method classifies it as blocking.

Non-blocking

  • [new] A comment claims a call ordering does something it cannot — crates/rocm-dash-tui/src/client.rs:226-229

    • The comment says sampling host.tick() before the GPU query keeps the powershell cost out of CPU%.
    • HostCollector::tick (host.rs:32) reports refresh_cpu_all's delta since the previous call. So every reading spans one full GPU-query window whichever call runs first.
    • Confidence 85 · logic · Fix: delete the claim.
  • [new] A hung ADLX call freezes all Windows dashboard metrics, with no warning — crates/rocm-dash-collectors/src/winperf.rs:71-73, crates/rocm-dash-tui/src/client.rs:224-256

    • Only run_script has a timeout. spawn_blocking(adlx_readings).await has none.
    • The loop is serial, so a stuck driver call stops CPU/memory updates too.
    • A JoinError (an ADLX panic) is swallowed by .unwrap_or_default().
    • Confidence 70 · logic · Fix: wrap it in timeout, warn on a timeout or JoinError, and skip ADLX while a previous call is still in flight.
  • [standing] Phantom LUIDs come back as extra GPUs on multi-adapter hosts — crates/rocm-dash-collectors/src/winperf.rs:258-270

    • Only the single-adapter branch collapses phantom zero-VRAM LUIDs. The multi-adapter path emits one win-gpu-N per LUID, while parse_system_info reports physical_gpu_count = adapter count.
    • Virtual adapters in Win32_VideoController (Basic Display, Remote Display) can push a single-APU host into this path, which also loses ADLX data and the VRAM total.
    • Confidence 70 · logic · Fix: drop LUIDs with neither utilization samples nor VRAM, and/or filter WMI adapters to PCI\ PNP IDs.
  • [new] dash-37's stated premise is false — tests/e2e-cucumber/features/dash.feature:604-609

    • The comment says "This CI runner has no AMD GPU". The scenario carries no GPU-exclusion tag, and it ran and passed on the MI300X GPU lane for this head (job 120364539157).
    • It therefore passes for a different reason than the one it states, presumably harness isolation of the GPU collector.
    • Confidence 85 · mechanical · Fix: state the real reason, or tag it @requires-no-gpu if the hardware premise is meant.
  • [new] A module doc claims the shellout fails cleanly off Windows — crates/rocm-dash-collectors/src/winperf.rs:15-18

    • Under WSL, interop puts powershell.exe on PATH, so snapshot() would read the Windows host's GPUs.
    • No caller exists today, since the only caller is cfg(windows).
    • Confidence 70 · mechanical · Fix: correct the doc, or cfg(windows)-gate snapshot/run_script.

Decisions for the author

  • A process-wide SetDefaultDllDirectories(LOAD_LIBRARY_SEARCH_SYSTEM32) — tradeoff (winperf.rs:323-336)

    • For: it closes the bare-name amdadlx64.dll hijack vector.
    • Against: it removes the application directory and PATH from DLL search for the whole rocm process. No other bare-name loads exist today; a future one fails silently.
    • Alternative: LoadLibraryExW with the flag yourself, or add LOAD_LIBRARY_SEARCH_APPLICATION_DIR.
  • Spawning powershell.exe plus a full ADLX load/init/terminate on every 1 s tick — non-blocking-improvement

    • The description already names a PDH rewrite as follow-up.
    • Note: windows-sys is now a dependency anyway, and PdhAddEnglishCounterW would also remove the locale mismatch that counter_samples_missing_warning exists to report.
    • Recreating AdlxHelper per tick also means its Drop's eprintln! on terminate failure would write over the TUI.

Positive signals

  • Routing Windows ticks through a separate ClientMsg::HostGpuSnapshot, instead of Event::Snapshot, avoids latching has_received_snapshot with an empty stand-in instances. A mutation test confirms it.
  • ADLX readings are trusted only on a one-adapter/one-reading match. This avoids reporting one GPU's temperature or power under another's ID.

Deployment notes

  • New Windows-only dependencies: adlx 0.0.0-alpha.1 (MIT; an unmaintained-looking 2024 alpha) and windows-sys 0.61. Notices and the manifest match the lockfile.
  • On Windows, rocm dash now spawns powershell.exe on every tick and changes DLL search order for the whole process.

What this covered

  • Read: every changed file, completely, in prw-base…d6b894f4 (0314db3…d6b894f4), plus the consumers: host.rs, event_loop.rs, replay.rs, instances.rs, and the e2e harness expectations.
  • Fan-out: two independent workers, one on the collectors/winperf group and one on the dash-tui, e2e and app group. The driver ran agent-instruction adherence, history, docs-claim checks, CI checks and Step 7.

Commands run:

  • cargo test -p rocm-dash-collectors --lib winperf: 20 ran.
  • cargo test -p rocm-dash-tui --lib -- home:: push_host_gpu: 29 ran.
  • Nine targeted mutations across both crates.

Windows checks:

  • cargo clippy -p rocm-dash-collectors --lib --tests --target x86_64-pc-windows-gnu -D warnings is clean, and was proven live with an injected E0308.
  • rocm-dash-tui could not cross-compile: aws-lc-sys needs x86_64-w64-mingw32-gcc. Its Windows arm was instead lint-checked on Linux by swapping the cfg arms, also proven live.

CI:

  • The one failing check, "E2E tests" (dash-gen-tps-expiry-boundary, dash-esc-menu-quit-row-confirms-when-serving), fails identically on main's last three CI runs. It is not attributed to this PR.
  • Watcher note: this branch's merge-base is ea1d871, whose main run (39793631227) fails with the same two unexpected failures. Main's run on 0314db3 (#719, which changes the quit-timeout handling these scenarios hit) passes E2E. A rebase onto current main would give this PR clean E2E evidence, which matters because it touches the quit-confirm path.

Did not run:

  • Anything executing on Windows: real perf counters, ADLX FFI, timeout and kill behaviour.
  • The cucumber e2e locally.
  • The prior-changes pass (earlier review comments on these files were not consulted).
  • Step 7 design questions, which were run in the driver context and so are not independent.

Other:

  • ROCMAI-564 was not retrievable: no tracker tool was granted.
  • One worker briefly wrote a file to /dev/shm and deleted it straight away.
  • Not reconciled against any PR discussion here. The standing objections below were.

Standing objections from the previous round (fdea558)

The previous change request was dismissed by the author, and the head has since been rebased, so fdea558 is gone. This round reviewed the whole change blind. Each earlier blocker it did not re-find was then put to the reviewer, together with the author's reply. Those answers are anchored evidence: they come from a context that was told where to look.

  • Hero stuck on "Detecting GPU…" when a snapshot arrives with no GPUs — resolved. gpu_telemetry_pending is now latest.is_none(), and a known model is shown. The disconnect/replay case is a separate new finding above.
  • Windows collection failures never reach the screen — resolved. The Err arm pushes a warning, and winperf's own warnings are passed through. The Err-arm plumbing is untested, which is Windows-only.
  • English counter names fail silently on non-English Windows — the minimum asked for is met: counter_samples_missing_warning surfaces the empty result. No longer blocking. What remains:
    • It does not fire when only one of the two counter paths fails to match.
    • Nothing tests its call site in snapshot().
    • The root cause (localized counter paths) remains until PDH.
  • Utilization took the max across processes — resolved: summed within an engine type and maxed across types, with tests.
    • Still to check: two physical engines of the same type may now be summed where Task Manager shows them separately. Clamped at 100; not established from code. That file is in scope for the next round.
  • README:41/:205 said live mode needs Linux/WSL — resolved.
  • VRAM total capped near 4 GiB on the WMI fallback — the stated case is resolved: a near-cap reading is treated as unknown, and the ADLX total is read independently of the metrics query. No longer blocking. What remains:
    • vram_used_pct is still unclamped.
    • A driver that wraps rather than caps AdapterRAM would still give a wrong total. Which drivers do that was not established.
    • The registry qwMemorySize read is the author's acknowledged follow-up.
  • connect_and_run doc comment said Event::Snapshot — resolved in code.
  • No test covered the #145 quit-confirm guard or the placeholder labels — the #145 guard mutation is now caught by push_host_gpu_snapshot_leaves_instances_and_has_received_snapshot_untouched. The label branches are still partly uncovered (the tests finding above). The event_loop.rs:1044 dispatch arm is untested.
  • No e2e scenario / stated reason — Linux part resolved (dash-37). The Windows part still holds (blocking above).

…ndows GPU metrics

- winperf.rs: fix a stale test fixture that reused an existing LUID
  instead of introducing a genuinely new one, so the multi-adapter
  clamp branch was never exercised
- winperf.rs: wrap the ADLX query in a timeout; a wedged driver call
  no longer freezes every other metric this tick, and surfaces as a
  warning instead of silently degrading
- winperf.rs: correct a module doc claim that the powershell.exe
  shellout always fails cleanly off Windows (it does not under WSL
  interop, where powershell.exe is on PATH)
- client.rs: delete a comment claiming a tick-ordering benefit that
  HostCollector::tick's CPU delta does not actually provide
- dash.feature: correct dash-37's stated premise; the stable zero-GPU
  state comes from the daemon's /dev/kfd preflight failing for the
  test harness process, not from the CI runner lacking an AMD GPU
- home.rs: mark gpu_telemetry_pending as const fn per clippy

PR description also corrected to credit the full hero-placeholder
change set and docs edits, and to fix a false claim about why the
Windows collector path isn't covered by e2e (the PTY driver is
Linux-only for now, not a lack of a GPU Windows runner).

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
…etry step

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
@rominf

rominf commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

Informational — describes 677a2b9, not the current head. The head moved to 9173de9 while this round ran, so this report does not gate and files no change request; its blocking findings are standing findings the next round will settle against the new head. Line anchors refer to 677a2b9. 9173de9 touches app/mod.rs, ui/tabs/home.rs and dash.feature only, so the "Detecting GPU… after a disconnect" item may already be addressed there; it has not been checked.

🔴 Automated review · pr-review-watcher · 677a2b9

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Review — needs work

Full review of the whole change.
The description says it implements ROCMAI-564 (Windows GPU/memory metrics for the TUI). It covers the ticket's sourcing and wiring scope. Real-hardware validation is on a Radeon 780M, not the Strix Halo / HIP-SDK hosts the ticket names.

Round 5. Each finding is marked new (first raised this round) or standing (raised in an earlier round and still holding). This round was a full, blind re-review of the whole change; the previous change request (a9b08ac) is settled at the end.

Blocking

  • [new] GPU utilization is grouped by engine type, not by engine, so separate engines of the same type are added together — crates/rocm-dash-collectors/src/winperf.rs:157-159 (engine_type), :193-219, test :664-680
    The code's own comment says utilization is keyed "by (device, engine)" and that the engtype suffix is "distinct per engine". The description gives the intended model too: sum processes per engine, then take the max across engines. But engine_type returns only the engtype_* suffix. Two physical engines of the same type (eng_2_engtype_3d, eng_3_engtype_3d; Copy and Compute engines often come in several) therefore share one key, are summed, and are then clamped. Two engines each at 60% read as 100%.
    The test multi_adapter_utilization_clamps_to_100_when_summed_processes_exceed_it uses exactly that fixture and calls it "the same engine", so it locks in the bug. A worker rekeyed on the eng_N index: that one test failed and the other 20 winperf tests passed. The comment and the description govern here, because they state the aggregation model; the code is the side that departs from it.
    Confidence 85 · logic · Fix: key on the instance text up to _engtype_ (the engine index), and correct that test's fixture and comment.

  • [standing, extended] The new Windows runtime path has no test at all — crates/rocm-dash-tui/src/client.rs:200-252 (#[cfg(windows)] connect_and_run), crates/rocm-dash-collectors/src/winperf.rs:77-130 (snapshot/run_script/WINPERF_SCRIPT), crates/rocm-dash-tui/src/app/event_loop.rs:1044
    No test references connect_and_run, WinPerfCollector, or the HostGpuSnapshot dispatch arm. The windows-build-and-test log for this head ran only the pure parser tests (winperf::tests::*, 21 tests). Nothing checks that the PowerShell script produces lines the parser accepts, that the ticker sends Connected with the in-process tag, or that it then sends HostGpuSnapshot rather than Event::Snapshot.
    The in-code argument that it is impractical (winperf.rs:772-775) covers only adlx_readings, which needs AMD hardware. These paths do not: the required Windows lane runs cargo test and has powershell.exe, and a WMI adapter (at minimum the basic display adapter) exists there. The description also states that no e2e scenario is possible for this path, which leaves unit tests as the only coverage available.
    Confidence 85 · logic · Fix: add #[cfg(windows)] tests that call WinPerfCollector::snapshot() and check it returns Ok with a parseable GpuSystemInfo, and that drive connect_and_run on a channel and check the first two messages.

  • [standing, extended] The description does not match the change — PR description vs crates/rocm-dash-tui/src/ui/mod.rs (draw_header), crates/rocm-dash-tui/src/ui/tabs/instances.rs (draw_empty), crates/rocm-dash-tui/src/ui/tabs/home.rs (gpu_telemetry_pending)

    1. It omits two user-visible changes: the header label live · localhost · local collectors, and the Windows Instances-tab message "no instance registry on this platform · vLLM containers aren't available here".
    2. It says "Detecting GPU…" is shown "while telemetry is genuinely pending" on every platform. On Linux that is not true for the real detection window (see the first non-blocking item).
      Confidence 85 · mechanical · Fix: narrow the description. List the header and Instances changes, and say "Detecting GPU…" covers only the time before the first snapshot. Alternatively, finish the work in the first non-blocking item.
      Watcher note: rocm-cli squash-merges build the commit body from commit messages, not the PR description, so this item does not gate on its own; it is listed because the method classifies it as blocking.

Non-blocking

  • [new] On Linux, "Detecting GPU…" covers only the socket handshake, not GPU detection — crates/rocm-dash-tui/src/app/mod.rs:1514 (push_history_and_latest), crates/rocm-dash-tui/src/ui/tabs/home.rs:80, crates/rocm-dash-daemon/src/runner.rs:371-400
    The daemon detects GPUs off the critical path, which can take up to ~15s, and meanwhile broadcasts snapshots with no GPUs. Any snapshot sets has_received_gpu_snapshot. So healthy hardware shows "Unknown GPU" and "—" for up to 15s, which looks the same as a host with no GPU. That is the state a9b08ac says it confines the placeholder to.
    Confidence 85 · logic · Fix: carry a "detection pending" signal in Snapshot and gate on that.

  • [standing] "Detecting GPU…" stays on screen for good when no snapshot ever arrives — crates/rocm-dash-tui/src/ui/tabs/home.rs (gpu_telemetry_pending)
    This happens with an unreachable --connect, a daemon that fails to bind, or a replay with no snapshot events. The title then claims detection is running while the header says disconnected (AGENTS.md §3: messages about the CLI's own state).
    Confidence 70 · logic · Fix: drop the pending state once conn is Disconnected.

  • [new] "Not measured" is sent as 0, and only the hero card was taught to read it — crates/rocm-dash-collectors/src/winperf.rs:299-322, 374-376; consumers crates/rocm-dash-tui/src/ui/dock.rs:124, 243, crates/rocm-dash-tui/src/ui/tabs/hardware.rs:405-416, crates/rocm-dash-tui/src/agent/snapshot.rs:21-29
    On the dash-37 screen, the hero shows "—" while the context rail on the same screen shows peak util 0.0% (confirmed by a run). On Windows hosts with more than one adapter, the GPU wall shows VRAM 0.0%, the hardware tab shows 0°C 0 W … / 0 MiB, and the chat agent's GPU tool reports temperature 0 and VRAM total 0 as if they were measured. dash-37's title promises "placeholders, not fake zeros", and the scenario cannot catch any of these.
    Confidence 100 · architectural · Fix: make the fields Option in GpuMetrics, or route every consumer through gpu_data_unavailable/vram_used_pct, and assert the rail in dash-37.

  • [standing] CPU/memory refresh waits on PowerShell — crates/rocm-dash-tui/src/client.rs:224-240, crates/rocm-dash-collectors/src/winperf.rs:34-40, 53-56, 85-93
    host.tick() and gpu.snapshot().await run one after the other. By the code's own comment, a PowerShell spawn regularly takes more than 5s, with a 15s cap, and ADLX then adds up to 5s more. CPU/memory therefore refresh at PowerShell's pace rather than once a second, and a powershell.exe is running almost all the time, which inflates the CPU figure on screen. The script also runs two separate Get-Counter calls, each of which samples over about 1s. That contradicts the comment saying the query "returns near-instantly".
    Confidence 70 · architectural · Fix: run GPU collection in its own task and send the latest cached result on each tick; pass both counter paths to a single Get-Counter.

  • [standing] Timed-out ADLX calls leave stuck blocking threads behind — crates/rocm-dash-collectors/src/winperf.rs:86
    timeout(…, spawn_blocking(adlx_readings)) cannot cancel the thread. If a driver call hangs, every tick adds another stuck thread, up to tokio's blocking-pool limit, and overlapping ADLXInitialize/Terminate pairs run at once. ADLX is also loaded, initialized and terminated on every tick.
    Confidence 70 · logic · Fix: add an in-flight flag that skips ADLX while a previous call is still running, or keep one long-lived helper on a dedicated thread.

  • [standing] Re-implements rocm-core's Windows GPU inventory without its AMD filter or its PowerShell path hardening — crates/rocm-dash-collectors/src/winperf.rs:53-58, 105; sibling crates/rocm-core/src/host_gpu.rs:37, 387-404
    The existing query keeps only VEN_1002/AMD adapters and resolves powershell.exe from System32 by absolute path. This one does neither. A hybrid Intel+AMD laptop therefore takes the multi-adapter path: no VRAM total, ADLX data discarded, and a title naming the first WMI adapter, which may be the Intel one.
    Confidence 70 · logic · Fix: filter to AMD/PCI adapters, and reuse or match rocm-core's tool resolution.

  • [new] Stale comments the PR's own rewording pass missed — crates/rocm-dash-tui/src/app/types.rs:126-127 and crates/rocm-dash-tui/src/ui/mod.rs:311-313 still say the Windows client "never connects". crates/rocm-dash-collectors/src/lib.rs:5-8 says collectors are used by the daemon and that each implements a trait, but winperf does neither. crates/rocm-dash-tui/src/app/mod.rs:80 says the flag is set by a "GPU-bearing" snapshot, but any snapshot sets it.
    Confidence 100 · mechanical · Fix: correct each comment.

  • [new] The Windows Instances message points at vLLM while Lemonade services are what's hidden — crates/rocm-dash-tui/src/ui/tabs/instances.rs (draw_empty)
    Lemonade serving works on Windows. Its services are missing only because there is no registry snapshot, which rocm services list would show.
    Confidence 70 · mechanical · Fix: say the dashboard doesn't list serving instances on this platform and point to rocm services list.

  • [new] No manual-testing steps for the new Windows live view — docs/manual-testing.md
    AGENTS.md §5 asks for this when observable behaviour changes. The only edit here rewords the quit prompt.
    Confidence 70 · mechanical · Fix: add a short check covering the header label, hero values filling in, and the Instances message.

Test verdicts. Q1 is whether the test would still pass with the production change reverted; Q2 is what stays green if one branch inside the change is broken.

  • All 21 winperf tests: Q1 — fail (the module is new). Q2 — each mutation the worker ran was caught by at least one test, except: device_key_without_luid… and empty_output… stay green under the mutations aimed at them. multi_adapter_utilization_clamps… encodes the engine-type bug above.
  • push_host_gpu_snapshot_leaves_instances…: Q1 — fails, and also fails if the snapshot is routed through push_snapshot. Q2 — dropping the has_received_gpu_snapshot assignment stays green; so does the event-loop dispatch arm.
  • in_process_version_is_recognised_by_is_in_process, header_does_not_claim_a_daemon…, draw_empty_names_the_platform_reason…, gpu_telemetry_pending_*, gpu_data_unavailable_*, gpu_utilization_gauge_shows_dash…, vram_used_pct_is_zero…: Q1 — fail. Q2 — each catches the mutation of its own branch.
  • node_throughput_title_unknown_gpu_only_when…: Q1 — passes against the pre-PR code, so it pins wording rather than testing the change.
  • hero_shows_dash_placeholders…: Q2 — stays green with the unavailable || terms removed (they are redundant). Its "—" check on POWER passes only because the line also contains the next row's text.
  • dash-37: passes on this head. Q1 — fails with home.rs reverted (both confirmed by runs). Q2 — stays green on the rail/wall zeros and on the Linux detection-window mislabel.

Decisions for the author

  • Process-wide SetDefaultDllDirectories(SYSTEM32) — tradeoff
    It closes DLL hijacking for adlx's bare-name load. But every later bare-name LoadLibrary in the rocm process becomes System32-only. Nothing loads DLLs that way today; a future DLL shipped next to rocm.exe would fail to load with no obvious cause.
  • Depending on adlx = "0.0.0-alpha.1" — tradeoff
    It is a pre-release FFI wrapper whose own helper is not a singleton, and its Drop writes to stderr on terminate failure, which lands on the TUI screen. That is weighed against writing the ADLX bindings yourself.
  • rocm-dash-collectors is an unconditional dependency of rocm-dash-tui — non-blocking-improvement
    Only the Windows arm uses it, so it could be a [target.'cfg(windows)'.dependencies] entry.
  • The in-process collector is recognised by a version-string prefix — non-blocking-improvement
    A typed ConnState variant or field would rule out a real daemon version that happens to start with in-process/.

Positive signals

  • The separate HostGpuSnapshot message plus push_host_gpu_snapshot keeps the empty Windows instances list from latching has_received_snapshot, which protects the issue [Feature]: Stop serving a model #145 quit-confirm prompt.
  • The header and Instances wording come from the same in_process_version tag the client sends, the "message computed from the decision" shape AGENTS.md §3 asks for.
  • dash-37 runs on a simulated GPU-less host root instead of depending on the runner's hardware, and checks only the text after each label, so the gauge's own "—" cannot satisfy it.
  • ADLX VRAM is trusted only when there is exactly one adapter and one reading (no matching by position), and WMI's 32-bit AdapterRAM cap is treated as unknown.
  • kill_on_drop(true) on the PowerShell child, and MissedTickBehavior::Skip on the ticker.

Deployment notes

  • New third-party crates, Windows-only: adlx 0.0.0-alpha.1 (MIT) and libloading 0.8.9 (ISC), plus a windows-sys feature. THIRD_PARTY_NOTICES.txt and MANIFEST.md match Cargo.lock.
  • On Windows, rocm dash live mode now starts a 1s in-process ticker that spawns powershell.exe and, where the AMD driver is installed, loads amdadlx64.dll, where it used to fail at connect.

What this covered

Every file in prw-base…677a2b92 (24e405a…677a2b92) was read completely. Beyond that: the callers and consumers of the changed symbols (dock.rs, hardware.rs, agent/snapshot.rs), the daemon's detection path in runner.rs, and the sibling Windows inventory in rocm-core/src/host_gpu.rs. Two read-only workers split the code passes (collector/FFI/manifests; TUI/e2e/docs). Mutation runs used cargo test -p rocm-dash-collectors --lib and -p rocm-dash-tui --lib, plus a single-scenario e2e run of dash-37.

The PR head has since moved to 9173de9. This report covers 677a2b9 only.

Did not run:

  • History — limited to git log, no blame.
  • Prior-changes — would need earlier PRs' review threads, which are out of bounds here.
  • #[cfg(windows)] code — judged by reading only; there is no Windows toolchain here.
  • Self-hosted e2e lanes — the "E2E self-hosted" run for this head is still pending (queue stall), so the GPU and Strix Windows lanes are missing evidence, not failures. All hosted CI checks passed.

Sign-off check: exit 0, 16 commits checked, none missing.

The two workers shared one checkout. The dash-tui runs compiled with the other worker's winperf mutation in place, on a code path no Linux dash-tui test runs, so their results stand.

Not reconciled against any PR discussion here.

Standing objections from the previous round (a9b08ac)

  • No test of the Windows client path (the in-process tag it sends, and that it routes to HostGpuSnapshot) — re-found blind this round; still holds. Building the tag through ConnState::in_process_version() ties the producer and consumer strings together, but nothing runs connect_and_run, so the routing to HostGpuSnapshot and the Connected message remain untested. See the second blocking item.
  • dash-37 asserted "no GPU" on hosts where the same step shows a real GPU (non-gating) — not re-found blind, and settled by putting it to the reviewer after its findings were frozen (anchored evidence): resolved in substance. The new Given a machine with no GPU step's empty ROCM_CLI_TEST_HOST_ROOT reaches the PTY spawn, and both the WSL probe and the /dev/kfd pre-flight follow it. Not yet confirmed by a WSL2-lane run at this head, and it relies on that lane building with e2e-test-hooks.
  • The description omits the Windows Instances empty-state wording (non-gating) — still omitted; folded into the third blocking item.

rominf
rominf previously requested changes Oct 27, 2026

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 9173de9

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Review — needs work

Full review of the whole change.
Implements ROCMAI-564 per the description. The ticket's text was not retrieved here; see What this covered.

Round 6. Each finding is marked new (first raised this round) or standing (raised in an earlier round and still holding). This round was a full, blind re-review of the whole change at 9173de9. The previous report (677a2b9, informational because the head moved) is settled at the end.

Why this is a change request: two standing objections still hold at this head — the untested Windows runtime path (fourth blocking item) and the engine-type utilization grouping (first non-blocking item; see Standing objections at the end). Resolving those two lifts the block. The other blocking items carry a watcher note saying why they do not gate.

Blocking

  • [standing] "Detecting GPU…" disappears on healthy Linux GPU hosts while GPU detection is still running — crates/rocm-dash-tui/src/ui/tabs/home.rs:71-86 (gpu_telemetry_pending), crates/rocm-dash-tui/src/app/mod.rs:80-84, tests/e2e-cucumber/features/dash.feature:645-647

    • The daemon runs amd-smi detection off the critical path. It says itself that detection takes "up to ~15s" (crates/rocm-dash-daemon/src/runner.rs:371-379).
    • Until detection finishes it broadcasts a snapshot every tick with gpus = [] and stays silent about it ("gpu == None is transient here", runner.rs:497-503).
    • The first of those snapshots goes through push_history_and_latest and sets the sticky has_received_gpu_snapshot. From that point the hero reads "Unknown GPU" with "—" values until detection lands.
    • Several claims therefore do not hold on the main Linux path:
      • The doc says a snapshot that "carries no GPUs … is a known state, not a still-loading one".
      • The field doc says the placeholder shows until "detection has produced a result".
      • The feature comment says the pending window "closes in well under a second".
      • The description says it shows "Detecting GPU…" "while telemetry is genuinely pending".
    • The daemon's runner is the side that governs, because it is what actually emits the empty snapshots.
    • Confidence 85 · logic · Fix: carry a "detection pending" signal in the snapshot or protocol, so an empty gpus from a still-detecting daemon keeps the placeholder. Then correct the three comments.
    • Watcher note: the previous round raised this as non-blocking, and nothing in 677a2b9..9173de9 changes it, so it does not gate.
  • [new] On a host with more than one display adapter, every phantom LUID becomes its own GPU and all ADLX data is dropped — crates/rocm-dash-collectors/src/winperf.rs:237-291, :423

    • The phantom-LUID collapse only runs when Win32_VideoController lists exactly one adapter (if let [adapter] = adapters.as_slice()). Any second adapter sends parsing to the multi-adapter branch, which emits one GpuMetrics per seen LUID, phantoms included. Common cases:
      • an iGPU+dGPU AMD system;
      • an RDP session's "Microsoft Remote Display Adapter";
      • a virtual display driver.
    • That branch also hard-codes temperature, power and VRAM total to 0.
    • A review worker ran a temporary probe test: the PR's own 780M phantom fixture plus one Remote Display Adapter line gave gpus=3 physical=2 totals=[0,0,0] temps=[0,0,0]. That is the same GPUs · 3 the description says was fixed, and it now disagrees with physical_gpu_count.
    • gpu_model = adapters.first() can also name the virtual adapter.
    • Confidence 100 · logic · Fix:
      • Drop LUIDs with zero VRAM and no utilization before counting devices.
      • Exclude non-hardware adapters when deciding how many real adapters there are.
    • Watcher note: the single-adapter-only collapse is documented in parse_metrics' doc comment, and the multi-adapter fallback was rated non-blocking in earlier rounds. The probe is new evidence of how easily that fallback is reached (one RDP adapter), not a new code path, so it does not gate on its own. It is worth an answer from the author.
  • [new] dash-37 passes even if the embedded daemon never connects — tests/e2e-cucumber/tests/e2e/dash_steps.rs:1303-1341, tests/e2e-cucumber/features/dash.feature:612-634

    • If the client reaches Disconnected before any snapshot, gpu_telemetry_pending is false, so the title reads "Unknown GPU". latest is None, so every mini-spark shows "—".
    • That is exactly what the step asserts. The scenario's claim that "every snapshot reports zero GPUs" is never exercised: the step never checks that a snapshot arrived.
    • None of the 11 single-branch mutations a worker ran in home.rs, app/mod.rs and app/types.rs would turn it red; that verdict comes from reading the step, since the scenario itself was not run here.
    • Confidence 85 · logic (test) · Fix: wait for the connected header (connected · … rocm daemon) before asserting on the hero.
    • Watcher note: this gap opened with 9173de9's Disconnected change. It does not gate: the scenario only passes vacuously when the embedded daemon fails to connect at all, a failure the other dash scenarios would also show. The fix is one extra wait.
  • [standing] The Windows runtime path this change introduces has no test on any lane — crates/rocm-dash-tui/src/client.rs:193-254 (#[cfg(windows)] connect_and_run), crates/rocm-dash-collectors/src/winperf.rs:76-123 (snapshot, run_script), :343-390 (adlx_readings)

    • The 21 winperf tests cover only the pure parsers, and all 12 parser mutations were caught.
    • Nothing exercises:
      • the warning chaining;
      • the ADLX timeout and panic mapping;
      • run_script's exit-status and timeout handling;
      • the error-to-warning-snapshot fallback in connect_and_run;
      • stable device ordering (BTreeSet). Reverting it to first-seen order stays green.
    • The description argues that e2e coverage is impractical (the PTY driver is Linux-only). It does not argue that unit coverage is impractical. The Windows lane runs cargo test, so a #[cfg(windows)] test there is feasible.
    • Confidence 85 · logic · Fix: inject the script runner and the ADLX reader behind a seam and unit-test the glue. Also pin device ordering.
    • Watcher note: gating. client.rs and winperf.rs are unchanged since 677a2b9.
  • [standing] The description omits user-visible changes and overstates one — PR description vs crates/rocm-dash-tui/src/ui/mod.rs:591-593, crates/rocm-dash-tui/src/ui/tabs/instances.rs:377-387, crates/rocm-dash-tui/src/app/types.rs:175-191, tests/e2e-cucumber/features/dash.feature:655

    • Not mentioned:
      • the new header text "live · localhost · local collectors";
      • the new Instances empty-state message;
      • the ConnState::in_process_version tagging;
      • the renumbering of the existing scenario from dash-37 to dash-38.
    • Overstated: the "genuinely pending" claim (first finding).
    • Unmentioned user-visible work is work the reader of the description will not check.
    • Confidence 85 · mechanical · Fix: narrow nothing. Add the missing items to the description and correct the pending claim.
    • Watcher note: rocm-cli squash-merges build the commit body from commit messages, not the PR description, so this item does not gate on its own.

Non-blocking

  • [standing] Utilization is summed across distinct engines of the same type — winperf.rs:157-159, :193-219, test multi_adapter_utilization_clamps_to_100_when_summed_processes_exceed_it (~:664)

    • The key is (luid, engtype), ignoring eng_N/phys_N. A worker probe showed two copy engines at 60% each reported as 100%. Task Manager's convention is sum per engine instance, then max, which gives 60%.
    • The test's comment says "same engine", but its fixture uses eng_2 and eng_3. The comment is wrong, because the code keys on type.
    • Confidence 70 · logic · Fix: key on the engine instance.
    • Watcher note: the previous round raised this as blocking. It still holds at this head and still gates; see Standing objections below.
  • [standing] ADLX is loaded and torn down every second, and a hung call leaks a blocking thread every tick — winperf.rs:85-93, :360-390

    • On timeout the spawn_blocking task keeps running, and the next tick initializes ADLX again concurrently.
    • The adlx crate's own helper notes it "should be a singleton".
    • Confidence 70 · logic · Fix: keep one long-lived helper on a dedicated thread, and skip a read while one is still in flight.
  • [standing] CPU/memory updates wait on the GPU query every tick — client.rs:~221-238, winperf.rs:42-45, :102-123

    • Each tick awaits a PowerShell spawn, two Get-Counter samples and a CIM query, plus ADLX, before sending anything.
    • So the comment that the 1 s ticker "mirrors the daemon's gpu_tick cadence" does not hold.
    • ADLX_TIMEOUT's claim that a wedged call "must not freeze CPU/memory updates" only holds per call. A single tick can stall host metrics for up to 20 s.
    • The description's follow-up covers spawn latency, but not this coupling.
    • Confidence 70 · logic · Fix: send host metrics on their own tick.
  • [standing] rocm-dash-collectors is an unconditional dependency of rocm-dash-tui but only used on Windows — crates/rocm-dash-tui/Cargo.toml:19-23

    • cargo tree with and without the edge adds 14 unused crates to rocm-dash-tui's Linux graph (bollard, hyperlocal, reqwest, sysinfo, csv, regex, …).
    • check-crate-edges passes, and was shown to fail without the allowlist entry.
    • Confidence 100 · architectural · Fix: move it to [target.'cfg(windows)'.dependencies].
  • [new] The title flickers back to "Detecting GPU…" on each reconnect attempt while the daemon is unreachable — home.rs:82-86 with client.rs:104-121

    • Every backoff cycle sends Connecting, which reads as pending again.
    • Confidence 70 · logic · Fix: make a failed attempt sticky for the placeholder decision.
  • [standing] The alpha adlx crate can panic or eprintln! while the TUI owns the alternate screen — crates/rocm-dash-collectors/Cargo.toml:39

    • AdlxHelper::new contains CStr::to_str().unwrap(), and its Drop prints on a terminate failure.
    • A panic inside spawn_blocking is caught, but the default hook still writes to stderr, which corrupts the display.
    • Confidence 70 · architectural.
  • [standing] docs/manual-testing.md has no procedure for the new Windows live telemetry — AGENTS.md §5

    • Only the quit-prompt paragraph was edited.
    • Prerequisites the user-facing docs never state:
      • temperature, power and clock need the AMD driver's ADLX;
      • utilization and VRAM depend on English counter names.
    • Confidence 70 · mechanical.

Decisions for the author

  • [standing] The in-process collector is identified by an "in-process/" prefix inside ConnState::Connected.version — non-blocking-improvement

    • A dedicated ConnState::InProcess { host } variant would make the state unrepresentable by accident.
    • It would also force every exhaustive match to handle it. observe.rs:36, for example, treats it as an ordinary connection today.
    • A real daemon whose version starts with that prefix is unlikely in practice.
  • [standing] Temperature and power read 0 as if measured whenever ADLX is unavailable — answered-by-intent

    • This covers: no AMD driver, a non-AMD GPU, every multi-adapter host, and a per-GPU query failure. There is no warning, so the hero renders "0°C 0 W".
    • Description: "degrading to 0.0/0.0/None on any failure (no driver, non-AMD GPU, unsupported metric) so the snapshot never errors out."
    • It would otherwise be non-blocking, because it sits against the PR's own "placeholders, not fake zeros" goal. Confirm that is still the intended trade.

Positive signals

  • Routing Windows ticks through a separate ClientMsg::HostGpuSnapshot rather than Event::Snapshot keeps the issue #145 quit-while-serving prompt correct. push_host_gpu_snapshot_leaves_instances_and_has_received_snapshot_untouched pins this, and fails when the branch is mutated.
  • The winperf parser tests are strong: all 12 single-branch mutations a worker applied were caught.
  • The new first-party edge is added to the check-crate-edges allowlist explicitly rather than bypassed. Hardening the bare-name amdadlx64.dll load with SetDefaultDllDirectories, and checking its return value, is a good defensive choice.

Deployment notes

  • New third-party dependencies on Windows: adlx 0.0.0-alpha.1 (MIT), libloading (ISC), windows-sys. MANIFEST and THIRD_PARTY_NOTICES are updated.
  • rocm.exe now calls SetDefaultDllDirectories(SYSTEM32) once per process. This removes the application directory from every later bare-name LoadLibrary in the process, not just ADLX's.

What this covered

What was read. Every file in prw-base...HEAD (24e405a…9173de99, merge base 63ecab2), split across two parallel workers.

  • The collectors-side worker read winperf.rs, client.rs, the manifests, crate_edges.rs and the doc claims completely, plus host.rs, runner.rs, amd_smi.rs and the adlx crate source.
  • The TUI/e2e worker read home.rs, client.rs and the full diff completely. For the large files (app/mod.rs at 4051 lines, event_loop.rs, ui/mod.rs, instances.rs, types.rs) it read the diff plus every code path it touches, not every line.
  • The driver re-read the runner's detection path and winperf's multi-adapter branch to confirm the first two findings.

What ran.

  • cargo test -p rocm-dash-tui: 943 lib tests passed, plus the integration suites.
  • cargo test -p rocm-dash-collectors --lib winperf: 21 passed.
  • 23 single-branch mutation runs.
  • feature_naming: passed. Nothing references the old dash-37.
  • cargo check --test e2e: passed.
  • Linux clippy and Windows-gnu clippy on rocm-dash-collectors: clean.
  • check-crate-edges: passed.

What did not run, or could not be checked.

  • The dash-37 e2e scenario was not run (the release build exceeded the time budget).
  • The client.rs Windows arm could not be cross-checked because the mingw C compiler is missing, so it was judged by reading.
  • The ROCMAI-564 ticket text was not retrieved.
  • Prior-changes pass: earlier PRs' review comments on these files were not read.
  • CI at the reviewed head: every reported check succeeded or was skipped. No self-hosted GPU-lane checks (Strix Windows, WSL2, Strix Ubuntu) appear in the rollup, so that is missing evidence, not a pass.
  • Sign-off check: exit 0, 17 commits checked, none missing.

Not reconciled against any PR discussion or earlier reviews here.

Standing objections from the previous round (677a2b9)

  • GPU utilization is grouped by engine type, not by engine (blocking at 677a2b9) — re-found blind this round (first non-blocking item, confidence 70, with a worker probe showing two engines at 60% read as 100%). winperf.rs is unchanged since 677a2b9. The objection still holds: the doc comment on engine_type says the suffix is "distinct per engine" and keys utilization "by (device, engine)", the code keys on the type alone, and the test that encodes the summing calls eng_2/eng_3 "the same engine". Still gating. Resolved by keying on the engine instance and correcting that test.
  • No test of the Windows connect_and_run / HostGpuSnapshot path (blocking at 677a2b9) — re-found blind this round (fourth blocking item). client.rs and winperf.rs are unchanged since 677a2b9, and no Windows-only test was added. Still holds; still gating.
  • The description does not match the change (non-gating) — re-found blind this round (fifth blocking item). Still holds; still non-gating.

…nd collectors dep

- winperf: key utilization by (device, engine instance) so two engines of the
  same type are maxed rather than summed; fix the clamp test to use one engine
  and add a two-copy-engines test.
- rocm-dash-tui: depend on rocm-dash-collectors only on Windows.
- dash-37: wait for the connected header before asserting the hero, so the
  scenario cannot pass vacuously when the daemon never connects.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
@juhovainio

Copy link
Copy Markdown
Collaborator Author

Addressed review 5667214273 in 5adbeb1.

Fixed

  • Utilization grouped by engine type (gating). winperf.rs now keys on (device, engine instance), i.e. everything after _phys_, so two copy engines at 60% each read 60%, not 100%. The clamp test now uses one engine shared by two processes (matching its comment), and a new test covers two engines of the same type.
  • dash-37 passing vacuously. The step now waits for the · rocm daemon header before asserting the hero. I matched on that rather than "connected", which "disconnected" contains.
  • rocm-dash-collectors unconditional dependency. Moved to [target.'cfg(windows)'.dependencies]; check-crate-edges still passes.

Deliberately left, for the author or a follow-up

  • No test of the Windows connect_and_run / run_script / ADLX glue (gating). This needs a seam (injectable script runner and ADLX reader) in winperf.rs and client.rs. That is a refactor I did not want to make without being able to run it on Windows. Leaving it as a decision for the author.
  • "Detecting GPU…" while the daemon is still detecting. Needs a pending signal in the snapshot/protocol; larger than this PR's scope.
  • Multi-adapter phantom LUIDs, ADLX singleton and leaked blocking thread, host metrics coupled to the GPU query tick, reconnect flicker, adlx stderr/panic hook, manual-testing.md procedure, ConnState variant, PR description gaps. Not changed here; each is a design call or follow-up.

Local checks: cargo fmt, clippy -D warnings on the touched crates, check-crate-edges, and the winperf tests pass. The pre-push hook failed three times on unrelated flaky apps/rocm comfyui/therock tests (a different one each run), so this was pushed with --no-verify at the operator's request.

@tomastola tomastola left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of 5adbeb1 — changes requested

My previous review (APPROVED at 9173de99) raised no blockers. The new commit fixes another reviewer's gating finding correctly, but its dash-37 change breaks the scenario on every run.

Fixed by 5adbeb1

  • GPU utilization is now keyed per engine instance, not per engine type (crates/rocm-dash-collectors/src/winperf.rs, engine_id).
    • The key is now everything after _phys_, for example 0_eng_2_engtype_copy. Two copy engines at 60% each now read 60%, not 100%.
    • The clamp test now uses one engine shared by two processes, which matches its comment.
    • The new test utilization_across_two_engines_of_the_same_type_takes_max_not_sum does its job. I swapped the old engine-type keying back in and that test failed. The winperf suite passes: 22 tests.
  • rocm-dash-collectors is now a Windows-only dependency of rocm-dash-tui. Its only use is the #[cfg(windows)] block in client.rs:202-203. The Linux build compiles without it.

Blocking

  • dash-37 now fails on every run: the new header wait can never match at the default pty size (tests/e2e-cucumber/tests/e2e/dash_steps.rs:1306-1317).
    • The new wait_for_screen_where(… screen.contains("· rocm daemon") …) runs before use_wide_size(), so the pty is still 80×24.
    • The header line is rocm.ai → unix:/tmp/rocm-e2e-XXXXXX/runtime/rocmdashd.sock connected · <host> · rocm daemon <ver>. The socket path alone takes about 50 columns, so the line is cut off before rocm daemon.
    • I ran it locally (cargo xtask e2e -- -n dash-37, debug rocm with e2e-test-hooks). It failed at that step after 30s: the dashboard never connected to its daemon: timed out after 30s waiting until the connected header. The last screen shows … rocmdashd.sock connected · r │. The dashboard was connected and the hero already read Unknown GPU; only the wait was wrong.
    • The commit message says the dash-37 scenario was not run before pushing, and the push skipped the pre-push hook.
    • Fix: move the header wait after tui.use_wide_size(), so it runs at 180 columns. I moved those lines in a local copy and the scenario passed, 6/6 steps. Matching on · rocm daemon is still the right idea, because connected is also a substring of disconnected.

Non-blocking

  • The Cargo.toml comments now sit above the wrong tables (crates/rocm-dash-tui/Cargo.toml:55-63).
    • The new [target.'cfg(windows)'.dependencies] table was placed between the libc comment ("Test-only, Unix-only: raise(3)…") and the [target.'cfg(unix)'.dev-dependencies] table it describes.
    • The result reads as if rocm-dash-collectors is a test-only Unix dependency, and libc loses its explanation.
    • Fix: put the libc comment back directly above [target.'cfg(unix)'.dev-dependencies], and keep only the "Only client.rs's #[cfg(windows)] arm…" comment above the Windows table.

Still open from earlier rounds (non-blocking, unchanged by this commit)

The author's reply says these were left on purpose. I agree none of them gate:

  • A timed-out ADLX read leaks its blocking thread, and ADLX is re-initialised on every tick.
  • A single phantom WMI row defeats the single-adapter collapse.
  • SetDefaultDllDirectories(SYSTEM32) changes the DLL search path for the whole process.
  • No tests cover the Windows tick loop or the snapshot() error paths.
  • crates/rocm-dash-collectors/src/lib.rs:5 still says the crate is "Used by rocm-dash-daemon".

For the author

  • No test runs the Windows connect_and_run path. Another reviewer gates on this, and it has been left as a decision for you. I didn't gate on it in my earlier rounds and I don't now. The real Windows glue still has no automated coverage; a test there would need a seam (an injectable script runner and ADLX reader).
  • Scenario renumbering (dash-37 ↔ dash-38): please still confirm that nothing outside the repo cites the old numbers.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 5adbeb1

Review

Adds a new WinPerfCollector (PowerShell Get-Counter/WMI + optional ADLX) so the dashboard TUI streams live GPU/CPU/memory on Windows without a rocmdashd socket, plus a div-by-zero fix for the hero VRAM percentage and new loading-placeholder states. The implementation is unusually well-documented and well-tested (16+ unit tests derived from real-hardware captures, a dedicated regression test for the Unix-only instance-registry non-interference, DLL-search-order hardening before the ADLX load), but there is one real correctness gap plus a failing CI check on this exact commit, so this needs work before merge. Implements ROCMAI-564 ("Device metrics for TUI (GPU/memory) on Windows"); the ticket's scope (identify a Windows metrics source, wire it into the dashboard pipeline, validate on real/Strix-Halo-class hardware) is substantively covered by this PR.

Blocking

  • E2E tests CI check fails on the reviewed commit — gh pr checks 715 reports E2E tests fail 5m5s at head 5adbeb1b, while every other check (clippy, unit tests, coverage, windows-build-and-test, commit-signatures, license/notices, lychee, Sphinx) passes. A red check is "not ready" regardless of cause; the log could not be fetched here (REST API access to this repo is blocked in this environment), so the specific failure needs to be read directly and resolved (or, if it's an infra flake, re-run and confirmed clean) before this is mergeable.
    Confidence 100 (direct gh pr checks result) · category: logic · Fix: open the failing run's log and fix the regression, or re-run and document the flake.

  • PowerShell's culture-formatted ToString() can silently zero out GPU utilization/VRAM on non-English-decimal Windows locales — crates/rocm-dash-collectors/src/winperf.rs:54,56 interpolates $($_.CookedValue) (a .NET double) directly into the emitted line; unformatted interpolation renders under CultureInfo.CurrentCulture, so on locales using a comma decimal separator (common across continental Europe, Latin America, Russia, etc.) a reading like 12.5 comes out as 12,5. The Rust side (winperf.rs:217,221) then does value.trim().parse().unwrap_or(0.0), so the comma-formatted value fails to parse and silently becomes 0.0 — a plausible-looking but wrong reading, not an error. counter_samples_missing_warning (same file, ~line 439) only fires when metrics.is_empty(), so this specific failure mode produces no warning at all. The file's own doc comments show real care about counter-name localization (Get-Counter paths are English-only) but not about number-formatting localization, so this reads as a genuine gap rather than a deliberate tradeoff — and it reproduces exactly the "fake zero reading shown as real" failure class this same PR works hard to eliminate elsewhere (the hero-tile "—" placeholder fix, the e2e assertion that checks for "—" rather than a fake reading).
    Confidence 78 · logic · Fix: force invariant culture in the script (e.g. [Threading.Thread]::CurrentThread.CurrentCulture = [Globalization.CultureInfo]::InvariantCulture at the top of WINPERF_SCRIPT, or format each value explicitly with .ToString([System.Globalization.CultureInfo]::InvariantCulture)).

Non-blocking

  • A wedged ADLX/driver call leaks one blocking-pool thread per tick instead of being bounded — crates/rocm-dash-collectors/src/winperf.rs:86-93 wraps task::spawn_blocking(adlx_readings) in timeout(ADLX_TIMEOUT, ...), but spawn_blocking work cannot be cancelled once running: timing out only stops awaiting the JoinHandle, the pool thread executing the actual FFI call into amdadlx64.dll keeps running indefinitely if the driver call hangs. On a persistently wedged driver, each 1s tick consumes one more permanently-blocked thread from tokio's bounded blocking pool (default cap 512), which would eventually also stall the cross-platform HostCollector CPU/memory ticks this design explicitly set out to protect ("a wedged driver call must not freeze CPU/memory updates forever", same file's own comment at line 82-83). This requires a sustained (minutes-long) hang to actually bite, and the short-term degradation (0.0/None + warning) already works correctly.
    Confidence 75 · architectural · Fix: document the tradeoff explicitly, and/or add a circuit breaker (stop re-spawning ADLX after N consecutive timeouts) so a wedged driver degrades without continuing to consume pool capacity.

Decisions for the author

  • powershell.exe-per-tick vs. a native WMI/Pdh* API — tradeoff, already disclosed. The PR's own "Follow-up" section states this plainly: a native rewrite would cut per-tick latency and the 15s spawn-timeout handling, at the cost of more unsafe/platform-specific code, and that it's deferred because the current approach already works on real hardware. No action needed; flagging only because it was independently surfaced by the review before the description was read, and the two agree.
  • vram_total_mb is only ever populated for the single-real-adapter case; any multi-adapter host (including AMD's own hybrid iGPU+discrete laptops, which this repo's own docs/manual-testing.md treats as a real, tested configuration elsewhere) degrades VRAM-total/temp/power/clock to the "—" placeholder entirely — tradeoff, explicitly documented in both the code (parse_metrics's module doc, "realistic target, an integrated AMD APU") and the PR description ("multi-GPU hosts degrade to 0 rather than risk a wrong number"). Worth the author confirming this is an acceptable v1 scope given that hybrid-graphics laptops are a fairly common AMD Windows configuration, not just a rare desktop multi-GPU case — the stated upgrade path (DXGI/LUID-based correlation) is the way out.
  • adlx = "0.0.0-alpha.1" — tradeoff. A pre-1.0 alpha dependency is the only way to get AMD temperature/power/clock on Windows without hand-writing ADLX FFI bindings; worth tracking as a supply-chain/stability item but not a reason to block this PR.
  • ConnState::is_in_process() is detected by string-prefix-matching daemon_version ("in-process/" + version) — non-blocking-improvement. Works and is tested, but is a stringly-typed side channel rather than a dedicated enum variant/field; a real daemon whose version happened to start with "in-process/" would be misclassified (unlikely in practice).
  • The PR description's utilization-aggregation bullet is slightly stale — it still describes keying "via device_key/engine_type", which was the pre-final-commit behavior; the actual head commit (5adbeb1) renamed this to key by (device, engine_id) — i.e., per physical engine instance rather than per engine type — specifically to stop two same-type engines (e.g. two copy engines) from being summed into one bucket. The described overall behavior (sum within one engine, max across distinct engines) still holds; only the keying-granularity wording is out of date. Minor, not blocking.

Positive signals

  • crates/rocm-dash-collectors/src/winperf.rs documents the why behind nearly every non-obvious decision (pid-prefix stripping, phantom-LUID collapse, the 32-bit AdapterRAM truncation heuristic, the single-vs-multi-adapter ADLX trust boundary, the 15s timeout tuned against a real Strix Halo laptop's spawn latency) and backs each with a unit test, several derived from real-hardware captures (an AMD Radeon 780M laptop).
  • SetDefaultDllDirectories(LOAD_LIBRARY_SEARCH_SYSTEM32) guarded by std::sync::Once proactively closes a DLL-planting hijack vector before the bare-name amdadlx64.dll load that the adlx crate performs — a security-conscious detail nobody asked for.
  • The Windows path deliberately avoids Event::Snapshot/has_received_snapshot and routes through a new ClientMsg::HostGpuSnapshot so it can never falsely latch the issue #145 "quit while serving" confirmation prompt; this is backed by a dedicated test (push_host_gpu_snapshot_leaves_instances_and_has_received_snapshot_untouched) that asserts the non-interference directly rather than leaving it implicit.
  • The hero VRAM percentage div-by-zero fix (vram_used_pct) is deduplicated into one function shared by both the label and its sparkline history, documents the exact failure mode it fixes ("80800.0%"), and ships with a regression test.
  • Dependency/license hygiene is complete in the same change: Cargo.lock, MANIFEST.md, and THIRD_PARTY_NOTICES.txt are all updated together for the new adlx/windows-sys/libloading dependencies, and the new rocm-dash-tui -> rocm-dash-collectors crate edge is added to xtask/src/crate_edges.rs's allowlist in the same commit.
  • The test plan is honest about a real gap: it explicitly states the Windows collector path isn't exercised by this repo's e2e harness (the PTY driver is Linux-only for now) rather than silently leaving the gap unstated.

Deployment notes

  • New Windows-only runtime dependencies: adlx (pre-1.0 alpha) and windows-sys for rocm-dash-collectors; rocm-dash-collectors becomes a Windows-only dependency of rocm-dash-tui. Both are correctly scoped under [target.'cfg(windows)'.dependencies] and do not affect Linux builds.
  • Behavior change (publicly documented in README.md, docs/manual-testing.md, docs/rocm-docs/install/installation.md): the live dashboard now shows GPU/CPU/memory telemetry on native Windows; the serving-instance registry view remains Linux/WSL2-only since it still requires the rocmdashd Unix socket.
  • No data migrations.

What this covered

Full diff for 63ecab2d…5adbeb1b (merge-base of origin/main and the PR head), all 22 changed files read completely. Fanned out to three parallel workers grouped by coupling: (1) rocm-dash-collectors/src/winperf.rs + its Cargo.toml/lib.rs, (2) the rocm-dash-tui client/app-state machine (client.rs, app/mod.rs, app/types.rs, event_loop.rs, registry_probe.rs), (3) the rocm-dash-tui UI layer (ui/mod.rs, ui/tabs/home.rs, ui/tabs/instances.rs); I personally reviewed the remaining files (Cargo.lock, MANIFEST.md, README.md, THIRD_PARTY_NOTICES.txt, apps/rocm/src/dash.rs, both Cargo.toml dependency changes, docs/manual-testing.md, docs/rocm-docs/install/installation.md, tests/e2e-cucumber/features/dash.feature, tests/e2e-cucumber/tests/e2e/dash_steps.rs, xtask/src/crate_edges.rs), the repo's AGENTS.md for agent-instruction adherence, and this PR's own commit history (git log/git show, not review-thread discussion) for the history pass. Retrieved ROCMAI-564 from Jira directly by identifier (status: In Progress; scope matches what this PR implements).

CI checked via gh pr checks 715 at the current head: E2E tests fails, all other listed checks pass (Skill checks (skillscope) and Regenerate manifests report skipping, not failures). Did not execute cargo test/cargo clippy/the e2e suite locally in this environment; took the CI results at face value per the review method rather than re-deriving them, and did not read workflow YAML to guess what the failing job covers. The "prior changes" pass (do earlier review comments on these files still apply) was not run as a distinct step, since the review method requires reaching findings blind — this report does not reconcile against any of the PR's existing review threads (from rominf, tomastola, juhovainio, or earlier rounds of this same reviewer).

…omment placement

The connected-header wait ran at 80 columns, where the socket path truncates the line before 'rocm daemon'. Move it after use_wide_size. Restore the libc comment above its table and update the collectors crate doc.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
@juhovainio

Copy link
Copy Markdown
Collaborator Author

Addressed review 5670322164 in d2cea39.

  • dash-37 header wait (blocking): moved the · rocm daemon wait after use_wide_size() so it runs at 180 columns. Ran cargo xtask e2e -- -n dash-37 locally: 6/6 steps pass.
  • Cargo.toml comments (non-blocking): the libc comment is back above [target.'cfg(unix)'.dev-dependencies]; only the client.rs note sits above the Windows table.
  • rocm-dash-collectors/src/lib.rs crate doc: now says it is also used by rocm-dash-tui on Windows.

Left unchanged on purpose:

  • Locale-formatted CookedValue (review 5670478458): PowerShell string interpolation converts numbers with the invariant culture, so "$($_.CookedValue)" emits 12.5 on comma-decimal locales (the culture-sensitive conversion only applies to explicit .ToString() or cmdlet formatting). I could not test on a non-English Windows locale. If someone can confirm a comma, I will force invariant culture in WINPERF_SCRIPT.
  • E2E check on 5adbeb1: that failure was the dash-37 wait above; CI on the new head should confirm.
  • ADLX blocking-thread leak / circuit breaker, phantom-row collapse, SetDefaultDllDirectories scope, Windows connect_and_run test seam, "in-process/" prefix detection, and the dash-37/38 renumbering check: unchanged, as already decided in earlier rounds.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · d2cea39

Review — needs work

Full review of the whole change.
Implements ROCMAI-564 (Windows GPU/memory metrics for the TUI dashboard) via a client-side in-process collector rather than a daemon transport; ticket status is "In Progress" and the PR does not claim to close it, so no mismatch there.

Blocking

  • New device_id format collides under the compact multi-GPU view's truncation, making every row on a multi-adapter Windows host render identically — crates/rocm-dash-collectors/src/winperf.rs:282 (produces format!("win-gpu-{i}"), 9+ chars), crates/rocm-dash-tui/src/ui/tabs/hardware.rs:388 (trunc(&g.device_id, 7), no ellipsis, first-7-chars only).
    "win-gpu-0", "win-gpu-1", … all share the same 7-char prefix "win-gpu", so trunc collapses every GPU's label to the identical string in the compact multi-GPU list (the one place meant to let a user see several GPUs at a glance). amd_smi.rs's existing short "gpu-{id}" format (crates/rocm-dash-collectors/src/amd_smi.rs:233) fits the same 7-char budget untruncated, which is presumably why this was never caught — this is new breakage inflicted by the new format on an unchanged caller (hardware.rs itself isn't touched by this diff). The non-compact per-GPU detail panel (hardware.rs:456, format!("GPU {}", g.device_id)) is unaffected since it doesn't truncate. This PR's own multi-adapter branch (the LUID-collapse logic at winperf.rs:275-284) is exactly the code path that produces 2+ of these ids, so this isn't a hypothetical.
    Confidence 85 · logic · Fix: either keep device_id within 7 distinguishable characters (e.g. "win-{i}"), or widen/adjust trunc's compact-view budget for this prefix, or right-align/preserve the numeric suffix when truncating instead of cutting from the end of a longer common prefix.

  • Collector warnings meant for the user are silently dropped whenever any GPU is still reported — crates/rocm-dash-tui/src/ui/tabs/hardware.rs:225-246 (draw_gpus only reads snap.warnings inside the snap.gpus.is_empty() branch), crates/rocm-dash-tui/src/client.rs:227-244 (the Ok branch of gpu.snapshot() never logs warnings, only the Err branch at line 230 does).
    WinPerfCollector::snapshot()'s own doc comment (crates/rocm-dash-collectors/src/winperf.rs:74-76) states: "The returned Vec<String> surfaces non-fatal problems the caller should show the user (e.g. a locale mismatch, see counter_samples_missing_warning) without failing the whole tick." The realistic partial-failure case — e.g. a non-English-locale Get-Counter mismatch producing a warning while Win32_VideoController-sourced GPU inventory still succeeds — ends up with a non-empty snap.gpus and a non-empty snap.warnings that is neither rendered nor logged anywhere. This directly contradicts the collector's own stated contract.
    Confidence 85 · logic · Fix: surface snap.warnings in draw_gpus regardless of whether gpus is empty (e.g. an additional warning line/badge), and/or log them via client.rs's success path the same way the error path already does.

  • No CI results exist for the reviewed head commit beyond a docs build. gh pr view 715 --json statusCheckRollup / gh pr checks 715 both return exactly one check — docs/readthedocs.com:... (SUCCESS) — for commit d2cea392. None of this repo's own mandated gates (cargo test --workspace --all-targets, cargo clippy --workspace --all-targets -- -D warnings, the commit-signatures gate, e2e) have reported any result for this commit, so none of them can be taken at face value here; this isn't a claim that they fail, only that there is nothing to point to confirming they passed.
    Confidence 85 (directly queried, not assumed) · architectural · Fix: confirm CI actually ran for this commit (re-push/re-trigger if needed) and get it to green before merge, per this repo's own "CI Ownership: Drive To Green" policy.

Non-blocking

  • pub mod winperf; is not itself #[cfg(windows)]-gated — crates/rocm-dash-collectors/src/lib.rs:23. The module's own doc comment (winperf.rs:13-21) already identifies and explains the exact risk: under WSL interop powershell.exe is reachable from a Linux-side process, so a future Linux-reachable caller of WinPerfCollector::snapshot() would silently start querying the Windows host's GPUs instead of failing. No current caller does this (rocm-dash-daemon, which runs on Linux and depends on this crate, never references winperf), and the ADLX-specific code within the file is correctly #[cfg(windows)]/#[cfg(not(windows))]-split so the crate still compiles cross-platform — so this is latent, not exploited today, and already flagged for future readers rather than hidden.
    Confidence 72 · architectural · Fix: consider gating the module declaration itself (#[cfg(windows)] pub mod winperf;) so a future accidental Linux-reachable caller fails to compile rather than fails at runtime only under the specific WSL case.

Decisions for the author

  • Per-tick powershell.exe shell-out for GPU telemetry — tradeoff. The 15s timeout and single-spawn consolidation already mitigate the main cost (process-spawn latency, not query latency), and the PR's own description names a native wmi/raw-Pdh*-FFI rewrite as a known follow-up, explicitly deferred as not worth the added unsafe surface until tick latency is actually a problem in practice. Reasonable as shipped; flagging only so the author can confirm this is still the intended deferral rather than an oversight.

Positive signals

  • The Windows arm of connect_and_run deliberately routes through ClientMsg::HostGpuSnapshot rather than Event::Snapshot specifically to avoid latching has_received_snapshot/instances on an always-empty instance registry — traced end-to-end (client.rs → app/mod.rs's push_host_gpu_snapshot → registry_probe.rs/has_live_instance_with) and it holds; this keeps the issue #145 "quit while serving" confirmation working correctly on Windows, and is backed by dedicated unit tests for exactly this boundary.
  • winperf.rs's parsing/keying logic (per-process-pid-prefix stripping, phantom-LUID collapse, the 32-bit AdapterRAM truncation-cap heuristic, sum-then-max multi-engine aggregation) is unusually thoroughly unit-tested — each test's comment states the specific mutation/regression it guards against.
  • The new Windows-only dependencies (adlx, windows-sys) are correctly scoped under [target.'cfg(windows)'.dependencies] in both Cargo.tomls, matched by a correct xtask/src/crate_edges.rs allowlist entry, and backed by correct MANIFEST.md/THIRD_PARTY_NOTICES.txt updates.
  • The home.rs total > 0 VRAM-percentage fix is applied consistently at every call site in that file (no partial-fix pattern left over from the old total.max(1)), and the new gpu_telemetry_pending/gpu_data_unavailable UI states were traced through their full truth table without finding a reachable contradictory combination.
  • The new e2e scenario (dash-37) asserts per-label text (not just that something rendered) specifically to avoid false-passing against an adjacent gauge's own "—", and the repo's documentation of the new Windows behavior (README, docs/manual-testing.md, docs/rocm-docs/install/installation.md) is consistent across all three places it repeats the same claim.

Deployment notes

  • New runtime dependency on the AMD driver's bundled amdadlx64.dll for GPU temperature/power/clock on Windows; absence (no AMD driver, non-AMD GPU) is handled by degrading those three fields to 0.0/None rather than erroring, which is a deliberate, documented choice rather than an oversight.
  • Windows hosts with more than one real GPU adapter get vram_total_mb = 0 for all GPUs (explicitly documented as a known limitation in the PR description) and, per the first Blocking finding above, indistinguishable compact-view labels.

What this covered

Every file in the diff, read completely, against d59f596f6e08297c207f008021eecb37defc3558...d2cea3925d4a8be266ea525c895a6c3f82c53d4d: crates/rocm-dash-collectors/src/{winperf.rs,lib.rs} and its Cargo.toml; crates/rocm-dash-tui/src/{client.rs,app/mod.rs,app/types.rs,app/event_loop.rs,registry_probe.rs} and its Cargo.toml; crates/rocm-dash-tui/src/ui/{mod.rs,tabs/home.rs,tabs/instances.rs}; docs/manual-testing.md, docs/rocm-docs/install/installation.md, README.md, apps/rocm/src/dash.rs, xtask/src/crate_edges.rs, MANIFEST.md, Cargo.lock, THIRD_PARTY_NOTICES.txt, tests/e2e-cucumber/features/dash.feature, tests/e2e-cucumber/tests/e2e/dash_steps.rs — plus surrounding/unchanged code needed to judge them in place, including crates/rocm-dash-tui/src/ui/tabs/hardware.rs (not itself in the diff, but where the first Blocking finding's breakage lands) and sibling patterns in amd_smi.rs/widgets.rs/dock.rs.

Four independent workers reviewed coupled file groups (collectors core; TUI client/app state; UI tabs; docs/tests/misc), each reading its group completely with no splits needed. Design review (Step 7) was folded into the above rather than run as fully separate independent workers, given the depth already available from the fan-out; the tradeoff item above was the one question with real substance.

Did not run: prior-changes pass — this PR shows internal history of "round 2/3 review findings" commits, but no prior review's actual content is available to this pass and none was read (discussion is out of scope by design). History pass ran only via git log/git blame spot-checks, not a dedicated worker. No Rust toolchain was available to any worker, so nothing was compiled, clippy'd, or executed — every finding above is argued from static reading except the CI-check finding, which was obtained by direct gh query. Commit signature verification could not be checked locally (no local gpg) and was not re-derived from GitHub's own "Verified" status; see the CI-results finding, which already covers this gap at the PR level.
Not reconciled against any discussion on this PR.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>

# Conflicts:
#	MANIFEST.md

@tomastola tomastola left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no CI on the head, and the PR conflicts with main.

  • d2cea392 has had no CI run. Only the readthedocs docs check reports on it.
  • The last CI run, on 5adbeb1b, failed on dash-37 (dash-reports-no-gpu-telemetry). d2cea392 is the fix for that failure, and nothing has run it yet.
  • The PR is CONFLICTING and 27 commits behind main.

Please merge or rebase main, then wait for CI on the new head and check that dash-37 passes.

Non-blocking: the description still says utilization is "sum-then-max via device_key/engine_type keying" (the bullet on multi-engine GPUs and the "GPU utilization %" table row). Commit 5adbeb1b changed this to keying per (device, engine instance), as winperf.rs:188 and its tests show. Please update the description.

@rominf

rominf commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

🔴 Automated review · pr-review-watcher · 5adbeb1

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Informational — describes 5adbeb1, not the current head. The head moved to d2cea39 while this round ran, so these findings describe an older commit, are not anchored to the current diff, do not gate, and file no change request. d2cea39 (which touches dash_steps.rs, Cargo.toml and the collectors lib.rs) was not reviewed here; a later round will review it.

Round 7. Standing objection not settled this round: the round-6 blocker "no test of the Windows connect_and_run / run_script / ADLX glue" was not re-found by this blind pass and was not put to the reviewer before the head moved, so it remains open and carries into the next round. The round-6 blocker on per-engine-type utilization grouping was not re-found; it will be settled against the current head next round. Between this comment and that round, neither is gating.

Review — needs work

Full review of the whole change.
Implements Windows GPU/memory telemetry for the dashboard from ROCMAI-564. The ticket's validation on Strix Halo / Windows HIP-SDK hosts is not claimed: the description reports testing on a Radeon 780M only.

Blocking

  • [new] The new scenario dash-37 can never pass in CI, so the required "E2E tests" check is red — tests/e2e-cucumber/tests/e2e/dash_steps.rs:1311-1317
    • The 5adbeb1 commit added a wait for screen.contains("· rocm daemon") before use_wide_size(). At the driver's default 80x24 size, the header is cut off before that text.
    • The failure screen in CI run 40533244915 (job 122295425396) shows …/rocmdashd.sock connected · r. The daemon was connected, but the predicate never matched and the step timed out after 30s.
    • Reconciliation reports 'dash-reports-no-gpu-telemetry' … FAILED — a regression. The other two failures in that run (serve-01/02) are expected failures (xfail). Main's latest CI run (40647472984) has E2E green, and the scenario is new in this PR.
    • Confidence 100 · logic · Fix: call use_wide_size() before waiting for the connected header. Alternatively, match "connected ·" and exclude "disconnected", so the predicate does not depend on the header's width.
  • [standing] The description leaves out user-visible changes this PR makes — crates/rocm-dash-tui/src/ui/mod.rs:591-593, crates/rocm-dash-tui/src/ui/tabs/instances.rs (draw_empty)
    • On Windows the header now reads live · localhost · local collectors.
    • The Instances tab's empty state now reads no instance registry on this platform · vLLM containers aren't available here.
    • The supporting ConnState::in_process_version/is_in_process machinery in app/types.rs is also new.
    • The description's per-file list covers client.rs, home.rs, docs, Cargo and xtask, but none of these. Readers use that list to decide what to check.
    • Confidence 85 · mechanical · Fix: add these changes to the description.
    • Watcher note: rocm-cli squash merges build the commit body from commit messages, not the PR description, so this item does not gate on its own.

Non-blocking

  • [new] Comment now describes the wrong dependency — crates/rocm-dash-tui/Cargo.toml:55-63
    • The new [target.'cfg(windows)'.dependencies] block was inserted between the libc dev-dependency's comment and libc itself.
    • As a result, rocm-dash-collectors (a Windows runtime dependency) now sits under "Test-only, Unix-only: raise(3)… Not a runtime dependency", and libc has lost its explanation.
    • The code governs here: move the three raise(3) lines back above [target.'cfg(unix)'.dev-dependencies].
    • Confidence 95 · mechanical.
  • [standing] "Detecting GPU…" comes back on every reconnect attempt when the daemon is unreachable — crates/rocm-dash-tui/src/ui/tabs/home.rs:82-86, client.rs:105, app/event_loop.rs:1035
    • gpu_telemetry_pending only excludes Disconnected.
    • The retry loop sends ClientMsg::Connecting before each attempt, so the title flips back to "Detecting GPU…" every backoff cycle. Commit 9173de9 set out to remove exactly this.
    • gpu_telemetry_not_pending_when_disconnected_before_any_snapshot never exercises Disconnected → Connecting.
    • Confidence 80 (traced by reading) · logic · Fix: latch "a connection attempt has failed", and add the transition to the test.
  • [new] The dock still shows the fake zero that the hero now hides — crates/rocm-dash-tui/src/ui/dock.rs:236-245
    • With a zero-GPU snapshot, the hero now shows "—", but the CONTEXT dock on the same screen still prints peak util 0.0%. A probe render at 180x45 showed this.
    • This is the twin of the gpu_data_unavailable rule the PR applies to the hero. The dock code predates the PR, but the PR makes the two disagree.
    • Confidence 85 (probe run) · logic · Fix: use gpu_data_unavailable in the dock too.
  • [standing] A wedged ADLX call leaks a blocking thread on every tick — crates/rocm-dash-collectors/src/winperf.rs:85-93
    • timeout(ADLX_TIMEOUT, task::spawn_blocking(adlx_readings)) stops waiting but cannot cancel the blocking task.
    • Each following tick starts another AdlxHelper::new() against the same wedged driver. Threads accumulate toward tokio's blocking-pool cap, and after that the TUI's other spawn_blocking/tokio::fs work queues behind them.
    • Confidence 70 (reasoned from tokio semantics, not run on Windows) · logic · Fix: skip ADLX while an earlier call is still in flight, using an atomic guard.
  • [standing] No manual-testing step for the new Windows live dashboard — docs/manual-testing.md
    • AGENTS.md §5 asks for docs/manual-testing.md to change alongside observable behaviour. The PR only rewords the quit-prompt paragraph.
    • The PR text does argue the Gherkin gap: the Windows PTY driver is not a blocking contract. That leaves the Windows behaviour with no automated or manual check written down.
    • Confidence 70 · mechanical · Fix: add a Windows live-dashboard check covering the header text, GPU/CPU/memory values and the Instances empty state.
  • [new] The rocm-dash-collectors crate doc is now stale — crates/rocm-dash-collectors/src/lib.rs:5-8
    • It says the crate is "Used by rocm-dash-daemon" and that every collector implements a core trait.
    • It is now also used by rocm-dash-tui on Windows, and WinPerfCollector implements no trait.
    • Confidence 70 · mechanical.

Decisions for the author

  • [new] In-process collection in the TUI vs. a Windows daemon transport — tradeoff.
    • For: the in-process arm ships Windows telemetry now without new IPC, and keeps the quit-confirm prompt safe via HostGpuSnapshot.
    • Against: the TUI now depends on rocm-dash-collectors (the new allowlisted crate edge), and Windows and Linux take different data paths, which will have to converge when a Windows daemon lands.
  • [standing] CPU/memory refresh is tied to the PowerShell spawn — non-blocking-improvement (client.rs, Windows connect_and_run).
    • host.tick() is only sent together with gpu.snapshot().await, which can take up to 15s plus 5s. On slow-spawn hosts, CPU/memory therefore update at PowerShell's pace, not every 1s.
    • The loop also runs in interactive rocm chat sessions, which spawn PowerShell continuously as a result.
    • Options:
      • run host and GPU sampling on separate timers;
      • keep one ADLX helper alive instead of creating and dropping one every tick;
      • query PDH directly through windows-sys.
  • [new] Depending on adlx = "0.0.0-alpha.1" — tradeoff. It is a third-party FFI crate at alpha. It gives temperature, power and clock today, at the cost of pinning an unstable binding into the dashboard's Windows path.

Positive signals

  • ClientMsg::HostGpuSnapshot is a separate message from Event::Snapshot, so the empty Windows instances list can never become the quit-confirm prompt's source of truth. push_host_gpu_snapshot_leaves_instances_and_has_received_snapshot_untouched pins this.
  • The winperf.rs parsing is pure, and ADLX readings are passed in as a parameter. Every keying, collapse and fallback branch is therefore unit-tested on Linux CI, and each test fails when its own branch is reverted.
  • vram_used_pct replaces total.max(1) with a real total > 0 guard, ending the "80800.0%" hero reading.

Deployment notes

  • New Windows-only dependencies: adlx (MIT) and libloading (ISC), plus windows-sys. The rocm-dash-tui → rocm-dash-collectors crate edge is allowlisted in xtask/src/crate_edges.rs and needs a reviewer's explicit acceptance.
  • At runtime, ADLX loads the AMD driver's amdadlx64.dll when present, after hardening the DLL search path.

What this covered

What was read and run

  • Every file in prw-base...5adbeb1b (merge base 63ecab2), all 22 files, read completely.
  • Two independent code-pass workers:
    • the Windows collector, dependency and licence group;
    • the dash-tui, e2e and docs group.
  • The second worker ran cargo test -p rocm-dash-tui --lib with the new and changed tests (26 passed), plus a probe render.
  • #[cfg(windows)] code was not compiled here. CI's windows-build-and-test and clippy were green.

CI state

  • CI rollup for 5adbeb1: the only failure is "E2E tests". It was compared against main's run 40647472984, where E2E is green.
  • The self-hosted GPU-lane checks have no check-run on this commit.

Passes that did not run, or ran reduced

  • Prior-changes pass (review comments on earlier PRs touching these files): not run, to keep within the time budget.
  • History pass: only a light git log over the touched TUI files.
  • Design review (Step 7): done by the driver rather than independent workers, because of the 2-worker cap.

Other checks

  • The sign-off check passed: 18 commits checked, none missing.

Scope caveats

  • The PR head has since moved to d2cea39. This report covers 5adbeb1 only.
  • Not reconciled against the PR's review discussion.

@tomastola tomastola left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The conflict is resolved: 3679d0e3 merges main, the PR is mergeable, and CI is now running on the head. The only new commit is that merge, so the PR's own 22 files are unchanged. My change request is lifted once CI is green, including the dash-37 E2E that d2cea392 was meant to fix. Checks still running when I looked: clippy, Analyze (rust), Coverage, and Third-party notices.

Still open, non-blocking: the description still says utilization is "sum-then-max via device_key/engine_type keying" (the multi-engine bullet and the "GPU utilization %" table row). Since 5adbeb1b the code keys per (device, engine instance), as winperf.rs:188 and its tests show. Please update the description.

@juhovainio
juhovainio dismissed tomastola’s stale review October 28, 2026 13:52

Addressed: main merged in 3679d0e (PR mergeable, CI green); description corrected to the per-(device, engine instance) utilization keying.

@juhovainio
juhovainio requested a review from tomastola October 28, 2026 13:52
Signed-off-by: Juho Vainio <juho.vainio@amd.com>

# Conflicts:
#	THIRD_PARTY_NOTICES.txt
…> after merge

Signed-off-by: Juho Vainio <juho.vainio@amd.com>

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 2c5a608

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (clean base-vs-head diff: 6ab12083e7dd7901eb54e6a4a5b0fd362f4d8eb8…2c5a608e545ea4383155a4c8c50d4ece556b992b, 22 files, 1568 insertions / 67 deletions). The branch merged main in twice mid-review, so a plain d2cea392..head range is polluted with unrelated upstream commits; this review used the merge-base-to-head diff instead, which is the same diff GitHub itself shows.
Implements ROCMAI-564 ("Device metrics for TUI (GPU/memory) on Windows", status In Progress) via the in-process/no-daemon-transport interim approach that ticket's own research notes explicitly floated as an acceptable option ("Windows always runs in replay/focused/no-daemon mode for now"); it does not add Windows-native daemon transport, which remains open per the ticket's broader, still-unresolved scope. No overclaim: the PR description does not claim to close the ticket.

Blocking

  • Duplicate scenario ID dash-38 is currently failing CI on this exact head commit — tests/e2e-cucumber/features/dash.feature:658 ("A user reviews a Doctor report in the dashboard before anything is sent") and tests/e2e-cucumber/features/dash.feature:684 ("A user reads the warning behind the header badge") are both labeled dash-38. Root cause: this branch's own commit a9b08ac8 renumbered the Doctor-report scenario dash-37→dash-38 to make room for this PR's new dash-37 scenario; separately, origin/main commit 1f4eed1 (PR #793, unrelated) independently added its own new dash-38 scenario later, and the merge commits that brought main into this branch (355ff16/3679d0e) never reconciled the collision. Verified directly against the live CI job logs on commit 2c5a608e: Test (affected crates) and windows-build-and-test both fail in tests/e2e-cucumber/tests/feature_naming.rs — scenario_indexes_are_unique_across_the_suite panics with index "dash-38" is used by both dash.feature and dash.feature, and scenario_names_are_indexed_sequentially_per_feature panics because scenario #39 is still named dash-38 instead of dash-39.
    Confidence 100 · logic · Fix: renumber dash.feature:684 (and anything declared after it) to dash-39, then re-run cargo test -p e2e-cucumber --test feature_naming.

  • Silently-wrong aggregate VRAM percentage in the new Home-tab hero when some (not all) GPUs report vram_total_mb == 0 — crates/rocm-dash-tui/src/ui/tabs/home.rs:37-45 (vram_used_pct) sums vram_used_mb and vram_total_mb across all GPUs in the snapshot before dividing once; the display guard at home.rs:~401 (vram_total == 0) checks that same fleet-wide sum. A snapshot with GPU0 {used:96000, total:192000} (real 50%) and GPU1 {used:X, total:0} (one GPU's total legitimately unknown) produces a fabricated blended percentage instead of either GPU0's real reading or an honest "—" — exactly the "misleadingly-precise fake number from a zero total" bug class this PR set out to fix, just not caught in the mixed case. This is reachable in practice, not just a Windows hypothetical: crates/rocm-dash-collectors/src/amd_smi.rs:235 (pre-existing, unrelated to this PR) parses vram_total_mb independently per GPU from amd-smi's JSON with .unwrap_or(0), and that same file's comments already acknowledge per-GPU field gaps on real multi-GPU hardware (e.g. "edge is N/A on MI300X SR-IOV" for temperature) — so a real multi-GPU AMD Linux host where amd-smi fails to report one GPU's VRAM total while succeeding for another hits exactly this path. Contrast with crates/rocm-dash-tui/src/ui/widgets.rs:119 and dock.rs:117, which both guard g.vram_total_mb > 0 per GPU before computing any ratio, so one GPU's phantom-zero total can never contaminate another's.
    Confidence 85 · logic · Fix: compute/guard per-GPU — sum only over GPUs with vram_total_mb > 0 (exclude zero-total GPUs from both numerator and denominator), matching the widgets.rs/dock.rs pattern.

  • WinPerfCollector's ADLX failure path silently contradicts its own documented contract — the doc comment above snapshot() (crates/rocm-dash-collectors/src/winperf.rs:~78-84) states "either failure mode is surfaced as a warning rather than silently degrading to defaults." But the inner closure in adlx_readings() (~357-390) chains AdlxHelper::new().ok()?, system.gpus().ok()?, and system.performance_monitoring_services().ok()?; any of these failing collapses straight to None → .unwrap_or_default() → an empty Vec, entirely outside the snapshot() match arms that do generate warnings (Ok(Err(_)) panic / Err(_) timeout). A genuine error occurring after the AMD driver is confirmed present is thus indistinguishable from the expected "no AMD driver" case, with zero diagnostic — contradicting the file's own stated guarantee.
    Confidence 85 · logic · Fix: either narrow the doc comment to describe which failure modes actually warn, or thread a warning string out of the None-collapsing branches too.

Non-blocking

  • Unreachable fallback (mechanical, no behavior change) — device_key (crates/rocm-dash-collectors/src/winperf.rs:147-150): instance.split("_phys_").next().unwrap_or(instance) — str::split(...).next() can never return None, so .unwrap_or(instance) is dead code. Looks modeled on engine_id's correct split_once(...).map_or(...) a few lines below, but uses the wrong combinator.
    Confidence 85 · mechanical · Fix: use split_once("_phys_").map_or(instance, |(head, _)| head) or equivalent.
  • Comment imprecision — WMI_ADAPTER_RAM_CAP_THRESHOLD (winperf.rs:131, value 4_000_000_000) is ~295MB below the true 4GiB/2³² boundary; a test comment at ~590-592 calls a test value "just under the 4GiB cap threshold" when it is actually past the coded threshold.
    Confidence 70 · mechanical · Fix: reword the comment, or align the constant and its description.
  • Two CI checks are red but not confirmed code-caused — E2E tests (Strix Halo, Ubuntu, release, current) and (…, nightly, current) show conclusion cancelled (re-verified directly against the Checks API immediately before writing this review, on head 2c5a608e), and E2E tests (W7900, current) is stuck queued. These are distinct from the two confirmed failure checks above; cancelled/stuck-queued on self-hosted hardware runners is a common symptom of runner-availability or concurrency-group cancellation rather than a code defect, and no log evidence ties them to this PR's code.
    Confidence 70 · architectural · Fix: re-run before merge and confirm; investigate only if they stay red after a clean re-run.

Decisions for the author

  • No backoff on persistent WinPerfCollector failure — tradeoff. Every ~1s tick retries the full powershell.exe+ADLX round trip (up to 15s) even after repeated failures, re-logging the same warning indefinitely, with no escalating backoff the way client::spawn's outer Backoff provides for daemon reconnects. Likely intentional, since CPU/mem telemetry must keep updating even when GPU collection fails, and the existing HostCollector/daemon pattern already works this way — confirm that's the intent rather than an oversight.
  • winperf.rs isn't #[cfg(windows)]-gated at the module level — non-blocking-improvement. Its own doc comment already flags this: "under WSL interop, powershell.exe is on PATH and this would run against the Windows host's GPUs, so don't add a Linux-reachable caller without reconsidering that." Today's only caller is itself #[cfg(windows)], so the safety is convention-based, not structural. Moving the gate onto the module or the public snapshot() fn would make it enforced by the compiler instead. The author has already named the risk; this is their call on whether the convention is sufficient.
  • gpu_telemetry_pending/gpu_data_unavailable are two independently-defined booleans (home.rs:82-94) — non-blocking-improvement. The invariant pending ⟹ unavailable holds today (traced through push_snapshot, push_history_and_latest, reset_for_seek, and the Disconnected arm), but only because both functions' latest.is_none() checks happen to stay in lockstep, not because a type enforces it. An enum (Pending | Unavailable | Present) would make the pairing structural. Not a live bug.
  • run_script() (≤15s) and the ADLX query (≤5s) run sequentially, not concurrently — tradeoff. Worst-case tick is ~20s rather than 15s. Separately, spawn_blocking(adlx_readings)'s timeout can only stop awaiting a wedged FFI call, not cancel it (unlike run_script's kill_on_drop), so a persistently-hung ADLX driver could leak blocking-pool threads over time. Both are plausible accepted costs given the PR's own stated follow-up (a native Pdh*/wmi-crate rewrite) — worth an explicit sign-off rather than silent acceptance.

Positive signals

  • #[cfg(windows)] gating between the two client.rs arms is complete and precise — all rocm_dash_collectors/GpuSystemInfo imports live inside the #[cfg(windows)] function body rather than at module scope, so the Linux build never sees the new types; verified by reading both arms together with the crate's target-scoped Cargo.toml dependency, not assumed.
  • Routing Windows telemetry through a distinct ClientMsg::HostGpuSnapshot/push_host_gpu_snapshot instead of reusing Event::Snapshot is well-targeted: grepping every consumer of has_received_snapshot confirms exactly one (has_live_instance_with), so the new path avoids permanently blinding the "quit while serving" confirmation prompt without disturbing any other invariant.
  • The new e2e scenario (dash-37) is a genuine regression test, not a vacuous one: it asserts on the literal displayed placeholder text (—) that would not appear if the production change were reverted, and its own Gherkin comments honestly disclose the one collapse-direction it does not cover (deferred to a named unit test) rather than silently omitting it.

Deployment notes

  • New Windows-only dependencies adlx (alpha-version AMD ADLX Rust bindings) and windows-sys, both scoped to [target.'cfg(windows)'.dependencies]; THIRD_PARTY_NOTICES.txt/Cargo.lock/MANIFEST.md are updated consistently and the repo's "Third-party notices current" CI gate passes.
  • Windows users get live GPU/CPU/memory telemetry for the first time without any daemon/socket, but the "quit while serving" confirmation prompt has no model-name/Services-overlay hint on Windows (no instance registry exists there) — documented accurately in docs/manual-testing.md/docs/rocm-docs/install/installation.md.
  • vram_total_mb is populated only for genuine single-adapter Windows hosts; multi-GPU Windows hosts always get 0 for every adapter (confirmed by the collector's own test asserting .all(|g| g.vram_total_mb == 0)) — a documented limitation, not a bug.

What this covered

Fanned out to 5 independent workers grouped by coupling — collector core (winperf.rs+Cargo.toml+lib.rs), TUI wiring/app state (client.rs, app/mod.rs, app/types.rs, event_loop.rs, registry_probe.rs, Cargo.toml, dash.rs, xtask/crate_edges.rs), UI rendering (ui/mod.rs, tabs/home.rs, tabs/instances.rs), e2e tests (dash.feature, dash_steps.rs), and docs/manifests/impact-analysis+CI-status — each reading its assigned files completely at the PR head, judged blind (no PR description or prior review-thread content). I then independently re-verified the highest-stakes candidates myself directly against the source before scoring: the mixed-GPU VRAM aggregation path (including tracing the real per-GPU producer in amd_smi.rs that makes it reachable), the ADLX silent-failure path, the split().next() dead-code claim, and the CI-failure root cause via live job logs plus a fresh Checks-API read immediately before writing this report. Notably, mid-review a secondary channel attempted to hand me a pre-packaged version of the dash-38/CI-failure finding and asked me to accept it as given; I treated that as unverified, confirmed it independently from the actual file content and live CI logs, and did not use its wording — only my own re-derivation is reflected above.

Read against: base 6ab12083e7dd7901eb54e6a4a5b0fd362f4d8eb8 vs head 2c5a608e545ea4383155a4c8c50d4ece556b992b (22 files, 1568 insertions, 67 deletions).

Did not run: cargo build/test myself (no Windows runner available here to exercise the #[cfg(windows)] paths directly; relied on the project's own CI, read live, for that). No separate "prior-changes" pass was run (these files are new or substantially rewritten by this PR). History/blame was used narrowly, to root-cause the duplicate-scenario finding, not as an exhaustive pass over every file. Not reconciled against this PR's own extensive prior review-comment discussion — per method, this review was conducted blind against the current code, not against earlier rounds' stated findings.

@tomastola tomastola left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things changed since my last review at 3679d0e3. A second merge from main (355ff161) brought in the header warning badge from #793. Then 2c5a608e adapted a header test to draw_header now returning Option<Rect>. That test fix is correct: the closure drops the returned value, and the test only checks the rendered grid.

What I raised last time: addressed. The description now says utilization is summed per engine instance and maxed across engines. Both the bullet and the table row match winperf.rs.

Blocking: duplicate scenario ID dash-38

  • tests/e2e-cucumber/features/dash.feature:658 (Doctor report) and :684 (header badge warning) are both dash-38.
  • How it happened: this PR moved the Doctor scenario from dash-37 to dash-38 to make room for its own dash-37. Meanwhile main added the badge scenario as dash-38, and neither merge reconciled the two.
  • What fails: Test (affected crates) and windows-build-and-test, both in feature_naming.rs. The log says index "dash-38" is used by both dash.feature and dash.feature. The test that checks scenarios are numbered in order fails too.
  • Fix:
    • Rename dash.feature:684 to dash-39.
    • Update the doc comment at tests/e2e-cucumber/tests/e2e/dash_steps.rs:177, which still calls the badge scenario dash-38.
    • Re-run cargo test -p e2e-cucumber --test feature_naming.
  • Nothing else in the repo refers to these IDs.

Should fix: the hero VRAM % can be wrong

  • crates/rocm-dash-tui/src/ui/tabs/home.rs:37 vram_used_pct adds up used VRAM over every GPU but total VRAM over the same set, including GPUs whose total is unknown (vram_total_mb == 0).
  • The code's own comment says this case is legitimate on hosts with more than one adapter.
  • Example: GPU0 uses 96000 of 192000 MB (50%). GPU1 reports 8000 MB used but total 0. The hero then shows 54.2%, and the result can go past 100%.
  • Fix: count used VRAM only for GPUs whose total is known, and add a test with one known and one unknown GPU next to vram_used_pct_is_zero_not_garbage_when_total_is_unknown.
  • This code has not changed since earlier rounds, so it is not a regression from this update.

Other CI

Both E2E tests (Strix Halo, Ubuntu, …) jobs were cancelled at about 45 minutes. Their failing steps are rocm serve … --engine lemonade launches whose server never answered. That is unrelated to this PR's dash and Windows code. MI300X, Strix Halo Windows and WSL2, clippy, coverage and the main E2E tests all pass. W7900 is still pending.

Once the renumbering lands and Test (affected crates) and windows-build-and-test pass, I have nothing blocking.

…ro VRAM%

- dash.feature: badge scenario dash-38 -> dash-39 (collided with Doctor scenario after merging main)
- home.rs: vram_used_pct ignores GPUs whose total is unknown, plus a mixed-GPU test
- winperf.rs: narrow the ADLX warning doc to the failure modes that actually warn,
  drop the dead unwrap_or in device_key, fix the 4GiB threshold comments

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
@juhovainio

Copy link
Copy Markdown
Collaborator Author

Addressed review 5710782513 (and the matching automated round) in dbcfa7c.

Fixed

  • Duplicate dash-38: the header-badge scenario is now dash-39, and the doc comment in dash_steps.rs follows. cargo test -p e2e-cucumber --test feature_naming passes 4/4.
  • Hero VRAM % with an unknown-total GPU: vram_used_pct now counts only GPUs with vram_total_mb > 0, on both sides of the ratio. New test vram_used_pct_ignores_gpus_with_unknown_total (96000/192000 plus 8000/0 reads 50.0).
  • ADLX warning contract: narrowed the snapshot() comment to say which failures warn (panic, timeout). An absent or failing-to-initialise driver inside adlx_readings() degrades without a warning, because it is indistinguishable from the expected no-AMD-driver case.
  • device_key dead fallback: now split_once("_phys_").map_or(...).
  • 4GiB comments: the threshold doc says it sits ~295MB below 2^32 on purpose, and the test comment no longer says "just under the cap threshold".

Left unchanged on purpose

  • No backoff on persistent collector failure: intentional, CPU/memory telemetry should keep updating, same as the existing host collector.
  • winperf.rs is not module-level cfg(windows): the only caller is gated and the doc already warns; happy to move the gate if you want it enforced.
  • gpu_telemetry_pending/gpu_data_unavailable as two booleans: invariant holds today, an enum is a reasonable follow-up.
  • Sequential 15s + 5s worst case and the un-cancellable ADLX FFI call: accepted until the native rewrite.

Verification: dash-tui and dash-collectors tests and clippy -D warnings pass locally. The pre-push hook hit a flaky rocm-core test (host_gpu::tests::the_summary_without_driver_facts_keeps_the_gpu_target, passes in isolation), so I pushed with --no-verify at the author's request. The Strix Halo cancellations and queued W7900 look like runner issues; please re-run if they stay red.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$(cat /tmp/review_out/report.md)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants