Skip to content

test(e2e): drive the Lemonade recovery scenario with a planted runtime (ROCMAI-669) - #351

Open
rominf wants to merge 1 commit into
mainfrom
chore/remove-e2e-test-hooks
Open

rominf wants to merge 1 commit into
mainfrom
chore/remove-e2e-test-hooks

Conversation

@rominf

@rominf rominf commented Sep 7, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #349.

Re-scoped on 2026-10-26. This PR no longer removes the e2e-test-hooks feature, and it no longer touches the dashboard clock:

What remains is one consumer converted: the Lemonade recovery scenario.

Problem

@id:serve-lemonade-preparation-recovery (serve-18) needs Lemonade's llama.cpp backend install to fail twice, so it can pin the retry count and the recovery command. It got that failure from two #[cfg(feature = "e2e-test-hooks")] branches in product code:

  • engines/lemonade/src/lib.rs failed the Install RPC when ROCM_E2E_LEMONADE_BACKEND_INSTALL_FAILURE was set;
  • apps/rocm/src/serve_cmd.rs waived rocm serve's no-GPU pre-flight for the same variable.

So the scenario tested a code path that exists only in test builds.

Change

Both branches are deleted. The failure now happens where a broken install fails, at a non-zero child process.

  • The runtime. tests/e2e-cucumber/src/bin/fake-lemonade.rs is a [[bin]] of the suite, planted as the runtime's lemonade and lemond:

    • lemonade backends reports llamacpp:rocm as installable;
    • lemonade backends install exits non-zero;
    • lemond stays alive until killed.

    The install manifest is written beside it. Its env_id is derived from rocm_deps::LEMONADE_VERSION (a dev-dependency, so no crate-edge entry). serve skips its own install only when Detect reports exactly that id; a placeholder silently runs the real multi-gigabyte install instead.

  • The GPU. The pre-flight runs unchanged, against one KFD GPU node planted under the simulated host root (ROCM_CLI_TEST_HOST_ROOT). The root holds nothing else, so the scenario needs no GPU, and a GPU runner's own devices do not take part. XDG_RUNTIME_DIR also points inside the scenario, so the fake server writes nothing outside it.

  • The gate. @requires-no-gpu → @requires-os:linux.

  • The engine's unit test reads the fixture's backends table with include_str!, instead of a copy kept in step by a comment.

Workflow and contract-test comments that cited the deleted seam as their example now cite the host root.

Why this doesn't weaken verification

resolve_runtime accepts the planted manifest because it only checks that the manifest parses and that lemond is a file. No checksum, version or signature is on this path; the sha-pinned archive download lives in prepare_embeddable, which serve does not reach once Detect matches. The no-GPU pre-flight is no longer waived. It now passes for the same reason it passes on a real GPU host.

Test plan

  • serve-18 passes locally via cargo xtask e2e on a GPU-less Linux (WSL2) host.
  • Mutation checks:
    • fake install succeeds: the scenario fails, with no retry announcement;
    • GPU not planted: the scenario fails with its named pre-flight message;
    • fixture table reports installed: the engine unit test fails;
    • tag changed to @requires-no-gpu or removed: the tag pin fails.
  • Gate:
    • fmt;
    • both clippy invocations on Linux and on the Windows GNU target, and Windows cargo check -D warnings;
    • rocm-engine-lemonade --all-targets, rocm --bin rocm, e2e-cucumber --lib + feature_naming, check --test e2e;
    • xtask tests, xtask manifest --check, check-crate-edges.

Related

@rominf

rominf commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto current main (6ad833f1) and repaired. Head is now 1c2600ba, a single commit.

What the rebase dropped

The two ci(windows) commits at the base of this branch, both fully landed on main as da157ca3 (#342). Checked against main's files rather than the commit messages — main's version is a strict superset: same $targetDir block in ci.yml, plus two assertions this branch's copy lacked (it pins both exported binary paths, and pins the Test-Path/throw fail-fast guard). Nothing was unlanded, so nothing was carried forward.

That also retires the "Depends on #342" note at the top of the description — #342 has merged.

What the rebase preserved

Four conflicts, each resolved by keeping both sides rather than taking one:

sccache (#125) is untouched: 87 occurrences in ci.yml on both main and this head.

Three things needed updating because main moved underneath the original patch:

  • The MI350P lanes in e2e-selfhosted.yml and nightly.yml (ci(e2e): add an MI350P self-hosted E2E lane #341) still passed --features rocm/e2e-test-hooks. Left alone they would have failed the repurposed contract test.
  • install_lifecycle.feature's CONSTRAINT comment described the hook-carrying/hook-free lane split this PR abolishes, so it is removed.
  • The dashboard clock — see below.

The dash clock was a second consumer

#380 added a #[cfg(feature = "e2e-test-hooks")] gate in apps/rocm/src/dash.rs after this branch was cut, so deleting the feature broke clippy -D warnings (unexpected cfg condition value) and would have silently disabled the deterministic clock for dash-08/dash-09 on the blocking lane.

Rather than keep the feature alive for it, the same technique this PR already uses for Lemonade is applied to dash. The principle: what must not diverge between the tested and the shipped binary is the compile. That is not the same as "no test affordances" — a path- or state-based affordance is present identically in every build and inert until something plants the state, whereas a #[cfg(feature)] makes the tested binary a different binary.

So rocm dash now reads the directive from <ROCM_CLI_DATA_DIR>/telemetry/test-clock-offset, inside the data root it already resolves, and the harness plants it exactly as it plants the Lemonade manifest. No env var, no feature. The daemon side needed no change — RunnerOptions::test_clock_offset_path already ships unconditionally.

The existence check there is load-bearing rather than an optimisation: that field selects wall time only on None, so handing the daemon a path whose file is absent would leave production on the default FreeRunning(0) logical clock.

Why serve-18 was failing

Not infrastructure — a real defect in this PR. All three red lanes reported the same single unexpected failure, and the logs showed the planted runtime being ignored while a real multi-gigabyte backend install ran and succeeded, so serve returned 0.

Before serve prepares an engine it calls Detect and skips its own install only when the reported env_id equals the lemonade-embeddable-{LEMONADE_VERSION} that managed_engine_runtime_id builds. The planted manifest carried the sentinel "e2e-planted", which does not fail loudly — it just lets serve fall through to the real install. The manifest now derives that id from rocm_deps::LEMONADE_VERSION rather than copying it, so a version bump cannot reintroduce the fallthrough.

With the install skipped, serve_http resolves the planted runtime and reaches ensure_best_llamacpp_backend, whose backends install runs the fixture and fails at a non-zero child process — which is the premise the scenario asserts.

The planting location was already correct: this scenario does not opt into the shared runtime tree, so the CLI resolves no env root and lemonade_root is the isolated engine dir. (Planting into the shared tree would have been wrong anyway — other scenarios serve from it.)

Status of the three previously-red lanes

Lane Before Now
E2E tests (GPU) → now E2E tests (MI300X) 1 unexpected failure green — serve-18 passes, 0 unexpected failures
E2E tests (rad3 R9700) 1 unexpected failure green — serve-18 passes, 0 unexpected failures
E2E tests (Strix Halo, Windows) 1 unexpected failure still queued at time of writing

Also green on this head: E2E tests (mock lane), E2E tests (MI350P) — serve-18 passes there too — and E2E tests (Strix Halo, WSL2). Every required check is green, including clippy, windows-build-and-test, Test (affected crates) and Commit signatures + sign-off. E2E tests (Strix Halo, Ubuntu) is also still queued.

Verified locally

cargo clippy --workspace --all-targets -- -D warnings and cargo clippy -p e2e-cucumber --test e2e -- -D warnings clean; cargo fmt --all --check clean; cargo check -p e2e-cucumber --test e2e explicitly (that target is test = false); feature_naming 4/4, so no stale @id/index rows after the rebase; xtask 138/138.

dash-08 and dash-09 were run on this Linux host and pass, as does the whole dash feature (16 scenarios, 102 steps, 0 failures). Worth noting that a local run cannot reproduce the concurrency of the mock lane, which is the load that made these two flaky before — the lane itself is the authority, and it is green on this head.

cargo test --workspace --all-targets is clean apart from three rocm-dash-tui agent tests failing with cannot create token cache dir: Permission denied. That is a sandbox filesystem restriction in my environment, in a crate this diff does not touch. I did not verify them against an unmodified main checkout, so I am reporting the cause rather than asserting they are pre-existing.

Not verified locally: serve-18 itself — this host has no usable AMD GPU, which is precisely why the scenario is now @requires-gpu. The three GPU lanes above are what exercised it.

@rominf
rominf marked this pull request as ready for review September 22, 2026 13:12
@rominf
rominf requested a review from a team as a code owner September 22, 2026 13:13
@rominf
rominf requested a review from tomastola September 22, 2026 13:13

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran a code review pass on this in addition to my own read of the diff, and it surfaced two things worth looking at before merge.

The main one: dash_test_clock_offset_path in apps/rocm/src/dash.rs now checks for a file on every rocm dash launch, in every build, not just E2E builds. If that file (telemetry/test-clock-offset) ever ends up in a real user's data directory (synced or copied from a dev or CI machine, some future bug that touches that path), the daemon silently drops onto a frozen logical clock instead of wall time. There's no log line when this happens, only a doc comment saying it should never occur outside the E2E harness. A one line warn log when the file is found and used would make this fail loudly instead of silently, without touching the test seam design itself.

The second one is minor, and actually already flagged by the author in a comment: the clock file name and its data/telemetry subpath are duplicated as literals between dash.rs and the E2E's dash_steps.rs, kept in sync only by that comment. Not asking to block on this one, just noting it since the review turned it up.

CI is otherwise clean aside from the Strix Halo Windows and Ubuntu self hosted lanes, which I checked are flaky on main independent of this PR (3 of the last 4 completed main runs failed or were cancelled on those two lanes), so that's not a signal against this change.

Comment thread apps/rocm/src/dash.rs Outdated
/// `TestClockDirective::FreeRunning(0)` logical clock. Returning the path
/// unconditionally would therefore take production off `Utc::now()`.
fn dash_test_clock_offset_path(paths: &AppPaths) -> Option<PathBuf> {
let path = paths.telemetry_state_dir().join(DASH_TEST_CLOCK_FILE);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every build now takes this path, not just E2E builds. If telemetry/test-clock-offset ever exists in a real user's data dir, the dashboard silently freezes its telemetry clock instead of using wall time, with nothing logged. Worth a warn log when this file is found and used, so a stray file fails loudly instead of silently skewing displayed timestamps.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and fixed in 7841012 — this is the inherent cost of the state-based seam and it should not be paid silently.

dash_test_clock_offset_path now logs at WARN when it finds the file, naming the path:

WARN rocm::dash: dashboard telemetry clock is in TEST mode, not wall time: displayed
timestamps come from a logical clock read from this file. Only the E2E harness creates
it; delete it to restore wall-clock timestamps.
  test_clock_file=/…/telemetry/test-clock-offset

Why tracing and not stderr. logging.rs makes the client subscriber file-only on purpose — the TUI owns the raw-mode terminal and a stray stdout/stderr write corrupts the display. The only production caller of this function is maybe_spawn_embedded_daemon on the rocm dash path, so stderr is exactly the wrong channel here; a line printed there would also land on the PTY grid that the dash-* scenarios parse and assert against. The WARN lands in ~/.rocm/logs/rocm-cli.log.<date>, which is where you would look when investigating "my timestamps are wrong" — and it survives the alt-screen switch, which a pre-launch eprintln would not.

Verified live, not just by unit test. Planting telemetry/test-clock-offset under a throwaway ROCM_CLI_DATA_DIR and launching the release rocm dash under a pty produced exactly the record above in the client log; the same launch with no file planted produced an empty log. logging::init runs at the top of run(), before subcommand dispatch, so the subscriber is installed before this callsite is reached, and DEFAULT_FILTER (warn,rocm=info,…) admits the rocm::dash target.

Tests. Two, using a thread-local capturing subscriber (tracing::subscriber::with_default, so a concurrently-running test cannot see or pollute the records): one asserts the WARN fires and names the file when it is present, one asserts nothing at all is logged when it is absent — a warning on every launch would be noise and would drain the signal out of the first.

Both were mutation-probed rather than just observed green: deleting the tracing::warn! turns the first test red (expected a WARN record, got: ), and moving the warn above the existence check so it fires unconditionally turns the second red. So neither passes vacuously.

The existence check itself is unchanged — cycle_timestamp picks wall time only on None, so returning Some for an absent file would put production on FreeRunning(0). Only the logging is new.

/// `<ROCM_CLI_DATA_DIR>/telemetry/test-clock-offset` and falls back to wall time
/// when it is absent, so a rename on either side silently drops the whole
/// mechanism rather than failing loudly.
const DASH_CLOCK_OFFSET_FILE: &str = "test-clock-offset";

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This risk is already spelled out in the comment above (rename on either side silently breaks the mechanism). Not blocking, just flagging that the fix (a shared constant, or a debug_assert tying the two together) is still open.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Closed this one too, with the debug_assert-style option rather than the shared constant — reasoning, since you offered both:

A shared constant is not reachable here. apps/rocm is a binary crate, so the suite cannot import DASH_TEST_CLOCK_FILE at all; making it importable means hosting it in a library and adding a first-party edge from e2e-cucumber to that library. #422 landed xtask check-crate-edges on main while this branch sat, which deliberately makes every such edge a reviewed decision — spending one on a single string, to couple the black-box suite to a production crate it otherwise doesn't link, seemed like the wrong trade.

So instead dash::tests::e2e_harness_plants_the_file_rocm_dash_reads in apps/rocm/src/dash.rs reads the step file's source with include_str! and pins the literals to what rocm dash actually resolves. That is the same cross-tree reach the suite already makes (installer_fixture.rs does it for install.ps1), it adds no dependency, and it runs in the every-PR unit lane rather than only where the dash scenarios run.

It checks two things:

  • the harness's DASH_CLOCK_OFFSET_FILE declaration parsed out of the source, compared against DASH_TEST_CLOCK_FILE — parsed, not substring-matched, so an unrelated constant containing the same text can't satisfy it;
  • every directory between the data root and the file, derived from AppPaths::telemetry_state_dir() and required to appear in order inside dash_clock_path's own body. Anchoring on the builder rather than the whole file matters: a stray .join("telemetry") elsewhere in those ~1.2k lines would otherwise satisfy it. Taking the whole relative path rather than its last component matters too — telemetry_state_dir() being moved (say to data/dash/telemetry) leaves the leaf name intact, and an earlier version of this test passed under that mutation while the mechanism was broken.

Mutation-probed in all four directions: renaming the constant on the production side, renaming it on the harness side, relocating telemetry_state_dir(), and moving the .join out of the builder while leaving a decoy elsewhere in the file — each fails with a message naming the mismatch.

It does impose two constraints on that file, both written into the comment above the constant: keep the declaration on one line, and keep dash_clock_path building the path from .join(..) links. If either is broken the guard fails loudly with an explanatory message rather than passing silently, which is the failure mode I wanted.

One related thing the rebase surfaced, since it is the same gate: this PR's own e2e-cucumber -> rocm-deps edge (the pinned Lemonade version behind the planted runtime's env_id) was not in the allowlist, so check-crate-edges failed on the rebased branch even though the rebase itself was conflict-free. That entry is added in 7841012.

@rominf
rominf force-pushed the chore/remove-e2e-test-hooks branch from 1c2600b to 7841012 Compare September 24, 2026 08:05
@rominf

rominf commented Sep 24, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks — both findings were right, and the first one is the real cost of choosing a state-based seam over a compile-time one, so it shouldn't be paid silently. Pushed as 7841012, on top of a rebase onto current main.

The warn log. dash_test_clock_offset_path now logs at WARN when it finds the file, naming the path and saying the dashboard clock is in test mode rather than wall time. It goes through tracing (file-only client log) rather than stderr, because the only caller sits on the rocm dash path where the TUI owns the terminal — details and the live end-to-end evidence are in the thread. The existence check itself is untouched.

The duplicated literals. Took the debug_assert-style option rather than the shared constant: apps/rocm is a binary crate, so a shared constant would mean a new first-party crate edge, which #422's check-crate-edges guard (landed on main while this sat) exists to make a reviewed decision. A unit test reads the step file's source instead and pins both literals — anchored inside the harness's own path builder, and covering the whole directory chain, so a relocation is caught as well as a rename. Reasoning and the mutation probes are in that thread.

Rebase. Onto 8788394c, conflict-free — but not therefore free. #224's removal of the WSL setup script survives (the deliberate "no ROCDXG provisioning here" notes in both WSL lanes are intact, and the only remaining references to the script are the assertions that it must not be named), and git grep e2e-test-hooks finds no #[cfg(feature = ...)] anywhere — only two doc-comment mentions, both added by this PR to explain what replaced the seam. What the clean rebase did hide is that this PR's own e2e-cucumber -> rocm-deps edge was missing from the new allowlist, so cargo xtask check-crate-edges failed on the rebased branch; that entry is added here.

Ran locally on this head: dash-08 and dash-09 green (2 scenarios, 22 steps, 0 unexpected failures) — they are on the blocking every-PR lane, so I didn't want to push a dash change without executing them. Plus both CI clippy invocations, fmt --check, cargo check --workspace --all-targets and -p e2e-cucumber --test e2e separately, xtask manifest --check, xtask check-crate-edges, smoke_local.py, and the workspace test suite.

Also closed out one loose end from the previous description: the three rocm-dash-tui agent-test failures in my sandbox are now established as pre-existing rather than assumed — the same three fail identically on a clean checkout of origin/main.

Leaving both threads open for you to close.

@rominf

rominf commented Sep 24, 2026

Copy link
Copy Markdown
Collaborator Author

CI status correction — two things changed after this review was written

Flagging this because the review reasonably discounted the self-hosted lanes, and both facts behind that have since moved. Neither was wrong at the time.

1. E2E tests (Strix Halo, Windows) became a required check today (2026-09-24). When this review was filed it was advisory, so discounting it was correct then. It is now in branch protection's required-status-check list, so a failure there blocks the merge.

2. That lane has since gone green on main three times running. The review noted 3 of the last 4 completed main runs failed or were cancelled on these lanes — accurate when written. The precise picture for the Windows lane specifically:

main run Strix Halo, Windows
2026-09-23 07:59 → 15:06 (5 runs) never completed (runs cancelled by supersession)
2026-09-23 17:08 success
2026-09-24 05:48 success
2026-09-24 07:10 success

The first completed result landed about three hours after this review. So the flakiness read was sound at the time and simply predates the evidence.

The Ubuntu lane is a different story and the original read still holds there: it fails at GPU preflight under contention, is red on main today, and is not required. Not chasing it.

What that means for this PR

E2E tests (Strix Halo, Windows) is failing here on a real regression in this PR's own scenario, not lane noise:

  • serve-18 / @id:serve-lemonade-preparation-recovery reports serve unexpectedly succeeded — rc 0, empty stderr.
  • The backend install is never attempted, so the failure the scenario exists to assert never happens. The one Installing Lemonade llamacpp:rocm backend... line in the log is the lane's pre-warm step, not the scenario.
  • It is Windows-specific: E2E tests (MI300X) and E2E tests (rad3 R9700) are green on this same head, so it is neither a rebase-wide break nor a side effect of the two review fixes (a warn! in dash.rs, #[cfg(test)] tests, a comment, and one crate-edge allowlist entry — none reachable from rocm serve).

This was invisible until now because the Windows lane was cancelled on the previous head, so this PR has never had Windows data for the planted-runtime rework.

Working hypothesis, not yet confirmed: #404 changed llamacpp:rocm backend alignment to depend on the active ROCm SDK, and this scenario's planted runtime deliberately reports llamacpp rocm as installable to make the engine attempt an install. If that alignment now short-circuits when no SDK is active, the install would be skipped exactly as observed. Investigating that interaction now — I will report back either way rather than leaving it as a guess.

Not proposing to retag or weaken the scenario while the cause is unknown.

@rominf

rominf commented Sep 24, 2026

Copy link
Copy Markdown
Collaborator Author

Follow-up: my #404 hypothesis was wrong, and the real cause is a pre-existing Windows defect

Correcting the guess I posted above before it misleads anyone: #404 is not involved. It touched crates/rocm-core/src/lib.rs, docs/engine-plugins.md, engines/lemonade/src/lib.rs, runtime_setup.feature and runtime_steps.rs — not apps/rocm/src/main.rs, where the divergence actually lives. Apologies for pointing at it; I flagged it as unconfirmed and it did not survive contact with the code.

The real mechanism

spawn_managed_engine_child in apps/rocm/src/main.rs is asymmetric across platforms, and has been since the initial import — months before any recent change:

  • non-Windows: command.spawn() → sleep(200ms) → child.try_wait(); if the child already exited it bail!s with managed_engine_startup_failure_detail(status, &record.log_path), which tails the child's service log into the parent's error.
  • Windows: rocm_core::spawn_detached_no_inherit(...), which returns a bare PID. There is no Child, and therefore no liveness or exit check whatsoever.

After the spawn the parent waits up to 45s for HTTP readiness, records the status, and returns Ok(...) regardless. So on Windows a managed engine that dies during startup yields exit code 0 with readiness: starting — exactly the observed serve unexpectedly succeeded, rc 0, empty stderr, ~48s.

This is a real user-visible defect on main, independent of this PR: on Windows, rocm serve reports success when the managed engine died at startup.

Why this PR surfaced it

The deleted e2e-test-hooks seam injected its failure at the top of install_response — inside the Install RPC, which ensure_self_managed_engine_ready calls in the parent before any spawn. It failed synchronously on every platform, so the Windows gap was never exercised.

The planted runtime deliberately moves the failure to the fake's backends install, which is reached from the engine's Serve path — inside the detached child. serve-18 therefore now depends on managed-spawn startup-failure detection, which Windows does not implement. The scenario is correct; it is asserting behaviour the platform does not provide.

Why there is no test-side fix

Both seam-side options are closed, checked rather than assumed:

  • Run serve in the foreground. run_attached_service calls the same spawn_managed_engine_child. Every rocm serve path on Windows goes through the same detached primitive.
  • Make the failure happen in the parent's Install RPC instead. install_response → prepare_embeddable_with calls ensure_cached_archive(..., expected_sha256, download) before its needs_extraction check, so the Install RPC always demands the sha256-pinned embeddable archive. A planted runtime cannot forge that, and forcing this path would trigger the real multi-gigabyte download this design exists to avoid.

What I am not doing

I am not retagging or weakening serve-18 while the cause is understood but unfixed, and I am not bolting a production Windows fix onto a test-infrastructure PR — that change deserves its own PR, its own tests, and a maintainer with a Windows host.

Flagging it here for a maintainer decision rather than picking one unilaterally. Happy to open a separate issue for the spawn_managed_engine_child gap if that is wanted.

Observed vs inferred, stated plainly: every file/line and git fact above is observed. That the fake's backends install failure occurs inside the detached child and is invisible to the parent is inferred from the two branches plus the rc 0 / empty-parent-stderr evidence — I have no Windows host and did not read the child's service log, which the job log does not include.

@siloteemu

Copy link
Copy Markdown

🔴 Automated review · pr-review-watcher · 7841012

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Summary

Deletes the e2e-test-hooks cargo feature of the rocm crate and both seams it gated, replacing them with state-based affordances: the Lemonade recovery scenario now drives a planted fake runtime whose backends install exits non-zero, and rocm dash reads its logical-clock directive from <data>/telemetry/test-clock-offset (warning at WARN when present) instead of a compiled-in env-var branch. No blocking findings: the code itself is in good shape, and the two red checks at this head are recorded as an observation rather than as a finding of mine, since I could not attribute either to this diff. Verified: in a scratch copy of the tree I ran the dash unit-test module (22/22 pass, including all three new tests) plus two mutation experiments — renaming the production DASH_TEST_CLOCK_FILE constant, and relocating the harness's telemetry path segment — and the new cross-file guard correctly fails on both, so it is not vacuous; I separately confirmed AppPaths exposes exactly the three fields the new test constructs and telemetry_state_dir() is data_dir/telemetry, that managed_engine_runtime_id("lemonade") builds exactly the lemonade-embeddable-{LEMONADE_VERSION} string the planted manifest writes, that LemonadeInstallManifest's eight fields match the planted JSON name-for-name with no serde renames, that the engine's path helpers resolve to where the harness plants, that Duration::from_mins compiles on the pinned 1.96.0 toolchain, that the feature deletion leaves no live references anywhere in the tree, and that the workflow-contract and crate-edges checks pass; leak and prompt-injection scans of the diff were clean. The full suite, the e2e suite and anything needing a GPU were not run here. CI conclusions this review worked from: 26 success, 1 skipped, 2 failure. Blocking: 0 · Non-blocking: 6.

🚫 Blocking (must fix before merge)

None.

Non-blocking

  • Two checks are failing at this head and one is pending. I could not determine whether the failures are real regressions from this change or infrastructure noise, and I am deliberately not naming or guessing at jobs — but the change touches several workflow files and the contract tests that police them, so a lane-level failure is exactly the shape a regression here would take, and the two logs are worth reading directly rather than assuming they are unrelated.

  • engines/lemonade/src/lib.rs:~6806 — e2e_fake_runtime_table_drives_a_rocm_backend_install hardcodes a copy of the fixture's backends table rather than reading it, so editing the table in tests/e2e-cucumber/src/bin/fake-lemonade.rs would break the now-GPU-gated scenario while this test stays green; the doc comment's "keep this table byte-identical" is an unenforced instruction, which is precisely the pattern this same PR mechanically eliminates on the dash side — mirror that by include_str!-ing the fixture and extracting its printed rows.

  • apps/rocm/src/dash.rs:105 — the new WARN goes to tracing, which on the rocm dash path lands in a log file, so a user staring at skewed timestamps still sees nothing on screen; the terminal-corruption reasoning is sound, but a line in the TUI's own status area would actually reach the person the warning is for.

  • tests/e2e-cucumber/features/model_serving.feature:187 — nothing asserts this scenario still carries @requires-gpu, so a future tag edit would silently change which lane runs it; the repo already has the precedent for pinning a tag against the real feature file in tests/e2e-cucumber/src/expectation.rs.

  • engines/lemonade/src/lib.rs:~642 — pre-existing, but this PR makes the path routine: the ensure_best_llamacpp_backend failure returns via ? without terminating the spawned lemond, unlike the two sibling error paths just below it that do; in the scenario the harness's own teardown reaps it, so nothing leaks in test, but the asymmetry in production code is worth a follow-up.

  • xtask/src/workflow_contract.rs:~1292 — the ci.yml lifecycle lane's build check remains a substring match while the two other workflows now get the stricter whole-line one; the asymmetry predates this PR, but this is the change that introduces the stronger form and could cheaply extend it.

@siloteemu siloteemu left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 7841012

This is a formal review recording our position on the record. It is deliberately non-gating: this automation files no approval, so no approving review will appear here whatever the outcome, and the merge decision stays with a human reviewer.

Blocking: 0 · Non-blocking: 6. This round reviewed removal of a compile-time test seam in favour of state-based affordances, with the new cross-file guard shown to be non-vacuous by two mutation experiments run in a scratch copy.

Check conclusions this review worked from, read at this exact head immediately before publishing: 26 success, 1 skipped, 2 failure.

The full findings, including the non-blocking items, are in the review comment posted alongside this one.

@rominf

rominf commented Sep 30, 2026

Copy link
Copy Markdown
Collaborator Author

Splitting this up — it has three blockers welded together

Rebasing this onto current main (b0d598d0) surfaced a second structural problem, and taken with the Windows one it convinced me this PR is the wrong container for the work. Marking it draft rather than leaving it sitting as ready for review.

What the rebase found

The rebase itself was nearly clean — one additive conflict in tests/e2e-cucumber/Cargo.toml, where main added the fake-tailscale bin and this branch adds fake-lemonade. But a clean rebase is not a working one.

#421 landed two new consumers of the feature this PR deletes, in the newest commit on main:

  • apps/rocm/src/comfyui.rs:36,41 — comfyui_source_archive_url(), behind ROCM_CLI_COMFYUI_SOURCE_ARCHIVE_URL_OVERRIDE
  • apps/rocm/src/main.rs:9549,9554 — torch_runtime_dep_checks_disabled(), behind ROCM_CLI_DISABLE_TORCH_RUNTIME_DEP_CHECKS

cargo clippy --workspace --all-targets -- -D warnings fails on the rebased branch with four unexpected cfg condition value errors. And silencing them by simply deleting the attributes would be worse than the compile error: the #[cfg(not(..))] bodies become the only ones, so #421's ComfyUI scenario would fetch the real GitHub archive instead of its loopback fixture, and therock_steps.rs would run real system package installs. Both are on the every-PR mock lane.

That is the same thing that happened with #380's dash clock while this branch sat. Two consumers added in the time this has been open — the removal is chasing a target that keeps moving, which is itself a signal.

The three blockers

Piece Blocker
dash logical clock none — mergeable today
Lemonade planted runtime the Windows managed-spawn gap
e2e-test-hooks removal a moving consumer list; correctly the last step, not the first

Bundled, they all inherit the worst one.

What I have done

  • refactor(dash): read the test clock from state, not a compiled-in seam #466 — the dash clock conversion on its own, rebased onto current main. Both review threads here informed it and it carries both fixes: the WARN, and the cross-file guard pinning the harness literals. dash-08/dash-09 pass locally, both clippy invocations and fmt are clean, and the guard is mutation-probed. It has no GPU or Windows dependency, so it can land independently.
  • serve: on Windows a managed engine that dies at startup is reported as success #467 — the spawn_managed_engine_child gap, filed as its own issue as offered above. On Windows there is no liveness check after the detached spawn, and start_managed_service returns Ok(..) regardless of the readiness outcome, so rocm serve exits 0 when the managed engine dies at startup. That is a real user-visible defect on main independent of this PR, and it is what serve-18 is now unlucky enough to depend on.

What happens to this one

Parked as draft, not closed — the planted-runtime work is sound and I would rather re-land it than rewrite it. It needs #467 fixed first, because the scenario legitimately asserts behaviour Windows does not currently implement, and the Windows lane is now a required check.

The feature removal should come last, as a small mechanical PR converting the remaining consumers in one go, so it lands before main can overtake it again.

I have not pushed the rebase here — it does not compile for the reason above, and a broken head would be worse than a stale one. It is kept locally if it is useful later.

Nothing here needs a decision from a reviewer right now; #466 is the piece that is ready.

@rominf
rominf marked this pull request as draft September 30, 2026 10:44
`@id:serve-lemonade-preparation-recovery` (serve-18) needs Lemonade's
llama.cpp backend install to fail twice, so it can pin the retry count and
the recovery command. Two `#[cfg(feature = "e2e-test-hooks")]` branches
provided that: one in the Lemonade engine's Install RPC that failed on
`ROCM_E2E_LEMONADE_BACKEND_INSTALL_FAILURE`, and one in `rocm serve` that
waived its no-GPU pre-flight for the same variable. Both are deleted. The
`e2e-test-hooks` feature stays: it has other consumers (the simulated host
root among them), and removing it is the last step of #349, not this one.

The failure now happens where a broken install fails, at a non-zero child
process:

- `fake-lemonade`, a `[[bin]]` of the suite, is planted as the runtime's
  `lemonade` and `lemond`. `lemonade backends` reports `llamacpp:rocm` as
  installable and `lemonade backends install` exits non-zero; `lemond` stays
  alive until killed.
- The install manifest is written beside it. Its `env_id` is derived from
  `rocm_deps::LEMONADE_VERSION` (a dev-dependency, so no crate-edge entry):
  `serve` skips its own install only when `Detect` reports exactly that id,
  and a placeholder silently runs the real multi-gigabyte install instead.
- The pre-flight runs unchanged against one KFD GPU node planted under the
  simulated host root. The root holds nothing else, so the scenario needs no
  GPU and a GPU runner's devices do not take part. `XDG_RUNTIME_DIR` points
  inside the scenario too, so the fake server writes nothing outside it.

The scenario is now `@requires-os:linux` instead of `@requires-no-gpu`. It
still runs on the GPU-less every-PR lane, and now on the Linux GPU lanes and
WSL2 as well. It stays off Windows: the host root re-roots Linux paths only,
and there `rocm serve` does not notice a managed engine dying at startup
(#467), so the failure would never reach the user. A test in
`expectation.rs` pins that gate against the real tag line.

The engine's unit test now reads the backends table out of the fixture's
source with `include_str!`, instead of a copy kept in step by a comment.
Workflow and contract-test comments that cited the deleted seam as their
example now cite the host root.

Checked by mutation: the scenario fails when the fake install succeeds, and
fails with its named pre-flight message when the GPU is not planted.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf

rominf commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Head is now e97c827c. The PR is re-scoped and rebuilt on main (1c33248c), and the description is rewritten to match. It stays a draft.

What changed in scope.

What carried over. Compared with the old diff over the kept files:

  • Kept, with prose changes only: fake-lemonade, the planted manifest with env_id derived from rocm_deps, the serving_steps.rs changes, removal of the seam and the bypass, and the engine unit test.
  • Different:
    • the table is now a raw const read via include_str!;
    • rocm-deps is a dev-dependency, so the crate-edges allowlist entry is not needed;
    • the serve runs through run_rocm_with_scenario_env, because it now carries the host root;
    • the Lemonade crate keeps its feature, since it now only forwards to rocm-core;
    • comments are ASCII, for the fix(output): replace non-ASCII punctuation with ASCII in CLI output text #683 contract.
  • Dropped: the dash files, the workflow/xtask feature removal, and the install_lifecycle.feature note, which is still true.

One design change from the old version: no GPU needed. Previously, removing the pre-flight bypass forced serve-18 onto @requires-gpu, which lost the every-PR cadence. Instead, the harness now plants one KFD GPU node under the simulated host root, so the unchanged pre-flight passes on any Linux host. The tag becomes @requires-os:linux:

Verified locally on a GPU-less host. Mutations:

  • fake install succeeds: the scenario fails;
  • GPU not planted: the scenario fails with its named pre-flight message.

siloteemu's non-blocking items (automated review on 7841012c):

Item Disposition
two failing checks at that head that was serve-18 on Strix Windows (#467), analysed in the two comments above it. The re-scoped scenario no longer runs on Windows.
engine test copies the fixture table fixed: the_e2e_fake_runtime_table_drives_a_rocm_backend_install reads BACKENDS_TABLE from fake-lemonade.rs via include_str!; editing the fixture's table fails it (mutation-checked)
WARN not visible in the TUI dashboard clock, moved to #466; out of scope here
nothing pins serve-18's gate tag fixed: the_lemonade_recovery_scenario_runs_on_every_linux_lane_and_not_on_windows in expectation.rs reads the real tag line; changing it to @requires-no-gpu or dropping it fails (mutation-checked). The tag pinned is @requires-os:linux, not @requires-gpu, because of the design change above.
lemond left running when ensure_best_llamacpp_backend fails follow-up filed as #729 (also covers wait_for_lemonade_cli_status's early return)
contract asymmetry: substring vs whole-line build check moot here: this PR no longer edits those assertions, so it stays with the feature-removal step

Why still a draft. It no longer has to come last: it removes no feature, and the scenario does not need #437 to pass. I am keeping it a draft as asked until #437 (the fix for #467) lands. Once that happens it can come out of draft as-is, and extending serve-18 to Windows would be a follow-up.

Gate on this head:

  • fmt;
  • both clippy invocations, on Linux and on the Windows GNU target, plus Windows cargo check -D warnings;
  • rocm-engine-lemonade, rocm --bin rocm, e2e-cucumber --lib + feature_naming, check --test e2e;
  • xtask tests, manifest --check, check-crate-edges;
  • serve-18 via cargo xtask e2e.

@rominf
rominf marked this pull request as ready for review October 26, 2026 12:46
@rominf rominf changed the title test(e2e): drive the Lemonade recovery scenario with a planted runtime test(e2e): drive the Lemonade recovery scenario with a planted runtime (ROCMAI-669) Oct 26, 2026

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · e97c827

Review — needs work

Full review of the whole change (12 files, 1c33248c318c72a9e837627ff71476c8583f1930...e97c827c0b69d684f64524794ed3d94116e86ae0), fanned out across three coupled-file-group workers plus centralized cross-cutting checks. Part of #349 (open tracking issue), explicitly re-scoped to convert one consumer of the e2e-test-hooks seam rather than remove the feature — the description says so and the diff matches that scope exactly. PR title cites ROCMAI-669; that identifier is an internal tracker reference not retrieved for this review.

Blocking

  • CI check "E2E tests (MI300X, current)" is red at the reviewed head commit — CI status for e97c827c0b69d684f64524794ed3d94116e86ae0 (not a source location; run: https://github.com/ROCm/rocm-cli/actions/runs/40315449640/job/121546925225).
    gh pr view 351 --json statusCheckRollup,headRefOid confirms headRefOid matches the reviewed commit and this check's conclusion is FAILURE, while every other GPU/OS E2E lane (Strix Halo ×4, W7900, Windows, WSL2) passed on the same run. The job log itself returned 403 in this environment, so the root cause could not be read directly — but the mechanism this PR introduces is a plausible direct cause, not just an unrelated flake: the scenario's gate changed from @requires-no-gpu to @requires-os:linux, which is exactly what newly exposes serve-18 to a real-GPU lane (MI300X) for the first time. Previously this scenario never ran there at all. Worth ruling out before assuming it's infra noise, since (unlike the Strix Halo/Windows self-hosted flakiness a prior reviewer already characterized as pre-existing) this specific lane has no established flake history against this scenario — it's new exposure.
    Confidence 100 · architectural · Fix: pull the MI300X job log and confirm whether the planted fake runtime / simulated host-root GPU node interacts badly with a lane that also has a real AMD GPU and real Lemonade state, before merging.

Non-blocking

  • Stale "scripted failure seams" wording in an assert message, contradicting the doc comment immediately above it — xtask/src/workflow_contract.rs:296 (the assert! message inside assert_prebuilt_e2e_lanes_enable_test_hooks), vs. the doc comment at xtask/src/workflow_contract.rs:273-274.
    This PR correctly rewrote the function's doc comment from "the scripted Lemonade backend-install failure" to "the simulated host root, ROCM_CLI_TEST_HOST_ROOT" (the compiled-in failure branch is exactly what this PR removes). The assert! panic message below it in the same function was not updated and still reads "...without it the suite's scripted failure seams are compiled out:\n{block}", describing the now-deleted mechanism. A future reader hitting this assertion failure gets pointed at machinery that no longer exists.
    Confidence 85 · mechanical · Fix: update the assert message to match the revised doc comment (e.g. "...without it the suite's test seams, such as the simulated host root, are compiled out").

Decisions for the author

  • fake-lemonade.rs's run_lemonade_cli silently succeeds on any subcommand it doesn't recognize — tradeoff. This keeps the fixture small and avoids breaking the scenario if the engine issues some other incidental call, but it also means a future, unexpected invocation shape from the real engine (a new flag, a new subcommand) would pass through unnoticed rather than failing loudly, which could mask a drift between the fixture and the real lemonade CLI contract it stands in for. The alternative (fail loudly on anything unrecognized) was deliberately not chosen here (the code comments explain the three calls that matter and say "any other subcommand: succeed quietly"); confirm that's still the intended tradeoff as the fixture's surface grows.

Positive signals

  • The new backend_alignment.rs unit test pulls fake-lemonade.rs's BACKENDS_TABLE via include_str! rather than a hand-copied literal, so the fixture and the parser it's meant to drive cannot silently drift apart — verified the extraction is byte-exact and the parser assertions are correct against it.
  • The planted runtime's env_id (format!("lemonade-embeddable-{}", rocm_deps::LEMONADE_VERSION) in the test fixture) is verified byte-identical to the format string the real installer uses in engines/lemonade/src/install.rs:201, and the match against Detect was traced through to confirm the scenario actually skips the real multi-gigabyte install rather than silently falling through to it — exactly the property the PR's description claims.
  • Every cross-referenced issue in the description (#467, #437, #729, #510, #466) checks out factually against live GitHub state — titles, open/merged status, and the relationships claimed between them are all accurate.

Deployment notes

Both removed code paths (engines/lemonade/src/lib.rs's scripted install failure, apps/rocm/src/serve_cmd.rs's pre-flight waiver) were compiled only under the e2e-test-hooks feature, so this change has no behavioral effect on release/production builds.

What this covered

Every file in the diff, read completely at the PR head plus surrounding code (the full serve() pre-flight, the Lemonade install/detect flow, the E2eWorld harness, the tag-resolution machinery in expectation.rs), via three independent fanned-out workers grouped by contract coupling (engine/fixture contract; harness-and-steps; feature-tag-and-CI-gating), each of which activated the review-code skill itself and verified its own candidates before reporting. Centralized checks on top: live CI status for the exact head commit, a repo-wide grep confirming no dead references remain to the removed env var/feature seam, independent tracing of the env_id format-string parity between the fixture and the real installer, verification of the AGENTS.md crate-edge and GitHub-Actions-pinning conventions against this diff, and DCO/commit-quality checks (single well-described commit, signed off).

Did not run: a deep git log/blame history pass beyond confirming this is a single commit with no revision churn to review; a prior-PR-review-comments pass on these files (not retrievable at reasonable cost here); local build or test execution (no Rust toolchain available in this environment) — relied on live CI results instead, per the skill's guidance to take CI at face value rather than re-deriving its coverage.

Not reconciled against any PR discussion or review threads — those were deliberately excluded as input to this review's findings.

@tomastola tomastola left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings.

Non-blocking

  • MI300X lane red is unrelated to this PR. E2E tests (MI300X, current) (job 121546925225) fails only on therock-next-09 (a vLLM uv pip install --reinstall step), which this diff does not touch. serve-18 passed on that same lane. The identical single unexpected failure appears on main (runs 40360152181, 40357677968, 40354126258) and on other branches in the same window (e.g. jobs 121795242877, 121882265448, 121704235820). It should not gate this PR.
  • The fake lemond never exits (tests/e2e-cucumber/src/bin/fake-lemonade.rs, run_lemond). The comment says rocm serve terminates it when the serve fails, but the description says #729 tracks a lemond left running when the backend install fails during serve, which is this scenario's path. Until #729 lands, every Linux lane run can leave a sleeping process behind, and the scenario now runs on every Linux lane. Consider bounding the fake's lifetime, and aligning the comment with what serve does today.
  • The planted env_id repeats the format string managed_engine_runtime_id uses (plant_failing_lemonade_runtime in tests/e2e-cucumber/tests/e2e.rs; apps/rocm/src/engines_cmd.rs:323). Deriving the version from rocm_deps covers a version bump but not a change to the lemonade-embeddable- prefix. If Detect stopped matching, serve would start the real multi-gigabyte install instead of failing fast. An assertion that the install attempted was the fake's (it already prints a distinctive stderr line) would make that fail quickly.
  • tests/e2e-cucumber/README.md (around lines 244-251) lists the fixture binaries (fake-uv, fake-python, fake-tailscale) and does not mention fake-lemonade.

Decisions for the author

  • The engine unit test reads the e2e crate's fake-lemonade.rs with include_str! and extracts the raw-string literal textually. The table is then checked on every unit-test lane with no copy to drift. The cost is that a product crate's unit tests depend on a path in tests/ and on the literal's exact spelling. Please confirm that is intended.
  • The e2e crate gains a rocm-deps dev-dependency, while the README's "Black-box only … No imports from the rocm-cli codebase" principle stays as written. rocm-core is already such an edge. Either confirm this extension or adjust the README line.
  • serve-18 now runs on every Linux lane (GPU and GPU-less) and WSL2, where it previously ran only on the no-GPU lane. The new gate is pinned in expectation.rs. This is more cadence, and each run starts the fake server noted above.

Good things

  • The tag gate is pinned against the real feature-file tag line, so an edit fails a test.
  • The no-GPU pre-flight is no longer waived; it runs against a GPU planted under the simulated host root.
  • Both e2e-test-hooks branches are removed cleanly, and no references to ROCM_E2E_LEMONADE_BACKEND_INSTALL_FAILURE remain.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants