fix(lemonade): restore packaged backend pins when a kept alignment pin stops installing

README.md

@@ -750,6 +750,15 @@ rocm engines shell <engine>   [--runtime-id KEY | --env-id ID] [--shell PATH]
 
 Supported engines: `lemonade`, `vllm`.
 
+`rocm engines install lemonade` also keeps Lemonade's llama.cpp backend paired
+with the ROCm SDK in use, and rewrites the two pins it owns
+(`therock.version` and `llamacpp.rocm-stable` in `resources/backend_versions.json`)
+to do so. If installing an already-pinned backend fails, it restores those two
+pins from the cached Lemonade archive (it does not download), prints a warning,
+and installs once more; see
+[Lemonade backend alignment](https://github.com/ROCm/rocm-cli/blob/main/docs/engine-plugins.md#lemonade-backend-alignment-on-engine-install) for the details and for
+`ROCM_CLI_DISABLE_LEMONADE_BACKEND_ALIGNMENT`, which turns both off.
+
 `--yes` approves required system-package installs, such as OpenMPI for vLLM,
 without asking.
 

crates/rocm-core/src/lib.rs

@@ -337,9 +337,11 @@ pub fn unix_time_millis() -> u128 {
         .as_millis()
 }
 
-/// Replace `path`'s content by writing a sibling temp file and renaming it
-/// over the destination: an interrupted write (full disk, crash, killed
-/// process) can never leave a truncated file in place, and a reader sees
+/// Replace `path`'s content with `bytes`, atomically.
+///
+/// Writes a sibling temp file and renames it over the destination: an
+/// interrupted write (full disk, crash, killed process) can never leave a
+/// truncated file in place, and a reader sees
 /// either the old content or the new content. The destination is NOT removed
 /// first: `fs::rename` replaces an existing file on both supported platforms
 /// (it is `MoveFileEx` with `MOVEFILE_REPLACE_EXISTING` on Windows), and
@@ -359,7 +361,7 @@ pub fn unix_time_millis() -> u128 {
 /// - Replacing a destination another process holds open without delete
 ///   sharing: on Windows the rename then fails with a sharing violation,
 ///   this returns that error, and the old destination stays in place.
-pub(crate) fn write_atomic(path: &Path, bytes: &[u8], tmp_ext: &str) -> Result<()> {
+pub fn write_atomic(path: &Path, bytes: &[u8], tmp_ext: &str) -> Result<()> {
     let tmp_path = path.with_extension(format!(
         "{tmp_ext}.tmp-{}-{}",
         unix_time_millis(),

docs/architecture.md

@@ -111,7 +111,7 @@ What else stays in [`lib.rs`](../crates/rocm-core/src/lib.rs), beside module dec
   - `TORCH_ALIGNMENT_DISABLED_ENV`/`torch_alignment_disabled` — the one read of the torch-alignment opt-out, shared by the CLI and the vLLM engine so the two cannot disagree.
   - `VLLM_GPU_MEMORY_UTILIZATION_HINT` — a single wording contract between the CLI, the vLLM engine, and this crate's own `fix-16-vllm-oom` diagnosis; `rocm-core` is the only crate all of them can reach.
 - **amd-smi binary resolution** (`resolve_amd_smi_binary*`) — belongs in its own `amd_smi.rs` module: locating the tool is a distinct concern that only reads [`managed_runtime.rs`](../crates/rocm-core/src/managed_runtime.rs)'s registry records. The move is deferred to ROCMAI-723 and sequenced after the open [#447](https://github.com/ROCm/rocm-cli/pull/447), which adds amd-smi invocation code to the same stretch of `lib.rs`; moving the family first would make that PR relocate its additions mid-review. Until then `VLLM_GPU_MEMORY_UTILIZATION_HINT` stays physically inside that stretch; the extraction leaves it behind at the root.
-- **`write_atomic`** — not permanent. It is the crate-private temp-then-rename writer [`config.rs`](../crates/rocm-core/src/config.rs), [`automation.rs`](../crates/rocm-core/src/automation.rs), and [`service_record.rs`](../crates/rocm-core/src/service_record.rs) use for their atomic replaces. The open [#610](https://github.com/ROCm/rocm-cli/pull/610) adds a shared, stronger atomic writer to this crate for `rocm` and `rocmd`, and the open [#541](https://github.com/ROCm/rocm-cli/pull/541) moves `RocmCliConfig::save` onto it. Once #610 lands, the automation and service-record callers follow and `write_atomic` is deleted (ROCMAI-724). That is a behavior change, not a pure move, so it is not done here.
+- **`write_atomic`** — not permanent. It is the temp-then-rename writer [`config.rs`](../crates/rocm-core/src/config.rs), [`automation.rs`](../crates/rocm-core/src/automation.rs), and [`service_record.rs`](../crates/rocm-core/src/service_record.rs) use for their atomic replaces. The open [#610](https://github.com/ROCm/rocm-cli/pull/610) adds a shared, stronger atomic writer to this crate for `rocm` and `rocmd`, and the open [#541](https://github.com/ROCm/rocm-cli/pull/541) moves `RocmCliConfig::save` onto it. It is `pub`, visible to other crates, because `rocm-engine-lemonade` also writes Lemonade's `backend_versions.json` through it, in [`install.rs`](../engines/lemonade/src/install.rs). Once #610 lands, the automation and service-record callers follow; `write_atomic` is deleted only after that Lemonade caller has moved to the shared writer too (ROCMAI-724). That is a behavior change, not a pure move, so it is not done here.
 
 Once those two deferred moves land, [`lib.rs`](../crates/rocm-core/src/lib.rs) holds only module declarations, re-exports, and the crate-root primitives above — the end state of the modularization effort for this crate (EAI-7768, planned last: highest fan-in, since every app and engine crate depends on it, but lowest novelty, since the sibling modules already prove the pattern).
 

docs/engine-plugins.md

@@ -66,6 +66,54 @@ was pinned before the attempt — the packaged pin on a fresh install or after
 `--reinstall`, but whatever alignment last wrote otherwise — rather than
 being kept.
 
+A pin an earlier alignment kept can go stale: the llama.cpp or TheRock build it
+names may later be withdrawn upstream. When the pin already equals the active
+ROCm version the install does not attempt another alignment, so if installing
+that pinned backend fails, `rocm engines install lemonade` restores the two pins
+alignment writes (`therock.version` and `llamacpp.rocm-stable`) in
+`resources/backend_versions.json` from the embeddable archive it was extracted
+from (digest-verified; the file is rewritten atomically and every other entry
+in it is kept) and installs once more against Lemonade's own pins, printing a
+warning that says so. This changes what a plain install does: it can rewrite
+those two pins and replace the selected backend, where it used to report the
+failure and leave both alone.
+
+The restore never downloads. It reads the cached copy of the embeddable archive
+that the same install verified a moment earlier (under `downloads/` next to the
+runtime tree). If that copy is gone or no longer matches its digest, the restore
+is skipped, a warning says why, and the original install error is reported. The
+same applies when the pin file exists but cannot be read (it is left as it is,
+never treated as absent), when the pins already hold the packaged values, or when the
+restore fails for any other reason: there is no second attempt and the original
+install error is reported unchanged.
+
+The retry is a forced reinstall of the selected backend into the shared tree.
+Like alignment's own revert, it sets the existing ROCm backend directories aside
+first and puts them back if the retry fails. The ROCm backend entries are listed
+before the first attempt, and after a failed retry anything that was not in that
+list (a partial build the failed attempts left) is removed once the directories
+are back. Any install failure starts the restore, including a transient network
+or disk error, so when the second attempt fails too, its error is reported (the
+first error is in the warning) and the previous `backend_versions.json` is put
+back.
+
+Known limitation, shared with alignment's own revert: between setting the
+directories aside and putting them back, the working backend exists only under a
+`.pre-alignment` name. If the process is killed in that window the directory
+stays there and nothing recovers it automatically, because the runtime tree is
+shared between processes and a leftover aside cannot be told from another
+process's install in progress. Run `rocm engines install lemonade --reinstall`
+to rebuild the tree.
+
+The previous `backend_versions.json` is held in memory only while the retry runs.
+If the process is killed in that window the packaged pins stay on disk and the
+previous bytes are lost; the next install runs alignment against the packaged
+pins again. The recovery applies only to the case above (the kept pin equals the
+active ROCm version): with no active SDK, an unreadable pin file, or a failed
+backend probe, the install uses what the file pins and does not restore.
+
+Setting the variable below skips this recovery along with the alignment.
+
 Set `ROCM_CLI_DISABLE_LEMONADE_BACKEND_ALIGNMENT` to keep whatever
 `backend_versions.json` already pins and skip the rewrite — any value works,
 including an empty one, since the variable being set is the signal:

docs/manual-testing.md

@@ -293,6 +293,23 @@ in `MSVCP140.dll`), the Visual C++ runtime on the machine is probably older than
 fix-21-msvc-redist-outdated` prints the update plan and changes nothing. After
 the update, restart Lemonade and repeat the serve command above.
 
+To check the recovery from a stale Lemonade backend pin (Windows or Linux with
+a managed Lemonade runtime already installed and an active managed ROCm SDK):
+the recovery only runs when the pinned `therock.version` equals the active
+SDK's version; with no active SDK, or a different version, the install takes
+other paths and simply fails or realigns. Edit
+`resources/backend_versions.json` in the Lemonade runtime tree so that
+`therock.version` is the active SDK's version and `llamacpp.rocm-stable` is a
+build tag Lemonade does not ship, add an unrelated key, remove the installed
+ROCm backend directory under `bin/llamacpp/`, and run
+`rocm engines install lemonade`. Expected result:
+
+- A warning says the pinned llama.cpp backend could not be installed and that the
+  pins Lemonade ships were restored; the command exits 0 and installs the
+  packaged backend.
+- `therock.version` and `llamacpp.rocm-stable` in `backend_versions.json` hold the
+  packaged values again, and the unrelated key is still there.
+
 While the log stream is attached, verify detach and stop behave differently:
 
 - Press `Ctrl+D`. The stream ends and the shell prompt returns, printing a

docs/testing.md

@@ -1059,6 +1059,26 @@ and Lemonade on non-Windows are all fully supported. Teaching the Windows
 detached-spawn primitive to carry a value-typed override (so Windows Lemonade can
 serve public with auth) is tracked as a separate follow-up.
 
+Lemonade backend pin restore focused tests (the restore after a kept alignment
+pin stops installing; `docs/engine-plugins.md`, section "Lemonade backend
+alignment on engine install"):
+
+```bash
+cargo test -p rocm-engine-lemonade restore
+cargo test -p rocm-engine-lemonade equal_pin
+cargo test -p rocm-engine-lemonade failed_retry
+```
+
+They assert each warning together with the real state it describes: the pin
+file, the ROCm backend directories (including a partial build the first attempt
+left), and that a missing or mismatched cached archive triggers no download.
+No Gherkin scenario covers the restore: it needs a runtime tree that already
+holds a stale pin and a live backend download, which only a gated hardware lane
+has. The Windows Strix Halo e2e lane's prewarm step (`rocm engines install
+lemonade` against the shared tree) reaches the restore only on a runner whose
+shared tree still holds a stale pin; on a healthy runner the install succeeds and
+the restore never runs, so a green lane does not show that it works.
+
 Engine recipe adapter contract focused tests:
 
 ```bash