Skip to content

test(therock): stop the update-json guard test mutating process PATH - #784

Open
jussielo-amd wants to merge 2 commits into
mainfrom
fix/650-python-resolver-path-race
Open

jussielo-amd wants to merge 2 commits into
mainfrom
fix/650-python-resolver-path-race

Conversation

@jussielo-amd

Copy link
Copy Markdown
Collaborator

Fixes #650

Problem

render_update_json_installs_the_suppression_guard_around_resolution steers render_update_json's Python-interpreter resolution onto its PATH-unusable branch by overwriting the process-wide PATH env var, under PROCESS_ENV_TEST_LOCK. cargo test runs every test as a thread in one process, so any sibling test thread that spawns a bare-name, PATH-resolved binary (tar, sleep, powershell) during that window can fail with "No such file or directory" / "program not found". Two commits this week alone (bbd80bf5, and the one squashed into 92d82dcd) each retrofitted one more previously-missed tar/PATH-spawning test onto PROCESS_ENV_TEST_LOCK after the race reproduced again, including once during unrelated verification work. That pattern only serializes around the mutation — it never removes it, so a new test that spawns a PATH-resolved binary can reintroduce the race at any time.

Cause

#253 fixed this same problem for two other python-resolver tests by passing the resolver's environment in as a value (PythonResolverEnv) instead of reading the process PATH at the use site. Those tests call the low-level resolver directly, so they could just construct one. This test exercises the higher-level render_update_json instead, and the call chain between them (render_update_json → runtime_update_plan → resolve_latest_for_manifest → resolve_python_launcher) has production call sites of its own, so threading an env parameter through it would touch signatures used well beyond this one test.

Change

Add a #[cfg(test)] thread-local override for resolve_python_launcher's PythonResolverEnv, mirroring the SuppressProgressOutput/PROGRESS_LINE_SINK seam already in this file: a test can point resolution at a fixture directory without ever touching the real PATH, and because it's thread-local, nothing about it can race a sibling test thread. Production builds always see None and fall straight through to the real process environment, unchanged. The target test's PATH/ROCM_CLI_PYTHON mutation and its unsafe blocks for them are gone entirely; it still takes PROCESS_ENV_TEST_LOCK for the one remaining real mutation (ROCM_CLI_DISABLE_MANAGED_PYTHON_BOOTSTRAP, which nothing else in the file reads concurrently).

Two tests back the new seam: one asserts resolving through the override never mutates process-wide PATH (the property this issue is about), and one proves the override's nesting/restore semantics, mirroring suppress_progress_output_contract's equivalent test for SuppressProgressOutput.

Verification

  • cargo test -p rocm --bin rocm run 15 consecutive times — all green (1346/1347 passed, 0 failed each run), confirming the race no longer reproduces.
  • cargo test --workspace --no-fail-fast and cargo clippy --workspace --all-targets -- -D warnings both clean.
  • An independent fresh-context review (4 parallel passes: correctness/logic, AGENTS.md adherence, history, code-comment compliance) found no blocking issues; its two non-blocking findings (the Clone derive not #[cfg(test)]-gated, and comments not explaining the remaining lock/override split) are addressed in the second commit.

Scope and scenarios

Test-only change with no user-observable behavior, so there is no Gherkin scenario to add (AGENTS.md: purely internal changes need none).

render_update_json_installs_the_suppression_guard_around_resolution
steered render_update_json's Python resolution onto its PATH-unusable
branch by overwriting the process-wide PATH env var under
PROCESS_ENV_TEST_LOCK. cargo test runs every test as a thread in one
process, so any sibling test thread spawning a bare-name PATH-resolved
binary (tar, sleep, powershell) during that window could fail with
ENOENT/"program not found" -- seen repeatedly (#650), most recently
on 2026-10-27 during unrelated verification, and patched around twice
more this week (bbd80bf, 8f0e37e) by retrofitting yet another
tar/PATH-spawning test onto the same lock. That whack-a-mole only
serializes around the mutation; it never removes it.

#253 fixed this same problem for two other python-resolver tests by
passing the resolver's environment in as a value (PythonResolverEnv)
instead of reading the process PATH at the use site. Those tests call
the low-level resolver directly, so they could just construct one.
This test exercises the higher-level render_update_json instead, and
the call chain between them has production call sites of its own, so
threading an env parameter through it would touch signatures used
well beyond this one test.

Add a #[cfg(test)] thread-local override for resolve_python_launcher's
PythonResolverEnv instead, mirroring the SuppressProgressOutput/
PROGRESS_LINE_SINK seam already in this file: a test can now point
resolution at a fixture directory without ever touching the real PATH,
and because it's thread-local, nothing about it can race a sibling
test thread. Production builds always see None and fall straight
through to the real process environment, unchanged.

Add a regression test proving the property this closes: resolving
through the override never mutates process-wide PATH.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Independent review of the #650 fix surfaced three small gaps:

- PythonResolverEnv's Clone derive was unconditional even though the
  only clone() call is #[cfg(test)]; gate it the same way the rest of
  the seam already is.
- OverridePythonResolverEnv's doc comment claimed nested use composes
  correctly (mirroring SuppressProgressOutput) with nothing proving
  it; add the same bookkeeping test SuppressProgressOutput has.
- Neither PythonResolverEnv's doc comment nor the rewritten test's
  comment on PROCESS_ENV_TEST_LOCK mentioned why that lock is still
  taken now that PATH is no longer mutated there.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
@jussielo-amd
jussielo-amd requested a review from a team as a code owner October 28, 2026 17:05
@jussielo-amd
jussielo-amd requested a review from rominf October 28, 2026 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: render_update_json_installs_the_suppression_guard_around_resolution mutates process PATH, racing tar-spawning tests

1 participant