Repository navigation
test(therock): stop the update-json guard test mutating process PATH - #784
Open
jussielo-amd wants to merge 2 commits into
Open
jussielo-amd wants to merge 2 commits into
jussielo-amd wants to merge 2 commits into
Conversation
render_update_json_installs_the_suppression_guard_around_resolution steered render_update_json's Python resolution onto its PATH-unusable branch by overwriting the process-wide PATH env var under PROCESS_ENV_TEST_LOCK. cargo test runs every test as a thread in one process, so any sibling test thread spawning a bare-name PATH-resolved binary (tar, sleep, powershell) during that window could fail with ENOENT/"program not found" -- seen repeatedly (#650), most recently on 2026-10-27 during unrelated verification, and patched around twice more this week (bbd80bf, 8f0e37e) by retrofitting yet another tar/PATH-spawning test onto the same lock. That whack-a-mole only serializes around the mutation; it never removes it. #253 fixed this same problem for two other python-resolver tests by passing the resolver's environment in as a value (PythonResolverEnv) instead of reading the process PATH at the use site. Those tests call the low-level resolver directly, so they could just construct one. This test exercises the higher-level render_update_json instead, and the call chain between them has production call sites of its own, so threading an env parameter through it would touch signatures used well beyond this one test. Add a #[cfg(test)] thread-local override for resolve_python_launcher's PythonResolverEnv instead, mirroring the SuppressProgressOutput/ PROGRESS_LINE_SINK seam already in this file: a test can now point resolution at a fixture directory without ever touching the real PATH, and because it's thread-local, nothing about it can race a sibling test thread. Production builds always see None and fall straight through to the real process environment, unchanged. Add a regression test proving the property this closes: resolving through the override never mutates process-wide PATH. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Independent review of the #650 fix surfaced three small gaps: - PythonResolverEnv's Clone derive was unconditional even though the only clone() call is #[cfg(test)]; gate it the same way the rest of the seam already is. - OverridePythonResolverEnv's doc comment claimed nested use composes correctly (mirroring SuppressProgressOutput) with nothing proving it; add the same bookkeeping test SuppressProgressOutput has. - Neither PythonResolverEnv's doc comment nor the rewritten test's comment on PROCESS_ENV_TEST_LOCK mentioned why that lock is still taken now that PATH is no longer mutated there. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #650
Problem
render_update_json_installs_the_suppression_guard_around_resolutionsteersrender_update_json's Python-interpreter resolution onto its PATH-unusable branch by overwriting the process-widePATHenv var, underPROCESS_ENV_TEST_LOCK.cargo testruns every test as a thread in one process, so any sibling test thread that spawns a bare-name, PATH-resolved binary (tar,sleep,powershell) during that window can fail with "No such file or directory" / "program not found". Two commits this week alone (bbd80bf5, and the one squashed into92d82dcd) each retrofitted one more previously-missed tar/PATH-spawning test ontoPROCESS_ENV_TEST_LOCKafter the race reproduced again, including once during unrelated verification work. That pattern only serializes around the mutation — it never removes it, so a new test that spawns a PATH-resolved binary can reintroduce the race at any time.Cause
#253 fixed this same problem for two other python-resolver tests by passing the resolver's environment in as a value (
PythonResolverEnv) instead of reading the processPATHat the use site. Those tests call the low-level resolver directly, so they could just construct one. This test exercises the higher-levelrender_update_jsoninstead, and the call chain between them (render_update_json→runtime_update_plan→resolve_latest_for_manifest→resolve_python_launcher) has production call sites of its own, so threading an env parameter through it would touch signatures used well beyond this one test.Change
Add a
#[cfg(test)]thread-local override forresolve_python_launcher'sPythonResolverEnv, mirroring theSuppressProgressOutput/PROGRESS_LINE_SINKseam already in this file: a test can point resolution at a fixture directory without ever touching the realPATH, and because it's thread-local, nothing about it can race a sibling test thread. Production builds always seeNoneand fall straight through to the real process environment, unchanged. The target test'sPATH/ROCM_CLI_PYTHONmutation and itsunsafeblocks for them are gone entirely; it still takesPROCESS_ENV_TEST_LOCKfor the one remaining real mutation (ROCM_CLI_DISABLE_MANAGED_PYTHON_BOOTSTRAP, which nothing else in the file reads concurrently).Two tests back the new seam: one asserts resolving through the override never mutates process-wide
PATH(the property this issue is about), and one proves the override's nesting/restore semantics, mirroringsuppress_progress_output_contract's equivalent test forSuppressProgressOutput.Verification
cargo test -p rocm --bin rocmrun 15 consecutive times — all green (1346/1347 passed, 0 failed each run), confirming the race no longer reproduces.cargo test --workspace --no-fail-fastandcargo clippy --workspace --all-targets -- -D warningsboth clean.Clonederive not#[cfg(test)]-gated, and comments not explaining the remaining lock/override split) are addressed in the second commit.Scope and scenarios
Test-only change with no user-observable behavior, so there is no Gherkin scenario to add (AGENTS.md: purely internal changes need none).