Skip to content

test(proc-lifecycle): property-test the process-tree walk - #800

Draft
rominf wants to merge 1 commit into
mainfrom
test/proptest-proc-lifecycle
Draft

rominf wants to merge 1 commit into
mainfrom
test/proptest-proc-lifecycle

Conversation

@rominf

@rominf rominf commented Oct 30, 2026

Copy link
Copy Markdown
Collaborator
  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. — N/A, test-only.
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md. — N/A.
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3. — N/A, no message changes.

Why

A tree stop signals every PID that collect_process_tree returns for the service's root PID: terminate_verified walks once to snapshot the tree it will wait on and again to pick the signal targets. Since #499 routed rocmd's managed-service stop through terminate_verified, that one walk decides the kill set for both rocm services stop and rocmd. A wrong walk either leaves a GPU worker running or signals a process that is not the service's, and until now the walk had no test of its own: the existing tree tests only use small real process trees, where cycles and other odd tables never occur.

What changes

  • Refactor, no behaviour change. The traversal moves out of collect_process_tree into walk_process_tree, which takes an already-built parent-to-children table. collect_process_tree still reads /proc and hands the table over. The walk's doc now states its ordering: root first, every other PID after its parent, no PID twice.
  • Property tests (crates/rocm-core/src/process_tree_properties.rs, Unix-only, mounted as a child module of process.rs like examine_driver_proptests.rs):
    • Generator. Tables of one to six PIDs over a six-PID alphabet (including 0 and 1, so init's ppid == 0 shape occurs), with unconstrained parents, so shared parents, cycles and self-parents come up in a large share of cases. A consistent process tree has no cycle, but the table is built from one stat read per PID at different moments, so a cycle from a PID reused mid-scan is rare but cannot be ruled out, and the walk must terminate on it. A self-parent cannot come from /proc and is included as the one-node case of the same cycle.
    • walk_matches_reachability_oracle: the walk's PID set equals an independent breadth-first reachability oracle; the root leads; no PID is listed twice.
    • parents_are_listed_before_their_children: every member after the root comes after its parent, as the walk documents.
    • Each walk runs on a thread with a 5 s bound. A walk that spins on a cycle fails with its input instead of hanging the job (the process exits rather than letting proptest shrink by re-running a spinning walk; under nextest that ends one test).
    • generator_reaches_the_interesting_shapes guards against a vacuous suite. Over 1024 deterministic draws the generator reaches a shared parent in 567, a self-parent in 443, a cycle of two or more PIDs in 238, and a root lying on such a cycle (the shape where the walk's seen check matters) in 116; the floors are about half of each.
    • 4096 cases by default, the same depth therock_version_proptests.rs uses for cheap pure functions; PROPTEST_CASES overrides it, as in examine_driver_proptests.rs.

No new dependencies: proptest is already a rocm-core dev-dependency, so Cargo.lock, MANIFEST.md and the third-party notices are untouched.

Can the properties fail?

Each mutant was applied to the walk (or the generator), the suite run, and the original restored:

Mutation Caught by
Walk direct children only (grandchildren left behind) reachability oracle
Push a PID before the seen check (a PID signalled twice) oracle, parent-order
Seed the stack with root + 1 (a stranger adopted) oracle, parent-order
Drop the seen check 5 s bound: exits 101 printing the offending table
Sort everything after the root in descending order (child before parent) parent-order only
Generator forces ppid >= pid (no cycles) generator guard: cycle of two or more PIDs: 0/1024, floor 120

The walk itself passes every property on main; no defect was found in it.

Scope

  • Not covered here, and unchanged: the /proc side of the walk (read_parent_pid and the table assembly in collect_process_tree); the test's children_of mirrors that table builder rather than exercising it. A parse property for read_parent_pid (comm names containing ) or spaces) would be a natural follow-up.
  • rocmd's former ps-based descendant walk is not tested here because fix(rocmd): verify process identity before stopping a managed service #499 removed it; fix(rocmd): verify process identity before stopping a managed service #499 also ships the tests that a recorded PID whose start time no longer matches is refused, and that its subtree survives (stop_managed_service_refuses_a_pid_whose_recorded_identity_no_longer_matches, stop_managed_service_leaves_the_subtree_of_a_refuted_pid_running).

AGENTS.md §3

Test-only apart from a private refactor: no CLI output, exit code, file layout or runtime selection changes, so no Gherkin scenario is needed.

Verification

On Linux (WSL2):

  • cargo test -p e2e-cucumber -p rocm -p rocm-core -p rocm-dash-daemon -p rocm-engine-aim -p rocm-engine-lemonade -p rocm-engine-protocol -p rocm-engine-vllm -p rocmd -p xtask (the set cargo xtask affected selects): exit 0, 3501 passed.
  • cargo clippy --locked --workspace --all-targets -- -D warnings: exit 0.
  • cargo fmt --all -- --check: exit 0.
  • prek run --from-ref origin/main --to-ref HEAD: all hooks pass except the hawkeye license-header hook, which could not run here because the locally installed hawkeye is older than the version licenserc.toml targets; the new file carries the same SPDX header as its siblings, and the License header check (hawkeye) CI job uses the pinned version.

Not verified locally: Windows. The new code is cfg(not(windows)) throughout, so a Windows build does not compile it.

A tree stop signals every PID that `collect_process_tree` returns for
the root: `terminate_verified` walks once to snapshot the tree it will
wait on and again to pick the signal targets. Since the managed-service
stop in `rocmd` was routed through `terminate_verified` (#499), that
walk decides the kill set for both `rocm services stop` and `rocmd`. A
wrong walk either leaves a GPU worker running or signals a process that
is not the service's.

The walk is now a separate `walk_process_tree` over a parent-to-children
table, so it can run against synthetic tables instead of the live
`/proc`; `collect_process_tree` still builds the table from `/proc` and
hands it over, with no change in behaviour.

The properties generate tables of one to six PIDs over a six-PID
alphabet, so shared parents, cycles and self-parents come up often. A
consistent process tree has no cycle, but the table is built from one
`stat` read per PID at different moments, so a cycle from a PID reused
mid-scan is rare but cannot be ruled out, and the walk must terminate on
it. The properties check the walk against an independent breadth-first
oracle (exact reachable set, root first, no PID listed twice) and that
every member after the root comes after its parent, as the walk now
documents. Each walk runs under a time bound, so a walk that spins on a
cycle fails with its input instead of hanging the job. A companion test
asserts the generator keeps reaching each of those shapes, including a
root that lies on a cycle of two or more PIDs, so the suite cannot go
vacuous. 4096 cases by default; `PROPTEST_CASES` overrides that.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant