bash_watch_test.rs spawns background tasks that block on a sentinel file, and releases them with a bare statement at the end of the happy path. Any non-linear exit — an assertion panic, or a test runner killing tests in flight — skips the release, and the spawned shell then polls forever at 20 Hz for a file that can no longer be created.
Found on this host as a real orphan: pid busy-waiting since the previous day, cwd deleted, 20 polls/sec, from a gate run whose report was otherwise clean.
(Line numbers against v0.55.1; symbols are the stable reference.)
Mechanism
The fixture spawns a shell that blocks on a sentinel (release_gate_command, :74):
format!("while [ ! -f {} ]; do sleep 0.05; done; printf '%s\\n' {}", ...)The release is a plain call at the end of the test body — e.g. bash_status_distinguishes_erased_watched_task_from_never_existing_task (:392):
let dir = configure_background(&mut aft); // -> tempfile::TempDir
let release = dir.path().join("erased-status-release");
let task_id = spawn(&mut aft, &release_gate_command(&release, "never-reached-erased-status"));
// ~30 lines of assertions ...
release_task(&release); // :436 <-- only runs if every assert passed
assert!(aft.shutdown().success());release_task is just fs::write(path, "go") (:91). It is never wrapped in a Drop guard, scopeguard, or defer — 7 call sites, all bare statements on the happy path.
Why the orphan is permanent rather than merely late: configure_background returns a tempfile::TempDir (:9). On panic, unwinding drops it and deletes the directory — which is where the sentinel would have been written. The spawned shell is now waiting on a path inside a directory that no longer exists, so no later run, cleanup pass, or manual retry can ever satisfy it. It polls until the machine reboots or someone kills it by hand.
Two ways in, both real:
- Assertion panic — any of the ~30 asserts between spawn and release.
- Runner abort —
.config/nextest.toml sets fail-fast = true at the default profile, so the first failure anywhere cancels the run and kills in-flight tests. The test never reaches its release line.
Evidence
Observed orphan (argv verbatim):
while [ ! -f /tmp/.tmp9tN7oT/erased-status-release ]; do sleep 0.05; done; printf '%s\n' never-reached-erased-status
cwd deleted, ~20 polls/sec, running since the prior day. The sentinel name and the never-reached-... payload both pin it to :395/:398. It originated in a gate run that reported nothing unusual — a leaked process leaves no trace in a green gate report, which is why this has likely been happening for a while without anyone noticing. It only surfaced here because an unrelated service audit listed stray processes in the cgroup.
Fix
Tie the release to scope exit rather than to reaching the end of the body — a Drop guard holding the path, so unwinding writes the sentinel:
struct ReleaseOnDrop(PathBuf);
impl Drop for ReleaseOnDrop {
fn drop(&mut self) { let _ = fs::write(&self.0, "go"); }
}Belt-and-braces options, if you want the runner-kill case covered too (a SIGKILL runs no destructor): give the gate loop a bounded iteration count, or have the spawned command exit on a deadline as well as on the sentinel. The Drop guard alone fixes the panic path, which is the one that produced this orphan.
Related
Same class as #282 (a resource created with no owner tracking its lifecycle), one level up: process instead of directory. #282's transient cache dirs were bounded in bytes but unbounded in count; these are bounded in CPU per instance but unbounded in count and effectively immortal.
bash_watch_test.rsspawns background tasks that block on a sentinel file, and releases them with a bare statement at the end of the happy path. Any non-linear exit — an assertion panic, or a test runner killing tests in flight — skips the release, and the spawned shell then polls forever at 20 Hz for a file that can no longer be created.Found on this host as a real orphan: pid busy-waiting since the previous day, cwd deleted, 20 polls/sec, from a gate run whose report was otherwise clean.
(Line numbers against v0.55.1; symbols are the stable reference.)
Mechanism
The fixture spawns a shell that blocks on a sentinel (
release_gate_command,:74):The release is a plain call at the end of the test body — e.g.
bash_status_distinguishes_erased_watched_task_from_never_existing_task(:392):release_taskis justfs::write(path, "go")(:91). It is never wrapped in aDropguard,scopeguard, ordefer— 7 call sites, all bare statements on the happy path.Why the orphan is permanent rather than merely late:
configure_backgroundreturns atempfile::TempDir(:9). On panic, unwinding drops it and deletes the directory — which is where the sentinel would have been written. The spawned shell is now waiting on a path inside a directory that no longer exists, so no later run, cleanup pass, or manual retry can ever satisfy it. It polls until the machine reboots or someone kills it by hand.Two ways in, both real:
.config/nextest.tomlsetsfail-fast = trueat the default profile, so the first failure anywhere cancels the run and kills in-flight tests. The test never reaches its release line.Evidence
Observed orphan (argv verbatim):
cwddeleted, ~20 polls/sec, running since the prior day. The sentinel name and thenever-reached-...payload both pin it to:395/:398. It originated in a gate run that reported nothing unusual — a leaked process leaves no trace in a green gate report, which is why this has likely been happening for a while without anyone noticing. It only surfaced here because an unrelated service audit listed stray processes in the cgroup.Fix
Tie the release to scope exit rather than to reaching the end of the body — a
Dropguard holding the path, so unwinding writes the sentinel:Belt-and-braces options, if you want the runner-kill case covered too (a SIGKILL runs no destructor): give the gate loop a bounded iteration count, or have the spawned command exit on a deadline as well as on the sentinel. The
Dropguard alone fixes the panic path, which is the one that produced this orphan.Related
Same class as #282 (a resource created with no owner tracking its lifecycle), one level up: process instead of directory. #282's transient cache dirs were bounded in bytes but unbounded in count; these are bounded in CPU per instance but unbounded in count and effectively immortal.