Skip to content

[ML] PR D: linear child ownership and fault-injected lifecycle for Sandbox2 - #3187

Open
valeriy42 wants to merge 1 commit into
feature/sandbox2-pr-c-fs-net-policyfrom
feature/sandbox2-pr-d-lifecycle
Open

valeriy42 wants to merge 1 commit into
feature/sandbox2-pr-c-fs-net-policyfrom
feature/sandbox2-pr-d-lifecycle

Conversation

@valeriy42

@valeriy42 valeriy42 commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Rebuilds CSandboxedProcessSpawner around an explicit lifecycle state machine (Prepared -> Launched -> IdentityCaptured -> Registered -> Monitoring -> TerminationRequested -> CleanupRequired -> Reaped/Failed), stacked on #3184.

  • Non-throwing kill-and-reap guard, armed immediately after launch and PID capture, holding its own owning reference to the sandbox handle so cleanup is order-independent during unwinding.
  • Four injectable seams (pidfd acquisition, registry allocation, monitor-thread launch, sandbox completion) for deterministic fault injection.
  • Explicit pidfd outcome classification: a kernel lacking pidfd support (ENOSYS) is the only case that falls back to signalling the sandboxee through its owned monitor handle (SIGKILL, identity-safe, no numeric-PID lookup); every other pidfd error is treated as a resource/identity failure and fails registration outright. Numeric kill(pid, ...) does not appear anywhere in this file.
  • A CAS-controlled one-shot latch replacing two-boolean coordination for the timeout-vs-completion race.
  • CSandboxedProcessSpawnerLifecycleTest_Linux.cc: fault-injection coverage for every pidfd class, allocation/resource failures, stale-generation protection, PID-reuse identity binding, the CAS race, descriptor-baseline cleanup, and destructor-latency.

Notes

  • Not yet compiled or run on Linux/Sandbox2 in this session at the time of opening; being verified against a real Linux devbox in parallel.
  • E_Launched/E_CleanupRequired lifecycle states are declared but intentionally left unassigned (no natural single point without broader restructuring) — not silently dropped.
  • The SIGKILL-only assumption for the owned-monitor termination path (no graceful-SIGTERM variant is buildable against the pinned Sandbox2 dependency version) was cross-checked against the vendored monitor source during review.

…ted lifecycle

Replaces numeric-PID process control with an identity-bound spawner for
sandboxed pytorch_inference children, because a sandboxed child's PID can
be reused by an unrelated process while a stale monitor or a delayed
terminate call is still in flight.

CSandboxedProcessSpawner (new) drives every live child through an explicit
state machine (Prepared -> Launched -> IdentityCaptured -> Registered ->
Monitoring -> TerminationRequested -> CleanupRequired -> Reaped/Failed):

- A non-throwing kill-and-reap guard is armed immediately once a sandboxed
  process is launched and its PID captured, before any operation that could
  throw (registry insertion, monitor-thread construction/detach). It holds
  its own owning reference to the sandbox handle so cleanup is safe
  regardless of the destruction order of other locals during unwinding, and
  disarms only after registry insertion and monitor handoff have both
  succeeded.
- Four injectable seams (pidfd acquisition, registry allocation,
  monitor-thread launch, sandbox completion) allow every failure class to
  be exercised deterministically in tests, without waiting on real resource
  exhaustion.
- pidfd-acquisition outcomes are classified explicitly: a kernel that lacks
  pidfd support (ENOSYS) is the only case that falls back to signalling the
  sandboxee through its owned monitor handle (SIGKILL, identity-safe, no
  numeric-PID lookup); every other acquisition failure (ESRCH, EMFILE,
  ENFILE, ...) is treated as a resource/identity error and fails
  registration outright rather than silently choosing a weaker termination
  mechanism. Numeric kill(pid, ...) does not appear anywhere in this file.
- A one-shot CAS latch resolves the race between a timeout and the
  sandbox's own completion signal, replacing independent-boolean
  coordination with a single atomic decision that has exactly one winner.

CSandboxedProcessSpawnerLifecycleTest_Linux.cc adds fault-injection
coverage for every pidfd-acquisition outcome, allocation/resource
failures during registration and monitor launch, protection against a
stale monitor mutating a newer registration for a reused PID, protection
against signalling the wrong process after PID reuse, the CAS race
(deterministic interleavings, no wall-clock polling), descriptor-count
cleanup after every case, and destructor latency with a live child still
running.

Reviewed through several fault-injection and use-after-free fix rounds
this session; not yet compiled or executed against a real Linux/Sandbox2
toolchain.
@valeriy42
valeriy42 force-pushed the feature/sandbox2-pr-d-lifecycle branch from bed4aba to fa81668 Compare September 9, 2026 20:18
@valeriy42

Copy link
Copy Markdown
Contributor Author

Devbox verification (Linux x86_64, real Sandbox2 toolchain): `ml_test_sandbox` filtered to `CSandboxedProcessSpawnerLifecycleTest_Linux` — 13/13 test cases passed, 372/372 assertions, exit code 0.

This surfaced and fixed two real defects that macOS-only syntax checks couldn't catch:

  • Two BOOST_TEST_REQUIRE call sites compared non-streamable types (std::future_status, a std::map iterator), which is a hard compile error for that macro (it tries to print both operands on failure). Switched to plain BOOST_REQUIRE.
  • The forkserver warm-up used to stabilize the descriptor-count baseline was wired as a BOOST_GLOBAL_FIXTURE, whose constructor runs before Boost.Test finishes its own framework setup. Forking a real process that early corrupted the framework's internal state (every test case genuinely passed, but the run still reported a spurious "Incorrect setup: no test case executed" with a non-zero exit code). Moved the warm-up to a std::call_once-guarded call from inside the normal per-case fixture instead.

Also ran clang-format via the CI's own check image and fixed several workspace-planning-doc citations that had leaked into comments/commit messages (not resolvable outside this internal effort's own tracking) — both now clean.

Buildkite (macOS/Windows legs, full cross-platform CI) still pending.

@elasticsearchmachine

Copy link
Copy Markdown

Pinging @elastic/ml-core (Team:ML)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants