Replace forked processes that fail to boot

[sapandiwakar]

What

Track each forked process until its boot callbacks finish. If a child does not report readiness within a configurable startup timeout, terminate it and let the existing reap-and-replacement path replace that configured process.

This applies only to fork mode. The default startup timeout is five minutes.

Why

ForkSupervisor currently replaces children only when waitpid2(..., WNOHANG) reports that they exited. A child that remains alive while blocked during boot or registration is never returned by waitpid2, so the supervisor can retain an unusable child indefinitely without replacing it.

For a dispatcher, this means the process can remain present while never reaching its polling loop. Due scheduled jobs then remain in solid_queue_scheduled_executions instead of being moved to ready executions.

This addresses the concrete recovery gap observed in #751. It does not attempt to detect a process that successfully booted and later stopped making progress.

Reproduction

A deterministic local reproduction is to block a dispatcher in an existing startup hook:

SolidQueue::Dispatcher.on_start { sleep 60 }

Start Solid Queue in fork mode with a dispatcher configured.

On main, the dispatcher child remains alive but never reaches its polling loop. Because it has not exited, waitpid2(..., WNOHANG) does not return its PID and the supervisor never replaces it.

On this branch, the behavior can be observed quickly by temporarily lowering the timeout:

SolidQueue.process_startup_timeout = 2.seconds
SolidQueue::Dispatcher.on_start { sleep 60 }

The supervisor logs the startup timeout, terminates that dispatcher PID, and starts its replacement. The replacement also times out while the hook remains blocked, demonstrating that each configured slot is continuously supervised.

The regression test uses the same failure shape by blocking a worker during registration. It verifies that the stalled child is terminated and replaced while a healthy sibling keeps the same PID.

How

• Create a readiness pipe for each fork.
• Have the child signal readiness after all boot callbacks complete.
• Track each unready PID using a monotonic startup timestamp.
• Terminate only the PID that exceeds the timeout, then reuse the existing replacement path.
• Close readiness descriptors during successful startup, replacement, and shutdown.
• Emit a warning containing the affected child PID.

A timed-out child is killed rather than gracefully terminated because a process blocked during boot cannot reach its run loop to complete a graceful stop.

Tests

• A fork blocked during registration is terminated and replaced.
• A healthy sibling is not restarted when another child times out.
• A child that becomes ready before the deadline survives a complete supervision cycle.
• Startup timeout warnings include the actual child PID.

Relates to #751.