Forked worker misses forwarded TERM, keeps claiming jobs until SIGKILL

[nerisa]

Environment

• solid_queue 1.4.0, Rails 7.x, Ruby 3.x
• Workers run as Kubernetes Deployments, scaled by an HPA that reacts to queue depth with an aggressive scale-up (0s stabilization window) so pods can go from "just provisioned" to "asked to terminate" within seconds — either because the HPA scales back down almost immediately after a burst, or because the node backing the pod (we run these workers on Spot-backed node groups) gets reclaimed. A rollout can also mass-replace pods faster than they finish booting. terminationGracePeriodSeconds is generous (100s+) specifically to give in-flight jobs a chance to finish, but that budget is wasted if the worker never reacts to the signal at all.

The race

Solid Queue's supervisor only acts on a received signal inside its supervise loop, via process_signal_queue — that loop runs after start_processes has already forked workers (lib/solid_queue/supervisor.rb):

def start
  boot              # registers the supervisor's own trap (enqueue-only)
  run_start_hooks   # can take a while (app boot, cache warmup, etc.)
  start_processes   # forks workers
  launch_maintenance_task
  supervise         # <- first place a queued signal is actually processed
end

The supervisor's trap itself only enqueues (lib/solid_queue/supervisor/signals.rb):

trap(signal) { signal_queue << signal; interrupt }

If TERM arrives during run_start_hooks (a slow app boot is common), it sits queued until supervise runs perform_graceful_termination, which forwards TERM to the just-forked workers via term_forks (lib/solid_queue/fork_supervisor.rb). Because this forwarding happens within microseconds of the fork, it lands before the child has run its own register_signal_handlers (lib/solid_queue/processes/supervised.rb, called from boot, i.e. after fork). The freshly-forked worker still holds the inherited copy of the supervisor's enqueue-only trap at the moment the signal arrives — so the forwarded TERM gets pushed onto a signal_queue that only supervisors ever drain. A worker process never calls process_signal_queue, so the signal is silently dropped: @stopped is never set, shutting_down? stays false, and the poller (lib/solid_queue/processes/poller.rb) keeps calling poll → ReadyExecution.claim indefinitely.

The worker keeps claiming and running jobs in a pod that Kubernetes believes is already draining, until the grace period expires and the pod is SIGKILLed. The in-flight claimed job is orphaned; five minutes later another supervisor's maintenance task detects the stale heartbeat and fails it with SolidQueue::Processes::ProcessPrunedError — for a job that may have completed or been mid-flight, not one that genuinely crashed.

We confirmed this with production logs across multiple pods, e.g. (paraphrased, timestamps redacted to relative offsets):

T+0.000  Started Supervisor(fork) pid: 7        # supervisor's trap already armed
T+0.010  preStop sends TERM to supervisor pid
T+2.900  Started Worker pid: 152                # forked ~3s later, after TERM already queued
T+2.970  Starting perform job                   # worker claims a job AFTER the TERM
T+101.7  Starting perform job                   # STILL claiming new jobs seconds before kill
T+102.0  pod killed, exit code 137 (SIGKILL)     # no graceful shutdown log lines at all

Notably absent from the whole window: Supervisor terminated gracefully, Supervisor terminated immediately, or Shutdown Worker — the shutdown path never runs at all for that worker.

There is a second, related manifestation of the same underlying gap: if termination is requested even earlier — while the app itself (not just the supervisor) is still booting, before on_start hooks have run — nothing in solid_queue's boot path checks for a pending termination, so the boot completes and a worker gets forked into a pod that's already being torn down.

Related issues

• Finished jobs ending up as failed with SolidQueue::Processes::ProcessPrunedError #589 — "Finished jobs ending up as failed with ProcessPrunedError" (open, no root-cause diagnosis)
• Need to retry jobs that died with ProcessPrunedError #591 — "Need to retry jobs that died with ProcessPrunedError" (open, asks for a retry mechanism rather than a fix)
• SolidQueue::Processes::ProcessPrunedError shown in "Failed jobs" and not being released back to their queues #422 — closed, similar symptom reported without a fork-timing/signal analysis

None of these identify the fork-timing/signal-forwarding race described above, so we're filing a new issue rather than commenting on an existing one.

How we worked around it (without patching the gem)

We close the race entirely from the outside, using only public hooks and our own container lifecycle scripts:

1. Worker announces "I can handle a graceful TERM" via a ready file, written from an after_boot hook (on_process_start) — which runs only once the worker's own TERM → stop trap is armed. The file's existence is the signal that this specific process (not just "a supervisor is up") is ready to shut down cleanly.
2. Our preStop hook waits for that ready file before sending TERM to the supervisor (bounded wait, falls back to a timeout marker if the worker never becomes ready). This guarantees TERM is only forwarded once the worker has already replaced the inherited enqueue-only trap with its own — closing the fork-timing race without touching solid_queue's source.
3. Liveness probing is kept on the supervisor's pidfile, not the ready file, so a healthy-but-still-booting supervisor isn't killed by kubelet while waiting for step 1.
4. For the earlier boot-time variant: our preStop hook writes a separate "terminating" marker immediately, before waiting for readiness. An on_start hook (on_supervisor_start, which runs before start_processes forks anything) checks that marker and exits before forking a worker at all if termination was already requested.

All of this lives in our app's solid_queue hooks and our container's start/stop scripts — nothing in lib/solid_queue/ is modified.

What would help upstream

We think this is worth fixing natively rather than requiring every Kubernetes deployer to reinvent the ready-file dance:

• Have the worker (child) re-arm its signal trap as the very first thing after fork, before the parent's term_forks has a chance to signal it — or have the supervisor briefly block sending TERM to a just-forked child until it acknowledges it's ready (e.g. via a pipe/socket rather than relying on trap timing).
• Alternatively, expose an official "worker ready" hook/notification (what we built ourselves as a ready file) so operators don't need out-of-band file-based coordination to know when it's safe to forward TERM.
• Document the boot-time gap (TERM received before on_start hooks run) and consider having the supervisor check for a pending signal before start_processes runs.

Happy to share our hook implementation if useful as a reference.