Distinguish a zombie postmaster from a live one in get_pgpid() - #1173
Merged
Conversation
…id() get_pgpid() probed liveness with a bare kill(pid, 0) == 0. A process that has exited but not yet been reaped (a zombie) still holds a process-table entry, so kill(pid, 0) reports it alive even though it will never do anything again -- e.g. when Postgres is SIGKILLed mid-startup, before it gets to write PM_STATUS_READY. get_pgpid() is read from five different processes: the postgres controller sub-process (the actual parent, since it fork()s + execv()s postgres directly in service_postgres_start()), the keeper/node-active process, the monitor reporting service, and one-shot CLI commands -- all sharing the same on-disk postmaster.pid file. Because every one of them reaches the same false alive conclusion from the same buggy probe, none of them ever decides Postgres needs restarting: the controller sees nothing to do, the keeper waits forever for a ready status a dead process will never write, and the monitor never gets a heartbeat. pid_is_alive() resolves this with waitpid(pid, WNOHANG) first: from the controller (the true parent), this reaps the zombie right there and reports not-alive, or reports 0 when the child is genuinely still starting up (slow-startup behavior unchanged). From every other caller, waitpid() on a pid that isn't their own child returns -1/ECHILD immediately, falling back to the historical kill(pid, 0) probe -- no behavior change there. Verified with a standalone harness exercising the exact failure mode (fork + exit without reaping): kill(pid, 0) reports the zombie alive as expected (reproducing the bug), pid_is_alive() correctly reports it dead, including on a repeated call after the process has been reaped. Also verified end to end: clean build, docker-check (citus_indent) and banned.h.sh both clean, and a full Docker pgaftest run of basic_operation.pgaf (27/27) and timeline_fork_report_lsn_deadlock.pgaf (6/6) -- both exercise get_pgpid() heavily across ordinary start/stop and failover paths with no regression.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1166.
Background
The reporter fault-injected a
SIGKILLagainst Postgres mid-startup (before it writesPM_STATUS_READY) and found the node hangs indefinitely:postmaster.pidstill points at a pidpsshows as<defunct>, health stays 0, no FSM progress, no monitor heartbeat, untilpg_autoctlis restarted by hand.Root cause, confirmed against current
main:get_pgpid()(src/bin/common/pgsetup.c) probes liveness with a barekill(pid, 0) == 0. A zombie -- exited but not yet reaped -- still holds a process-table entry, so the kernel reports it alive even though it will never do anything again.get_pgpid()is read from five different processes sharing the same on-diskpostmaster.pidfile: the postgres controller sub-process (the actual parent -- itfork()s +execv()s postgres directly inservice_postgres_start()), the keeper/node-active process, the monitor reporting service, and one-shot CLI commands. Because every one of them reaches the same false "alive" conclusion from the same buggy probe, none of them ever decides Postgres needs restarting: the controller sees nothing to do, the keeper waits forever for a ready status a dead process will never write, and the monitor never gets a heartbeat. That's the "forever" in the reported symptom -- not that the zombie is technically unreapable, but that nothing ever concludes it needs replacing.Fix
pid_is_alive()resolves the ambiguity withwaitpid(pid, WNOHANG)first:waitpid()on a pid that isn't their own child returns-1/ECHILDimmediately, falling back to the historicalkill(pid, 0)probe -- no behavior change there.This matches the reporter's own proposed fix.
Verification
kill(pid, 0)reports the zombie alive as expected (reproducing the bug),pid_is_alive()correctly reports it dead, including on a repeated call after the process has actually been reaped.make docker-check(citus_indent) andci/banned.h.shboth clean.basic_operation.pgaf(27/27) andtimeline_fork_report_lsn_deadlock.pgaf(6/6) -- both exerciseget_pgpid()heavily across ordinary start/stop and failover paths, no regression.The reporter's own fault-injection scenario (
blade kill --signal 9mid-startup) isn't independently reproduced end-to-end here -- it needs precise timing against a real Postgres startup under a fault-injection tool -- so this is verified via a standalone harness that isolates and reproduces the exact process-liveness ambiguity, plus the broader regression suite for the surrounding start/stop/failover paths.