Skip to content

Distinguish a zombie postmaster from a live one in get_pgpid() - #1173

Merged
dimitri merged 1 commit into
mainfrom
fix/1166-zombie-postmaster-detection
Jul 28, 2026
Merged

Distinguish a zombie postmaster from a live one in get_pgpid()#1173
dimitri merged 1 commit into
mainfrom
fix/1166-zombie-postmaster-detection

Conversation

@dimitri

@dimitri dimitri commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Fixes #1166.

Background

The reporter fault-injected a SIGKILL against Postgres mid-startup (before it writes PM_STATUS_READY) and found the node hangs indefinitely: postmaster.pid still points at a pid ps shows as <defunct>, health stays 0, no FSM progress, no monitor heartbeat, until pg_autoctl is restarted by hand.

Root cause, confirmed against current main: get_pgpid() (src/bin/common/pgsetup.c) probes liveness with a bare kill(pid, 0) == 0. A zombie -- exited but not yet reaped -- still holds a process-table entry, so the kernel reports it alive even though it will never do anything again.

get_pgpid() is read from five different processes sharing the same on-disk postmaster.pid file: the postgres controller sub-process (the actual parent -- it fork()s + execv()s postgres directly in service_postgres_start()), the keeper/node-active process, the monitor reporting service, and one-shot CLI commands. Because every one of them reaches the same false "alive" conclusion from the same buggy probe, none of them ever decides Postgres needs restarting: the controller sees nothing to do, the keeper waits forever for a ready status a dead process will never write, and the monitor never gets a heartbeat. That's the "forever" in the reported symptom -- not that the zombie is technically unreapable, but that nothing ever concludes it needs replacing.

Fix

pid_is_alive() resolves the ambiguity with waitpid(pid, WNOHANG) first:

  • From the controller (the true parent), this reaps the zombie right there and reports not-alive, or reports the child is genuinely still starting up (slow-startup behavior unchanged).
  • From every other caller, waitpid() on a pid that isn't their own child returns -1/ECHILD immediately, falling back to the historical kill(pid, 0) probe -- no behavior change there.

This matches the reporter's own proposed fix.

Verification

  • Standalone harness exercising the exact failure mode (fork + exit without reaping): kill(pid, 0) reports the zombie alive as expected (reproducing the bug), pid_is_alive() correctly reports it dead, including on a repeated call after the process has actually been reaped.
  • Clean build, make docker-check (citus_indent) and ci/banned.h.sh both clean.
  • Full Docker pgaftest run of basic_operation.pgaf (27/27) and timeline_fork_report_lsn_deadlock.pgaf (6/6) -- both exercise get_pgpid() heavily across ordinary start/stop and failover paths, no regression.

The reporter's own fault-injection scenario (blade kill --signal 9 mid-startup) isn't independently reproduced end-to-end here -- it needs precise timing against a real Postgres startup under a fault-injection tool -- so this is verified via a standalone harness that isolates and reproduces the exact process-liveness ambiguity, plus the broader regression suite for the surrounding start/stop/failover paths.

…id()

get_pgpid() probed liveness with a bare kill(pid, 0) == 0. A process that
has exited but not yet been reaped (a zombie) still holds a process-table
entry, so kill(pid, 0) reports it alive even though it will never do
anything again -- e.g. when Postgres is SIGKILLed mid-startup, before it
gets to write PM_STATUS_READY.

get_pgpid() is read from five different processes: the postgres controller
sub-process (the actual parent, since it fork()s + execv()s postgres
directly in service_postgres_start()), the keeper/node-active process, the
monitor reporting service, and one-shot CLI commands -- all sharing the
same on-disk postmaster.pid file. Because every one of them reaches the
same false alive conclusion from the same buggy probe, none of them ever
decides Postgres needs restarting: the controller sees nothing to do, the
keeper waits forever for a ready status a dead process will never write,
and the monitor never gets a heartbeat.

pid_is_alive() resolves this with waitpid(pid, WNOHANG) first: from the
controller (the true parent), this reaps the zombie right there and
reports not-alive, or reports 0 when the child is genuinely still
starting up (slow-startup behavior unchanged). From every other caller,
waitpid() on a pid that isn't their own child returns -1/ECHILD
immediately, falling back to the historical kill(pid, 0) probe -- no
behavior change there.

Verified with a standalone harness exercising the exact failure mode
(fork + exit without reaping): kill(pid, 0) reports the zombie alive as
expected (reproducing the bug), pid_is_alive() correctly reports it dead,
including on a repeated call after the process has been reaped. Also
verified end to end: clean build, docker-check (citus_indent) and
banned.h.sh both clean, and a full Docker pgaftest run of
basic_operation.pgaf (27/27) and timeline_fork_report_lsn_deadlock.pgaf
(6/6) -- both exercise get_pgpid() heavily across ordinary start/stop and
failover paths with no regression.
@dimitri dimitri self-assigned this Jul 28, 2026
@dimitri dimitri added the bug Something isn't working label Jul 28, 2026
@dimitri
dimitri merged commit ef7aa1d into main Jul 28, 2026
73 checks passed
@dimitri
dimitri deleted the fix/1166-zombie-postmaster-detection branch July 28, 2026 14:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

keeper deadlocks on a zombie postmaster: get_pgpid() uses kill(pid, 0), which cannot distinguish a zombie from a live process

1 participant