Skip to content

Fix #1168: wait_primary -> draining/demote_timeout assignment had no KeeperFSM edge - #1169

Merged
dimitri merged 2 commits into
mainfrom
fix/1168-wait-primary-draining
Jul 27, 2026
Merged

Fix #1168: wait_primary -> draining/demote_timeout assignment had no KeeperFSM edge#1169
dimitri merged 2 commits into
mainfrom
fix/1168-wait-primary-draining

Conversation

@dimitri

@dimitri dimitri commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

What

Fixes #1168: the monitor could assign a primary already at (converged)
wait_primary a goal state (draining, then demote_timeout) that
KeeperFSM[] has no edge for -- the keeper would fatal and retry
forever on it, and the cluster only recovered via the unrelated,
purely time-based drain-timeout self-heal, drain_timeout_ms late.

Includes:

  • tests/tap/specs/wait_primary_draining_deadlock.pgaf, a pgaftest
    spec that reproduces the bug end-to-end against a real 3-node
    cluster (primary + sync secondary + async secondary), added to
    tests/tap/schedule.
  • The actual fix in src/monitor/group_state_machine.c and
    src/monitor/node_metadata.c.

The bug, precisely

Reported against PR #1165 (the #774 fix):
#1165 (comment)

  1. node1 (primary), node2 (sync secondary), node3 (async,
    candidate-priority 50).
  2. node2 fails. Async standbys never count toward
    secondaryQuorumNodesCount, so this alone reassigns node1 to
    wait_primary for real, even with node3 healthy throughout.
  3. node1 itself becomes unhealthy while sitting at that converged
    wait_primary. The auto-failover trigger in
    ProceedGroupStateFromContext fires against node3 and used to
    unconditionally assign node1 a goal state with no KeeperFSM[]
    edge from wait_primary (only PRIMARY/JOIN_PRIMARY/
    APPLY_SETTINGS have those edges).

The fix, in three coordinated parts

  1. The trigger no longer reassigns primaryNode's goal at all when
    it's already converged to wait_primary -- there is nothing
    reachable to reassign it to, and no live standby to gracefully
    drain in the first place (wait_primary never had synchronous
    quorum).
  2. The unconditional prepare_promotion -> stop_replication follow-on
    rule likewise skips reassigning primaryNode's goal in that case
    (still unconditionally moves the candidate to stop_replication).
  3. A new NodeIsWaitPrimaryPresumedDead() applies the exact same
    drain_timeout_ms safety margin as NodeIsDrainTimeExpired before
    the completion rule commits the one real, reachable
    wait_primary -> demoted transition -- deliberately not shortened
    to the trigger's own (shorter) unhealthyTimeoutMs, and
    deliberately not gated on pgIsRunning (the common crash/partition
    case leaves a stale "running" self-report, so requiring it to be
    false first would make the wait permanent for exactly that case).

Anchoring that safety margin took two attempts: the first version
measured elapsed time from primaryNode's own reportTime, which
seemed right by analogy to NodeIsDrainTimeExpired -- but reportTime
keeps getting refreshed by the primary's own keeper the moment it
reconnects and resumes polling, even if it never actually converges on
anything, turning the bounded wait into a permanent stall. Caught this
by re-running this spec with the primary reconnecting mid-flight
(exactly the scenario a real network blip would produce) and watching
promotion never complete. Fixed by anchoring on the candidate's own
stateChangeTime instead, which only moves when the monitor reassigns
the candidate's own goal -- untouched by the old primary's unrelated
activity, and fixed in place the moment the candidate converges to
stop_replication.

Verification

  • pg_regress + isolation suites: 13 + 6 tests, unchanged, all green.
  • multi_async.pgaf: 27 tests, including the existing drain-timeout
    self-heal path at the same ~35s timing, all green.
  • demote_timeout_wait_primary_deadlock.pgaf: 3 tests, all green.
  • This PR's own spec (wait_primary_draining_deadlock.pgaf): asserts
    the fatal never happens and the cluster still converges within the
    same safety margin. Run twice for stability, both green.

dimitri added a commit that referenced this pull request Jul 27, 2026
…eachable goal state

The two-node/general auto-failover trigger in ProceedGroupStateFromContext
unconditionally assigned a healthy candidate's primary goal state
"draining" (and, one tick later, an unconditional follow-on rule
re-assigned it "demote_timeout"), neither of which KeeperFSM[] has an
edge for from wait_primary -- only PRIMARY/JOIN_PRIMARY/APPLY_SETTINGS
have those edges. The primary's keeper would fatal and retry forever on
that specific assignment; the cluster recovered anyway, but only via the
unrelated, purely time-based drain-timeout self-heal, drain_timeout_ms
late.

Root issue was reported against PR #1165 (the #774 fix):
#1165 (comment)

Fix, in three coordinated parts:

1. The trigger no longer reassigns primaryNode's goal at all when it's
   already converged to wait_primary -- there is nothing reachable to
   reassign it to, and no live standby to gracefully drain in the first
   place (wait_primary never had synchronous quorum).

2. The unconditional prepare_promotion -> stop_replication follow-on
   rule likewise skips reassigning primaryNode's goal in that case
   (still unconditionally moves the candidate to stop_replication).

3. A new NodeIsWaitPrimaryPresumedDead() applies the exact same
   drain_timeout_ms safety margin as NodeIsDrainTimeExpired before the
   completion rule commits the one real, reachable
   wait_primary -> demoted transition -- deliberately not shortened to
   the trigger's own (shorter) unhealthyTimeoutMs, and deliberately not
   gated on pgIsRunning (the common crash/partition case leaves a stale
   "running" self-report, so requiring it to be false first would make
   the wait permanent for exactly that case).

Anchoring that safety margin took two attempts: the first version
measured elapsed time from primaryNode's own reportTime, which seemed
right by analogy to NodeIsDrainTimeExpired -- but reportTime keeps
getting refreshed by the primary's own keeper the moment it reconnects
and resumes polling, even if it never actually converges on anything,
turning the bounded wait into a permanent stall. Caught this by
re-running the #1169 repro spec with the primary reconnecting mid-flight
(exactly the scenario a real network blip would produce) and watching
promotion never complete. Fixed by anchoring on the *candidate's* own
stateChangeTime instead, which only moves when the monitor reassigns
the candidate's own goal -- untouched by the old primary's unrelated
activity, and fixed in place the moment the candidate converges to
stop_replication.

Verified: pg_regress + isolation suites (13+6 tests, unchanged),
multi_async.pgaf (27 tests, including the existing drain-timeout
self-heal path at the same ~35s timing), demote_timeout_wait_primary_deadlock.pgaf
(3 tests), and the #1169 repro spec updated to assert the fix (no fatal
log, cluster still converges within the same safety margin) -- all
green, run twice for stability.
@dimitri dimitri changed the title tests: add pgaftest spec reproducing #1168 (wait_primary -> draining/demote_timeout) Fix #1168: monitor no longer assigns wait_primary an unreachable goal state Jul 27, 2026
@dimitri dimitri changed the title Fix #1168: monitor no longer assigns wait_primary an unreachable goal state Fix #1168: wait_primary -> draining/demote_timeout assignment had no KeeperFSM edge Jul 27, 2026
@dimitri dimitri self-assigned this Jul 27, 2026
@dimitri dimitri added the bug Something isn't working label Jul 27, 2026
dimitri added 2 commits July 27, 2026 20:28
3-node cluster (node1 primary, node2 sync secondary, node3 async
secondary with candidate-priority 50 > 0). node2 fails, zeroing
secondaryQuorumNodesCount even though node3 stays healthy (async
standbys never count toward it), so node1 gets reassigned wait_primary.
node1 then itself goes unhealthy while at that converged wait_primary:
the auto-failover trigger in ProceedGroupStateFromContext fires against
node3 and hands node1 a goal (draining, then demote_timeout within the
same polling cycle) that has no KeeperFSM[] edge from wait_primary.

Verified against current origin/main: node1's keeper fatals/retries
with "does not know how to reach state" before self-healing via the
unrelated, purely time-based drain-timeout mechanism (goal re-targeted
straight to demoted). Ran twice locally via 'pgaftest run' against a
freshly built pg17 image, both green -- this spec documents the current
(buggy) recovery path and will need its assertions updated once #1168
is actually fixed (drop the log-contains check, assert node1 goes
straight to demoted without the intermediate fatal).
…eachable goal state

The two-node/general auto-failover trigger in ProceedGroupStateFromContext
unconditionally assigned a healthy candidate's primary goal state
"draining" (and, one tick later, an unconditional follow-on rule
re-assigned it "demote_timeout"), neither of which KeeperFSM[] has an
edge for from wait_primary -- only PRIMARY/JOIN_PRIMARY/APPLY_SETTINGS
have those edges. The primary's keeper would fatal and retry forever on
that specific assignment; the cluster recovered anyway, but only via the
unrelated, purely time-based drain-timeout self-heal, drain_timeout_ms
late.

Root issue was reported against PR #1165 (the #774 fix):
#1165 (comment)

Fix, in three coordinated parts:

1. The trigger no longer reassigns primaryNode's goal at all when it's
   already converged to wait_primary -- there is nothing reachable to
   reassign it to, and no live standby to gracefully drain in the first
   place (wait_primary never had synchronous quorum).

2. The unconditional prepare_promotion -> stop_replication follow-on
   rule likewise skips reassigning primaryNode's goal in that case
   (still unconditionally moves the candidate to stop_replication).

3. A new NodeIsWaitPrimaryPresumedDead() applies the exact same
   drain_timeout_ms safety margin as NodeIsDrainTimeExpired before the
   completion rule commits the one real, reachable
   wait_primary -> demoted transition -- deliberately not shortened to
   the trigger's own (shorter) unhealthyTimeoutMs, and deliberately not
   gated on pgIsRunning (the common crash/partition case leaves a stale
   "running" self-report, so requiring it to be false first would make
   the wait permanent for exactly that case).

Anchoring that safety margin took two attempts: the first version
measured elapsed time from primaryNode's own reportTime, which seemed
right by analogy to NodeIsDrainTimeExpired -- but reportTime keeps
getting refreshed by the primary's own keeper the moment it reconnects
and resumes polling, even if it never actually converges on anything,
turning the bounded wait into a permanent stall. Caught this by
re-running the #1169 repro spec with the primary reconnecting mid-flight
(exactly the scenario a real network blip would produce) and watching
promotion never complete. Fixed by anchoring on the *candidate's* own
stateChangeTime instead, which only moves when the monitor reassigns
the candidate's own goal -- untouched by the old primary's unrelated
activity, and fixed in place the moment the candidate converges to
stop_replication.

Verified: pg_regress + isolation suites (13+6 tests, unchanged),
multi_async.pgaf (27 tests, including the existing drain-timeout
self-heal path at the same ~35s timing), demote_timeout_wait_primary_deadlock.pgaf
(3 tests), and the #1169 repro spec updated to assert the fix (no fatal
log, cluster still converges within the same safety margin) -- all
green, run twice for stability.
@dimitri
dimitri force-pushed the fix/1168-wait-primary-draining branch from 9acaa35 to eb01e22 Compare July 27, 2026 18:31
@dimitri
dimitri merged commit 89d7164 into main Jul 27, 2026
73 checks passed
@dimitri
dimitri deleted the fix/1168-wait-primary-draining branch July 27, 2026 19:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Two-node auto-failover trigger can assign draining to a primary already at wait_primary (no KeeperFSM edge)

1 participant