Fix #1168: wait_primary -> draining/demote_timeout assignment had no KeeperFSM edge - #1169
Merged
Conversation
dimitri
added a commit
that referenced
this pull request
Jul 27, 2026
…eachable goal state The two-node/general auto-failover trigger in ProceedGroupStateFromContext unconditionally assigned a healthy candidate's primary goal state "draining" (and, one tick later, an unconditional follow-on rule re-assigned it "demote_timeout"), neither of which KeeperFSM[] has an edge for from wait_primary -- only PRIMARY/JOIN_PRIMARY/APPLY_SETTINGS have those edges. The primary's keeper would fatal and retry forever on that specific assignment; the cluster recovered anyway, but only via the unrelated, purely time-based drain-timeout self-heal, drain_timeout_ms late. Root issue was reported against PR #1165 (the #774 fix): #1165 (comment) Fix, in three coordinated parts: 1. The trigger no longer reassigns primaryNode's goal at all when it's already converged to wait_primary -- there is nothing reachable to reassign it to, and no live standby to gracefully drain in the first place (wait_primary never had synchronous quorum). 2. The unconditional prepare_promotion -> stop_replication follow-on rule likewise skips reassigning primaryNode's goal in that case (still unconditionally moves the candidate to stop_replication). 3. A new NodeIsWaitPrimaryPresumedDead() applies the exact same drain_timeout_ms safety margin as NodeIsDrainTimeExpired before the completion rule commits the one real, reachable wait_primary -> demoted transition -- deliberately not shortened to the trigger's own (shorter) unhealthyTimeoutMs, and deliberately not gated on pgIsRunning (the common crash/partition case leaves a stale "running" self-report, so requiring it to be false first would make the wait permanent for exactly that case). Anchoring that safety margin took two attempts: the first version measured elapsed time from primaryNode's own reportTime, which seemed right by analogy to NodeIsDrainTimeExpired -- but reportTime keeps getting refreshed by the primary's own keeper the moment it reconnects and resumes polling, even if it never actually converges on anything, turning the bounded wait into a permanent stall. Caught this by re-running the #1169 repro spec with the primary reconnecting mid-flight (exactly the scenario a real network blip would produce) and watching promotion never complete. Fixed by anchoring on the *candidate's* own stateChangeTime instead, which only moves when the monitor reassigns the candidate's own goal -- untouched by the old primary's unrelated activity, and fixed in place the moment the candidate converges to stop_replication. Verified: pg_regress + isolation suites (13+6 tests, unchanged), multi_async.pgaf (27 tests, including the existing drain-timeout self-heal path at the same ~35s timing), demote_timeout_wait_primary_deadlock.pgaf (3 tests), and the #1169 repro spec updated to assert the fix (no fatal log, cluster still converges within the same safety margin) -- all green, run twice for stability.
3-node cluster (node1 primary, node2 sync secondary, node3 async secondary with candidate-priority 50 > 0). node2 fails, zeroing secondaryQuorumNodesCount even though node3 stays healthy (async standbys never count toward it), so node1 gets reassigned wait_primary. node1 then itself goes unhealthy while at that converged wait_primary: the auto-failover trigger in ProceedGroupStateFromContext fires against node3 and hands node1 a goal (draining, then demote_timeout within the same polling cycle) that has no KeeperFSM[] edge from wait_primary. Verified against current origin/main: node1's keeper fatals/retries with "does not know how to reach state" before self-healing via the unrelated, purely time-based drain-timeout mechanism (goal re-targeted straight to demoted). Ran twice locally via 'pgaftest run' against a freshly built pg17 image, both green -- this spec documents the current (buggy) recovery path and will need its assertions updated once #1168 is actually fixed (drop the log-contains check, assert node1 goes straight to demoted without the intermediate fatal).
…eachable goal state The two-node/general auto-failover trigger in ProceedGroupStateFromContext unconditionally assigned a healthy candidate's primary goal state "draining" (and, one tick later, an unconditional follow-on rule re-assigned it "demote_timeout"), neither of which KeeperFSM[] has an edge for from wait_primary -- only PRIMARY/JOIN_PRIMARY/APPLY_SETTINGS have those edges. The primary's keeper would fatal and retry forever on that specific assignment; the cluster recovered anyway, but only via the unrelated, purely time-based drain-timeout self-heal, drain_timeout_ms late. Root issue was reported against PR #1165 (the #774 fix): #1165 (comment) Fix, in three coordinated parts: 1. The trigger no longer reassigns primaryNode's goal at all when it's already converged to wait_primary -- there is nothing reachable to reassign it to, and no live standby to gracefully drain in the first place (wait_primary never had synchronous quorum). 2. The unconditional prepare_promotion -> stop_replication follow-on rule likewise skips reassigning primaryNode's goal in that case (still unconditionally moves the candidate to stop_replication). 3. A new NodeIsWaitPrimaryPresumedDead() applies the exact same drain_timeout_ms safety margin as NodeIsDrainTimeExpired before the completion rule commits the one real, reachable wait_primary -> demoted transition -- deliberately not shortened to the trigger's own (shorter) unhealthyTimeoutMs, and deliberately not gated on pgIsRunning (the common crash/partition case leaves a stale "running" self-report, so requiring it to be false first would make the wait permanent for exactly that case). Anchoring that safety margin took two attempts: the first version measured elapsed time from primaryNode's own reportTime, which seemed right by analogy to NodeIsDrainTimeExpired -- but reportTime keeps getting refreshed by the primary's own keeper the moment it reconnects and resumes polling, even if it never actually converges on anything, turning the bounded wait into a permanent stall. Caught this by re-running the #1169 repro spec with the primary reconnecting mid-flight (exactly the scenario a real network blip would produce) and watching promotion never complete. Fixed by anchoring on the *candidate's* own stateChangeTime instead, which only moves when the monitor reassigns the candidate's own goal -- untouched by the old primary's unrelated activity, and fixed in place the moment the candidate converges to stop_replication. Verified: pg_regress + isolation suites (13+6 tests, unchanged), multi_async.pgaf (27 tests, including the existing drain-timeout self-heal path at the same ~35s timing), demote_timeout_wait_primary_deadlock.pgaf (3 tests), and the #1169 repro spec updated to assert the fix (no fatal log, cluster still converges within the same safety margin) -- all green, run twice for stability.
dimitri
force-pushed
the
fix/1168-wait-primary-draining
branch
from
July 27, 2026 18:31
9acaa35 to
eb01e22
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes #1168: the monitor could assign a primary already at (converged)
wait_primarya goal state (draining, thendemote_timeout) thatKeeperFSM[]has no edge for -- the keeper would fatal and retryforever on it, and the cluster only recovered via the unrelated,
purely time-based drain-timeout self-heal,
drain_timeout_mslate.Includes:
tests/tap/specs/wait_primary_draining_deadlock.pgaf, a pgaftestspec that reproduces the bug end-to-end against a real 3-node
cluster (primary + sync secondary + async secondary), added to
tests/tap/schedule.src/monitor/group_state_machine.candsrc/monitor/node_metadata.c.The bug, precisely
Reported against PR #1165 (the #774 fix):
#1165 (comment)
node1(primary),node2(sync secondary),node3(async,candidate-priority 50).node2fails. Async standbys never count towardsecondaryQuorumNodesCount, so this alone reassignsnode1towait_primaryfor real, even withnode3healthy throughout.node1itself becomes unhealthy while sitting at that convergedwait_primary. The auto-failover trigger inProceedGroupStateFromContextfires againstnode3and used tounconditionally assign
node1a goal state with noKeeperFSM[]edge from
wait_primary(onlyPRIMARY/JOIN_PRIMARY/APPLY_SETTINGShave those edges).The fix, in three coordinated parts
primaryNode's goal at all whenit's already converged to
wait_primary-- there is nothingreachable to reassign it to, and no live standby to gracefully
drain in the first place (
wait_primarynever had synchronousquorum).
prepare_promotion -> stop_replicationfollow-onrule likewise skips reassigning
primaryNode's goal in that case(still unconditionally moves the candidate to
stop_replication).NodeIsWaitPrimaryPresumedDead()applies the exact samedrain_timeout_mssafety margin asNodeIsDrainTimeExpiredbeforethe completion rule commits the one real, reachable
wait_primary -> demotedtransition -- deliberately not shortenedto the trigger's own (shorter)
unhealthyTimeoutMs, anddeliberately not gated on
pgIsRunning(the common crash/partitioncase leaves a stale "running" self-report, so requiring it to be
false first would make the wait permanent for exactly that case).
Anchoring that safety margin took two attempts: the first version
measured elapsed time from
primaryNode's ownreportTime, whichseemed right by analogy to
NodeIsDrainTimeExpired-- butreportTimekeeps getting refreshed by the primary's own keeper the moment it
reconnects and resumes polling, even if it never actually converges on
anything, turning the bounded wait into a permanent stall. Caught this
by re-running this spec with the primary reconnecting mid-flight
(exactly the scenario a real network blip would produce) and watching
promotion never complete. Fixed by anchoring on the candidate's own
stateChangeTimeinstead, which only moves when the monitor reassignsthe candidate's own goal -- untouched by the old primary's unrelated
activity, and fixed in place the moment the candidate converges to
stop_replication.Verification
pg_regress+ isolation suites: 13 + 6 tests, unchanged, all green.multi_async.pgaf: 27 tests, including the existing drain-timeoutself-heal path at the same ~35s timing, all green.
demote_timeout_wait_primary_deadlock.pgaf: 3 tests, all green.wait_primary_draining_deadlock.pgaf): assertsthe fatal never happens and the cluster still converges within the
same safety margin. Run twice for stability, both green.