pg_autoctl: retry perform_failover on transient monitor deadlocks (#1004)

[dimitri]

Fixes #1004 (reopening/re-closing with an actual fix, since the original close had no linked PR or commit).

Background

#1004 reported pg_autoctl perform switchover failing outright on a monitor-side Postgres deadlock:

ERROR Monitor ERROR:  deadlock detected
ERROR Monitor DETAIL:  Process 18419 waits for ShareLock on transaction 3711; blocked by process 18418.
ERROR Monitor Process 18418 waits for ExclusiveLock on advisory lock [16385,1338977919,0,11]; blocked by process 18419.
ERROR Monitor CONTEXT:  while updating tuple (0,10) in relation "node"
ERROR SQL query: SELECT pgautofailover.perform_failover($1, $2)
FATAL Failed to perform failover/switchover, see above for details

and asked for pg_autoctl to retry transient errors like this rather than fail the whole command.

Three separate things were going on here, and I checked all three against current main:

1. The deadlock's root cause (inconsistent lock-acquisition order across different node-mutating monitor functions) is already fixed, by pgaftest: DSL/runner improvements + monitor fixes #1150 (LockNodeGroupAndFetch()/LockNodeGroupAndFetchByName(), unifying every node-mutating entry point onto a single, consistently-ordered lock-then-fetch helper). That PR never referenced Deadlocks during pg_auto_failover operations #1004, so the issue never got closed with an explanation.

2. The retry-policy request is not fixed. pg_autoctl has had a retry mechanism for exactly this SQLSTATE class since 2020 (monitor_retryable_error() in monitor.c, covering deadlock_detected/serialization_failure/etc, added in Implement a retry policy for deadlocks and other transient failures. #359) -- monitor_start_maintenance, monitor_stop_maintenance, and monitor_register_node all use it. But monitor_perform_failover() and monitor_perform_failover_allow_data_loss() -- the exact functions behind the reporter's failing command -- were never wired into it. Confirmed by reading current main: both just run their query once and fail hard on any error.

3. The secondary "stuck at demote_timeout forever" symptom the reporter also described (a production failover that left the old primary permanently parked at demote_timeout, requiring a manual restart) is a different mechanism entirely -- a state-machine deadlock (no legal KeeperFSM[] transition out of an assigned goal state), not a Postgres lock deadlock. That's already fixed too, independently: Fix #1025: self-fenced primary deadlocks forever on an unreachable goal state #1158 (Possible FAILURE STATE in State Machine #1025) and Fix #774: monitor can assign a primary an unreachable goal state during failover #1165 (How to recover from a current/assigned state combination without a possible state transition path? #774) both root-cause and close variants of a primary landing in an unreachable goal state during failover. Not the same bug as 1/2 above, but worth noting since the reporter conflated the two symptoms in the original issue.

Fix

• monitor_perform_failover() / monitor_perform_failover_allow_data_loss() (monitor.c) now take a bool *mayRetry out-parameter, following the exact convention already used by monitor_start_maintenance/monitor_stop_maintenance: set it and return false when monitor_retryable_error() says the failure is transient.
• cli_perform_failover() (cli_perform.c, backing both pg_autoctl perform failover and perform switchover) wraps the call in the same ConnectionRetryPolicy loop enable/disable maintenance already use.
• demoapp.c's own call site updated for the new signature (it already retries on its own schedule via its outer loop; no behavior change there).

Verification

• Clean build (make -C src/bin/pg_autoctl).
• make docker-check (citus_indent) and ci/banned.h.sh both clean.
• Full Docker run of basic_operation.pgaf: 27/27 pass, including test_020_multiple_manual_failover_verify_replication_slots which exercises perform_failover directly -- confirms no regression to the normal, non-deadlock path.

The deadlock itself is timing-sensitive and no longer reliably reproducible now that #1150 fixed the underlying lock-ordering bug, so this is a defense-in-depth fix for the residual case (a transient deadlock/serialization failure from some other, not-yet-identified race) rather than something with its own new regression test that manufactures the exact race.