fix(pool): fence stale automatic-primary checkouts

[farnoy]

This is an AI-assisted PR. Consider this an early draft for now and feel free to recommend significant changes.

I ran into an issue where PgDog was not routing queries to my role=auto instances after an AWS-induced RDS Aurora failover event. The failover was caused by excessive load on CPU & drained freeable memory. After the event, I've seen a lot of SQLSTATE 25006s - indicative of PgDog routing writer queries to what it is now a read replica. I believe PgDog should be adjusted to keep track of the latest & best evidence as to who's the primary.

With the fix applied, I was able to avoid these errors in reproduction scenarios, using an automated testing harness. I only tested this patch with the settings I use: role=auto, pooler_mode=session, query_parser=off, and having two app-side connection pools using the pgdog.role=replica/primary option.

I will follow up with timing measurements. This patch series successfully avoids 25006 errors, but I haven't yet measured the unavailability window from the client perspective.

---

Summary

During a managed PostgreSQL failover, PgDog can retain an automatic target's old Primary role after the backend that supplied that evidence disappears. When the database returns as a reader, a fresh session-pool checkout can then bind to that reader. The client receives SQLSTATE 25006 (cannot execute ... in a read-only transaction), and session pooling can keep returning the same bad backend until the frontend session is replaced.

This change makes automatic-primary selection require valid, non-replica role evidence and adds an independent checkout-time pg_is_in_recovery() fence before handing a backend to a write client. A backend found to be in recovery is closed, its automatic-role evidence is cleared, and selection retries another qualified target.

PgDog version

Reproduced on the incident-era PgDog revision (eff27d42) and on upstream commit 5e8b8858. The proposed change is based on v0.1.54 (7b40c0c2), whose additional commit changes only version metadata.

Description

The failure requires a narrow sequence:

1. PgDog is using role = auto with session pooling and has identified a backend as the automatic primary.
2. A managed PostgreSQL failover causes the old writer's connections to close while monitoring and topology information are changing.
3. The old database instance returns or remains reachable as a standby during a short stale-role window.
4. PgDog still has the target labelled Primary, or reuses that target before fresh writer evidence is available.
5. New client sessions are checked out against that target and are pinned by session pooling.

The resulting error is produced after a query reaches the wrong backend. A query-level retry therefore does not guarantee a fresh backend when session pooling has pinned the stale binding.

The failure is easiest to observe with sustained concurrent client traffic, short requests that identify the backend role, and a failover triggered while the clients are reconnecting. The application authentication method is not the root cause. The tested setup used IAM backend authentication and query_parser = "off", but the relevant conditions are automatic role selection, session-pool rebinding, and a stale reader/writer topology window.

How it was observed

The test used one synchronized failover event for all matrix arms. Each arm ran the same client workload and recorded, for every checkout:

• the PgDog/frontend connection identity;
• the backend address and process identity;
• pg_is_in_recovery() / writer-versus-reader role;
• query errors, including genuine 25006 responses; and
• whether the client later rebound to a healthy writer.

The failover was considered relevant only when direct database role checks and the client observations bracketed the same topology change. This distinguishes a stale-reader rebind from a separate case where an already-held backend is demoted after a query has already been sent.

The compact matrix compared five arms on the same failover event:

Arm	Purpose
Incident-era vanilla	Positive control matching the affected PgDog behavior
Upstream socket-liveness fix (#1318)	Tests whether detecting closed idle sockets is sufficient
Upstream #1323 (b050a570) bundled change	Starts unelected Auto targets as replicas and waits for a primary on write checkout; this arm does not isolate those two behaviors
Current upstream baseline at the time of testing (5e8b885).	Current baseline
Proposed port	Qualified-primary evidence plus checkout-time backend fencing

The final slim series was then run in five synchronized events of the same matrix. The counts below are totals across those completed events; the proposed arm had 60 paired clients per event. The socket-liveness comparator (#1318) was also present in the matrix, but its partial-run total is omitted from this concise summary.

Arm	Stale reader binds	Genuine 25006	Clients proven wedged within the observation window	Writer binds
Incident-era vanilla	31	334	12	—
Upstream #1323 (b050a570) bundled change	2	55	2	—
Current upstream	1	28	1	—
Proposed two-commit series	0	0	0	300/300

These are strict observation-window totals and descriptive developmental evidence, not a general safety proof or a formal treatment-efficacy claim. The formal recovery deadline right-censored the completed event outcomes. The preceding ten-event broad prototype showed the same directional contrast, but those results are retained as developmental evidence only.

Root cause

Automatic role detection and checkout treated cached role state as sufficient to select a writer. Monitoring could lose, clear, or fail to refresh the LSN/recovery evidence without revoking the target's cached Primary role. A later checkout therefore had no independent guard against receiving a backend that was now in recovery.

The important distinction is between:

• a backend that was already handed to a client and is later demoted; and
• a new checkout that can still be prevented from binding to a known standby.

This PR addresses the second, incident-relevant path. It does not claim to migrate an in-flight PostgreSQL session or replay an ambiguous query.

Fix

The series is intentionally split into two reviewable commits:

1. Automatic-primary role selection now requires valid, non-replica role evidence. Unavailable, timed-out, malformed, or failed monitoring evidence revokes stale automatic-primary qualification.
2. Before an automatic-primary backend is returned, PgDog checks pg_is_in_recovery() within the existing checkout timeout. A standby is force-closed, its cached automatic-role evidence is cleared, and PgDog retries another qualified primary. Probe failures fail closed rather than handing the client an unverified writer.

The checkout probe is the correctness barrier. Evidence revocation prevents the load balancer from repeatedly selecting the same stale target while the topology converges.

Testing

Local validation on the final slim series:

• formatting and diff checks: passed;
• workspace check: passed;
• all-workspace, all-target Clippy with warnings denied: passed;
• seven focused automatic-role/evidence/checkout regressions: passed (the broader 19-test focused set also passed);
• full disposable-PostgreSQL suite: 2,384 passed, 7 skipped;
• release build and IAM/session/parser-off configuration check: passed;
• offline listener, OpenMetrics, and clean-shutdown smoke test: passed.

Synchronized failover rerun: five completed events in the same five-arm matrix; the partial, non-claimable results are summarized above. Raw artifacts and provenance are retained by the submitter and are available to reproduce the summary if maintainers want the full diagnostic detail.

Configuration

No new configuration is required. The behavior applies to automatic-role targets. Static primary and replica configurations retain their existing semantics.

Scope and limitations

This series targets fresh stale-reader checkouts after a topology change. It does not make arbitrary in-flight PostgreSQL operations replayable, and it does not transfer transaction state, cursors, temporary objects, prepared statements, or advisory locks between servers. A query whose outcome is ambiguous when a backend connection dies still requires the client/application's existing error-handling policy.

The checkout probe adds a bounded round trip for automatic-primary checkouts. That is an intentional availability/correctness tradeoff during failover: PgDog may return a checkout error while writer identity cannot be verified, rather than risk sending a write to a standby.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}