fix(pool): bound the on-release ping so a dead connection can't leak its permit#4350
Open
rslowinski wants to merge 1 commit into
Open
Conversation
…its permit When a PoolConnection is dropped, the pool spawns a task that pings the connection before returning it to the idle queue. If the peer has gone away silently (no RST/FIN — e.g. a NAT/firewall dropped the flow), that ping is sent successfully and then waits forever for a response that never arrives. Because the ping runs inside the drop-spawned task, the unbounded wait strands that task's DecrementSizeGuard, permanently leaking the connection's pool permit. Once this has happened max_connections times, every subsequent acquire() fails with PoolTimedOut and the pool never recovers. Bound the on-release ping with a timeout, mirroring the existing close-on-drop path (CLOSE_ON_DROP_TIMEOUT). On timeout the connection is discarded via close_hard() (a local socket shutdown that cannot block), which releases the permit so the pool can open a replacement. Adds a regression test that drives a real PgPool against an in-process fake server which completes the handshake then goes silent; it needs no DATABASE_URL and runs on the tokio runtime cells.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4349.
Problem
When a
PoolConnectionis dropped, the pool spawns a task (return_to_pool) thatpings the connection before returning it to the idle queue. If the peer has gone away silently — no RST/FIN, e.g. a NAT/firewall expired the flow, or a query was abandoned viatokio::time::timeout— the ping request is written successfully and then the read waits forever for a response that never comes.Because that wait happens inside the drop-spawned task, it strands the task's
DecrementSizeGuard, so the connection's pool permit is never released. Once this has happenedmax_connectionstimes, every subsequentacquire()fails withPoolTimedOutand the pool never recovers without a process restart. There are no error logs, because nothing errors — the future simply never completes.#4349 has the full write-up and a standalone reproduction.
Fix
Bound the on-release
ping()with a timeout, following the pattern already used for theclose_on_droppath (CLOSE_ON_DROP_TIMEOUTintake_and_close). On timeout the connection is treated as unresponsive and discarded withclose_hard()— a local socket shutdown that cannot itself block on the dead peer — which releases the permit so the pool can open a replacement. No public API change; the bound is a siblingconstof the existing one (5s).The
close_on_droppath was already bounded this way; this closes the same gap on the default return-to-pool path.Test
tests/postgres/pool.rsdrives a realPgPoolagainst an in-process fake server that completes the startup handshake and then goes silent (keeps reading, never replies) — the exact state of a silently-dropped flow. It checks out a connection, drops it (arming the on-release ping against the silent peer), and asserts the nextacquire()succeeds by opening a fresh connection rather than failing withPoolTimedOut.DATABASE_URL(connects only to its own local listener), so it runs in CI.required-features = ["postgres", "runtime-tokio"](it uses a tokio fake server), and skipped under the other runtime matrix cells.PoolTimedOutatacquire_timeout— verified locally by reverting the fix.Notes for reviewers
CLOSE_ON_DROP_TIMEOUTrather than introduce a config knob; happy to make it configurable or derive it fromacquire_timeoutinstead if you'd prefer.close_hard()(socket shutdown only). If you'd rather also guardclose_hard/closeagainst a hypothetically-blocking shutdown, I can wrap those too, but they don't await the peer today.This change was developed with AI assistance; I've reviewed it end-to-end and can speak to the design and trade-offs.