Skip to content

fix(pool): bound the on-release ping so a dead connection can't leak its permit#4350

Open
rslowinski wants to merge 1 commit into
transact-rs:mainfrom
rslowinski:fix/pool-permit-leak-unresponsive-connection
Open

fix(pool): bound the on-release ping so a dead connection can't leak its permit#4350
rslowinski wants to merge 1 commit into
transact-rs:mainfrom
rslowinski:fix/pool-permit-leak-unresponsive-connection

Conversation

@rslowinski

@rslowinski rslowinski commented Jul 23, 2026

Copy link
Copy Markdown

Fixes #4349.

Problem

When a PoolConnection is dropped, the pool spawns a task (return_to_pool) that pings the connection before returning it to the idle queue. If the peer has gone away silently — no RST/FIN, e.g. a NAT/firewall expired the flow, or a query was abandoned via tokio::time::timeout — the ping request is written successfully and then the read waits forever for a response that never comes.

Because that wait happens inside the drop-spawned task, it strands the task's DecrementSizeGuard, so the connection's pool permit is never released. Once this has happened max_connections times, every subsequent acquire() fails with PoolTimedOut and the pool never recovers without a process restart. There are no error logs, because nothing errors — the future simply never completes.

#4349 has the full write-up and a standalone reproduction.

Fix

Bound the on-release ping() with a timeout, following the pattern already used for the close_on_drop path (CLOSE_ON_DROP_TIMEOUT in take_and_close). On timeout the connection is treated as unresponsive and discarded with close_hard() — a local socket shutdown that cannot itself block on the dead peer — which releases the permit so the pool can open a replacement. No public API change; the bound is a sibling const of the existing one (5s).

The close_on_drop path was already bounded this way; this closes the same gap on the default return-to-pool path.

Test

tests/postgres/pool.rs drives a real PgPool against an in-process fake server that completes the startup handshake and then goes silent (keeps reading, never replies) — the exact state of a silently-dropped flow. It checks out a connection, drops it (arming the on-release ping against the silent peer), and asserts the next acquire() succeeds by opening a fresh connection rather than failing with PoolTimedOut.

  • Needs no DATABASE_URL (connects only to its own local listener), so it runs in CI.
  • Gated to the tokio runtime cells via required-features = ["postgres", "runtime-tokio"] (it uses a tokio fake server), and skipped under the other runtime matrix cells.
  • Passes with the fix (~5s, the ping bound); without the fix it fails with PoolTimedOut at acquire_timeout — verified locally by reverting the fix.

Notes for reviewers

  • I reused a hardcoded 5s bound to match CLOSE_ON_DROP_TIMEOUT rather than introduce a config knob; happy to make it configurable or derive it from acquire_timeout instead if you'd prefer.
  • The timeout arm calls close_hard() (socket shutdown only). If you'd rather also guard close_hard/close against a hypothetically-blocking shutdown, I can wrap those too, but they don't await the peer today.

This change was developed with AI assistance; I've reviewed it end-to-end and can speak to the design and trade-offs.

…its permit

When a PoolConnection is dropped, the pool spawns a task that pings the
connection before returning it to the idle queue. If the peer has gone
away silently (no RST/FIN — e.g. a NAT/firewall dropped the flow), that
ping is sent successfully and then waits forever for a response that
never arrives. Because the ping runs inside the drop-spawned task, the
unbounded wait strands that task's DecrementSizeGuard, permanently
leaking the connection's pool permit. Once this has happened
max_connections times, every subsequent acquire() fails with
PoolTimedOut and the pool never recovers.

Bound the on-release ping with a timeout, mirroring the existing
close-on-drop path (CLOSE_ON_DROP_TIMEOUT). On timeout the connection is
discarded via close_hard() (a local socket shutdown that cannot block),
which releases the permit so the pool can open a replacement.

Adds a regression test that drives a real PgPool against an in-process
fake server which completes the handshake then goes silent; it needs no
DATABASE_URL and runs on the tokio runtime cells.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dropping a PoolConnection whose connection is silently dead leaks the pool permit forever (return_to_pool cleanup never completes)

1 participant