pg-pool 3.14: ended client (_ending/_ended true, socket closed) stays in _clients and is counted as busy forever

[ddoyle196]

Summary

pg-pool 3.14.0 / pg 8.23.0: after _remove() has ended a client (_ending and _ended true, socket destroyed, backend gone from pg_stat_activity), the client is sometimes still present in pool._clients. It is not in _idle, so it is counted as busy against max forever. Nothing ever removes it, because every later reaper (idleTimeoutMillis, maxLifetimeSeconds) acts only on release, and the client is never released again. Enough of these and pool.connect() waits time out with timeout exceeded when trying to connect while Postgres shows a handful of backends.

Every path in pg-pool that calls client.end() filters the client out of _clients first (_remove, the newClient connect-error path, the onConnect error path), so I cannot find how this state is reached by reading the source. I am filing this with the captured state in case it is recognisable.

Environment

• pg 8.23.0, pg-pool 3.14.0 (both current), Node 24, Linux container on Railway, PostgreSQL 18.6 over the private network (no proxy; verified from the Postgres side that established sockets on 5432 equal backends).
• Pool options: max: 20, min: 2, idleTimeoutMillis: 10000, connectionTimeoutMillis: 10000, maxLifetimeSeconds: 600, query_timeout: 30000, keepAlive: true, keepAliveInitialDelayMillis: 10000, ssl: { rejectUnauthorized: false }, options: "-c jit=off", and a Client subclass that only wraps connect(callback) to time the handshake (it calls the original callback exactly once).
• A pool.on('connect') hook that runs one SET query on each new client.
• Sentry's vendored @opentelemetry/instrumentation-pg is active (wraps Pool.prototype.connect, Client.prototype.connect and Client.prototype.query; the wrappers pass callbacks through).
• Two other pools in the process (max: 10 and max: 3) with the same options never showed this.

How it was observed

Every 30s a monitor walks pool._clients, skips anything in pool._idle, and logs any entry with _ending === true, then drops it from _clients (to keep production alive). Before that reconciliation, the same condition was visible as pool.totalCount exceeding the number of backends from this host in pg_stat_activity by exactly the number of such clients, over a 12.5 hour run in which totalCount climbed monotonically to max and waiting then went non-zero. Restarting the process was the only thing that cleared it.

The four captured clients

All appear within ~2 minutes of process start, during a burst of concurrent pool.query() / pool.connect() / pool.query-via-drizzle-transaction work from a warm-up job plus the first user requests. All four had been used normally (5 to 49 releases) before being ended. Fields are read straight off the pg Client and the pool; heldMs is our own checkout stamp (null = not currently checked out by anyone); hasRelease = typeof client.release === 'function'; errorListeners = client.listenerCount('error'); expired = pool._expired.has(client).

2026-09-08T15:53:07Z {"pid":1226948,"connecting":false,"connected":true,"ending":true,"ended":true,"readyForQuery":true,"queryable":true,"activeQuery":null,"queued":0,"poolUseCount":49,"hasRelease":true,"expired":false,"errorListeners":1,"socketDestroyed":true,"socketReadyState":"closed","bytesRead":110983,"bytesWritten":28056,"poolClients":4,"poolIdle":2,"poolPending":0}
2026-09-08T16:07:20Z {"pid":1228398,"connecting":false,"connected":true,"ending":true,"ended":true,"readyForQuery":true,"queryable":true,"activeQuery":null,"queued":0,"poolUseCount":5,"hasRelease":true,"expired":false,"errorListeners":1,"socketDestroyed":true,"socketReadyState":"closed","bytesRead":3742,"bytesWritten":2850,"poolClients":3,"poolIdle":1,"poolPending":0}
2026-09-08T16:39:37Z {"pid":1231638,"connecting":false,"connected":true,"ending":true,"ended":true,"readyForQuery":true,"queryable":true,"activeQuery":null,"queued":0,"poolUseCount":18,"hasRelease":true,"expired":false,"errorListeners":1,"socketDestroyed":true,"socketReadyState":"closed","bytesRead":130618,"bytesWritten":12229,"poolClients":5,"poolIdle":3,"poolPending":0}
2026-09-08T16:44:12Z {"pid":1232110,"connecting":false,"connected":true,"ending":true,"ended":true,"readyForQuery":true,"queryable":true,"activeQuery":null,"queued":0,"poolUseCount":10,"hasRelease":true,"expired":false,"errorListeners":1,"socketDestroyed":true,"socketReadyState":"closed","bytesRead":7391,"bytesWritten":5038,"poolClients":5,"poolIdle":3,"poolPending":0}

Reading of that state: queryable: true and no active query mean end() took the graceful connection.end() branch; errorListeners: 1 and hasRelease: true mean the client had been through _release() (idle listener re-attached) and not re-acquired since; expired: false rules out the maxLifetimeSeconds path (and the process was under 600s old). That is the shape of the idleTimeoutMillis reaper calling _remove(), with _clients still containing the client afterwards.

Ruled out

• Application code, drizzle-orm (node-postgres driver), kysely, better-auth, and the Sentry pg wrappers: none call client.end() on a pooled client, and none touch _clients.
• The connect-timeout path (fix(pg-pool): honor connectionTimeoutMillis and don't leak timed-out connections #3711 / fix(pg-pool): honor connectionTimeoutMillis and prevent connection leaks/hangs #3721 look related but different): these clients completed their handshake and served queries.
• Not reproducible locally against a real Postgres with the same options and a heavier burst, but without TLS and without the Sentry wrappers.

Question

Is there a known ordering in _release / _remove / _pulseQueue under concurrent release and checkout where this._clients = this._clients.filter(...) can be followed by an assignment that reinstates the removed client? If not, what would you like instrumented? I can add a _clients write trace to the production process and report back.

[ddoyle196]

Two more, and these are NOT within a boot burst: the process had been up for 12 and 20 minutes and was serving ordinary user traffic (a few requests per second). Same shape as the four above: used normally, then ended by the idle reaper, still listed in _clients.

2026-09-08T17:17:48Z {"heldMs":null,"pid":1235411,"connecting":false,"connected":true,"ending":true,"ended":true,"readyForQuery":true,"queryable":true,"activeQuery":null,"queued":0,"poolUseCount":8,"hasRelease":true,"expired":false,"errorListeners":1,"socketDestroyed":true,"socketReadyState":"closed","bytesRead":2937,"bytesWritten":4467,"poolClients":6,"poolIdle":4,"poolPending":0}
2026-09-08T17:25:18Z {"heldMs":null,"pid":1236055,"connecting":false,"connected":true,"ending":true,"ended":true,"readyForQuery":true,"queryable":true,"activeQuery":null,"queued":0,"poolUseCount":148,"hasRelease":true,"expired":false,"errorListeners":1,"socketDestroyed":true,"socketReadyState":"closed","bytesRead":545550,"bytesWritten":116940,"poolClients":5,"poolIdle":3,"poolPending":0}

So the condition is load-related rather than startup-related. Roughly 1-2 per hour of active use here.

[ddoyle196]

Update with a strong correlation. Overnight, with ordinary low traffic, a new one of these appeared every ten minutes on the dot (10:11:31, 10:21:33, 10:31:34, 10:41:30, 10:51:32, 11:01:37 UTC), each on a client with ~120 releases, i.e. the one warm connection this app keeps between bursts. Our pool had maxLifetimeSeconds: 600. So the sequence per client is: 600s of normal use, the lifetime timer fires while it is idle, pg-pool retires it via _acquireClient(client, new PendingItem((err, client, clientRelease) => clientRelease()), idleListener, false) -> _release -> _remove -> end(), and afterwards the client is still in _clients (not in _idle, expired: false because _release deleted it from _expired).

I could not reproduce the lifetime path leaving a client behind locally (Node 20, same options, TLS, traffic every 150ms, lifetime 3s: clean every time), so there is still a production-only ingredient; production is Node 24 on Linux. We have set maxLifetimeSeconds back to 0 to confirm the ten-minute cases stop. The boot-time cases in the original report did not involve the lifetime timer (process age < 600s), so there are at least two ways in.