- pg: 8.23.0
- Runtime: Cloudflare Workers (
workerd) via pg-cloudflare, nodejs_compat
- PostgreSQL: Neon, reached through Cloudflare Hyperdrive (transaction-mode pooler)
Summary
Connection.prototype.sync() sets this._ending = true, and nothing ever sets it back. Since sync() runs after every query in the extended protocol, _ending is permanently true from the first query onward, for a client that is not ending at all.
reportStreamError then silently drops ECONNRESET and EPIPE for the rest of that connection's life:
// lib/connection.js:54-59
const reportStreamError = function (error) {
// errors about disconnections should be ignored during disconnect
if (self._ending && (error.code === 'ECONNRESET' || error.code === 'EPIPE')) {
return
}
self.emit('error', error)
}// lib/connection.js:199-202
sync() {
this._ending = true
this._send(syncBuffer)
}The comment says during disconnect, and the flag no longer means that. Recovery from a mid-request teardown therefore rests entirely on the socket's close -> end -> _errorAllQueries path.
Where we hit it
With pipeline: true (8.22+) that recovery path races the pipelined bookkeeping, and an in-flight query's promise is sometimes never settled. Not rejected, not resolved: the awaiting code waits for ever.
In production this presented as a sign-in that hung indefinitely. The HTTP request never completed, so nothing was ever logged, while pg_stat_activity on the server showed the connection idle in Client/ClientRead with its last query answered, for fourteen hours. The server had replied; the reply never reached the promise.
Reproduction
A Worker opening a fresh Client per round against a real Hyperdrive origin, running a sign-in-shaped chain of several sequential queries with a transaction interleaved on the same client (begin / set_config / ... / commit), 400 rounds per variant:
| variant |
hung for ever |
pipeline: true + interleaved transaction |
2 / 400 |
pipeline: false + interleaved transaction |
0 / 400 |
pipeline: true, no transaction |
0 / 400 |
Three things are needed together: pipelining on, more than one statement genuinely in flight (which the interleaved transaction guarantees), and a real origin teardown. A local proxy issuing RST/FIN rejected cleanly every time; only the real pooler, with its own connection recycling, produced the race at scale.
In both hangs the client-side state was:
events: ["client.error:Connection terminated unexpectedly", "client.end"]
readyForQuery: false
_activeQuery: still set
_ended: false
So the end handler had run while _activeQuery was still set: the queue never reached _errorAllQueries for that query.
Why we think the two are the same story
_ending being permanently true means the ordinary error road is closed before the socket-close road is even reached, so everything depends on a path that, on this runtime, is delivered asynchronously by cloudflare:sockets' .closed and races _pulsePipelinedQueryQueue. We did not pin the lost rejection to a single line inside that race, and we would rather say so than guess. The _ending behaviour above, however, is plain in the source and looks wrong on its own terms regardless of runtime.
Possibly the same family as #3707, where a missing Sync leaves ReadyForQuery unarriving and wedges the connection.
What we did
Turned pipeline off. The same sign-in step that never returned now answers in 88-115 ms, and the hang has not recurred.
Happy to run further instrumented variants of the probe if that would help; the environment is reproducible on our side.
workerd) viapg-cloudflare,nodejs_compatSummary
Connection.prototype.sync()setsthis._ending = true, and nothing ever sets it back. Sincesync()runs after every query in the extended protocol,_endingis permanently true from the first query onward, for a client that is not ending at all.reportStreamErrorthen silently dropsECONNRESETandEPIPEfor the rest of that connection's life:The comment says during disconnect, and the flag no longer means that. Recovery from a mid-request teardown therefore rests entirely on the socket's
close->end->_errorAllQueriespath.Where we hit it
With
pipeline: true(8.22+) that recovery path races the pipelined bookkeeping, and an in-flight query's promise is sometimes never settled. Not rejected, not resolved: the awaiting code waits for ever.In production this presented as a sign-in that hung indefinitely. The HTTP request never completed, so nothing was ever logged, while
pg_stat_activityon the server showed the connectionidleinClient/ClientReadwith its last query answered, for fourteen hours. The server had replied; the reply never reached the promise.Reproduction
A Worker opening a fresh
Clientper round against a real Hyperdrive origin, running a sign-in-shaped chain of several sequential queries with a transaction interleaved on the same client (begin/set_config/ ... /commit), 400 rounds per variant:pipeline: true+ interleaved transactionpipeline: false+ interleaved transactionpipeline: true, no transactionThree things are needed together: pipelining on, more than one statement genuinely in flight (which the interleaved transaction guarantees), and a real origin teardown. A local proxy issuing RST/FIN rejected cleanly every time; only the real pooler, with its own connection recycling, produced the race at scale.
In both hangs the client-side state was:
So the
endhandler had run while_activeQuerywas still set: the queue never reached_errorAllQueriesfor that query.Why we think the two are the same story
_endingbeing permanently true means the ordinary error road is closed before the socket-close road is even reached, so everything depends on a path that, on this runtime, is delivered asynchronously bycloudflare:sockets'.closedand races_pulsePipelinedQueryQueue. We did not pin the lost rejection to a single line inside that race, and we would rather say so than guess. The_endingbehaviour above, however, is plain in the source and looks wrong on its own terms regardless of runtime.Possibly the same family as #3707, where a missing Sync leaves
ReadyForQueryunarriving and wedges the connection.What we did
Turned
pipelineoff. The same sign-in step that never returned now answers in 88-115 ms, and the hang has not recurred.Happy to run further instrumented variants of the probe if that would help; the environment is reproducible on our side.