Skip to content

proxy/server: finish the RPC when the client half-closes - #628

Open
sfc-gh-pkowalewski wants to merge 1 commit into
mainfrom
pkowalewski/proxy-dispatch-donechan-deadlock
Open

proxy/server: finish the RPC when the client half-closes#628
sfc-gh-pkowalewski wants to merge 1 commit into
mainfrom
pkowalewski/proxy-dispatch-donechan-deadlock

Conversation

@sfc-gh-pkowalewski

Copy link
Copy Markdown
Collaborator

What is wrong

When the request channel closes, dispatch calls ClientCloseAll and returns straight away, which leaves doneChan without a reader. Any target stream that finishes after that point blocks forever trying to report itself, so its wg.Done never runs, TargetStreamSet.Wait never returns and Proxy never returns.

The RPC therefore only ends when the client disconnects, and the status it reports is whatever the dead transport gives back. In practice that means almost every proxied call finishes as Unavailable: transport is closing, even though the call itself succeeded. Anything reading that status is misled: on our staging proxy 97.9% of proxy spans were recorded as errors, against roughly 3% real failures in the gRPC metrics.

The change

Keep reading doneChan after the client half-closes, and return once the stream set has drained. TargetStreamSet.Empty is new and reports whether any stream is still expected to report completion.

Nothing changes while the client is still sending. Streams are only ever added in Add and removed in Remove, and Remove is called from the doneChan handler alone, so an empty set means every goroutine has finished and Wait returns immediately.

Testing

proxy/proxy/rpcstatus_test.go asserts the status gRPC reports for the proxy RPC, which is the value tracing exporters turn into a span status. It covers a single target, several targets, and a server-streaming method. All three fail on the current code with Unavailable: transport is closing and pass with this change.

go test ./proxy/... -race -count=2 passes. One unrelated pre-existing failure in ./server (TestServerWithUnixCredentials, supplementary group subtests) also fails on an unmodified checkout on macOS.

Deployed on our staging proxy, this moved proxy RPCs recorded as failed from 97.9% to zero over several thousand calls.

When the request channel closed, dispatch called ClientCloseAll and returned
straight away, which left doneChan without a reader. Any target stream that
finished after that blocked forever trying to report itself, so its wg.Done
never ran, TargetStreamSet.Wait never returned and Proxy never returned. The
RPC only ended once the client disconnected, and it therefore reported
"Unavailable: transport is closing" nearly every time.

Keep reading doneChan after the client half-closes and return once the stream
set has drained. Nothing changes while the client is still sending.

On our staging proxy this took the share of proxy RPCs recorded as failed from
97.9% to zero, measured over several thousand calls.
@sfc-gh-pkowalewski
sfc-gh-pkowalewski force-pushed the pkowalewski/proxy-dispatch-donechan-deadlock branch from b845289 to b38a236 Compare August 8, 2026 09:19
@sfc-gh-pkowalewski
sfc-gh-pkowalewski marked this pull request as ready for review August 8, 2026 09:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants