RemoteServer: an execute ticket can be stranded when a connection drops mid-flight, hanging serverPool's destructor forever

[Yaraslaut]

Summary

RemoteServer's execute-ordering ticket mechanism (takeExecuteTicket/awaitExecuteTurn/releaseExecuteTicket, remote.hpp ~1725-1800) is documented as structurally protected against a stranded ticket via ExecuteTicketGuard's RAII release, closing two prior instances of this exact bug class (#348, #351). A third occurrence reproduces reliably under morph::net::SocketBackend/SocketServer specifically, when many concurrent execute calls race a connection drop.

Reproduced ~5/15 runs (33%) with a deliberately modest stress shape (3 producer threads x 15 calls, dropped mid-stream) -- not a rare edge case.

Symptom

A ThreadPoolExecutor (the RemoteServer's own pool) hangs forever in its destructor, joining a worker thread that is itself blocked forever in RemoteServer::awaitExecuteTurn, waiting for an earlier ticket for the same model that was apparently taken but never released.

Evidence

Captured with gdb -p <pid> -batch -ex "thread apply all bt" on a genuinely hung run (Mull/clang-19 build, WSL Ubuntu 24.04, this issue is not toolchain-specific):

Thread 2 (serverPool worker):
#5  std::condition_variable::wait<...>
#6  morph::backend::RemoteServer::awaitExecuteTurn (mid=..., ticket=30) at remote.hpp:1752
#7  morph::backend::RemoteServer::ExecuteTicketGuard::awaitTurn () at remote.hpp:1015
#8  morph::backend::RemoteServer::dispatchExecute (...) at remote.hpp:1559
#9  morph::backend::RemoteServer::dispatchMessage (...) at remote.hpp:1325
...

Thread 1 (test's main thread):
#4  std::thread::join()
#5  morph::exec::ThreadPoolExecutor::~ThreadPoolExecutor () at executor.hpp:80
#6  <TEST_CASE body> at test_socket_backend.cpp:463

Ticket #30 never becomes current, meaning some earlier ticket for the same modelId was taken (in handleImpl's peek-decode) but its posted task's ExecuteTicketGuard never ran to release it. The run's own debug log shows dispatchMessage executing calls out of strict ticket order, and several call ids from that iteration never appear in the log at all before the hang.

What's been ruled out

• Not a linker/multi-TU artifact (unrelated to the Mull investigation in mutation: a Mull survivor is not evidence of a missing assertion — four sampled survivors all turn the suite red when applied by hand #434) -- this is a real runtime hang, confirmed with a live process and gdb, not a mutation-testing measurement question.
• Not ThreadPoolExecutor's own documented "concurrent-with-destruction post is lost" caveat (executor.hpp's ~ThreadPoolExecutor() doc comment) -- SocketServer::close() synchronously join()s every client thread before returning (socket_server.hpp ~93, ~117-121), and handleImpl's _pool.post() call happens synchronously on that same client thread before it can exit -- so every ticket-taking post() in this scenario necessarily completes before close() returns, and nothing can post() afterward. The queue-drain-on-destruction guarantee should therefore hold.
• Not a classic condition-variable lost-wakeup -- awaitExecuteTurn's cv.wait(lock, predicate) re-checks its predicate on entry under the same mutex releaseExecuteTicket mutates nextToRun under, so a notify_all() that fires strictly before a waiter's wait() call is still observed correctly by that waiter's own initial predicate check.

Not yet done

• The actual mechanism by which a ticket's owning task fails to reach its ExecuteTicketGuard's release (or reaches it but the release doesn't unblock the correct waiter) has not been isolated further -- the three ruled-out mechanisms above narrow it, but the real path has not been found by reading dispatchExecute/dispatchMessage alone.
• Not reproduced yet against SimulatedRemoteBackend or QtWebSocketBackend in isolation, to see whether this is specific to morph::net's connection-teardown shape (client threads joined synchronously, _regMtx/model lookups racing the same teardown) or general to any transport that can drop a connection mid-flight with executes still in the ticket queue.
• No fix attempted. Given An execute refused by the shutdown gate takes an ordering ticket it never releases, contradicting remote.hpp's own stated invariant #348/A throw out of dispatchExecute strands the execute-ordering ticket, the same way #348's shutdown gate did #351 both closed this exact bug class by making ticket release structural rather than per-call-site, the right fix likely lives in the same place -- but where the current structural guarantee actually breaks down under this transport's teardown ordering needs to be found first.

Repro

Reduced test (not yet merged, on a local branch): tests/net/test_socket_backend.cpp, TEST_CASE("SocketBackend: execute() racing a disconnect never leaves a Completion unresolved"). 3 iterations of: connect, spawn 3 threads each issuing 15 execute() calls against a shared SbEchoModel instance, drop the connection 2ms in, join the producer threads, assert every call settled client-side, then let the iteration's RemoteServer/ThreadPoolExecutor go out of scope. Hangs in that teardown roughly 1 run in 3.

🤖 Filed with Claude Code