Repository navigation
Join the connection's worker thread on close instead of at GC - #470
Conversation
close(), delete() and invalidate() drained the ThreadPool but left its worker running; it was only joined in ~ThreadPool, when the last shared_ptr (JS object, prepared statements, pending promise tasks) was released. Apps that open/close repeatedly in a long-lived process kept one idle thread per closed database until GC (an iOS background kill had 1,006 idle do_work threads). - ThreadPool::shutdown(): sets done under the mutex, wakes and joins the worker; idempotent and safe to call concurrently. close/delete/ invalidate call it after wait_finished(); ~ThreadPool calls it too. - queue_work on a shut pool throws (callers queue from a Promise executor, so the promise rejects instead of never settling). - A task that drops the last reference to its own pool no longer self-joins (std::system_error -> terminate): the worker detaches and do_work returns without touching the freed pool. Fixes OP-Engineering#469 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for this! I've gone through the changes. Can you take a look at the following Bugs1. The second op-sqlite/cpp/OPThreadPool.cpp Line 46 in 53f027b If a second caller (e.g. 2. The self-release flag is set in op-sqlite/cpp/OPThreadPool.cpp Line 67 in 53f027b Whenever 3. Line 425 in 53f027b On a remote-only libsql/Turso connection, 4. Tasks still in the queue at shutdown are dropped and their promises never settle op-sqlite/cpp/OPThreadPool.cpp Line 106 in 53f027b
5. Prepared statements behave inconsistently after
Cleanups6. The self-join guard can't be reached today op-sqlite/cpp/OPThreadPool.cpp Line 128 in 53f027b No queued task holds a 7. op-sqlite/cpp/OPThreadPool.hpp Line 50 in 53f027b Both are set together under the same lock, so 8. Line 367 in 53f027b Moving the drain into 9. The pool's error message doesn't match the other "database is closed" errors op-sqlite/cpp/OPThreadPool.cpp Line 85 in 53f027b
10. The destructor comment is wrong, and there's no test for the new path op-sqlite/cpp/OPThreadPool.cpp Line 38 in 53f027b Destructors don't run when a constructor throws. If creating a later thread fails, the threads already started are never joined and the program calls #2 and #4 are the most likely to show up in practice as hangs. |
…join Addresses the review of OP-Engineering#470 (points 1-4 and 6-10). shutdown() is now the single way to stop a pool. It refuses new work, lets the worker run every task that is already queued (so no promise is left unsettled and a later wait_finished() cannot block), then joins. close(), delete() and invalidate() call just that instead of wait_finished() followed by shutdown(). - Concurrent callers: the join runs under join_mutex, so a second shutdown() (or the destructor after it) returns only once the first caller's join has completed, never while the worker still uses the mutex, condition variable or queue. - Called on the worker itself (e.g. a database invalidated from a task), shutdown() does not join: it runs the remaining queued tasks inline and returns. do_work() then releases `busy` as usual and exits, and a later shutdown() or the destructor joins it. The thread_local self-release flag and the detach branch are gone; the header documents the rule that makes them unnecessary: a task must never own its pool (none does: tasks capture the database or statement by raw pointer). - `shut_down` is dropped; `done` is the only flag and is now read under the mutex only. - queue_work() on a stopped pool throws ThreadPoolStoppedError ("[op-sqlite] thread pool is stopped"). promisify and flushPendingReactiveQueries map it to the same "[op-sqlite][<fn>] database is closed" error as throw_if_closed, thrown from the Promise executor so the promise rejects. promisify takes the function name for that. - delete() checks the remote-only case before invalidating, interrupting or stopping the pool, so an unsupported delete() leaves the database open and usable. - The constructor joins any thread it already started if a later one fails to start: the destructor does not run when a constructor throws. scripts/thread_pool_check.cpp (run by scripts/test-thread-pool.sh, under TSan and ASan) covers 300 closed-but-referenced pools, queue_work after shutdown, draining queued tasks on shutdown() and in the destructor, two concurrent shutdowns followed by the destructor, shutdown() from a task on the worker, and a constructor whose thread cannot start. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Addresses point 5 of the review of OP-Engineering#470 (and the test asked for in 10). A statement keeps the raw connection handle it was prepared on. After close() the async path rejected only because the pool was stopped, while executeSync() and bindSync() still ran against the closed handle, which for libsql is already freed. PreparedStatementHostObject now holds a weak_ptr to its OPDatabase and checks it at the top of bind, bindSync, execute and executeSync, before anything touches the handle. A closed (or already destroyed) database throws the same "[op-sqlite][<fn>] database is closed" error as OPDatabase::throw_if_closed, with no `code`, whatever the pool's state. The JS wrapper for stmt.execute is now async, as bind already is, so the native throw reaches the caller as a rejection, the same way db.execute reports a closed database. executeSync/bindSync throw synchronously. example/src/tests/preparedStatements.ts covers both paths. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Thanks for the careful review. You were right on all ten points. I pushed two commits: 1. Concurrent 2. 3. 4. Queued tasks at shutdown. 5. Prepared statements after 6. Self-join guard. Removed. I checked first: no queued task owns the pool. 7. 8. Repeated 9. Error shape. The pool throws its own 10. Constructor and test. You're right that the destructor doesn't run if the constructor throws. The constructor now catches, stops and joins any thread it already started, and rethrows. I fixed the destructor comment too. Test (f) exhausts the process's threads (macOS caps it at ~4096) and checks that the constructor throws Testing. Not verified. I haven't run the new JS tests: I have no example-app build on this machine, so they mirror the structure of the existing ones. The CI run for this push is waiting for your approval to start; once it runs, the iOS and Android jobs will exercise them. The JSI translation units ( |
The pool only ever runs one thread per connection, and after OP-Engineering#470 nothing calls wait_finished() any more: shutdown() drains the queue and joins. That makes most of the pool's machinery unnecessary. - One std::thread, declared last, so a failed start needs no cleanup. This replaces the thread vector and the constructor try/catch. - shutdown() runs under std::call_once: idempotent, and a concurrent caller blocks until the first caller's join has finished. This replaces join_mutex. - shutdown() from the worker's own thread throws std::system_error before entering call_once, instead of running the rest of the queue inline. No task does this; the header documents the rule. This replaces worker_ids and the inline-drain path. - busy and wait_finished() are removed. Renamed ThreadPool -> WorkerThread (OPWorkerThread.*), ThreadPoolStoppedError -> WorkerThreadStoppedError, the thread_pool members -> worker_thread, and the harness/script to scripts/worker_thread_check.cpp / scripts/test-worker-thread.sh. Check (e) now expects shutdown() from the worker to throw while another thread is already shutting down. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes #469.
Problem.
close(),delete()andinvalidate()callwait_finished()but leave the pool's worker running. It is only joined in~ThreadPool, i.e. when the lastshared_ptris released: the JS object is collected, and every prepared statement and pending promise task sharing the pool is gone. A long-lived process that opens and closes databases repeatedly accumulates one idle thread per closed database until GC. We hit an iOS background kill (0x8BADF00D) with 1,006 idledo_workthreads.Changes (
cpp/OPThreadPool.*,cpp/OPDatabase.cpp):ThreadPool::shutdown()setsdoneunderwork_queue_mutex, so a worker between its predicate check and its wait can't miss the wake-up. It swaps the threads out under the lock and joins them. Ashut_downflag makes it idempotent and safe to call concurrently.close,deleteandinvalidatecall it right after their existingwait_finished(), and~ThreadPoolcalls it as well.queue_workon a shut pool throws"[op-sqlite] database is closed". The pool is shared, so a prepared statement used afterclose()would otherwise queue onto a pool with no worker and its promise would never settle. Everyqueue_workcall site is inside a Promise executor host function (promisify,flushPendingReactiveQueries), so the throw becomes a rejection.~ThreadPoolused to run on that worker and join itself:std::system_error("Resource deadlock avoided"), thenstd::terminate. That worker now detaches, anddo_workreturns without touching the freed pool (athread_localflag).Testing. The standalone harness in the repro gist compiles
cpp/OPThreadPool.cppand runs under-fsanitize=threadand-fsanitize=address.main@ 5e756ecstd::terminateWe've been running the equivalent patch on 17.1.2 in our app; it was reviewed there by a second engineer, who mutation-tested it (dropping the self-release guard or detaching instead of joining is caught by ASan/TSan).
Update (after review). The review's ten points are addressed in
deb3846(pool + database: concurrentshutdown()waits for the join, drain on shutdown, no self-join,delete()ordering, onedoneflag, generic pool error mapped to thethrow_if_closedshape, constructor-throw handling) andbaa94ba(prepared statements check the database on every entry point afterclose(), plus the JS test). The harness now lives atscripts/thread_pool_check.cppwithscripts/test-thread-pool.shrunning it under-fsanitize=threadand-fsanitize=address; every check has a watchdog so a hang fails instead of stalling.clang-formathas been run on the changed lines. Details point by point in the comment below.