Skip to content

Join the connection's worker thread on close instead of at GC - #470

Merged
ospfranco merged 3 commits into
OP-Engineering:mainfrom
Glenn444:fix/close-joins-thread-pool
Oct 26, 2026
Merged

ospfranco merged 3 commits into
OP-Engineering:mainfrom
Glenn444:fix/close-joins-thread-pool

Conversation

@Glenn444

@Glenn444 Glenn444 commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #469.

Problem. close(), delete() and invalidate() call wait_finished() but leave the pool's worker running. It is only joined in ~ThreadPool, i.e. when the last shared_ptr is released: the JS object is collected, and every prepared statement and pending promise task sharing the pool is gone. A long-lived process that opens and closes databases repeatedly accumulates one idle thread per closed database until GC. We hit an iOS background kill (0x8BADF00D) with 1,006 idle do_work threads.

Changes (cpp/OPThreadPool.*, cpp/OPDatabase.cpp):

  • ThreadPool::shutdown() sets done under work_queue_mutex, so a worker between its predicate check and its wait can't miss the wake-up. It swaps the threads out under the lock and joins them. A shut_down flag makes it idempotent and safe to call concurrently. close, delete and invalidate call it right after their existing wait_finished(), and ~ThreadPool calls it as well.
  • queue_work on a shut pool throws "[op-sqlite] database is closed". The pool is shared, so a prepared statement used after close() would otherwise queue onto a pool with no worker and its promise would never settle. Every queue_work call site is inside a Promise executor host function (promisify, flushPendingReactiveQueries), so the throw becomes a rejection.
  • No self-join. If releasing a task destroys its own pool, which happens when the task held the last reference, ~ThreadPool used to run on that worker and join itself: std::system_error ("Resource deadlock avoided"), then std::terminate. That worker now detaches, and do_work returns without touching the freed pool (a thread_local flag).

Testing. The standalone harness in the repro gist compiles cpp/OPThreadPool.cpp and runs under -fsanitize=thread and -fsanitize=address.

300 closed pools, still referenced queue after close last ref released on the worker
main @ 5e756ec 300 live threads queued, never runs std::terminate
this PR 0–1 extra threads throws (promise rejects) clean

We've been running the equivalent patch on 17.1.2 in our app; it was reviewed there by a second engineer, who mutation-tested it (dropping the self-release guard or detaching instead of joining is caught by ASan/TSan).

Update (after review). The review's ten points are addressed in deb3846 (pool + database: concurrent shutdown() waits for the join, drain on shutdown, no self-join, delete() ordering, one done flag, generic pool error mapped to the throw_if_closed shape, constructor-throw handling) and baa94ba (prepared statements check the database on every entry point after close(), plus the JS test). The harness now lives at scripts/thread_pool_check.cpp with scripts/test-thread-pool.sh running it under -fsanitize=thread and -fsanitize=address; every check has a watchdog so a hang fails instead of stalling. clang-format has been run on the changed lines. Details point by point in the comment below.

close(), delete() and invalidate() drained the ThreadPool but left its
worker running; it was only joined in ~ThreadPool, when the last
shared_ptr (JS object, prepared statements, pending promise tasks) was
released. Apps that open/close repeatedly in a long-lived process kept
one idle thread per closed database until GC (an iOS background kill had
1,006 idle do_work threads).

- ThreadPool::shutdown(): sets done under the mutex, wakes and joins the
  worker; idempotent and safe to call concurrently. close/delete/
  invalidate call it after wait_finished(); ~ThreadPool calls it too.
- queue_work on a shut pool throws (callers queue from a Promise
  executor, so the promise rejects instead of never settling).
- A task that drops the last reference to its own pool no longer
  self-joins (std::system_error -> terminate): the worker detaches and
  do_work returns without touching the freed pool.

Fixes OP-Engineering#469

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ospfranco

Copy link
Copy Markdown
Contributor

Thanks for this! I've gone through the changes. Can you take a look at the following

Bugs

1. The second shutdown() caller can free memory the worker is still using

if (shut_down) {

If a second caller (e.g. ~ThreadPool) sees shut_down == true, it returns straight away while the first caller may still be in notify_all()/join(). The destructor then frees the mutex, condvar and queue while the worker is still waiting on them. The header says shutdown() is "safe to call concurrently", but it isn't yet. Later callers need to wait until the join has finished.

2. The self-release flag is set in shutdown() rather than in ~ThreadPool

thread.detach();

Whenever shutdown() runs on the worker thread, do_work() returns early without decrementing busy, even if the pool isn't being destroyed (e.g. an OPDatabase whose invalidate() or destructor runs on the worker). After that, every wait_finished() waits forever and the JS thread freezes. Only the destructor knows this is being freed, so the flag should be set there.

3. delete() stops the worker before checking whether delete is supported

thread_pool->shutdown();

On a remote-only libsql/Turso connection, delete() throws "not supported", but the pool has already been shut down. The database stays open, yet every async prepared-statement call now rejects with "database is closed". Please move the remote-only check above invalidated, interrupt(), wait_finished() and shutdown().

4. Tasks still in the queue at shutdown are dropped and their promises never settle

// Only wake up if there are elements in the queue or the

do_work() exits as soon as done is set, even with work left in the queue. Any shutdown() not preceded by wait_finished() (including the one in the destructor) leaves JS awaits hanging, and a later wait_finished() on that pool blocks forever.

5. Prepared statements behave inconsistently after close()

if (name == "bindSync") {

stmt.execute() now rejects, but executeSync()/bindSync() still run against the closed handle, which for libsql has already been freed. A closed check in PreparedStatementHostObject (or OPDatabase) would cover both paths, instead of relying on the pool's state.

Cleanups

6. The self-join guard can't be reached today

// Releasing the task may have destroyed this pool (it held the last

No queued task holds a shared_ptr<ThreadPool> (promisify and the reactive-flush task don't capture it), so the thread_local flag/detach branch never runs. I'd drop it and document a rule instead: tasks must not own the pool.

7. shut_down duplicates done

bool shut_down{false};

Both are set together under the same lock, so if (done) return; does the same job with one fewer flag to keep in sync.

8. wait_finished(); shutdown(); is repeated in close, delete and invalidate

thread_pool->shutdown();

Moving the drain into shutdown() itself (or a single drain_and_shutdown()) would also fix #4.

9. The pool's error message doesn't match the other "database is closed" errors

throw std::runtime_error("[op-sqlite] database is closed");

queue_work throws [op-sqlite] database is closed, which differs from throw_if_closed's [op-sqlite][execute] ... format and has no code field. JS code that matches on the message or reads error.code will get a different shape. I'd have the pool throw its own generic error and let callers map it to the existing one.

10. The destructor comment is wrong, and there's no test for the new path

// main-thread/runtime generation) can exit gracefully. This will be executed if

Destructors don't run when a constructor throws. If creating a later thread fails, the threads already started are never joined and the program calls std::terminate. Also, could you add a test in example/src/tests for stmt.execute() after db.close()?

#2 and #4 are the most likely to show up in practice as hangs.

Glenn444 and others added 2 commits October 24, 2026 21:10
…join

Addresses the review of OP-Engineering#470 (points 1-4 and 6-10).

shutdown() is now the single way to stop a pool. It refuses new work,
lets the worker run every task that is already queued (so no promise is
left unsettled and a later wait_finished() cannot block), then joins.
close(), delete() and invalidate() call just that instead of
wait_finished() followed by shutdown().

- Concurrent callers: the join runs under join_mutex, so a second
  shutdown() (or the destructor after it) returns only once the first
  caller's join has completed, never while the worker still uses the
  mutex, condition variable or queue.
- Called on the worker itself (e.g. a database invalidated from a task),
  shutdown() does not join: it runs the remaining queued tasks inline and
  returns. do_work() then releases `busy` as usual and exits, and a later
  shutdown() or the destructor joins it. The thread_local self-release
  flag and the detach branch are gone; the header documents the rule
  that makes them unnecessary: a task must never own its pool (none
  does: tasks capture the database or statement by raw pointer).
- `shut_down` is dropped; `done` is the only flag and is now read under
  the mutex only.
- queue_work() on a stopped pool throws ThreadPoolStoppedError
  ("[op-sqlite] thread pool is stopped"). promisify and
  flushPendingReactiveQueries map it to the same
  "[op-sqlite][<fn>] database is closed" error as throw_if_closed, thrown
  from the Promise executor so the promise rejects. promisify takes the
  function name for that.
- delete() checks the remote-only case before invalidating, interrupting
  or stopping the pool, so an unsupported delete() leaves the database
  open and usable.
- The constructor joins any thread it already started if a later one
  fails to start: the destructor does not run when a constructor throws.

scripts/thread_pool_check.cpp (run by scripts/test-thread-pool.sh, under
TSan and ASan) covers 300 closed-but-referenced pools, queue_work after
shutdown, draining queued tasks on shutdown() and in the destructor, two
concurrent shutdowns followed by the destructor, shutdown() from a task
on the worker, and a constructor whose thread cannot start.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Addresses point 5 of the review of OP-Engineering#470 (and the test asked for in 10).

A statement keeps the raw connection handle it was prepared on. After
close() the async path rejected only because the pool was stopped, while
executeSync() and bindSync() still ran against the closed handle, which
for libsql is already freed.

PreparedStatementHostObject now holds a weak_ptr to its OPDatabase and
checks it at the top of bind, bindSync, execute and executeSync, before
anything touches the handle. A closed (or already destroyed) database
throws the same "[op-sqlite][<fn>] database is closed" error as
OPDatabase::throw_if_closed, with no `code`, whatever the pool's state.

The JS wrapper for stmt.execute is now async, as bind already is, so the
native throw reaches the caller as a rejection, the same way db.execute
reports a closed database. executeSync/bindSync throw synchronously.

example/src/tests/preparedStatements.ts covers both paths.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Glenn444

Copy link
Copy Markdown
Contributor Author

Thanks for the careful review. You were right on all ten points. I pushed two commits: deb3846 covers the pool and the database (1–4, 6–10), and baa94ba covers prepared statements (5) plus the JS test. Point by point:

1. Concurrent shutdown(). The join now runs under a dedicated join_mutex. A second caller, including ~ThreadPool after an explicit shutdown(), blocks until the first caller's join has finished, so it can't return while the worker is still using the mutex, condvar or queue. Test (d) runs two concurrent shutdown()s while the worker is inside a 30 ms task, then deletes the pool straight after the second one returns (20 rounds). Each caller must see the task finished when it returns. If I put back the early return, the test fails: "the second shutdown() returned before the join completed", and then the worker aborts on the freed mutex, under both ASan and TSan.

2. shutdown() on the worker. The thread_local flag is gone. shutdown() checks whether it's running on one of the pool's own threads (ids recorded in the constructor). If it is, it doesn't join, since a thread can't join itself. It runs the remaining queued tasks inline and returns, so as on any other thread, nothing else runs after it returns. do_work() then decrements busy as usual and exits, and a later shutdown() or the destructor joins it. Test (e) calls shutdown() from a task with 10 tasks queued behind it. It checks that all 10 have run by the time shutdown() returns, that wait_finished() returns (twice), that queue_work throws, and that a later shutdown() plus the destructor are clean. Putting back the 53f027b behaviour (flag set in shutdown(), do_work returns without --busy) makes wait_finished() hang, and the watchdog reports it. Joining unconditionally gives std::system_error: Resource deadlock avoided, then std::terminate.

3. delete() ordering. The remote-only check now comes first, after the argument check and before invalidated, interrupt() and shutdown(). An unsupported delete() leaves the database open and usable. This is native-only, and there is no remote setup in the example tests, so it's covered by inspection only.

4. Queued tasks at shutdown. do_work() exits only once done is set and the queue is empty, so every queued task runs and every promise settles. Test (c) parks the worker, queues 50 more tasks and calls shutdown() with no wait_finished() first: 51/51 run. It does the same for the destructor-only path. Putting back if (done) break; drops them: 1/51 run, and the next wait hangs.

5. Prepared statements after close(). PreparedStatementHostObject now holds a weak_ptr<OPDatabase>. bind, bindSync, execute and executeSync check it before touching the handle, and a closed or destroyed database throws the same error as throw_if_closed. This no longer depends on the pool's state. I also made the JS stmt.execute wrapper async, like bind already is, so the error arrives as a rejection the same way db.execute reports a closed database. One side effect: the existing "statement has been freed" throw from execute also becomes a rejection.

6. Self-join guard. Removed. I checked first: no queued task owns the pool. promisify's executor captures thread_pool, but the task it queues doesn't, and the task's lambdas capture the database or statement by raw pointer. The header now states the rule: tasks must not own the pool.

7. shut_down. Removed. done is the only flag, and it's now read only under the mutex, so it no longer needs to be atomic.

8. Repeated wait_finished(); shutdown();. shutdown() drains and then joins. close, delete and invalidate call only shutdown().

9. Error shape. The pool throws its own ThreadPoolStoppedError ("[op-sqlite] thread pool is stopped"). promisify (which now takes the function name) and flushPendingReactiveQueries catch it and throw "[op-sqlite][<fn>] database is closed" from the Promise executor. That's the same std::runtime_error → host-function conversion throw_if_closed goes through, so JS gets the same message and no code. Both throw_if_closed and the mapping build the message with one database_closed_message() helper. In practice the mapping is a backstop, because every entry point checks the database first.

10. Constructor and test. You're right that the destructor doesn't run if the constructor throws. The constructor now catches, stops and joins any thread it already started, and rethrows. I fixed the destructor comment too. Test (f) exhausts the process's threads (macOS caps it at ~4096) and checks that the constructor throws std::system_error without terminating. With one thread per pool, that only exercises the first thread failing. I also checked the "later thread fails" path separately, with a local build of the pool at number_of_threads = 2 and exactly one free thread slot: it throws cleanly. Without the catch it calls std::terminate. That variant isn't committed, because it needs a source edit. For JS, example/src/tests/preparedStatements.ts has two new tests. stmt.execute() after db.close() must return a promise that rejects with "[op-sqlite][execute] database is closed" (not a synchronous throw) and no code. executeSync()/bindSync() after db.close() must throw the same error.

Testing. scripts/thread_pool_check.cpp extends the harness from the gist and lives outside cpp/, because the podspec and Package.swift glob cpp/**/*.cpp. scripts/test-thread-pool.sh builds it against cpp/OPThreadPool.cpp and runs it under -fsanitize=thread and -fsanitize=address. Every check runs under a watchdog, so a hang counts as a failure. Locally (macOS, Apple clang) it's all green, five runs per sanitizer, with no sanitizer reports. Check (a) is still there: 300 closed pools that are still referenced leave 0 extra threads. I ran clang-format (the repo's .clang-format) on the files that were clean upstream, and only on the changed lines of OPDatabase.cpp/OPUtils.*, so unrelated lines don't churn.

Not verified. I haven't run the new JS tests: I have no example-app build on this machine, so they mirror the structure of the existing ones. The CI run for this push is waiting for your approval to start; once it runs, the iOS and Android jobs will exercise them. The JSI translation units (OPDatabase.cpp, OPUtils.cpp, OPPreparedStatementHostObject.cpp) pass -fsyntax-only for the default, libsql and Turso backends against RN JSI headers, but I didn't build or link them, and I haven't tested on a device. Points 3 and 5 have no native test, because the harness has no JSI runtime.

@ospfranco
ospfranco merged commit baa94ba into OP-Engineering:main Oct 26, 2026
10 checks passed
github-actions Bot pushed a commit to CasaZurigo/op-sqlite that referenced this pull request Nov 2, 2026
The pool only ever runs one thread per connection, and after OP-Engineering#470
nothing calls wait_finished() any more: shutdown() drains the queue and
joins. That makes most of the pool's machinery unnecessary.

- One std::thread, declared last, so a failed start needs no cleanup.
  This replaces the thread vector and the constructor try/catch.
- shutdown() runs under std::call_once: idempotent, and a concurrent
  caller blocks until the first caller's join has finished. This
  replaces join_mutex.
- shutdown() from the worker's own thread throws std::system_error
  before entering call_once, instead of running the rest of the queue
  inline. No task does this; the header documents the rule. This
  replaces worker_ids and the inline-drain path.
- busy and wait_finished() are removed.

Renamed ThreadPool -> WorkerThread (OPWorkerThread.*),
ThreadPoolStoppedError -> WorkerThreadStoppedError, the thread_pool
members -> worker_thread, and the harness/script to
scripts/worker_thread_check.cpp / scripts/test-worker-thread.sh.
Check (e) now expects shutdown() from the worker to throw while
another thread is already shutting down.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: close() leaves the connection's worker thread running until GC (iOS background kill with 1,006 idle threads)

2 participants