Flaky Rust test: notification_poll_failures_do_not_stop_the_client hits database is locked #1455
Description
@brandur notification_poll_failures_do_not_stop_the_client in rust/riverqueue/tests/resilience_sqlite.rs (added with the Rust port in #1442) fails intermittently in CI with SQLite's database is locked:
thread 'notification_poll_failures_do_not_stop_the_client' panicked at riverqueue/tests/resilience_sqlite.rs:208:10:
called `Result::unwrap()` on an `Err` value: Database(SqliteError { code: 5, message: "database is locked" })
Line 208 is the unwrap in the job_state helper, which the test calls from wait_until while waiting for the job to complete.
It's shown up in about 6 of the ~12 Rust workflow runs since #1442 merged, 8 jobs in all, always with the same panic. It doesn't follow a particular PostgreSQL or Rust version:
- Generate port conformance fixtures from River's Go implementation #1451,
postgres (15): https://github.com/riverqueue/river/actions/runs/37464710415/job/112272738126 - Add a conformance harness for River's Go, Rust, and JavaScript clients #1435,
postgres (17): https://github.com/riverqueue/river/actions/runs/37464713159/job/112272746416 - Add a conformance harness for River's Go, Rust, and JavaScript clients #1435,
postgres (14)andrust_versions (1.96)in run https://github.com/riverqueue/river/actions/runs/37464710877 - Dependabot runs 37380202575 (
rust_versions (1.97)), 37376861834 (postgres (18)), and 37376860109 (postgres (15),rust_versions (1.97))
The likely cause is the test's own setup. It opens its pool in rollback-journal (Delete) mode with a 20 ms busy timeout, so that the outbox poll fails quickly while a second connection holds BEGIN EXCLUSIVE. The client and the test's job_state reads share that pool. After the exclusive lock is released, the client keeps writing (fetch polls every 20 ms with a 1 ms cooldown, plus the insert and completion), and in rollback-journal mode a committing writer blocks readers. On a loaded runner a commit can take longer than 20 ms, so job_state gets SQLITE_BUSY and its unwrap panics. The client.insert just before the wait uses the same pool and has the same exposure.
Two possible fixes: read job state for the assertions through a separate pool with a normal busy timeout, or have the wait_until condition treat a busy error as "not yet" and keep polling.
Activity
brandur commentedon Oct 6, 2026
Didn't have a super strong opinion on the fix, but Codex seem to have a slight preference for the first option (and I suppose that is one that would have less contention), so went with that. #1457
Fix intermittent Rust test case (riverqueue#1457)