Skip to content

Let jobs acquire their concurrency lock when dispatched - #1

Open
ceritium wants to merge 1 commit into
mainfrom
defer-concurrency-lock-acquisition
Open

ceritium wants to merge 1 commit into
mainfrom
defer-concurrency-lock-acquisition

Conversation

@ceritium

@ceritium ceritium commented Oct 4, 2026 •

Copy link
Copy Markdown
Owner

When Solid Queue shares the app's database (no connects_to) and the enqueue isn't deferred to after commit, enqueueing a due job with limits_concurrency acquires its semaphore (SELECT ... FOR UPDATE or insert) inside the caller's transaction. The row lock lasts until that transaction ends, so:

  1. Another enqueue with the same key waits for it.
  2. A worker finishing a job with that key waits to signal the semaphore.
  3. Two transactions enqueueing two keys in opposite order deadlock, raised as SolidQueue::Job::EnqueueError and aborting the caller's transaction.

Reproduction against 1.7.0 (Rails 8.1.4, PostgreSQL 17), single file: https://gist.github.com/ceritium/8d733493fef1c18f8f143fb8a2144c60
With transactions held 3 s, (1) and (2) wait ~2.5 s and (3) deadlocks. FORK=1 runs this branch, where none of them happens.

The change

A job enqueued with a wait: that isn't due yet already avoids this: it's inserted as scheduled and the dispatcher acquires the lock later, in its own short transaction. This makes that path available without a delay, through an option:

  • :on_enqueue (default): current behaviour.
  • :on_dispatch: insert the job as scheduled and already due; the dispatcher acquires the lock on its next poll.
  • :outside_transaction: :on_dispatch when a transaction is open on Solid Queue's connection, :on_enqueue otherwise.
config.solid_queue.concurrency_lock_acquisition = :outside_transaction
limits_concurrency key: ->(record) { record }, acquire: :on_dispatch
MyJob.set(acquire: :on_enqueue).perform_later(record)

The job row is still written in the caller's transaction; only the semaphore moves out. Trade-offs, documented in the README: up to polling_interval of extra latency, a dispatcher is required, and on_conflict: :discard happens at dispatch (as for scheduled jobs).

Related: rails#812 fixes a deadlock between dispatchers on that same path.

Tests: model tests for each mode, precedence and bulk enqueue, and an integration test per effect above showing it with :on_enqueue and not with :outside_transaction (skipped on SQLite). Full suite green on PostgreSQL; MySQL not run.

@ceritium
ceritium force-pushed the defer-concurrency-lock-acquisition branch 2 times, most recently from f0abcb8 to a1e9177 Compare October 4, 2026 13:06
When Solid Queue shares the app's connection and a concurrency-limited
job is enqueued inside a transaction, acquiring the semaphore happens
inside that transaction: Semaphore#wait selects the row FOR UPDATE, or
inserts it, and the lock lasts until the caller's transaction ends.
Until then, any other enqueue with the same key waits, and so does the
worker signalling the semaphore after running a job with that key. Two
transactions that enqueue jobs for two keys in opposite order deadlock,
which surfaces as SolidQueue::Job::EnqueueError and aborts the caller's
transaction.

A job enqueued with a wait that isn't due yet when it's inserted doesn't
have the problem: it's scheduled, and the dispatcher acquires its
semaphore later in its own transaction. This makes that available on
purpose, without a delay. A new option says when a job acquires its
concurrency lock:

- :on_enqueue, the default and the current behaviour.
- :on_dispatch, enqueuing the job as a scheduled job that is already
  due, for the dispatcher to acquire the lock once it's committed.
- :outside_transaction, :on_dispatch when there's an open transaction on
  Solid Queue's connection, :on_enqueue otherwise.

It can be set globally (config.solid_queue.concurrency_lock_acquisition),
per job (limits_concurrency acquire:) or per enqueue (set(acquire:)),
the most specific one winning. The job row is still inserted in the
caller's transaction, so it's still committed or rolled back with it.

Whether there's an open transaction is checked before the job's own
insert, which opens one, and only joinable transactions count, so the
test fixtures' transaction doesn't.

Jobs without concurrency controls have no lock to defer, so they skip
the transaction check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lceyt66zciMB7d4Yv7UgDd
@ceritium
ceritium force-pushed the defer-concurrency-lock-acquisition branch from e0e0c9c to 97ef544 Compare October 15, 2026 21:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant