Do not retry a write that cannot succeed, on a queue with one writer - #145
Merged
Conversation
Within an hour of the Redis write queue going live it retried two jobs that had no chance: write-failed UNIQUE constraint failed: authors.slug attempts=1,2,3 write-failed aborted due to timeout id=30 21:29:27, 21:29:58, 21:30:29 Worker concurrency is 1 and cannot be raised -- SQLite has one writer -- so a retry is not merely wasted effort. The failing job holds **the cluster's only writer** for each of its attempts while every other write queues behind it. Job 30 held it for over a minute. A constraint or syntax error is deterministic: the same statements against the same data fail identically for ever, so all three attempts were certain to fail and the second and third bought nothing but blocking. Those now raise BullMQ's `UnrecoverableError`, which stops the retries -- the caller learns at once and the writer is released. Transport failures and timeouts still retry. Those genuinely can go differently on the next attempt, and retrying them is most of why the queue was wanted. The classification is `isStatementError`, already used by the folder to decide when splitting a combined transaction can isolate one bad caller. The job body moves into an exported `runWriteJob` so the retry decision can be tested against a fake client, without a broker. The BullMQ plumbing is not the part with judgement in it. Not addressed here: three attempts at a thirty-second timeout still costs the writer ninety seconds. That is a real cost and a defensible policy -- shortening it trades durability for throughput -- so it wants deciding rather than smuggling into this change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happened
Within an hour of the Redis write queue going live, it retried two jobs that had no chance of succeeding:
Worker concurrency is 1 and cannot be raised — SQLite permits one writer. So a retry here is not merely wasted effort: the failing job holds the cluster's only writer for each of its attempts while every other write queues behind it. Job 30 held it for over a minute.
The fix
A constraint or syntax error is deterministic — the same statements against the same data fail identically for ever — so all three attempts were certain to fail, and the second and third bought nothing but blocking.
Those now raise BullMQ's
UnrecoverableError, which stops the retries: the caller learns at once and the writer is released.Transport failures and timeouts still retry. Those can genuinely go differently on the next attempt, and retrying them is most of why the queue was wanted. The classification is
isStatementError, already used by the folder to decide when splitting a combined transaction can isolate one bad caller.Testability
The job body moves into an exported
runWriteJobso the retry decision can be tested against a fake client without standing up a broker. The retry decision is the part with judgement in it; the BullMQ plumbing is not.Full suite: 1,152 tests, 0 failures.
Deliberately not in this PR
Three attempts at a thirty-second timeout still costs the writer ninety seconds. That is a real cost and a defensible policy — shortening it trades durability for throughput — so it wants deciding rather than smuggling in here.
Also still open: the
authors.slugrace itself.claimAuthorSlugreads the taken slugs and inserts later, so two concurrent crawls can claim the same slug. It predates the queue (it was failing whole feed crawls hours earlier); this change only stops it wasting the writer three times over.