Skip to content

Do not retry a write that cannot succeed, on a queue with one writer - #145

Merged
ralyodio merged 1 commit into
mainfrom
worktree-no-poison-retries
Aug 19, 2026
Merged

Do not retry a write that cannot succeed, on a queue with one writer#145
ralyodio merged 1 commit into
mainfrom
worktree-no-poison-retries

Conversation

@ralyodio

Copy link
Copy Markdown
Contributor

What happened

Within an hour of the Redis write queue going live, it retried two jobs that had no chance of succeeding:

write-failed  UNIQUE constraint failed: authors.slug   attempts=1
write-failed  UNIQUE constraint failed: authors.slug   attempts=2
write-failed  UNIQUE constraint failed: authors.slug   attempts=3

write-failed  aborted due to timeout  id=30  21:29:27
write-failed  aborted due to timeout  id=30  21:29:58
write-failed  aborted due to timeout  id=30  21:30:29

Worker concurrency is 1 and cannot be raised — SQLite permits one writer. So a retry here is not merely wasted effort: the failing job holds the cluster's only writer for each of its attempts while every other write queues behind it. Job 30 held it for over a minute.

The fix

A constraint or syntax error is deterministic — the same statements against the same data fail identically for ever — so all three attempts were certain to fail, and the second and third bought nothing but blocking.

Those now raise BullMQ's UnrecoverableError, which stops the retries: the caller learns at once and the writer is released.

Transport failures and timeouts still retry. Those can genuinely go differently on the next attempt, and retrying them is most of why the queue was wanted. The classification is isStatementError, already used by the folder to decide when splitting a combined transaction can isolate one bad caller.

Testability

The job body moves into an exported runWriteJob so the retry decision can be tested against a fake client without standing up a broker. The retry decision is the part with judgement in it; the BullMQ plumbing is not.

Full suite: 1,152 tests, 0 failures.

Deliberately not in this PR

Three attempts at a thirty-second timeout still costs the writer ninety seconds. That is a real cost and a defensible policy — shortening it trades durability for throughput — so it wants deciding rather than smuggling in here.

Also still open: the authors.slug race itself. claimAuthorSlug reads the taken slugs and inserts later, so two concurrent crawls can claim the same slug. It predates the queue (it was failing whole feed crawls hours earlier); this change only stops it wasting the writer three times over.

Within an hour of the Redis write queue going live it retried two jobs that had
no chance:

  write-failed  UNIQUE constraint failed: authors.slug   attempts=1,2,3
  write-failed  aborted due to timeout  id=30  21:29:27, 21:29:58, 21:30:29

Worker concurrency is 1 and cannot be raised -- SQLite has one writer -- so a
retry is not merely wasted effort. The failing job holds **the cluster's only
writer** for each of its attempts while every other write queues behind it. Job
30 held it for over a minute.

A constraint or syntax error is deterministic: the same statements against the
same data fail identically for ever, so all three attempts were certain to fail
and the second and third bought nothing but blocking. Those now raise BullMQ's
`UnrecoverableError`, which stops the retries -- the caller learns at once and
the writer is released.

Transport failures and timeouts still retry. Those genuinely can go differently
on the next attempt, and retrying them is most of why the queue was wanted.
The classification is `isStatementError`, already used by the folder to decide
when splitting a combined transaction can isolate one bad caller.

The job body moves into an exported `runWriteJob` so the retry decision can be
tested against a fake client, without a broker. The BullMQ plumbing is not the
part with judgement in it.

Not addressed here: three attempts at a thirty-second timeout still costs the
writer ninety seconds. That is a real cost and a defensible policy -- shortening
it trades durability for throughput -- so it wants deciding rather than
smuggling into this change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ralyodio
ralyodio merged commit 3223fa7 into main Aug 19, 2026
3 checks passed
@ralyodio
ralyodio deleted the worktree-no-poison-retries branch August 19, 2026 21:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant