Repository navigation
19.0 rdm concurrency retry cap - #4
Open
richard-willdooit wants to merge 3 commits into
Open
richard-willdooit wants to merge 3 commits into
richard-willdooit wants to merge 3 commits into
Conversation
The typical transaction serialization errors are retried, but the conversion from OperationalError to RetryableJobError happens in the runners, outside Job.perform() - and perform() is the only place max_retries is consulted. retry_postpone() simply writes a new eta and requeues, so a job that keeps colliding is retried for ever while its retry counter climbs past the limit and it never reaches the failed state. Seen in production on a long report-building job: it sat at try 10 of a maximum of 5, each attempt running for some eight minutes, until it was closed by hand. Convert the error inside Job.perform() instead, so it falls into the RetryableJobError handling that is already there and is subject to max_retries like any other retryable error. The duplicated handling in queue_job's controller and in queue_job_cron_jobrunner then comes out, and PG_RETRY moves beside DEFAULT_MAX_RETRIES. _enqueue_dependent_jobs keeps its own OperationalError handling: that is a separate retry loop around the dependency update, not the job itself. Tests: a serialization failure is retried and gives up at max_retries, and any other OperationalError is not retried at all.
…y delay Follow-up to the review on OCA#1009. A job that keeps hitting serialization failures was waiting 5 seconds before each retry, ignoring its specific retry_pattern. Now, we pass no delay at all when the job function declares a retry_pattern, and the pattern will apply as it does for any other RetryableJobError. Jobs without a pattern are still passed PG_RETRY and still wait 5 seconds. The constant cannot simply be dropped for them: _get_retry_seconds() would then fall back to RETRY_INTERVAL - ten minutes - for a conflict that usually clears at once. A job function that declares a retry_pattern will change behaviour: a serialization failure now waits what the pattern says. That is the point of it; the job has said how it wants to be retried. The surprise is a pattern that does not cover the early tries - {3: 180} waits RETRY_INTERVAL on tries 1 and 2, so ten minutes where it used to be five seconds. PG_RETRY is also re-exported from queue_job.controllers.main, so that imports from its old home keep working. queue_job_cron_jobrunner had a hardcoded seconds=5, discarding the delay the error asks for, so a retry_pattern would be ignored under that runner; it now passes err.seconds like the controller does. Tests: with no retry_pattern a serialization failure still waits PG_RETRY, and with one it waits what the pattern says.
perform() converts the retryable PostgreSQL concurrency errors into RetryableJobError, which is what subjects them to max_retries. But a job function rarely raises one itself: it writes, and PostgreSQL reports the conflict when that write reaches the database. The runners call perform() inside a savepoint, and leaving the savepoint is what flushes, so the failure was raised after perform() had returned. Flushing at the end of perform()'s try puts the failure back where it can be converted. Both new tests error with a bare OperationalError without this change, in the frames the report showed: the savepoint's __exit__ flushing on the way out of _try_perform_job.
richard-willdooit
force-pushed
the
19.0
branch
from
November 20, 2026 15:19
e3a6efc to
ed5c640
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.