Skip to content

19.0 rdm concurrency retry cap - #4

Open
richard-willdooit wants to merge 3 commits into
19.0from
19.0-rdm-concurrency_retry_cap
Open

richard-willdooit wants to merge 3 commits into
19.0from
19.0-rdm-concurrency_retry_cap

Conversation

@richard-willdooit

Copy link
Copy Markdown

No description provided.

The typical transaction serialization errors are retried, but the conversion
from OperationalError to RetryableJobError happens in the runners, outside
Job.perform() - and perform() is the only place max_retries is consulted.
retry_postpone() simply writes a new eta and requeues, so a job that keeps
colliding is retried for ever while its retry counter climbs past the limit
and it never reaches the failed state.

Seen in production on a long report-building job: it sat at try 10 of a
maximum of 5, each attempt running for some eight minutes, until it was
closed by hand.

Convert the error inside Job.perform() instead, so it falls into the
RetryableJobError handling that is already there and is subject to
max_retries like any other retryable error. The duplicated handling in
queue_job's controller and in queue_job_cron_jobrunner then comes out, and
PG_RETRY moves beside DEFAULT_MAX_RETRIES.

_enqueue_dependent_jobs keeps its own OperationalError handling: that is a
separate retry loop around the dependency update, not the job itself.

Tests: a serialization failure is retried and gives up at max_retries, and
any other OperationalError is not retried at all.
…y delay

Follow-up to the review on OCA#1009.

A job that keeps hitting serialization failures was waiting 5 seconds before
each retry, ignoring its specific retry_pattern.

Now, we pass no delay at all when the job function declares a retry_pattern,
and the pattern will apply as it does for any other RetryableJobError. Jobs
without a pattern are still passed PG_RETRY and still wait 5 seconds.

The constant cannot simply be dropped for them: _get_retry_seconds() would then
fall back to RETRY_INTERVAL - ten minutes - for a conflict that usually clears
at once.

A job function that declares a retry_pattern will change behaviour: a
serialization failure now waits what the pattern says. That is the point of it;
the job has said how it wants to be retried. The surprise is a pattern that
does not cover the early tries - {3: 180} waits RETRY_INTERVAL on tries 1 and
2, so ten minutes where it used to be five seconds.

PG_RETRY is also re-exported from queue_job.controllers.main, so that imports
from its old home keep working.

queue_job_cron_jobrunner had a hardcoded seconds=5, discarding the delay the
error asks for, so a retry_pattern would be ignored under that runner; it now
passes err.seconds like the controller does.

Tests: with no retry_pattern a serialization failure still waits PG_RETRY, and
with one it waits what the pattern says.
perform() converts the retryable PostgreSQL concurrency errors into
RetryableJobError, which is what subjects them to max_retries. But a job
function rarely raises one itself: it writes, and PostgreSQL reports the
conflict when that write reaches the database.

The runners call perform() inside a savepoint, and leaving the savepoint
is what flushes, so the failure was raised after perform() had returned.

Flushing at the end of perform()'s try puts the failure back where it can
be converted.

Both new tests error with a bare OperationalError without this change, in
the frames the report showed: the savepoint's __exit__ flushing on the way
out of _try_perform_job.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant