Skip to content

[FIX] queue_job: apply max_retries to PostgreSQL concurrency errors - #1009

Open
richard-willdooit wants to merge 3 commits into
OCA:19.0from
WilldooIT:19.0-rdm-concurrency_retry_cap
Open

richard-willdooit wants to merge 3 commits into
OCA:19.0from
WilldooIT:19.0-rdm-concurrency_retry_cap

Conversation

@richard-willdooit

Copy link
Copy Markdown

The problem

The typical PostgreSQL transaction serialization errors are retried, but the conversion from OperationalError to RetryableJobError happens in the runners, outside Job.perform() — and perform() is the only place max_retries is consulted.

  • queue_job/controllers/main.py wraps _try_perform_job and raises RetryableJobError from there; the surrounding except RetryableJobError calls retry_postpone(), which writes a new eta and requeues.
  • queue_job_cron_jobrunner/models/queue_job.py does the same inline, calling postpone() + set_pending(reset_retry=False) directly.

Neither consults max_retries. perform() has already incremented retry by the time the error surfaces, so a job that keeps colliding is retried for ever while its counter climbs past the limit, and it never reaches failed.

Seen in production on a long report-building job: it sat at try 10 of a maximum of 5, each attempt running for about eight minutes, until it was closed by hand.

The change

Convert the error inside Job.perform() instead, so it falls into the RetryableJobError handling already there and is subject to max_retries like any other retryable error:

self.retry += 1
try:
    try:
        ...
        self.result = self.func(*tuple(self.args), **self.kwargs)
    except OperationalError as err:
        if err.pgcode not in PG_CONCURRENCY_ERRORS_TO_RETRY:
            raise
        raise RetryableJobError(
            ustr(err.pgerror, errors="replace"), seconds=PG_RETRY
        ) from err
except RetryableJobError as err:
    ...  # unchanged: ignore_retry, infinite retries, max_retries

The duplicated handling then comes out of both runners, and PG_RETRY moves beside DEFAULT_MAX_RETRIES in job.py. Net effect is a deletion: one conversion in one place, and both runners behave identically.

_enqueue_dependent_jobs keeps its own OperationalError handling — that is a separate retry loop around the dependency update, not around the job.

Behaviour

before after
serialization failure, under max_retries postponed postponed, unchanged
serialization failure, at max_retries postponed for ever FailedJobError, job fails
any other OperationalError fails the job fails the job, unchanged
max_retries = 0 infinite infinite, unchanged

Tests

test_queue_job gains a raise_pg_concurrency option on testing_method, backed by a small OperationalError subclass — psycopg2 exposes pgcode read-only and errors.lookup() leaves it unset, so the code has to be carried that way. Two cases in test_job.py, mirroring the existing test_retryable_error:

  • a serialization failure retries and gives up at max_retries
  • any other OperationalError is not retried

Full suite on 19.0: 186 tests, 0 failed, 0 errors (queue_job, test_queue_job, queue_job_cron_jobrunner).

@oca-cla-bot

oca-cla-bot Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

@OCA-git-bot

Copy link
Copy Markdown
Contributor

Hi @guewen, @sbidoul,
some modules you are maintaining are being modified, check this out!

The typical transaction serialization errors are retried, but the conversion
from OperationalError to RetryableJobError happens in the runners, outside
Job.perform() - and perform() is the only place max_retries is consulted.
retry_postpone() simply writes a new eta and requeues, so a job that keeps
colliding is retried for ever while its retry counter climbs past the limit
and it never reaches the failed state.

Seen in production on a long report-building job: it sat at try 10 of a
maximum of 5, each attempt running for some eight minutes, until it was
closed by hand.

Convert the error inside Job.perform() instead, so it falls into the
RetryableJobError handling that is already there and is subject to
max_retries like any other retryable error. The duplicated handling in
queue_job's controller and in queue_job_cron_jobrunner then comes out, and
PG_RETRY moves beside DEFAULT_MAX_RETRIES.

_enqueue_dependent_jobs keeps its own OperationalError handling: that is a
separate retry loop around the dependency update, not the job itself.

Tests: a serialization failure is retried and gives up at max_retries, and
any other OperationalError is not retried at all.

@simahawk simahawk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM


_logger = logging.getLogger(__name__)

PG_RETRY = 5 # seconds

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe worth importing it from where it stands now for backward compat.
Not a blocker from my POV.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah maybe worth in case someone customized the default of 5 seconds.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@guewen

Happy to add the re-export, so imports from controllers.main keep working. But it only keeps the import working. If someone changed the value at the old location, job.py won't see it - the delay would quietly go back to 5 seconds.

I'd rather offer a supported way to do it, which ties into your other comment: use the job's retry_pattern when it has one, and fall back to PG_RETRY otherwise.

@simahawk

Copy link
Copy Markdown
Contributor

@guewen ping

@OCA-git-bot

Copy link
Copy Markdown
Contributor

This PR has the approved label and has been created more than 5 days ago. It should therefore be ready to merge by a maintainer (or a PSC member if the concerned addon has no declared maintainer). 🤖

Comment thread queue_job/job.py Outdated
# limit, so it retries for ever.
if err.pgcode not in PG_CONCURRENCY_ERRORS_TO_RETRY:
raise
raise RetryableJobError(err.pgerror, seconds=PG_RETRY) from err

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should keep this forced PG_RETRY or just use the same behavior as other retries.
Originally, the serialization error were not checking the max retries on purpose, because I considered they had to succeed at some point. But then, 5 seconds of wait can be long and there is no override mechanism for this, but if we do remove seconds=PG_RETRY, we get the default retry which is also 5 seconds, but allow to be customized in the job function.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @guewen

Removing it wouldn't give 5 seconds: with no retry_pattern on the job function, _get_retry_seconds falls back to RETRY_INTERVAL, which is 10 minutes, so a serialization conflict that clears almost instantly would wait 10 minutes. Jobs that do define a pattern usually tune it for slower failures.

I'd suggest keeping PG_RETRY as the default and letting a job's retry_pattern override it when one is set (seconds=None if self.job_config.retry_pattern else PG_RETRY). That keeps current behaviour for everyone else and gives the override mechanism you mention.

Happy to push that if it works for you.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, I misread. I like the idea to use the retry pattern if set or fallback on the constant

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, I misread. I like the idea to use the retry pattern if set or fallback on the constant

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@guewen Implemented.

…y delay

Follow-up to the review on OCA#1009.

A job that keeps hitting serialization failures was waiting 5 seconds before
each retry, ignoring its specific retry_pattern.

Now, we pass no delay at all when the job function declares a retry_pattern,
and the pattern will apply as it does for any other RetryableJobError. Jobs
without a pattern are still passed PG_RETRY and still wait 5 seconds.

The constant cannot simply be dropped for them: _get_retry_seconds() would then
fall back to RETRY_INTERVAL - ten minutes - for a conflict that usually clears
at once.

A job function that declares a retry_pattern will change behaviour: a
serialization failure now waits what the pattern says. That is the point of it;
the job has said how it wants to be retried. The surprise is a pattern that
does not cover the early tries - {3: 180} waits RETRY_INTERVAL on tries 1 and
2, so ten minutes where it used to be five seconds.

PG_RETRY is also re-exported from queue_job.controllers.main, so that imports
from its old home keep working.

queue_job_cron_jobrunner had a hardcoded seconds=5, discarding the delay the
error asks for, so a retry_pattern would be ignored under that runner; it now
passes err.seconds like the controller does.

Tests: with no retry_pattern a serialization failure still waits PG_RETRY, and
with one it waits what the pattern says.
Comment thread queue_job/job.py
else:
env_context_manager = nullcontext()
with env_context_manager:
self.result = self.func(*tuple(self.args), **self.kwargs)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm testing this PR on a real project and found that the retry is not working.

SerializationFailure moves a job state to "Failed" without any retries.

 Traceback (most recent call last):
    File "/odoo/external-src/queue/queue_job/controllers/main.py", line 166, in _runjob
      cls._try_perform_job(env, job)
    File "/odoo/external-src/queue/queue_job/controllers/main.py", line 108, in _try_perform_job
      with env.cr.savepoint():
           ^^^^^^^^^^^^^^^^^^
    File "/odoo/src/odoo/sql_db.py", line 116, in __exit__
      self.close(rollback=exc_type is not None)
    File "/odoo/src/odoo/sql_db.py", line 120, in close
      self._close(rollback)
    File "/odoo/src/odoo/sql_db.py", line 146, in _close
      self._cr.flush()
    File "/odoo/src/odoo/sql_db.py", line 182, in flush
      self.transaction.flush()
    File "/odoo/src/odoo/orm/environments.py", line 592, in flush
      self.default_env.flush_all()
    File "/odoo/src/odoo/orm/environments.py", line 388, in flush_all
      self[model_name].flush_model()
    File "/odoo/src/odoo/orm/models.py", line 6383, in flush_model
      self._flush()
    File "/odoo/src/odoo/orm/models.py", line 6451, in _flush
      model.browse(some_ids)._write_multi(vals_list)
    File "/odoo/src/odoo/orm/models.py", line 4600, in _write_multi
      self.env.execute_query(SQL(
    File "/odoo/src/odoo/orm/environments.py", line 534, in execute_query
      self.cr.execute(query)
    File "/odoo/src/odoo/sql_db.py", line 440, in execute
      self._obj.execute(query, params)
  psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update

I think the problem is here. perform executes the code but it's not flushing -- so the error is only raised on the caller method right after the savepoint exits. It's never caught by this try.. except.

We need here either an explicit flush, or move the savepoint here (replacing the nullcontext maybe).

And certainly a test because CI it's not catching this regression

perform() converts the retryable PostgreSQL concurrency errors into
RetryableJobError, which is what subjects them to max_retries. But a job
function rarely raises one itself: it writes, and PostgreSQL reports the
conflict when that write reaches the database.

The runners call perform() inside a savepoint, and leaving the savepoint
is what flushes, so the failure was raised after perform() had returned.

Flushing at the end of perform()'s try puts the failure back where it can
be converted.

Both new tests error with a bare OperationalError without this change, in
the frames the report showed: the savepoint's __exit__ flushing on the way
out of _try_perform_job.
@richard-willdooit

Copy link
Copy Markdown
Author

Thanks @ivantodorovich - reproduced.

Fixed by flushing at the end of perform()'s try, rather than moving the savepoint - _try_perform_job keeps set_done/store outside it on purpose, so that a rollback releases the row locks before the failure is handled from another cursor.

Two tests, around a job function that writes without flushing: one unit test that it is retried and then capped at max_retries, and one through RunJobController._try_perform_job.

The end-to-end one goes through _try_perform_job rather than _runjob because _runjob's postpone branch calls env.cr.rollback(), which a TransactionCase cursor refuses.

One thing unchanged: set_done, store and env.flush_all() run outside the savepoint, so a conflict on the job's own row is still unconverted and would fail the job, where the outer except OperationalError this PR removes used to postpone it. The runner holds that row locked, so I think it is unreachable in practice.

richard-willdooit added a commit to WilldooIT/queue that referenced this pull request Nov 20, 2026
…y delay

Follow-up to the review on OCA#1009.

A job that keeps hitting serialization failures was waiting 5 seconds before
each retry, ignoring its specific retry_pattern.

Now, we pass no delay at all when the job function declares a retry_pattern,
and the pattern will apply as it does for any other RetryableJobError. Jobs
without a pattern are still passed PG_RETRY and still wait 5 seconds.

The constant cannot simply be dropped for them: _get_retry_seconds() would then
fall back to RETRY_INTERVAL - ten minutes - for a conflict that usually clears
at once.

A job function that declares a retry_pattern will change behaviour: a
serialization failure now waits what the pattern says. That is the point of it;
the job has said how it wants to be retried. The surprise is a pattern that
does not cover the early tries - {3: 180} waits RETRY_INTERVAL on tries 1 and
2, so ten minutes where it used to be five seconds.

PG_RETRY is also re-exported from queue_job.controllers.main, so that imports
from its old home keep working.

queue_job_cron_jobrunner had a hardcoded seconds=5, discarding the delay the
error asks for, so a retry_pattern would be ignored under that runner; it now
passes err.seconds like the controller does.

Tests: with no retry_pattern a serialization failure still waits PG_RETRY, and
with one it waits what the pattern says.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

mod:queue_job_cron_jobrunner Module queue_job_cron_jobrunner mod:queue_job Module queue_job mod:test_queue_job Module test_queue_job series:19.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants