Repository navigation
[FIX] queue_job: apply max_retries to PostgreSQL concurrency errors #1009
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: 19.0
Are you sure you want to change the base?
Changes from all commits
b0b8956
d0c7291
fe7c206
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -12,8 +12,11 @@ | |
| from datetime import datetime, timedelta | ||
| from random import randint | ||
|
|
||
| from psycopg2 import OperationalError | ||
|
|
||
| import odoo | ||
| from odoo.models import BaseModel | ||
| from odoo.service.model import PG_CONCURRENCY_ERRORS_TO_RETRY | ||
|
|
||
| from .exception import FailedJobError, NoSuchJobError, RetryableJobError | ||
|
|
||
|
|
@@ -57,6 +60,7 @@ def _rebind_to_cr(value, cr): | |
|
|
||
| DEFAULT_PRIORITY = 10 # used by the PriorityQueue to sort the jobs | ||
| DEFAULT_MAX_RETRIES = 5 | ||
| PG_RETRY = 5 # seconds, when the job function defines no retry_pattern | ||
| RETRY_INTERVAL = 10 * 60 # seconds | ||
|
|
||
| _logger = logging.getLogger(__name__) | ||
|
|
@@ -511,12 +515,34 @@ def perform(self): | |
| """ | ||
| self.retry += 1 | ||
| try: | ||
| if self.job_config.allow_commit: | ||
| env_context_manager = self.in_temporary_env() | ||
| else: | ||
| env_context_manager = nullcontext() | ||
| with env_context_manager: | ||
| self.result = self.func(*tuple(self.args), **self.kwargs) | ||
| try: | ||
| if self.job_config.allow_commit: | ||
| env_context_manager = self.in_temporary_env() | ||
| else: | ||
| env_context_manager = nullcontext() | ||
| with env_context_manager: | ||
| self.result = self.func(*tuple(self.args), **self.kwargs) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm testing this PR on a real project and found that the retry is not working. SerializationFailure moves a job state to "Failed" without any retries. I think the problem is here. We need here either an explicit flush, or move the savepoint here (replacing the nullcontext maybe). And certainly a test because CI it's not catching this regression |
||
| # Most job functions write without flushing, so the conflicting | ||
| # update is reported when their work reaches the database. The | ||
| # runners call perform() inside a savepoint, and leaving it is | ||
| # what flushes, so the failure would be raised after this try | ||
| # has been left and escape the conversion below. Flush here | ||
| # instead, while it can still become a RetryableJobError. | ||
| self.env.flush_all() | ||
| except OperationalError as err: | ||
| # The typical transaction serialization errors are retryable, | ||
| # and converting them here rather than in the runners is what | ||
| # subjects them to max_retries below: a job raising them from | ||
| # outside perform() is postponed without ever consulting the | ||
| # limit, so it retries for ever. | ||
| if err.pgcode not in PG_CONCURRENCY_ERRORS_TO_RETRY: | ||
| raise | ||
| # A job function that declares a retry_pattern has said how it | ||
| # wants to be retried, so leave the delay to it. | ||
| raise RetryableJobError( | ||
| err.pgerror, | ||
| seconds=None if self.job_config.retry_pattern else PG_RETRY, | ||
| ) from err | ||
| except RetryableJobError as err: | ||
| if err.ignore_retry: | ||
| self.retry -= 1 | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe worth importing it from where it stands now for backward compat.
Not a blocker from my POV.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah maybe worth in case someone customized the default of 5 seconds.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@guewen
Happy to add the re-export, so imports from controllers.main keep working. But it only keeps the import working. If someone changed the value at the old location, job.py won't see it - the delay would quietly go back to 5 seconds.
I'd rather offer a supported way to do it, which ties into your other comment: use the job's retry_pattern when it has one, and fall back to PG_RETRY otherwise.