You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi, thanks for this package — it's been working well for us.
I ran into an interesting edge case around the exit code of the crontask management command when the Redis lock can't be acquired. Looking at crontask.py:
exceptutils.LockNotOwnedErrorase:
capture_exception(e)
self.stderr.write(
"The lock is no longer owned by the scheduler. Shutting down."
)
exceptutils.LockErrorase:
capture_exception(e)
self.stderr.write("Another scheduler is already running.")
Bothexceptionsarecaught, reportedtoSentry, andwrittentostderr, buthandle() thenreturnsnormally — sotheprocessexitswithcode0.
Context: we run this on Fly.io Machines, with two replicas of the same process group (a primary + a standby) for HA. On deploy, both instances briefly race for the lock. The loser exits cleanly (as designed, I assume — it's not really an error, just "someone else already grabbed the lock"). The problem is that Fly's default machine restart policy is on-failure, which only restarts on a non-zero exit code. Since the loser exits 0, it's never restarted — it just sits stopped indefinitely until someone manually restarts it, even though crontask is meant to run continuously as a long-lived daemon.
I suspect the current behavior assumes an orchestrator that reconciles process count independently of exit code (e.g., a Kubernetes Deployment, which keeps N replicas alive regardless of container exit status) — which isn't the case on every platform.
Was exiting 0 here a deliberate choice (treating "lost the lock race" as a non-error, expected outcome), or would you be open to exiting non-zero in these cases (or making it configurable) so that exit-code-based supervisors can restart the instance too? Happy to open a PR if that's a welcome direction — just wanted to check the intent first.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Hi, thanks for this package — it's been working well for us.
I ran into an interesting edge case around the exit code of the crontask management command when the Redis lock can't be acquired. Looking at crontask.py:
Context: we run this on Fly.io Machines, with two replicas of the same process group (a primary + a standby) for HA. On deploy, both instances briefly race for the lock. The loser exits cleanly (as designed, I assume — it's not really an error, just "someone else already grabbed the lock"). The problem is that Fly's default machine restart policy is on-failure, which only restarts on a non-zero exit code. Since the loser exits 0, it's never restarted — it just sits stopped indefinitely until someone manually restarts it, even though crontask is meant to run continuously as a long-lived daemon.
I suspect the current behavior assumes an orchestrator that reconciles process count independently of exit code (e.g., a Kubernetes Deployment, which keeps N replicas alive regardless of container exit status) — which isn't the case on every platform.
Was exiting 0 here a deliberate choice (treating "lost the lock race" as a non-error, expected outcome), or would you be open to exiting non-zero in these cases (or making it configurable) so that exit-code-based supervisors can restart the instance too? Happy to open a PR if that's a welcome direction — just wanted to check the intent first.
Thanks!
All reactions