Repository navigation
fix(worker): wake the event loop on shutdown signals - #724
AlexeyShalaev wants to merge 1 commit into
Conversation
The worker's signal handler is installed with signal.signal and calls shutdown_event.set(). That only schedules the event's waiters, it doesn't wake the loop up. Without uvloop, a worker that is idle waiting for a message stays blocked in the selector, never shuts down, and the main process waits for it until it gets killed. Set the event with loop.call_soon_threadsafe, which also wakes the loop up, and create the loop before the handlers are installed. Closes taskiq-python#722
OsmnvAslan
left a comment
There was a problem hiding this comment.
Thanks for the clear write-up! I checked it locally (Python 3.14, with uvloop shadowed): the new test fails on master and passes here, and the full suite passes. With a real taskiq worker and the idle broker from the issue, SIGTERM leaves the worker running on master (still alive after 15 s, and broker.shutdown() never runs); with this PR it exits in under a second and broker.shutdown() runs. A SIGTERM that arrives while the tasks are still being imported, before the loop runs, still works too: the queued callback sets the event once the loop starts.
The diagnosis matches how CPython handles signals here: the handler interrupts epoll_wait(), runs, and the call is then retried (PEP 475), so a plain call_soon() from the handler doesn't wake the loop, while call_soon_threadsafe() writes to its self-pipe. A one-line comment on why it's call_soon_threadsafe() might keep someone from "simplifying" it back later. LGTM.
Closes #722
The worker's signal handler is registered with
signal.signaland callsshutdown_event.set(). That only schedules the event's waiters and doesn't wake the loop, so without uvloop a worker that is idle waiting for a message stays blocked in the selector. The main process then waits for it until it gets killed, andbroker.shutdown()doesn't run.The handler now sets the event through
loop.call_soon_threadsafe, which also wakes the loop. The loop is created before the handlers are installed, so it exists by the time a signal can arrive.The test starts a real worker the way the #686 test does and sits next to it to reuse its idle broker and CLI runner. A stub
uvloop.pyin the worker's directory makes it run on the default loop: CI installs uvloop, and with uvloop the test passes even without the fix. It sends SIGTERM to the main process and expects it to exit within 10 s. Without the fix it times out; with it, it passes on Python 3.10, 3.12 and 3.14. The full suite andprek run -apass.The repro from the issue, Python 3.12 without uvloop,
taskiq worker --workers 1, SIGTERM to the main process:broker.shutdown()runsListQueueBroker,socket_timeout=NoneMost of that second is the main process's 1 s polling loop. The worker starts shutting down the broker 3 to 4 ms after it gets the signal.