Repository navigation
Register Nexus operation handle tasks with the workflow instance - #1961
Closed
ningmao-hlyz wants to merge 1 commit into
Closed
ningmao-hlyz wants to merge 1 commit into
ningmao-hlyz wants to merge 1 commit into
Conversation
The Nexus operation handle creates its asyncio task directly, so unlike activity and child workflow handles it never joins the workflow instance's task registry. When the workflow is dropped (worker shutdown or eviction), the task is left pending and asyncio logs 'Task was destroyed but it is pending!' once per waiting workflow. Register the task like the other workflow tasks so it is cancelled with the rest and no longer logged.
Contributor
|
This issue was already assigned to an internal contributor. |
Author
|
Understood, thanks for the quick response. Leaving this with the internal contributor — feel free to reuse anything from the branch if it's useful. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1960
_NexusOperationHandlecreates its asyncio task directly (asyncio.Task(fn)), so unlike activity and child workflow handles it never joins the workflow instance's task registry. When the workflow is dropped (worker shutdown or eviction), that task is left pending and asyncio logsTask was destroyed but it is pending!once per waiting workflow, while a workflow parked on an activity or child workflow logs nothing.This registers the handle's task with the instance like the activity handle (
instance._register_task(self, name=...)at_workflow_instance.py:3400) and the child workflow handle (:3569). The task then joinsself._tasks, so the cancel-everything loop in_apply_remove_from_cachereaches it and_log_destroy_pendingis suppressed, same as the other workflow tasks. A changelog fragment is included underchangelog/fixed/.Verification
main, 76875b8, with only the new test applied):test_nexus_operation_handle_task_not_destroyed_on_shutdownfails in ~1.2s -- it starts a workflow that parks on a Nexus operation (no handler worker, matching the issue's reproduction), waits for the operation to appear in history, exits the worker block, then asserts no "Task was destroyed but it is pending!" record was captured; on unpatched code the assertion fails with the message present.tests/nexus/test_nexus_worker_shutdown.py: 5 passed (--workflow-environment local, dev server from the Temporal CLI).ruff format --check/ruff check --select Iclean with the pinned 0.15.22;mypy --namespace-packages --check-untyped-defsclean on both changed files.One note: with the fix, a workflow task that is evicted while parked on a Nexus operation now gets its handle task cancelled during eviction, so the
CancelledErrorbranch ofoperation_handle_fnmay add a cancel command as part of eviction. That mirrors what already happens for activity and child workflow tasks; I saw no errors from it in the tests, but it's the one behavioral edge a maintainer may want to look at.