Skip to content

Register Nexus operation handle tasks with the workflow instance - #1961

Closed
ningmao-hlyz wants to merge 1 commit into
temporalio:mainfrom
ningmao-hlyz:fix/nexus-handle-task-registration
Closed

ningmao-hlyz wants to merge 1 commit into
temporalio:mainfrom
ningmao-hlyz:fix/nexus-handle-task-registration

Conversation

@ningmao-hlyz

@ningmao-hlyz ningmao-hlyz commented Oct 13, 2026 •

Copy link
Copy Markdown

Fixes #1960

_NexusOperationHandle creates its asyncio task directly (asyncio.Task(fn)), so unlike activity and child workflow handles it never joins the workflow instance's task registry. When the workflow is dropped (worker shutdown or eviction), that task is left pending and asyncio logs Task was destroyed but it is pending! once per waiting workflow, while a workflow parked on an activity or child workflow logs nothing.

This registers the handle's task with the instance like the activity handle (instance._register_task(self, name=...) at _workflow_instance.py:3400) and the child workflow handle (:3569). The task then joins self._tasks, so the cancel-everything loop in _apply_remove_from_cache reaches it and _log_destroy_pending is suppressed, same as the other workflow tasks. A changelog fragment is included under changelog/fixed/.

Verification

  • Baseline (current main, 76875b8, with only the new test applied): test_nexus_operation_handle_task_not_destroyed_on_shutdown fails in ~1.2s -- it starts a workflow that parks on a Nexus operation (no handler worker, matching the issue's reproduction), waits for the operation to appear in history, exits the worker block, then asserts no "Task was destroyed but it is pending!" record was captured; on unpatched code the assertion fails with the message present.
  • With this change: the same test passes and no destroyed-task message is logged.
  • Full tests/nexus/test_nexus_worker_shutdown.py: 5 passed (--workflow-environment local, dev server from the Temporal CLI).
  • ruff format --check / ruff check --select I clean with the pinned 0.15.22; mypy --namespace-packages --check-untyped-defs clean on both changed files.

One note: with the fix, a workflow task that is evicted while parked on a Nexus operation now gets its handle task cancelled during eviction, so the CancelledError branch of operation_handle_fn may add a cancel command as part of eviction. That mirrors what already happens for activity and child workflow tasks; I saw no errors from it in the tests, but it's the one behavioral edge a maintainer may want to look at.

The Nexus operation handle creates its asyncio task directly, so unlike
activity and child workflow handles it never joins the workflow
instance's task registry. When the workflow is dropped (worker shutdown
or eviction), the task is left pending and asyncio logs 'Task was
destroyed but it is pending!' once per waiting workflow. Register the
task like the other workflow tasks so it is cancelled with the rest and
no longer logged.
@ningmao-hlyz
ningmao-hlyz requested a review from a team as a code owner October 13, 2026 21:34
@tconley1428

Copy link
Copy Markdown
Contributor

This issue was already assigned to an internal contributor.

@ningmao-hlyz

Copy link
Copy Markdown
Author

Understood, thanks for the quick response. Leaving this with the internal contributor — feel free to reuse anything from the branch if it's useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Nexus operation handle task logs "Task was destroyed but it is pending!" on worker shutdown/eviction

2 participants