feat: Blueapi plan pause - #1589
Open
Alexj9837 wants to merge 8 commits into
Open
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1589 +/- ##
==========================================
+ Coverage 95.96% 96.33% +0.37%
==========================================
Files 45 45
Lines 3317 3387 +70
==========================================
+ Hits 3183 3263 +80
+ Misses 134 124 -10 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
…ead signals Route resume() and cancel_active_task() through the worker thread's signal queue instead of mutating state directly from the caller, so state stays consistent across pause/resume/cancel transitions. Includes tests and a fix for cancel_active_task() when the RunEngine is already in a paused state.
- Resolve a paused task before stopping the worker, instead of leaving the RunEngine paused and the task incomplete forever with no thread left to resolve it. - Apply the latest cancel_active_task() request while paused instead of a stale queued one. - Prevent a race between the caller thread and worker thread when recording a cancelled task's outcome.
racing the worker thread's own finalization, which could report a completed task with no result.
Alexj9837
force-pushed
the
plan-pause
branch
2 times, most recently
from
July 14, 2026 11:04
5f85341 to
f6f82cb
Compare
Alexj9837
marked this pull request as ready for review
July 14, 2026 11:33
fajinyuan
reviewed
Aug 5, 2026
| return self._current.task_id | ||
| self._task_channel.put(CancelSignal(failure=failure, reason=reason)) | ||
| add_span_attributes( | ||
| {"Task aborted" if failure else "Task stopped": reason or ""} |
There was a problem hiding this comment.
should 'default_reason' be used instead of ""? the default reason for abort and stop are different
tpoliaw
requested changes
Aug 10, 2026
tpoliaw
left a comment
tpoliaw
left a comment
Contributor
There was a problem hiding this comment.
Taking a while to figure out what is going on with the task worker in general and what needed to change. There are a couple of blocking things (resumed plan results and exceptions) but feel free to ignore/question the rest.
Might need you to go over why the CancelSignal is needed when it wasn't before - I'm not sure I follow the logic.
| default_reason = "Task failed for unknown reason" | ||
| if failure: | ||
| default_reason = "Task failed for unknown reason" | ||
| with self._status_lock: |
Contributor
There was a problem hiding this comment.
Should this be re-using the status lock? As far as I can see, its other use is related to monitoring statuses from the run engine, not for modifying the status of the worker.
|
|
||
| def _apply_cancel(self, signal: "CancelSignal") -> None: | ||
| self._pending_cancel = None | ||
| default_reason = "Task failed for unknown reason" |
Contributor
There was a problem hiding this comment.
Is it really an unknown reason? If a user aborts a task wouldn't it be more useful to mark the task as "aborted by user" or similar?
| if signal.failure: | ||
| reason = signal.reason or default_reason | ||
| self._ctx.run_engine.abort(reason) | ||
| self._current.set_exception(Exception(reason)) |
Contributor
There was a problem hiding this comment.
Creating a generic exception here hides the fact it was aborted. Is it worth adding an Aborted outcome as an alternative to TaskResult/TaskError?
Or for a smaller change, add a mark_aborted method that creates a TaskError with the message.
| elif self._ctx.run_engine.state == "paused": | ||
| if self._current is not None: | ||
| try: | ||
| result = self._ctx.run_engine.resume() |
Contributor
There was a problem hiding this comment.
I think this is missing all the logging instrumentation that plans currently have. The process_task call above has the plan_tag_filter_context etc around it.
| result = self._ctx.run_engine.resume() | ||
| self._current.set_result(result) | ||
| except RunEngineInterrupted: | ||
| # Plan paused again immediately - not a failure, |
Contributor
There was a problem hiding this comment.
"paused again or aborted" and doesn't have to be immediately. It could be a failure (in the abort as failure sense) but the outcome would already have been set in that case.
I think we also need to handle exceptions here. Currently a task that fails after being resumed does not have its outcome set.
| finally: | ||
| if self._current_task_otel_context is not None: | ||
| if ( | ||
| self._current_task_otel_context is not None |
Contributor
There was a problem hiding this comment.
Should the context be left here or recreated when the task is resumed? I'm not sure how to check this but wouldn't this leave the resumed part of plan running with the telemetry of the original submit and lose trace of the resume happening? Maybe that is what we want?
| if self._current is not None: | ||
| try: | ||
| result = self._ctx.run_engine.resume() | ||
| self._current.set_result(result) |
Contributor
There was a problem hiding this comment.
resume returns a RunEngineResult not the return value of the plan
Suggested change
| self._current.set_result(result) | |
| self._current.set_result(result.plan_result) |
Contributor
|
Another thing to look at - running a count plan and then pausing/resuming several times, it occasionally fails with "Received multiple indices in a `stream_datum` document for one event"I'm not sure if it's related to this change or if there is a bug in the run engine/plan that has only just surfaced now pausing is possible. |
Alexj9837
force-pushed
the
plan-pause
branch
2 times, most recently
from
August 12, 2026 12:24
a5e8e0f to
70671d9
Compare
fixed a merge conflict error
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bluesky's RunEngine pauses via a RunEngineInterrupted, but blueapi was treating that exception like any other plan failure , pausing a task marked it as failed instead of leaving it resumable. On top of that, resume() and cancel_active_task() mutated RunEngine state directly from the calling thread, racing with the worker thread that owns it, which could leave a paused RunEngine stuck forever, apply a stale cancel, or double-set a task's outcome.