feat: introduce shouldRestartCauseTaskDisruption in SupervisorSpec#19700
feat: introduce shouldRestartCauseTaskDisruption in SupervisorSpec#19700Fly-Style wants to merge 3 commits into
shouldRestartCauseTaskDisruption in SupervisorSpec#19700Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds a new SupervisorSpec extension point to control whether supervisor restarts (stop+recreate) should gracefully stop managed tasks, enabling certain supervisor implementations to avoid disrupting running tasks during spec updates while preserving existing default behavior.
Changes:
- Added
SupervisorSpec#shouldRestartCauseTaskDisruption()with a default oftrueto preserve existing behavior. - Threaded the new flag through
SupervisorManagerso restarts can callSupervisor.stop(...)with either disruptive (true) or non-disruptive (false) behavior, while termination continues to always disrupt tasks.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
server/src/main/java/org/apache/druid/indexing/overlord/supervisor/SupervisorSpec.java |
Introduces the new default method used to decide whether restarts should disrupt managed tasks. |
indexing-service/src/main/java/org/apache/druid/indexing/overlord/supervisor/SupervisorManager.java |
Wires the new flag into supervisor restart/stop flows by passing it through to Supervisor.stop(...). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| /** | ||
| * Returns true if the supervisor restart should cause task disruption, false otherwise | ||
| */ | ||
| default boolean shouldRestartCauseTaskDisruption() |
There was a problem hiding this comment.
Whether to restart tasks or not will depend on the difference between the current spec and proposed spec.
In some cases, we might want to rollover tasks and in other cases, we might want to continue with the existing tasks.
| default boolean shouldRestartCauseTaskDisruption() | |
| default boolean shouldRolloverTasksOnRestart(SupervisorSpec proposedSpec); |
| private SupervisorSpec possiblyStopAndRemoveSupervisorInternal( | ||
| String id, | ||
| boolean writeTombstone, | ||
| boolean shouldRestartCauseTaskDisruption |
There was a problem hiding this comment.
| boolean shouldRestartCauseTaskDisruption | |
| boolean terminateTasks |
|
@Fly-Style , since the methods The SupervisorAction getActionOnUpdateTo(SupervisorSpec proposedSpec);
enum SupervisorAction
{
NONE, RESTART_SUPERVISOR, RESTART_SUPERVISOR_AND_TASKS
}The advantage is that implementers would have to implement just one method and not need to worry about how the two methods play together. The end result would be much more intuitive. cc: @jtuglu1 , do you have any thoughts? |
| * republish) the currently running tasks, false if the existing tasks should be left running as-is. | ||
| * Invoked on the running spec; default is conservative (always roll over). | ||
| */ | ||
| default boolean shouldRolloverTasksOnRestart(SupervisorSpec proposedSpec) |
FrankChen021
left a comment
There was a problem hiding this comment.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 0 |
| P2 | 1 |
| P3 | 0 |
| Total | 1 |
Reviewed 2 of 2 changed files.
Found one lifecycle race where an active autoscaler can still disrupt tasks during an opted-out restart.
This is an automated review by Codex GPT-5.6-Sol
| ); // where NoopSupervisorSpec is a tombstone | ||
| } | ||
| pair.lhs.stop(true); | ||
| pair.lhs.stop(terminateTasks); |
There was a problem hiding this comment.
[P2] Stop the autoscaler before a non-disruptive restart
When terminateTasks is false, the old autoscaler remains active throughout pair.lhs.stop(false). Its allocation executor can enqueue a DynamicAllocationTasksNotice ahead of the ShutdownNotice; that handler does not reject the STOPPING state and invokes gracefulShutdownInternal(), rolling over the tasks despite the spec explicitly opting out. Stop/remove the autoscaler before stopping the supervisor, or make allocation notices ignore STOPPING, to preserve the non-disruption guarantee.
This patch:
SupervisorSpec#shouldRestartCauseTaskDisruption()(defaulttrue, preserving current behavior) so a supervisor spec can opt out of disrupting its running tasks when the supervisor is stopped/recreated on restart. There are some supervisor which might not want to interrupt their tasks during spec update.terminate()continues to always passtrue(termination should always disrupt tasks).