Skip to content

feat: introduce shouldRestartCauseTaskDisruption in SupervisorSpec#19700

Open
Fly-Style wants to merge 3 commits into
apache:masterfrom
Fly-Style:supervisor-introduce-restart-quiet
Open

feat: introduce shouldRestartCauseTaskDisruption in SupervisorSpec#19700
Fly-Style wants to merge 3 commits into
apache:masterfrom
Fly-Style:supervisor-introduce-restart-quiet

Conversation

@Fly-Style

Copy link
Copy Markdown
Contributor

This patch:

  • adds SupervisorSpec#shouldRestartCauseTaskDisruption() (default true, preserving current behavior) so a supervisor spec can opt out of disrupting its running tasks when the supervisor is stopped/recreated on restart. There are some supervisor which might not want to interrupt their tasks during spec update.
  • terminate() continues to always pass true (termination should always disrupt tasks).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a new SupervisorSpec extension point to control whether supervisor restarts (stop+recreate) should gracefully stop managed tasks, enabling certain supervisor implementations to avoid disrupting running tasks during spec updates while preserving existing default behavior.

Changes:

  • Added SupervisorSpec#shouldRestartCauseTaskDisruption() with a default of true to preserve existing behavior.
  • Threaded the new flag through SupervisorManager so restarts can call Supervisor.stop(...) with either disruptive (true) or non-disruptive (false) behavior, while termination continues to always disrupt tasks.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File Description
server/src/main/java/org/apache/druid/indexing/overlord/supervisor/SupervisorSpec.java Introduces the new default method used to decide whether restarts should disrupt managed tasks.
indexing-service/src/main/java/org/apache/druid/indexing/overlord/supervisor/SupervisorManager.java Wires the new flag into supervisor restart/stop flows by passing it through to Supervisor.stop(...).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

/**
* Returns true if the supervisor restart should cause task disruption, false otherwise
*/
default boolean shouldRestartCauseTaskDisruption()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whether to restart tasks or not will depend on the difference between the current spec and proposed spec.

In some cases, we might want to rollover tasks and in other cases, we might want to continue with the existing tasks.

Suggested change
default boolean shouldRestartCauseTaskDisruption()
default boolean shouldRolloverTasksOnRestart(SupervisorSpec proposedSpec);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

private SupervisorSpec possiblyStopAndRemoveSupervisorInternal(
String id,
boolean writeTombstone,
boolean shouldRestartCauseTaskDisruption

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
boolean shouldRestartCauseTaskDisruption
boolean terminateTasks

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@kfaraz

kfaraz commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

@Fly-Style , since the methods requireRestart and the new one (shouldRolloverTasksOnRestart) are closely related, I wonder if we shouldn't just have a single method that returns a tri-state enum.

The requireRestart method could then be changed to:

SupervisorAction getActionOnUpdateTo(SupervisorSpec proposedSpec);

enum SupervisorAction
{
  NONE, RESTART_SUPERVISOR, RESTART_SUPERVISOR_AND_TASKS
}

The advantage is that implementers would have to implement just one method and not need to worry about how the two methods play together. The end result would be much more intuitive.

cc: @jtuglu1 , do you have any thoughts?

* republish) the currently running tasks, false if the existing tasks should be left running as-is.
* Invoked on the running spec; default is conservative (always roll over).
*/
default boolean shouldRolloverTasksOnRestart(SupervisorSpec proposedSpec)

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity Findings
P0 0
P1 0
P2 1
P3 0
Total 1

Reviewed 2 of 2 changed files.

Found one lifecycle race where an active autoscaler can still disrupt tasks during an opted-out restart.


This is an automated review by Codex GPT-5.6-Sol

); // where NoopSupervisorSpec is a tombstone
}
pair.lhs.stop(true);
pair.lhs.stop(terminateTasks);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Stop the autoscaler before a non-disruptive restart

When terminateTasks is false, the old autoscaler remains active throughout pair.lhs.stop(false). Its allocation executor can enqueue a DynamicAllocationTasksNotice ahead of the ShutdownNotice; that handler does not reject the STOPPING state and invokes gracefulShutdownInternal(), rolling over the tasks despite the spec explicitly opting out. Stop/remove the autoscaler before stopping the supervisor, or make allocation notices ignore STOPPING, to preserve the non-disruption guarantee.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants