Skip to content

feat: support scaling direction-aware cooldown for task auto-scalers#19286

Draft
jtuglu1 wants to merge 1 commit intoapache:masterfrom
jtuglu1:support-direction-aware-cooldown-for-autoscaler
Draft

feat: support scaling direction-aware cooldown for task auto-scalers#19286
jtuglu1 wants to merge 1 commit intoapache:masterfrom
jtuglu1:support-direction-aware-cooldown-for-autoscaler

Conversation

@jtuglu1
Copy link
Copy Markdown
Contributor

@jtuglu1 jtuglu1 commented Apr 9, 2026

Description

Adds support for configuring different cooldowns for scaling direction. While both scaling actions do cause temporary disruption to ingestion, scaling down can cause more disruption than scaling up due to having less resources than when you started to recover from lag. Therefore, to allow for aggressive scale up while having a more conservative scale-down approach, this adds configuration for cool down period for both directions. Cool down for a specific scaling direction is evaluated as follows:

  1. minTriggerScaleUpActionFrequencyMillis/minTriggerScaleDownActionFrequencyMillis
  2. minTriggerScaleActionFrequencyMillis
  3. The default of 600000.

This also does the following:

  1. Cleans up the core scaling logic to be more clear/readable
  2. Adds a custom ScaleActionSupplier which makes the contract clear for auto-scalers who want to implement it.
  3. Fixes a few notification bugs, namely emitting/logging metrics irrespective of whether we are actually planning to scale or not (e.g. auto-scaler function returns value != -1, but that value is equal to the current task count).

Release note

Adds support for configuring different cooldowns for scaling direction for streaming task auto-scalers.


This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@jtuglu1 jtuglu1 force-pushed the support-direction-aware-cooldown-for-autoscaler branch from c72942f to fc9c5cb Compare April 9, 2026 18:16
@jtuglu1 jtuglu1 force-pushed the support-direction-aware-cooldown-for-autoscaler branch 2 times, most recently from 52e327f to 164f891 Compare April 9, 2026 19:24
@jtuglu1 jtuglu1 marked this pull request as ready for review April 9, 2026 19:30
@jtuglu1 jtuglu1 requested a review from Fly-Style April 9, 2026 20:19
@jtuglu1 jtuglu1 force-pushed the support-direction-aware-cooldown-for-autoscaler branch from 164f891 to 7fd9d61 Compare April 10, 2026 00:38
@jtuglu1 jtuglu1 force-pushed the support-direction-aware-cooldown-for-autoscaler branch 2 times, most recently from 6fe7359 to ad264b0 Compare April 10, 2026 02:01
@jtuglu1 jtuglu1 requested a review from kfaraz April 10, 2026 02:04
@jtuglu1 jtuglu1 force-pushed the support-direction-aware-cooldown-for-autoscaler branch from ad264b0 to 757cbde Compare April 10, 2026 02:06
@jtuglu1 jtuglu1 marked this pull request as draft April 10, 2026 03:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants