Consolidate two disk util rebalance prechecks into one

[J-HowHuang]

Description

Originally the disk utilization precheck in table rebalance is easily confusing. Reduce the unnecessary information so that it's more intuitive.

Original Behavior

• Precheck diskUtilizationDuringRebalance: shows the highest disk utilization on each server it could get during the rebalance (i.e. the footprint, the worst case if all segments are added first, then removed)
• Precheck diskUtilizationAfterRebalance: shows the net disk utilization on each server after rebalance

These checks warn anyway regardless of lowDiskMode is set or not, where the lowDiskMode config is designed to solve the case when the first fails and the second passes.

New Behavior

Make it only one diskUtilization check. Pass when both the original diskUtilizationDuringRebalance and diskUtilizationAfterRebalance passes, or only diskUtilizationAfterRebalance passes yet lowDiskMode=true, downtime=false, fail otherwise.

Examples

1. Within threshold throughout

"diskUtilization": {
  "preCheckStatus": "PASS",
  "message": "Within threshold (<90%)"
}

2. Over threshold after the rebalance — ERROR regardless of config

"diskUtilization": {
  "preCheckStatus": "ERROR",
  "message": "UNSAFE. Servers with unsafe disk utilization AFTER rebalance (>90%): Server_localhost_8099 (92%)"
}

3. Over threshold only during the rebalance, lowDiskMode=false → ERROR, with the fix named

"diskUtilization": {
  "preCheckStatus": "ERROR",
  "message": "UNSAFE. Servers with unsafe disk utilization DURING rebalance (>90%): Server_localhost_8098 (93%). Enable lowDiskMode to delete segments before adding the new ones"
}

4. Same, lowDiskMode=true → PASS, transient peak is ruled out

"diskUtilization": {
  "preCheckStatus": "PASS",
  "message": "Within threshold (<90%) after rebalance. Some servers would go over it during the rebalance, but lowDiskMode avoids that transient disk usage"
}

5. Same, lowDiskMode=true AND downtime=true → still ERROR, because downtime replaces the IdealState in one go and skips the incremental path lowDiskMode acts on

"diskUtilization": {
  "preCheckStatus": "ERROR",
  "message": "UNSAFE. Servers with unsafe disk utilization during rebalance (>90%): Server_localhost_8098 (93%). lowDiskMode has no effect while downtime is enabled, disable downtime for it to delete segments before adding the new ones"
}

6. Disk usage info not yet collected — unchanged

"diskUtilization": {
  "preCheckStatus": "WARN",
  "message": "Disk usage info has not been updated. Try later or set controller.resource.utilization.checker.initial.delay to a shorter period"
}

7. rebalanceConfigOptions also flags the dead combination, even when disk is fine

"rebalanceConfigOptions": {
  "preCheckStatus": "WARN",
  "message": "lowDiskMode has no effect when downtime is enabled, disable downtime for segments to be deleted before the new ones are added."
}

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 66.98%. Comparing base (504efc4) to head (f2c9602).
⚠️ Report is 70 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19176      +/-   ##
============================================
+ Coverage     65.70%   66.98%   +1.27%     
  Complexity     1423     1423              
============================================
  Files          3439     3453      +14     
  Lines        218064   218948     +884     
  Branches      34679    34805     +126     
============================================
+ Hits         143289   146658    +3369     
+ Misses        63226    60570    -2656     
- Partials      11549    11720     +171     

Flag	Coverage Δ
custom-integration1	?
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-25	66.98% <100.00%> (+1.27%)	⬆️
lane-a	100.00% <ø> (?)
lane-b	0.00% <ø> (?)
temurin	66.98% <100.00%> (+1.27%)	⬆️
unittests	66.98% <100.00%> (+1.27%)	⬆️
unittests1	57.72% <ø> (+0.67%)	⬆️
unittests2	39.04% <100.00%> (+1.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[yashmayya]

Thanks for the improvement, I've left some comments

[yashmayya]

We'll be flagging servers where the existing used space is over the threshold even if diskUtilizationGain == 0? Seems wrong to say unsafe DURING rebalance / recommend enabling lowDiskMode which would do nothing in such a scenario.

[J-HowHuang]

In that case it would also fall into "unsafe AFTER rebalance", and will have the message of that.

[yashmayya]

Should we also check for bestEfforts which seems to also potentially violate lowDiskMode?

[J-HowHuang]

Afaik bestEfforts is orthogonal to lowDiskMode, can you explain your concern?

[yashmayya]

If the EV-IS progress stalls, IIUC bestEfforts will cause the rebalance algorithm to give up waiting and just move to the next step which could mean that we add new segments to servers before old ones are actually offloaded?

[J-HowHuang]

This is possible but in reality it should be pretty rare, because in lowDiskMode rebalancer first updates the IS with offload only and wait. No loading (segment added) is involved in this pass, so all of them should be completed in a short time, unless the rebalance config sets the wait timeout to 0 or a very small number.

Other than this risk, I think rebalance with both bestEfforts and lowDiskMode is a valid usage. Adding message here might confuse users that's my concern.