[SYSTEMDS-3922] Fix federated quantile VALUEPICK averaging for even-length input

[MegaByteTron]

Summary

Two related fixes for the SystemDS quantile-pick averaging convention on
even-length inputs. The federated fix is the primary contribution
([SYSTEMDS-3922]); the Spark fix was added at reviewer request as the
SP analog of the CP work done in [SYSTEMDS-3898].

Commit 1 — [SYSTEMDS-3922] Federated VALUEPICK averaging

The federated QuantilePickFEDInstruction only enabled the even-length
averaging branch for MEDIAN, while the CP reference path
(QuantilePickCPInstruction) applies the same averaging convention for
VALUEPICK by passing matBlock.getLength() % 2 == 0 to
MatrixBlock.pickValue.

As a result, quantile(A, p) on a federated matrix with an even row
count diverged from the CP reference (which compareResults(1e-9)
rejects), causing the federatedQuantile{1,2,3}{CP,SP} tests to fail.
They were previously marked @Ignore to hide the failure.

• Extend the average flag in processRowQPick to also cover
  VALUEPICK. IQM is intentionally excluded because it has its own
  interpolation path.
• Un-ignore the six previously-disabled tests
  (federatedQuantile{1,2,3}{CP,SP}) and drop the unused
  org.junit.Ignore import.

Commit 2 — [SYSTEMDS-3898] Spark VALUEPICK averaging

Added at David's (ywcb00) request in the PR discussion — the SP analog
of the CP work in 3ae6a77 and ebdabf1. Without it, running the same
DML program in CP vs SP mode returned different values on any
even-length input.

• Mirror the CP condition in QuantilePickSPInstruction.getWeightedQuantileSummary:
  when the input length is even and the picked key is not the last row,
  add the next value and divide by two.
• Add the key < mc.getRows() guard (SP analog of the
  pos < getNumRows() - 1 fix in ebdabf1) for boundary quantiles.
• Un-ignore testQuantileBugSP.
• Weighted branch (mc.getCols() == 2) still uses the same non-averaged
  key lookup and remains flagged by the existing TODO — same scope note
  as 3ae6a77.

Test plan

• FederatedQuantileTest — all 12 tests pass locally (CP + SP),
  including the six previously-ignored ones.
• federatedMedian{CP,SP}, federatedIQR{CP,SP},
  federatedQuantiles{CP,SP} still pass — IQM and MEDIAN
  paths untouched.
• QuantileTest — all 20+ tests pass locally, including the
  newly-enabled testQuantileBugSP.

Resolves https://issues.apache.org/jira/browse/SYSTEMDS-3922
Related: https://issues.apache.org/jira/browse/SYSTEMDS-3898

[ywcb00]

Hi @MegaByteTron, please have a look at the following two commits
3ae6a7764a
ebdabf15ba
, which fixed the respective mistake for CP execution. There are test cases named QuantileBug, which will trigger an error when implemented for FED and SP.
All the best,
David

[MegaByteTron]

Hi @ywcb00, thanks for the pointer!

I've added a second commit (4ace52d) tagged [SYSTEMDS-3898] that mirrors the CP fix pattern for the unweighted SP path in QuantilePickSPInstruction.getWeightedQuantileSummary — the length-based mc.getRows() % 2 == 0 condition from 3ae6a77, plus the boundary guard analog of ebdabf1. testQuantileBugSP is un-ignored and green locally alongside the rest of QuantileTest.

The weighted branch (mc.getCols() == 2) is intentionally left as-is to match the scope note from 3ae6a77 about weighted kernels needing a more involved implementation — the existing TODO stays in place there.

Let me know if you'd like anything adjusted.

[codecov]

Codecov Report

❌ Patch coverage is 81.81818% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.60%. Comparing base (895bbeb) to head (e64f3d0).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
.../instructions/spark/QuantilePickSPInstruction.java	80.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2497      +/-   ##
============================================
+ Coverage     71.58%   71.60%   +0.02%     
- Complexity    49872    49897      +25     
============================================
  Files          1602     1602              
  Lines        193154   193157       +3     
  Branches      37817    37818       +1     
============================================
+ Hits         138260   138308      +48     
+ Misses        44106    44067      -39     
+ Partials      10788    10782       -6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.