[SYSTEMDS-3894] Add OOC covariance and extend TSMM support

[tuluyhansozen]

Overview

This PR adds out-of-core execution for covariance and extends OOC transpose-self matrix multiplication (TSMM) to support right-side operations and multi-tile outputs.

The PR changes 12 files with 688 additions and 59 deletions, covering 5 runtime files, 3 Java test files, and 4 DML test scripts.

Implementation

OOC Covariance

• Adds CovarianceOOCInstruction for both unweighted cov(A, B) and weighted cov(A, B, W) operations.
• Registers covariance through OOCInstructionParser and the new OOCType.COV.
• Joins input streams by block index and aggregates the block-level covariance results into a scalar output.
• Validates matching block sizes for weighted covariance inputs.
• Ignores zero-weight covariance contributions to avoid invalid NaN results.

OOC TSMM

• Extends TSMMOOCInstruction to support multi-tile results for:
  • Left TSMM: t(X) %*% X
  • Right TSMM: X %*% t(X)
• Groups compatible input blocks and computes diagonal and off-diagonal output tiles.
• Produces mirrored off-diagonal tiles to preserve output symmetry.
• Aggregates partial tiles by output block index.
• Reuses a cached input stream for the required block joins and removes the temporary cache after completion.
• Preserves the existing single-output-tile fast path.

Testing

The targeted test suite contains 11 tests:

Covariance — 4 tests

• 2 unweighted covariance tests: dense and sparse.
• 2 weighted covariance tests: dense and sparse.
• Input dimensions: 1871 x 1.
• Block size: 1000, producing two input blocks.
• Dense sparsity: 0.65.
• Sparse sparsity: 0.05.
• Compares CP and OOC results with a tolerance of 1e-10.
• Verifies that the ooc_cov instruction is present in the heavy hitters.

TSMM — 7 tests

• Left single-tile: dense and sparse.
  • Input: 2143 x 123
  • Block size: 1000
  • Output: 123 x 123
• Right single-tile: dense.
  • Input: 123 x 2143
  • Block size: 1000
  • Output: 123 x 123
• Left multi-tile: dense and sparse.
  • Input: 1501 x 1301
  • Block size: 500
  • Output: 1301 x 1301 across 3 x 3 blocks
• Right multi-tile: dense and sparse.
  • Input: 1501 x 1301
  • Block size: 500
  • Output: 1501 x 1501 across 4 x 4 blocks
• Dense sparsity: 0.7.
• Sparse sparsity: 0.1.
• Compares CP and OOC results with a tolerance of 1e-8.
• Verifies ooc_tsmm execution, output metadata, multi-tile coverage, and off-diagonal symmetry.

Test Results

• Targeted covariance and TSMM tests: 11 passed, 0 failures, 0 errors, 0 skipped.
• Java formatter check: passed.
• Java Checkstyle: passed.
• Current GitHub status checks: 51/51 passed.

[codecov]

Codecov Report

❌ Patch coverage is 87.03704% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.59%. Comparing base (366a411) to head (b0620a5).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...ime/instructions/ooc/CovarianceOOCInstruction.java	82.60%	5 Missing and 3 partials ⚠️
...s/runtime/instructions/ooc/TSMMOOCInstruction.java	89.65%	3 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##               main    #2529   +/-   ##
=========================================
  Coverage     71.59%   71.59%           
- Complexity    49893    49922   +29     
=========================================
  Files          1602     1603    +1     
  Lines        193154   193251   +97     
  Branches      37817    37829   +12     
=========================================
+ Hits         138290   138361   +71     
- Misses        44089    44108   +19     
- Partials      10775    10782    +7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[tuluyhansozen]

@janniklinde we solved conflicts and failing checks and created a fresh clean pr.

[janniklinde]

Thank you for the PR. Overall, these changes look very good to me. I left some minor comments in the code @tuluyhansozen @122Astha @AdityaPandey2612

[janniklinde]

Please mark comments as resolved if you have addressed them

[tuluyhansozen]

Last commit is for format check and rebase from current main @janniklinde