tests: parallel batching for packages when running unit tests

[parthea]

A legacy bottleneck in the unit test presubmit causes a timeout/crash after 4.5 hours (log) when testing all packages. This is blocking PR #17585 (Python 3.15 testing for all packages).

This PR introduces dynamic parallel execution, dropping runtime from several hours to under 10 minutes.

Key Changes

• Add TEST_ALL_PACKAGES variable: When set to true, it forces the workflow to discover and execute testing across all repository packages regardless of the git diff.

• ci/run_conditional_tests.sh: Accepts optional explicit package paths to run to facilitate batching

• Adds a discover-packages job to the GH action workflow to batch testing of packages based on a configurable BATCH_SIZE (default: 5). It respects the TEST_ALL_PACKAGES variable which toggles testing of all packages, vs only the ones that have a diff.

• Parallel Execution: Splitting the workload across a dynamic matrix of runners. Coverage artifacts use unique index names to prevent overwrites.

[gemini-code-assist]

Code Review

This pull request updates ci/run_conditional_tests.sh to allow passing specific package paths as optional arguments to run tests only on those directories, defaulting to the entire packages directory if no arguments are provided. The review feedback highlights two key improvements: first, making the check for the "packages" directory robust against trailing slashes (e.g., "packages/") by stripping them before comparison; second, replacing the fragile parsing of ls with safer, more idiomatic Bash globbing.

[parthea]

/gemini review

[gemini-code-assist]

Code Review

This pull request updates ci/run_conditional_tests.sh to allow passing specific package paths as optional arguments to restrict testing to those directories, falling back to the default behavior of scanning the packages directory if no arguments are provided. The review feedback identifies two robustness issues: first, handling non-existent directory arguments by validating their existence and exiting early; second, preventing issues with unmatched glob patterns when a directory is empty by verifying that each resolved path actually exists before processing.