[SPARK-59821][CORE] Make Maven utility tests safe for concurrent runs

[qianlan717]

What changes were proposed in this pull request?

This follow-up to #58874 makes the Maven test utilities safe when the same suite runs concurrently.

• Compile generated Java classes in an invocation-specific temporary directory instead of the process working directory.
• Give each test JVM its own stable fake local Maven repository.
• Add a regression test that compiles the same class name concurrently into separate destinations.

Why are the changes needed?

SparkTestUtils.createCompiledClass currently asks javac to emit a fixed class name, such as MyLib.class, in the process working directory before moving it to the requested destination. MavenUtils.m2Path also uses the fixed relative path dummy/.m2/repository during tests.

When a test target is repeated concurrently, the JVMs can move or delete files needed by another JVM. This causes failures such as:

java.nio.file.NoSuchFileException:
  MyLib.class -> dummy/.m2/repository/.../MyLib.class

Using isolated temporary paths removes this cross-JVM interference without changing production Maven resolution.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Added a SparkTestUtilsSuite regression test that starts eight concurrent compilations of the same class name and verifies every output exists.

git diff --check passes.

The following focused command was attempted locally:

/usr/bin/sbt \
  "common-utils/testOnly org.apache.spark.util.SparkTestUtilsSuite" \
  "common-utils/testOnly org.apache.spark.util.MavenUtilsSuite"

The suites could not start because the local environment could not resolve Maven Central or the SBT plugin repositories (UnknownHostException). GitHub Actions will provide the executable test result.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-5)

[HyukjinKwon]

0 addressed, 0 remaining, 0 new. No findings in this same-commit re-review.

Verification

Independently traced every createCompiledClass and m2Path caller, including packaged-class output and Maven repository deletion/recreation. A direct eight-thread javac probe completed 400/400 same-name compilations successfully and preserved the flat destination contract. Focused SBT suites were blocked during project loading because this machine redirects Maven Central to localhost; upstream CI currently reports no failures.

[uros-b]

Thank you @qianlan717 and @HyukjinKwon! Please try to get CI run complete & green

[qianlan717]

@uros-b and @HyukjinKwon CI is green. Can you help merge this PR to master and branch-4.x? Thanks!

[HyukjinKwon]

Merge Summary:

• merged into master aeda065
• merged into branch-4.x b1a57d5

Posted by merge_spark_pr.py