[AURON #2017] [BUILD] Add Spark 4.x support to dev/reformat script.#2018
[AURON #2017] [BUILD] Add Spark 4.x support to dev/reformat script.#2018slfan1989 wants to merge 2 commits intoapache:masterfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This pull request adds support for Spark 4.0 and 4.1 to the dev/reformat script, enabling automated code formatting and style checks for the new Spark versions. The script is updated to automatically switch between JDK and Scala versions based on the Spark version being processed, as Spark 4.x requires JDK 17+ and Scala 2.13, while Spark 3.x uses JDK 8 and Scala 2.12.
Changes:
- Fixed Maven profile references for Flink and Iceberg to use correct versioned profile IDs
- Added Spark 4.0 and 4.1 to the version sweep array in the reformat script
- Implemented automatic JDK and Scala version switching logic for Spark 4.x vs 3.x
- Updated CI workflow to set up both JDK 8 and JDK 17
- Applied formatting changes to Scala files (indentation adjustments from running the reformatter)
Reviewed changes
Copilot reviewed 3 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| dev/reformat | Core changes: fixed Flink/Iceberg profiles, added Spark 4.0/4.1 to version array, added JDK/Scala switching logic |
| .github/workflows/style.yml | Added JDK 17 setup to support Spark 4.x formatting in CI |
| thirdparty/auron-uniffle/src/main/scala/org/apache/spark/sql/execution/auron/shuffle/uniffle/AuronUniffleShuffleReader.scala | Indentation formatting adjustment |
| spark-extension/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeParquetInsertIntoHiveTableBase.scala | Indentation formatting adjustment |
| spark-extension/src/main/scala/org/apache/spark/sql/auron/util/TaskContextHelper.scala | Indentation formatting adjustment |
| spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeHelper.scala | Indentation formatting adjustment |
| spark-extension/src/main/scala/org/apache/spark/sql/auron/AuronSparkSessionExtension.scala | Indentation formatting adjustment |
| spark-extension/src/main/scala/org/apache/spark/sql/auron/AuronConverters.scala | Indentation formatting adjustments |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@cxzl25 Thanks for reviewing this pr! I’ll continue refining this pr. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 8 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
slfan1989
force-pushed
the
auron-2017
branch
from
428854b to
5ac544f
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 8 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
slfan1989
force-pushed
the
auron-2017
branch
from
a096ee8 to
c099533
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 8 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 8 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…ipt. Signed-off-by: slfan1989 <slfan1989@apache.org>
…ipt. Signed-off-by: slfan1989 <slfan1989@apache.org>
Which issue does this PR close?
Closes #2017
Rationale for this change
With Spark 4.0 and 4.1 support added to the project, the
dev/reformatscript needs to be updated to handle formatting and style checks for these new versions. Spark 4.x requires JDK 17+ and Scala 2.13, while Spark 3.x uses JDK 8 and Scala 2.12. The script should automatically switch between these environments.What changes are included in this PR?
1. Fix Flink Maven profile
2.Add Spark 4.x support
3.Update CI workflow (.github/workflows/style.yml)
Are there any user-facing changes?
No.
How was this patch tested?
Verified automatic JDK switching works for Spark 3.x (JDK 8) and Spark 4.x (JDK 17)