[CELEBORN-2239] Support Spark 4.1

[SteNicholas]

What changes were proposed in this pull request?

Support Spark 4.1.

Why are the changes needed?

Spark 4.1.1 has already released, which release notes refer to Spark Release 4.1.1.

Does this PR resolve a correctness bug?

No.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

CI.

[SteNicholas]

Ping @pan3793, @turboFei.

[github-actions]

This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[SteNicholas]

@pan3793, I have bumped version to 4.1.1 and excluded netty dependencies from spark-sql dependency. PTAL.

[pan3793]

@SteNicholas AFAIK, Spark 4.1 uses Netty 4.2 exclusive classes, so integration test should fail (suppose you also exclude Netty 4.2 from test scope), in this case, I'm afraid the right path is - upgrading Netty 4.2 for Celeborn first, then try integration with Spark 4.1

[github-actions]

This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[SteNicholas]

@pan3793, after #3596 merged, please help to review the changes of this pull request.

[pan3793]

seems you forgot to update the CI build and test workflow

[SteNicholas]

@pan3793, sorry for missing CI of spark-4.1. The CI of spark-4.1 is passed. PTAL.

[SteNicholas]

Merged to main(v0.7.0).