[andygrove]

Which issue does this PR close?

Part of #4329.

Rationale for this change

Spark 3.4 has been deprecated since 1.0.0 and no longer gates a merge (#5885). This is the first, minimal step of dropping it: stop building it and stop testing it. The Spark 3.4 shims are left alone for a follow-up so this change stays easy to review.

What changes are included in this PR?

• Remove the spark-3.4 profile from pom.xml and spark/pom.xml, and from the contrib-delta enforcer's profile list.
• Remove the spark_3_4 Spark SQL job and the iceberg_1_8 job (Iceberg 1.8.1 ran against Spark 3.4.3) from ci.yml, along with their compute-changes.py filters and POLICY entries and the matching check-ci-config.py cases. The run-spark-3.4-tests label no longer gates anything.
• Remove Spark 3.4 from the Linux test matrix (linux-test-profiles.py), the Lint Java matrix and the snapshot publish variants.
• Remove Spark 3.4 from the release, docs-generation and golden-file scripts (dev/release/*, docs/build.sh, dev/generate-release-docs.sh, dev/regenerate-golden-files.sh).
• Update the contributor and user docs (CI tiers, workflow README, Spark SQL and Iceberg test guides, installation, compatibility pages) and drop the Spark 3.4 expression compatibility index page.

Deliberately not touched:

• spark/src/main/spark-3.4/, spark/src/test/spark-3.4/ and any other shim. Nothing builds them now; they can be deleted in a follow-up.
• dev/diffs/3.4.3.diff and dev/diffs/iceberg/1.8.1.diff. Nothing uses them now.
• The !spark/src/main/spark-3.4/** path excludes in compute-changes.py, which keep edits to the orphaned shim directory from triggering other jobs.

Follow-ups this leaves:

• The shared approved-plans-v1_4 / approved-plans-v2_7 golden-file base tier was only regenerable on the 3.4 profile (the other profiles write to their own -sparkN_M directory). The files still resolve correctly, so nothing needs regenerating now, but a future change that alters plans for every version has no way to update the base. Decide whether 3.5 becomes the base or the base is folded into -spark3_5.
• Branch protection should not require any 3.4 check; none is a distinct required context, as Required Checks aggregates the jobs.

How are these changes tested?

• python3 dev/ci/check-ci-config.py, check-versioned-source-parity.py and check-suites.py pass.
• ./mvnw -N help:evaluate -Dexpression=spark.version -Pspark-3.5 still resolves (3.5.9), and the root pom parses.
• The modified workflow files parse under actionlint (only the existing shellcheck style notes remain).
• Not run: a full Maven build or the Spark SQL suites. CI on this PR exercises the remaining profiles.

[mbutrovich]

Don't threaten me with a good time! Thanks @andygrove!

[comphead]

Thanks @andygrove, the profile, workflow, filter, POLICY and check-ci-config.py changes are consistent with each other. I extracted .github and dev/ci at this head and python3 dev/ci/check-ci-config.py prints CI config checks passed.

A few remaining references to what this PR removes that have no diff anchor, since the lines were not touched:

• docs/source/contributor-guide/ci.md line 301 still says the nightly run covers Iceberg "1.8, 1.9 and 1.10". The ci.yml comment on iceberg_1_11 (line 608) says "1.8/1.9/1.10 run nightly".
• docs/source/about/faq.md line 55 ("Spark 3.4, 3.5, 4.0, 4.1, and 4.2 ... Spark 3.4 is deprecated") and docs/source/about/gluten_comparison.md line 87 ("Comet supports Spark 3.4, 3.5, ...") still list 3.4 as supported, which now contradicts README.md and installation.md.
• docs/source/contributor-guide/iceberg-writes.md line 458 still lists "1.5.2 for Spark 3.4" as a profile pin, and docs/source/user-guide/latest/iceberg.md lines 656 and 711 describe "the Spark 3.4 profile" as if it still exists.

[comphead]

The removed Spark 3.4, JDK 17, Scala 2.12 row was the only place the Comet suites ran on Scala 2.12 (the Spark 3.5 row here is -Pscala-2.13). After this change Scala 2.12 only gets compiled by lint-java and runs CelebornReflectionCompatibilitySuite, plus Spark's own SQL suite for 3.5 when it runs against a 2.12 build of Comet. We still publish comet-spark-spark3.5_2.12, and the comment above this list says the goal is coverage of every Java, Scala and Spark version. Should the 3.5 nightly row move to -Pscala-2.12, or should a Spark 3.5, JDK 17, Scala 2.12 nightly row be added next to it? I have not run the matrix. This is from grepping scala-2.12 over .github and dev/ci at this head and at upstream/main.

[andygrove]

Added a Spark 3.5, JDK 17, Scala 2.12 nightly row next to the 2.13 one, so Scala 2.12 keeps running the Comet suites. Pushed in 3819297.

[comphead]

The accepted-version regex still includes 3\.4, while the error message below it and the default versions list no longer do. --spark-version 3.4 passes validation and then runs Maven with -Pspark-3.4, which does not exist any more. Could 3\.4| be dropped from the pattern?

[andygrove]

Dropped 3\.4| from the pattern, so --spark-version 3.4 now hits the invalid-version error. Pushed in 3819297.

[comphead]

The 3.4 table row, warning and download links are gone, but I could not find a place that tells an existing 3.4 user that the next release no longer builds it. migration-guide.md has no section for the version main is heading to (1.3.0-SNAPSHOT), and versioning_policy.md tells users on an unsupported Spark to stay on an earlier Comet release. Could this PR add a short Upgrading to Comet 1.3.0 entry saying Spark 3.4 is no longer built or published and that users on 3.4 should stay on the last release that has comet-spark-spark3.4_*? The JDK 11 removal has the same kind of note in the 1.1.0 section.

[andygrove]

Added an Upgrading to Comet 1.3.0 section to migration-guide.md saying Spark 3.4 is no longer built or published and that 3.4 users should stay on the last release with comet-spark-spark3.4_*. Pushed in 3819297.

[andygrove]

Thanks for the review. The stale references are fixed in 3819297: the nightly Iceberg list in ci.md and the ci.yml comment now read 1.9/1.10, faq.md and gluten_comparison.md no longer list Spark 3.4, and iceberg-writes.md and iceberg.md no longer describe a Spark 3.4 profile or the Iceberg 1.5.2 pin. I left the remaining prose in iceberg.md and iceberg-writes.md that describes Spark 3.4 write behavior, since it documents shim code that the shim-removal follow-up will delete.