Skip to content

chore: remove Spark 3.4 Maven profiles and stop running Spark 3.4 CI #7521

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
andygrove wants to merge 2 commits into
base: main
Choose a base branch
from
+104 −473
Open

chore: remove Spark 3.4 Maven profiles and stop running Spark 3.4 CI #7521

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1a5f85e
chore: remove Spark 3.4 Maven profiles and stop running Spark 3.4 CI
andygrove Nov 5, 2026
3819297
address review: keep Scala 2.12 coverage, drop remaining Spark 3.4 re…
andygrove Nov 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 17 additions & 32 deletions .github/workflows/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,25 +26,15 @@ ruleset in `.asf.yaml`. That splits CI into three tiers:
and one Iceberg version, both the default profile's.
- **Nightly tier** (`nightly`): the regression sweep of everything else, once
a day against `main` as it stands. The Comet test suites against the other
four Spark profiles, Spark SQL on Spark 3.5, 4.0 and 4.2, Iceberg
1.8/1.9/1.10, and the Delta contrib suites on Spark 3.5, 4.0 and 4.1. See
three Spark profiles, Spark SQL on Spark 3.5, 4.0 and 4.2, Iceberg
1.9/1.10, and the Delta contrib suites on Spark 3.5, 4.0 and 4.1. See
[Nightly tier](#nightly-tier) below for how a failure surfaces.

Every queue-only and nightly job has a `run-*` label that opts a pull request
into it early, listed in the diagram below. The Lint Java matrix compiles
every Spark profile on every pull request, so a shim that fails to build is
caught there; only the runtime suites wait for the queue or the nightly.

`spark_3_4` is in none of the tiers. Spark 3.4 is deprecated, so its Spark SQL
suite no longer gates a merge; it runs only when a pull request carries
`run-spark-3.4-tests`, or from a `workflow_dispatch`. Anyone who wants to
check a change against 3.4 can still do so, but note when that result starts
to block a merge. Adding the label fires a `labeled` event, and those runs
publish the advisory `Required Checks (label run)` name rather than the
required one, so a red 3.4 there changes nothing. It is the next push with
the label still applied that runs 3.4 under `Required Checks`, and with the
queue run gone that push is the only thing that makes a 3.4 failure blocking.

Heavy jobs have no `push` tier. The queue already tested the exact tree that
lands, so re-running them on push to main would double the cost of every
merge. Three jobs are still on `push`: `docs`, because it deploys to
Expand Down Expand Up @@ -118,17 +108,15 @@ tiers partition the list and that the `pr` tier is exactly the default profile.
delta_build_gate run-delta-build-gate
pyarrow_udf_test run-pyarrow-udf-tests
spark_4_1 run-spark-4.1-tests
label or dispatch only spark_4_1 sql_hive run-spark-4.1-hive-tests
---------------------- iceberg_1_11 run-iceberg-tests
spark_3_4 run-spark-3.4-tests
spark_4_1 sql_hive run-spark-4.1-hive-tests
iceberg_1_11 run-iceberg-tests
nightly tier, or PR with label
------------------------------
pr_build_linux run-all-spark-profiles
(other profiles)
spark_3_5 run-spark-3.5-tests
spark_4_0 run-spark-4.0-tests
spark_4_2 run-spark-4.2-tests
iceberg_1_8 run-iceberg-tests
iceberg_1_9 run-iceberg-tests
iceberg_1_10 run-iceberg-tests
delta_3_5 run-delta-tests
Expand Down Expand Up @@ -176,11 +164,9 @@ tiers partition the list and that the `pr` tier is exactly the default profile.
| `docs` | push to main, paths matched, **or** dispatch on `main` | `.asf.yaml`, `docs/**`, `docs.yaml` |
| `spark_3_5` | nightly, **or** PR with `run-spark-3.5-tests` | Spark 3.5 sources |
| `spark_4_1` | merge group, **or** PR with `run-spark-4.1-tests`; the `sql_hive` shards alone with `run-spark-4.1-hive-tests` | Spark 4.1 sources |
| `spark_3_4` | PR with `run-spark-3.4-tests`, or dispatch | Spark 3.4 sources |
| `spark_4_0` | nightly, **or** PR with `run-spark-4.0-tests` | Spark 4.0 sources |
| `spark_4_2` | nightly, **or** PR with `run-spark-4.2-tests` | Spark 4.2 sources |
| `iceberg_1_11` | merge group, **or** PR with `run-iceberg-tests` | Iceberg sources |
| `iceberg_1_8` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources |
| `iceberg_1_9` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources |
| `iceberg_1_10` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources |
| `required_checks` | always, after every job above except `docs` | none (always runs) |
Expand All @@ -194,8 +180,8 @@ safe to make a required check.
A pull request whose base is a release branch (`branch-N.M`) is the one
exception to the table: it runs every job the table puts in the PR, merge
group or nightly tier, because a release branch has no merge queue and no
nightly to run the last two later. `docs` stays push-only and `spark_3_4`
still needs its label, so a `labeled` run there adds `spark_3_4` or nothing.
nightly to run the last two later. `docs` stays push-only, and a `labeled`
run there adds nothing.
`changes` hands the base branch to `compute-changes.py` as `PR_BASE_REF`.
`check-ci-config.py` pins that wiring, because a dropped variable reads as an
empty string and would route the pull request as if it targeted `main`.
Expand Down Expand Up @@ -289,7 +275,7 @@ Things that follow from that, and are easy to break:
### Label events

`ci_label.yml` fires on `pull_request.types: [labeled]` and calls `ci.yml`
through `workflow_call`, so applying `run-spark-3.4-tests`,
through `workflow_call`, so applying `run-spark-3.5-tests`,
`run-spark-4.0-tests` or `run-iceberg-tests` starts the job that label gates
without needing a new push. GitHub cannot filter a `pull_request` trigger by
label name, so **every** label added to a PR starts a run, including labels
Expand Down Expand Up @@ -393,17 +379,16 @@ umbrella doesn't watch, or operate independently of the rest of CI:

## Reusable workflows (called by `ci.yml`)

| File | Called from `ci.yml` job(s) |
| --------------------------------- | --------------------------------------------------------------- |
| `pr_build_linux.yml` | `pr_build_linux` |
| `pr_build_macos.yml` | `pr_build_macos` |
| `pr_benchmark_check.yml` | `pr_benchmark_check` |
| `delta_build_gate.yml` | `delta_build_gate` |
| `delta_contrib_test.yml` | `delta_3_5`, `delta_4_0`, `delta_4_1` |
| `pyarrow_udf_test.yml` | `pyarrow_udf_test` |
| `docs.yaml` | `docs` |
| `spark_sql_test_reusable.yml` | `spark_3_4`, `spark_3_5`, `spark_4_0`, `spark_4_1`, `spark_4_2` |
| `iceberg_spark_test_reusable.yml` | `iceberg_1_8`, `iceberg_1_9`, `iceberg_1_10`, `iceberg_1_11` |
| File | Called from `ci.yml` job(s) |
| ----------------------------- | ------------------------------------- |
| `pr_build_linux.yml` | `pr_build_linux` |
| `pr_build_macos.yml` | `pr_build_macos` |
| `pr_benchmark_check.yml` | `pr_benchmark_check` |
| `delta_build_gate.yml` | `delta_build_gate` |
| `delta_contrib_test.yml` | `delta_3_5`, `delta_4_0`, `delta_4_1` |
| `pyarrow_udf_test.yml` | `pyarrow_udf_test` |
| `docs.yaml` | `docs` |
| `spark_sql_test_reusable.yml` | `iceberg_spark_test_reusable.yml` |

## Changing what runs when

Expand Down
47 changes: 2 additions & 45 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
# the default profile's.
# schedule the nightly regression sweep of everything else: the Linux
# test matrix against the other Spark profiles, Spark SQL on
# Spark 3.5, 4.0 and 4.2, and Iceberg 1.8/1.9/1.10, routed by
# Spark 3.5, 4.0 and 4.2, and Iceberg 1.9/1.10, routed by
# the same path filters over what landed on main since the last
# green nightly. A failure opens (or comments on) an issue
# labelled `ci-nightly-failure`.
Expand All @@ -44,9 +44,6 @@
# on the merge result. Opt-in `run-*` suites, pull requests from a branch of this
# repository and pull requests to a release branch are unaffected.
#
# Spark 3.4 is deprecated and sits outside all three tiers: it runs only when
# a pull request carries `run-spark-3.4-tests`, or from a manual dispatch.
#
# Release branches (`branch-N.M`) get neither the queue nor the nightly: the
# queue ruleset covers only the default branch, `push` below is main-only, and
# GitHub fires `schedule` only on the default branch. So a pull request that
Expand Down Expand Up @@ -290,13 +287,11 @@ jobs:
delta_gate: ${{ steps.compute.outputs.delta_gate }}
pyarrow_udf: ${{ steps.compute.outputs.pyarrow_udf }}
docs: ${{ steps.compute.outputs.docs }}
spark_3_4: ${{ steps.compute.outputs.spark_3_4 }}
spark_3_5: ${{ steps.compute.outputs.spark_3_5 }}
spark_4_0: ${{ steps.compute.outputs.spark_4_0 }}
spark_4_1: ${{ steps.compute.outputs.spark_4_1 }}
spark_4_1_hive: ${{ steps.compute.outputs.spark_4_1_hive }}
spark_4_2: ${{ steps.compute.outputs.spark_4_2 }}
iceberg_1_8: ${{ steps.compute.outputs.iceberg_1_8 }}
iceberg_1_9: ${{ steps.compute.outputs.iceberg_1_9 }}
iceberg_1_10: ${{ steps.compute.outputs.iceberg_1_10 }}
iceberg_1_11: ${{ steps.compute.outputs.iceberg_1_11 }}
Expand Down Expand Up @@ -485,29 +480,6 @@ jobs:
contents: write
uses: ./.github/workflows/docs.yaml

spark_3_4:
name: Spark SQL Tests (Spark 3.4)
needs: changes
# Spark 3.4 is deprecated, so this is the one test job the merge queue does
# not run: it needs the `run-spark-3.4-tests` label on a pull request, or a
# manual workflow_dispatch.
#
# It is still in `required_checks.needs`, but be precise about what that
# buys. Adding the label fires a `labeled` event, and on that event the
# aggregator deliberately publishes the advisory `Required Checks (label
# run)` name instead of the required one (see its comment below), so a red
# 3.4 there leaves the commit's existing verdict alone. 3.4 only counts
# toward the required `Required Checks` on a pull_request event that is not
# `labeled` -- a push with the label already applied. With no queue run left
# to catch it, that push is the only thing that turns a 3.4 failure into a
# merge blocker.
if: needs.changes.outputs.spark_3_4 == 'true'
uses: ./.github/workflows/spark_sql_test_reusable.yml
with:
spark-short: '3.4'
spark-full: '3.4.3'
java: 17

spark_3_5:
name: Spark SQL Tests (Spark 3.5)
needs: changes
Expand Down Expand Up @@ -602,19 +574,6 @@ jobs:
spark-full: '4.2.0'
java: 17

iceberg_1_8:
name: Iceberg Spark SQL Tests (Iceberg 1.8)
needs: changes
# Nightly by default; PRs need the `run-iceberg-tests` label.
if: needs.changes.outputs.iceberg_1_8 == 'true'
uses: ./.github/workflows/iceberg_spark_test_reusable.yml
with:
iceberg-short: '1.8'
iceberg-full: '1.8.1'
spark-short: '3.4'
spark-full: '3.4.3'
java: 17

iceberg_1_9:
name: Iceberg Spark SQL Tests (Iceberg 1.9)
needs: changes
Expand Down Expand Up @@ -646,7 +605,7 @@ jobs:
needs: changes
# Queue-only by default; PRs need the `run-iceberg-tests` label. Iceberg
# 1.11 is our only Spark 4.1 Iceberg coverage, which is why it is the one
# Iceberg version the queue runs while 1.8/1.9/1.10 run nightly.
# Iceberg version the queue runs while 1.9/1.10 run nightly.
if: needs.changes.outputs.iceberg_1_11 == 'true'
uses: ./.github/workflows/iceberg_spark_test_reusable.yml
with:
Expand Down Expand Up @@ -706,15 +665,13 @@ jobs:
- pr_benchmark_check
- delta_build_gate
- pyarrow_udf_test
- spark_3_4
- spark_3_5
- delta_3_5
- delta_4_0
- delta_4_1
- spark_4_0
- spark_4_1
- spark_4_2
- iceberg_1_8
- iceberg_1_9
- iceberg_1_10
- iceberg_1_11
Expand Down
3 changes: 0 additions & 3 deletions .github/workflows/pr_build_linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,9 +164,6 @@ jobs:
strategy:
matrix:
profile:
- name: "Spark 3.4, JDK 17, Scala 2.12"
java_version: "17"
maven_opts: "-Pspark-3.4 -Pscala-2.12"
- name: "Spark 3.5, JDK 17, Scala 2.12"
java_version: "17"
maven_opts: "-Pspark-3.5 -Pscala-2.12"
Expand Down
3 changes: 1 addition & 2 deletions .github/workflows/publish_snapshot.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -241,11 +241,10 @@ jobs:
done
}

# One build per published variant. Spark 3.4 and 3.5 are published for
# One build per published variant. Spark 3.5 is published for
# Scala 2.12, which downstream benchmarks still use; 4.0, 4.1 and 4.2
# default to Scala 2.13.
variants=(
"-Pspark-3.4 -Pscala-2.12"
"-Pspark-3.5 -Pscala-2.12"
"-Pspark-4.0"
"-Pspark-4.1"
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/spark_sql_writer_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ jobs:
run: |
# Map each supported Spark minor version to its full version + JDK.
# Mirrors ci.yml's per-version reusable invocations (default-on PR
# versions only; 3.4 and 4.0 are label-gated and not offered here).
# versions only; 4.0 is label-gated and not offered here).
case "${{ inputs.spark-version }}" in
3.5) spark_full=3.5.9; java=17 ;;
4.1) spark_full=4.1.3; java=17 ;;
Expand Down
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ benefits of Comet's acceleration capabilities without disrupting your Spark appl

## Getting Started

Comet supports Apache Spark 3.4, 3.5, 4.0, 4.1, and 4.2. See the
Comet supports Apache Spark 3.5, 4.0, 4.1, and 4.2. See the
[installation guide](https://datafusion.apache.org/comet/user-guide/installation.html) for the detailed
version, Java, and Scala compatibility matrix.

Expand Down
Loading