diff --git a/.github/workflows/README.md b/.github/workflows/README.md index d50ef7cd58a..ce4b3f26f8e 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -26,8 +26,8 @@ ruleset in `.asf.yaml`. That splits CI into three tiers: and one Iceberg version, both the default profile's. - **Nightly tier** (`nightly`): the regression sweep of everything else, once a day against `main` as it stands. The Comet test suites against the other - four Spark profiles, Spark SQL on Spark 3.5, 4.0 and 4.2, Iceberg - 1.8/1.9/1.10, and the Delta contrib suites on Spark 3.5, 4.0 and 4.1. See + three Spark profiles, Spark SQL on Spark 3.5, 4.0 and 4.2, Iceberg + 1.9/1.10, and the Delta contrib suites on Spark 3.5, 4.0 and 4.1. See [Nightly tier](#nightly-tier) below for how a failure surfaces. Every queue-only and nightly job has a `run-*` label that opts a pull request @@ -35,16 +35,6 @@ into it early, listed in the diagram below. The Lint Java matrix compiles every Spark profile on every pull request, so a shim that fails to build is caught there; only the runtime suites wait for the queue or the nightly. -`spark_3_4` is in none of the tiers. Spark 3.4 is deprecated, so its Spark SQL -suite no longer gates a merge; it runs only when a pull request carries -`run-spark-3.4-tests`, or from a `workflow_dispatch`. Anyone who wants to -check a change against 3.4 can still do so, but note when that result starts -to block a merge. Adding the label fires a `labeled` event, and those runs -publish the advisory `Required Checks (label run)` name rather than the -required one, so a red 3.4 there changes nothing. It is the next push with -the label still applied that runs 3.4 under `Required Checks`, and with the -queue run gone that push is the only thing that makes a 3.4 failure blocking. - Heavy jobs have no `push` tier. The queue already tested the exact tree that lands, so re-running them on push to main would double the cost of every merge. Three jobs are still on `push`: `docs`, because it deploys to @@ -118,9 +108,8 @@ tiers partition the list and that the `pr` tier is exactly the default profile. delta_build_gate run-delta-build-gate pyarrow_udf_test run-pyarrow-udf-tests spark_4_1 run-spark-4.1-tests - label or dispatch only spark_4_1 sql_hive run-spark-4.1-hive-tests - ---------------------- iceberg_1_11 run-iceberg-tests - spark_3_4 run-spark-3.4-tests + spark_4_1 sql_hive run-spark-4.1-hive-tests + iceberg_1_11 run-iceberg-tests nightly tier, or PR with label ------------------------------ pr_build_linux run-all-spark-profiles @@ -128,7 +117,6 @@ tiers partition the list and that the `pr` tier is exactly the default profile. spark_3_5 run-spark-3.5-tests spark_4_0 run-spark-4.0-tests spark_4_2 run-spark-4.2-tests - iceberg_1_8 run-iceberg-tests iceberg_1_9 run-iceberg-tests iceberg_1_10 run-iceberg-tests delta_3_5 run-delta-tests @@ -176,11 +164,9 @@ tiers partition the list and that the `pr` tier is exactly the default profile. | `docs` | push to main, paths matched, **or** dispatch on `main` | `.asf.yaml`, `docs/**`, `docs.yaml` | | `spark_3_5` | nightly, **or** PR with `run-spark-3.5-tests` | Spark 3.5 sources | | `spark_4_1` | merge group, **or** PR with `run-spark-4.1-tests`; the `sql_hive` shards alone with `run-spark-4.1-hive-tests` | Spark 4.1 sources | -| `spark_3_4` | PR with `run-spark-3.4-tests`, or dispatch | Spark 3.4 sources | | `spark_4_0` | nightly, **or** PR with `run-spark-4.0-tests` | Spark 4.0 sources | | `spark_4_2` | nightly, **or** PR with `run-spark-4.2-tests` | Spark 4.2 sources | | `iceberg_1_11` | merge group, **or** PR with `run-iceberg-tests` | Iceberg sources | -| `iceberg_1_8` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources | | `iceberg_1_9` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources | | `iceberg_1_10` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources | | `required_checks` | always, after every job above except `docs` | none (always runs) | @@ -194,8 +180,8 @@ safe to make a required check. A pull request whose base is a release branch (`branch-N.M`) is the one exception to the table: it runs every job the table puts in the PR, merge group or nightly tier, because a release branch has no merge queue and no -nightly to run the last two later. `docs` stays push-only and `spark_3_4` -still needs its label, so a `labeled` run there adds `spark_3_4` or nothing. +nightly to run the last two later. `docs` stays push-only, and a `labeled` +run there adds nothing. `changes` hands the base branch to `compute-changes.py` as `PR_BASE_REF`. `check-ci-config.py` pins that wiring, because a dropped variable reads as an empty string and would route the pull request as if it targeted `main`. @@ -289,7 +275,7 @@ Things that follow from that, and are easy to break: ### Label events `ci_label.yml` fires on `pull_request.types: [labeled]` and calls `ci.yml` -through `workflow_call`, so applying `run-spark-3.4-tests`, +through `workflow_call`, so applying `run-spark-3.5-tests`, `run-spark-4.0-tests` or `run-iceberg-tests` starts the job that label gates without needing a new push. GitHub cannot filter a `pull_request` trigger by label name, so **every** label added to a PR starts a run, including labels @@ -393,17 +379,16 @@ umbrella doesn't watch, or operate independently of the rest of CI: ## Reusable workflows (called by `ci.yml`) -| File | Called from `ci.yml` job(s) | -| --------------------------------- | --------------------------------------------------------------- | -| `pr_build_linux.yml` | `pr_build_linux` | -| `pr_build_macos.yml` | `pr_build_macos` | -| `pr_benchmark_check.yml` | `pr_benchmark_check` | -| `delta_build_gate.yml` | `delta_build_gate` | -| `delta_contrib_test.yml` | `delta_3_5`, `delta_4_0`, `delta_4_1` | -| `pyarrow_udf_test.yml` | `pyarrow_udf_test` | -| `docs.yaml` | `docs` | -| `spark_sql_test_reusable.yml` | `spark_3_4`, `spark_3_5`, `spark_4_0`, `spark_4_1`, `spark_4_2` | -| `iceberg_spark_test_reusable.yml` | `iceberg_1_8`, `iceberg_1_9`, `iceberg_1_10`, `iceberg_1_11` | +| File | Called from `ci.yml` job(s) | +| ----------------------------- | ------------------------------------- | +| `pr_build_linux.yml` | `pr_build_linux` | +| `pr_build_macos.yml` | `pr_build_macos` | +| `pr_benchmark_check.yml` | `pr_benchmark_check` | +| `delta_build_gate.yml` | `delta_build_gate` | +| `delta_contrib_test.yml` | `delta_3_5`, `delta_4_0`, `delta_4_1` | +| `pyarrow_udf_test.yml` | `pyarrow_udf_test` | +| `docs.yaml` | `docs` | +| `spark_sql_test_reusable.yml` | `iceberg_spark_test_reusable.yml` | ## Changing what runs when diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0fc66aba7f4..51d75a1b864 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -32,7 +32,7 @@ # the default profile's. # schedule the nightly regression sweep of everything else: the Linux # test matrix against the other Spark profiles, Spark SQL on -# Spark 3.5, 4.0 and 4.2, and Iceberg 1.8/1.9/1.10, routed by +# Spark 3.5, 4.0 and 4.2, and Iceberg 1.9/1.10, routed by # the same path filters over what landed on main since the last # green nightly. A failure opens (or comments on) an issue # labelled `ci-nightly-failure`. @@ -44,9 +44,6 @@ # on the merge result. Opt-in `run-*` suites, pull requests from a branch of this # repository and pull requests to a release branch are unaffected. # -# Spark 3.4 is deprecated and sits outside all three tiers: it runs only when -# a pull request carries `run-spark-3.4-tests`, or from a manual dispatch. -# # Release branches (`branch-N.M`) get neither the queue nor the nightly: the # queue ruleset covers only the default branch, `push` below is main-only, and # GitHub fires `schedule` only on the default branch. So a pull request that @@ -290,13 +287,11 @@ jobs: delta_gate: ${{ steps.compute.outputs.delta_gate }} pyarrow_udf: ${{ steps.compute.outputs.pyarrow_udf }} docs: ${{ steps.compute.outputs.docs }} - spark_3_4: ${{ steps.compute.outputs.spark_3_4 }} spark_3_5: ${{ steps.compute.outputs.spark_3_5 }} spark_4_0: ${{ steps.compute.outputs.spark_4_0 }} spark_4_1: ${{ steps.compute.outputs.spark_4_1 }} spark_4_1_hive: ${{ steps.compute.outputs.spark_4_1_hive }} spark_4_2: ${{ steps.compute.outputs.spark_4_2 }} - iceberg_1_8: ${{ steps.compute.outputs.iceberg_1_8 }} iceberg_1_9: ${{ steps.compute.outputs.iceberg_1_9 }} iceberg_1_10: ${{ steps.compute.outputs.iceberg_1_10 }} iceberg_1_11: ${{ steps.compute.outputs.iceberg_1_11 }} @@ -485,29 +480,6 @@ jobs: contents: write uses: ./.github/workflows/docs.yaml - spark_3_4: - name: Spark SQL Tests (Spark 3.4) - needs: changes - # Spark 3.4 is deprecated, so this is the one test job the merge queue does - # not run: it needs the `run-spark-3.4-tests` label on a pull request, or a - # manual workflow_dispatch. - # - # It is still in `required_checks.needs`, but be precise about what that - # buys. Adding the label fires a `labeled` event, and on that event the - # aggregator deliberately publishes the advisory `Required Checks (label - # run)` name instead of the required one (see its comment below), so a red - # 3.4 there leaves the commit's existing verdict alone. 3.4 only counts - # toward the required `Required Checks` on a pull_request event that is not - # `labeled` -- a push with the label already applied. With no queue run left - # to catch it, that push is the only thing that turns a 3.4 failure into a - # merge blocker. - if: needs.changes.outputs.spark_3_4 == 'true' - uses: ./.github/workflows/spark_sql_test_reusable.yml - with: - spark-short: '3.4' - spark-full: '3.4.3' - java: 17 - spark_3_5: name: Spark SQL Tests (Spark 3.5) needs: changes @@ -602,19 +574,6 @@ jobs: spark-full: '4.2.0' java: 17 - iceberg_1_8: - name: Iceberg Spark SQL Tests (Iceberg 1.8) - needs: changes - # Nightly by default; PRs need the `run-iceberg-tests` label. - if: needs.changes.outputs.iceberg_1_8 == 'true' - uses: ./.github/workflows/iceberg_spark_test_reusable.yml - with: - iceberg-short: '1.8' - iceberg-full: '1.8.1' - spark-short: '3.4' - spark-full: '3.4.3' - java: 17 - iceberg_1_9: name: Iceberg Spark SQL Tests (Iceberg 1.9) needs: changes @@ -646,7 +605,7 @@ jobs: needs: changes # Queue-only by default; PRs need the `run-iceberg-tests` label. Iceberg # 1.11 is our only Spark 4.1 Iceberg coverage, which is why it is the one - # Iceberg version the queue runs while 1.8/1.9/1.10 run nightly. + # Iceberg version the queue runs while 1.9/1.10 run nightly. if: needs.changes.outputs.iceberg_1_11 == 'true' uses: ./.github/workflows/iceberg_spark_test_reusable.yml with: @@ -706,7 +665,6 @@ jobs: - pr_benchmark_check - delta_build_gate - pyarrow_udf_test - - spark_3_4 - spark_3_5 - delta_3_5 - delta_4_0 @@ -714,7 +672,6 @@ jobs: - spark_4_0 - spark_4_1 - spark_4_2 - - iceberg_1_8 - iceberg_1_9 - iceberg_1_10 - iceberg_1_11 diff --git a/.github/workflows/pr_build_linux.yml b/.github/workflows/pr_build_linux.yml index 7f70f81122c..ef593ff4ed5 100644 --- a/.github/workflows/pr_build_linux.yml +++ b/.github/workflows/pr_build_linux.yml @@ -164,9 +164,6 @@ jobs: strategy: matrix: profile: - - name: "Spark 3.4, JDK 17, Scala 2.12" - java_version: "17" - maven_opts: "-Pspark-3.4 -Pscala-2.12" - name: "Spark 3.5, JDK 17, Scala 2.12" java_version: "17" maven_opts: "-Pspark-3.5 -Pscala-2.12" diff --git a/.github/workflows/publish_snapshot.yml b/.github/workflows/publish_snapshot.yml index e5d8eeea1d8..667a344870b 100644 --- a/.github/workflows/publish_snapshot.yml +++ b/.github/workflows/publish_snapshot.yml @@ -241,11 +241,10 @@ jobs: done } - # One build per published variant. Spark 3.4 and 3.5 are published for + # One build per published variant. Spark 3.5 is published for # Scala 2.12, which downstream benchmarks still use; 4.0, 4.1 and 4.2 # default to Scala 2.13. variants=( - "-Pspark-3.4 -Pscala-2.12" "-Pspark-3.5 -Pscala-2.12" "-Pspark-4.0" "-Pspark-4.1" diff --git a/.github/workflows/spark_sql_writer_tests.yml b/.github/workflows/spark_sql_writer_tests.yml index 6b5fc7f6d65..6b729cc488b 100644 --- a/.github/workflows/spark_sql_writer_tests.yml +++ b/.github/workflows/spark_sql_writer_tests.yml @@ -68,7 +68,7 @@ jobs: run: | # Map each supported Spark minor version to its full version + JDK. # Mirrors ci.yml's per-version reusable invocations (default-on PR - # versions only; 3.4 and 4.0 are label-gated and not offered here). + # versions only; 4.0 is label-gated and not offered here). case "${{ inputs.spark-version }}" in 3.5) spark_full=3.5.9; java=17 ;; 4.1) spark_full=4.1.3; java=17 ;; diff --git a/README.md b/README.md index 63827189dc0..5d666f20cfb 100644 --- a/README.md +++ b/README.md @@ -86,7 +86,7 @@ benefits of Comet's acceleration capabilities without disrupting your Spark appl ## Getting Started -Comet supports Apache Spark 3.4, 3.5, 4.0, 4.1, and 4.2. See the +Comet supports Apache Spark 3.5, 4.0, 4.1, and 4.2. See the [installation guide](https://datafusion.apache.org/comet/user-guide/installation.html) for the detailed version, Java, and Scala compatibility matrix. diff --git a/dev/ci/check-ci-config.py b/dev/ci/check-ci-config.py index 9823869bf8c..d4b87f5ceaf 100644 --- a/dev/ci/check-ci-config.py +++ b/dev/ci/check-ci-config.py @@ -120,13 +120,11 @@ "build_linux_all_profiles", "build_macos", "build_macos_full", - "spark_3_4", "spark_3_5", "spark_4_0", "spark_4_1", "spark_4_1_hive", "spark_4_2", - "iceberg_1_8", "iceberg_1_9", "iceberg_1_10", "iceberg_1_11", @@ -230,7 +228,6 @@ "spark_3_5", "spark_4_0", "spark_4_2", - "iceberg_1_8", "iceberg_1_9", "iceberg_1_10", "delta_3_5", @@ -238,18 +235,12 @@ "delta_4_1", "build_linux_all_profiles", } -# Spark 3.4 is deprecated and sits outside the queue and nightly tiers -# entirely: a label on a pull request, or a workflow_dispatch, and nothing -# else. Keeping it in its own set is what makes the `merge_group` and -# `schedule` cases below assert its absence rather than quietly accept it -# coming back. -SPARK_DEPRECATED = {"spark_3_4"} # One label opts a pull request into every Iceberg version, whichever tier # each sits in. -ICEBERG_OPT_IN = {"iceberg_1_8", "iceberg_1_9", "iceberg_1_10", "iceberg_1_11"} +ICEBERG_OPT_IN = {"iceberg_1_9", "iceberg_1_10", "iceberg_1_11"} # Likewise one label for every Delta profile. DELTA_OPT_IN = {"delta_3_5", "delta_4_0", "delta_4_1"} -ALL_JOBS = QUEUE_TIER | NIGHTLY_TIER | SPARK_DEPRECATED | {"docs"} +ALL_JOBS = QUEUE_TIER | NIGHTLY_TIER | {"docs"} assert not QUEUE_TIER & NIGHTLY_TIER, "a job is queue or nightly, never both" # The repository whose queue gates main; check_event_policy asserts this matches @@ -277,9 +268,8 @@ def fork_pr(action="synchronize", label=None, labels=(), base="main", repository # A manual run may exercise anything. ({"name": "workflow_dispatch"}, ALL_JOBS), # The merge queue is the authoritative gate: everything except the site - # deploy, which can only run once the commit is actually on main, the - # nightly tier, and the deprecated Spark 3.4 suite, which no longer gates - # a merge. + # deploy, which can only run once the commit is actually on main, and the + # nightly tier. ({"name": "merge_group"}, QUEUE_TIER), # The scheduled run is the nightly tier and nothing else. The queue already # ran everything in QUEUE_TIER against the tree that is now main, so a @@ -413,23 +403,6 @@ def fork_pr(action="synchronize", label=None, labels=(), base="main", repository }, {"spark_4_1_hive"}, ), - # An opt-in label present on a pushed commit adds just that suite. For the - # deprecated Spark 3.4 suite the label is the *only* way it ever runs on a - # pull request or the queue, so this case and the `labeled` one below are - # what keep it reachable at all. - ( - {"name": "pull_request", "action": "synchronize", "labels": ["run-spark-3.4-tests"]}, - PR_TIER | {"spark_3_4"}, - ), - ( - { - "name": "pull_request", - "action": "labeled", - "label": "run-spark-3.4-tests", - "labels": ["run-spark-3.4-tests"], - }, - {"spark_3_4"}, - ), # Spark 4.2 runs nightly, and its label brings that run forward onto a pull # request, the same shape as 3.5 and 4.0. It pulls in no other Spark version. ( @@ -481,14 +454,13 @@ def fork_pr(action="synchronize", label=None, labels=(), base="main", repository "name": "pull_request", "action": "labeled", "label": "dependencies", - "labels": ["dependencies", "run-spark-3.4-tests"], + "labels": ["dependencies", "run-spark-3.5-tests"], }, set(), ), # A pull request against a release branch runs the queue and nightly tiers # as well, because a release branch has no queue and no nightly to run them - # later. The site deploy still never runs from a pull request, and the - # deprecated Spark 3.4 suite still waits for its label. + # later. The site deploy still never runs from a pull request. ( {"name": "pull_request", "action": "opened", "labels": [], "base": "branch-1.1"}, QUEUE_TIER | NIGHTLY_TIER, @@ -497,27 +469,6 @@ def fork_pr(action="synchronize", label=None, labels=(), base="main", repository {"name": "pull_request", "action": "synchronize", "labels": [], "base": "branch-2.0"}, QUEUE_TIER | NIGHTLY_TIER, ), - ( - { - "name": "pull_request", - "action": "synchronize", - "labels": ["run-spark-3.4-tests"], - "base": "branch-1.1", - }, - QUEUE_TIER | NIGHTLY_TIER | SPARK_DEPRECATED, - ), - # There the commit run already covered everything a label gates except - # Spark 3.4, so a `labeled` run adds Spark 3.4 or nothing. - ( - { - "name": "pull_request", - "action": "labeled", - "label": "run-spark-3.4-tests", - "labels": ["run-spark-3.4-tests"], - "base": "branch-1.1", - }, - SPARK_DEPRECATED, - ), ( { "name": "pull_request", diff --git a/dev/ci/compute-changes.py b/dev/ci/compute-changes.py index f07abdb8329..4dc6eca5003 100644 --- a/dev/ci/compute-changes.py +++ b/dev/ci/compute-changes.py @@ -198,34 +198,6 @@ "spark/src/main/scala/org/apache/comet/expressions/**", "spark/src/main/spark-*/**", ], - "spark_3_4": [ - "native/**/src/**", - "native/**/Cargo.toml", - "native/Cargo.lock", - "common/src/main/**", - "common/pom.xml", - "spark/src/main/**", - "!spark/src/main/spark-3.5/**", - "!spark/src/main/spark-4.0/**", - "!spark/src/main/spark-4.1/**", - "!spark/src/main/spark-4.2/**", - "!spark/src/main/spark-4.x/**", - "!spark/src/main/scala/org/apache/comet/GenerateDocs.scala", - "spark/pom.xml", - "dev/diffs/3.4.3.diff", - "pom.xml", - "rust-toolchain.toml", - ".github/workflows/ci.yml", - ".github/workflows/spark_sql_test_reusable.yml", - "dev/ci/spark-sql-modules.py", - ".github/actions/setup-builder/**", - ".github/actions/setup-spark-builder/**", - ".github/actions/upload-artifact-retry/**", - ".github/actions/download-artifact-retry/**", - ".github/actions/maven-bootstrap/**", - ".mvn/**", - "mvnw", - ], "spark_3_5": [ "native/**/src/**", "native/**/Cargo.toml", @@ -343,31 +315,6 @@ # ci.yml folds the two outputs into the reusable workflow's `modules` # input. Populated below, after the dict, so the two lists cannot drift. "spark_4_1_hive": [], - "iceberg_1_8": [ - "native/**/src/**", - "native/**/Cargo.toml", - "native/Cargo.lock", - "common/src/main/**", - "common/pom.xml", - "spark/src/main/**", - "!spark/src/main/scala/org/apache/comet/GenerateDocs.scala", - "spark/pom.xml", - "dev/diffs/iceberg/**", - "pom.xml", - "rust-toolchain.toml", - ".github/workflows/ci.yml", - ".github/workflows/iceberg_spark_test_reusable.yml", - ".github/actions/setup-builder/**", - ".github/actions/setup-iceberg-builder/**", - "dev/ci/iceberg-test-shards.gradle", - "dev/ci/check-iceberg-shards.py", - "dev/ci/test-iceberg-shards.py", - ".github/actions/upload-artifact-retry/**", - ".github/actions/download-artifact-retry/**", - ".github/actions/maven-bootstrap/**", - ".mvn/**", - "mvnw", - ], "iceberg_1_9": [ "native/**/src/**", "native/**/Cargo.toml", @@ -514,8 +461,8 @@ # `schedule` only on the default branch. A tier there would not defer a suite, # it would drop it. And the tiers exist because of main's volume, which a # release branch does not have: branch-1.0 took 28 pull requests in its first -# two months. "push" jobs (the site deploy) and label-only jobs (Spark 3.4) -# keep their rules on a release branch. +# two months. "push" jobs (the site deploy) and any label-only job keep their +# rules on a release branch. POLICY = { # The one test job that also runs on push to main, and only because of # actions/cache scoping: a pull request can restore caches saved on its @@ -573,11 +520,6 @@ # docs deploys to asf-site, so it must not run from a pull request or from # the queue's throwaway branch. "docs": ["push"], - # Spark 3.4 is deprecated, so it is the one test job outside the queue - # tier: a failure there no longer blocks a merge. It stays runnable on - # demand -- the label on a pull request, or a workflow_dispatch -- so - # anyone who wants to check a change against 3.4 still can. - "spark_3_4": ["label:run-spark-3.4-tests"], # Spark 4.1 is the default build profile and the one Spark SQL suite the # queue runs; 3.5, 4.0 and 4.2 run nightly, or on a pull request with # their label. @@ -607,9 +549,8 @@ # until someone thinks to ask for it. "spark_4_2": ["nightly", "label:run-spark-4.2-tests"], # Same shape for Iceberg: 1.11 is the only Spark 4.1 coverage, so it is - # the one Iceberg version the queue runs; the three older versions run - # nightly. One label opts a pull request into all four. - "iceberg_1_8": ["nightly", "label:run-iceberg-tests"], + # the one Iceberg version the queue runs; the older versions run + # nightly. One label opts a pull request into all three. "iceberg_1_9": ["nightly", "label:run-iceberg-tests"], "iceberg_1_10": ["nightly", "label:run-iceberg-tests"], "iceberg_1_11": ["queue", "label:run-iceberg-tests"], @@ -727,7 +668,7 @@ def release_branch_allows(job, event): automatic = any(tier in tiers for tier in RELEASE_BRANCH_TIERS) # The opened/synchronize run already ran every automatic job at this # commit, so a `labeled` run adds only a job that nothing else runs there, - # which leaves the label-only Spark 3.4 suite. + # which leaves only the label-only jobs, of which there are currently none. if event.get("action") == "labeled": return not automatic and event.get("label") in gates return automatic or any(label in event.get("labels", []) for label in gates) diff --git a/dev/ci/linux-test-profiles.py b/dev/ci/linux-test-profiles.py index 741ef7d3b1e..011c518df12 100644 --- a/dev/ci/linux-test-profiles.py +++ b/dev/ci/linux-test-profiles.py @@ -46,7 +46,7 @@ # `run-all-spark-profiles` label). The goal of the list is coverage of every # Java, Scala and Spark version without testing every combination. PROFILES = [ - {"name": "Spark 3.4, JDK 17, Scala 2.12", "java_version": "17", "maven_opts": "-Pspark-3.4 -Pscala-2.12", "tier": "nightly"}, + {"name": "Spark 3.5, JDK 17, Scala 2.12", "java_version": "17", "maven_opts": "-Pspark-3.5 -Pscala-2.12", "tier": "nightly"}, {"name": "Spark 3.5, JDK 17, Scala 2.13", "java_version": "17", "maven_opts": "-Pspark-3.5 -Pscala-2.13", "tier": "nightly"}, {"name": "Spark 4.0, JDK 21", "java_version": "21", "maven_opts": "-Pspark-4.0", "tier": "nightly"}, # The default build profile, and the one a contributor builds locally. diff --git a/dev/generate-release-docs.sh b/dev/generate-release-docs.sh index 8ba19a875b7..2a214d6b683 100755 --- a/dev/generate-release-docs.sh +++ b/dev/generate-release-docs.sh @@ -57,7 +57,7 @@ TEMPLATE_DIR="${COMPAT_EXPR_DIR}/_category_template" # and a source-tree subdirectory under compatibility/expressions/. # Every run also rewrites the shared configs.md and expressions.md pages, so the last # entry decides their content. Keep the default build profile (spark-4.1) last. -SPARK_PROFILES=(spark-3.4 spark-3.5 spark-4.0 spark-4.2 spark-4.1) +SPARK_PROFILES=(spark-3.5 spark-4.0 spark-4.2 spark-4.1) echo "Compiling and generating documentation content..." for profile in "${SPARK_PROFILES[@]}"; do diff --git a/dev/regenerate-golden-files.sh b/dev/regenerate-golden-files.sh index 1feb88e60d5..40d3d384db0 100755 --- a/dev/regenerate-golden-files.sh +++ b/dev/regenerate-golden-files.sh @@ -22,7 +22,7 @@ # Usage: ./dev/regenerate-golden-files.sh [--spark-version ] # # Options: -# --spark-version Only regenerate for specified Spark version (3.4, 3.5, 4.0, 4.1, or 4.2) +# --spark-version Only regenerate for specified Spark version (3.5, 4.0, 4.1, or 4.2) # If not specified, regenerates for all versions. # # Examples: @@ -77,8 +77,9 @@ build_native() { # Regenerate golden files for a specific Spark version. CometPlanStabilitySuite's # afterAll() prunes any query directory whose contents match what its fallback # chain resolves to, so each version-suffixed directory ends up with only the -# queries whose plans diverge from the previous tier. The base (3.4) profile has -# no fallback chain, so the prune is a no-op there. +# queries whose plans diverge from the previous tier. The shared base tier was +# generated by the Spark 3.4 profile, which no longer exists, so this script does not +# regenerate it. regenerate_golden_files() { local spark_version=$1 @@ -113,7 +114,7 @@ main() { echo "Usage: $0 [--spark-version ]" echo "" echo "Options:" - echo " --spark-version Only regenerate for specified Spark version (3.4, 3.5, 4.0, 4.1, or 4.2)" + echo " --spark-version Only regenerate for specified Spark version (3.5, 4.0, 4.1, or 4.2)" echo " If not specified, regenerates for all versions." exit 0 ;; @@ -127,9 +128,9 @@ main() { # Validate target version if specified if [ -n "$target_version" ]; then - if [[ ! "$target_version" =~ ^(3\.4|3\.5|4\.0|4\.1|4\.2)$ ]]; then + if [[ ! "$target_version" =~ ^(3\.5|4\.0|4\.1|4\.2)$ ]]; then echo "[ERROR] Invalid Spark version: $target_version" - echo "[ERROR] Supported versions: 3.4, 3.5, 4.0, 4.1, 4.2" + echo "[ERROR] Supported versions: 3.5, 4.0, 4.1, 4.2" exit 1 fi fi @@ -149,7 +150,7 @@ main() { if [ -n "$target_version" ]; then versions=("$target_version") else - versions=("3.4" "3.5" "4.0" "4.1" "4.2") + versions=("3.5" "4.0" "4.1" "4.2") fi # Regenerate for each version diff --git a/dev/release/build-release-comet.sh b/dev/release/build-release-comet.sh index fcf181ba54e..36834dba3dd 100755 --- a/dev/release/build-release-comet.sh +++ b/dev/release/build-release-comet.sh @@ -237,8 +237,6 @@ pushd "$BUILD_DIR" LOCAL_REPO=$(mktemp -d /tmp/comet-staging-repo-XXXXX) -./mvnw "-Dmaven.repo.local=${LOCAL_REPO}" -P spark-3.4 -P scala-2.12 -DskipTests install -./mvnw "-Dmaven.repo.local=${LOCAL_REPO}" -P spark-3.4 -P scala-2.13 -DskipTests install ./mvnw "-Dmaven.repo.local=${LOCAL_REPO}" -P spark-3.5 -P scala-2.12 -DskipTests install ./mvnw "-Dmaven.repo.local=${LOCAL_REPO}" -P spark-3.5 -P scala-2.13 -DskipTests install # The spark-4.x profiles pin their own Scala 2.13.x patch versions to match the diff --git a/dev/release/verify-release-candidate.sh b/dev/release/verify-release-candidate.sh index df150835ee3..9f7e49aa3e3 100755 --- a/dev/release/verify-release-candidate.sh +++ b/dev/release/verify-release-candidate.sh @@ -111,7 +111,7 @@ test_source_distribution() { RUSTFLAGS="-Ctarget-cpu=native" cargo build --release popd # test with the latest supported version of Spark - ./mvnw verify -Prelease -DskipTests -P"spark-3.4" -Dmaven.gitcommitid.skip=true + ./mvnw verify -Prelease -DskipTests -P"spark-3.5" -Dmaven.gitcommitid.skip=true } TEST_SUCCESS=no diff --git a/docs/build.sh b/docs/build.sh index efaaa0f94ab..fa87486331c 100755 --- a/docs/build.sh +++ b/docs/build.sh @@ -53,7 +53,7 @@ TEMPLATE_DIR="${COMPAT_EXPR_DIR}/_category_template" # docs/source/user-guide/latest/compatibility/expressions/. # Every run also rewrites the shared configs.md and expressions.md pages, so the last # entry decides their content. Keep the default build profile (spark-4.1) last. -SPARK_PROFILES=(spark-3.4 spark-3.5 spark-4.0 spark-4.2 spark-4.1) +SPARK_PROFILES=(spark-3.5 spark-4.0 spark-4.2 spark-4.1) for profile in "${SPARK_PROFILES[@]}"; do echo "Generating compatibility pages for ${profile}..." diff --git a/docs/source/about/faq.md b/docs/source/about/faq.md index 29c39910ed1..a06435e2c3b 100644 --- a/docs/source/about/faq.md +++ b/docs/source/about/faq.md @@ -52,7 +52,7 @@ not documented, please [file an issue](https://github.com/apache/datafusion-come ### Which Spark and Java versions does Comet support? -Spark 3.4, 3.5, 4.0, 4.1, and 4.2, on Java 17 or later. Spark 3.4 is deprecated. The +Spark 3.5, 4.0, 4.1, and 4.2, on Java 17 or later. The [installation guide](../user-guide/latest/installation.md#supported-spark-versions) lists the exact Spark, Java, and Scala versions, and the [Spark Version Compatibility](../user-guide/latest/compatibility/spark-versions.md) page lists known diff --git a/docs/source/about/gluten_comparison.md b/docs/source/about/gluten_comparison.md index 9e129ec30bb..80025b6deb3 100644 --- a/docs/source/about/gluten_comparison.md +++ b/docs/source/about/gluten_comparison.md @@ -84,7 +84,7 @@ the community behind each engine alongside the technical differences. Both projects target a similar set of Spark releases. -Comet supports Spark 3.4, 3.5, 4.0, 4.1, and 4.2 in production builds. See the +Comet supports Spark 3.5, 4.0, 4.1, and 4.2 in production builds. See the [Spark version compatibility guide] for the exact patch versions and JDK/Scala combinations. [Spark version compatibility guide]: /user-guide/latest/compatibility/spark-versions.md diff --git a/docs/source/contributor-guide/ci.md b/docs/source/contributor-guide/ci.md index 9088ddbcf22..1be7a134128 100644 --- a/docs/source/contributor-guide/ci.md +++ b/docs/source/contributor-guide/ci.md @@ -37,7 +37,6 @@ flowchart LR PR([pull request]) --> PRTIER PR -. with label .-> QUEUE PR -. with label .-> NIGHTLY - PR -. with label .-> S34 MQ([merge queue]) --> PRTIER MQ --> QUEUE CRON([schedule
06:00 UTC]) --> NIGHTLY @@ -45,8 +44,7 @@ flowchart LR PRTIER["PR tier
Linux build, lint, Rust tests
TPC-H / TPC-DS
Comet suites, Spark 4.1"] QUEUE["Queue tier, on top of the PR tier
Spark SQL, Spark 4.1
Iceberg 1.11
macOS build and Comet suites
Benchmark check, Delta gate
PyArrow UDF, Spark 4.0 / 4.1 / 4.2"] - NIGHTLY["Nightly tier
Comet suites, Spark 3.4 / 3.5 / 4.0 / 4.2
Spark SQL, Spark 3.5 / 4.0 / 4.2
Iceberg 1.8 / 1.9 / 1.10
Delta contrib, Spark 3.5 / 4.0 / 4.1"] - S34["Neither tier
Spark SQL, Spark 3.4"] + NIGHTLY["Nightly tier
Comet suites, Spark 3.5 / 4.0 / 4.2
Spark SQL, Spark 3.5 / 4.0 / 4.2
Iceberg 1.9 / 1.10
Delta contrib, Spark 3.5 / 4.0 / 4.1"] CACHE["Cache-refresh-only mode
the three cache-writing jobs, plus Lint"] ``` @@ -63,11 +61,10 @@ Suite by suite: | Benchmark compile and lint check | with label | yes | no | | Delta contrib build gate | with label | yes | no | | PyArrow UDF tests, Spark 4.0 / 4.1 / 4.2 | with label | yes | no | -| Comet test suites, Spark 3.4 / 3.5 / 4.0 / 4.2 | with label | no | yes | +| Comet test suites, Spark 3.5 / 4.0 / 4.2 | with label | no | yes | | Spark SQL tests, Spark 3.5 / 4.0 / 4.2 | with label | no | yes | -| Iceberg Spark SQL tests, Iceberg 1.8 / 1.9 / 1.10 | with label | no | yes | +| Iceberg Spark SQL tests, Iceberg 1.9 / 1.10 | with label | no | yes | | Delta contrib tests, Spark 3.5 / 4.0 / 4.1 | with label | no | yes | -| Spark SQL tests, Spark 3.4 | with label | no | no | The **PR tier** is the fast feedback loop while a change is being iterated on. The **queue tier** is the authoritative gate: everything the PR tier runs plus the suites that must pass before @@ -90,16 +87,6 @@ runner-minutes a push and about 40. If you add a job to `pr_build_linux.yml`, gi `if: ${{ !inputs.cache-refresh-only }}` unless it writes a cache that `main` needs; `dev/ci/check-ci-config.py` fails the build if you forget. -Spark 3.4 is the one suite in neither tier. [Spark 3.4 support is deprecated](../user-guide/latest/compatibility/spark-versions.md#spark-34), -so its Spark SQL suite no longer gates a merge. It remains available on demand: apply the -`run-spark-3.4-tests` label to run it against a pull request, or trigger `ci.yml` from the Actions -page with **Run workflow**, which runs every suite regardless of tier. A contributor touching Spark -3.4 code can still get a verdict before merging, but applying the label is only half of that: the -run it starts is advisory. Label the pull request and then push, and Spark 3.4 joins the required -verdict like any other suite. See -[Opting a pull request into a suite the PR tier skips](#opting-a-pull-request-into-a-suite-the-pr-tier-skips) -for the mechanics. - For a pull request from a fork, the pull request column's "yes" rows run on your fork rather than here; see [Pull requests from forks](#pull-requests-from-forks). @@ -184,14 +171,13 @@ Each suite outside the PR tier has a label that runs it on a pull request: | Label | Runs | | -------------------------- | ----------------------------------------------------- | | `run-macos-tests` | macOS build and Comet test suites | -| `run-all-spark-profiles` | Comet test suites against Spark 3.4 / 3.5 / 4.0 / 4.2 | +| `run-all-spark-profiles` | Comet test suites against Spark 3.5 / 4.0 / 4.2 | | `run-benchmark-check` | Benchmark compile and lint check | | `run-delta-build-gate` | Delta contrib build gate | | `run-pyarrow-udf-tests` | PyArrow UDF tests against Spark 4.0/4.1/4.2 | | `run-delta-tests` | Delta contrib tests against Spark 3.5, 4.0 and 4.1 | | `run-spark-4.1-tests` | Spark SQL tests against Spark 4.1, every module | | `run-spark-4.1-hive-tests` | Spark SQL tests against Spark 4.1, sql_hive only | -| `run-spark-3.4-tests` | Spark SQL tests against Spark 3.4 | | `run-spark-3.5-tests` | Spark SQL tests against Spark 3.5 | | `run-spark-4.0-tests` | Spark SQL tests against Spark 4.0 | | `run-spark-4.2-tests` | Spark SQL tests against Spark 4.2 | @@ -199,8 +185,7 @@ Each suite outside the PR tier has a label that runs it on a pull request: For a queue-tier suite the label only brings the run forward; the queue would have run it anyway before the change landed. For a nightly-tier suite the label is what gets a verdict _before_ the -change lands: without it, the first run is the nightly after the merge. For Spark 3.4 the label is -the only way the suite runs at all. +change lands: without it, the first run is the nightly after the merge. Apply a label from the pull request sidebar, or from the command line: @@ -216,12 +201,12 @@ read alongside the commit run without replacing it. For a queue-tier suite, that separate name costs nothing: the merge queue runs the suite again before the change lands, so a failure a label run surfaced still blocks the merge later. A -nightly-tier suite and Spark 3.4 have no queue run behind them. A red `Required Checks (label run)` +nightly-tier suite has no queue run behind it. A red `Required Checks (label run)` leaves an earlier green `Required Checks` in place and the pull request mergeable, so treat a label run as feedback to read, not as a gate. For one of those suites to count toward the required verdict the label has to already be on the pull request when a commit is pushed, which is what the next push gives you, since the label stays applied. A nightly-tier regression that does slip -through is caught by the next nightly run and filed as an issue; a Spark 3.4 one is not. +through is caught by the next nightly run and filed as an issue. The label stays on the pull request, so every later push runs the suite as part of the normal PR run. Remove the label once it has served its purpose. To re-run the suite at the same commit, @@ -233,7 +218,7 @@ does not cover. Some examples: - a change to the serde, the planner, or a native operator, where the Comet test suites pass but Spark's own SQL suite is the thing that would catch a behavior difference; `run-spark-4.1-tests` runs it against the default profile -- code under `spark/src/main/spark-3.4/`, `spark-3.5/`, `spark-4.0/` or the shared `spark-3.x/` +- code under `spark/src/main/spark-3.5/`, `spark-4.0/` or the shared `spark-3.x/` directory, or any change to `CometExprShim` and friends; `run-all-spark-profiles` runs the Comet test suites against every Spark version rather than 4.1 alone (the Lint Java matrix already compiles every Spark profile on every pull request, so this is for runtime differences) @@ -248,11 +233,6 @@ does not cover. Some examples: - a change to the benchmark sources under `spark/src/test/scala/org/apache/spark/sql/benchmark` or to `native/*/benches` -Reach for `run-spark-3.4-tests` deliberately, since nothing else will run it: a change to -`spark/src/main/spark-3.4/`, to `dev/diffs/3.4.3.diff`, or to shared code whose Spark 3.4 behavior -you are unsure of. Without the label, a Spark 3.4 regression will not be caught before the change -lands. - Labels that gate nothing, such as `dependencies` or the type labels, also start a run. That run executes nothing and finishes in a minute. It is a consequence of GitHub firing the `labeled` event for every label; see the workflows README for why it is handled this way. @@ -318,7 +298,7 @@ re-armed automatically. `ci.yml` also runs on a schedule, at 06:00 UTC every day, against what landed on `main` since the last successful scheduled run. That run executes only the nightly tier: the Comet test suites against the Spark profiles other than 4.1, the Spark SQL suites for Spark 3.5, 4.0 and 4.2, the Iceberg suites for -1.8, 1.9 and 1.10, and the Delta contrib suites for Spark 3.5, 4.0 and 4.1, each only when the day's +1.9 and 1.10, and the Delta contrib suites for Spark 3.5, 4.0 and 4.1, each only when the day's changes touched files it covers. The queue already ran everything else against the same tree, so nothing in the queue tier is repeated. A day with no merges, or with only documentation changes, runs nothing. @@ -415,7 +395,7 @@ gh run view "$run" --repo apache/datafusion-comet --json jobs \ A healthy run over a day of normal merges reports about 53 successes — nine Spark SQL shards for each of 3.5, 4.0 and 4.2, plus the Iceberg shards — with the handful of skips being the suites that -belong to the queue tier rather than the nightly one — Spark 3.4, Spark 4.1 +belong to the queue tier rather than the nightly one — Spark 4.1 and Iceberg 1.11. If everything is skipped, open the run's `Detect changes` job: it logs the `Nightly base:` commit it diffed against and the list of changed files, which is enough to tell a genuinely quiet day from a base that has drifted. @@ -431,9 +411,8 @@ On `main`, the tiers hold the expensive suites back for the queue and the nightl test every change. A release branch has neither, so holding a suite back there would mean never running it. A pull request that targets a release branch, such as a backport, therefore runs the PR, queue and nightly tiers together. Release branches get few pull requests, so this costs little. The -path filters still apply, so a documentation-only change runs none of the heavy suites. The Spark -SQL suite for Spark 3.4 still needs its `run-spark-3.4-tests` label, and the other `run-*` labels -have nothing to add there. +path filters still apply, so a documentation-only change runs none of the heavy suites. The +`run-*` labels have nothing to add there. A release branch runs the workflow files committed on it, so a change to these rules on `main` reaches a release branch only if it is backported. `branch-1.0` predates the tiers and follows its diff --git a/docs/source/contributor-guide/development.md b/docs/source/contributor-guide/development.md index 8efec3ea7c3..16a458d180b 100644 --- a/docs/source/contributor-guide/development.md +++ b/docs/source/contributor-guide/development.md @@ -447,7 +447,7 @@ from a clean IntelliJ configuration: 6. Open the repository root as a new project in IntelliJ IDEA. 7. In the Maven tool window, enable the `spark-4.0` and `jdk17` profiles. Disable conflicting Spark - version profiles such as `spark-3.4` or `spark-3.5` if IntelliJ selected them. + version profiles such as `spark-3.5` if IntelliJ selected them. 8. Re-import or reload the Maven project. 9. Build the project in IntelliJ. @@ -543,7 +543,6 @@ The tests can be run with: ```sh export SPARK_HOME=`pwd` -./mvnw -Dsuites="org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite" -Pspark-3.4 -nsu test ./mvnw -Dsuites="org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite" -Pspark-3.5 -nsu test ./mvnw -Dsuites="org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite" -Pspark-4.0 -nsu test ``` @@ -552,7 +551,6 @@ and ```sh export SPARK_HOME=`pwd` -./mvnw -Dsuites="org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite" -Pspark-3.4 -nsu test ./mvnw -Dsuites="org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite" -Pspark-3.5 -nsu test ./mvnw -Dsuites="org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite" -Pspark-4.0 -nsu test ``` @@ -562,7 +560,6 @@ To regenerate the golden files, you can run the following commands. ```sh export SPARK_HOME=`pwd` -SPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -Dsuites="org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite" -Pspark-3.4 -nsu test SPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -Dsuites="org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite" -Pspark-3.5 -nsu test SPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -Dsuites="org.apache.spark.sql.comet.CometTPCDSV1_4_PlanStabilitySuite" -Pspark-4.0 -nsu test ``` @@ -571,7 +568,6 @@ and ```sh export SPARK_HOME=`pwd` -SPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -Dsuites="org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite" -Pspark-3.4 -nsu test SPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -Dsuites="org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite" -Pspark-3.5 -nsu test SPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -Dsuites="org.apache.spark.sql.comet.CometTPCDSV2_7_PlanStabilitySuite" -Pspark-4.0 -nsu test ``` diff --git a/docs/source/contributor-guide/iceberg-spark-tests.md b/docs/source/contributor-guide/iceberg-spark-tests.md index 9e35a82881c..5a7508e2665 100644 --- a/docs/source/contributor-guide/iceberg-spark-tests.md +++ b/docs/source/contributor-guide/iceberg-spark-tests.md @@ -79,8 +79,8 @@ Clone Apache Iceberg locally and apply the diff file from Comet against the matc ```shell git clone git@github.com:apache/iceberg.git apache-iceberg cd apache-iceberg -git checkout apache-iceberg-1.8.1 -git apply ../datafusion-comet/dev/diffs/iceberg/1.8.1.diff +git checkout apache-iceberg-1.9.1 +git apply ../datafusion-comet/dev/diffs/iceberg/1.9.1.diff ``` ## 3. Run Iceberg Spark Tests @@ -112,26 +112,26 @@ regenerate: ```shell cd apache-iceberg -git reset --hard apache-iceberg-1.8.1 && git clean -fd -git apply ../datafusion-comet/dev/diffs/iceberg/1.8.1.diff +git reset --hard apache-iceberg-1.9.1 && git clean -fd +git apply ../datafusion-comet/dev/diffs/iceberg/1.9.1.diff # Make changes, then run spotless to fix formatting ./gradlew spotlessApply # Stage any new or deleted files, then generate the diff git add -A -git diff apache-iceberg-1.8.1 > ../datafusion-comet/dev/diffs/iceberg/1.8.1.diff +git diff apache-iceberg-1.9.1 > ../datafusion-comet/dev/diffs/iceberg/1.9.1.diff ``` -Repeat for each Iceberg version (1.8.1, 1.9.1, 1.10.0, 1.11.0). The file contents differ between versions, so each +Repeat for each Iceberg version (1.9.1, 1.10.0, 1.11.0). The file contents differ between versions, so each diff must be generated against its own tag. ## Running Tests in CI -The `iceberg_1_8`, `iceberg_1_9`, `iceberg_1_10` and `iceberg_1_11` jobs in `ci.yml` apply these diffs and -run the three Gradle targets above against each Iceberg version, all with Java 17. Iceberg 1.8.1 runs against -Spark 3.4.3; Iceberg 1.9.1 and 1.10.0 run against Spark 3.5.9; Iceberg 1.11.0 runs against Spark 4.1.3. -Iceberg 1.11.0 runs in the merge queue; 1.8.1, 1.9.1 and 1.10.0 run once a night against `main`. All four +The `iceberg_1_9`, `iceberg_1_10` and `iceberg_1_11` jobs in `ci.yml` apply these diffs and +run the three Gradle targets above against each Iceberg version, all with Java 17. Iceberg 1.9.1 and 1.10.0 run +against Spark 3.5.9; Iceberg 1.11.0 runs against Spark 4.1.3. +Iceberg 1.11.0 runs in the merge queue; 1.9.1 and 1.10.0 run once a night against `main`. All three run earlier on a pull request labeled `run-iceberg-tests`; none runs on an unlabeled pull request. Each job calls `iceberg_spark_test_reusable.yml`, which holds the build and test job logic. See [.github/workflows/README.md](https://github.com/apache/datafusion-comet/blob/main/.github/workflows/README.md) diff --git a/docs/source/contributor-guide/iceberg-writes.md b/docs/source/contributor-guide/iceberg-writes.md index 373f3c91df5..01da96ff8ef 100644 --- a/docs/source/contributor-guide/iceberg-writes.md +++ b/docs/source/contributor-guide/iceberg-writes.md @@ -454,8 +454,8 @@ the writer: run the write suites and the Iceberg Spark tests. The pin policy is | `CometIcebergWriteBenchmark` | Native versus iceberg-java for unpartitioned, clustered, fanout and copy-on-write delete writes. It checks each arm's plan before timing it. | | `CometIcebergEndToEndBenchmark` | An Iceberg read, hash repartition and Iceberg write against stock Spark, over primitive, struct, array, map and nested columns. It checks each arm's plan and the rows it wrote before timing it. | -The Comet suites run against the Iceberg version each Spark profile pins in `spark/pom.xml`: 1.5.2 -for Spark 3.4, 1.8.1 for 3.5, 1.10.0 for 4.0, and 1.11.0 for 4.1. No Iceberg runtime is published +The Comet suites run against the Iceberg version each Spark profile pins in `spark/pom.xml`: 1.8.1 +for Spark 3.5, 1.10.0 for 4.0, and 1.11.0 for 4.1. No Iceberg runtime is published for Spark 4.2, so the 4.2 profile builds against the 4.0 runtime and the Iceberg suites skip there (`icebergAvailable` is false on Spark 4.2). Only the default profile runs on every pull request. Behaviour that differs between Iceberg versions, such as reflection targets, metrics conventions diff --git a/docs/source/contributor-guide/release_process.md b/docs/source/contributor-guide/release_process.md index 0dac5641c8a..e0820d2fead 100644 --- a/docs/source/contributor-guide/release_process.md +++ b/docs/source/contributor-guide/release_process.md @@ -195,7 +195,7 @@ match a bare version too. Prefer patterns such as `comet-spark-spark3.5_2.12-*.j The release branch has the same CI workflows as `main`, so a `-SNAPSHOT`-only glob in a test harness or script fails only after the release branch is cut. The version bump pull request is where it shows up. It changes the `pom.xml` files, so it runs the suites that find the jar by file name or version, such as the PyArrow UDF tests and -the Spark SQL and Iceberg suites. The Spark SQL suite for Spark 3.4 runs only with the `run-spark-3.4-tests` label. +the Spark SQL and Iceberg suites. ### Update Version in main @@ -294,8 +294,7 @@ change log file into `main`. Once the generated docs, version bump, and change log have merged to the release branch, run every CI suite against it. Each pull request ran against the branch as it stood when its run started, so nothing has yet tested -the branch with all of them merged. Pull requests there also skip the Spark SQL suite for Spark 3.4 unless it is -labeled, and Miri, which runs only on a schedule on `main`. +the branch with all of them merged. Pull requests there also skip Miri, which runs only on a schedule on `main`. A dispatch runs every job in the release branch's own `ci.yml`, including `docs`, which publishes the website. So first check that the branch limits that job to `main`: the `if:` this prints must require @@ -314,7 +313,7 @@ gh workflow run miri.yml --repo apache/datafusion-comet --ref branch-0.13 ``` A dispatched `ci.yml` run ignores the tiers and the path filters. It runs every suite in the -[tier table](ci.md#three-tiers), including the Spark SQL suite for Spark 3.4, which sits outside every tier. +[tier table](ci.md#three-tiers). `miri.yml` runs the unsafe code checks, which are not part of `ci.yml`. Expect the runs to take a few hours. A failed dispatched run does not open a `ci-nightly-failure` issue, so check the result yourself. This prints the diff --git a/docs/source/contributor-guide/spark-sql-tests.md b/docs/source/contributor-guide/spark-sql-tests.md index 989cd85bf93..f2775a04c09 100644 --- a/docs/source/contributor-guide/spark-sql-tests.md +++ b/docs/source/contributor-guide/spark-sql-tests.md @@ -37,14 +37,14 @@ Here is an overview of the changes that we need to make to Spark: Comet's cache format although it is disabled by default. The serializer has to be a default for the whole JVM because Spark keeps the first cache serializer it loads for the rest of the JVM -Here are the steps involved in running the Spark SQL tests with Comet, using Spark 3.4.3 for this example. +Here are the steps involved in running the Spark SQL tests with Comet, using Spark 3.5.9 for this example. `dev/local-ci.sh` runs all of the steps below the way CI runs them: ```shell dev/local-ci.sh spark # every matrix row dev/local-ci.sh spark catalyst # one row -dev/local-ci.sh spark 3.4 catalyst # a non-default Spark version +dev/local-ci.sh spark 3.5 catalyst # a non-default Spark version ``` See [Continuous Integration](ci.md#reproducing-a-suite-failure-locally). The manual steps below @@ -55,7 +55,7 @@ are still the reference, and are what you want when creating or updating a diff Run `make release` in Comet to install the Comet JAR into the local Maven repository, specifying the Spark version. ```shell -PROFILES="-Pspark-3.4" make release +PROFILES="-Pspark-3.5" make release ``` ## 2. Clone Spark and Apply Diff @@ -65,9 +65,9 @@ Clone Apache Spark locally and apply the diff file from Comet. Note: this is a shallow clone of a tagged Spark commit and is not suitable for general Spark development. ```shell -git clone -b 'v3.4.3' --single-branch --depth 1 git@github.com:apache/spark.git apache-spark +git clone -b 'v3.5.9' --single-branch --depth 1 git@github.com:apache/spark.git apache-spark cd apache-spark -git apply ../datafusion-comet/dev/diffs/3.4.3.diff +git apply ../datafusion-comet/dev/diffs/3.5.9.diff ``` ## 3. Run Spark SQL Tests @@ -115,7 +115,7 @@ ENABLE_COMET=true ENABLE_COMET_ONHEAP=true sbt -J-Xmx4096m -Dspark.test.includeS 2. Set `ENABLE_COMET=true` and `ENABLE_COMET_ONHEAP=true` in environment variables. Without `ENABLE_COMET_ONHEAP`, Comet stays disabled, because the tests do not enable off-heap memory. ![img.png](img.png) -3. After the above tests are configured, spark tests can be run with debugging enabled on spark/comet code. Note that Comet is added as a dependency and the classes are readonly while debugging from Spark. Any new changes to Comet are to be built and deployed locally through the command (`PROFILES="-Pspark-3.4" make release`) +3. After the above tests are configured, spark tests can be run with debugging enabled on spark/comet code. Note that Comet is added as a dependency and the classes are readonly while debugging from Spark. Any new changes to Comet are to be built and deployed locally through the command (`PROFILES="-Pspark-3.5" make release`) ## Creating a diff file for a new Spark version @@ -194,21 +194,6 @@ earlier on a pull request carrying the `run-spark-3.5-tests` / `run-spark-4.0-te request. Apply the 3.5, 4.0 or 4.2 label when a change could behave differently on those versions, since without it the first verdict is the nightly run's, after the change has landed. -Spark 3.4 is deprecated and is not run by the merge queue. It runs only when a pull request carries -the `run-spark-3.4-tests` label, or when `ci.yml` is dispatched manually from the Actions page. -Apply the label if your change touches `spark/src/main/spark-3.4/`, `dev/diffs/3.4.3.diff`, or -shared code whose Spark 3.4 behavior you are unsure of: - -```shell -gh pr edit --add-label run-spark-3.4-tests -``` - -The run that command starts is advisory: it reports under `Required Checks (label run)`, not the -`Required Checks` status `main` requires, so a Spark 3.4 failure there does not block the merge on -its own. The label stays applied, so the next push you make runs Spark 3.4 as part of the required -verdict. Read the label run's result before merging — with Spark 3.4 out of the merge queue, -nothing else will. - See [Continuous Integration](ci.md) for how the tiers and labels work. When bringing up a new Spark version, the version needs its own job in `.github/workflows/ci.yml` plus entries in `FILTERS` and `POLICY` in `dev/ci/compute-changes.py`. diff --git a/docs/source/user-guide/latest/compatibility/expressions/index.md b/docs/source/user-guide/latest/compatibility/expressions/index.md index 2681261ce13..dbfe346b2f8 100644 --- a/docs/source/user-guide/latest/compatibility/expressions/index.md +++ b/docs/source/user-guide/latest/compatibility/expressions/index.md @@ -26,16 +26,12 @@ version you are running: ```{toctree} :maxdepth: 2 -Spark 3.4 Spark 3.5 Spark 4.0 Spark 4.1 Spark 4.2 ``` -Spark 3.4 support is deprecated and will be removed in a future release; see -[Spark Version Compatibility](../spark-versions.md#spark-34). - Expressions that are not 100% Spark-compatible fall back to Spark by default, except those with a JVM codegen-dispatch path, which stay in Comet's native pipeline and match Spark exactly. Set `spark.comet.expression.EXPRNAME.allowIncompatible=true`, where `EXPRNAME` is diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/index.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/index.md deleted file mode 100644 index 0550dfb0c66..00000000000 --- a/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/index.md +++ /dev/null @@ -1,46 +0,0 @@ - - -# Expression Compatibility (Spark 3.4) - -Compatibility notes for Comet running on Apache Spark 3.4. Expressions that are not 100% -Spark-compatible fall back to Spark by default, except those with a JVM codegen-dispatch -path, which stay in Comet's native pipeline and match Spark exactly. Set -`spark.comet.expression.EXPRNAME.allowIncompatible=true`, where `EXPRNAME` is the Spark -expression class name, to run Comet's native implementation despite its differences -from Spark. See the [Comet Supported Expressions Guide](../../../expressions.md) -for more information on this configuration setting. - -Spark 3.4 support is deprecated and will be removed in a future release; see -[Spark Version Compatibility](../../spark-versions.md#spark-34). - -```{toctree} -:maxdepth: 1 - -aggregate -array -datetime -map -math -misc -string -struct -url -cast -``` diff --git a/docs/source/user-guide/latest/compatibility/spark-versions.md b/docs/source/user-guide/latest/compatibility/spark-versions.md index 3ae6fb1b336..ffbbfc1059c 100644 --- a/docs/source/user-guide/latest/compatibility/spark-versions.md +++ b/docs/source/user-guide/latest/compatibility/spark-versions.md @@ -26,25 +26,6 @@ compatibility guide. For the rule that governs how long each Spark minor is supp [Apache Spark version support section](../../../about/versioning_policy.md#apache-spark-version-support) of the versioning policy. -## Spark 3.4 - -Spark 3.4.3 is supported with Java 17 and Scala 2.12/2.13. - -```{warning} -Spark 3.4 support is deprecated as of the 1.0.0 release and will be removed in a future release. -Comet continues to build and publish Spark 3.4 binaries in the meantime, but Apache Spark's own -SQL test suite no longer runs against Spark 3.4 automatically: it runs only when a contributor -opts a pull request into it. Regressions specific to Spark 3.4 are therefore more likely to reach a -release than on the other supported versions. We recommend moving to Spark 3.5 or later. -``` - -### Known Limitations - -- **Extra `SparkException` layer in Parquet schema mismatch errors**: when a Parquet read is - rejected because a file's type cannot be converted to the requested type, the error's cause chain - has one more `SparkException` layer than Spark's own reader produces. - See [Parquet](../parquet.md#schema-mismatch-handling) for details. - ## Spark 3.5 Spark 3.5.9 is supported with Java 17 and Scala 2.12/2.13. diff --git a/docs/source/user-guide/latest/iceberg.md b/docs/source/user-guide/latest/iceberg.md index 13f811a68c6..f788ab1da9c 100644 --- a/docs/source/user-guide/latest/iceberg.md +++ b/docs/source/user-guide/latest/iceberg.md @@ -651,9 +651,8 @@ normal `remove_orphan_files` maintenance. When one task fails, the tasks that had already completed leave committed-nothing data files too. The committer collects each task's commit message as that task finishes, so on a job failure it aborts with the completed messages and deletes their data files through the table -`FileIO`. (From Iceberg 1.6 on, Iceberg's own `SparkWrite.abort` skips cleanup unless a commit -failed with a cleanable error, so on the stock path those files are left for -`remove_orphan_files`. Iceberg 1.5, which the Spark 3.4 profile pins, deletes them.) A +`FileIO`. (Iceberg's own `SparkWrite.abort` skips cleanup unless a commit failed with a +cleanable error, so on the stock path those files are left for `remove_orphan_files`.) A failure during the driver-side commit itself behaves exactly as on the stock path: the commit messages carry genuine `SparkWrite$TaskCommit` objects, so Iceberg's own `SparkWrite.abort` cleanup (which deletes the files listed in the commit messages for cleanable failures) applies @@ -707,9 +706,8 @@ a data file but not what any reader computes from it: per task. The native writer zero-pads the task attempt id and counts from `00000`; iceberg-java leaves the id unpadded and counts from `00001`. - Partition directory names match iceberg-java 1.8+'s `PartitionSpec.partitionToPath` for every - partition type the native writer accepts. On Iceberg 1.5.x, - which the Spark 3.4 profile pins, iceberg-java itself spelled `timestamp` and `timestamptz` - directories with `LocalDateTime.toString()` / `OffsetDateTime.toString()` + partition type the native writer accepts. Earlier iceberg-java releases spelled `timestamp` and + `timestamptz` directories with `LocalDateTime.toString()` / `OffsetDateTime.toString()` (`ts=1969-12-31T23:59:58.500Z`) and left the partition field name unescaped; Comet uses the 1.8+ spelling on every profile. Distinct partition values still get distinct directories in all cases, and no reader parses these names: files are resolved through committed manifests. diff --git a/docs/source/user-guide/latest/installation.md b/docs/source/user-guide/latest/installation.md index 3ee1562a215..434b9b0d07b 100644 --- a/docs/source/user-guide/latest/installation.md +++ b/docs/source/user-guide/latest/installation.md @@ -47,15 +47,8 @@ Other versions may work well enough for development and evaluation purposes. Comet requires JDK 17 or later. JDK 11 is no longer supported as of the 1.1.0 release. -```{warning} -Spark 3.4 support is deprecated as of the 1.0.0 release and will be removed in a future release. -Apache Spark's own SQL test suite no longer runs against Spark 3.4 automatically; it runs only on -demand. We recommend moving to Spark 3.5 or later. -``` - | Spark Version | Java Version | Scala Version | Comet Tests in CI | Spark SQL Tests in CI | | ------------- | ------------ | ------------- | ----------------- | --------------------- | -| 3.4.3 | 17 | 2.12/2.13 | Nightly | On demand | | 3.5.9 | 17 | 2.12/2.13 | Nightly | Nightly | | 4.0.4 | 17/21 | 2.13 | Nightly | Nightly | | 4.1.3 | 17/21 | 2.13 | Before merge | Before merge | @@ -65,9 +58,7 @@ Note that we do not test the full matrix of supported Java and Scala versions in "Before merge" in the table above means the suite must pass before a change is merged. "Nightly" means the suite runs once a day against the `main` branch, so a regression it finds is caught after the change -has been merged rather than before. "On demand" means the suite does not run automatically at all. A -contributor can still run it against an individual pull request, but Spark 3.4 is no longer covered by -default. +has been merged rather than before. Note that Comet may not fully work with proprietary forks of Apache Spark such as the Spark versions offered by Cloud Service Providers. @@ -93,7 +84,6 @@ records the commit each snapshot was built from. The following artifacts are published: -- `comet-spark-spark3.4_2.12` - `comet-spark-spark3.5_2.12` - `comet-spark-spark4.0_2.13` - `comet-spark-spark4.1_2.13` @@ -133,8 +123,6 @@ repository describing your environment, and [build from source] for your target Here are the direct links for downloading the Comet $COMET_VERSION jar file. -- [Comet plugin for Spark 3.4 / Scala 2.12](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.4_2.12/$COMET_VERSION/comet-spark-spark3.4_2.12-$COMET_VERSION.jar) -- [Comet plugin for Spark 3.4 / Scala 2.13](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.4_2.13/$COMET_VERSION/comet-spark-spark3.4_2.13-$COMET_VERSION.jar) - [Comet plugin for Spark 3.5 / Scala 2.12](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.5_2.12/$COMET_VERSION/comet-spark-spark3.5_2.12-$COMET_VERSION.jar) - [Comet plugin for Spark 3.5 / Scala 2.13](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.5_2.13/$COMET_VERSION/comet-spark-spark3.5_2.13-$COMET_VERSION.jar) - [Comet plugin for Spark 4.0 / Scala 2.13](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark4.0_2.13/$COMET_VERSION/comet-spark-spark4.0_2.13-$COMET_VERSION.jar) diff --git a/docs/source/user-guide/latest/migration-guide.md b/docs/source/user-guide/latest/migration-guide.md index feaa15aab06..5c360e31348 100644 --- a/docs/source/user-guide/latest/migration-guide.md +++ b/docs/source/user-guide/latest/migration-guide.md @@ -57,6 +57,16 @@ Treat setting one of these keys as a temporary measure. If you find you cannot s legacy behavior, please open an issue describing your use case so it can be considered before the key is removed. +## Upgrading to Comet 1.3.0 + +Comet `1.3.0` makes no behavior changes that need a `spark.comet.legacy.*` key. The change below +needs none either, but check whether it applies to your deployment. + +Comet `1.3.0` no longer builds or publishes support for Spark 3.4. Spark 3.4 has been deprecated +since Comet `1.0.0`. If you run Spark 3.4, stay on the last Comet release that publishes +`comet-spark-spark3.4_*`. See [Installing Comet](installation.md) for the supported Java, Scala, +and Spark versions. + ## Upgrading to Comet 1.2.0 Comet `1.2.0` makes no behavior changes that need a `spark.comet.legacy.*` key. The changes below diff --git a/pom.xml b/pom.xml index a215e2a818f..49efb28cf0a 100644 --- a/pom.xml +++ b/pom.xml @@ -105,17 +105,16 @@ under the License. 1.12.262 3.4.1 + in this pom sets delta.spark.version to its matching Delta release. The delta.artifact + property names the Delta artifact (delta-spark for the 3.x/4.x lines). The defaults + match the default spark-4.1 profile and only serve invocations that activate no + Spark profile (e.g. `mvn -Pdelta spotless:apply`). This is a separate property from + `delta.version` (the in-tree `-Pcontrib-delta` contrib) on purpose, so bumping one + pairing cannot silently change the other. --> delta-spark 4.3.1 - delta-core - 2.4.0 - 17 - ${java.version} - ${java.version} - - - spark-3.5 diff --git a/spark/pom.xml b/spark/pom.xml index 16a43db3200..071140fd5ef 100644 --- a/spark/pom.xml +++ b/spark/pom.xml @@ -344,63 +344,6 @@ under the License. - - spark-3.4 - - - - com.amazonaws - aws-java-sdk-s3 - ${aws-java-sdk-v1.version} - provided - - - commons-logging - commons-logging - - - - - com.amazonaws - aws-java-sdk-dynamodb - ${aws-java-sdk-v1.version} - provided - - - commons-logging - commons-logging - - - - - org.apache.iceberg - iceberg-spark-runtime-${spark.version.short}_${scala.binary.version} - 1.5.2 - test - - - - org.eclipse.jetty - jetty-server - 9.4.53.v20231009 - test - - - org.eclipse.jetty - jetty-servlet - 9.4.53.v20231009 - test - - - - spark-3.5