diff --git a/.github/workflows/README.md b/.github/workflows/README.md index d50ef7cd58a..67f591ae3ce 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -16,14 +16,16 @@ ruleset in `.asf.yaml`. That splits CI into three tiers: - **PR tier** (`pr`): fast feedback while a change is being iterated on. The Linux build, with the Comet test suites run against the default Spark - profile (4.1) only, and nothing else. For a pull request from a fork this + profile (4.1) only, plus one Spark version and one Iceberg version: Spark + SQL on Spark 4.1 (catalyst and sql_core) and Iceberg 1.11, the default + profile's. For a pull request from a fork this tier runs on the contributor's fork instead of here; see [Fork runs](#fork-runs). - **Queue tier** (`queue`): the authoritative gate. The PR tier plus the macOS build, the benchmark compile check, the Delta contrib build gate, the - PyArrow UDF suite, Spark SQL on Spark 4.1 and Iceberg 1.11, evaluated - against the merge result rather than against the PR head. One Spark version - and one Iceberg version, both the default profile's. + PyArrow UDF suite and the Spark 4.1 `sql_hive` shards, evaluated against + the merge result rather than against the PR head. It runs the same Spark and + Iceberg versions as the PR tier and no others. - **Nightly tier** (`nightly`): the regression sweep of everything else, once a day against `main` as it stands. The Comet test suites against the other four Spark profiles, Spark SQL on Spark 3.5, 4.0 and 4.2, Iceberg @@ -114,12 +116,11 @@ tiers partition the list and that the `pr` tier is exactly the default profile. --------------- ----------------- --------------------------- pr_build_linux (+ push, cache only) docs pr_build_macos run-macos-tests (Spark 4.1 profile only) (+ push, cache only) - pr_benchmark_check run-benchmark-check - delta_build_gate run-delta-build-gate + spark_4_1 (catalyst, sql_core) pr_benchmark_check run-benchmark-check + iceberg_1_11 delta_build_gate run-delta-build-gate pyarrow_udf_test run-pyarrow-udf-tests - spark_4_1 run-spark-4.1-tests label or dispatch only spark_4_1 sql_hive run-spark-4.1-hive-tests - ---------------------- iceberg_1_11 run-iceberg-tests + ---------------------- spark_3_4 run-spark-3.4-tests nightly tier, or PR with label ------------------------------ @@ -175,11 +176,11 @@ tiers partition the list and that the `pr` tier is exactly the default profile. | `delta_4_1` | nightly, **or** PR with `run-delta-tests` | Delta contrib and native sources | | `docs` | push to main, paths matched, **or** dispatch on `main` | `.asf.yaml`, `docs/**`, `docs.yaml` | | `spark_3_5` | nightly, **or** PR with `run-spark-3.5-tests` | Spark 3.5 sources | -| `spark_4_1` | merge group, **or** PR with `run-spark-4.1-tests`; the `sql_hive` shards alone with `run-spark-4.1-hive-tests` | Spark 4.1 sources | +| `spark_4_1` | PR or merge group for catalyst and `sql_core`; the `sql_hive` shards in the merge group, **or** PR with `run-spark-4.1-hive-tests` | Spark 4.1 sources | | `spark_3_4` | PR with `run-spark-3.4-tests`, or dispatch | Spark 3.4 sources | | `spark_4_0` | nightly, **or** PR with `run-spark-4.0-tests` | Spark 4.0 sources | | `spark_4_2` | nightly, **or** PR with `run-spark-4.2-tests` | Spark 4.2 sources | -| `iceberg_1_11` | merge group, **or** PR with `run-iceberg-tests` | Iceberg sources | +| `iceberg_1_11` | PR or merge group, paths matched | Iceberg sources | | `iceberg_1_8` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources | | `iceberg_1_9` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources | | `iceberg_1_10` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources | @@ -207,7 +208,7 @@ largest share of the runner-hours CI spends (about 60% in the week to 2026-10-28, label runs included). A fork pull request therefore does not run the PR tier on ASF's runners. The contributor's fork runs it, and the merge queue, which stays on ASF, runs the queue tier on the merge result before -anything lands. That is the PR tier plus Spark SQL on 4.1, Iceberg 1.11, macOS and +anything lands. That is the PR tier plus the Spark 4.1 `sql_hive` shards, macOS and the other queue-tier jobs, with path filters still applied; the nightly-only and label-only suites are not part of it. The design follows `apache/datafusion` (#25591) and `apache/arrow-rs` (#11788). Three files carry it: @@ -328,12 +329,11 @@ These rules keep those runs from corrupting the PR's status: still going. Skipping the job would not help either, since a skipped check run still carries the name and still counts as passing. -`run-spark-4.1-tests` and `run-spark-4.1-hive-tests` both feed the single -`spark_4_1` call. The first sets both of its POLICY outputs and runs every -module; the second sets only `spark_4_1_hive` and runs only the `sql_hive` -shards. Adding the hive label to a pull request that already carries the suite -label re-runs just the hive rows, since a `labeled` run selects what the new -label gates. +Two POLICY outputs feed the single `spark_4_1` call. `spark_4_1` is in the PR +tier and covers catalyst and the `sql_core` shards; `spark_4_1_hive` is in the +queue tier and covers the `sql_hive` shards. `run-spark-4.1-hive-tests` sets +the second on a pull request; on a `labeled` run that is the only output set, +so the call runs only the hive rows. The opt-in labels have to exist in repository settings before they can be applied; `contains()` on a label nobody can add is simply always false, which diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0fc66aba7f4..295e0ba410c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -22,13 +22,14 @@ # Merging goes through GitHub's merge queue (see `rulesets` in `.asf.yaml`), so # there are three tiers: # -# pull_request fast feedback. The Linux build only, with its test matrix -# run against the default Spark profile (4.1) alone. +# pull_request fast feedback. The Linux build, with its test matrix run +# against the default Spark profile (4.1) alone, plus Spark +# SQL (catalyst and sql_core) on Spark 4.1 and Iceberg 1.11. # merge_group the authoritative gate. The PR tier plus the macOS build, # the benchmark compile check, the Delta contrib build gate, -# the PyArrow UDF suite, Spark SQL on Spark 4.1 and Iceberg -# 1.11, evaluated against the merge result rather than the -# PR head. One Spark version and one Iceberg version, both +# the PyArrow UDF suite and the Spark SQL sql_hive shards, +# evaluated against the merge result rather than the PR +# head. One Spark version and one Iceberg version, both # the default profile's. # schedule the nightly regression sweep of everything else: the Linux # test matrix against the other Spark profiles, Spark SQL on @@ -571,12 +572,11 @@ jobs: spark_4_1: name: Spark SQL Tests (Spark 4.1) needs: changes - # Queue-only by default, like every other Spark SQL suite. Two POLICY - # outputs feed one call, so the queue gets every module from a single - # 40-minute build instead of two. `spark_4_1` covers catalyst and the - # sql_core shards; `spark_4_1_hive` adds the sql_hive shards. On a pull - # request `run-spark-4.1-tests` sets both, and `run-spark-4.1-hive-tests` - # sets only the second, which then runs only the hive rows. + # Two POLICY outputs feed one call, so the queue gets every module from a + # single 40-minute build instead of two. `spark_4_1` covers catalyst and + # the sql_core shards and is in the PR tier; `spark_4_1_hive` adds the + # sql_hive shards and is queue-only. On a pull request + # `run-spark-4.1-hive-tests` sets the second. if: needs.changes.outputs.spark_4_1 == 'true' || needs.changes.outputs.spark_4_1_hive == 'true' uses: ./.github/workflows/spark_sql_test_reusable.yml with: @@ -644,9 +644,10 @@ jobs: iceberg_1_11: name: Iceberg Spark SQL Tests (Iceberg 1.11) needs: changes - # Queue-only by default; PRs need the `run-iceberg-tests` label. Iceberg - # 1.11 is our only Spark 4.1 Iceberg coverage, which is why it is the one - # Iceberg version the queue runs while 1.8/1.9/1.10 run nightly. + # Runs on every PR whose paths match, and in the queue. Iceberg 1.11 is our + # only Spark 4.1 Iceberg coverage, which is why it is the one Iceberg + # version the PR and queue tiers run while 1.8/1.9/1.10 run nightly (or + # earlier on a PR with the `run-iceberg-tests` label). if: needs.changes.outputs.iceberg_1_11 == 'true' uses: ./.github/workflows/iceberg_spark_test_reusable.yml with: diff --git a/AGENTS.md b/AGENTS.md index 60af781d677..30bd2441449 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -49,22 +49,25 @@ remote changes before retrying; do not bypass it with `--force`. See ## Checking a change against CI A green pull request does not mean a change is safe to queue. The pull request tier runs the Comet -suites against the default Spark profile only. For a pull request from a fork it runs on the fork, -where `CI (fork)` reports it as advisory, and `Required Checks` here passes without a build. Spark's -own SQL suite and the Iceberg suites first report in the merge queue, where a failure evicts the -pull request and blocks everyone else's merges, or in the nightly run, after the change has already -landed. +suites, Spark's own SQL suite (catalyst and `sql_core`) and the Iceberg 1.11 suite against the +default Spark profile only. For a pull request from a fork it runs on the fork, where `CI (fork)` +reports it as advisory, and `Required Checks` here passes without a build. Everything else first +reports in the merge queue (Spark SQL `sql_hive`), where a failure evicts the pull request and +blocks everyone else's merges, or in the nightly run (the other Spark and Iceberg versions), after +the change has already landed. -So when a change touches the serde, the planner, a native operator, a Spark shim, an Iceberg code -path, or anything under `dev/diffs/`, get a verdict first. Either run the suite locally: +So when a change touches code those suites cover but the pull request tier does not run, such as +Hive table support, a Spark shim for a version other than 4.1, or an Iceberg code path that differs +across versions, get a verdict first. Either run the suite locally: ```shell -dev/local-ci.sh spark sql_core-1 # one matrix row, or `spark` for all of them -dev/local-ci.sh iceberg shard-2 # one shard, or `iceberg` for every target +dev/local-ci.sh spark 3.5 sql_core-1 # a nightly-tier version, named explicitly +dev/local-ci.sh iceberg 1.10 shard-2 # one shard, or `iceberg` for every target ``` -or apply the matching `run-*` label so CI runs it instead. Which changes warrant which suite, the -label names, and the script's caveats are in +or apply the matching `run-*` label so CI runs it instead. Do not do either for Spark SQL 4.1 +(catalyst and `sql_core`) or Iceberg 1.11, which every pull request already runs. Which changes +warrant which suite, the label names, and the script's caveats are in [Continuous Integration](docs/source/contributor-guide/ci.md). Pick one rather than both. A full local job is hours of compute and tens of GB of disk, so prefer diff --git a/dev/ci/check-ci-config.py b/dev/ci/check-ci-config.py index 9823869bf8c..0edbae114ac 100644 --- a/dev/ci/check-ci-config.py +++ b/dev/ci/check-ci-config.py @@ -207,15 +207,15 @@ # where "allowed" ignores path filters. Written out longhand rather than # derived from POLICY, so that a change to the routing has to be stated twice # and cannot be made by accident. -# The PR tier is the Linux build and nothing else. Every Spark SQL and Iceberg -# suite waits for the queue or the nightly run, or for its label. -PR_TIER = {"build_linux", "build_linux_full"} -# The queue adds one Spark version (4.1, the default profile) and one Iceberg -# version (1.11, the only Spark 4.1 coverage), plus the build-level gates. +# The PR tier is the Linux build plus one Spark version (4.1, the default +# profile, catalyst and sql_core only) and one Iceberg version (1.11, the only +# Spark 4.1 coverage). Every other Spark SQL and Iceberg suite waits for the +# nightly run, or for its label. +PR_TIER = {"build_linux", "build_linux_full", "spark_4_1", "iceberg_1_11"} +# The queue runs the same Spark and Iceberg versions as the PR tier, adds the +# Spark 4.1 sql_hive shards, and adds the build-level gates. QUEUE_TIER = PR_TIER | { - "spark_4_1", "spark_4_1_hive", - "iceberg_1_11", "build_macos", "build_macos_full", "benchmark", @@ -244,9 +244,9 @@ # `schedule` cases below assert its absence rather than quietly accept it # coming back. SPARK_DEPRECATED = {"spark_3_4"} -# One label opts a pull request into every Iceberg version, whichever tier -# each sits in. -ICEBERG_OPT_IN = {"iceberg_1_8", "iceberg_1_9", "iceberg_1_10", "iceberg_1_11"} +# One label opts a pull request into every nightly Iceberg version. Iceberg +# 1.11 is in the PR tier, so it needs no label. +ICEBERG_OPT_IN = {"iceberg_1_8", "iceberg_1_9", "iceberg_1_10"} # Likewise one label for every Delta profile. DELTA_OPT_IN = {"delta_3_5", "delta_4_0", "delta_4_1"} ALL_JOBS = QUEUE_TIER | NIGHTLY_TIER | SPARK_DEPRECATED | {"docs"} @@ -372,22 +372,9 @@ def fork_pr(action="synchronize", label=None, labels=(), base="main", repository }, {"build_linux_all_profiles"}, ), - # Spark 4.1 is queue-only too. Two labels feed its one call: the suite - # label selects every module, the hive label only the sql_hive shards. - # Neither label pulls in any other Spark version. - ( - {"name": "pull_request", "action": "synchronize", "labels": ["run-spark-4.1-tests"]}, - PR_TIER | {"spark_4_1", "spark_4_1_hive"}, - ), - ( - { - "name": "pull_request", - "action": "labeled", - "label": "run-spark-4.1-tests", - "labels": ["run-spark-4.1-tests"], - }, - {"spark_4_1", "spark_4_1_hive"}, - ), + # Spark 4.1's catalyst and sql_core shards are in the PR tier. Its sql_hive + # shards are queue-only, and their label brings them forward without + # pulling in any other Spark version. ( {"name": "pull_request", "action": "synchronize", "labels": ["run-spark-4.1-hive-tests"]}, PR_TIER | {"spark_4_1_hive"}, @@ -401,17 +388,20 @@ def fork_pr(action="synchronize", label=None, labels=(), base="main", repository }, {"spark_4_1_hive"}, ), - # Adding the hive label on top of the suite label re-runs only the hive - # rows: a `labeled` run selects what the new label gates, and the suite - # label's earlier run already covered every module at this commit. + # The retired suite label gates nothing now, so it must not start a second + # pipeline or add anything to a pushed commit's run. ( { "name": "pull_request", "action": "labeled", - "label": "run-spark-4.1-hive-tests", - "labels": ["run-spark-4.1-tests", "run-spark-4.1-hive-tests"], + "label": "run-spark-4.1-tests", + "labels": ["run-spark-4.1-tests"], }, - {"spark_4_1_hive"}, + set(), + ), + ( + {"name": "pull_request", "action": "synchronize", "labels": ["run-spark-4.1-tests"]}, + PR_TIER, ), # An opt-in label present on a pushed commit adds just that suite. For the # deprecated Spark 3.4 suite the label is the *only* way it ever runs on a @@ -547,8 +537,8 @@ def fork_pr(action="synchronize", label=None, labels=(), base="main", repository # because a fork cannot see the labels on the pull request. (fork_pr(), set()), ( - fork_pr(labels=["run-spark-4.1-tests", "run-delta-tests"]), - {"spark_4_1", "spark_4_1_hive"} | DELTA_OPT_IN, + fork_pr(labels=["run-spark-4.1-hive-tests", "run-delta-tests"]), + {"spark_4_1_hive"} | DELTA_OPT_IN, ), # The label brings the non-default Spark profiles forward here, while the # fork's run covers the default profile and the lints, so between them every diff --git a/dev/ci/compute-changes.py b/dev/ci/compute-changes.py index f07abdb8329..6d2551ac3e4 100644 --- a/dev/ci/compute-changes.py +++ b/dev/ci/compute-changes.py @@ -583,21 +583,23 @@ # their label. "spark_3_5": ["nightly", "label:run-spark-3.5-tests"], "spark_4_0": ["nightly", "label:run-spark-4.0-tests"], - # No Spark SQL suite runs on a plain pull request. Spark 4.1 was the last - # one in the PR tier, first whole (issue #5870 pulled the sql_hive shards - # out) and then catalyst and sql_core alone. What changed is how often a - # pull request is pushed: with agent-driven review and agent-driven - # replies to review, a PR now goes through several more rounds before it - # is queued, and each round paid for the whole 4.1 build. The queue still - # runs every shard before anything lands; the two labels bring the run - # forward. `run-spark-4.1-tests` selects the whole suite, so it appears on - # both outputs; `run-spark-4.1-hive-tests` selects only the hive shards. - "spark_4_1": ["queue", "label:run-spark-4.1-tests"], - "spark_4_1_hive": [ - "queue", - "label:run-spark-4.1-tests", - "label:run-spark-4.1-hive-tests", - ], + # Spark 4.1 is the default build profile, and its catalyst and sql_core + # shards are the one Spark SQL suite every pull request runs. For a pull + # request from a fork that is the fork's runners, not ASF's: the PR tier + # runs there (see UPSTREAM below). The suite was out of the PR tier while + # the PR tier ran upstream, because agent-driven review rounds meant each + # pull request paid for the whole 4.1 build several times over. On a fork + # that repetition costs ASF nothing, and running the suite there means a + # contributor sees a 4.1 failure before the queue does, rather than having + # to ask for a label. The queue runs the same shards again on the merge + # result, and it stays the gate. + # + # The sql_hive shards stay in the queue alone. They are four more shards + # on every push for a surface few changes touch, and the queue still runs + # them before anything lands; `run-spark-4.1-hive-tests` brings them + # forward onto a pull request. + "spark_4_1": ["pr", "queue"], + "spark_4_1_hive": ["queue", "label:run-spark-4.1-hive-tests"], # Spark 4.2 support is experimental, but the suite passes, so it sits in # the nightly tier with the other non-default versions rather than being # reachable only on demand. Nightly is the right tier for it twice over: @@ -607,12 +609,13 @@ # until someone thinks to ask for it. "spark_4_2": ["nightly", "label:run-spark-4.2-tests"], # Same shape for Iceberg: 1.11 is the only Spark 4.1 coverage, so it is - # the one Iceberg version the queue runs; the three older versions run - # nightly. One label opts a pull request into all four. + # the one Iceberg version every pull request and the queue run, for the + # same reason as Spark 4.1 above. The three older versions run nightly, + # and one label brings all three forward onto a pull request. "iceberg_1_8": ["nightly", "label:run-iceberg-tests"], "iceberg_1_9": ["nightly", "label:run-iceberg-tests"], "iceberg_1_10": ["nightly", "label:run-iceberg-tests"], - "iceberg_1_11": ["queue", "label:run-iceberg-tests"], + "iceberg_1_11": ["pr", "queue"], # The Delta contrib suites all run nightly. The PR tier's strict warnings # job compiles the contrib against Spark 3.5, so a change that breaks its # build on Spark 3.5 fails on its own pull request. One label opts a pull diff --git a/docs/source/contributor-guide/ci.md b/docs/source/contributor-guide/ci.md index 9088ddbcf22..716edb03c8d 100644 --- a/docs/source/contributor-guide/ci.md +++ b/docs/source/contributor-guide/ci.md @@ -43,8 +43,8 @@ flowchart LR CRON([schedule
06:00 UTC]) --> NIGHTLY PUSH([push to main]) --> CACHE - PRTIER["PR tier
Linux build, lint, Rust tests
TPC-H / TPC-DS
Comet suites, Spark 4.1"] - QUEUE["Queue tier, on top of the PR tier
Spark SQL, Spark 4.1
Iceberg 1.11
macOS build and Comet suites
Benchmark check, Delta gate
PyArrow UDF, Spark 4.0 / 4.1 / 4.2"] + PRTIER["PR tier
Linux build, lint, Rust tests
TPC-H / TPC-DS
Comet suites, Spark 4.1
Spark SQL, Spark 4.1 catalyst and sql_core
Iceberg 1.11"] + QUEUE["Queue tier, on top of the PR tier
Spark SQL, Spark 4.1 sql_hive
macOS build and Comet suites
Benchmark check, Delta gate
PyArrow UDF, Spark 4.0 / 4.1 / 4.2"] NIGHTLY["Nightly tier
Comet suites, Spark 3.4 / 3.5 / 4.0 / 4.2
Spark SQL, Spark 3.5 / 4.0 / 4.2
Iceberg 1.8 / 1.9 / 1.10
Delta contrib, Spark 3.5 / 4.0 / 4.1"] S34["Neither tier
Spark SQL, Spark 3.4"] CACHE["Cache-refresh-only mode
the three cache-writing jobs, plus Lint"] @@ -56,9 +56,9 @@ Suite by suite: | ------------------------------------------------- | ------------ | ----------- | ------- | | Linux build, lint, Rust tests, TPC-H/TPC-DS | yes | yes | no | | Comet test suites, Spark 4.1 | yes | yes | no | -| Spark SQL tests, Spark 4.1, catalyst and sql_core | with label | yes | no | +| Spark SQL tests, Spark 4.1, catalyst and sql_core | yes | yes | no | | Spark SQL tests, Spark 4.1, sql_hive | with label | yes | no | -| Iceberg Spark SQL tests, Iceberg 1.11 | with label | yes | no | +| Iceberg Spark SQL tests, Iceberg 1.11 | yes | yes | no | | macOS build and Comet test suites | with label | yes | no | | Benchmark compile and lint check | with label | yes | no | | Delta contrib build gate | with label | yes | no | @@ -71,8 +71,9 @@ Suite by suite: The **PR tier** is the fast feedback loop while a change is being iterated on. The **queue tier** is the authoritative gate: everything the PR tier runs plus the suites that must pass before -a change lands, evaluated against the merge result rather than the pull request head. The queue -runs one Spark version and one Iceberg version, the ones the default build profile targets. The +a change lands, evaluated against the merge result rather than the pull request head. The pull +request and the queue run the same Spark version and the same Iceberg version, the ones the default +build profile targets, so a green pull request run predicts the queue's verdict on them. The **nightly tier** runs the other Spark and Iceberg versions and every Delta contrib suite once a day against `main` as it stands; see [Nightly runs](#nightly-runs) below. Nothing in the queue tier runs again @@ -121,8 +122,8 @@ the project spends on CI. So for a pull request from a fork into `main`, the PR fork's** GitHub runners, not on the Apache Software Foundation's shared ones. What stays here is the preflight checks, `Required Checks`, and any `run-*` suite a maintainer opts the pull request into. The merge queue, which also stays here, runs the queue tier on the merge result before anything -lands: the PR tier plus Spark SQL on 4.1, Iceberg 1.11, macOS and the other queue-tier suites in the -table above. The nightly-only suites and the opt-in `run-*` suites are not part of it, and path +lands: the PR tier plus the Spark 4.1 `sql_hive` shards, macOS and the other queue-tier suites in +the table above. The nightly-only suites and the opt-in `run-*` suites are not part of it, and path filters still apply. This is the arrangement `apache/datafusion` and `apache/arrow-rs` use. To get it working: @@ -134,8 +135,9 @@ To get it working: 3. **Push your branch.** Each push to a branch of your fork starts a `Comet CI (fork run)` in your fork's **Actions** tab, and a newer push to the same branch cancels the run in flight. It runs the PR tier for the files your branch changes relative to upstream `main`, which is what the pull - request will show: the Linux build, lints, Rust tests, TPC-H and TPC-DS, and the Comet test suites - against Spark 4.1. If that comparison cannot be made, it runs the whole tier. + request will show: the Linux build, lints, Rust tests, TPC-H and TPC-DS, the Comet test suites + against Spark 4.1, and the Spark SQL (catalyst and sql_core) and Iceberg 1.11 suites when the + change touches what they cover. If that comparison cannot be made, it runs the whole tier. On the pull request, a `CI (fork)` check links to that run and mirrors its result. It is first written when the upstream run finishes. After that, a pull request updated in the last six hours @@ -158,7 +160,10 @@ Some things to know: `main` that adds nothing to upstream, such as syncing your fork, selects no suite, so it warms nothing, and a branch run rebuilds the native library. A run of the PR tier on a fork with cold caches took about an hour of wall time and 240 runner-minutes, of which the native build alone was - 23 minutes. The same run with warm caches took 50 minutes and 213 runner-minutes. + 23 minutes. The same run with warm caches took 50 minutes and 213 runner-minutes. A run that also + routes to the Spark SQL and Iceberg suites took 89 minutes and about 650 runner-minutes. Their + shards run in parallel with the Linux build's test matrix, but the Spark SQL job first builds its + own native library and test classes, which took 45 minutes. - **Scheduled runs stay upstream.** The nightly run, Miri, the stale-PR closer and the weekly CodeQL scan run in `apache/datafusion-comet` only. If your fork has scheduled workflows enabled they start no job. Miri can still be run on your fork by dispatching it. @@ -189,13 +194,12 @@ Each suite outside the PR tier has a label that runs it on a pull request: | `run-delta-build-gate` | Delta contrib build gate | | `run-pyarrow-udf-tests` | PyArrow UDF tests against Spark 4.0/4.1/4.2 | | `run-delta-tests` | Delta contrib tests against Spark 3.5, 4.0 and 4.1 | -| `run-spark-4.1-tests` | Spark SQL tests against Spark 4.1, every module | | `run-spark-4.1-hive-tests` | Spark SQL tests against Spark 4.1, sql_hive only | | `run-spark-3.4-tests` | Spark SQL tests against Spark 3.4 | | `run-spark-3.5-tests` | Spark SQL tests against Spark 3.5 | | `run-spark-4.0-tests` | Spark SQL tests against Spark 4.0 | | `run-spark-4.2-tests` | Spark SQL tests against Spark 4.2 | -| `run-iceberg-tests` | Iceberg Spark SQL tests against every Iceberg version | +| `run-iceberg-tests` | Iceberg Spark SQL tests against 1.8, 1.9 and 1.10 | For a queue-tier suite the label only brings the run forward; the queue would have run it anyway before the change landed. For a nightly-tier suite the label is what gets a verdict _before_ the @@ -231,8 +235,9 @@ Use a label when a change is likely to behave differently on a version or platfo does not cover. Some examples: - a change to the serde, the planner, or a native operator, where the Comet test suites pass but - Spark's own SQL suite is the thing that would catch a behavior difference; `run-spark-4.1-tests` - runs it against the default profile + Spark's own SQL suite is the thing that would catch a behavior difference; the PR tier runs it + against the default profile, and `run-spark-3.5-tests` / `run-spark-4.0-tests` / + `run-spark-4.2-tests` run it against the others - code under `spark/src/main/spark-3.4/`, `spark-3.5/`, `spark-4.0/` or the shared `spark-3.x/` directory, or any change to `CometExprShim` and friends; `run-all-spark-profiles` runs the Comet test suites against every Spark version rather than 4.1 alone (the Lint Java matrix already diff --git a/docs/source/contributor-guide/iceberg-scheduler-failures.md b/docs/source/contributor-guide/iceberg-scheduler-failures.md index da077f12e4c..b8c9aee0e2c 100644 --- a/docs/source/contributor-guide/iceberg-scheduler-failures.md +++ b/docs/source/contributor-guide/iceberg-scheduler-failures.md @@ -229,6 +229,7 @@ stability, and retain each invocation's evidence. ## Regression checks Compilation and formatting checks do not replace execution on the required cluster. An Iceberg -write-path change also needs an upstream Iceberg regression verdict before merging, using the -`run-iceberg-tests` label or an appropriate `dev/local-ci.sh iceberg` shard as described in +write-path change also needs an upstream Iceberg regression verdict before merging. Iceberg 1.11 +runs on every matching pull request. For the older versions, use the `run-iceberg-tests` label +(1.8, 1.9 and 1.10) or an appropriate `dev/local-ci.sh iceberg` shard as described in [Continuous Integration](ci.md). That regression run does not replace this scheduler suite. diff --git a/docs/source/contributor-guide/iceberg-spark-tests.md b/docs/source/contributor-guide/iceberg-spark-tests.md index 9e35a82881c..6c5a2f7e63c 100644 --- a/docs/source/contributor-guide/iceberg-spark-tests.md +++ b/docs/source/contributor-guide/iceberg-spark-tests.md @@ -131,8 +131,9 @@ diff must be generated against its own tag. The `iceberg_1_8`, `iceberg_1_9`, `iceberg_1_10` and `iceberg_1_11` jobs in `ci.yml` apply these diffs and run the three Gradle targets above against each Iceberg version, all with Java 17. Iceberg 1.8.1 runs against Spark 3.4.3; Iceberg 1.9.1 and 1.10.0 run against Spark 3.5.9; Iceberg 1.11.0 runs against Spark 4.1.3. -Iceberg 1.11.0 runs in the merge queue; 1.8.1, 1.9.1 and 1.10.0 run once a night against `main`. All four -run earlier on a pull request labeled `run-iceberg-tests`; none runs on an unlabeled pull request. Each job +Iceberg 1.11.0 runs on every pull request whose paths match and in the merge queue; 1.8.1, 1.9.1 and +1.10.0 run once a night against `main`. Those three run earlier on a pull request labeled +`run-iceberg-tests`. Each job calls `iceberg_spark_test_reusable.yml`, which holds the build and test job logic. See [.github/workflows/README.md](https://github.com/apache/datafusion-comet/blob/main/.github/workflows/README.md) for how the pull-request, merge-queue and nightly tiers differ. @@ -162,8 +163,9 @@ and failure propagation. The fixture does not compile Spark or Iceberg. Apply the `run-iceberg-tests` label to a pull request whenever it touches the Iceberg scan or write path, reflection code (`org.apache.comet.iceberg.IcebergReflection`), or other logic whose behavior can differ across Iceberg versions. The Comet test suites in the Linux build do not exercise Iceberg's -own Spark tests, so without the label the first Iceberg 1.11 verdict is the merge queue's, and the -first verdict on the older Iceberg versions is the nightly run's, after the change has landed. +own Spark tests, but the Iceberg 1.11 job does and every pull request gets its verdict. Without the +label, the first verdict on the older Iceberg versions is the nightly run's, after the change has +landed. ### Which writer ran each Iceberg write diff --git a/docs/source/contributor-guide/spark-sql-tests.md b/docs/source/contributor-guide/spark-sql-tests.md index 989cd85bf93..3a6b426f76d 100644 --- a/docs/source/contributor-guide/spark-sql-tests.md +++ b/docs/source/contributor-guide/spark-sql-tests.md @@ -188,10 +188,11 @@ git diff v3.5.6 > ../datafusion-comet/dev/diffs/3.5.6.diff ## Running Tests in CI The easiest way to run the tests is to open a pull request against Comet and let CI run them. Spark -4.1 runs in the merge queue; Spark 3.5, 4.0 and 4.2 run once a night against `main`. Each also runs -earlier on a pull request carrying the `run-spark-3.5-tests` / `run-spark-4.0-tests` / -`run-spark-4.1-tests` / `run-spark-4.2-tests` label. No Spark SQL suite runs on an unlabeled pull -request. Apply the 3.5, 4.0 or 4.2 label when a change could behave differently on those versions, +4.1 (catalyst and sql_core) runs on every pull request and again in the merge queue, which also runs +the sql_hive shards; `run-spark-4.1-hive-tests` brings those forward onto a pull request. Spark 3.5, +4.0 and 4.2 run once a night against `main`, and each also runs earlier on a pull request carrying +the `run-spark-3.5-tests` / `run-spark-4.0-tests` / `run-spark-4.2-tests` label. Apply the 3.5, 4.0 +or 4.2 label when a change could behave differently on those versions, since without it the first verdict is the nightly run's, after the change has landed. Spark 3.4 is deprecated and is not run by the merge queue. It runs only when a pull request carries