Skip to content

ci: run Spark SQL 4.1 and Iceberg 1.11 on every pull request #7516

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
andygrove merged 4 commits into from
Nov 7, 2026
Merged

ci: run Spark SQL 4.1 and Iceberg 1.11 on every pull request #7516

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
bd0ddf0
ci: run Spark SQL 4.1 and Iceberg 1.11 on every pull request
andygrove Nov 5, 2026
d5b7569
docs: update remaining queue-only references for Spark SQL 4.1 and Ic…
andygrove Nov 5, 2026
11a5c9b
Merge branch 'main' into andygrove/pr-tier-spark-sql-iceberg
andygrove Nov 6, 2026
6a9e3db
docs: update remaining tier descriptions and cover the retired label …
andygrove Nov 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 17 additions & 17 deletions .github/workflows/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,16 @@ ruleset in `.asf.yaml`. That splits CI into three tiers:

- **PR tier** (`pr`): fast feedback while a change is being iterated on.
The Linux build, with the Comet test suites run against the default Spark
profile (4.1) only, and nothing else. For a pull request from a fork this
profile (4.1) only, plus one Spark version and one Iceberg version: Spark
SQL on Spark 4.1 (catalyst and sql_core) and Iceberg 1.11, the default
profile's. For a pull request from a fork this
tier runs on the contributor's fork instead of here; see
[Fork runs](#fork-runs).
- **Queue tier** (`queue`): the authoritative gate. The PR tier plus the
macOS build, the benchmark compile check, the Delta contrib build gate, the
PyArrow UDF suite, Spark SQL on Spark 4.1 and Iceberg 1.11, evaluated
against the merge result rather than against the PR head. One Spark version
and one Iceberg version, both the default profile's.
PyArrow UDF suite and the Spark 4.1 `sql_hive` shards, evaluated against
the merge result rather than against the PR head. It runs the same Spark and
Iceberg versions as the PR tier and no others.
- **Nightly tier** (`nightly`): the regression sweep of everything else, once
a day against `main` as it stands. The Comet test suites against the other
four Spark profiles, Spark SQL on Spark 3.5, 4.0 and 4.2, Iceberg
Expand Down Expand Up @@ -114,12 +116,11 @@ tiers partition the list and that the `pr` tier is exactly the default profile.
--------------- ----------------- ---------------------------
pr_build_linux (+ push, cache only) docs pr_build_macos run-macos-tests
(Spark 4.1 profile only) (+ push, cache only)
pr_benchmark_check run-benchmark-check
delta_build_gate run-delta-build-gate
spark_4_1 (catalyst, sql_core) pr_benchmark_check run-benchmark-check
iceberg_1_11 delta_build_gate run-delta-build-gate
pyarrow_udf_test run-pyarrow-udf-tests
spark_4_1 run-spark-4.1-tests
label or dispatch only spark_4_1 sql_hive run-spark-4.1-hive-tests
---------------------- iceberg_1_11 run-iceberg-tests
----------------------
spark_3_4 run-spark-3.4-tests
nightly tier, or PR with label
------------------------------
Expand Down Expand Up @@ -175,11 +176,11 @@ tiers partition the list and that the `pr` tier is exactly the default profile.
| `delta_4_1` | nightly, **or** PR with `run-delta-tests` | Delta contrib and native sources |
| `docs` | push to main, paths matched, **or** dispatch on `main` | `.asf.yaml`, `docs/**`, `docs.yaml` |
| `spark_3_5` | nightly, **or** PR with `run-spark-3.5-tests` | Spark 3.5 sources |
| `spark_4_1` | merge group, **or** PR with `run-spark-4.1-tests`; the `sql_hive` shards alone with `run-spark-4.1-hive-tests` | Spark 4.1 sources |
| `spark_4_1` | PR or merge group for catalyst and `sql_core`; the `sql_hive` shards in the merge group, **or** PR with `run-spark-4.1-hive-tests` | Spark 4.1 sources |
| `spark_3_4` | PR with `run-spark-3.4-tests`, or dispatch | Spark 3.4 sources |
| `spark_4_0` | nightly, **or** PR with `run-spark-4.0-tests` | Spark 4.0 sources |
| `spark_4_2` | nightly, **or** PR with `run-spark-4.2-tests` | Spark 4.2 sources |
| `iceberg_1_11` | merge group, **or** PR with `run-iceberg-tests` | Iceberg sources |
| `iceberg_1_11` | PR or merge group, paths matched | Iceberg sources |
| `iceberg_1_8` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources |
| `iceberg_1_9` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources |
| `iceberg_1_10` | nightly, **or** PR with `run-iceberg-tests` | Iceberg sources |
Expand Down Expand Up @@ -207,7 +208,7 @@ largest share of the runner-hours CI spends (about 60% in the week to
2026-10-28, label runs included). A fork pull request therefore does not run
the PR tier on ASF's runners. The contributor's fork runs it, and the merge
queue, which stays on ASF, runs the queue tier on the merge result before
anything lands. That is the PR tier plus Spark SQL on 4.1, Iceberg 1.11, macOS and
anything lands. That is the PR tier plus the Spark 4.1 `sql_hive` shards, macOS and
the other queue-tier jobs, with path filters still applied; the nightly-only and
label-only suites are not part of it. The design follows `apache/datafusion` (#25591) and `apache/arrow-rs`
(#11788). Three files carry it:
Expand Down Expand Up @@ -328,12 +329,11 @@ These rules keep those runs from corrupting the PR's status:
still going. Skipping the job would not help either, since a skipped check
run still carries the name and still counts as passing.

`run-spark-4.1-tests` and `run-spark-4.1-hive-tests` both feed the single
`spark_4_1` call. The first sets both of its POLICY outputs and runs every
module; the second sets only `spark_4_1_hive` and runs only the `sql_hive`
shards. Adding the hive label to a pull request that already carries the suite
label re-runs just the hive rows, since a `labeled` run selects what the new
label gates.
Two POLICY outputs feed the single `spark_4_1` call. `spark_4_1` is in the PR
tier and covers catalyst and the `sql_core` shards; `spark_4_1_hive` is in the
queue tier and covers the `sql_hive` shards. `run-spark-4.1-hive-tests` sets
the second on a pull request; on a `labeled` run that is the only output set,
so the call runs only the hive rows.

The opt-in labels have to exist in repository settings before they can be
applied; `contains()` on a label nobody can add is simply always false, which
Expand Down
29 changes: 15 additions & 14 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,13 +22,14 @@
# Merging goes through GitHub's merge queue (see `rulesets` in `.asf.yaml`), so
# there are three tiers:
#
# pull_request fast feedback. The Linux build only, with its test matrix
# run against the default Spark profile (4.1) alone.
# pull_request fast feedback. The Linux build, with its test matrix run
# against the default Spark profile (4.1) alone, plus Spark
# SQL (catalyst and sql_core) on Spark 4.1 and Iceberg 1.11.
# merge_group the authoritative gate. The PR tier plus the macOS build,
# the benchmark compile check, the Delta contrib build gate,
# the PyArrow UDF suite, Spark SQL on Spark 4.1 and Iceberg
# 1.11, evaluated against the merge result rather than the
# PR head. One Spark version and one Iceberg version, both
# the PyArrow UDF suite and the Spark SQL sql_hive shards,
# evaluated against the merge result rather than the PR
# head. One Spark version and one Iceberg version, both
# the default profile's.
# schedule the nightly regression sweep of everything else: the Linux
# test matrix against the other Spark profiles, Spark SQL on
Expand Down Expand Up @@ -571,12 +572,11 @@ jobs:
spark_4_1:
name: Spark SQL Tests (Spark 4.1)
needs: changes
# Queue-only by default, like every other Spark SQL suite. Two POLICY
# outputs feed one call, so the queue gets every module from a single
# 40-minute build instead of two. `spark_4_1` covers catalyst and the
# sql_core shards; `spark_4_1_hive` adds the sql_hive shards. On a pull
# request `run-spark-4.1-tests` sets both, and `run-spark-4.1-hive-tests`
# sets only the second, which then runs only the hive rows.
# Two POLICY outputs feed one call, so the queue gets every module from a
# single 40-minute build instead of two. `spark_4_1` covers catalyst and
# the sql_core shards and is in the PR tier; `spark_4_1_hive` adds the
# sql_hive shards and is queue-only. On a pull request
# `run-spark-4.1-hive-tests` sets the second.
if: needs.changes.outputs.spark_4_1 == 'true' || needs.changes.outputs.spark_4_1_hive == 'true'
uses: ./.github/workflows/spark_sql_test_reusable.yml
with:
Expand Down Expand Up @@ -644,9 +644,10 @@ jobs:
iceberg_1_11:
name: Iceberg Spark SQL Tests (Iceberg 1.11)
needs: changes
# Queue-only by default; PRs need the `run-iceberg-tests` label. Iceberg
# 1.11 is our only Spark 4.1 Iceberg coverage, which is why it is the one
# Iceberg version the queue runs while 1.8/1.9/1.10 run nightly.
# Runs on every PR whose paths match, and in the queue. Iceberg 1.11 is our
# only Spark 4.1 Iceberg coverage, which is why it is the one Iceberg
# version the PR and queue tiers run while 1.8/1.9/1.10 run nightly (or
# earlier on a PR with the `run-iceberg-tests` label).
if: needs.changes.outputs.iceberg_1_11 == 'true'
uses: ./.github/workflows/iceberg_spark_test_reusable.yml
with:
Expand Down
25 changes: 14 additions & 11 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,22 +49,25 @@ remote changes before retrying; do not bypass it with `--force`. See
## Checking a change against CI

A green pull request does not mean a change is safe to queue. The pull request tier runs the Comet
suites against the default Spark profile only. For a pull request from a fork it runs on the fork,
where `CI (fork)` reports it as advisory, and `Required Checks` here passes without a build. Spark's
own SQL suite and the Iceberg suites first report in the merge queue, where a failure evicts the
pull request and blocks everyone else's merges, or in the nightly run, after the change has already
landed.
suites, Spark's own SQL suite (catalyst and `sql_core`) and the Iceberg 1.11 suite against the
default Spark profile only. For a pull request from a fork it runs on the fork, where `CI (fork)`
reports it as advisory, and `Required Checks` here passes without a build. Everything else first
reports in the merge queue (Spark SQL `sql_hive`), where a failure evicts the pull request and
blocks everyone else's merges, or in the nightly run (the other Spark and Iceberg versions), after
the change has already landed.

So when a change touches the serde, the planner, a native operator, a Spark shim, an Iceberg code
path, or anything under `dev/diffs/`, get a verdict first. Either run the suite locally:
So when a change touches code those suites cover but the pull request tier does not run, such as
Hive table support, a Spark shim for a version other than 4.1, or an Iceberg code path that differs
across versions, get a verdict first. Either run the suite locally:

```shell
dev/local-ci.sh spark sql_core-1 # one matrix row, or `spark` for all of them
dev/local-ci.sh iceberg shard-2 # one shard, or `iceberg` for every target
dev/local-ci.sh spark 3.5 sql_core-1 # a nightly-tier version, named explicitly
dev/local-ci.sh iceberg 1.10 shard-2 # one shard, or `iceberg` for every target
```

or apply the matching `run-*` label so CI runs it instead. Which changes warrant which suite, the
label names, and the script's caveats are in
or apply the matching `run-*` label so CI runs it instead. Do not do either for Spark SQL 4.1
(catalyst and `sql_core`) or Iceberg 1.11, which every pull request already runs. Which changes
warrant which suite, the label names, and the script's caveats are in
[Continuous Integration](docs/source/contributor-guide/ci.md).

Pick one rather than both. A full local job is hours of compute and tens of GB of disk, so prefer
Expand Down
58 changes: 24 additions & 34 deletions dev/ci/check-ci-config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,15 +207,15 @@
# where "allowed" ignores path filters. Written out longhand rather than
# derived from POLICY, so that a change to the routing has to be stated twice
# and cannot be made by accident.
# The PR tier is the Linux build and nothing else. Every Spark SQL and Iceberg
# suite waits for the queue or the nightly run, or for its label.
PR_TIER = {"build_linux", "build_linux_full"}
# The queue adds one Spark version (4.1, the default profile) and one Iceberg
# version (1.11, the only Spark 4.1 coverage), plus the build-level gates.
# The PR tier is the Linux build plus one Spark version (4.1, the default
# profile, catalyst and sql_core only) and one Iceberg version (1.11, the only
# Spark 4.1 coverage). Every other Spark SQL and Iceberg suite waits for the
# nightly run, or for its label.
PR_TIER = {"build_linux", "build_linux_full", "spark_4_1", "iceberg_1_11"}
# The queue runs the same Spark and Iceberg versions as the PR tier, adds the
# Spark 4.1 sql_hive shards, and adds the build-level gates.
QUEUE_TIER = PR_TIER | {
"spark_4_1",
"spark_4_1_hive",
"iceberg_1_11",
"build_macos",
"build_macos_full",
"benchmark",
Expand Down Expand Up @@ -244,9 +244,9 @@
# `schedule` cases below assert its absence rather than quietly accept it
# coming back.
SPARK_DEPRECATED = {"spark_3_4"}
# One label opts a pull request into every Iceberg version, whichever tier
# each sits in.
ICEBERG_OPT_IN = {"iceberg_1_8", "iceberg_1_9", "iceberg_1_10", "iceberg_1_11"}
# One label opts a pull request into every nightly Iceberg version. Iceberg
# 1.11 is in the PR tier, so it needs no label.
ICEBERG_OPT_IN = {"iceberg_1_8", "iceberg_1_9", "iceberg_1_10"}
# Likewise one label for every Delta profile.
DELTA_OPT_IN = {"delta_3_5", "delta_4_0", "delta_4_1"}
ALL_JOBS = QUEUE_TIER | NIGHTLY_TIER | SPARK_DEPRECATED | {"docs"}
Expand Down Expand Up @@ -372,22 +372,9 @@ def fork_pr(action="synchronize", label=None, labels=(), base="main", repository
},
{"build_linux_all_profiles"},
),
# Spark 4.1 is queue-only too. Two labels feed its one call: the suite
# label selects every module, the hive label only the sql_hive shards.
# Neither label pulls in any other Spark version.
(
{"name": "pull_request", "action": "synchronize", "labels": ["run-spark-4.1-tests"]},
PR_TIER | {"spark_4_1", "spark_4_1_hive"},
),
(
{
"name": "pull_request",
"action": "labeled",
"label": "run-spark-4.1-tests",
"labels": ["run-spark-4.1-tests"],
},
{"spark_4_1", "spark_4_1_hive"},
),
# Spark 4.1's catalyst and sql_core shards are in the PR tier. Its sql_hive
# shards are queue-only, and their label brings them forward without
# pulling in any other Spark version.
(
{"name": "pull_request", "action": "synchronize", "labels": ["run-spark-4.1-hive-tests"]},
PR_TIER | {"spark_4_1_hive"},
Expand All @@ -401,17 +388,20 @@ def fork_pr(action="synchronize", label=None, labels=(), base="main", repository
},
{"spark_4_1_hive"},
),
# Adding the hive label on top of the suite label re-runs only the hive
# rows: a `labeled` run selects what the new label gates, and the suite
# label's earlier run already covered every module at this commit.
# The retired suite label gates nothing now, so it must not start a second
# pipeline or add anything to a pushed commit's run.
Comment on lines +391 to +392

@comphead comphead Nov 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment says the retired label must not "add anything to a pushed commit's run", but the case below only covers the labeled event. Could you add a synchronize case with labels: ["run-spark-4.1-tests"] that expects PR_TIER? Without it nothing in the checker would notice if the label started gating spark_4_1_hive again. I evaluated compute() for that event on the head and got only PR_TIER, so the case should pass as is.

@andygrove andygrove Nov 6, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Added a synchronize case with labels: ["run-spark-4.1-tests"] that expects PR_TIER, in 6a9e3db. check-ci-config.py passes.

(
{
"name": "pull_request",
"action": "labeled",
"label": "run-spark-4.1-hive-tests",
"labels": ["run-spark-4.1-tests", "run-spark-4.1-hive-tests"],
"label": "run-spark-4.1-tests",
"labels": ["run-spark-4.1-tests"],
},
{"spark_4_1_hive"},
set(),
),
(
{"name": "pull_request", "action": "synchronize", "labels": ["run-spark-4.1-tests"]},
PR_TIER,
),
# An opt-in label present on a pushed commit adds just that suite. For the
# deprecated Spark 3.4 suite the label is the *only* way it ever runs on a
Expand Down Expand Up @@ -547,8 +537,8 @@ def fork_pr(action="synchronize", label=None, labels=(), base="main", repository
# because a fork cannot see the labels on the pull request.
(fork_pr(), set()),
(
fork_pr(labels=["run-spark-4.1-tests", "run-delta-tests"]),
{"spark_4_1", "spark_4_1_hive"} | DELTA_OPT_IN,
fork_pr(labels=["run-spark-4.1-hive-tests", "run-delta-tests"]),
{"spark_4_1_hive"} | DELTA_OPT_IN,
),
# The label brings the non-default Spark profiles forward here, while the
# fork's run covers the default profile and the lints, so between them every
Expand Down
Loading