[andygrove]

Which issue does this PR close?

No issue filed. Follows on from #7311, which moved the PR tier of fork pull requests onto the contributor's fork.

Rationale for this change

In the week from 2026-10-29 to 2026-11-05, Comet CI used about 2,600 runner-hours on ASF runners. Pull requests were 1,333 of them, the merge queue 1,003, and the nightly run 243. Even after #7311, fork pull requests into main used about 450 runner-hours in the 5.5 days after it landed. Nearly all of that came from run-* labels. A fork run can't see the labels, so labeled suites still run upstream, and because the labels stay on the pull request they run again on every push. run-spark-4.1-tests and run-iceberg-tests are among the most common, because without them a pull request gets no Spark SQL or Iceberg verdict until the merge queue.

The PR tier now runs on the fork for fork pull requests, so Spark SQL 4.1 and Iceberg 1.11 can run there at no cost to ASF runners. The merge queue already gates on exactly these versions. With this change a pull request gets the queue's Spark and Iceberg verdict before it is queued, which removes the main reason to apply those two labels, and it should also cut the queue runs that fail on a 4.1 or 1.11 problem. In the same week, 174 runner-hours went to failed queue runs and 154 to entries cancelled behind them.

What changes are included in this PR?

• dev/ci/compute-changes.py:
  • spark_4_1 (catalyst and sql_core) and iceberg_1_11 move from ["queue", label] to ["pr", "queue"]. Path filters still apply.
  • spark_4_1_hive stays queue-only, with run-spark-4.1-hive-tests to bring it forward.
• Labels:
  • run-spark-4.1-tests is retired. It gates nothing now, and a labeled run for it starts nothing. The label can be deleted from repository settings after merge.
  • run-iceberg-tests now selects the three nightly Iceberg versions (1.8, 1.9, 1.10).
• dev/ci/check-ci-config.py: PR_TIER, QUEUE_TIER, ICEBERG_OPT_IN and the label policy cases are updated to the new tiers.
• .github/workflows/README.md, the spark_4_1 comment in ci.yml, docs/source/contributor-guide/ci.md and spark-sql-tests.md are updated to describe the new tiers.

Nothing else changes for the other tiers. The nightly run still covers Spark SQL 3.5, 4.0 and 4.2, Iceberg 1.8, 1.9 and 1.10, the Delta suites and the other Comet profiles. Pull requests that target a release branch already ran every tier and are unaffected.

What this costs:

• Fork runs take longer, since the Spark SQL and Iceberg shards run alongside the Linux test matrix.
• A pull request from a branch of this repository now pays for these two suites upstream on each push when its paths match. That was 13 runs in the 5.5 days measured.

How are these changes tested?

python3 dev/ci/check-ci-config.py passes. It evaluates POLICY against the event cases in that file, which this PR updates: a plain pull request, a fork push, a fork pull request, the merge queue, the nightly run, release branches, and the remaining 4.1 hive and Iceberg label cases. test-iceberg-shards.py, test-fork-ci-status.cjs and check-versioned-source-parity.py also pass. This PR's own fork run is the end-to-end check: it changes dev/ci/**, so it routes to the Spark SQL 4.1 and Iceberg 1.11 suites on the fork.

[rich7420]

LGTM. Local policy checks and 53 helper tests passed. Current-head fork CI reports successful Spark SQL 4.1 and Iceberg 1.11 jobs.

[mbutrovich]

Thanks @andygrove.

Should this PR also update the places outside the diff that still describe Spark SQL 4.1 and Iceberg 1.11 as queue-only? As I read them, these now give the wrong guidance:

• AGENTS.md says Spark's own SQL suite and the Iceberg suites "first report in the merge queue", and tells agents to run dev/local-ci.sh spark sql_core-1 or apply a run-* label to get a verdict first. The PR tier now runs the 4.1 catalyst and sql_core shards and Iceberg 1.11, so an agent that follows it spends local compute on a run CI already does.
• iceberg-spark-tests.md says Iceberg 1.11 runs in the merge queue and none runs on an unlabeled pull request, and that without run-iceberg-tests the first Iceberg 1.11 verdict is the merge queue's.
• The iceberg_1_11 comment in ci.yml still says the job is queue-only and needs run-iceberg-tests on a pull request.

[andygrove]

Thanks @mbutrovich, good catches. d5b7569 updates all three places: AGENTS.md now says the PR tier runs Spark SQL 4.1 (catalyst and sql_core) and Iceberg 1.11, and points agents at the nightly-tier versions and sql_hive for local runs and labels. iceberg-spark-tests.md describes 1.11 as running on every matching PR and run-iceberg-tests as bringing forward 1.8, 1.9 and 1.10. The iceberg_1_11 comment in ci.yml now describes the PR and queue tiers. The fork-run figures in ci.md are updated too, per your inline suggestion.

[mbutrovich]

Thanks for the quick iteration @andygrove!

[andygrove]

Thanks for the reviews @rich7420 and @mbutrovich

[comphead]

Thanks @andygrove. The routing change reads correctly to me and I have not run anything beyond python3 dev/ci/check-ci-config.py, which passes on the head. Required Checks already lists spark_4_1 and iceberg_1_11 in needs, so no check name changes for .asf.yaml.

Two places outside the diff still describe the old tiers, in addition to the ones @mbutrovich listed:

• The tier summary at the top of .github/workflows/ci.yml (lines 25 to 31) still says the pull_request tier is "The Linux build only" and that the merge_group tier adds "Spark SQL on Spark 4.1 and Iceberg 1.11".
• docs/source/contributor-guide/iceberg-scheduler-failures.md (around line 233) tells an Iceberg write-path change to get an upstream regression verdict with run-iceberg-tests or a local shard. 1.11 now comes with the PR, so the label only adds 1.8, 1.9 and 1.10 there.

This PR also conflicts with #7521 (and #7520) in compute-changes.py, check-ci-config.py, ci.md, iceberg-spark-tests.md and .github/workflows/README.md. The conflicts look mechanical, but #7521 drops iceberg_1_8, so whichever lands second needs ICEBERG_OPT_IN and the "1.8, 1.9 and 1.10" wording in the label table, AGENTS.md and the README to say 1.9 and 1.10.

[comphead]

There are 42 open pull requests that carry run-spark-4.1-tests (all from forks). Until now the label set both spark_4_1 and spark_4_1_hive, so their upstream runs included the sql_hive shards. After this change a push to any of them runs only catalyst and sql_core, and the hive shards wait for the queue, with no message saying so. Should the PR description tell those authors to apply run-spark-4.1-hive-tests if they want the hive shards early? Or would keeping label:run-spark-4.1-tests on spark_4_1_hive until the label is deleted be less surprising?

[comphead]

This comment says the retired label must not "add anything to a pushed commit's run", but the case below only covers the labeled event. Could you add a synchronize case with labels: ["run-spark-4.1-tests"] that expects PR_TIER? Without it nothing in the checker would notice if the label started gating spark_4_1_hive again. I evaluated compute() for that event on the head and got only PR_TIER, so the case should pass as is.