CI: Reduce workflow overhead - #3962
kevinjqliu wants to merge 4 commits into
Conversation
Overlap validation while retaining one-at-a-time ALLGREEN squash merges. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Apply the existing workflow/ref concurrency policy to policy and documentation checks, cancelling active predecessors only for pull requests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use ubuntu-slim for eleven validation, reporting, artifact-merging, documentation, and maintenance jobs that do not require Docker or a full VM. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Build and serve documentation with the docs group without installing the unrelated default test dependency stack. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
The dependency-heavy integration report may exceed the ubuntu-slim 15-minute job limit and block the required gate.
Pull request overview
Reduces CI and merge-queue overhead through lighter documentation dependencies, slimmer runners, concurrency cancellation, and parallel queue builds.
Changes:
- Restricts documentation commands to the
docsdependency group. - Moves lightweight jobs to
ubuntu-slimand cancels redundant PR runs. - Increases merge-queue build parallelism from 1 to 2.
File summaries
| File | Summary |
|---|---|
Makefile |
Limits documentation commands to docs dependencies. |
.github/workflows/zizmor.yml |
Adds workflow concurrency controls. |
.github/workflows/svn-build-artifacts.yml |
Moves artifact merging to a slim runner. |
.github/workflows/stale.yml |
Moves stale processing to a slim runner. |
.github/workflows/python-release.yml |
Moves validation jobs to slim runners. |
.github/workflows/python-release-docs.yml |
Moves documentation release work to a slim runner. |
.github/workflows/python-integration.yml |
Moves selected integration jobs to slim runners; the dependency-heavy report job may exceed the 15-minute limit. |
.github/workflows/python-ci.yml |
Moves the aggregate gate to a slim runner. |
.github/workflows/pypi-build-artifacts.yml |
Moves artifact merging to a slim runner. |
.github/workflows/nightly-pypi-build.yml |
Moves version setup to a slim runner. |
.github/workflows/license_check.yml |
Adds workflow concurrency controls. |
.github/workflows/codeql.yml |
Adds concurrency controls and a slim runner. |
.github/workflows/check-md-link.yml |
Adds workflow concurrency controls. |
.github/workflows/asf-allowlist-check.yml |
Adds concurrency controls and a slim runner. |
.asf.yaml |
Allows two merge-queue entries to build concurrently. |
Review details
Suppressed comments (1)
.github/workflows/python-integration.yml:155
ubuntu-slimhas a hard 15-minute job limit, but this job performs a fulluv sync --group dev(including heavyweight development dependencies such as PySpark) before generating the report. A cold-cache run can hit that limit and fail the requiredpython-integration-requiredgate; keep this dependency-heavy report on a full runner, or reduce the install to only the report's dependencies before moving it to slim.
runs-on: ubuntu-slim
- Files reviewed: 15/15 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
rambleraptor
left a comment
rambleraptor
left a comment
There was a problem hiding this comment.
Thanks for doing this! Excited to see better CI performance, especially on the merge queue.
| asf-allowlist-check: | ||
| runs-on: ubuntu-24.04 | ||
| runs-on: ubuntu-slim |
There was a problem hiding this comment.
is it possible to still be explicit about this versioning?
There was a problem hiding this comment.
The versioned Ubuntu images use 4 CPUs / 16 GB ram vs 1 CPU / 5 GB ram for slim. We're probably not using the additional resources, so might as well not waste them.
https://docs.github.com/en/actions/reference/runners/github-hosted-runners
There was a problem hiding this comment.
was just looking at apache/iceberg-verification#5 (comment) and thought it would make sense to see if possible is all
There was a problem hiding this comment.
like is it possible to have a pinned slim 24.04
Grouping a few changes together in this PR. Overall, this helps improve CI and merge queue.
max_entries_to_buildfrom1to2. This allows merge queue to run in paralleldocs-serveanddocs-buildubuntu-slim. Hopefully this will reduce contention waiting for runnerconcurrencysetting to cancel redundant runs in the same PR