Skip to content

GH-50046: [CI][C++] Improve caching with apache/infrastructure-actions/stash and more general cache keys#50047

Open
rok wants to merge 6 commits into
apache:mainfrom
rok:change_ccache_keys
Open

GH-50046: [CI][C++] Improve caching with apache/infrastructure-actions/stash and more general cache keys#50047
rok wants to merge 6 commits into
apache:mainfrom
rok:change_ccache_keys

Conversation

@rok
Copy link
Copy Markdown
Member

@rok rok commented May 26, 2026

Rationale for this change

C++ CI jobs currently key ccache/Docker volume caches with hashFiles('cpp/**'). This creates new immutable GitHub Actions cache entries whenever any C++ file changes, even though ccache already handles per-file invalidation internally.

Recent CI logs show that when caches are restored, C++ jobs get high ccache hit rates, but GitHub cache restore misses are common and cause large CI runtime regressions.

What changes are included in this PR?

  • Replace actions/cache with apache/infrastructure-actions/stash/restore and apache/infrastructure-actions/stash/save.
  • Remove hashFiles('cpp/**') from cache keys.
  • Use stable restore prefixes per job/config.

Are these changes tested?

By CI?

Are there any user-facing changes?

CI users should see faster builds..

Copilot AI review requested due to automatic review settings May 26, 2026 21:01
@github-actions github-actions Bot added the awaiting committer review Awaiting committer review label May 26, 2026
@github-actions
Copy link
Copy Markdown

⚠️ GitHub issue #50046 has been automatically assigned in GitHub to PR creator.

@github-actions github-actions Bot added the CI: Extra: C++ Run extra C++ CI label May 26, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors C++ CI cache handling to improve ccache reuse. Instead of using actions/cache with cache keys that include hashFiles('cpp/**'), it splits cache restore/save into separate steps, uses github.run_id as the unique save key, and only saves new caches when running on main. This avoids creating new cache entries per PR (which can cause eviction) while still letting PR builds benefit from the stable restore prefix.

Changes:

  • Replace actions/cache@v5 with paired actions/cache/restore@v5 and actions/cache/save@v5 steps across C++ workflows.
  • Drop hashFiles('cpp/**') from cache keys; use ${{ github.run_id }} as the save key and a stable prefix as the restore-key.
  • Gate Save steps with if: github.ref == 'refs/heads/main' so only main pushes write new cache entries.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
.github/workflows/cpp.yml Splits Docker volume and ccache caching into restore/save pairs for docker, macOS, and MinGW jobs; saves only from main.
.github/workflows/cpp_windows.yml Splits Windows ccache caching into restore/save; saves only from main.
.github/workflows/cpp_extra.yml Same restore/save split applied to docker-extra, JNI (linux + macOS), ODBC linux/macOS/MSVC jobs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@rok rok force-pushed the change_ccache_keys branch from 9776486 to 1711a31 Compare May 26, 2026 23:11
Copilot AI review requested due to automatic review settings May 26, 2026 23:12
@rok rok force-pushed the change_ccache_keys branch from 1711a31 to fa05f88 Compare May 26, 2026 23:12
@rok rok requested a review from kou May 26, 2026 23:14
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment thread .github/workflows/cpp_extra.yml Outdated
@rok rok force-pushed the change_ccache_keys branch from fa05f88 to 4a893c1 Compare May 26, 2026 23:15
Copilot AI review requested due to automatic review settings May 27, 2026 11:45
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

@rok rok marked this pull request as ready for review May 27, 2026 13:53
@kou
Copy link
Copy Markdown
Member

kou commented Jun 1, 2026

https://github.com/apache/infrastructure-actions/tree/main/stash
apache/infrastructure-actions has a similar action that is originally developed by @assignUser.
Should we try this?

Copilot AI review requested due to automatic review settings June 1, 2026 21:51
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

Comment thread .github/workflows/cpp.yml
Comment thread .github/workflows/cpp_windows.yml
Comment thread .github/workflows/cpp_extra.yml
Comment thread .github/workflows/cpp_extra.yml
Comment thread .github/workflows/cpp_extra.yml
@rok
Copy link
Copy Markdown
Member Author

rok commented Jun 1, 2026

@kou I think just tweaking ccache could have worked, but infrastructure-actions/stash gives us 5 day eviction and no size limit so a safer long term approach.

I updated the PR to use infrastructure-actions/stash. I am not sure if there's another way to test it but by mergeing?

@github-actions github-actions Bot removed the awaiting committer review Awaiting committer review label Jun 1, 2026
@github-actions github-actions Bot added the awaiting changes Awaiting changes label Jun 1, 2026
Copy link
Copy Markdown
Member

@kou kou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you update the PR title and description?

Let's try this. If this doesn't work, we can fix this or revert this later.

Comment thread .github/workflows/cpp.yml Outdated
@pitrou
Copy link
Copy Markdown
Member

pitrou commented Jun 2, 2026

If we switch to apache/infrastructure-actions/stash, do we still need to restrict cache saves to git main?

@github-actions github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jun 2, 2026
@rok rok changed the title GH-50046: [CI][C++] Improve ccache reuse by saving to cache only from main GH-50046: [CI][C++] Improve caching with apache/infrastructure-actions/stash and more general cache keys Jun 2, 2026
@github-actions github-actions Bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Jun 2, 2026
@rok
Copy link
Copy Markdown
Member Author

rok commented Jun 2, 2026

Thanks for the review @kou & @pitrou. I updated the PR title/description and allowed for cache to be stored on non-main branches as well.

@rok rok requested review from kou and pitrou June 2, 2026 09:56
Comment thread .github/workflows/cpp.yml Outdated
Comment thread .github/workflows/cpp_extra.yml Outdated
Comment thread .github/workflows/cpp_extra.yml Outdated
Copilot AI review requested due to automatic review settings June 2, 2026 12:04
@github-actions github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jun 2, 2026
@rok
Copy link
Copy Markdown
Member Author

rok commented Jun 2, 2026

@pitrou any further comments or shall we merge and see what happens?

@rok rok requested a review from pitrou June 2, 2026 12:06
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment thread .github/workflows/cpp.yml
@github-actions github-actions Bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Jun 2, 2026
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Co-authored-by: Rok Mihevc <rok@mihevc.org>
@rok rok force-pushed the change_ccache_keys branch from 4187aed to f465a72 Compare June 2, 2026 12:20
@github-actions github-actions Bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Jun 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting changes Awaiting changes CI: Extra: C++ Run extra C++ CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants