Skip to content

Add S3 cache provider - #365

Open
gautamg795 wants to merge 1 commit into
Swatinem:masterfrom
get-convex:s3-cache-provider-upstream
Open

Add S3 cache provider#365
gautamg795 wants to merge 1 commit into
Swatinem:masterfrom
get-convex:s3-cache-provider-upstream

Conversation

@gautamg795

@gautamg795 gautamg795 commented Jul 21, 2026

Copy link
Copy Markdown

This adds cache-provider: s3 alongside the existing github and warpbuild providers, storing caches in an S3 bucket of your choosing. Useful for self-hosted runners on AWS, where a bucket avoids the 10 GB GitHub cache limit and transfers stay inside the VPC. Closes #286.

Usage

- uses: Swatinem/rust-cache@v2
  with:
    cache-provider: s3
    s3-bucket: ${{ vars.CACHE_BUCKET }}

Notes

  • Objects are namespaced by prefix, repository, and a hash of the cache paths + compression method (mirroring @actions/cache's version), with the full cache key as the object name.
  • Restore probes the exact key with a HeadObject/GetObject, then falls back to the newest object matching each restore-key prefix (paginated ListObjectsV2, newest LastModified wins).
  • Uploads are multipart via @aws-sdk/lib-storage; downloads fetch 16 MB byte ranges concurrently with retries and an If-Match ETag guard against concurrent overwrites.
  • Archive creation/extraction reuses the internal tar helpers from @actions/cache (resolved explicitly in the rollup config so they're bundled), so the on-disk behavior is identical to the GitHub provider.

The README includes a security warning: S3 has no equivalent of GitHub's branch-scoped cache isolation, and this can't be enforced client-side (a malicious job holding write-capable credentials can bypass the action and call S3 directly). Write access must be scoped at the credential level — e.g. an OIDC role restricted to trusted refs — and the docs say so explicitly.

Testing

  • Unit tests for provider selection and S3 listing pagination (npm test).
  • An end-to-end test (npm run test:s3-e2e, also a new workflow) that runs the built dist/restore.js/dist/save.js against MinIO in a container: verifies restored content by hash, that multipart upload engaged, that an exact hit skips the re-save, and that restore-key fallback picks the newest entry. Unlike the credential-gated warpbuild workflow, this runs on any fork or PR — no AWS credentials needed.

Full disclosure: this code was written with AI assistance. It has been reviewed by us and has been running in production CI at Convex for over a week before being submitted here.

Add `cache-provider: s3` alongside the github and warpbuild providers,
storing caches in an S3 bucket via the standard AWS SDK credential
chain. New `s3-bucket` and `s3-prefix` inputs; an empty bucket warns
and falls back to the github provider, so a workflow using an unset
repository variable still works in forks without S3 credentials.

Restores probe the exact key first and fall back to the newest object
matching each restore key prefix. Downloads fetch byte ranges
concurrently with retries; uploads are multipart. Archive creation and
extraction reuse the internal tar helpers from @actions/cache, bundled
through a small rollup resolver.

Includes unit tests for provider selection and S3 listing pagination,
plus an end-to-end test that saves and restores through the built
action against MinIO, verifying content integrity, multipart transfer,
and restore-key fallback.
Repository owner deleted a comment from teesaelee1998-ux Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request] Add S3 provider

2 participants