CI: cache Docker Hub image pulls in-cluster for the k8s Jenkins agents - #4974
Open
michaelsembwever wants to merge 3 commits into
Open
CI: cache Docker Hub image pulls in-cluster for the k8s Jenkins agents#4974michaelsembwever wants to merge 3 commits into
michaelsembwever wants to merge 3 commits into
Conversation
In a Helm deployed CI, each CI agent runs its own docker-in-docker daemon on an emptyDir, blank to begin with. Every agent therefore pulls the same build and test images afresh; the largest, `cassandra-ubuntu-test`, is ~35Gi. At hundreds of concurrent agents this is slow, wastes bandwidth, and trips pull-rate limits. Add an in-cluster Docker Hub pull-through cache (a registry:2 proxy) and point the agents' `dind` at it via `--registry-mirror`. Each image is then fetched from Docker Hub once and served to every agent over the cluster LAN; `dind` falls back to Docker Hub directly if the cache is unreachable. Docker's `--registry-mirror` can only mirror Docker Hub, not an arbitrary registry such as apache.jfrog.io. So the prefetch in the Jenkinsfile is repointed from the apache.jfrog.io names to those same Docker Hub names. Patch changes: - `.jenkins/k8s/docker-cache.yaml`: the pull-through cache (Deployment, Service, PVC), pinned to the always-on controller node. - `.jenkins/k8s/jenkins-deployment.yaml`: `dind` `--registry-mirror` in all three agent templates. - `.build/run-ci`: `-setup/-only-setup` applies the cache; teardown removes it. It injects the cache PVC's `storageClassName` from the Jenkins values' `persistence.storageClass`, so a site declares its storage class once and the PVC binds on clusters (such as EKS) that mark no default StorageClass. - `.jenkins/Jenkinsfile`: prefetch the Docker Hub image names. The cache runs a single replica for now. It converts repeated WAN pulls into one origin fetch plus LAN reads; if the pod's egress becomes the bottleneck this can be scaled with replicas or an object-storage backend. patch by Mick Semb Wever; reviewed by xxx for CASSANDRA-21541
michaelsembwever
force-pushed
the
mck/21541/5.0
branch
from
July 29, 2026 18:56
325860d to
5e931d0
Compare
netudima
reviewed
Jul 29, 2026
netudima
reviewed
Jul 29, 2026
netudima
reviewed
Jul 29, 2026
michaelsembwever
marked this pull request as ready for review
July 29, 2026 20:03
netudima
approved these changes
Jul 29, 2026
Member
Author
|
Another commit added: |
- fetchDockerImages: detect the dind registry mirror via `docker info`;
fall back to apache.jfrog.io when none is configured (dark launch)
- run-ci: keep the module under pylint's 1000-line cap by folding
resolve_storage_class into apply_docker_cache and merging the two
kubectl-cp download helpers
- jenkins-test.sh: validate multi-document manifests (docker-cache.yaml)
with safe_load_all
michaelsembwever
force-pushed
the
mck/21541/5.0
branch
from
July 30, 2026 15:17
3c5967c to
97aa689
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In a Helm deployed CI, each CI agent runs its own docker-in-docker daemon on an emptyDir, blank to begin with. Every agent therefore pulls the same build and test images afresh; the largest,
cassandra-ubuntu-test, is ~35Gi. At hundreds of concurrent agents this is slow, wastes bandwidth, and trips pull-rate limits.Add an in-cluster Docker Hub pull-through cache (a registry:2 proxy) and point the agents'
dindat it via--registry-mirror. Each image is then fetched from Docker Hub once and served to every agent over the cluster LAN;dindfalls back to Docker Hub directly if the cache is unreachable.Docker's
--registry-mirrorcan only mirror Docker Hub, not an arbitrary registry such as apache.jfrog.io. So the prefetch in the Jenkinsfile is repointed from the apache.jfrog.io names to those same Docker Hub names.Patch changes:
.jenkins/k8s/docker-cache.yaml: the pull-through cache (Deployment, Service, PVC), pinned to the always-on controller node..jenkins/k8s/jenkins-deployment.yaml:dind--registry-mirrorin all three agent templates..build/run-ci:--setup/--only-setupapplies the cache; teardown removes it..jenkins/Jenkinsfile: prefetch the Docker Hub image names.The cache runs a single replica for now. It converts repeated WAN pulls into one origin fetch plus LAN reads; if the pod's egress becomes the bottleneck this can be scaled with replicas or an object-storage backend.