Repository navigation
atecontroller: scope controller cache to deployments managed by workerpools - #181
danielle-mustillo wants to merge 1 commit into
Conversation
iplay88keys
left a comment
There was a problem hiding this comment.
🤖 AI-generated review. I reviewed the findings before posting.
Looks good. I checked that only WorkerPoolReconciler reads Deployments from the cache, and that pool Deployments have carried ate.dev/worker-pool since v1.0.0. I also confirmed TestDeploymentCacheScope fails when the Deployment cache has no label filter. Two optional suggestions inline.
…rpools The WorkerPool controller currently caches all Deployments in a cluster. Therefore the controller's memory incorrectly grows proportionately to the number of Deployments in the cluster, not to the size of the resources managed by agent-substrate as is expected. Before this change, on very large clusters with thousands of unrelated deployments, the controller's memory can grow to consume hundreds megabytes needlessly. This commit changes the ate-controller cache to only cache Deployments with the label "ate.dev/worker-pool", a label already added to every Deployment managed by the WorkerPool controller. Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
c01f4eb to
7e0c5cf
Compare
| log := log.FromContext(ctx) | ||
| log.Info("Reconciling worker pool") | ||
|
|
||
| // The cache holds only Deployments with workerPoolLabel (CacheOptions). One |
| // The cache holds only Deployments with workerPoolLabel (CacheOptions). One | ||
| // without it reads as not found here, and the apply at the end labels it. | ||
| existing := &appsv1.Deployment{} | ||
| if err := r.Get(ctx, types.NamespacedName{Name: wp.Name, Namespace: wp.Namespace}, existing); err != nil { |
There was a problem hiding this comment.
entirely unrelated and we shouldn't fix this here, but we can just client.ObjectKeyFromObject(wp) instead of the direct NN declaration
| var ( | ||
| cfg *rest.Config | ||
| k8sClient client.Client | ||
| // cachedClient reads from the manager cache, as the reconcilers do. |
| // Deployments that carry workerPoolLabel. Other kinds are cached in full. | ||
| func CacheOptions(egressMITMCAPool types.NamespacedName) cache.Options { | ||
| hasWorkerPoolLabel, err := labels.NewRequirement(workerPoolLabel, selection.Exists, nil) | ||
| utilruntime.Must(err) |
There was a problem hiding this comment.
nit: calling utilruntime.Must seems incorrect here looking at the rest of the main.go flow. for scheme registration, fine, but as you get further into the main.go flow, it uses explicit error handling
There was a problem hiding this comment.
Not necessarily opposed to a new file here, but the cache_test.go is insufficient for the existing secret cache functionality. TestManagerOptionsScopeCache also exists from main_test.go as well. And then we're modifying that test case as well with these diffs.
Why
The controller caches every Deployment in the cluster needlessly. Thousands of unrelated Deployments in a cluster can cost hundreds of megabytes in controller memory usage. This PR changes the controller-runtime configuration to cache only Deployments labeled
ate.dev/worker-pool. The controller already sets that label on Deployments.What changed
controllers.CacheOptionsnow selects Deployments by label.main.goand the controller tests now usecontrollers.CacheOptions.Scope
Alternatives Considered
A predicate on
Ownsdoes not reduce memory. A predicate filters after the informer stores objects.Blast Radius
Only the controller's Deployment cache changes. An upgrade needs no action.
Verification
go test -race ./cmd/atecontroller/...passes.mainAI use: AI-assisted (Claude Code).