Skip to content

atecontroller: scope controller cache to deployments managed by workerpools - #181

Open
danielle-mustillo wants to merge 1 commit into
mainfrom
danielle-mustillo/scopeDeployCacheFork
Open

danielle-mustillo wants to merge 1 commit into
mainfrom
danielle-mustillo/scopeDeployCacheFork

Conversation

@danielle-mustillo

@danielle-mustillo danielle-mustillo commented Nov 18, 2026 •

Copy link
Copy Markdown

Why

The controller caches every Deployment in the cluster needlessly. Thousands of unrelated Deployments in a cluster can cost hundreds of megabytes in controller memory usage. This PR changes the controller-runtime configuration to cache only Deployments labeled ate.dev/worker-pool. The controller already sets that label on Deployments.

What changed

  • controllers.CacheOptions now selects Deployments by label.
  • main.go and the controller tests now use controllers.CacheOptions.

Scope

Alternatives Considered

A predicate on Owns does not reduce memory. A predicate filters after the informer stores objects.

Blast Radius

Only the controller's Deployment cache changes. An upgrade needs no action.

Verification

Unrelated Deployments main This PR
0 10.2 MiB 10.2 MiB
10,000 199.0 MiB 10.2 MiB
20,000 350.5 MiB 10.2 MiB
30,000 516.2 MiB 10.2 MiB

AI use: AI-assisted (Claude Code).

  • Tests pass
  • Docs updated (none needed)

@danielle-mustillo
danielle-mustillo marked this pull request as ready for review November 18, 2026 15:15

@iplay88keys iplay88keys left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-generated review. I reviewed the findings before posting.

Looks good. I checked that only WorkerPoolReconciler reads Deployments from the cache, and that pool Deployments have carried ate.dev/worker-pool since v1.0.0. I also confirmed TestDeploymentCacheScope fails when the Deployment cache has no label filter. Two optional suggestions inline.

Comment thread cmd/atecontroller/main_test.go Resolved
…rpools

The WorkerPool controller currently caches all Deployments in a cluster.
Therefore the controller's memory incorrectly grows proportionately to
the number of Deployments in the cluster, not to the size of the
resources managed by agent-substrate as is expected.

Before this change, on very large clusters with thousands of unrelated
deployments, the controller's memory can grow to consume hundreds
megabytes needlessly.

This commit changes the ate-controller cache to only cache Deployments
with the label "ate.dev/worker-pool", a label already added to every
Deployment managed by the WorkerPool controller.

Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
@danielle-mustillo
danielle-mustillo force-pushed the danielle-mustillo/scopeDeployCacheFork branch from c01f4eb to 7e0c5cf Compare November 18, 2026 17:14

@timflannagan timflannagan left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple quick comments

log := log.FromContext(ctx)
log.Info("Reconciling worker pool")

// The cache holds only Deployments with workerPoolLabel (CacheOptions). One

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: No need for this comment

// The cache holds only Deployments with workerPoolLabel (CacheOptions). One
// without it reads as not found here, and the apply at the end labels it.
existing := &appsv1.Deployment{}
if err := r.Get(ctx, types.NamespacedName{Name: wp.Name, Namespace: wp.Namespace}, existing); err != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

entirely unrelated and we shouldn't fix this here, but we can just client.ObjectKeyFromObject(wp) instead of the direct NN declaration

var (
cfg *rest.Config
k8sClient client.Client
// cachedClient reads from the manager cache, as the reconcilers do.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks unused to me?

// Deployments that carry workerPoolLabel. Other kinds are cached in full.
func CacheOptions(egressMITMCAPool types.NamespacedName) cache.Options {
hasWorkerPoolLabel, err := labels.NewRequirement(workerPoolLabel, selection.Exists, nil)
utilruntime.Must(err)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: calling utilruntime.Must seems incorrect here looking at the rest of the main.go flow. for scheme registration, fine, but as you get further into the main.go flow, it uses explicit error handling

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not necessarily opposed to a new file here, but the cache_test.go is insufficient for the existing secret cache functionality. TestManagerOptionsScopeCache also exists from main_test.go as well. And then we're modifying that test case as well with these diffs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants