Problem Description

reconcileOutdatedHelmChartMetric (controllers/helmchart_outdated_check.go:177) clears an outdated_helm_chart series only when its chart was checked again in the current pass. A key that was outdated in the previous pass and not checked in this one is carried forward unchanged. That is right when the repository fetch failed, but the same branch also catches a chart that nothing references anymore: its ClusterSummary was deleted, or the chart was removed from the profile. Such a key is never checked again, so it stays in previouslyOutdatedHelmCharts and its series stays at 1 until the addon-controller restarts. The comment on outdatedHelmChartGauge (controllers/metrics.go:155) says series of charts that are no longer referenced don't linger.

To reproduce on main, this test marks a chart outdated in one pass, then runs five passes in which nothing references it. It fails: the chart is still tracked, and its series is still exposed.

func TestOutdatedSeriesOfUnreferencedChart(t *testing.T) {
	key := outdatedHelmChartMetricKey{profileKind: "ClusterProfile", profileName: "gone", clusterType: "Capi",
		clusterNamespace: "default", clusterName: "deleted-cluster", chartName: "nginx",
		releaseNamespace: "web", releaseName: "nginx"}

	pass := newOutdatedHelmChartPassState()
	pass.markChecked(&key, true)
	reconcileOutdatedHelmChartMetric(pass)

	for range 5 {
		reconcileOutdatedHelmChartMetric(newOutdatedHelmChartPassState())
	}

	if _, ok := previouslyOutdatedHelmCharts[key]; ok {
		t.Errorf("chart no longer referenced is still tracked as outdated")
	}
}

We should tell the two cases apart: keep carrying forward a chart that is still referenced but could not be fetched, and clear one that no ClusterSummary references anymore. The pass could record the keys it collected from the ClusterSummaries, so reconcileOutdatedHelmChartMetric knows which keys are still referenced.

System Information

CLUSTERAPI VERSION: n/a
SVELTOS VERSION: main (9bc39ad7); the checker is the same since v1.13.0 (#1883)
KUBERNETES VERSION: n/a

Logs

None.