Skip to content

feat: Health monitoring of the resources a profile deploys - #2037

Merged
gianlucam76 merged 1 commit into
projectsveltos:mainfrom
gianlucam76:health-monitoring
Oct 15, 2026
Merged

gianlucam76 merged 1 commit into
projectsveltos:mainfrom
gianlucam76:health-monitoring

Conversation

@gianlucam76

Copy link
Copy Markdown
Member

Adds an opt-in healthMonitoring section to the shared profile spec (ClusterProfile, Profile, ClusterPromotion):

    spec:
      healthMonitoring:
        enabled: true
        defaultChecks: true      # default
        healthChecks: []         # ClusterProfile only

When enabled, once every feature of a ClusterSummary is Provisioned, addon-controller creates one HealthCheck and one ClusterHealthCheck for that ClusterSummary. The existing pipeline does the rest: healthcheck-manager deploys the HealthCheck to the managed cluster, sveltos-agent evaluates it, and the HealthCheckReports reach the management cluster (also in pull mode and agentless).

This answers "what is unhealthy on this cluster among what Sveltos deployed", separately from validateHealths, which gates a ClusterSummary reaching Provisioned at deploy time only.

  • Default checks cover Deployment, StatefulSet, DaemonSet, Job and PersistentVolumeClaim. The Lua rules are embedded (controllers/healthmonitoring/health.lua) and report Healthy, Progressing, Degraded or Suspended.

  • Only resources deployed by the profile are evaluated. Raw resources are selected by the projectsveltos.io/reason label and matched on the owner kind and name annotations. Helm chart content is matched on the Helm release annotations of the releases the ClusterSummary manages.

  • healthChecks adds user-written HealthChecks to the same ClusterHealthCheck. defaultChecks: false monitors only those.

  • Only a ClusterProfile can set healthChecks. A HealthCheck is cluster scoped and referenced by name, and its report is shared by every profile referencing it on a cluster, so a namespaced Profile could learn about resources outside its namespace. For a Profile the list is ignored, the default checks apply, and a Warning event HealthChecksIgnored says so. CRD validation rejects healthChecks when enabled is false and defaultChecks: false with no healthChecks.

  • The generated objects are named hm-<12 hex of the SHA-256 of the ClusterSummary UID>, carry the ClusterSummary UID label and namespace/name annotations, and are removed when monitoring is disabled, when nothing is left to monitor, or when the ClusterSummary is deleted. Removal only touches objects carrying that ClusterSummary's UID label, so a user's HealthCheck is never deleted.

  • Health monitoring is reconciled before shouldReconcile, so it works for a OneTime ClusterSummary with nothing left to deploy. The ClusterSummary update predicate reacts to features becoming Provisioned and to changes in the managed Helm releases.

  • DryRun ClusterSummaries are skipped.

  • A HealthCheck only evaluates objects that exist: a deleted resource is not reported (drift detection covers that).

  • Helm chart resources outside the release namespace are only partly matched.

  • sveltos-agent caches every object of each monitored kind in the cluster, about 20 KB of heap per object, independent of how many profiles enable monitoring.

Adds an opt-in `healthMonitoring` section to the shared profile spec
(ClusterProfile, Profile, ClusterPromotion):

```yaml
    spec:
      healthMonitoring:
        enabled: true
        defaultChecks: true      # default
        healthChecks: []         # ClusterProfile only
```

When enabled, once every feature of a ClusterSummary is Provisioned, addon-controller creates one HealthCheck and one ClusterHealthCheck
for that ClusterSummary. The existing pipeline does the rest: healthcheck-manager deploys the HealthCheck to the managed cluster,
sveltos-agent evaluates it, and the HealthCheckReports reach the management cluster (also in pull mode and agentless).

This answers "what is unhealthy on this cluster among what Sveltos deployed", separately from `validateHealths`, which gates a ClusterSummary
reaching Provisioned at deploy time only.

- Default checks cover Deployment, StatefulSet, DaemonSet, Job and PersistentVolumeClaim. The Lua rules are embedded
(`controllers/healthmonitoring/health.lua`) and report Healthy, Progressing, Degraded or Suspended.
- Only resources deployed by the profile are evaluated. Raw resources are selected by the `projectsveltos.io/reason` label and
matched on the owner kind and name annotations. Helm chart content is matched on the Helm release annotations of the releases
the ClusterSummary manages.
- `healthChecks` adds user-written HealthChecks to the same ClusterHealthCheck. `defaultChecks: false` monitors only those.
- Only a ClusterProfile can set `healthChecks`. A HealthCheck is cluster scoped and referenced by name, and its report is shared by
every profile referencing it on a cluster, so a namespaced Profile could learn about resources outside its namespace. For a Profile
the list is ignored, the default checks apply, and a Warning event `HealthChecksIgnored` says so.
CRD validation rejects `healthChecks` when `enabled` is false and `defaultChecks: false` with no `healthChecks`.
- The generated objects are named `hm-<12 hex of the SHA-256 of the ClusterSummary UID>`, carry the ClusterSummary UID label and
namespace/name annotations, and are removed when monitoring is disabled, when nothing is left to monitor, or when the ClusterSummary
is deleted. Removal only touches objects carrying that ClusterSummary's UID label, so a user's HealthCheck is never deleted.
- Health monitoring is reconciled before `shouldReconcile`, so it works for a OneTime ClusterSummary with nothing left to deploy.
The ClusterSummary update predicate reacts to features becoming Provisioned and to changes in the managed Helm releases.
- DryRun ClusterSummaries are skipped.

- A HealthCheck only evaluates objects that exist: a deleted resource is not reported (drift detection covers that).
- Helm chart resources outside the release namespace are only partly matched.
- sveltos-agent caches every object of each monitored kind in the cluster, about 20 KB of heap per object, independent of how many profiles enable monitoring.
@gianlucam76
gianlucam76 merged commit 7767789 into projectsveltos:main Oct 15, 2026
12 checks passed
@gianlucam76
gianlucam76 deleted the health-monitoring branch October 15, 2026 07:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant