Repository navigation
feat: Health monitoring of the resources a profile deploys - #2037
Merged
Merged
Conversation
Adds an opt-in `healthMonitoring` section to the shared profile spec
(ClusterProfile, Profile, ClusterPromotion):
```yaml
spec:
healthMonitoring:
enabled: true
defaultChecks: true # default
healthChecks: [] # ClusterProfile only
```
When enabled, once every feature of a ClusterSummary is Provisioned, addon-controller creates one HealthCheck and one ClusterHealthCheck
for that ClusterSummary. The existing pipeline does the rest: healthcheck-manager deploys the HealthCheck to the managed cluster,
sveltos-agent evaluates it, and the HealthCheckReports reach the management cluster (also in pull mode and agentless).
This answers "what is unhealthy on this cluster among what Sveltos deployed", separately from `validateHealths`, which gates a ClusterSummary
reaching Provisioned at deploy time only.
- Default checks cover Deployment, StatefulSet, DaemonSet, Job and PersistentVolumeClaim. The Lua rules are embedded
(`controllers/healthmonitoring/health.lua`) and report Healthy, Progressing, Degraded or Suspended.
- Only resources deployed by the profile are evaluated. Raw resources are selected by the `projectsveltos.io/reason` label and
matched on the owner kind and name annotations. Helm chart content is matched on the Helm release annotations of the releases
the ClusterSummary manages.
- `healthChecks` adds user-written HealthChecks to the same ClusterHealthCheck. `defaultChecks: false` monitors only those.
- Only a ClusterProfile can set `healthChecks`. A HealthCheck is cluster scoped and referenced by name, and its report is shared by
every profile referencing it on a cluster, so a namespaced Profile could learn about resources outside its namespace. For a Profile
the list is ignored, the default checks apply, and a Warning event `HealthChecksIgnored` says so.
CRD validation rejects `healthChecks` when `enabled` is false and `defaultChecks: false` with no `healthChecks`.
- The generated objects are named `hm-<12 hex of the SHA-256 of the ClusterSummary UID>`, carry the ClusterSummary UID label and
namespace/name annotations, and are removed when monitoring is disabled, when nothing is left to monitor, or when the ClusterSummary
is deleted. Removal only touches objects carrying that ClusterSummary's UID label, so a user's HealthCheck is never deleted.
- Health monitoring is reconciled before `shouldReconcile`, so it works for a OneTime ClusterSummary with nothing left to deploy.
The ClusterSummary update predicate reacts to features becoming Provisioned and to changes in the managed Helm releases.
- DryRun ClusterSummaries are skipped.
- A HealthCheck only evaluates objects that exist: a deleted resource is not reported (drift detection covers that).
- Helm chart resources outside the release namespace are only partly matched.
- sveltos-agent caches every object of each monitored kind in the cluster, about 20 KB of heap per object, independent of how many profiles enable monitoring.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an opt-in
healthMonitoringsection to the shared profile spec (ClusterProfile, Profile, ClusterPromotion):When enabled, once every feature of a ClusterSummary is Provisioned, addon-controller creates one HealthCheck and one ClusterHealthCheck for that ClusterSummary. The existing pipeline does the rest: healthcheck-manager deploys the HealthCheck to the managed cluster, sveltos-agent evaluates it, and the HealthCheckReports reach the management cluster (also in pull mode and agentless).
This answers "what is unhealthy on this cluster among what Sveltos deployed", separately from
validateHealths, which gates a ClusterSummary reaching Provisioned at deploy time only.Default checks cover Deployment, StatefulSet, DaemonSet, Job and PersistentVolumeClaim. The Lua rules are embedded (
controllers/healthmonitoring/health.lua) and report Healthy, Progressing, Degraded or Suspended.Only resources deployed by the profile are evaluated. Raw resources are selected by the
projectsveltos.io/reasonlabel and matched on the owner kind and name annotations. Helm chart content is matched on the Helm release annotations of the releases the ClusterSummary manages.healthChecksadds user-written HealthChecks to the same ClusterHealthCheck.defaultChecks: falsemonitors only those.Only a ClusterProfile can set
healthChecks. A HealthCheck is cluster scoped and referenced by name, and its report is shared by every profile referencing it on a cluster, so a namespaced Profile could learn about resources outside its namespace. For a Profile the list is ignored, the default checks apply, and a Warning eventHealthChecksIgnoredsays so. CRD validation rejectshealthCheckswhenenabledis false anddefaultChecks: falsewith nohealthChecks.The generated objects are named
hm-<12 hex of the SHA-256 of the ClusterSummary UID>, carry the ClusterSummary UID label and namespace/name annotations, and are removed when monitoring is disabled, when nothing is left to monitor, or when the ClusterSummary is deleted. Removal only touches objects carrying that ClusterSummary's UID label, so a user's HealthCheck is never deleted.Health monitoring is reconciled before
shouldReconcile, so it works for a OneTime ClusterSummary with nothing left to deploy. The ClusterSummary update predicate reacts to features becoming Provisioned and to changes in the managed Helm releases.DryRun ClusterSummaries are skipped.
A HealthCheck only evaluates objects that exist: a deleted resource is not reported (drift detection covers that).
Helm chart resources outside the release namespace are only partly matched.
sveltos-agent caches every object of each monitored kind in the cluster, about 20 KB of heap per object, independent of how many profiles enable monitoring.