Skip to content

Add prometheus metrics - #248

Open
alicefr wants to merge 8 commits into
bootc-dev:mainfrom
alicefr:prometheus
Open

alicefr wants to merge 8 commits into
bootc-dev:mainfrom
alicefr:prometheus

Conversation

@alicefr

@alicefr alicefr commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

Add prometheus metrics for better monitoring of bootc-operator operation. Added metrics:

  • bootcoperator_pool_nodes_total
  • bootcoperator_pool_nodes_updated
  • bootcoperator_pool_nodes_updated
  • bootcoperator_pool_nodes_updated
  • bootcoperator_pool_nodes_updated
  • bootcoperator_node_booted_image_info

Fixes: #102

Expose pool-level gauges (nodes total, updated, updating, degraded,
rollout paused) and a node-level info metric for the booted OS image.
The collector reads from the controller-runtime cache on each scrape.

Configure the metrics server with HTTPS and authentication/authorization
on port 8443, with flags for bind address and TLS cert directory.

Assisted-by: AI
Signed-off-by: Alice Frosi <afrosi@redhat.com>
Cover pool gauges, paused/not-paused rollout, node info metric,
orphan nodes, missing booted status, multiple pools, and empty cache.

Assisted-by: AI
Signed-off-by: Alice Frosi <afrosi@redhat.com>
Add kustomize and Helm chart resources for the metrics endpoint:
a Service targeting port 8443, an opt-in ServiceMonitor, RBAC for
the operator SA to perform token/access reviews, and a metrics-reader
ClusterRole for Prometheus scraping.

Assisted-by: AI
Signed-off-by: Alice Frosi <afrosi@redhat.com>
Extend the envtest suite to expose the manager cache and add two
tests that create pools and nodes against the envtest API server,
then verify the collector emits the expected metrics.

Assisted-by: AI
Signed-off-by: Alice Frosi <afrosi@redhat.com>
Create a ServiceAccount with the metrics-reader ClusterRole and run
a curl pod to scrape the metrics endpoint from inside the cluster,
the same way Prometheus would. Verify pool and node metrics are
present after a pool reaches steady state.

Assisted-by: AI
Signed-off-by: Alice Frosi <afrosi@redhat.com>
Document how to configure the monitoring using Prometheus and Grafana.

Assisted-by: AI
Signed-off-by: Alice Frosi <afrosi@redhat.com>
The released operator binary does not support --metrics-bind-address,
so deploying it with the current Helm chart crashes the operator.
Gate the flag and metrics port in sync-helm.sh on metrics.enabled,
and disable metrics when installing the released image in the upgrade
test. Also filter --metrics-bind-address from controllerExtraArgs to
prevent duplication.

Assisted-by: AI
Signed-off-by: Alice Frosi <afrosi@redhat.com>
The curl pod used to scrape the metrics endpoint was scheduled on
worker nodes, where Calico CNI is not fully initialized. This caused
FailedCreatePodSandBox errors and curl failures on K8s 1.36 and 1.37,
while 1.35 passed only due to timing.

Pin the curl pod to the control-plane node with a nodeSelector and
toleration so it runs where networking is stable and the metrics
service is reachable.

Assisted-by: AI
Signed-off-by: Alice Frosi <afrosi@redhat.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create prometheus metrics for the pool

1 participant