Document that scale set metric labels come from Prometheus relabeling - #4627
Document that scale set metric labels come from Prometheus relabeling#4627nikola-jokic wants to merge 2 commits into
Conversation
The Grafana dashboard sample filters on actions_github_com_scale_set_name and actions_github_com_scale_set_namespace, which are not emitted by ARC. They are produced by Prometheus relabeling of the listener pod labels. Document the required scrape configuration, clarify that adding these labels to listenerMetrics results in empty values, and add a troubleshooting section for empty panels. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Hello! Thank you for your contribution. Please review our contribution guidelines to understand the project's testing and code conventions. |
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The new documentation includes a Prometheus scrape example and troubleshooting guidance that contains a few concrete inaccuracies/misleading bits (notably recommending __meta_kubernetes_* labels for Grafana queries and a scrape config pattern that can fail in the described empty-label scenario).
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Lite
Findings: 2
New issues introduced by this change (3)
| Severity | Finding |
|---|---|
 |
docs/gha-runner-scale-set-controller/samples/grafana-dashboard/README.md — honor_labels: true prevents target labels produced by relabel_configs from overriding a… |
|
docs/gha-runner-scale-set-controller/samples/grafana-dashboard/README.md — The labelmap relabeling (__meta_kubernetes_pod_label_(.+)) copies all pod labels onto every… |
 |
docs/gha-runner-scale-set-controller/samples/grafana-dashboard/README.md — This example uses actions.github.com/scale-set-namespace=arc-runners in the label snippet above,… |
What changed in this PR
This PR updates the Grafana dashboard sample documentation to clarify that the actions_github_com_scale_set_name and actions_github_com_scale_set_namespace labels used by the dashboard are derived from Prometheus relabeling of listener pod labels (not emitted by ARC itself), and to help users troubleshoot empty dashboard panels caused by missing/empty scale set labels.
Changes:
- Adds a new section explaining how Prometheus maps listener pod labels into metric labels (with an example relabeling config).
- Adds guidance warning users not to add
actions_github_com_scale_set_*labels tolistenerMetrics(as they will be exported as empty values). - Adds a troubleshooting section and a note in the required metrics table pointing to the new explanation.
| File | Description |
|---|---|
| docs/gha-runner-scale-set-controller/samples/grafana-dashboard/README.md | Documents that scale set metric labels come from Prometheus relabeling and adds troubleshooting guidance for empty Grafana panels |
Suppressed comments (2)
docs/gha-runner-scale-set-controller/samples/grafana-dashboard/README.md:53
- The scrape config namespace list should match the example scale set namespace shown earlier (currently
arc-runnersin the label example).
- arc-systems
docs/gha-runner-scale-set-controller/samples/grafana-dashboard/README.md:81
__meta_kubernetes_pod_label_...labels are Prometheus service-discovery metadata labels and aren’t queryable in PromQL/Grafana (labels starting with__are dropped before ingestion). Suggesting users switch the dashboard to__meta_kubernetes_pod_label_actions_github_com_scale_set_namespacewill not work.
- Replace `actions_github_com_scale_set_name` and `actions_github_com_scale_set_namespace`
in your copy of the dashboard with labels that your setup already produces, such as
`namespace`, or the raw
`__meta_kubernetes_pod_label_actions_github_com_scale_set_namespace` metadata label.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The Prometheus scrape config example is easy to misapply as a complete job and could unintentionally scrape every pod in the namespace, so it should be clarified or constrained.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Lite
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
|
docs/gha-runner-scale-set-controller/samples/grafana-dashboard/README.md — The Prometheus snippet is presented as a full scrape_configs job (role: pod + namespace… |
Issues resolved since last review (3)
| Severity | Finding |
|---|---|
|
docs/gha-runner-scale-set-controller/samples/grafana-dashboard/README.md — This example uses actions.github.com/scale-set-namespace=arc-runners in the label snippet above,… View resolved comment |
|
docs/gha-runner-scale-set-controller/samples/grafana-dashboard/README.md — The labelmap relabeling (__meta_kubernetes_pod_label_(.+)) copies all pod labels onto every… View resolved comment |
|
docs/gha-runner-scale-set-controller/samples/grafana-dashboard/README.md — honor_labels: true prevents target labels produced by relabel_configs from overriding a… View resolved comment |
| The scrape configuration has to copy those metadata labels onto the scraped metrics. The | ||
| example below copies only the two labels the dashboard needs, scoped to the `arc-systems` | ||
| namespace: |
Fixes #4348
The Grafana dashboard sample filters several panels on
actions_github_com_scale_set_nameandactions_github_com_scale_set_namespace. These labels are not emitted by ARC — they are produced by Prometheus from the listener pod labels (actions.github.com/scale-set-name,actions.github.com/scale-set-namespace) via alabelmaprelabeling of__meta_kubernetes_pod_label_(.+).Users who add those label names to
listenerMetricsinvalues.yamlend up with empty label values, since the listener only knowsname,namespace,repository,organization,enterprise, and the job labels.This documents the explanation @kenmuse gave in #4348 (comment):
Startup Duration/Job Execution/Running JobspanelsDocs-only change.