feat(ci): report clusteralerts fired during release rollover - #2686
Draft
universal-itengineer wants to merge 6 commits into
Draft
feat(ci): report clusteralerts fired during release rollover#2686universal-itengineer wants to merge 6 commits into
universal-itengineer wants to merge 6 commits into
Conversation
The release e2e pipeline had no visibility into alerts of the module itself: nothing in the repo ever looked at ClusterAlert objects, and the rules in monitoring/prometheus-rules were only covered by promtool unit tests. A dedicated job now watches the nested cluster for the whole test and upgrade sequence and collects every firing D8Virtualization* alert. A ClusterAlert object exists only while the alert fires, so the watch polls instead of taking a single snapshot at the end. It stops on a ConfigMap marker placed in the nested cluster by a separate always-running job: runners share no filesystem, and the marker must also appear when test-new-release never started. The report job renders a table into the job summary, emits a warning annotation per alert and exits non-zero, which paints it red in the UI. Being continue-on-error, it leaves the workflow conclusion successful. Alerts are attributed to pre-upgrade, upgrade or post-upgrade by comparing the observation time with the upgrade window, so an alert that was already firing before the rollover is not mistaken for its result. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
The report job only depended on the upgrade job and on the watch, so in the run graph it sat next to the new-release tests instead of after them. Worse, if the watch job died early - on the kubeconfig step, say - the report started while the new-release tests were still running and declared that no alerts had been firing. It now depends on the test jobs as well, and an empty result is reported as "not monitored" instead of "nothing was firing" when the watch job did not succeed. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Pin the release-upgrade defaults to the versions under test and delete the nested cluster right after the ClusterAlerts report, so a test run does not hold the cluster until the nightly cleanup. Drop this commit before merge. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
universal-itengineer
force-pushed
the
chore/ci/check-clusteralerts-during-overroling-release
branch
from
August 10, 2026 18:52
d647fce to
3eeef5f
Compare
The release e2e config hardcoded a feature gate that the 1.9 release line never shipped, so ModulePullOverride validation failed and the module was never installed: the pipeline only reported that it timed out waiting for virtualization to become ready. Gates are now derived from the release under test. The upgrade narrows the list to what both releases support before switching the image tag, then enables everything the new release supports once its images are running, so each release is tested with all the gates it has. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
The release pipeline waited for one Evict operation per running virtual machine after every upgrade, and failed after 20 minutes when none appeared. Two releases that ship the same virt-handler and virt-launcher never move a virtual machine, so the wait could not be satisfied and reported a timeout instead of an upgrade that simply had nothing to migrate. The expectation is now derived from the workload image digests of both releases: unchanged images end the step with a message, and anything that cannot be determined keeps the previous waiting behaviour. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Runners force these actions onto Node.js 24 already and annotate every run with a deprecation warning. Moved each one to its first major built for Node.js 24: cache v4 to v5, setup-go v5 to v6, github-script v7 to v8, setup-kubectl and k8s-set-context v4 to v5. Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The release e2e pipeline had no visibility into alerts of the module itself: nothing in the repo ever looked at ClusterAlert objects, and the rules in monitoring/prometheus-rules were only covered by promtool unit tests.
A dedicated job now watches the nested cluster for the whole test and upgrade sequence and collects every firing D8Virtualization* alert. A ClusterAlert object exists only while the alert fires, so the watch polls instead of taking a single snapshot at the end. It stops on a ConfigMap marker placed in the nested cluster by a separate always-running job: runners share no filesystem, and the marker must also appear when test-new-release never started.
The report job renders a table into the job summary, emits a warning annotation per alert and exits non-zero, which paints it red in the UI. Being continue-on-error, it leaves the workflow conclusion successful. Alerts are attributed to pre-upgrade, upgrade or post-upgrade by comparing the observation time with the upgrade window, so an alert that was already firing before the rollover is not mistaken for its result.
Checklist
Changelog entries