NO-JIRA: Exclude KubeJobFailed for periodic-gathering jobs in openshift-insights

[petr-muller]

Mitigates impact of OCPBUGS-77314 / CCXDEV-16087.

Summary

• The periodic-gathering jobs in the openshift-insights namespace can fail because of the cluster-external API failures, causing KubeJobFailed alerts that make the "shouldn't report any alerts in firing state" e2e test fail.
• Uses a PromQL unless clause to exclude specifically KubeJobFailed alerts from openshift-insights namespace where job_name matches periodic-gathering-.*, while still catching any other unexpected alerts.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Tests
  • Refined alert-exclusion logic in Prometheus test queries to more precisely ignore specific transient job alerts, with explanatory comments and a temporary TODO for future reversion.

[openshift-ci-robot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

[coderabbitai]

Walkthrough

A Prometheus test in test/extended/prometheus/prometheus.go changed alert-matching logic: the previous exclusion set was replaced by a PromQL unless clause that conditionally excludes KubeJobFailed alerts originating from periodic-gathering jobs in openshift-insights.

Changes

Cohort / File(s)	Summary
Prometheus test test/extended/prometheus/prometheus.go	Replaced a simple alert-name exclusion matcher with a PromQL unless clause to exclude KubeJobFailed alerts from periodic-gathering jobs in openshift-insights; added explanatory comments and a TODO.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Test Structure And Quality	⚠️ Warning	Test assertion at line 885 lacks meaningful failure message, violating criterion 4 requiring diagnostic context in assertions.	Add failure message to assertion: change o.Expect(err).NotTo(o.HaveOccurred()) to o.Expect(err).NotTo(o.HaveOccurred(), "failed to verify no unexpected Prometheus alerts are firing")

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically describes the main change: excluding KubeJobFailed alerts for periodic-gathering jobs in openshift-insights, which matches the core modification in the PromQL test expression.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Stable And Deterministic Test Names	✅ Passed	All test names in the modified prometheus.go file use stable and deterministic strings without dynamic information, with PromQL query modifications correctly placed in test bodies.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[petr-muller]

/hold

Should not be needed anymore, just wanted to have a record

[openshift-ci-robot]

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

[neisw]

/lgtm

[openshift-ci-robot]

@petr-muller: This pull request explicitly references no jira issue.

Details

In response to this:

Summary

• The periodic-gathering jobs in the openshift-insights namespace can transiently fail, causing KubeJobFailed alerts that make the "shouldn't report any alerts in firing state" e2e test fail.
• Uses a PromQL unless clause to exclude specifically KubeJobFailed alerts from openshift-insights namespace where job_name matches periodic-gathering-.*, while still catching any other unexpected alerts.
• This is the most targeted exclusion possible: only this specific alert + namespace + job name pattern is excluded. Any other KubeJobFailed (different namespace/job) or any other alert in openshift-insights will still fail the test.

Test plan

• Verify the modified PromQL query compiles and is syntactically valid
• Run the "shouldn't report any alerts in firing state" test on a cluster with a failing periodic-gathering job in openshift-insights and confirm it no longer causes a test failure
• Verify that other KubeJobFailed alerts (in different namespaces) still cause the test to fail

🤖 Generated with Claude Code

Summary by CodeRabbit

• Tests
• Updated Prometheus alert exclusion logic in test queries to refine alert handling for specific monitoring scenarios.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[neisw]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: neisw, petr-muller

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• test/extended/prometheus/OWNERS [neisw,petr-muller]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[petr-muller]

/hold cancel

[openshift-ci-robot]

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

[petr-muller]

No signs of trouble in the jobs

/verified by CI

[openshift-ci-robot]

@petr-muller: This PR has been marked as verified by CI.

Details

In response to this:

No signs of trouble in the jobs

/verified by CI

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-trt]

Job Failure Risk Analysis for sha: 18f455a

Job Name	Failure Risk
pull-ci-openshift-origin-main-e2e-metal-ipi-ovn-ipv6	IncompleteTests Tests for this run (25) are below the historical average (2350): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

[petr-muller]

/override ci/prow/e2e-aws-ovn-fips

The job actually failed due to TRT-2560

[openshift-ci]

@petr-muller: Overrode contexts on behalf of petr-muller: ci/prow/e2e-aws-ovn-fips

Details

In response to this:

/override ci/prow/e2e-aws-ovn-fips

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[petr-muller]

/override ci/prow/e2e-gcp-ovn-upgrade ci/prow/e2e-metal-ipi-ovn-ipv6

[openshift-ci]

@petr-muller: Overrode contexts on behalf of petr-muller: ci/prow/e2e-gcp-ovn-upgrade, ci/prow/e2e-metal-ipi-ovn-ipv6

Details

In response to this:

/override ci/prow/e2e-gcp-ovn-upgrade ci/prow/e2e-metal-ipi-ovn-ipv6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-ci]

@petr-muller: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.