[swicken]

Proposed Changes

When an integration test fails, failsafe retries it up to three times (rerunFailingTestsCount in dotcms-integration/pom.xml). The retry should repeat only the failed methods, but for tests in the MainSuite* suites it repeated every method of the failing class. For example, one failed WebAssetHelperIntegrationTest method reran all 37 of its methods in merge-queue run 40712603717. A failed TagAPITest method shows up as all ten of its methods "flaking".

• Failsafe's JUnit 4 provider runs the retry as Request.aClass(<suite>).filterWith(<failed test descriptions>).
• MainBaseSuite wraps each class's runner in DotRunner, which extended Runner without implementing Filterable. JUnit's Filter.apply silently skips children that are not Filterable, so the filter kept the failing class but never narrowed it to the failed methods.
• DotRunner now implements Filterable and passes the filter to the runner it wraps. That runner, CustomDataProviderRunner, already filters per method and per data-provider row. Runs without a filter are unchanged.

Fixes #38308

Addendum: fail fast only in the merge queue (.github/workflows/cicd_comp_test-phase.yml)

The test matrix used to cancel every other suite as soon as one failed, unless the PR carried the CI : No Fail Fast label. That hides evidence:

• One flaky suite turned the other ~24 jobs into cancellations, which look like failures at a glance.
• A push only ever showed its first failure.

With Enterprise concurrency, and free standard minutes on a public repo, the runner time fail-fast saved is no longer worth that.

• Merge queue: fail-fast now applies only to merge_group runs. A failing group should be ejected quickly so the groups behind it rebuild.
• Everywhere else: pull requests, trunk, nightly and LTS runs let every suite finish, so the label is no longer needed.
• Phased OpenSearch runs still run to completion in every context, as before.

Checklist

• Tests
• Translations
• Security Implications Contemplated (add notes if applicable)

Additional Info

Verified locally with a throwaway suite, not committed, in the MainBaseSuite style. It held one plain class and one data-provider class, each with a method that fails on its first run.

• Before the change: the retry ran all 4 of 4 methods in each class (14 tests, 2 flakes).
• After the change: it ran only the failed method and the failed provider row (8 tests, 2 flakes).
• Normal run: a suite of SystemTableFactoryTest, ObjectMapperTest and TagAPITest ran the same 3, 2 and 45 tests as before.

The JUnit 5 suites (Junit5Suite*) need no change. The JUnit Platform provider already retries by each failed test's unique ID, which was checked the same way (details in #38308).

This changes only the cost of a retry, not whether retries happen. A test that passed its retry only because the rest of its class ran again will now fail. That means its methods depend on each other, which is worth seeing.

This PR fixes: #38308

[claude]

Claude finished @swicken's task in 59s —— View job

---

Code Review

Task

• Read both changed files and the diff against the PR commits
• Trace the filter delegation chain (DotRunner → CustomDataProviderRunner → ParentRunner)
• Review the workflow fail-fast change
• Post review

---

Verdict

No blocking issues. The fix is correct and minimal.

Analysis

MainBaseSuite.java — the actual fix

The root cause is accurate: JUnit's Filter.apply(Object child) only narrows children that are Filterable; a plain Runner is silently left whole. Since DotRunner extends Runner without Filterable, failsafe's rerun (Request.aClass(suite).filterWith(...)) kept the failing class but ran all its methods.

The delegation chain checks out:

• DotRunner.filter(filter) calls filter.apply(this.runner) where this.runner is a CustomDataProviderRunner.
• CustomDataProviderRunner extends DataProviderRunner → BlockJUnit4ClassRunner → ParentRunner, and ParentRunner implements Filterable. So Filter.apply dispatches to ParentRunner.filter, which prunes its children to the matching methods/data-provider rows and throws NoTestsRemainException if none match.
• DotRunner.filter rethrows NoTestsRemainException, which the enclosing Suite handles by dropping the runner — the correct behavior.

Normal, unfiltered runs never invoke filter, so they're untouched. This matches the before/after evidence in the PR description.

cicd_comp_test-phase.yml — fail-fast scoping

fail-fast now evaluates to true only on merge_group with no OpenSearch phase, and false everywhere else. The expression is well-formed and the behavior matches the commit message (queue ejects fast; PR/trunk/nightly/LTS/phased runs complete for full evidence). No issue. Note: I can't modify files under .github/workflows/ even if a change were wanted (GitHub App permission), but none is.

Notes (non-blocking)

• As the PR body itself calls out, a test that only passed on retry because its whole class reran again will now surface as a real failure. That's a desirable behavior change, not a regression — worth watching the first few merge-queue runs to confirm nothing was quietly leaning on intra-class ordering.

No issues found.
· branch issue-38308-failsafe-method-rerun

[yolabingo]

✅

[sfreudenthaler]

pipeline go burrrrrrr

[wezell]

Let's see this work.

[wezell]

or not :P

[wezell]

Let's see the flakes!