feat(ci): run integration suites under an ES→OS migration phase (#36320)

[fabrizzio-dotCMS]

Proposed Changes

Makes it possible to run the full integration battery under a chosen ES→OS migration phase — on a single phase or swept across all three — locally, per-PR, on-demand, and automatically on a schedule. Contributes to #36320 ("Make it possible to run MainSuite Tests in all 3 phases").

The opensearch-phase-suite Maven profile already existed (two clusters ES @9207 + OS 3.x @9201, sets DOT_FEATURE_FLAG_OPEN_SEARCH_PHASE, inherits the base suite includes). This PR wires that capability into every layer of the tooling. It does not change any product code, and default (phase 0) behavior is untouched everywhere.

Four ways to run a phased battery

Path	Trigger	Scope
Local	just test-integration-phase <N> / test-integration-phase-per-suite <N>	single phase
Per-PR (opt-in)	PR label OpenSearch:Phase-<N>	single phase
On-demand	Nightly workflow_dispatch → opensearch_phase dropdown (none/1/2/3/all)	single phase or sweep
Automatic	New weekly scheduled workflow (Sun 06:00 UTC)	sweep (all 3)

Local (justfile)

• test-integration-phase <N> [TEST] — full battery in a single JVM (one docker bring-up), or a narrowed run with a test arg.
• test-integration-phase-per-suite <N> — iterates the MainSuite/Junit5 suites one at a time (forkCount=1) under phase <N>, giving attributable per-suite pass/fail plus a summary (non-zero exit if any suite fails). This is the local triage tool for de-ES-ifying the ITs. (renamed from test-integration-phase-all — the sweep is per-suite, not per-phase.)

CI — shared engine (cicd_comp_test-phase.yml)

• New opensearch_phase input now accepts a single phase (2), a comma list (1,2,3), or the alias all (= 1,2,3).
• The matrix generator fans each integration MainSuite/Junit5 entry out once per requested phase, appending -Dopensearch.phase=<N> and suffixing name / stage_name per phase so job labels and the resulting build-reports-<stage_name> artifacts stay unique. The OpenSearch Upgrade Suite is never phased (it already drives OS via -Dopensearch.upgrade.test; combining the two profiles would collide their endpoint config).
• fail-fast is now conditional: kept true for normal runs, false for a phased sweep so every suite/phase runs to completion and failures are attributable per phase.
• No-op when off: empty / none / 0 yields an empty phase list → the matrix is byte-for-byte identical to today. Verified against the other callers that don't opt in (cicd_2-merge-queue, cicd_3-trunk, cicd_5-lts).

CI — per-PR (cicd_1-pr.yml)

• A PR carrying an OpenSearch:Phase-<N> label runs the integration battery under that phase (mirrors the existing PR: docker image label pattern). No label → phase 0 (current behavior). (The label family was aligned with the repo's OpenSearch area label — see Repo labels below.)

CI — on-demand (cicd_4-nightly.yml)

• workflow_dispatch opensearch_phase dropdown: default none (clearer than 0), options none/1/2/3/all.
• Any active phase value (1/2/3/all) skips the deployment job — a build exercised under a non-default migration phase is never promoted. The scheduled nightly (03:18) is unaffected: it runs with none.

CI — automatic (cicd_scheduled_opensearch-phase-sweep.yml, new)

• Weekly (Sun 06:00 UTC, off-peak and clear of the daily nightly): sweeps the integration battery across all 3 phases automatically, so migration-flow regressions are caught without any developer opting a PR in.
• Test-only: initialize → build → test → finalize → report. No deploy, no CLI build. Only the integration test type is enabled (other suites don't vary by phase). workflow_dispatch also allows picking a single phase.
• Serialized via a concurrency group (cancel-in-progress: false): only one sweep runs at a time — a manual dispatch fired while the weekly run is going is queued rather than run in parallel (a sweep is expensive and its result matters).
• Scheduled triggers only fire from the default branch, so it stays dormant until merged to main.

Repo labels (created out-of-band)

• Renamed the area label OpenSearch-Migration → OpenSearch (assignments preserved; nothing references it in code).
• Created the CI-trigger family OpenSearch:Phase-1 / -2 / -3 used by the per-PR trigger.

Checklist

• YAML validated; matrix-generation logic simulated against the real test-matrix.yml: none/''/0 → 7 entries / 0 phased (unchanged); 2 → 6 phased + upgrade untouched; all → 18 phased + upgrade, all 19 stage_names unique (no artifact collision).
• Shared-engine change confirmed no-op for callers that don't pass opensearch_phase (merge-queue, trunk, lts).
• Smoke-verified locally: just test-integration-phase 1 ContentletIndexAPIImplTest brings up both clusters, bootstraps OS under PHASE_1_DUAL_WRITE_ES_READS, and runs the suite.
• No product code changed (CI/tooling only).

Notes for reviewers

• The OpenSearch:Phase-<N> labels already exist in the repo (created alongside this PR) so the per-PR trigger is selectable.
• Impact of the new scheduled workflow on other jobs is minimal: it's independent and test-only (no shared state, no deploy, per-run artifacts); the only shared surface is cicd_comp_test-phase.yml, whose change is a verified no-op when the phase is off. The material cost is ~19 long integration jobs once a week, off-peak Sunday (no cron collision with the pentest/rolling-tags/nightly schedules).

Related: #36320, PR #36268 / #36266 (added the opensearch-phase-suite profile).

[claude]

Claude finished @fabrizzio-dotCMS's task in 1m 51s —— View job

---

Code Review — phased integration suite (CI/tooling only)

I reviewed the diff against origin/main. This is CI/tooling only, no product code. The matrix fan-out, the phaseable filter (correctly excludes the OpenSearch Upgrade Suite via the opensearch.upgrade.test marker), the PR-label expression, the nightly deploy-skip, and the justfile targets all check out. One latent inconsistency worth noting.

New Issues

• 🟡 Medium: .github/workflows/cicd_comp_test-phase.yml:231 — The fail-fast guard !contains(fromJSON('["1","2","3","all"]'), inputs.opensearch_phase) only recognizes the single-value / all forms, but the input's own description (lines 48–56) documents that a comma list like '1,2,3' is accepted, and the matrix generator (lines 133–141) does fan that out. For a comma-list input the matrix produces a multi-phase sweep, yet contains([...], '1,2,3') is false, so fail-fast stays true — the first failing suite would cancel the rest, defeating the stated "every suite/phase runs to completion, attributable per phase" goal. No current caller passes a comma list (PR label → single/''; nightly/sweep → choice), so this is latent, not live. Consider deriving the flag from "phase active" rather than an exact-match list, e.g. fail-fast: ${{ inputs.opensearch_phase == '' || inputs.opensearch_phase == 'none' || inputs.opensearch_phase == '0' }}. Fix this →

Notes (non-blocking, verified correct)

• cicd_comp_test-phase.yml:110-111 — passing opensearch_phase via env: rather than ${{ }} interpolation into the script is the right call for injection safety; the regex /^[1-3]$/ filter also hard-limits values.
• cicd_1-pr.yml:113-118 — the &&/|| label expression is the correct GitHub-expressions idiom (false && '1' → false, chain falls through to ''). Multiple OpenSearch:Phase-* labels resolve to the lowest phase; fine given the "single phase per PR" contract.
• cicd_4-nightly.yml:186 — deploy-skip if reads inputs.opensearch_phase, which is empty on the cron trigger → deployment proceeds as today; only manual 1/2/3/all skips it. Correct.
• The no-op claim for empty/none/0 holds: phaseList is empty → matrix is byte-identical, and fail-fast stays true.
• justfile test-integration-phase-per-suite uses set -uo pipefail (not -e) so the loop intentionally continues past a failing suite and exits non-zero via the failed array — correct for a triage sweep.

No blocking issues. The Medium item is a documentation/behavior mismatch that only bites if a future caller uses the comma-list form.