CI: Improve CRE Matrix

[kalverra]

Intent

Remove static test matrix building in CI. Convert CI tools and scripts to a single, well-tested Go CLI (tools/ci) instead of bash and yaml: test matrix discovery via AST, package sharding, changelog formatting, and integration-tests gating decisions all live under one ci command with unit tests. CRE smoke tests are renamed to a TestCRE_*_E2E convention so matrix discovery is pattern-based instead of grep-based.

Big Changes

Unified tools/ci Go CLI

New standalone Go module (tools/ci, cobra-based) replaces tools/ci-testshard and tools/ci/format_changelog, and absorbs the bash/jq/grep matrix building previously embedded in workflows. Subcommands: matrix (AST discovery of Test/Example functions + per-test topology/config mapping + JSON output), matrix setup (all matrices needed by integration-tests in one call), testshard list|verify (deterministic FNV sharding, ported 1:1 with tests), changelog format (changeset grouping/PR-body generation, parity with the old bash script), and gating (event/label-driven gate decisions + step-summary table). Dependency tree kept lean (fang removed; cobra + stdlib + testify only).

WHY: The old bash/jq/grep pipelines were fan-out points for silent breakage (regex drift between yaml and test names), impossible to unit test, and duplicated across four workflows. A CLI centralizes the logic and lets CI and local runs share one tested source of truth.

test_matrix passthrough + consolidated gating

Every system-test workflow now accepts a pre-computed test_matrix input; when supplied, the per-workflow define-test-matrix job is skipped and the matrix is consumed directly (inputs.test_matrix != '' && inputs.test_matrix || needs.define-test-matrix.outputs.matrix). integration-tests.yml collapses the three per-suite setup jobs into one test-setup job: changes + label signals feed the ci gating command, which emits all gate decisions (cre/ccip/regression/mixed-env/image builds), publishes a step-summary table, and matrix setup pre-computes all four matrices in a single step.

WHY: Gating logic previously existed in two shapes (inline bash + duplicated "form inputs" jobs) and matrices were recomputed separately per workflow. One gate job = one place to read decisions; passthrough input lets future callers (runs-on pre-scaling, batch dispatch) compute matrices without workflow duplication.

CRE test rename + AST-based discovery with per-test topologies

All CRE smoke tests renamed from Test_CRE_* to TestCRE_*_E2E; the CLI discovers them via go/parser AST (no grep), and perTestTopologies maps each test to its topology/config — Bucket B expands to four entries (base + 3 vault topologies), Solana/Aptos/Stellar/Sharding/Multi-Gateway to their dedicated configs, everything else to the default gateway-capabilities topology. TestMain and unit tests are excluded by pattern + explicit filter.

WHY: Grep-based discovery (grep -oP '^func \K(Test|Example)...') broke on formatting edge cases and silently drifted from renames. AST parsing is deterministic and testable, and the _E2E suffix makes the CI contract explicit. Multi-topology expansion restores the vault-config coverage (jwt-auth, optimizations, stall-purge) that the single-default-topology mapping would otherwise drop from CI.

Solana setup action pinning

setup-solana action now takes version/shasum inputs (defaults pinned), verifies the install script checksum in-line, and keys the CI cache on version+shasum instead of a repo file hash.

WHY: Repository-path coupling made the cache miss on unrelated edits; moving constants into inputs makes the pinning explicit and version-bumpable from callers.

Small Changes

• cre-mixed-env-tests.yaml and cre-regression-system-tests.yaml fallback matrices now call ci matrix --suite=... instead of duplicated jq/grep (removes the second copy of the test list and the per-test config map).
• Shared internal/paths (repo-root/cwd-relative resolution) and internal/githuboutput ($GITHUB_OUTPUT appending incl. delimited vars) helpers replace three duplicated implementations; both unit-tested.
• test_id is numeric across all suites (smoke previously embedded the test name); artifact naming unchanged.
• Multi-gateway test restores strict skip semantics unless TOPOLOGY_NAME contains multi-gateway.
• Doc comments/READMEs updated to the renamed tests (^TestCRE_ run patterns, HTTP-action run commands, gRPC source comments).

---

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

[github-actions]

👋 kalverra, thanks for creating this pull request!

To help reviewers, please consider creating future PRs as drafts first. This allows you to self-review and make any final changes before notifying the team.

Once you're ready, you can mark it as "Ready for review" to request feedback. Thanks!

[github-actions]

✅ No conflicts with other open PRs targeting develop

[trunk-io]

   

View Full Report ↗︎ ⋅ Docs

[Copilot]

Pull request overview

Risk Rating: HIGH

This PR migrates several CI utilities from ad-hoc bash/YAML logic into a dedicated, tested Go CLI (tools/ci) and updates workflows + CRE smoke tests to align with a new dynamic matrix discovery approach.

Changes:

• Added a new tools/ci Go module/CLI with subcommands for matrix discovery, test package sharding, and changelog formatting (plus unit tests).
• Updated GitHub workflows and scripts to call the new CLI instead of legacy scripts/tools.
• Renamed CRE smoke tests to a consistent TestCRE_..._E2E convention to support automated discovery.

Scrupulous human review recommended (high impact areas):

• .github/workflows/cre-system-tests.yaml end-to-end contract between “matrix generation output” and downstream job inputs (especially CRE topology/config/environment startup).
• CRE smoke test topology/config behavior when TOPOLOGY_NAME is unset (now common with the new matrix approach).
• Workflow changes for changeset/changelog generation to ensure outputs and artifact behavior match expectations.

Reviewed changes

Copilot reviewed 27 out of 29 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
tools/ci/wait-for-containers-to-stop.sh	Removed legacy bash helper (migration to Go tooling).
tools/ci/main.go	New Go CLI entrypoint for CI tooling.
tools/ci/main_test.go	Smoke tests for root CLI help + major subcommands.
tools/ci/internal/cmd/root.go	Cobra root command wiring + fang execution.
tools/ci/internal/cmd/matrix.go	matrix command wrapper over matrix discovery/output.
tools/ci/internal/cmd/testshard.go	testshard command wiring for list/verify.
tools/ci/internal/cmd/changelog.go	changelog format command wrapper.
tools/ci/internal/matrix/matrix.go	AST-based Go test discovery + matrix JSON output.
tools/ci/internal/matrix/matrix_test.go	Unit tests + a “real directory” scan test for matrix discovery.
tools/ci/internal/testshard/shard.go	Test package sharding + verification helpers.
tools/ci/internal/testshard/shard_test.go	Unit tests for sharding/list/verify behavior.
tools/ci/internal/changelog/format.go	Go implementation of changelog grouping + PR body output generation.
tools/ci/internal/changelog/format_test.go	Unit tests for changelog formatting + truncation behavior.
tools/ci/install_stellar	Removed legacy bash installer.
tools/ci/install_solana	Removed legacy bash installer.
tools/ci/format_changelog	Removed legacy bash changelog formatter.
tools/ci/go.mod	New nested Go module for the CI CLI.
tools/ci/go.sum	Dependency lockfile for the new CI CLI.
tools/ci-testshard/main.go	Removed legacy standalone testshard tool (replaced by tools/ci testshard).
tools/ci-testshard/main_test.go	Removed legacy tests for the standalone tool.
tools/bin/go_deployment_tests	Updated to call tools/ci testshard instead of the removed tool.
system-tests/tests/smoke/cre/multi_gateway_http_action_test.go	Renamed test to TestCRE_..._E2E + updated topology skip behavior.
system-tests/tests/smoke/cre/grpc_source_test.go	Renamed tests/comments to TestCRE_..._E2E.
system-tests/tests/smoke/cre/cresettings_override_test.go	Renamed test to TestCRE_..._E2E.
system-tests/tests/smoke/cre/cre_suite_test.go	Renamed suite tests + adjusted bucket/topology handling.
.github/workflows/cre-system-tests.yaml	Switched matrix generation from bash/jq to tools/ci matrix and adjusted downstream usage.
.github/workflows/cre-mixed-env-tests.yaml	Updated test name list to the new naming convention.
.github/workflows/changesets-preview-pr.yml	Switched changelog generation to tools/ci changelog format (after pnpm changeset version).
.github/actions/setup-solana/action.yml	Inlined Solana install script download + checksum verification; removed dependency on removed bash script.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 30 changed files in this pull request and generated no new comments.

Suppressed comments (3)

system-tests/tests/smoke/cre/multi_gateway_http_action_test.go:114

• The skip condition contradicts the preceding comment ("Skips unless TOPOLOGY_NAME contains "multi-gateway"") and the prior behavior: with TOPOLOGY_NAME unset/empty, this test will now run instead of skipping. That can cause unexpected failures when running the suite without explicitly selecting the multi-gateway topology.

// Skips unless TOPOLOGY_NAME contains "multi-gateway".
func TestCRE_V2_HTTP_Action_Multi_Gateway_E2E(t *testing.T) {
	if topology != "" && !isMultiGatewayTopology(topology) {
		t.Skipf("skipping multi-gateway HTTP action test: TOPOLOGY_NAME=%q does not match %q", topology, multiGatewayTopologyMarker)
	}

tools/ci/internal/testshard/shard.go:101

• ReadPackages already rejects duplicate package paths, so the seen map and the second loop checking seen[pkg] != 1 are unreachable and add unnecessary work/complexity. Verify can compute shard sizes without tracking duplicates here.

	for _, pkg := range packages {
		if seen[pkg] != 1 {
			return fmt.Errorf("package %q assigned %d times", pkg, seen[pkg])
		}
	}

.github/workflows/integration-tests.yml:253

• This step sets REF_NAME and REF_TYPE in the env: block, but then the script ignores them and re-reads GITHUB_REF_NAME / GITHUB_REF_TYPE. That makes the gating logic depend on GitHub-provided env vars instead of the explicitly provided values, and can lead to empty REF_NAME/REF_TYPE if those GitHub env vars are not present in a given event context.

          EVENT="${EVENT_NAME:-}"
          REF_TYPE="${GITHUB_REF_TYPE:-}"
          REF_NAME="${GITHUB_REF_NAME:-}"

[Copilot]

Pull request overview

Copilot reviewed 30 out of 32 changed files in this pull request and generated 2 comments.

Suppressed comments (5)

.github/workflows/integration-tests.yml:290

• Same issue as above: the CCIP gate uses parentheses without whitespace, which can break the [[ ... ]] expression parsing in bash.

          if [[ "$EVENT" == "workflow_dispatch" ]] || \
             [[ "$EVENT" == "push" && ("$CCIP_CHANGED" == "true" || "$REF_TYPE" == "tag") ]] || \
             [[ "$EVENT" == "merge_group" && "$CCIP_CHANGED" == "true" ]]; then

system-tests/tests/smoke/cre/grpc_source_test.go:51

• The local run command still references the old test name (Test_CRE_GRPCSource_Lifecycle) and won’t match after the rename.

// To run locally:
//  1. Start the test (it will start the environment automatically):
//     go test -timeout 20m -run "^Test_CRE_GRPCSource_Lifecycle$" ./smoke/cre/...

system-tests/tests/smoke/cre/cre_suite_test.go:36

• The instructions for running this suite locally still use the old test-name prefix ("^Test_CRE_"). After the renames, it should match the new "TestCRE_" prefix.

To execute tests start the local CRE first:
 1. Inside `core/scripts/cre/environment` directory: `go run . env restart --with-chip-ingress-stack` (deprecated: `--with-beholder`)
 2. Execute the tests in `system-tests/tests/smoke/cre`: `go test -timeout 15m -run "^Test_CRE_"`.
*/

system-tests/tests/smoke/cre/multi_gateway_http_action_test.go:113

• The comment immediately above this test still references the old function name (Test_CRE_V2_HTTP_Action_Multi_Gateway), which can make local runs/discovery confusing after the rename.

func TestCRE_V2_HTTP_Action_Multi_Gateway_E2E(t *testing.T) {

system-tests/tests/smoke/cre/cresettings_override_test.go:48

• The local run command in the header comment still uses the old test name (Test_CRE_CRESettings_Override) and won’t match after the rename to TestCRE_CRESettings_Override_E2E.

func TestCRE_CRESettings_Override_E2E(t *testing.T) {

[Copilot]

Pull request overview

Copilot reviewed 42 out of 44 changed files in this pull request and generated no new comments.

Suppressed comments (3)

tools/ci/internal/githuboutput/githuboutput.go:31

• AppendMultilineVar writes GitHub Actions output using a hard-coded EOF delimiter. If the value contains a line equal to EOF, the output file becomes malformed and downstream steps.<id>.outputs parsing can break. GitHub recommends using a unique delimiter that does not appear in the value (often generated per write).
  system-tests/tests/smoke/cre/multi_gateway_http_action_test.go:111
• The doc comment contradicts the code: it says the test “runs without an explicit … skip”, but the function immediately calls t.Skipf when the topology marker is missing. Rewording would avoid confusion for anyone running this locally.

// Skips unless TOPOLOGY_NAME contains "multi-gateway": the scenario requires the two-gateway
// link topology, so runs without an explicit multi-gateway TOPOLOGY_NAME skip.

system-tests/tests/smoke/cre/grpc_source_test.go:52

• The “To run locally” command uses ./smoke/cre/..., but this package lives under ./system-tests/tests/smoke/cre/.... As written, the command won’t match any packages when run from the repo root.

// To run locally:
//  1. Start the test (it will start the environment automatically):
//     go test -timeout 20m -run "^TestCRE_GRPCSource_Lifecycle_E2E$" ./smoke/cre/...
//

[github-actions]

CORA - Pending Reviewers

Codeowners Entry	Overall	Num Files	Owners
*	⏳	35	@smartcontractkit/foundations, @smartcontractkit/core
/integration-tests/**/*ccip*	⏳	3	@smartcontractkit/ccip-offchain, @smartcontractkit/core, @smartcontractkit/ccip
/.github/**	⏳	7	@smartcontractkit/devex-cicd, @smartcontractkit/devex-tooling, @smartcontractkit/core
go.mod	⏳	1	@smartcontractkit/core, @smartcontractkit/foundations
go.sum	⏳	1	@smartcontractkit/core, @smartcontractkit/foundations

Legend: ✅ Approved | ❌ Changes Requested | 💬 Commented | 🚫 Dismissed | ⏳ Pending | ❓ Unknown

For more details, see the full review summary.

[cl-sonarqube-production]

Quality Gate failed

Failed conditions
1 Security Hotspot
2 New Blocker Issues (required ≤ 0)
E Security Rating on New Code (required ≥ A)

See analysis details on SonarQube

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE