optimize: Tweak Integration Tests Flow for Speed + Cost - #23408
Open
kalverra wants to merge 4 commits into
Open
Conversation
Contributor
|
✅ No conflicts with other open PRs targeting |
Contributor
|
I see you updated files related to
|
kalverra
force-pushed
the
DX-4988/integration-test-speedups
branch
2 times, most recently
from
August 14, 2026 13:31
fad0539 to
0484b20
Compare
kalverra
marked this pull request as ready for review
August 14, 2026 14:22
product-security-plaid-production
Bot
requested review from
Tofel,
jmank88,
patrickhuie19 and
skudasov
August 14, 2026 14:22
Contributor
There was a problem hiding this comment.
Pull request overview
Risk Rating: MEDIUM — Changes are primarily test/CI infrastructure, but they affect critical integration workflows, caching behavior, and WASM build mechanics; regressions here can block merges and inflate CI cost.
This PR focuses on speeding up and simplifying the integration/system test pipeline (runner usage, caching, and test setup), while also making system tests cheaper by using faster scrypt settings and consolidating matrix prep/setup.
Changes:
- Switch Go WASM compilation in tests/system-tests to
chainlink-common/pkg/wasmbuildand add/extend.wasm-cachecaching in GitHub Actions. - Update system-test crypto key encryption to use
FastScryptParamsand enableInsecureFastScryptin generated node configs (test-only). - Refactor integration workflows: remove
enforce-ctf-version, compute CRE matrix in setup, and replace customctf-setup-gowithactions/setup-go.
Scrupulous human review recommended (high-impact areas):
.github/workflows/integration-tests.yml(matrix generation, cache keys, and job dependencies/conditions)system-tests/tests/smoke/cre/vault_don_test_helpers.go(new propagation wait semantics)core/internal/testutils/wasmtest/wasm.go(test build/caching behavior now delegated towasmbuild)
Reviewed changes
Copilot reviewed 68 out of 76 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| system-tests/tests/smoke/cre/vault_don_test_helpers.go | Registry update wait now polls gateway |
| system-tests/tests/go.mod | Bump chainlink-common & brotli |
| system-tests/tests/go.sum | go.sum updates for bumps |
| system-tests/lib/go.mod | Bump chainlink-common & brotli |
| system-tests/lib/go.sum | go.sum updates for bumps |
| system-tests/lib/crypto/stellar.go | Use FastScryptParams |
| system-tests/lib/crypto/sol.go | Use FastScryptParams |
| system-tests/lib/crypto/p2p.go | Use FastScryptParams |
| system-tests/lib/crypto/dkgrecipient.go | Use FastScryptParams |
| system-tests/lib/crypto/aptos.go | Use FastScryptParams |
| system-tests/lib/crypto/evm.go | Import formatting cleanup |
| system-tests/lib/cre/workflow/workflow.go | Import formatting cleanup |
| system-tests/lib/cre/workflow/secrets.go | Import formatting cleanup |
| system-tests/lib/cre/workflow/registry.go | Import formatting cleanup |
| system-tests/lib/cre/workflow/compile.go | Use wasmbuild.Compile for Go workflows |
| system-tests/lib/cre/types.go | Import ordering cleanup |
| system-tests/lib/cre/sharding/sharding.go | Import formatting cleanup |
| system-tests/lib/cre/grpc_source_mock/server.go | Import ordering cleanup |
| system-tests/lib/cre/grpc_source_mock/auth.go | Update JWT authenticator construction |
| system-tests/lib/cre/grpc_source_mock/auth_test.go | Add interceptor test |
| system-tests/lib/cre/features/vault/vault.go | Import ordering cleanup |
| system-tests/lib/cre/features/stellar/stellar.go | Import formatting cleanup |
| system-tests/lib/cre/features/stellar/stellar_read_fixture.go | Import formatting cleanup |
| system-tests/lib/cre/features/stellar/stellar_forwarder.go | Import formatting cleanup |
| system-tests/lib/cre/features/stellar/stellar_config.go | Import formatting cleanup |
| system-tests/lib/cre/features/solana/v2/solana.go | Import ordering cleanup |
| system-tests/lib/cre/features/http_trigger/http_trigger.go | Import formatting cleanup |
| system-tests/lib/cre/features/http_action/http_action.go | Import formatting cleanup |
| system-tests/lib/cre/features/evm/v2/evm.go | Import re-org / ordering |
| system-tests/lib/cre/features/don_time/don_time.go | Import formatting cleanup |
| system-tests/lib/cre/features/cron/cron.go | Import formatting cleanup |
| system-tests/lib/cre/features/consensus/v2/consensus.go | Import ordering cleanup |
| system-tests/lib/cre/features/aptos/aptos.go | Import formatting cleanup |
| system-tests/lib/cre/features/aptos/aptos_workers.go | Import ordering cleanup |
| system-tests/lib/cre/features/aptos/aptos_helpers.go | Import formatting cleanup |
| system-tests/lib/cre/features/aptos/aptos_forwarder.go | Add depguard nolints on test-only imports |
| system-tests/lib/cre/environment/state.go | Import formatting cleanup |
| system-tests/lib/cre/environment/jobs.go | Import ordering cleanup |
| system-tests/lib/cre/environment/environment.go | Import formatting cleanup |
| system-tests/lib/cre/environment/dons.go | Import formatting cleanup |
| system-tests/lib/cre/environment/create_jd_jobs_op.go | Import formatting cleanup |
| system-tests/lib/cre/environment/config/config.go | Import formatting cleanup |
| system-tests/lib/cre/environment/config/config_test.go | Import formatting cleanup |
| system-tests/lib/cre/environment/blockchains/tron/tron.go | Import formatting cleanup |
| system-tests/lib/cre/environment/blockchains/stellar/stellar.go | Import formatting cleanup |
| system-tests/lib/cre/environment/blockchains/solana/solana.go | Import ordering cleanup |
| system-tests/lib/cre/environment/blockchains/blockchains.go | Import formatting cleanup |
| system-tests/lib/cre/don/jobs/jobs.go | Import formatting cleanup |
| system-tests/lib/cre/don/config/config.go | Enable InsecureFastScrypt in core config |
| system-tests/lib/cre/don.go | Import ordering cleanup |
| system-tests/lib/cre/contracts/registry_pickup_wait.go | Import formatting cleanup |
| system-tests/lib/cre/contracts/keystone_test.go | Import ordering cleanup |
| system-tests/lib/cre/contracts/contracts.go | Import ordering cleanup |
| integration-tests/load/go.mod | Bump chainlink-common & brotli |
| integration-tests/load/go.sum | go.sum updates for bumps |
| integration-tests/go.mod | Bump chainlink-common & brotli |
| integration-tests/go.sum | go.sum updates for bumps |
| go.mod | Bump chainlink-common & brotli |
| go.sum | go.sum updates for bumps |
| devenv/go.mod | Bump geth/chainlink deps; add indirects |
| devenv/go.sum | go.sum updates for bumps |
| deployment/go.mod | Bump chainlink-common & brotli |
| deployment/go.sum | go.sum updates for bumps |
| core/services/job/wasm_file_spec_factory_test.go | Build WASM using wasmbuild.Compile |
| core/scripts/go.mod | Bump chainlink-common & brotli |
| core/scripts/go.sum | go.sum updates for bumps |
| core/internal/testutils/wasmtest/wasm.go | Replace bespoke cache/build logic with wasmbuild |
| core/internal/testutils/wasmtest/wasm_test.go | Simplify wasmtest coverage to smoke-check output |
| .github/workflows/integration-tests.yml | Restructure setup/matrix, setup-go, and WASM cache |
| .github/workflows/integration-in-memory-tests.yml | Replace ctf-setup-go with actions/setup-go |
| .github/workflows/cre-system-tests.yaml | Allow precomputed matrix; switch setup-go |
| .github/workflows/cre-regression-system-tests.yaml | Switch setup-go; GITHUB_OUTPUT write style |
| .github/workflows/cre-mixed-env-tests.yaml | Switch setup-go; YAML description formatting |
| .github/workflows/ci-core.yml | Extend WASM cache key inputs; whitespace cleanup |
| .github/workflows/ccip-system-tests.yaml | Switch setup-go usage |
| .github/AGENTS.md | Add guidance for writing to GITHUB_OUTPUT |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
jmank88
reviewed
Aug 14, 2026
Contributor
CORA - Pending Reviewers
Legend: ✅ Approved | ❌ Changes Requested | 💬 Commented | 🚫 Dismissed | ⏳ Pending | ❓ Unknown For more details, see the full review summary. |
kalverra
force-pushed
the
DX-4988/integration-test-speedups
branch
from
August 14, 2026 17:31
3074420 to
3a366f9
Compare
chainchad
previously approved these changes
Aug 14, 2026
kalverra
force-pushed
the
DX-4988/integration-test-speedups
branch
from
August 14, 2026 23:57
05d128c to
735c86c
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 80 out of 89 changed files in this pull request and generated no new comments.
Suppressed comments (1)
system-tests/lib/cre/workflow/compile.go:131
- compileGoWorkflow logs timing info to stderr unconditionally. In CI this can add significant noise (and cost) when compiling many workflows, and it’s inconsistent with compileTSWorkflow which only prints on error. Consider gating this behind an env var (or removing it) so normal test runs stay quiet.
Tofel
reviewed
Aug 17, 2026
| }' | ||
|
|
||
| test_names=$(grep -rh -oP '^func \K(Test|Example)[^(]+' system-tests/tests/smoke/cre/*_test.go | grep -v Test_Upgrade | grep -v '^TestMain$') | ||
| test_names=$(grep -rh -oP '^func \K(Test|Example)[^(]+' system-tests/tests/smoke/cre/*_test.go | grep -v Test_Upgrade | grep -v '^TestMain$' | grep -v '_LoadExpectedConfig$' | grep -v '_CRESettingDefaults' | grep -v '^TestMustMintVaultJWT' | grep -v '^TestPendingQueueStallWedge') |
Contributor
There was a problem hiding this comment.
why do we need these extra exclusions? a comment would be welcomed
Tofel
reviewed
Aug 17, 2026
| with: | ||
| go-project-path: ./integration-tests | ||
| module-name: github.com/smartcontractkit/chainlink-testing-framework/lib | ||
| enforce-semantic-tag: "true" |
Contributor
There was a problem hiding this comment.
what's the reason for removing this action? how will we make sure peps don't use branch commits?
Tofel
reviewed
Aug 17, 2026
| id: define-matrix | ||
| shell: bash | ||
| run: | | ||
| test_names=$(grep -rh -oP '^func \K(Test|Example)[^(]+' system-tests/tests/smoke/cre/*_test.go | grep -v Test_Upgrade | grep -v '^TestMain$' | grep -v '_LoadExpectedConfig$' | grep -v '_CRESettingDefaults' | grep -v '^TestMustMintVaultJWT' | grep -v '^TestPendingQueueStallWedge') |
Contributor
There was a problem hiding this comment.
please explain the exclusions... I can say that:
Test_Upgradeshould be excluded, because it contains tests that are to be run, when we test node upgrade compatibility tests
Tofel
reviewed
Aug 17, 2026
| }) | ||
| ') | ||
|
|
||
| echo "matrix=$tests" | tee -a "$GITHUB_OUTPUT" |
Contributor
There was a problem hiding this comment.
this bit really worries me... now peps will need to modify the topology in two places? and if they miss adding a new test here with new topology then that test will never run? I think we need a better solution.
also, why do we even want to pass precomputed matrix?
Tofel
reviewed
Aug 17, 2026
| return "", errors.Wrap(err, "failed to write WASM file") | ||
| } | ||
|
|
||
| fmt.Fprintf(os.Stderr, "compileGoWorkflow: %s took %s\n", workflowName, time.Since(compileStart)) |
Contributor
There was a problem hiding this comment.
should we keep this or is it debug?
Tofel
reviewed
Aug 17, 2026
| } | ||
|
|
||
| enc, err := key.ToEncryptedJSON(password, keystore.DefaultScryptParams) | ||
| enc, err := key.ToEncryptedJSON(password, keystore.FastScryptParams) |
Contributor
There was a problem hiding this comment.
what's the diff between default and fast?
Tofel
reviewed
Aug 17, 2026
| testLogger.Info().Msg("Waiting for registry syncer to propagate the on-chain config change to workflow nodes...") | ||
| var dbPort, nodeCount int | ||
| for _, nodeSet := range testEnv.Config.NodeSets { | ||
| if slices.Contains(nodeSet.Capabilities, cre.WorkflowDON) || slices.Contains(nodeSet.Capabilities, "workflow") { |
Contributor
There was a problem hiding this comment.
why do we need the second contains, if the value of cre.WorkflowDON constant is workflow?
Tofel
reviewed
Aug 17, 2026
| - default | ||
| - prefix(github.com/smartcontractkit/) | ||
| - prefix(github.com/smartcontractkit/chainlink) | ||
| - prefix(github.com/smartcontractkit/chainlink/) |
Contributor
There was a problem hiding this comment.
This is deliberate for historical reasons. Please don't undo
Suggested change
| - prefix(github.com/smartcontractkit/chainlink/) | |
| - prefix(github.com/smartcontractkit/chainlink) |
kalverra
force-pushed
the
DX-4988/integration-test-speedups
branch
from
August 17, 2026 18:59
76d77d8 to
17a111c
Compare
kalverra
force-pushed
the
DX-4988/integration-test-speedups
branch
2 times, most recently
from
August 17, 2026 20:41
c00157a to
bd134d0
Compare
kalverra
force-pushed
the
DX-4988/integration-test-speedups
branch
from
August 18, 2026 00:09
bd134d0 to
51d9c9a
Compare
kalverra
force-pushed
the
DX-4988/integration-test-speedups
branch
from
August 18, 2026 00:19
51d9c9a to
35d11d4
Compare
kalverra
force-pushed
the
DX-4988/integration-test-speedups
branch
from
August 18, 2026 00:49
35d11d4 to
d114281
Compare
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Squeeze out a little more
Integration Testsspeed.Big Changes
Replace
wasmtestReplace the old
wasmtestpackage with newwasmbuildfrom chainlink-common. Also replaces any remaininggo buildcalls in code to ad-hoc build wasm binaries.Gives us better performance and smarter caching, while distributing the process to the rest of the company (like
cre-cli).Small Changes
smartcontractkit/.github/actions/ctf-setup-gowithactions/setup-go@v7across workflow files. Drops wrapper action execution overhead and eliminates unused cache restore steps for precompiled test binaries.FastScryptParamsin system-test crypto key generation and enabledInsecureFastScrypt: truein base node configs. Speeds up node startup and reduces CPU/memory usage during local key generation.time.Sleepwith JSON-RPC polling against the vault gateway inupdateVaultCapabilityConfigInRegistry. Prevents unnecessary static wait times while verifying on-chain config propagation.enforce-ctf-versioncheck from integration test workflows. Cleans up legacy CI gates.