optimize: CRE Startup Times

[kalverra]

Intent

Optimize CRE test environment startup performance locally and in CI to reduce integration test runtime toward the <= 10m target.

Results

Cut runtime for Start Local CRE step in half!

Big Changes

CRE Docker Image Background Prefetching

Introduced a reusable composite action (.github/actions/prefetch-cre-images) that initiates non-blocking background docker pull operations for public and private images during early workflow setup steps, awaiting completion just before the local CRE environment start step.

Eliminates lazy container image pull latencies from testcontainers during test startup, cutting Start Local CRE execution time in CI from 53s to 21s.

Parallelized Environment & Blockchain Initialization

Overlapped independent setup phases: Chip Router startup runs concurrently with blockchain deployment via worker pool, blockchain deployment is parallelized across instances using errgroup, NewDonMetadata generation runs concurrently across node sets with cloned capability configs, and node funding executes concurrently across blockchains while maintaining per-chain nonce serialization.

Removes sequential blocking waits across environment components to substantially accelerate startup in multi-chain and multi-DON configurations.

Workflow Optimization Tooling Migration to octometrics

Removed legacy Python scripts (workflow_monitor.py, workflow_compare.py, and test suites) from .github/.agents/skills/optimize-workflow/ and updated skill guidance to use octometrics.

Consolidates CI performance monitoring and run comparisons into a dedicated, standard tool.

Small Changes

• Reduced capability registry sync poll interval from 2s to 250ms in system-tests/lib/cre/contracts/registry_pickup_wait.go. Speeds up capability registration pickup loops during CRE bootstrap.
• Wrapped worker pool in system-tests/lib/cre/environment/environment.go with a cancelable context. Ensures background Chip Router tasks terminate immediately on early return errors.
• Added set +e in prefetch-cre-images subshells. Guarantees exit codes are always written so the wait step accurately detects failed pulls.
• Pre-populated per-blockchain mutexes for node funding in system-tests/lib/cre/environment/dons.go. Avoids runtime mutex map write contention during concurrent funding.

[github-actions]

✅ No conflicts with other open PRs targeting develop

[trunk-io]

   

View Full Report ↗︎ ⋅ Docs

[Copilot]

Pull request overview

Risk Rating: MEDIUM

This PR optimizes CRE environment startup time (local + CI) by overlapping independent startup work, parallelizing some setup stages, and prefetching Docker images to avoid lazy pulls on the critical path.

Changes:

• Parallelize CRE topology DON metadata creation and blockchain deployments.
• Overlap Chip Router startup with blockchain startup.
• Prefetch required Docker images earlier in GitHub Actions workflows and wait for prefetch completion before “Start local CRE”.

Areas requiring scrupulous human review:

• Newly introduced goroutine-based parallelism in NewTopology and blockchain Start (index capture correctness, cancellation behavior).
• Composite action behavior for background docker pulls (failure reporting correctness and robustness).
• Startup-stage cancellation/cleanup semantics when an early stage fails.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
system-tests/lib/cre/topology.go	Parallelizes DON metadata creation for topology construction.
system-tests/lib/cre/environment/environment.go	Overlaps Chip Router startup with blockchain startup via worker pool.
system-tests/lib/cre/environment/dons.go	Refactors node funding to build tasks and fund in parallel with per-chain serialization.
system-tests/lib/cre/environment/blockchains/blockchains.go	Parallelizes blockchain deployments using errgroup.WithContext.
system-tests/lib/cre/contracts/registry_pickup_wait.go	Speeds up capability registry sync polling interval.
.github/workflows/cre-system-tests.yaml	Adds image prefetch steps (public + private) and a wait barrier before CRE startup.
.github/workflows/cre-regression-system-tests.yaml	Adds image prefetch steps (public + private) and a wait barrier before CRE startup.
.github/workflows/cre-mixed-env-tests.yaml	Adds image prefetch steps (including baseline image) plus wait barrier(s).
.github/actions/prefetch-cre-images/action.yml	New composite action to start background docker pulls and later wait for completion.
.github/.agents/skills/optimize-workflow/SKILL.md	Updates workflow optimization skill guidance (switches to octometrics, adjusts constraints).
.github/.agents/skills/optimize-workflow/scripts/workflow_monitor.py	Removes legacy workflow monitoring script.
.github/.agents/skills/optimize-workflow/scripts/workflow_compare.py	Removes legacy workflow comparison script.
.github/.agents/skills/optimize-workflow/scripts/test_workflow_monitor.py	Removes legacy tests for removed monitoring script.
.github/.agents/skills/optimize-workflow/scripts/test_workflow_compare.py	Removes legacy tests for removed comparison script.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

The merge-base changed after approval.

[Tofel]

do we need this struct? can't we call fmt.Print() inside the task instead? I'd rather keep this orchestration code as simple as possible to reduce mental fatigue

[Tofel]

nice job!

[cl-sonarqube-production]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube