Skip to content

Shard smoketests using nextest - #5612

Draft
joshua-spacetime wants to merge 27 commits into
masterfrom
joshua/shard-smoketests
Draft

Shard smoketests using nextest#5612
joshua-spacetime wants to merge 27 commits into
masterfrom
joshua/shard-smoketests

Conversation

@joshua-spacetime

@joshua-spacetime joshua-spacetime commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Description of Changes

The overall goals of this patch are to:

  1. Speed up CI workflows
  2. Avoid having to re-run full test suites due to a flaky test

This patch doesn't fully achieve these goals, but it makes decent progress towards them.

The main way it does that is by sharding the smoketests. This works as follows: There are two new smoketest build jobs - one for linux and one for windows. These jobs each build the following:

  1. The CLI
  2. Standalone
  3. The smoketest WASM modules
  4. The smoketest executables

These artifacts are uploaded, and the test jobs each download these artifacts and run a partition/shard of the smoketest executables.

If a smoketest in one of the shards fails, only that shard needs to be re-run. And nothing needs to be rebuilt. The job just restores the artifacts and re-runs the test executables.

Note, this patch only targets the public smoketests. I will do the same thing for the private smoketests in a follow on patch. And similarly after that, I will partition the test suite.

A few tangential changes that were included:

1. Build the cli and standalone once

Previously there were many different jobs that would build the cli and standalone. Now they are built once by the linux smoketest job and shared with the other test jobs.

2. Rust-cache only for windows build

This patch also includes some caching changes. In particular, only non-workspace dependencies for windows builds are cached.

Linux builds only take around 4 minutes without caching. Windows builds take almost 15 minutes without caching, but 4 minutes with caching. So it's extremely important to always hit the cache for windows builds. Therefore we want to be careful about how much we write to the cache to avoid thrashing it. The non-workspace dependencies already take up 1.8GB, so if the linux builds can get away without caching these dependencies, it's probably better.

3. Fix caching for unity builds

One thing I discovered is that the unity cache key was causing us to write a new (> 1GB) cache entry for unity builds on every single PR, even when no unity inputs changed. This is fixed now.

4. Split up test_all_templates

I split up the test_all_templates smoketest into 24 individual smoketest so that they could all be sharded evenly.

5. Upload --timings for smoketest builds

This was very helpful in determining why the windows builds were so much slower than linux.

API and ABI breaking changes

None

Expected complexity level and risk

...

Testing

N/A

@joshua-spacetime
joshua-spacetime force-pushed the joshua/shard-smoketests branch 2 times, most recently from 3522120 to cff37df Compare July 29, 2026 19:09
@joshua-spacetime
joshua-spacetime force-pushed the joshua/shard-smoketests branch from 515cf89 to 74f6dca Compare July 30, 2026 04:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant