Shard smoketests using nextest - #5612
Draft
joshua-spacetime wants to merge 27 commits into
Draft
Conversation
joshua-spacetime
force-pushed
the
joshua/shard-smoketests
branch
2 times, most recently
from
July 29, 2026 19:09
3522120 to
cff37df
Compare
joshua-spacetime
force-pushed
the
joshua/shard-smoketests
branch
from
July 30, 2026 04:51
515cf89 to
74f6dca
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of Changes
The overall goals of this patch are to:
This patch doesn't fully achieve these goals, but it makes decent progress towards them.
The main way it does that is by sharding the smoketests. This works as follows: There are two new smoketest build jobs - one for linux and one for windows. These jobs each build the following:
These artifacts are uploaded, and the test jobs each download these artifacts and run a partition/shard of the smoketest executables.
If a smoketest in one of the shards fails, only that shard needs to be re-run. And nothing needs to be rebuilt. The job just restores the artifacts and re-runs the test executables.
Note, this patch only targets the public smoketests. I will do the same thing for the private smoketests in a follow on patch. And similarly after that, I will partition the test suite.
A few tangential changes that were included:
1. Build the cli and standalone once
Previously there were many different jobs that would build the cli and standalone. Now they are built once by the linux smoketest job and shared with the other test jobs.
2. Rust-cache only for windows build
This patch also includes some caching changes. In particular, only non-workspace dependencies for windows builds are cached.
Linux builds only take around 4 minutes without caching. Windows builds take almost 15 minutes without caching, but 4 minutes with caching. So it's extremely important to always hit the cache for windows builds. Therefore we want to be careful about how much we write to the cache to avoid thrashing it. The non-workspace dependencies already take up 1.8GB, so if the linux builds can get away without caching these dependencies, it's probably better.
3. Fix caching for unity builds
One thing I discovered is that the unity cache key was causing us to write a new (> 1GB) cache entry for unity builds on every single PR, even when no unity inputs changed. This is fixed now.
4. Split up
test_all_templatesI split up the
test_all_templatessmoketest into 24 individual smoketest so that they could all be sharded evenly.5. Upload
--timingsfor smoketest buildsThis was very helpful in determining why the windows builds were so much slower than linux.
API and ABI breaking changes
None
Expected complexity level and risk
...
Testing
N/A