Combine all tests into a single suite to increase parallelism. NFC#26365
Open
sbc100 wants to merge 1 commit intoemscripten-core:mainfrom
Open
Combine all tests into a single suite to increase parallelism. NFC#26365sbc100 wants to merge 1 commit intoemscripten-core:mainfrom
sbc100 wants to merge 1 commit intoemscripten-core:mainfrom
Conversation
kripken
reviewed
Feb 28, 2026
test/runner.py
Outdated
| is_parallel = use_parallel_suite(m, tests) | ||
| if not suite: | ||
| suite = create_test_suite(is_parallel, options) | ||
| is_suite_parallel = is_parallel |
Member
There was a problem hiding this comment.
how do these two variables differ? from the name and from the code I'm having a hard time figuring it out.
Collaborator
Author
There was a problem hiding this comment.
The first module we find dictates if we are parallel or not.
If we later find a module that does not agree, we error out. I will update the names to be more clear.
This allows all tests to be run in parallel. For example:
```
Test suites: ['test_core', 'test_jslib', 'test_other']
Running test_core: (88 tests)
Using 88 parallel test processes
[88/88] test_asan_vector (test_core.core0.test_asan_vector) ... ok
----------------------------------------------------------------------
Ran 88 tests in 3.296s
OK (skipped=5)
Running test_jslib: (54 tests)
Using 54 parallel test processes
[54/54] test_jslib_aliases_closure_wasm64 (test_jslib.jslib.test_jslib_aliases_closure_wasm64) ... ok
----------------------------------------------------------------------
Ran 54 tests in 5.287s
OK
Running test_other: (41 tests)
Using 41 parallel test processes
[41/41] test_abspaths (test_other.other.test_abspaths) ... ok
----------------------------------------------------------------------
Ran 41 tests in 5.210s
OK
Total core time: 228.881s. Wallclock time: 13.795s. Parallelization: 16.59x.
====================
TEST SUMMARY
test_core: 88 run, 0 errors, 0 failures, 5 skipped
test_jslib: 54 run, 0 errors, 0 failures, 0 skipped
test_other: 41 run, 0 errors, 0 failures, 0 skipped
```
After:
```
$ ./test/runner jslib other.test_a* core0.test_a* --skip-slow
Running 183 tests
Using 128 parallel test processes
[183/183] test_abspaths (test_other.other.test_abspaths) ... ok
----------------------------------------------------------------------
Ran 183 tests in 7.459s
OK (skipped=5)
Total core time: 301.490s. Wallclock time: 7.459s. Parallelization: 40.42x.
```
Note the wall clock time is less since we ran all the tests in parallel.
If you try to mix parallel and non-parallel test module you now get an
error. e.g.:
```
$ ./test/runner core0 benchmark
runner: error: attempt to mix parallel and non-parallel test modules
```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
My motivation for this change is to allow for increased parallelism that is not limited to how
many tests are in a give module. I'm hoping to do more splitting up of test_other.py, for
example, and I don't want to loose parallism.
After this change all tests are combined into a single parallel test suite. For example:
After:
Note the wall clock time is less since we ran all the tests in parallel.
If you try to mix parallel and non-parallel test module you now get an error. e.g.: