Skip to content

Combine all tests into a single suite to increase parallelism. NFC#26365

Open
sbc100 wants to merge 1 commit intoemscripten-core:mainfrom
sbc100:test_runner
Open

Combine all tests into a single suite to increase parallelism. NFC#26365
sbc100 wants to merge 1 commit intoemscripten-core:mainfrom
sbc100:test_runner

Conversation

@sbc100
Copy link
Collaborator

@sbc100 sbc100 commented Feb 27, 2026

My motivation for this change is to allow for increased parallelism that is not limited to how
many tests are in a give module. I'm hoping to do more splitting up of test_other.py, for
example, and I don't want to loose parallism.

After this change all tests are combined into a single parallel test suite. For example:

$ ./test/runner jslib other.test_a* core0.test_a* --skip-slow
Test suites: ['test_core', 'test_jslib', 'test_other']
Running test_core: (88 tests)
Using 88 parallel test processes
[88/88] test_asan_vector (test_core.core0.test_asan_vector) ... ok

----------------------------------------------------------------------
Ran 88 tests in 3.296s

OK (skipped=5)
Running test_jslib: (54 tests)
Using 54 parallel test processes
[54/54] test_jslib_aliases_closure_wasm64 (test_jslib.jslib.test_jslib_aliases_closure_wasm64) ... ok

----------------------------------------------------------------------
Ran 54 tests in 5.287s

OK
Running test_other: (41 tests)
Using 41 parallel test processes
[41/41] test_abspaths (test_other.other.test_abspaths) ... ok

----------------------------------------------------------------------
Ran 41 tests in 5.210s

OK
Total core time: 228.881s. Wallclock time: 13.795s. Parallelization: 16.59x.
====================

TEST SUMMARY
    test_core: 88 run, 0 errors, 0 failures, 5 skipped
    test_jslib: 54 run, 0 errors, 0 failures, 0 skipped
    test_other: 41 run, 0 errors, 0 failures, 0 skipped

After:

$ ./test/runner jslib other.test_a* core0.test_a* --skip-slow
Running 183 tests
Using 128 parallel test processes
[183/183] test_abspaths (test_other.other.test_abspaths) ... ok

----------------------------------------------------------------------
Ran 183 tests in 7.459s

OK (skipped=5)
Total core time: 301.490s. Wallclock time: 7.459s. Parallelization: 40.42x.

Note the wall clock time is less since we ran all the tests in parallel.

If you try to mix parallel and non-parallel test module you now get an error. e.g.:

$ ./test/runner core0 benchmark
runner: error: attempt to mix parallel and non-parallel test modules

test/runner.py Outdated
is_parallel = use_parallel_suite(m, tests)
if not suite:
suite = create_test_suite(is_parallel, options)
is_suite_parallel = is_parallel
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how do these two variables differ? from the name and from the code I'm having a hard time figuring it out.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first module we find dictates if we are parallel or not.

If we later find a module that does not agree, we error out. I will update the names to be more clear.

This allows all tests to be run in parallel.  For example:

```
Test suites: ['test_core', 'test_jslib', 'test_other']
Running test_core: (88 tests)
Using 88 parallel test processes
[88/88] test_asan_vector (test_core.core0.test_asan_vector) ... ok

----------------------------------------------------------------------
Ran 88 tests in 3.296s

OK (skipped=5)
Running test_jslib: (54 tests)
Using 54 parallel test processes
[54/54] test_jslib_aliases_closure_wasm64 (test_jslib.jslib.test_jslib_aliases_closure_wasm64) ... ok

----------------------------------------------------------------------
Ran 54 tests in 5.287s

OK
Running test_other: (41 tests)
Using 41 parallel test processes
[41/41] test_abspaths (test_other.other.test_abspaths) ... ok

----------------------------------------------------------------------
Ran 41 tests in 5.210s

OK
Total core time: 228.881s. Wallclock time: 13.795s. Parallelization: 16.59x.
====================

TEST SUMMARY
    test_core: 88 run, 0 errors, 0 failures, 5 skipped
    test_jslib: 54 run, 0 errors, 0 failures, 0 skipped
    test_other: 41 run, 0 errors, 0 failures, 0 skipped
```

After:

```
$ ./test/runner jslib other.test_a* core0.test_a* --skip-slow
Running 183 tests
Using 128 parallel test processes
[183/183] test_abspaths (test_other.other.test_abspaths) ... ok

----------------------------------------------------------------------
Ran 183 tests in 7.459s

OK (skipped=5)
Total core time: 301.490s. Wallclock time: 7.459s. Parallelization: 40.42x.
```

Note the wall clock time is less since we ran all the tests in parallel.

If you try to mix parallel and non-parallel test module you now get an
error.  e.g.:

```
$ ./test/runner core0 benchmark
runner: error: attempt to mix parallel and non-parallel test modules
```
@sbc100 sbc100 changed the title Combine all tests into a single test suite. NFC Combine all tests into a single suite to increase parallelism. NFC Feb 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants