feat(test-runner): add Reporter.preprocessSuite() hook for test filtering#41100
feat(test-runner): add Reporter.preprocessSuite() hook for test filtering#41100Skn0tt wants to merge 6 commits into
Conversation
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
@Skn0tt Sweet, we'd love to dogfood this once it lands! We implemented time-based sharding using the A few comments:
|
Add tests locking in that Reporter.plan's suite contains only top-level projects (setup/dependency projects are not exposed), respects --project and --grep, and ignores --shard (built-in sharding runs after plan). Clarify the Reporter.plan.suite docs accordingly. Fixes: microsoft#40934
|
Thanks for the feedback. Sounds like what you have in mind is already there, i've added tests and extended docs to capture that. The API does allow excluding projects, but not project dependencies. |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
|
||
| const shardingReporters = reporters.filter(r => r.implementsSharding?.() ?? false); | ||
| if (shardingReporters.length > 1) | ||
| throw new Error(`Multiple reporters declare 'implementsSharding': ${shardingReporters.map(r => r.constructor?.name ?? 'reporter').join(', ')}. Only one reporter may handle sharding.`); |
There was a problem hiding this comment.
Not sure why this is an issue, let them plan as much as they want?
There was a problem hiding this comment.
All shards need to receive the same input tree, but if there's two reporters implementing sharding, then the second one will receive different inputs per shard. So for the same reason that our sharding cannot run together with 3rd party sharding, two 3rd party sharders cannot run together.
Test results for "MCP"7230 passed, 1103 skipped Merge workflow run. |
Test results for "tests 1"1 failed 2 flaky39608 passed, 771 skipped Merge workflow run. |
Skn0tt
changed the title
3 participants
Adds a
Reporter.plan(rootSuite)hook that runs betweenonConfigureandonBegin, letting reporters filter/modify tests usingskip/fixme/fail/excludedisposition methods onTestCaseandSuite.Reporters can also opt out of Playwright's built-in sharding by implementing
implementsSharding().Follow-up: expose
Suite.parallelModepublicly so third-party sharders can keeptest.describe.serial(...)chains together. Without it, custom sharding can only safely split tests that aren't in serial groups.Fixes #40934