From 6cd43c8c56469629cedfcddef94d3d7636aafb2b Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 10 Sep 2026 16:27:45 +0000 Subject: [PATCH 1/4] Initial plan From a60bde446d9c38045b49e5d385a2cc06046b68a9 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 10 Sep 2026 16:30:06 +0000 Subject: [PATCH 2/4] Document whole-run zero-tests verdict and global/per-module minimum-expected-tests Co-authored-by: meaghanlewis <10103121+meaghanlewis@users.noreply.github.com> --- .../microsoft-testing-platform-cli-options.md | 7 +++- ...rosoft-testing-platform-troubleshooting.md | 15 ++++++-- docs/core/tools/dotnet-test-mtp.md | 37 ++++++++++++++++++- 3 files changed, 53 insertions(+), 6 deletions(-) diff --git a/docs/core/testing/microsoft-testing-platform-cli-options.md b/docs/core/testing/microsoft-testing-platform-cli-options.md index 06c34b8bddc46..8ab4ad1b5933f 100644 --- a/docs/core/testing/microsoft-testing-platform-cli-options.md +++ b/docs/core/testing/microsoft-testing-platform-cli-options.md @@ -3,7 +3,7 @@ title: Microsoft.Testing.Platform (MTP) CLI options reference description: Find platform and extension command-line options for MTP in one place. author: Evangelink ms.author: amauryleve -ms.date: 09/02/2026 +ms.date: 09/10/2026 ai-usage: ai-assisted --- @@ -144,6 +144,11 @@ This article gives a central entry point for MTP command-line options. Specifies the minimum number of tests that must run. When the run executes fewer tests, including zero, it exits with code `9`. An explicit minimum supersedes `--zero-tests-policy`. + With `dotnet test`, this option applies to the whole run when it's specified before `--`, and to each test module when it's specified after `--`. For more information, see [Whole-run and per-module minimums](../tools/dotnet-test-mtp.md#whole-run-and-per-module-minimums). + + > [!NOTE] + > `--minimum-expected-tests 0` isn't an alias for `--ignore-exit-code 8`. To suppress the zero-tests exit code, use `--ignore-exit-code 8`. + - **`--no-banner`** Disables the startup banner, the copyright message, and the telemetry banner. The same effect can be achieved through the `TESTINGPLATFORM_NOBANNER` or `DOTNET_NOLOGO` [environment variables](./microsoft-testing-platform-config.md#environment-variables). diff --git a/docs/core/testing/microsoft-testing-platform-troubleshooting.md b/docs/core/testing/microsoft-testing-platform-troubleshooting.md index 112daffdd20c8..ec3dad5f531c4 100644 --- a/docs/core/testing/microsoft-testing-platform-troubleshooting.md +++ b/docs/core/testing/microsoft-testing-platform-troubleshooting.md @@ -3,7 +3,7 @@ title: Microsoft.Testing.Platform (MTP) troubleshooting description: Troubleshoot MTP issues, exit codes, and known problems. author: Evangelink ms.author: amauryleve -ms.date: 09/02/2026 +ms.date: 09/10/2026 ai-usage: ai-assisted --- @@ -26,17 +26,26 @@ MTP uses known exit codes to communicate test failure or app errors. Exit codes | `6` (no longer used) | Exit code `6` is no longer produced by the platform; it previously indicated that the test session was using a non-implemented feature. | | `7` | The exit code `7` indicates that a test session was unable to complete successfully, and likely crashed. It's possible that this was caused by a test session that was run via a test controller's extension point. | | `8` | The exit code `8` indicates that the test session ran zero tests under the strict `--zero-tests-policy`. | -| `9` | The exit code `9` indicates that the run executed fewer tests than `--minimum-expected-tests` requires, including zero tests. | +| `9` | The exit code `9` indicates that the run executed fewer tests than an explicit `--minimum-expected-tests` value requires, including zero tests. | | `10` | The exit code `10` indicates that the test adapter, Testing.Platform Test Framework, MSTest, NUnit, or xUnit, failed to run tests for an infrastructure reason unrelated to the test's self. An example is failing to create a fixture needed by tests. | | `11` | The exit code `11` indicates that the test process will exit if dependent process exits. | | `12` | The exit code `12` indicates that the test session was unable to run because the client does not support any of the supported protocol versions. | | `13` | The exit code `13` indicates that the test session was stopped due to reaching the specified number of maximum failed tests using `--maximum-failed-tests` command-line option. For more information, see [the Options section in MTP CLI options reference](microsoft-testing-platform-cli-options.md) | | `14` | The exit code `14` indicates that a compatible coverage collector published a failed coverage threshold evaluation. | -An explicit `--minimum-expected-tests` value supersedes `--zero-tests-policy`. Without the minimum option, strict zero-test handling continues to use exit code `8`. +An explicit `--minimum-expected-tests` value supersedes `--zero-tests-policy`. Without the minimum option, strict zero-test handling continues to use exit code `8`. Exit codes `8` and `9` remain distinct so that an unmet minimum isn't confused with a module that ran no tests. To enable verbose logging and troubleshoot issues, see [Diagnostic logging](#diagnostic-logging). +### Zero tests in a multi-module run + +When `dotnet test` runs several test modules, exit code `8` is a per-module signal, while the zero-tests verdict for the whole run is decided once from the total number of tests that ran. A module that matches no tests, for example because of `--test-modules` or a global `--filter`, exits with code `8`, but that code is normalized to success before the results are aggregated. A single empty module therefore doesn't fail the whole run, although the module keeps its `Exit code: 8` diagnostic in the output. + +An explicit `--minimum-expected-tests` value that isn't met yields exit code `9`, whether the minimum is global (specified before `--`) or per-module (specified after `--`). For more information, see [Whole-run and per-module minimums](../tools/dotnet-test-mtp.md#whole-run-and-per-module-minimums). + +> [!NOTE] +> This whole-run aggregation requires the .NET 11 SDK (11.0.1xx) or a later version. + ### Ignore specific exit codes MTP is designed to be strict by default but allows for configurability. As such, it's possible for users to decide which exit codes should be ignored (an exit code of `0` will be returned instead of the original exit code). diff --git a/docs/core/tools/dotnet-test-mtp.md b/docs/core/tools/dotnet-test-mtp.md index 0830474ec3f36..d94a2d3c2ad82 100644 --- a/docs/core/tools/dotnet-test-mtp.md +++ b/docs/core/tools/dotnet-test-mtp.md @@ -1,7 +1,7 @@ --- title: dotnet test command with Microsoft.Testing.Platform (MTP) description: The dotnet test command is used to execute unit tests in a given project using MTP. -ms.date: 09/02/2026 +ms.date: 09/10/2026 ai-usage: ai-assisted --- # dotnet test with Microsoft.Testing.Platform (MTP) @@ -98,7 +98,12 @@ With MTP, `dotnet test` operates faster than with VSTest. The test-related argum - **`--minimum-expected-tests `** - Specifies the minimum number of tests that must be executed. If the actual number of tests is less than the specified minimum, the test run fails with exit code 9. For more information about exit codes, see [MTP exit codes](../testing/microsoft-testing-platform-troubleshooting.md#exit-codes). + Specifies the minimum number of tests that must run across the whole run. If the total number of tests that ran is less than the specified minimum, the test run fails with exit code 9. For more information about exit codes, see [MTP exit codes](../testing/microsoft-testing-platform-troubleshooting.md#exit-codes). + + Because this option appears before `--`, it's a global (whole-run) option. To require a minimum for each test module instead, pass the option after `--` so that it's forwarded to every test module. For more information, see [Whole-run and per-module minimums](#whole-run-and-per-module-minimums). + + > [!NOTE] + > Whole-run aggregation of the minimum requires the .NET 11 SDK (11.0.1xx) or a later version. In earlier SDKs, the option applies to each test module. - [!INCLUDE [arch](includes/cli-arch.md)] @@ -190,6 +195,28 @@ The preceding example requires the [`Microsoft.Testing.Extensions.TrxReport`](ht The same parser behavior applies to `dotnet run` and `dotnet build`. For a detailed example, see [Forward arguments to the application](dotnet-run.md#forward-arguments-to-the-application) in the `dotnet run` reference. +## Whole-run and per-module minimums + +Starting with the .NET 11 SDK (11.0.1xx), the position of an argument determines its scope: + +- Arguments *before* `--` are global. The `dotnet test` orchestrator interprets them for the whole run. +- Arguments *after* `--` are local. `dotnet test` forwards them to each test module, so each module applies them independently. + +Because `--minimum-expected-tests` is available in both scopes, you can require a minimum for the whole run, for each module, or both: + +```dotnetcli +dotnet test --minimum-expected-tests 5 -- --minimum-expected-tests 2 +``` + +The preceding command requires at least 5 tests across the whole run and at least 2 tests in each test module. + +The zero-tests verdict for the whole run is decided once, from the total number of tests that ran. A module that matches no tests, for example because of `--test-modules` or a global `--filter`, exits with code 8 (`ZeroTests`), but that code is normalized to success before the results are aggregated. As a result, a single empty module no longer fails the whole run, although the module keeps its `Exit code: 8` diagnostic in the output for visibility. + +When you specify `--minimum-expected-tests` and the minimum isn't met, the run fails with exit code 9 (`MinimumExpectedTestsPolicyViolation`). This code is distinct from 8 so that a stricter global or per-module minimum isn't confused with an empty module. + +> [!NOTE] +> `--minimum-expected-tests 0` isn't an alias for `--ignore-exit-code 8`. To suppress the zero-tests exit code, use `--ignore-exit-code 8`. + ## Examples - Run the tests in the project or solution in the current directory: @@ -246,6 +273,12 @@ The same parser behavior applies to `dotnet run` and `dotnet build`. For a detai dotnet test --minimum-expected-tests 10 ``` +- Run the tests requiring at least 5 tests across the whole run and at least 2 tests in each test module: + + ```dotnetcli + dotnet test --minimum-expected-tests 5 -- --minimum-expected-tests 2 + ``` + - Run the tests in the `TestProject` project, providing the `-bl` (binary log) argument to `msbuild`: ```dotnetcli From 0534aa703414f01256b50fa52822d20202fc2049 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 10 Sep 2026 16:30:56 +0000 Subject: [PATCH 3/4] Address review feedback: imperative example wording and reduce duplicated prose Co-authored-by: meaghanlewis <10103121+meaghanlewis@users.noreply.github.com> --- .../testing/microsoft-testing-platform-troubleshooting.md | 4 +--- docs/core/tools/dotnet-test-mtp.md | 2 +- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/core/testing/microsoft-testing-platform-troubleshooting.md b/docs/core/testing/microsoft-testing-platform-troubleshooting.md index ec3dad5f531c4..d1e9958916d8d 100644 --- a/docs/core/testing/microsoft-testing-platform-troubleshooting.md +++ b/docs/core/testing/microsoft-testing-platform-troubleshooting.md @@ -39,9 +39,7 @@ To enable verbose logging and troubleshoot issues, see [Diagnostic logging](#dia ### Zero tests in a multi-module run -When `dotnet test` runs several test modules, exit code `8` is a per-module signal, while the zero-tests verdict for the whole run is decided once from the total number of tests that ran. A module that matches no tests, for example because of `--test-modules` or a global `--filter`, exits with code `8`, but that code is normalized to success before the results are aggregated. A single empty module therefore doesn't fail the whole run, although the module keeps its `Exit code: 8` diagnostic in the output. - -An explicit `--minimum-expected-tests` value that isn't met yields exit code `9`, whether the minimum is global (specified before `--`) or per-module (specified after `--`). For more information, see [Whole-run and per-module minimums](../tools/dotnet-test-mtp.md#whole-run-and-per-module-minimums). +When `dotnet test` runs several test modules, exit code `8` is a per-module signal, while the zero-tests verdict for the whole run is decided once from the total number of tests that ran. A single empty module therefore doesn't fail the whole run, although the module keeps its `Exit code: 8` diagnostic in the output. For more information, see [Whole-run and per-module minimums](../tools/dotnet-test-mtp.md#whole-run-and-per-module-minimums). > [!NOTE] > This whole-run aggregation requires the .NET 11 SDK (11.0.1xx) or a later version. diff --git a/docs/core/tools/dotnet-test-mtp.md b/docs/core/tools/dotnet-test-mtp.md index d94a2d3c2ad82..0ec0a19c0886a 100644 --- a/docs/core/tools/dotnet-test-mtp.md +++ b/docs/core/tools/dotnet-test-mtp.md @@ -273,7 +273,7 @@ When you specify `--minimum-expected-tests` and the minimum isn't met, the run f dotnet test --minimum-expected-tests 10 ``` -- Run the tests requiring at least 5 tests across the whole run and at least 2 tests in each test module: +- Require at least 5 tests across the whole run and at least 2 tests in each test module: ```dotnetcli dotnet test --minimum-expected-tests 5 -- --minimum-expected-tests 2 From 550a670a46a8d6809d73871669700fc310c59ceb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Fri, 11 Sep 2026 11:51:56 +0200 Subject: [PATCH 4/4] Correct MTP test verdict documentation Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../microsoft-testing-platform-cli-options.md | 8 ++--- ...rosoft-testing-platform-troubleshooting.md | 8 ++--- docs/core/tools/dotnet-test-mtp.md | 29 ++++++++++++++----- 3 files changed, 30 insertions(+), 15 deletions(-) diff --git a/docs/core/testing/microsoft-testing-platform-cli-options.md b/docs/core/testing/microsoft-testing-platform-cli-options.md index 8ab4ad1b5933f..c301b351bb1c4 100644 --- a/docs/core/testing/microsoft-testing-platform-cli-options.md +++ b/docs/core/testing/microsoft-testing-platform-cli-options.md @@ -3,7 +3,7 @@ title: Microsoft.Testing.Platform (MTP) CLI options reference description: Find platform and extension command-line options for MTP in one place. author: Evangelink ms.author: amauryleve -ms.date: 09/10/2026 +ms.date: 09/11/2026 ai-usage: ai-assisted --- @@ -142,12 +142,12 @@ This article gives a central entry point for MTP command-line options. - **`--minimum-expected-tests`** - Specifies the minimum number of tests that must run. When the run executes fewer tests, including zero, it exits with code `9`. An explicit minimum supersedes `--zero-tests-policy`. + Specifies a positive minimum number of tests that must run. When the run executes fewer tests, including zero, it exits with code `9`. An explicit minimum supersedes `--zero-tests-policy`. With `dotnet test`, this option applies to the whole run when it's specified before `--`, and to each test module when it's specified after `--`. For more information, see [Whole-run and per-module minimums](../tools/dotnet-test-mtp.md#whole-run-and-per-module-minimums). > [!NOTE] - > `--minimum-expected-tests 0` isn't an alias for `--ignore-exit-code 8`. To suppress the zero-tests exit code, use `--ignore-exit-code 8`. + > `--minimum-expected-tests 0` is invalid. To suppress the zero-tests exit code, use `--ignore-exit-code 8`. - **`--no-banner`** @@ -187,7 +187,7 @@ This article gives a central entry point for MTP command-line options. Controls whether a run that executes no tests because every test was skipped is treated as a failure. Valid values are `allow-skipped` (default) and `strict`. With `allow-skipped`, an all-skipped run succeeds. With `strict`, it fails with exit code `8`. An explicit `--minimum-expected-tests` value supersedes this policy and uses exit code `9` when the minimum isn't met. > [!NOTE] - > This option is available in MTP starting with version 2.3.0. + > This option is available in MTP starting with version 4.3.0. With `dotnet test`, pass the option after `--` to forward it to each test module. When you don't set a global minimum, the .NET 11 SDK determines the whole-run zero-tests verdict separately. For more information, see [Whole-run and per-module minimums](../tools/dotnet-test-mtp.md#whole-run-and-per-module-minimums). ## Extension options by scenario diff --git a/docs/core/testing/microsoft-testing-platform-troubleshooting.md b/docs/core/testing/microsoft-testing-platform-troubleshooting.md index d1e9958916d8d..220877844f534 100644 --- a/docs/core/testing/microsoft-testing-platform-troubleshooting.md +++ b/docs/core/testing/microsoft-testing-platform-troubleshooting.md @@ -3,7 +3,7 @@ title: Microsoft.Testing.Platform (MTP) troubleshooting description: Troubleshoot MTP issues, exit codes, and known problems. author: Evangelink ms.author: amauryleve -ms.date: 09/10/2026 +ms.date: 09/11/2026 ai-usage: ai-assisted --- @@ -25,7 +25,7 @@ MTP uses known exit codes to communicate test failure or app errors. Exit codes | `5` | The exit code `5` indicates that the command-line arguments passed to the test app were invalid. | | `6` (no longer used) | Exit code `6` is no longer produced by the platform; it previously indicated that the test session was using a non-implemented feature. | | `7` | The exit code `7` indicates that a test session was unable to complete successfully, and likely crashed. It's possible that this was caused by a test session that was run via a test controller's extension point. | -| `8` | The exit code `8` indicates that the test session ran zero tests under the strict `--zero-tests-policy`. | +| `8` | The exit code `8` indicates that the test session discovered no tests, or that every selected test was skipped under the strict `--zero-tests-policy`. | | `9` | The exit code `9` indicates that the run executed fewer tests than an explicit `--minimum-expected-tests` value requires, including zero tests. | | `10` | The exit code `10` indicates that the test adapter, Testing.Platform Test Framework, MSTest, NUnit, or xUnit, failed to run tests for an infrastructure reason unrelated to the test's self. An example is failing to create a fixture needed by tests. | | `11` | The exit code `11` indicates that the test process will exit if dependent process exits. | @@ -39,10 +39,10 @@ To enable verbose logging and troubleshoot issues, see [Diagnostic logging](#dia ### Zero tests in a multi-module run -When `dotnet test` runs several test modules, exit code `8` is a per-module signal, while the zero-tests verdict for the whole run is decided once from the total number of tests that ran. A single empty module therefore doesn't fail the whole run, although the module keeps its `Exit code: 8` diagnostic in the output. For more information, see [Whole-run and per-module minimums](../tools/dotnet-test-mtp.md#whole-run-and-per-module-minimums). +When `dotnet test` runs several test modules, exit code `8` is a per-module signal, while the zero-tests verdict for the whole run is decided once from the aggregated results. A single empty module therefore doesn't fail the whole run, although the module keeps its `Exit code: 8` diagnostic in the output. When you don't set a global minimum, an all-skipped whole run is treated as a zero-test run regardless of the per-module `--zero-tests-policy` value. For more information, see [Whole-run and per-module minimums](../tools/dotnet-test-mtp.md#whole-run-and-per-module-minimums). > [!NOTE] -> This whole-run aggregation requires the .NET 11 SDK (11.0.1xx) or a later version. +> This whole-run zero-tests verdict requires the .NET 11 SDK or a later version. ### Ignore specific exit codes diff --git a/docs/core/tools/dotnet-test-mtp.md b/docs/core/tools/dotnet-test-mtp.md index 0ec0a19c0886a..a5b738ebf2f4e 100644 --- a/docs/core/tools/dotnet-test-mtp.md +++ b/docs/core/tools/dotnet-test-mtp.md @@ -1,7 +1,7 @@ --- title: dotnet test command with Microsoft.Testing.Platform (MTP) description: The dotnet test command is used to execute unit tests in a given project using MTP. -ms.date: 09/10/2026 +ms.date: 09/11/2026 ai-usage: ai-assisted --- # dotnet test with Microsoft.Testing.Platform (MTP) @@ -98,12 +98,12 @@ With MTP, `dotnet test` operates faster than with VSTest. The test-related argum - **`--minimum-expected-tests `** - Specifies the minimum number of tests that must run across the whole run. If the total number of tests that ran is less than the specified minimum, the test run fails with exit code 9. For more information about exit codes, see [MTP exit codes](../testing/microsoft-testing-platform-troubleshooting.md#exit-codes). + Specifies a positive minimum number of tests for the whole run. If the aggregated test count is less than the specified minimum, the test run fails with exit code 9. The global count includes skipped tests. For more information about exit codes, see [MTP exit codes](../testing/microsoft-testing-platform-troubleshooting.md#exit-codes). Because this option appears before `--`, it's a global (whole-run) option. To require a minimum for each test module instead, pass the option after `--` so that it's forwarded to every test module. For more information, see [Whole-run and per-module minimums](#whole-run-and-per-module-minimums). > [!NOTE] - > Whole-run aggregation of the minimum requires the .NET 11 SDK (11.0.1xx) or a later version. In earlier SDKs, the option applies to each test module. + > The global minimum requires the .NET 10 SDK (10.0.100) or a later version. - [!INCLUDE [arch](includes/cli-arch.md)] @@ -197,7 +197,7 @@ The same parser behavior applies to `dotnet run` and `dotnet build`. For a detai ## Whole-run and per-module minimums -Starting with the .NET 11 SDK (11.0.1xx), the position of an argument determines its scope: +For `--minimum-expected-tests`, the `--` separator determines the option's scope: - Arguments *before* `--` are global. The `dotnet test` orchestrator interprets them for the whole run. - Arguments *after* `--` are local. `dotnet test` forwards them to each test module, so each module applies them independently. @@ -210,12 +210,27 @@ dotnet test --minimum-expected-tests 5 -- --minimum-expected-tests 2 The preceding command requires at least 5 tests across the whole run and at least 2 tests in each test module. -The zero-tests verdict for the whole run is decided once, from the total number of tests that ran. A module that matches no tests, for example because of `--test-modules` or a global `--filter`, exits with code 8 (`ZeroTests`), but that code is normalized to success before the results are aggregated. As a result, a single empty module no longer fails the whole run, although the module keeps its `Exit code: 8` diagnostic in the output for visibility. +The two scopes count skipped tests differently: -When you specify `--minimum-expected-tests` and the minimum isn't met, the run fails with exit code 9 (`MinimumExpectedTestsPolicyViolation`). This code is distinct from 8 so that a stricter global or per-module minimum isn't confused with an empty module. +| Scope | Do skipped tests count toward the minimum? | +| --- | --- | +| Global | Yes. The `dotnet test` aggregated total includes skipped tests. | +| Per module | No. MTP excludes skipped tests from the number of tests that ran. | + +Starting with the .NET 11 SDK, the zero-tests verdict for the whole run is decided once from the aggregated results. A module that matches no tests, for example because of `--test-modules` or a global `--filter`, exits with code 8 (`ZeroTests`), but that code is normalized to success before the results are aggregated. As a result, a single empty module doesn't fail the whole run, although the module keeps its `Exit code: 8` diagnostic in the output for visibility. + +MTP 4.3.0 and later versions provide `--zero-tests-policy `. The default value, `allow-skipped`, lets an all-skipped module succeed. The `strict` value treats skipped tests as not run, so an all-skipped module exits with code 8. Pass the option after `--` to forward it to each test module: + +```dotnetcli +dotnet test -- --zero-tests-policy strict +``` + +When you don't set a global minimum, the .NET 11 SDK determines the whole-run zero-tests verdict separately. An all-skipped whole run exits with code 8 regardless of the per-module `--zero-tests-policy` value. + +When you specify `--minimum-expected-tests` and the minimum isn't met, the run fails with exit code 9 (`MinimumExpectedTestsPolicyViolation`). This code is distinct from 8 so that a stricter global or per-module minimum isn't confused with an empty module. For a per-module minimum to return code 9 when the module runs zero tests, the test module must use MTP 4.4.0 or a later version. > [!NOTE] -> `--minimum-expected-tests 0` isn't an alias for `--ignore-exit-code 8`. To suppress the zero-tests exit code, use `--ignore-exit-code 8`. +> `--minimum-expected-tests 0` is invalid. To suppress the zero-tests exit code, use `--ignore-exit-code 8`. ## Examples