Skip to content

feat(cli): --extensions/--exclude-extensions scenario selection #420

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pcarleton wants to merge 1 commit into
base: main
Choose a base branch
from
+394 −55
Open

feat(cli): --extensions/--exclude-extensions scenario selection #420

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ npx @modelcontextprotocol/conformance client --command "<client-command>" --scen
- `--suite` - Run a suite of tests in parallel: `all`, `core`, `extensions`, `backcompat`, `auth`, `metadata`, `draft` (scenarios targeting the in-progress draft spec), or `sep-835`
- `--spec-version <version>` - Filter scenarios by spec version (e.g., `2025-11-25`, `2026-07-28`; `draft` is accepted as an alias for the current draft identifier). The draft version selects the latest dated release plus any draft-only scenarios. When omitted, the version is inferred from the scenario's spec applicability (draft-only scenarios run at the draft version, everything else at the latest dated release); an explicitly requested version outside a scenario's applicability window skips the scenario (exit 0) unless `--force` is passed
- `--force` - Run a scenario even if it is not applicable at the requested `--spec-version`
- `--extensions <ids>` / `--exclude-extensions <ids>` - Select which protocol extensions' scenarios run within the chosen suite (mutually exclusive; spec-timeline scenarios are unaffected, and a lone `--scenario` runs unfiltered). `--extensions` keeps only the listed extensions (`none` deselects them all); `--exclude-extensions` runs everything but the listed ones (`none` deselects nothing). IDs are comma-separated and may omit the `io.modelcontextprotocol/` prefix, e.g. `--exclude-extensions auth/dpop,auth/wif` or `--extensions oauth-client-credentials`. Deselected scenarios are listed in the run output; asking to include an extension with no scenarios in the suite is an error
- `--expected-failures <path>` - Path to YAML baseline file of known failures (see [Expected Failures](#expected-failures))
- `--timeout` - Timeout in milliseconds (default: 30000)
- `--verbose` - Show verbose output
Expand All @@ -83,6 +84,7 @@ npx @modelcontextprotocol/conformance server --url <url> [--scenario <scenario>]
- `--url` - URL of the server to test
- `--scenario <scenario>` - Test scenario to run (e.g., "server-initialize"). Runs all available scenarios by default
- `--suite <suite>` - Suite to run: "active" (default; excludes pending and draft-spec scenarios), "all", "draft" (scenarios targeting the in-progress draft spec), or "pending"
- `--extensions <ids>` / `--exclude-extensions <ids>` - Select which protocol extensions' scenarios run within the chosen suite (mutually exclusive; spec-timeline scenarios are unaffected, and a lone `--scenario` runs unfiltered). `--extensions` keeps only the listed extensions (`none` deselects them all); `--exclude-extensions` runs everything but the listed ones (`none` deselects nothing). IDs are comma-separated and may omit the `io.modelcontextprotocol/` prefix, e.g. `--exclude-extensions tasks` or `--extensions tasks`. Deselected scenarios are listed in the run output; asking to include an extension with no scenarios in the suite is an error
- `--expected-failures <path>` - Path to YAML baseline file of known failures (see [Expected Failures](#expected-failures))
- `--verbose` - Show verbose output

Expand Down
230 changes: 180 additions & 50 deletions src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/usr/bin/env node

import { Command } from 'commander';
import { Command, Option } from 'commander';
import { ZodError } from 'zod';
import { promises as fs } from 'fs';
import {
Expand Down Expand Up @@ -33,9 +33,13 @@ import {
getScenarioSpecVersions,
listClientScenariosForAuthorizationServer,
listClientScenariosForAuthorizationServerForSpec,
resolveSpecVersion
resolveSpecVersion,
resolveExtensionIds,
filterScenariosByExtensions,
getScenarioExtensionId
} from './scenarios';
import type { SpecVersion } from './scenarios';
import type { ExtensionSelection } from './scenarios';
import { ConformanceCheck } from './types';
import {
AuthorizationServerOptionsSchema,
Expand Down Expand Up @@ -78,6 +82,90 @@ function filterScenariosBySpecVersion(
return allScenarios.filter((s) => allowed.has(s));
}

const EXTENSION_SELECTION_NOTE =
'Note: --extensions/--exclude-extensions apply to --suite selection; a lone --scenario runs regardless.';

/** Add the shared --extensions/--exclude-extensions option pair to a command. */
function withExtensionSelectionOptions(cmd: Command): Command {
return cmd
.option(
'--extensions <ids>',
'Comma-separated extension IDs: keep only these extensions\' scenarios in the selected suite (spec-timeline scenarios are unaffected; "none" deselects all extension scenarios)'
)
.addOption(
new Option(
'--exclude-extensions <ids>',
'Comma-separated extension IDs whose scenarios are deselected from the selected suite ("none" deselects nothing; mutually exclusive with --extensions)'
).conflicts('extensions')
);
}

/**
* Parse `--extensions` / `--exclude-extensions` (commander enforces their
* mutual exclusion) into an extension selection, or undefined when neither
* is given (all extension scenarios stay selected, the historical behavior).
* `none` is accepted by both: `--extensions none` deselects every extension
* scenario, `--exclude-extensions none` deselects nothing.
*/
function parseExtensionSelection(options: {
extensions?: string;
excludeExtensions?: string;
}): ExtensionSelection | undefined {
if (options.extensions !== undefined) {
return { mode: 'include', ids: resolveExtensionIds(options.extensions) };
}
if (options.excludeExtensions !== undefined) {
return {
mode: 'exclude',
ids: resolveExtensionIds(options.excludeExtensions)
};
}
return undefined;
}

/**
* Apply the extension selection to a suite's scenario list, logging what was
* deselected. An empty result is an error: a run whose selection removes
* every scenario (or whose --spec-version already removed all extension
* scenarios the selection asked for) should fail loudly, not exit 0 having
* tested nothing.
*/
function applyExtensionSelection(
scenarios: string[],
selection: ExtensionSelection | undefined
): string[] {
if (!selection) return scenarios;
const { kept, dropped } = filterScenariosByExtensions(scenarios, selection);
if (dropped.length > 0) {
console.log(
`Deselected ${dropped.length} extension scenario(s): ${dropped.join(', ')}\n`
);
}
if (selection.mode === 'include') {
// Asking to include an extension that contributes nothing to the
// selected suite (wrong suite, or --spec-version already removed its
// scenarios) must not silently pass as if it were tested. Excluding an
// absent extension, by contrast, is a satisfied no-op — including when
// the suite was already empty before the selection.
const unmatched = selection.ids.filter(
(id) => !kept.some((name) => getScenarioExtensionId(name) === id)
);
if (unmatched.length > 0) {
console.error(
`No scenarios for extension(s) ${unmatched.join(', ')} in the selected suite; check --suite and --spec-version`
);
process.exit(1);
}
}
if (kept.length === 0 && dropped.length > 0) {
console.error(
'The extension selection deselected every scenario in the suite; check --suite and --extensions/--exclude-extensions'
);
process.exit(1);
}
return kept;
}

const program = new Command();

program
Expand All @@ -86,28 +174,33 @@ program
.version(packageJson.version);

// Client command - tests a client implementation against scenarios
program
.command('client')
.description(
'Run conformance tests against a client implementation or start interactive mode'
)
.option('--command <command>', 'Command to run the client')
.option('--scenario <scenario>', 'Scenario to test')
.option('--suite <suite>', 'Run a suite of tests in parallel (e.g., "auth")')
.option('--timeout <ms>', 'Timeout in milliseconds', '30000')
.option(
'--expected-failures <path>',
'Path to YAML file listing expected failures (baseline)'
)
.option('-o, --output-dir <path>', 'Save results to this directory')
.option(
'--spec-version <version>',
'Filter scenarios by spec version (cumulative for date versions)'
)
.option(
'--force',
'Run a scenario even if it is not applicable at the requested --spec-version'
)
withExtensionSelectionOptions(
program
.command('client')
.description(
'Run conformance tests against a client implementation or start interactive mode'
)
.option('--command <command>', 'Command to run the client')
.option('--scenario <scenario>', 'Scenario to test')
.option(
'--suite <suite>',
'Run a suite of tests in parallel (e.g., "auth")'
)
.option('--timeout <ms>', 'Timeout in milliseconds', '30000')
.option(
'--expected-failures <path>',
'Path to YAML file listing expected failures (baseline)'
)
.option('-o, --output-dir <path>', 'Save results to this directory')
.option(
'--spec-version <version>',
'Filter scenarios by spec version (cumulative for date versions)'
)
.option(
'--force',
'Run a scenario even if it is not applicable at the requested --spec-version'
)
)
.option('--verbose', 'Show verbose output')
.action(async (options) => {
try {
Expand All @@ -118,6 +211,8 @@ program
? resolveSpecVersion(options.specVersion)
: undefined;

const extensionSelection = parseExtensionSelection(options);

// Handle suite mode
if (options.suite) {
if (!options.command) {
Expand Down Expand Up @@ -152,6 +247,7 @@ program
'client'
);
}
scenarios = applyExtensionSelection(scenarios, extensionSelection);
console.log(
`Running ${suiteName} suite (${scenarios.length} scenarios) in parallel...\n`
);
Expand Down Expand Up @@ -270,6 +366,10 @@ program
process.exit(1);
}

if (extensionSelection) {
console.log(EXTENSION_SELECTION_NOTE);
}

// Validate options with Zod for single scenario mode
const validated = ClientOptionsSchema.parse(options);

Expand Down Expand Up @@ -338,32 +438,34 @@ program
});

// Server command - tests a server implementation
program
.command('server')
.description('Run conformance tests against a server implementation')
.requiredOption('--url <url>', 'URL of the server to test')
.option(
'--scenario <scenario>',
'Scenario to test (defaults to active suite if not specified)'
)
.option(
'--suite <suite>',
'Suite to run: "active" (default, excludes pending and draft), "all", "draft", or "pending"',
'active'
)
.option(
'--expected-failures <path>',
'Path to YAML file listing expected failures (baseline)'
)
.option('-o, --output-dir <path>', 'Save results to this directory')
.option(
'--spec-version <version>',
'Filter scenarios by spec version (cumulative for date versions)'
)
.option(
'--force',
'Run a scenario even if it is not applicable at the requested --spec-version'
)
withExtensionSelectionOptions(
program
.command('server')
.description('Run conformance tests against a server implementation')
.requiredOption('--url <url>', 'URL of the server to test')
.option(
'--scenario <scenario>',
'Scenario to test (defaults to active suite if not specified)'
)
.option(
'--suite <suite>',
'Suite to run: "active" (default, excludes pending and draft), "all", "draft", or "pending"',
'active'
)
.option(
'--expected-failures <path>',
'Path to YAML file listing expected failures (baseline)'
)
.option('-o, --output-dir <path>', 'Save results to this directory')
.option(
'--spec-version <version>',
'Filter scenarios by spec version (cumulative for date versions)'
)
.option(
'--force',
'Run a scenario even if it is not applicable at the requested --spec-version'
)
)
.option('--verbose', 'Show verbose output (JSON instead of pretty print)')
.action(async (options) => {
try {
Expand All @@ -376,8 +478,14 @@ program
? resolveSpecVersion(options.specVersion)
: undefined;

const extensionSelection = parseExtensionSelection(options);

// If a single scenario is specified, run just that one
if (validated.scenario) {
if (extensionSelection) {
console.log(EXTENSION_SELECTION_NOTE);
}

const result = await runServerConformanceTest(
validated.url,
validated.scenario,
Expand Down Expand Up @@ -441,6 +549,7 @@ program
'server'
);
}
scenarios = applyExtensionSelection(scenarios, extensionSelection);

console.log(
`Running ${suite} suite (${scenarios.length} scenarios) against ${validated.url}\n`
Expand Down Expand Up @@ -699,10 +808,26 @@ program
'--spec-version <version>',
'Filter scenarios by spec version (cumulative for date versions)'
)
.option(
'--extensions <ids>',
'Preview an extension selection: list only these extensions\' scenarios plus spec-timeline scenarios ("none" hides all extension scenarios)'
)
.addOption(
new Option(
'--exclude-extensions <ids>',
"Preview an extension deselection: hide these extensions' scenarios (mutually exclusive with --extensions)"
).conflicts('extensions')
)
.action((options) => {
const specVersionFilter = options.specVersion
? resolveSpecVersion(options.specVersion)
: undefined;
const extensionSelection = parseExtensionSelection(options);
// Listing is a preview: filter silently, no fail-loud checks.
const applySelection = (names: string[]): string[] =>
extensionSelection
? filterScenariosByExtensions(names, extensionSelection).kept
: names;

if (
options.server ||
Expand All @@ -717,6 +842,7 @@ program
'server'
);
}
serverScenarios = applySelection(serverScenarios);
serverScenarios.forEach((s) => {
const v = getScenarioSpecVersions(s);
console.log(` - ${s}${v ? ` [${v}]` : ''}`);
Expand All @@ -739,6 +865,7 @@ program
'client'
);
}
clientScenarioNames = applySelection(clientScenarioNames);
clientScenarioNames.forEach((s) => {
const v = getScenarioSpecVersions(s);
console.log(` - ${s}${v ? ` [${v}]` : ''}`);
Expand All @@ -764,6 +891,9 @@ program
'authorization'
);
}
authorizationServerScenarios = applySelection(
authorizationServerScenarios
);
authorizationServerScenarios.forEach((s) => {
const v = getScenarioSpecVersions(s);
console.log(` - ${s}${v ? ` [${v}]` : ''}`);
Expand Down
Loading
Loading