feat(cli): --extensions/--exclude-extensions scenario selection - #420
Open
pcarleton wants to merge 1 commit into
Open
feat(cli): --extensions/--exclude-extensions scenario selection#420pcarleton wants to merge 1 commit into
pcarleton wants to merge 1 commit into
Conversation
Extension scenarios (DPoP, WIF, client-credentials, EMA, tasks) are
optional by definition, but --suite all selects them unconditionally,
so SDKs that don't implement an extension end up baselining its
scenarios as expected failures — a file whose semantics ("burns down
per milestone") misrepresent optional surface as conformance debt.
Give the runner explicit extension selection instead:
- --extensions <ids>: keep only the listed extensions' scenarios in
the selected suite ("none" deselects them all). Asking for an
extension with no scenarios in the suite is an error, so a wrong
suite or a --spec-version that already filtered them out cannot
silently pass as tested.
- --exclude-extensions <ids>: drop the listed extensions' scenarios
and run the rest ("none" deselects nothing; excluding an absent
extension is a satisfied no-op). Mutually exclusive with
--extensions via commander conflicts.
IDs may omit the io.modelcontextprotocol/ prefix (auth/dpop, tasks).
Spec-timeline scenarios always pass through, an explicit --scenario
runs unfiltered (with a note when combined with the flags), deselected
scenarios are logged, an empty value errors (an unset CI variable must
not silently change which scenarios run), and a selection that
deselects every scenario in a non-empty suite fails loudly. The list
command accepts both flags as a silent dry-run preview, and sdk-runner
passes them through to the client/server invocations. The
authorization command is untouched: no extension-tagged AS scenarios
exist, so the flags would be a guaranteed error there.
Verified end-to-end: typescript-sdk client --suite all
--exclude-extensions auth/dpop,auth/wif passes 439/0 with no
expected-failures file, and python-sdk server --suite all
--extensions none passes 137/0 likewise.
commit: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Extension scenarios (DPoP, WIF, client-credentials, EMA, tasks) are optional by definition, but
--suite allselects them unconditionally. SDKs that don't implement an extension end up baselining its scenarios inexpected-failures.yml— a file whose stated semantics ("burns down per milestone", "MUST be removed as they're fixed") misrepresent optional surface as conformance debt. Both typescript-sdk and python-sdk currently carry DPoP/WIF and tasks entries for exactly this reason.Change
Explicit, flexible runner config for which extensions run:
--extensions <ids>— keep only these extensions' scenarios in the selected suite (nonedeselects them all). Asking for an extension that contributes no scenarios to the suite is an error, so a wrong suite or a--spec-versionthat already filtered it out can't silently pass as tested.--exclude-extensions <ids>— run everything but these (nonedeselects nothing; excluding an absent extension is a satisfied no-op). Mutually exclusive with--extensions(commanderconflicts).Semantics:
auth/dpop,tasks,io.modelcontextprotocol/tasks).--scenarioruns unfiltered (a note is printed when combined with the flags).--extensions "$UNSET_VAR"must not silently change which scenarios run); a selection that deselects every scenario in a non-empty suite fails loudly.listaccepts both flags as a dry-run preview of a selection;sdkpasses them through to the child client/server invocations (including empty values, so the child's guard fires, and with the same immediate conflict check).authorizationis untouched — no extension-tagged AS scenarios exist, so the flags would be a guaranteed error there.Verification
client --suite all --exclude-extensions auth/dpop,auth/wif→ 439 passed, 0 failed with no expected-failures file (today this requires baselining 3 scenarios).server --suite all --extensions none→ 137 passed, 0 failed with no baseline (today: 9 baselined tasks scenarios).noneon both flags, conflicts, unknown IDs, include-with-no-match, spec-version interactions, suite emptied by selection,--scenariocombinations).src/scenarios/extension-filter.test.ts; full suite 459/459; typecheck + lint clean.Follow-up once released: both SDKs can swap their DPoP/WIF/tasks baseline entries for
--exclude-extensionslegs (the stale-entry enforcement forces the cleanup in the same pin-bump PR).