feat(sdk-runner): expand KNOWN_SDKS to all tier 1 SDKs - #417
Conversation
commit: |
- go-sdk: point at the relocated conformance/ fixtures (the old examples/server/conformance path is gone on main), add the everything-client fixture that now exists upstream, and wire conformance/baseline.yml as the expected-failures baseline. - python-sdk: new entry for main, which targets the 2026-07-28 revision (same uv workspace layout as the v1.x entry). - csharp-sdk: new entry driving the ConformanceClient / ConformanceServer fixtures via the .NET 10 SDK. The client takes the scenario as argv[0], so the command bridges MCP_CONFORMANCE_SCENARIO into argv; the server exposes the SEP-2575 stateless lifecycle at /stateless.
The example now shows field overrides on a built-in entry rather than a one-off ref, so the introducing sentence shouldn't promise a ref override.
5fccb2e to
02cbfae
Compare
|
Ran every entry end-to-end from this branch's build to show the config works per SDK: Toolchain prereqs on the runner box: .NET 10 SDK (csharp-sdk go-sdk — client 411/411, server 115/115 (exit 1 only from now-stale baseline entries)$ node dist/index.js sdk go-sdk@main --mode client --suite all --spec-version 2026-07-28 --timeout 60000
[sdk] Cloning https://github.com/modelcontextprotocol/go-sdk.git -> .sdk-under-test/go-sdk/main
[sdk] HEAD is e761950
[sdk] Building: go build -o ./.conformance-server ./conformance/everything-server && go build -o ./.conformance-client ./conformance/everything-client
[sdk] conformance client --command ./.conformance-client --suite all --timeout 60000 --spec-version 2026-07-28 --expected-failures .../go-sdk/main/conformance/baseline.yml
...
✓ auth/iss-normalized: 15 passed, 0 failed
✓ sep-2322-client-request-state: 5 passed, 0 failed
✓ http-custom-headers: 18 passed, 0 failed
✓ json-schema-ref-no-deref: 1 passed, 0 failed
Total: 411 passed, 0 failed, 0 warnings
Stale baseline entries (now passing - remove from baseline):
✓ json-schema-ref-no-deref$ node dist/index.js sdk go-sdk@main --mode server --suite all --spec-version 2026-07-28 \
--server-cmd './.conformance-server -http=localhost:3000' # SEP-2575 override from the entry comment
[sdk] Starting server: ./.conformance-server -http=localhost:3000
[sdk] Server ready at http://localhost:3000
...
✓ input-required-result-validate-input: 2 passed, 0 failed
Total: 115 passed, 0 failed
Stale baseline entries (now passing - remove from baseline):
✓ server-statelessBoth "stale" flags are the baseline machinery working as intended: go-sdk's python-sdk (new entry, main) — client 371/371, server 114/114, exit 0 both$ node dist/index.js sdk python-sdk@main --mode client --suite all --spec-version 2026-07-28 --timeout 60000 \
--expected-failures .../.github/actions/conformance/expected-failures.2026-07-28.yml
[sdk] Cloning https://github.com/modelcontextprotocol/python-sdk.git -> .sdk-under-test/python-sdk/main
[sdk] HEAD is dcd9c1ee
[sdk] Building: uv sync --frozen --all-extras --all-packages
[sdk] conformance client --command uv run --frozen python .github/actions/conformance/client.py ...
...
✓ auth/authorization-server-migration: 27 passed, 0 failed
✓ sep-2322-client-request-state: 5 passed, 0 failed
✓ http-custom-headers: 18 passed, 0 failed
✓ json-schema-ref-no-deref: 1 passed, 0 failed
Total: 371 passed, 0 failed, 0 warnings$ node dist/index.js sdk python-sdk@main --mode server --suite all --spec-version 2026-07-28 \
--expected-failures .../.github/actions/conformance/expected-failures.2026-07-28.yml
[sdk] Starting server: uv run --frozen mcp-everything-server --port 3000
[sdk] Server ready at http://localhost:3000/mcp
...
✓ input-required-result-tampered-state: 1 passed, 0 failed
✓ input-required-result-validate-input: 2 passed, 0 failed
Total: 114 passed, 0 failedcsharp-sdk (new entry) — client 311/321, server 90/110; failures are SDK-side findings, not entry misconfiguration$ node dist/index.js sdk csharp-sdk@main --mode client --suite all --spec-version 2026-07-28 --timeout 60000
[sdk] Cloning https://github.com/modelcontextprotocol/csharp-sdk.git -> .sdk-under-test/csharp-sdk/main
[sdk] HEAD is 950ac7c
[sdk] Building: dotnet build tests/ModelContextProtocol.ConformanceClient -c Release -f net10.0 && dotnet build tests/ModelContextProtocol.ConformanceServer -c Release -f net10.0
[sdk] conformance client --command bash -c 'exec dotnet artifacts/bin/ModelContextProtocol.ConformanceClient/Release/net10.0/ModelContextProtocol.ConformanceClient.dll "$MCP_CONFORMANCE_SCENARIO" "$1"' conformance-client ...
...
✓ auth/token-endpoint-auth-none: 16 passed, 0 failed
✗ auth/authorization-server-migration: 13 passed, 1 failed
✓ auth/metadata-issuer-mismatch: 3 passed, 0 failed
✗ sep-2322-client-request-state: 0 passed, 5 failed
✓ http-custom-headers: 18 passed, 0 failed
✗ json-schema-ref-no-deref: 0 passed, 1 failed
Total: 311 passed, 10 failed, 0 warnings$ node dist/index.js sdk csharp-sdk@main --mode server --suite all --spec-version 2026-07-28 \
--server-url http://localhost:3000/stateless # 2026-07-28 override from the entry comment
[sdk] Starting server: dotnet artifacts/bin/ModelContextProtocol.ConformanceServer/Release/net10.0/ModelContextProtocol.ConformanceServer.dll --urls http://localhost:3000
[sdk] Server ready at http://localhost:3000/stateless
...
✓ caching: 7 passed, 0 failed
✓ http-header-validation: 13 passed, 0 failed
✓ input-required-result-multi-round: 3 passed, 0 failed
Total: 90 passed, 20 failedThe 30 failing checks are genuine csharp-sdk conformance gaps, and the same scenarios pass through the identical plumbing on the other SDKs. The dominant cause is that csharp-sdk still implements the pre-modelcontextprotocol/modelcontextprotocol#3002 discover shape ( typescript-sdk / typescript-sdk-v1 (entries unchanged) — re-validated as a control, incl. the
|
Expands
KNOWN_SDKSso every Tier 1 SDK (TypeScript, Python, C#, Go — per the SDK tiering table) can be driven byconformance sdk <name> --mode client|serverwith no CLI overrides:examples/server/conformance, which no longer exists onmain— fixtures moved toconformance/everything-{client,server}. Also adds the client fixture (go-sdk now has one) and wiresconformance/baseline.ymlas the expected-failures baseline. The server command pins-stateless=falseto match go-sdk's ownscripts/server-conformance.sh; a SEP-2575 run overrides--server-cmd(documented in the entry comment and README example).main, which targets 2026-07-28. Same uv-workspace layout as the existingpython-sdk-v1entry; defaults to the dated-spec baseline (expected-failures.yml), withexpected-failures.2026-07-28.ymlavailable via--expected-failuresfor--spec-version 2026-07-28runs.tests/ModelContextProtocol.ConformanceClient/ConformanceServervia the .NET 10 SDK. Two quirks handled in the entry: the client takes the scenario asargv[0]rather than readingMCP_CONFORMANCE_SCENARIO, so the command bridges the env var into argv; and the repo usesUseArtifactsOutput, so binaries land underartifacts/bin/. The server exposes the SEP-2575 stateless lifecycle at/stateless(override--server-urlfor 2026-07-28 runs).Also refreshes the README's one-off-override example, which referenced a
mcp_go_client_oauthbuild tag that no longer exists in go-sdk.Verified by running the full matrix (client+server × all five entries) locally at
--suite all --spec-version 2026-07-28; go-sdk and python-sdk main come back fully green (go's baseline now has two stale entries to remove upstream), and the csharp entries produce clean per-scenario results (311/321 client, 90/110 server checks).