Skip to content

feat(sdk-runner): expand KNOWN_SDKS to all tier 1 SDKs - #417

Merged
pcarleton merged 2 commits into
mainfrom
paulc/sdk-runner-tier1-coverage
Jul 27, 2026
Merged

feat(sdk-runner): expand KNOWN_SDKS to all tier 1 SDKs#417
pcarleton merged 2 commits into
mainfrom
paulc/sdk-runner-tier1-coverage

Conversation

@pcarleton

Copy link
Copy Markdown
Member

Expands KNOWN_SDKS so every Tier 1 SDK (TypeScript, Python, C#, Go — per the SDK tiering table) can be driven by conformance sdk <name> --mode client|server with no CLI overrides:

  • go-sdk: the entry pointed at examples/server/conformance, which no longer exists on main — fixtures moved to conformance/everything-{client,server}. Also adds the client fixture (go-sdk now has one) and wires conformance/baseline.yml as the expected-failures baseline. The server command pins -stateless=false to match go-sdk's own scripts/server-conformance.sh; a SEP-2575 run overrides --server-cmd (documented in the entry comment and README example).
  • python-sdk (new entry): main, which targets 2026-07-28. Same uv-workspace layout as the existing python-sdk-v1 entry; defaults to the dated-spec baseline (expected-failures.yml), with expected-failures.2026-07-28.yml available via --expected-failures for --spec-version 2026-07-28 runs.
  • csharp-sdk (new entry): drives tests/ModelContextProtocol.ConformanceClient / ConformanceServer via the .NET 10 SDK. Two quirks handled in the entry: the client takes the scenario as argv[0] rather than reading MCP_CONFORMANCE_SCENARIO, so the command bridges the env var into argv; and the repo uses UseArtifactsOutput, so binaries land under artifacts/bin/. The server exposes the SEP-2575 stateless lifecycle at /stateless (override --server-url for 2026-07-28 runs).

Also refreshes the README's one-off-override example, which referenced a mcp_go_client_oauth build tag that no longer exists in go-sdk.

Verified by running the full matrix (client+server × all five entries) locally at --suite all --spec-version 2026-07-28; go-sdk and python-sdk main come back fully green (go's baseline now has two stale entries to remove upstream), and the csharp entries produce clean per-scenario results (311/321 client, 90/110 server checks).

@pkg-pr-new

pkg-pr-new Bot commented Jul 27, 2026

Copy link
Copy Markdown

Open in StackBlitz

npx https://pkg.pr.new/@modelcontextprotocol/conformance@417

commit: 02cbfae

- go-sdk: point at the relocated conformance/ fixtures (the old
  examples/server/conformance path is gone on main), add the
  everything-client fixture that now exists upstream, and wire
  conformance/baseline.yml as the expected-failures baseline.
- python-sdk: new entry for main, which targets the 2026-07-28
  revision (same uv workspace layout as the v1.x entry).
- csharp-sdk: new entry driving the ConformanceClient /
  ConformanceServer fixtures via the .NET 10 SDK. The client takes the
  scenario as argv[0], so the command bridges MCP_CONFORMANCE_SCENARIO
  into argv; the server exposes the SEP-2575 stateless lifecycle at
  /stateless.
The example now shows field overrides on a built-in entry rather than a
one-off ref, so the introducing sentence shouldn't promise a ref
override.
@pcarleton
pcarleton force-pushed the paulc/sdk-runner-tier1-coverage branch from 5fccb2e to 02cbfae Compare July 27, 2026 12:38
@pcarleton

Copy link
Copy Markdown
Member Author

Ran every entry end-to-end from this branch's build to show the config works per SDK: npm run build, then node dist/index.js sdk <name> --mode client|server --suite all --spec-version 2026-07-28 (plus the documented per-SDK overrides for the SEP-2575 legs). Output below is trimmed from real runs on 2026-07-27 against the alpha.10 scenario set.

Toolchain prereqs on the runner box: .NET 10 SDK (csharp-sdk global.json), go ≥ 1.25 (an older local go auto-fetches the toolchain via GOTOOLCHAIN), uv, node ≥ 20.

go-sdk — client 411/411, server 115/115 (exit 1 only from now-stale baseline entries)
$ node dist/index.js sdk go-sdk@main --mode client --suite all --spec-version 2026-07-28 --timeout 60000
[sdk] Cloning https://github.com/modelcontextprotocol/go-sdk.git -> .sdk-under-test/go-sdk/main
[sdk] HEAD is e761950
[sdk] Building: go build -o ./.conformance-server ./conformance/everything-server && go build -o ./.conformance-client ./conformance/everything-client
[sdk] conformance client --command ./.conformance-client --suite all --timeout 60000 --spec-version 2026-07-28 --expected-failures .../go-sdk/main/conformance/baseline.yml
...
✓ auth/iss-normalized: 15 passed, 0 failed
✓ sep-2322-client-request-state: 5 passed, 0 failed
✓ http-custom-headers: 18 passed, 0 failed
✓ json-schema-ref-no-deref: 1 passed, 0 failed

Total: 411 passed, 0 failed, 0 warnings

Stale baseline entries (now passing - remove from baseline):
  ✓ json-schema-ref-no-deref
$ node dist/index.js sdk go-sdk@main --mode server --suite all --spec-version 2026-07-28 \
    --server-cmd './.conformance-server -http=localhost:3000'   # SEP-2575 override from the entry comment
[sdk] Starting server: ./.conformance-server -http=localhost:3000
[sdk] Server ready at http://localhost:3000
...
✓ input-required-result-validate-input: 2 passed, 0 failed

Total: 115 passed, 0 failed

Stale baseline entries (now passing - remove from baseline):
  ✓ server-stateless

Both "stale" flags are the baseline machinery working as intended: go-sdk's conformance/baseline.yml predicted these entries would start passing once the suite picked up modelcontextprotocol/modelcontextprotocol#3002, which alpha.10 did — the cleanup belongs upstream in go-sdk.

python-sdk (new entry, main) — client 371/371, server 114/114, exit 0 both
$ node dist/index.js sdk python-sdk@main --mode client --suite all --spec-version 2026-07-28 --timeout 60000 \
    --expected-failures .../.github/actions/conformance/expected-failures.2026-07-28.yml
[sdk] Cloning https://github.com/modelcontextprotocol/python-sdk.git -> .sdk-under-test/python-sdk/main
[sdk] HEAD is dcd9c1ee
[sdk] Building: uv sync --frozen --all-extras --all-packages
[sdk] conformance client --command uv run --frozen python .github/actions/conformance/client.py ...
...
✓ auth/authorization-server-migration: 27 passed, 0 failed
✓ sep-2322-client-request-state: 5 passed, 0 failed
✓ http-custom-headers: 18 passed, 0 failed
✓ json-schema-ref-no-deref: 1 passed, 0 failed

Total: 371 passed, 0 failed, 0 warnings
$ node dist/index.js sdk python-sdk@main --mode server --suite all --spec-version 2026-07-28 \
    --expected-failures .../.github/actions/conformance/expected-failures.2026-07-28.yml
[sdk] Starting server: uv run --frozen mcp-everything-server --port 3000
[sdk] Server ready at http://localhost:3000/mcp
...
✓ input-required-result-tampered-state: 1 passed, 0 failed
✓ input-required-result-validate-input: 2 passed, 0 failed

Total: 114 passed, 0 failed
csharp-sdk (new entry) — client 311/321, server 90/110; failures are SDK-side findings, not entry misconfiguration
$ node dist/index.js sdk csharp-sdk@main --mode client --suite all --spec-version 2026-07-28 --timeout 60000
[sdk] Cloning https://github.com/modelcontextprotocol/csharp-sdk.git -> .sdk-under-test/csharp-sdk/main
[sdk] HEAD is 950ac7c
[sdk] Building: dotnet build tests/ModelContextProtocol.ConformanceClient -c Release -f net10.0 && dotnet build tests/ModelContextProtocol.ConformanceServer -c Release -f net10.0
[sdk] conformance client --command bash -c 'exec dotnet artifacts/bin/ModelContextProtocol.ConformanceClient/Release/net10.0/ModelContextProtocol.ConformanceClient.dll "$MCP_CONFORMANCE_SCENARIO" "$1"' conformance-client ...
...
✓ auth/token-endpoint-auth-none: 16 passed, 0 failed
✗ auth/authorization-server-migration: 13 passed, 1 failed
✓ auth/metadata-issuer-mismatch: 3 passed, 0 failed
✗ sep-2322-client-request-state: 0 passed, 5 failed
✓ http-custom-headers: 18 passed, 0 failed
✗ json-schema-ref-no-deref: 0 passed, 1 failed

Total: 311 passed, 10 failed, 0 warnings
$ node dist/index.js sdk csharp-sdk@main --mode server --suite all --spec-version 2026-07-28 \
    --server-url http://localhost:3000/stateless   # 2026-07-28 override from the entry comment
[sdk] Starting server: dotnet artifacts/bin/ModelContextProtocol.ConformanceServer/Release/net10.0/ModelContextProtocol.ConformanceServer.dll --urls http://localhost:3000
[sdk] Server ready at http://localhost:3000/stateless
...
✓ caching: 7 passed, 0 failed
✓ http-header-validation: 13 passed, 0 failed
✓ input-required-result-multi-round: 3 passed, 0 failed

Total: 90 passed, 20 failed

The 30 failing checks are genuine csharp-sdk conformance gaps, and the same scenarios pass through the identical plumbing on the other SDKs. The dominant cause is that csharp-sdk still implements the pre-modelcontextprotocol/modelcontextprotocol#3002 discover shape (DiscoverResult requires body serverInfo, so ConnectAsync throws on the alpha.10 mock — that alone accounts for tools_call, the five sep-2322-* checks, and json-schema-ref-no-deref on the client side); the rest are scope step-up / AS-migration on the client and SEP-2575 _meta strictness, JSON Schema 2020-12 keyword preservation, and DNS-rebinding host validation on the server.

typescript-sdk / typescript-sdk-v1 (entries unchanged) — re-validated as a control, incl. the @<ref> path
$ node dist/index.js sdk typescript-sdk@main --mode client --suite all --spec-version 2026-07-28 --timeout 60000
...
Total: 375 passed, 0 failed, 0 warnings
Baseline check passed: all failures are expected.

$ node dist/index.js sdk typescript-sdk@main --mode server --suite all --spec-version 2026-07-28
...
Total: 114 passed, 0 failed
Stale baseline entries (now passing - remove from baseline):
  ✓ server-stateless        # same #3002 story as go-sdk; cleanup belongs in typescript-sdk

The alias + ref-override path also works (this is the v1 line's 2026-07-28 integration branch, at its declared 2025-11-25 target):

$ node dist/index.js sdk typescript-sdk-v1@v1.x-2026-07-28 --mode client --suite all --timeout 60000
[sdk] conformance client --command npx tsx test/conformance/src/everythingClient.ts ... --spec-version 2025-11-25 --expected-failures .../test/conformance/conformance-baseline.yml
...
Total: 246 passed, 0 failed, 0 warnings

Full logs and per-check checks.json results are on the runner box if anyone wants the unabridged versions.

@pcarleton
pcarleton merged commit 2a705a8 into main Jul 27, 2026
8 checks passed
@pcarleton
pcarleton deleted the paulc/sdk-runner-tier1-coverage branch July 27, 2026 13:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant