Skip to content

refactor(benchsdk): split SDK into @benchsdk/api, @benchsdk/worker, and @benchsdk/client umbrella - #336

Open
dtice25 wants to merge 4 commits into
masterfrom
devin/benchsdk-v2-migration
Open

refactor(benchsdk): split SDK into @benchsdk/api, @benchsdk/worker, and @benchsdk/client umbrella#336
dtice25 wants to merge 4 commits into
masterfrom
devin/benchsdk-v2-migration

Conversation

@dtice25

@dtice25 dtice25 commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR begins the benchsdk v2 migration by splitting the SDK into focused, single-purpose packages:

  • @benchsdk/api — typed REST client (createBenchmarkClient) and shared platform data types. No worker runtime.
  • @benchsdk/worker — the runWorker engine, BenchmarkReporter, createSystemMetricsCollector, and participant helpers. Depends on @benchsdk/api.
  • @benchsdk/client — remains a backwards-compatible umbrella package. It re-exports from @benchsdk/api and @benchsdk/worker, and its createBenchmarkClient returns a client that still exposes runWorker(options) so existing consumers and tests compile unchanged.

What changed

// @benchsdk/api — thin REST + types only
export { createBenchmarkClient, BenchmarkApiError } from '@benchsdk/api';

// @benchsdk/worker — worker runtime only
export { runWorker, BenchmarkReporter, claimBenchmarkReporter,
        createSystemMetricsCollector, filterParticipantsByEnv, selectParticipants } from '@benchsdk/worker';

// @benchsdk/client — backwards-compatible umbrella
import { createBenchmarkClient } from '@benchsdk/client';
const client = createBenchmarkClient({ apiKey });
await client.runWorker({ ... }); // still works
  • Extracted packages/benchsdk/src/client.ts into packages/benchsdk-api/src/client.ts with the REST methods only.
  • Extracted the runWorker implementation into packages/benchsdk-worker/src/worker.ts as a standalone runWorker(client, options) function.
  • Copied reporter.ts, metrics.ts, and participants.ts into @benchsdk/worker.
  • packages/benchsdk/src/client.ts is now a thin wrapper that creates an @benchsdk/api client and attaches runWorker.
  • packages/benchsdk/src/reporter.ts, metrics.ts, and participants.ts are now re-export shims from @benchsdk/worker.
  • Added the two new packages to pnpm-workspace.yaml and their dependencies to pnpm-lock.yaml.
  • Updated packages/benchsdk/src/__tests__/public-api.contract.test.ts so VAL-SDK-090 parses the new .d.ts barrel shape, which now contains both local type declarations and re-exports from workspace packages.

Verification

  • pnpm -r --filter "./packages/**" build passes for @benchsdk/api, @benchsdk/worker, @benchsdk/client, and @benchsdk/runner.
  • pnpm typecheck passes.
  • pnpm --filter @benchsdk/client test passes (119 passed, 1 skipped).
  • pnpm --filter @benchsdk/runner test passes (86 passed).
  • pnpm -r --filter "./packages/benchsdk" --filter "./packages/benchsdk-api" --filter "./packages/benchsdk-worker" lint passes (runner lint is missing eslint in its dev deps, pre-existing).

What comes next

  • Migrate @benchsdk/runner to import from @benchsdk/api and @benchsdk/worker directly instead of @benchsdk/client.
  • Move the display/scoring manifest types into @benchsdk/api so both benchmarks and benchmarks-platform share one source of truth.
  • Add typed metric sinks (metrics.timing(...) / metrics.count(...)) and platform-side scoring.

This is a foundation-only PR; no benchmark schema or runner behavior is changed yet.

Link to Devin session: https://app.devin.ai/sessions/e5c1507360b24f6f89327ccc975ae961
Requested by: @dtice25


Open in Devin Review

…nd @benchsdk/client umbrella

- Add @benchsdk/api: typed REST client and shared platform types.
- Add @benchsdk/worker: runWorker runtime, BenchmarkReporter, metrics, participants.
- Keep @benchsdk/client as a backwards-compatible umbrella that re-exports from the focused packages and exposes client.runWorker().
- Update pnpm workspace and lockfile for the two new packages.
- Adjust public-api.contract.test.ts to parse the new barrel .d.ts shape with workspace re-exports.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@dtice25 dtice25 self-assigned this Aug 19, 2026
@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@open-cla

open-cla Bot commented Aug 19, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

…orker artifacts are available

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
devin-ai-integration[bot]

This comment was marked as resolved.

…t.client matches type, and build all packages in release workflow

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
devin-ai-integration[bot]

This comment was marked as resolved.

… scale image

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant