Skip to content

feat(benchsdk): add optional display manifest to BenchmarkConfig and upload it on upsert - #335

Open
dtice25 wants to merge 2 commits into
masterfrom
devin/1787085805-display-config
Open

feat(benchsdk): add optional display manifest to BenchmarkConfig and upload it on upsert#335
dtice25 wants to merge 2 commits into
masterfrom
devin/1787085805-display-config

Conversation

@dtice25

@dtice25 dtice25 commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

This is the first end-to-end slice of making a *.bench.ts file the single source of truth for both benchmark execution and how the platform displays it.

  • BenchmarkConfig now accepts an optional display manifest (BenchmarkDisplayConfig) that declares:
    • metrics: key, label, unit, decimal places, and direction (lower-better | higher-better).
    • steps: human-readable labels and order for lifecycle steps.
    • overview: default ranked metric and default layout (ranking | cards | chart | leaderboard).
  • defineBenchmarkConfig validates the manifest (non-empty keys/labels, valid directions, no duplicate metric keys).
  • The new types are exported from @benchsdk/runner so authors get autocomplete.
  • runner.ts now uploads the manifest into the platform's existing benchmarks.config JSONB field via client.upsertBenchmark(..., { config: { display: ... } }).
  • benchmarks/browser/browser.bench.ts is updated with a concrete display manifest as a reference implementation.
export const config = defineBenchmarkConfig({
  // ... existing fields
  display: {
    metrics: [
      { key: 'totalMs', label: 'Total lifecycle', unit: 'ms', direction: 'lower-better', decimals: 0 },
      { key: 'createMs', label: 'Session creation', unit: 'ms', direction: 'lower-better', decimals: 0 },
    ],
    steps: [
      { key: 'create', label: 'Create session' },
      { key: 'connect', label: 'Connect CDP' },
    ],
    overview: { defaultMetric: 'totalMs', defaultLayout: 'ranking' },
  },
});

The platform counterpart (computesdk/benchmarks-platform#145) consumes this config.display to render labels, ranking direction, and default views without hard-coding the benchmark.

Link to Devin session: https://app.devin.ai/sessions/a45e3a08012c47e989ccfc4c055275ac
Requested by: @dtice25


Open in Devin Review

…hmark

- Add optional BenchmarkConfig.display spec with metrics, steps, and overview defaults.
- Validate display manifest in defineBenchmarkConfig.
- Upload display config into benchmarks.config JSONB on upsert.
- Export display types from @benchsdk/runner.
- Add display manifest to browser-lifecycle benchmark.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@dtice25 dtice25 self-assigned this Aug 18, 2026
@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@open-cla

open-cla Bot commented Aug 18, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 1 additional finding.

Open in Devin Review

…arks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant