diff --git a/.github/workflows/verify.yml b/.github/workflows/verify.yml new file mode 100644 index 00000000..b3109896 --- /dev/null +++ b/.github/workflows/verify.yml @@ -0,0 +1,40 @@ +# Cross-adapter capture-parity gate (Layer A). Runs the committed golden +# fixtures through the reader and snapshots them โ€” fast, no browser. Fixture +# regeneration (`pnpm fixtures:regen`) needs a real browser and is a deliberate +# maintainer/nightly step, not part of this gate. +name: Verify + +on: + push: + branches: + - main + pull_request: + +jobs: + verify: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + - name: Use Node.js + uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # v6.2.0 + with: + node-version-file: .nvmrc + - name: Cache pnpm modules + uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4 + env: + cache-name: cache-pnpm-modules + with: + path: ~/.pnpm-store + key: ${{ runner.os }}-verify-${{ env.cache-name }}-${{ hashFiles('**/pnpm-lock.yaml') }} + restore-keys: | + ${{ runner.os }}-verify-${{ env.cache-name }}- + - uses: pnpm/action-setup@41ff72655975bd51cab0327fa583b6e92b6d3061 # v4.2.0 + with: + version: 10.26.1 + run_install: false + - name: Install dependencies + run: pnpm install --frozen-lockfile + - name: ๐Ÿ” Lint harness + run: pnpm exec eslint test + - name: ๐Ÿงช Cross-adapter capture parity + run: pnpm verify diff --git a/CLAUDE.md b/CLAUDE.md index 71bd7d1f..89268c0f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -248,6 +248,7 @@ Documented divergences from the conventions above. They exist today as debt to b - Nightwatch native asserts are buffered at call time and emitted in one batch at test-end (Nightwatch exposes no per-assertion execution/outcome hook reachable from the plugin โ€” `client.queue.tree` / `client.reporter` aren't on `browser`). Streaming mid-run would flash every assert green before its outcome is known, so rows are held and reconciled against `results.testcases[*].assertions` at `afterEach` (the flat `results.assertions` only reflects the last testcase). The exporter re-sorts commands by timestamp (`buildActionEvents`) so the batched rows land at their real timeline positions rather than clustering at flush time. `actual` is parsed from the failure message (`but got: โ€ฆ`, failures only) into a collapsed `{passed, expected, actual}` result; `expected` is the assertion arg โ€” the label still mirrors the call args, not the derived values. - Nightwatch's BDD `describe/it` interface fires the plugin's global `beforeEach`/`afterEach` **once per module** (with an empty `currentTest.name`), not per `it` โ€” Nightwatch runs the individual `it`s internally with no per-testcase hook or event reachable from the plugin (the lib emits only transport-level events). So `traceGranularity:'test'` records a single slice boundary (the first test) and collapses to one session-scoped slice; retry-aware/per-test retention (`tracePolicy`) degrades to session scope for this interface. Per-test slicing works for adapters/styles that expose per-test hooks (WDIO mocha/cucumber, Selenium mocha; likely Nightwatch's exports-object style, unverified). Session-granularity trace is unaffected. **No per-test support has been built for this interface** โ€” real per-`it` slicing needs a hook Nightwatch doesn't surface; deferred to a full-picture pass, not attempted yet. - **Empirically confirmed** (BDD example, `mode:'trace'` + `traceGranularity:'test'` + `tracePolicy:'retain-on-failure'` + `screenshot:'on'` + `video:'retain-on-failure'`): the one collapsed slice is keyed to the **first** test's uid (the artifacts manifest's `tests[]` still carries every testcase with correct `state`, so metadata capture is fine โ€” it's the *slice/artifact* keying that collapses). Consequence: with a passing first test and a failing later test, the screenshot **produces** (policy `'on'` captures regardless of outcome, keyed to the first test) but the **trace zip and video are dropped** โ€” `retain-on-failure` evaluates the first (passing) test's outcome, and the actually-failing test never gets its own slice to retain. The per-test produce-only path itself (screenshot/video write + manifest) is correct; only the BDD slice-keying limits it. +- **Nightwatch cucumber: no `trace.zip` is written in trace mode** (surfaced by the `test/` verification harness, 2026-07-27). Root cause is hook ordering, not granularity: the plugin's cucumber `After` hook runs at `order: -1` (deliberately *after* Nightwatch's browser-quit hook, so it can read final assertion status โ€” see `helpers/cucumberHooks.cts`). By the time `finalizeCucumberScenario` runs its capture/`flushTestSlice`, the per-scenario browser session is already torn down, so `#flushSpecTrace`/`#writeTraceIfNeeded` bail on the absent `sessionCapturer.metadata?.sessionId` and nothing is written โ€” under **either** `traceGranularity`. (`'session'` also has no single live session at run-end because cucumber uses one session per scenario.) Per-scenario metadata capture and **live mode are unaffected**; only the trace-mode zip export is missing. A proper fix must capture/write each scenario's trace **before the browser quits** (e.g. from the last `AfterStep` or a pre-quit hook), which needs a rework + browser-iterated verification. A runnable example exists at `examples/nightwatch/cucumber/` (live-capable); the harness keeps the `nightwatch-cucumber` entry `planned` until this lands. - Service renders expect-webdriverio matchers as single `expect.` rows by **folding**, not stack/depth suppression (the old `#assertionDepth`/`#matcherStarted`/self-heal machinery is gone). The matcher's value-read (`toHaveText`โ†’`getText`, `toExist`โ†’`isExisting`, โ€ฆ) is captured as a normal command; `afterAssertion` then coalesces the synthesized `expect.*` row into that read in place โ€” inheriting its callSource, screenshot, and timeline position โ€” and the fold replaces **by timestamp, never a public `id`**: `id` is the per-worker `commandCounter`, which resets per spec, so stamping one lets the app's id-first `replaceCommand` swap a same-id row from another spec (duplicate rows + a fold from another spec vanishing, in multi-spec live mode). `beforeAssertion` arms the pending matcher (depth-counted so aliases like `toBeChecked`โ†’`toBeSelected` fold once); a matcher that **hard-throws** โ€” element never resolves, so expect-webdriverio's `waitUntil` rethrows and `afterAssertion` never fires โ€” is synthesized at `afterTest`/`afterStep` from the throwing read, so a failing assertion renders as `expect.` whether or not the element existed. Two limits: its error is then the read's (`Can't call getText on โ€ฆ element wasn't found`), not an assertion-phrased message; and `MATCHER_READ_COMMANDS` is a hand-maintained allowlist, so a matcher whose read isn't listed leaves its raw read visible alongside the `expect.*` row. Plain-value jest matchers (`expect(x).toBe(y)`) don't fire the ewdio hooks, so they aren't captured as rows. ### File-size (raw line counts; soft cap is 500 logic lines) diff --git a/examples/nightwatch/cucumber/features/login.feature b/examples/nightwatch/cucumber/features/login.feature new file mode 100644 index 00000000..05fdd88d --- /dev/null +++ b/examples/nightwatch/cucumber/features/login.feature @@ -0,0 +1,15 @@ +Feature: Example site smoke test + + Mirrors the Nightwatch BDD smoke-test: open a site, wait for the body to be + visible, and assert the page title. Runs as a Cucumber scenario so the + cross-adapter harness has a nightwatch-cucumber trace fixture. + + Scenario Outline: I can open a site and read its title + Given I navigate to "" + When the page body becomes visible + Then the page title contains "" + + Examples: + | url | title | + | https://example.com | Example | + | https://example.org | Example | diff --git a/examples/nightwatch/cucumber/features/step_definitions/login.steps.js b/examples/nightwatch/cucumber/features/step_definitions/login.steps.js new file mode 100644 index 00000000..00e11202 --- /dev/null +++ b/examples/nightwatch/cucumber/features/step_definitions/login.steps.js @@ -0,0 +1,18 @@ +// Step definitions for the example-site smoke scenario. +// +// Nightwatch's Cucumber integration injects the Nightwatch `browser` onto the +// World, so steps reach it via `this.browser` โ€” hence regular (non-arrow) +// functions so `this` binds to the World. +const { Given, When, Then } = require('@cucumber/cucumber') + +Given(/^I navigate to "([^"]*)"$/, async function (url) { + await this.browser.url(url) +}) + +When(/^the page body becomes visible$/, async function () { + await this.browser.waitForElementVisible('body', 5000) +}) + +Then(/^the page title contains "([^"]*)"$/, async function (title) { + await this.browser.assert.titleContains(title) +}) diff --git a/examples/nightwatch/cucumber/nightwatch.cucumber.conf.cjs b/examples/nightwatch/cucumber/nightwatch.cucumber.conf.cjs new file mode 100644 index 00000000..52318614 --- /dev/null +++ b/examples/nightwatch/cucumber/nightwatch.cucumber.conf.cjs @@ -0,0 +1,69 @@ +// Simple import - just require the package +const path = require('node:path') +const nightwatchDevtools = require('@wdio/nightwatch-devtools').default +const { cucumberHooksPath } = require('@wdio/nightwatch-devtools') + +const featuresDir = path.resolve(__dirname, 'features') + +module.exports = { + // Resolve relative to this config file so the paths hold regardless of CWD. + src_folders: [path.resolve(featuresDir, 'step_definitions')], + output_folder: false, // Skip generating nightwatch reports for this example + custom_commands_path: [], + custom_assertions_path: [], + + test_runner: { + type: 'cucumber', + options: { + feature_path: featuresDir, + // Registers the DevTools Before/After/Step scenario hooks so the plugin + // sees each scenario as a test unit. + require: [cucumberHooksPath] + } + }, + + webdriver: { + start_process: true, + // server_path: '/opt/homebrew/bin/chromedriver', + port: 9515 + }, + + test_settings: { + default: { + // Ensure all tests run even if one fails + skip_testcases_on_fail: false, + + desiredCapabilities: { + browserName: 'chrome', + // Required for chromedriver to expose the BiDi WebSocket channel. + // Without this, attachBidiHandlers silently fails and the perf-log + // fallback takes over. + webSocketUrl: true, + 'goog:chromeOptions': { + args: [ + '--headless', + '--no-sandbox', + '--disable-dev-shm-usage', + '--window-size=1600,900' + ] + }, + 'goog:loggingPrefs': { performance: 'ALL' } + }, + // bidi: opt-in WebDriver BiDi capture for console + network. When + // attached, the per-command Chrome perf-log network path is gated off to + // avoid duplicate entries. + globals: nightwatchDevtools({ + port: 3000, + // Cucumber exposes per-scenario hooks, so each scenario is captured as + // its own test unit. Trace when the harness sets DEVTOOLS_MODE=trace, + // otherwise stream live to the backend/UI. NOTE: trace mode does not yet + // emit a zip here โ€” the cucumber After hook runs after the browser + // session is torn down, so there's no live session to capture/write + // from. See CLAUDE.md known debt. Live mode works. + mode: process.env.DEVTOOLS_MODE === 'trace' ? 'trace' : 'live', + traceGranularity: 'session', + bidi: true + }) + } + } +} diff --git a/examples/selenium/cucumber-test/features/support/setup.js b/examples/selenium/cucumber-test/features/support/setup.js index 18f1e762..907dfb90 100644 --- a/examples/selenium/cucumber-test/features/support/setup.js +++ b/examples/selenium/cucumber-test/features/support/setup.js @@ -6,7 +6,10 @@ import { DevTools } from '@wdio/selenium-devtools' +// mode defaults to 'live' (the demo default); the verification harness sets +// DEVTOOLS_MODE=trace to produce a golden fixture from this same example. DevTools.configure({ + mode: process.env.DEVTOOLS_MODE === 'trace' ? 'trace' : 'live', screencast: { enabled: true, quality: 70, maxWidth: 1280, maxHeight: 720 }, headless: true }) diff --git a/examples/wdio/mocha/wdio.trace.conf.ts b/examples/wdio/mocha/wdio.trace.conf.ts new file mode 100644 index 00000000..3c39e5ea --- /dev/null +++ b/examples/wdio/mocha/wdio.trace.conf.ts @@ -0,0 +1,53 @@ +import type { Options } from '@wdio/types' + +// Trace-mode variant of wdio.conf.ts, used by the verification harness +// (`pnpm fixtures:regen`) to produce a deterministic golden trace.zip. Same +// capabilities and specs as the live config; session granularity writes one zip +// per run. Kept separate so the demo config's live default stays untouched. +export const config: Options.Testrunner = { + runner: 'local', + autoCompileOpts: { + autoCompile: true, + tsNodeOpts: { + project: './tsconfig.json', + transpileOnly: true + } + }, + specs: ['./specs/**/*.e2e.ts'], + exclude: [], + maxInstances: 1, + capabilities: [ + { + browserName: 'chrome', + 'goog:chromeOptions': { + args: [ + '--headless', + '--disable-gpu', + '--remote-allow-origins=*', + '--window-size=1600,900' + ] + } + } + ], + logLevel: 'warn', + bail: 0, + baseUrl: 'http://localhost', + waitforTimeout: 10000, + connectionRetryTimeout: 120000, + connectionRetryCount: 3, + services: [ + [ + 'devtools', + { + mode: 'trace' as const, + traceGranularity: 'session' as const + } + ] + ], + framework: 'mocha', + reporters: ['spec'], + mochaOpts: { + ui: 'bdd', + timeout: 60000 + } +} diff --git a/package.json b/package.json index a2d08b44..08e39f88 100644 --- a/package.json +++ b/package.json @@ -19,6 +19,11 @@ "test:coverage": "vitest run --coverage", "test:ui": "vitest --ui", "test:debug": "node --inspect-brk ./node_modules/.bin/vitest run", + "verify": "vitest run --config test/vitest.config.ts", + "verify:watch": "vitest --config test/vitest.config.ts", + "verify:update": "vitest run --config test/vitest.config.ts -u", + "verify:ui": "wdio run test/visual/wdio.conf.ts", + "fixtures:regen": "tsx test/regen.ts", "lint": "pnpm --parallel lint", "watch": "pnpm build --watch", "changeset:gen": "bash .changeset/changeset-gen.sh" @@ -43,6 +48,7 @@ "@vitest/browser": "^4.1.8", "@vitest/coverage-v8": "^4.1.8", "@wdio/allure-reporter": "^9.29.1", + "@wdio/visual-service": "^10.1.0", "autoprefixer": "^10.5.0", "eslint": "^10.4.1", "eslint-config-prettier": "^10.1.8", diff --git a/packages/nightwatch-devtools/package.json b/packages/nightwatch-devtools/package.json index 1830cb38..f4ba6a8c 100644 --- a/packages/nightwatch-devtools/package.json +++ b/packages/nightwatch-devtools/package.json @@ -35,6 +35,7 @@ "lint": "eslint .", "example": "nightwatch -c ../../examples/nightwatch/nightwatch.conf.cjs", "example:retry": "nightwatch -c ../../examples/nightwatch/nightwatch.conf.cjs --retries 1", + "example:cucumber": "nightwatch -c ../../examples/nightwatch/cucumber/nightwatch.cucumber.conf.cjs", "prepublishOnly": "pnpm build" }, "keywords": [ @@ -59,6 +60,7 @@ "yazl": "^2.5.1" }, "devDependencies": { + "@cucumber/cucumber": "^13.0.0", "@types/node": "25.9.3", "@types/ws": "^8.18.1", "@wdio/devtools-core": "workspace:^", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index bff88351..c9f908e1 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -15,7 +15,7 @@ importers: dependencies: '@wdio/cli': specifier: 9.28.0 - version: 9.28.0(@types/node@25.9.3)(expect-webdriverio@5.6.8)(puppeteer-core@21.11.0) + version: 9.28.0(@types/node@25.9.3)(expect-webdriverio@5.7.0)(puppeteer-core@21.11.0) devDependencies: '@changesets/cli': specifier: ^2.31.0 @@ -44,6 +44,9 @@ importers: '@wdio/allure-reporter': specifier: ^9.29.1 version: 9.29.1 + '@wdio/visual-service': + specifier: ^10.1.0 + version: 10.1.0(webdriverio@9.28.0(puppeteer-core@21.11.0)) autoprefixer: specifier: ^10.5.0 version: 10.5.0(postcss@8.5.15) @@ -264,7 +267,7 @@ importers: version: 11.2.0 '@wdio/cli': specifier: 9.28.0 - version: 9.28.0(@types/node@25.9.3)(expect-webdriverio@5.6.8)(puppeteer-core@21.11.0) + version: 9.28.0(@types/node@25.9.3)(expect-webdriverio@5.7.0)(puppeteer-core@21.11.0) '@wdio/devtools-app': specifier: workspace:^ version: link:../app @@ -362,7 +365,7 @@ importers: version: link:../core '@wdio/globals': specifier: 9.28.0 - version: 9.28.0(expect-webdriverio@5.6.8)(webdriverio@9.28.0(puppeteer-core@21.11.0)) + version: 9.28.0(expect-webdriverio@5.7.0)(webdriverio@9.28.0(puppeteer-core@21.11.0)) tsup: specifier: ^8.5.1 version: 8.5.1(@microsoft/api-extractor@7.58.9(@types/node@25.9.3))(jiti@2.7.0)(postcss@8.5.15)(tsx@4.22.4)(typescript@6.0.3)(yaml@2.9.0) @@ -371,7 +374,7 @@ importers: version: 6.0.3 vitest: specifier: ^4.0.16 - version: 4.1.9(@types/node@25.9.3)(@vitest/coverage-v8@4.1.9)(happy-dom@20.10.4)(jsdom@24.1.3)(vite@8.0.16(@types/node@25.9.3)(esbuild@0.28.0)(jiti@2.7.0)(tsx@4.22.4)(yaml@2.9.0)) + version: 4.1.9(@types/node@25.9.3)(@vitest/coverage-v8@4.1.9)(happy-dom@20.10.4)(jsdom@24.1.3)(vite@8.0.16(@types/node@25.9.3)(esbuild@0.27.7)(jiti@2.7.0)(tsx@4.22.4)(yaml@2.9.0)) packages/nightwatch-devtools: dependencies: @@ -412,6 +415,9 @@ importers: specifier: ^2.5.1 version: 2.5.1 devDependencies: + '@cucumber/cucumber': + specifier: ^13.0.0 + version: 13.0.0 '@types/node': specifier: 25.9.3 version: 25.9.3 @@ -622,7 +628,7 @@ importers: version: link:../shared '@wdio/globals': specifier: 9.28.0 - version: 9.28.0(expect-webdriverio@5.6.8)(webdriverio@9.28.0(puppeteer-core@21.11.0)) + version: 9.28.0(expect-webdriverio@5.7.0)(webdriverio@9.28.0(puppeteer-core@21.11.0)) '@wdio/protocols': specifier: 9.28.0 version: 9.28.0 @@ -2755,6 +2761,16 @@ packages: expect-webdriverio: ^5.6.5 webdriverio: ^9.0.0 + '@wdio/globals@9.29.1': + resolution: {integrity: sha512-F96BKppx4HGD64v+s57TM4K4zaxqUCg2RXHk6sjB2xrSa7P+d2VYV28ID2ddF7iSodVfPlpa1i2/jy8jMGrU8w==} + engines: {node: '>=18.20.0'} + peerDependencies: + expect-webdriverio: ^5.6.5 + webdriverio: ^9.0.0 + + '@wdio/image-comparison-core@2.1.0': + resolution: {integrity: sha512-o9w80bkMmgbUuPKoJYbuf2sKh/xyrfVyITftmVJuIaqIPehwriwGF7ZU7sS9JiXAdoqM17Wdy7MdpO3mOgsjZA==} + '@wdio/local-runner@9.28.0': resolution: {integrity: sha512-cuITB31LRvEuOfhVNtkqBC0LQsQQTp7jumV8l7YJgchdmNI2gwyRaeKR35HxPpR8Awi5QeBfSkFxipwP8JQIWg==} engines: {node: '>=18.20.0'} @@ -2824,6 +2840,9 @@ packages: resolution: {integrity: sha512-VDqUaXpR8oOZSs26dy06Y2LhmA8bldsXDHeZ36n8SfW+Bq0miG0RRxou7aqx7sifVbbsuxrbBPXvmK+40uAIbQ==} engines: {node: '>=18.20.0'} + '@wdio/visual-service@10.1.0': + resolution: {integrity: sha512-GsWItEpETBRy5Ymuh8HOEX1pMDME2hxSgUBW4UlSHCmKTSOcUEfQosjAx3XVMucpR5SosPIiOB1/4teEi4qLhw==} + '@wdio/xvfb@9.28.0': resolution: {integrity: sha512-9NUB6rYS7uOzE+attbvX7frfvXdf/FhinHBL2hJtBW81/lcSm3TPqgFskKVKiUooXfdTf4bvrRYn1+MVVGFFXA==} engines: {node: '>=18'} @@ -4161,6 +4180,14 @@ packages: '@wdio/logger': ^9.0.0 webdriverio: ^9.0.0 + expect-webdriverio@5.7.0: + resolution: {integrity: sha512-jLOTrJoPBC3Wtd83ryHMbRcEajMGtAnn6OWtSnoJoDLt16nHvxVdjcI4Bc0n+1KAOr8DvQtlJ55f0M48kJtEpw==} + engines: {node: '>=20'} + peerDependencies: + '@wdio/globals': ^9.0.0 + '@wdio/logger': ^9.0.0 + webdriverio: ^9.0.0 + expect@30.4.1: resolution: {integrity: sha512-PMARsyh/JtqC20HoGqlFcIlQAyqUtW4PlI1rup1uhYJtKuwAjbvWi3GQMAn+STdHum/dk8xrKfUM1+5SAwpolA==} engines: {node: ^18.14.0 || ^20.0.0 || ^22.0.0 || >=24.0.0} @@ -4204,6 +4231,9 @@ packages: fast-levenshtein@2.0.6: resolution: {integrity: sha512-DCXu6Ifhqcks7TZKY3Hxp3y6qphY5SJZmrWMDrKcERSOXWQdMhU9Ig/PYrzyw/ul9jOIyh0N4M0tbC5hodg8dw==} + fast-png@8.0.0: + resolution: {integrity: sha512-gCysNasJ8KEMgfdYIKd/wTDo6ENK1PWT0RJO7O+0pgmuHPw2O6tA1WvdxFRJoLf9V8yFYpG0FA1YgI8X97OhJA==} + fast-querystring@1.1.2: resolution: {integrity: sha512-g6KuKWmFXc0fID8WWH0jit4g0AGBoJhCkJMb1RmbsSEUNvQ+ZC8D6CUZ+GtF8nMzSPXnhiePyyqqipzNNEnHjg==} @@ -4749,6 +4779,9 @@ packages: resolution: {integrity: sha512-4gd7VpWNQNB4UKKCFFVcp1AVv+FMOgs9NKzjHKusc8jTMhd5eL1NqQqOpE0KzMds804/yHlglp3uxgluOqAPLw==} engines: {node: '>= 0.4'} + iobuffer@6.0.1: + resolution: {integrity: sha512-SZWYkWNfjIXIBYSDpXDYIgshqtbOPsi4lviawAEceR1Kqk+sHDlcQjWrzNQsii80AyBY0q5c8HCTNjqo74ul+Q==} + ip-address@10.2.0: resolution: {integrity: sha512-/+S6j4E9AHvW9SWMSEY9Xfy66O5PWvVEJ08O0y5JGyEKQpojb0K0GKpz/v5HJ/G0vi3D2sjGK78119oXZeE0qA==} engines: {node: '>= 12'} @@ -6016,6 +6049,10 @@ packages: piscina@4.9.2: resolution: {integrity: sha512-Fq0FERJWFEUpB4eSY59wSNwXD4RYqR+nR/WiEVcZW8IWfVBxJJafcgTEZDQo8k3w0sUarJ8RyVbbUF4GQ2LGbQ==} + pixelmatch@7.2.0: + resolution: {integrity: sha512-xhcb4yHu9sM/G7foGzoLtXYcC0zHEaOXXjRKhGup0fw78Nf2Tkiapv4EQyMzrbcmQPsllAI7DbFY2UT7PlI9Pg==} + hasBin: true + pkg-dir@4.2.0: resolution: {integrity: sha512-HRDzbaKjC+AOWVXxAU/x54COGeIv9eb+6CkDSQoNTt4XyWoIJvuPsXizxu/Fr23EiekbtZwmh1IcIG/l/a10GQ==} engines: {node: '>=8'} @@ -7669,7 +7706,7 @@ snapshots: '@babel/types': 7.29.7 '@jridgewell/remapping': 2.3.5 convert-source-map: 2.0.0 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) gensync: 1.0.0-beta.2 json5: 2.2.3 semver: 6.3.1 @@ -7828,7 +7865,7 @@ snapshots: '@babel/parser': 7.29.7 '@babel/template': 7.29.7 '@babel/types': 7.29.7 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) transitivePeerDependencies: - supports-color @@ -8509,7 +8546,7 @@ snapshots: '@eslint/config-array@0.23.5': dependencies: '@eslint/object-schema': 3.0.5 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) minimatch: 10.2.5 transitivePeerDependencies: - supports-color @@ -9193,7 +9230,7 @@ snapshots: '@puppeteer/browsers@2.13.2': dependencies: - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) extract-zip: 2.0.1 progress: 2.0.3 proxy-agent: 6.5.0 @@ -9611,7 +9648,7 @@ snapshots: '@typescript-eslint/types': 8.61.1 '@typescript-eslint/typescript-estree': 8.61.1(typescript@6.0.3) '@typescript-eslint/visitor-keys': 8.61.1 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) eslint: 10.5.0(jiti@2.7.0) typescript: 6.0.3 transitivePeerDependencies: @@ -9621,7 +9658,7 @@ snapshots: dependencies: '@typescript-eslint/tsconfig-utils': 8.61.1(typescript@6.0.3) '@typescript-eslint/types': 8.61.1 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) typescript: 6.0.3 transitivePeerDependencies: - supports-color @@ -9640,7 +9677,7 @@ snapshots: '@typescript-eslint/types': 8.61.1 '@typescript-eslint/typescript-estree': 8.61.1(typescript@6.0.3) '@typescript-eslint/utils': 8.61.1(eslint@10.5.0(jiti@2.7.0))(typescript@6.0.3) - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) eslint: 10.5.0(jiti@2.7.0) ts-api-utils: 2.5.0(typescript@6.0.3) typescript: 6.0.3 @@ -9655,7 +9692,7 @@ snapshots: '@typescript-eslint/tsconfig-utils': 8.61.1(typescript@6.0.3) '@typescript-eslint/types': 8.61.1 '@typescript-eslint/visitor-keys': 8.61.1 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) minimatch: 10.2.5 semver: 7.8.4 tinyglobby: 0.2.17 @@ -9794,6 +9831,14 @@ snapshots: chai: 6.2.2 tinyrainbow: 3.1.0 + '@vitest/mocker@4.1.9(vite@8.0.16(@types/node@25.9.3)(esbuild@0.27.7)(jiti@2.7.0)(tsx@4.22.4)(yaml@2.9.0))': + dependencies: + '@vitest/spy': 4.1.9 + estree-walker: 3.0.3 + magic-string: 0.30.21 + optionalDependencies: + vite: 8.0.16(@types/node@25.9.3)(esbuild@0.27.7)(jiti@2.7.0)(tsx@4.22.4)(yaml@2.9.0) + '@vitest/mocker@4.1.9(vite@8.0.16(@types/node@25.9.3)(esbuild@0.28.0)(jiti@2.7.0)(tsx@4.22.4)(yaml@2.9.0))': dependencies: '@vitest/spy': 4.1.9 @@ -9893,6 +9938,39 @@ snapshots: - supports-color - utf-8-validate + '@wdio/cli@9.28.0(@types/node@25.9.3)(expect-webdriverio@5.7.0)(puppeteer-core@21.11.0)': + dependencies: + '@vitest/snapshot': 2.1.9 + '@wdio/config': 9.28.0 + '@wdio/globals': 9.28.0(expect-webdriverio@5.7.0)(webdriverio@9.28.0(puppeteer-core@21.11.0)) + '@wdio/logger': 9.18.0 + '@wdio/protocols': 9.28.0 + '@wdio/types': 9.28.0 + '@wdio/utils': 9.28.0 + async-exit-hook: 2.0.1 + chalk: 5.6.2 + chokidar: 4.0.3 + create-wdio: 9.28.0(@types/node@25.9.3) + dotenv: 17.4.2 + import-meta-resolve: 4.2.0 + lodash.flattendeep: 4.4.0 + lodash.pickby: 4.6.0 + lodash.union: 4.6.0 + read-pkg-up: 10.1.0 + tsx: 4.22.4 + webdriverio: 9.28.0(puppeteer-core@21.11.0) + yargs: 17.7.2 + transitivePeerDependencies: + - '@types/node' + - bare-abort-controller + - bare-buffer + - bufferutil + - expect-webdriverio + - puppeteer-core + - react-native-b4a + - supports-color + - utf-8-validate + '@wdio/config@8.41.0': dependencies: '@wdio/logger': 8.38.0 @@ -9952,6 +10030,23 @@ snapshots: expect-webdriverio: 5.6.8(@wdio/globals@9.28.0)(@wdio/logger@9.29.1)(webdriverio@9.28.0(puppeteer-core@21.11.0)) webdriverio: 9.28.0(puppeteer-core@21.11.0) + '@wdio/globals@9.28.0(expect-webdriverio@5.7.0)(webdriverio@9.28.0(puppeteer-core@21.11.0))': + dependencies: + expect-webdriverio: 5.7.0(@wdio/globals@9.28.0)(@wdio/logger@9.29.1)(webdriverio@9.28.0(puppeteer-core@21.11.0)) + webdriverio: 9.28.0(puppeteer-core@21.11.0) + + '@wdio/globals@9.29.1(expect-webdriverio@5.7.0)(webdriverio@9.28.0(puppeteer-core@21.11.0))': + dependencies: + expect-webdriverio: 5.7.0(@wdio/globals@9.29.1)(@wdio/logger@9.18.0)(webdriverio@9.28.0(puppeteer-core@21.11.0)) + webdriverio: 9.28.0(puppeteer-core@21.11.0) + + '@wdio/image-comparison-core@2.1.0': + dependencies: + '@wdio/logger': 9.29.1 + '@wdio/types': 9.29.1 + fast-png: 8.0.0 + pixelmatch: 7.2.0 + '@wdio/local-runner@9.28.0(@wdio/globals@9.28.0)(webdriverio@9.28.0(puppeteer-core@21.11.0))': dependencies: '@types/node': 25.9.3 @@ -10119,6 +10214,16 @@ snapshots: - react-native-b4a - supports-color + '@wdio/visual-service@10.1.0(webdriverio@9.28.0(puppeteer-core@21.11.0))': + dependencies: + '@wdio/globals': 9.29.1(expect-webdriverio@5.7.0)(webdriverio@9.28.0(puppeteer-core@21.11.0)) + '@wdio/image-comparison-core': 2.1.0 + '@wdio/logger': 9.29.1 + '@wdio/types': 9.29.1 + expect-webdriverio: 5.7.0(@wdio/globals@9.29.1)(@wdio/logger@9.29.1)(webdriverio@9.28.0(puppeteer-core@21.11.0)) + transitivePeerDependencies: + - webdriverio + '@wdio/xvfb@9.28.0': dependencies: '@wdio/logger': 9.18.0 @@ -10149,7 +10254,7 @@ snapshots: agent-base@6.0.2: dependencies: - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) transitivePeerDependencies: - supports-color @@ -11252,7 +11357,7 @@ snapshots: edgedriver@6.3.0: dependencies: - '@wdio/logger': 9.29.1 + '@wdio/logger': 9.18.0 '@zip.js/zip.js': 2.8.26 decamelize: 6.0.1 edge-paths: 3.0.5 @@ -11597,7 +11702,7 @@ snapshots: '@types/estree': 1.0.9 ajv: 6.15.0 cross-spawn: 7.0.6 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) escape-string-regexp: 4.0.0 eslint-scope: 9.1.2 eslint-visitor-keys: 5.0.1 @@ -11709,6 +11814,36 @@ snapshots: jest-matcher-utils: 30.4.1 webdriverio: 9.28.0(puppeteer-core@21.11.0) + expect-webdriverio@5.7.0(@wdio/globals@9.28.0)(@wdio/logger@9.29.1)(webdriverio@9.28.0(puppeteer-core@21.11.0)): + dependencies: + '@vitest/snapshot': 4.1.9 + '@wdio/globals': 9.28.0(expect-webdriverio@5.7.0)(webdriverio@9.28.0(puppeteer-core@21.11.0)) + '@wdio/logger': 9.29.1 + deep-eql: 5.0.2 + expect: 30.4.1 + jest-matcher-utils: 30.4.1 + webdriverio: 9.28.0(puppeteer-core@21.11.0) + + expect-webdriverio@5.7.0(@wdio/globals@9.29.1)(@wdio/logger@9.18.0)(webdriverio@9.28.0(puppeteer-core@21.11.0)): + dependencies: + '@vitest/snapshot': 4.1.9 + '@wdio/globals': 9.29.1(expect-webdriverio@5.7.0)(webdriverio@9.28.0(puppeteer-core@21.11.0)) + '@wdio/logger': 9.18.0 + deep-eql: 5.0.2 + expect: 30.4.1 + jest-matcher-utils: 30.4.1 + webdriverio: 9.28.0(puppeteer-core@21.11.0) + + expect-webdriverio@5.7.0(@wdio/globals@9.29.1)(@wdio/logger@9.29.1)(webdriverio@9.28.0(puppeteer-core@21.11.0)): + dependencies: + '@vitest/snapshot': 4.1.9 + '@wdio/globals': 9.29.1(expect-webdriverio@5.7.0)(webdriverio@9.28.0(puppeteer-core@21.11.0)) + '@wdio/logger': 9.29.1 + deep-eql: 5.0.2 + expect: 30.4.1 + jest-matcher-utils: 30.4.1 + webdriverio: 9.28.0(puppeteer-core@21.11.0) + expect@30.4.1: dependencies: '@jest/expect-utils': 30.4.1 @@ -11724,7 +11859,7 @@ snapshots: extract-zip@2.0.1: dependencies: - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) get-stream: 5.2.0 yauzl: 2.10.0 optionalDependencies: @@ -11763,6 +11898,11 @@ snapshots: fast-levenshtein@2.0.6: {} + fast-png@8.0.0: + dependencies: + fflate: 0.8.3 + iobuffer: 6.0.1 + fast-querystring@1.1.2: dependencies: fast-decode-uri-component: 1.0.1 @@ -11993,7 +12133,7 @@ snapshots: geckodriver@6.1.0: dependencies: - '@wdio/logger': 9.29.1 + '@wdio/logger': 9.18.0 '@zip.js/zip.js': 2.8.26 decamelize: 6.0.1 http-proxy-agent: 7.0.2 @@ -12055,7 +12195,7 @@ snapshots: dependencies: basic-ftp: 5.3.1 data-uri-to-buffer: 6.0.2 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) transitivePeerDependencies: - supports-color @@ -12063,7 +12203,7 @@ snapshots: dependencies: basic-ftp: 5.3.1 data-uri-to-buffer: 8.0.0 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) transitivePeerDependencies: - supports-color @@ -12261,35 +12401,35 @@ snapshots: http-proxy-agent@7.0.2: dependencies: agent-base: 7.1.4 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) transitivePeerDependencies: - supports-color http-proxy-agent@9.0.0: dependencies: agent-base: 9.0.0 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) transitivePeerDependencies: - supports-color https-proxy-agent@5.0.1: dependencies: agent-base: 6.0.2 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) transitivePeerDependencies: - supports-color https-proxy-agent@7.0.6: dependencies: agent-base: 7.1.4 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) transitivePeerDependencies: - supports-color https-proxy-agent@9.0.0: dependencies: agent-base: 9.0.0 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) transitivePeerDependencies: - supports-color @@ -12370,6 +12510,8 @@ snapshots: hasown: 2.0.4 side-channel: 1.1.0 + iobuffer@6.0.1: {} + ip-address@10.2.0: {} ip-regex@4.3.0: {} @@ -12588,7 +12730,7 @@ snapshots: istanbul-lib-source-maps@5.0.6: dependencies: '@jridgewell/trace-mapping': 0.3.31 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) istanbul-lib-coverage: 3.2.2 transitivePeerDependencies: - supports-color @@ -13051,7 +13193,7 @@ snapshots: lighthouse-logger@2.0.2: dependencies: - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) marky: 1.3.0 transitivePeerDependencies: - supports-color @@ -13680,7 +13822,7 @@ snapshots: dependencies: '@tootallnate/quickjs-emscripten': 0.23.0 agent-base: 7.1.4 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) get-uri: 6.0.5 http-proxy-agent: 7.0.2 https-proxy-agent: 7.0.6 @@ -13692,7 +13834,7 @@ snapshots: pac-proxy-agent@9.0.1: dependencies: agent-base: 9.0.0 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) get-uri: 8.0.0 http-proxy-agent: 9.0.0 https-proxy-agent: 9.0.0 @@ -13858,6 +14000,10 @@ snapshots: optionalDependencies: '@napi-rs/nice': 1.1.1 + pixelmatch@7.2.0: + dependencies: + pngjs: 7.0.0 + pkg-dir@4.2.0: dependencies: find-up: 4.1.0 @@ -13969,7 +14115,7 @@ snapshots: proxy-agent@6.3.1: dependencies: agent-base: 7.1.4 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) http-proxy-agent: 7.0.2 https-proxy-agent: 7.0.6 lru-cache: 7.18.3 @@ -13982,7 +14128,7 @@ snapshots: proxy-agent@6.5.0: dependencies: agent-base: 7.1.4 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) http-proxy-agent: 7.0.2 https-proxy-agent: 7.0.6 lru-cache: 7.18.3 @@ -13995,7 +14141,7 @@ snapshots: proxy-agent@8.0.1: dependencies: agent-base: 9.0.0 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) http-proxy-agent: 9.0.0 https-proxy-agent: 9.0.0 lru-cache: 7.18.3 @@ -14492,7 +14638,7 @@ snapshots: simple-update-notifier@2.0.0: dependencies: - semver: 7.8.1 + semver: 7.8.4 sirv@3.0.2: dependencies: @@ -14515,7 +14661,7 @@ snapshots: socks-proxy-agent@10.0.0: dependencies: agent-base: 9.0.0 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) socks: 2.8.9 transitivePeerDependencies: - supports-color @@ -14523,7 +14669,7 @@ snapshots: socks-proxy-agent@8.0.5: dependencies: agent-base: 7.1.4 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) socks: 2.8.9 transitivePeerDependencies: - supports-color @@ -14732,7 +14878,7 @@ snapshots: cosmiconfig: 9.0.2(typescript@6.0.3) css-functions-list: 3.3.3 css-tree: 3.2.1 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) fast-glob: 3.3.3 fastest-levenshtein: 1.0.16 file-entry-cache: 11.1.3 @@ -14995,7 +15141,7 @@ snapshots: cac: 6.7.14 chokidar: 4.0.3 consola: 3.4.2 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) esbuild: 0.27.7 fix-dts-default-cjs-exports: 1.0.1 joycon: 3.1.1 @@ -15128,7 +15274,7 @@ snapshots: '@rollup/pluginutils': 5.4.0(rollup@4.62.0) '@volar/typescript': 2.4.28 compare-versions: 6.1.1 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) kolorist: 1.8.0 local-pkg: 1.2.1 magic-string: 0.30.21 @@ -15271,6 +15417,21 @@ snapshots: optionalDependencies: rollup: 4.62.0 + vite@8.0.16(@types/node@25.9.3)(esbuild@0.27.7)(jiti@2.7.0)(tsx@4.22.4)(yaml@2.9.0): + dependencies: + lightningcss: 1.32.0 + picomatch: 4.0.4 + postcss: 8.5.15 + rolldown: 1.0.3 + tinyglobby: 0.2.17 + optionalDependencies: + '@types/node': 25.9.3 + esbuild: 0.27.7 + fsevents: 2.3.3 + jiti: 2.7.0 + tsx: 4.22.4 + yaml: 2.9.0 + vite@8.0.16(@types/node@25.9.3)(esbuild@0.28.0)(jiti@2.7.0)(tsx@4.22.4)(yaml@2.9.0): dependencies: lightningcss: 1.32.0 @@ -15286,6 +15447,36 @@ snapshots: tsx: 4.22.4 yaml: 2.9.0 + vitest@4.1.9(@types/node@25.9.3)(@vitest/coverage-v8@4.1.9)(happy-dom@20.10.4)(jsdom@24.1.3)(vite@8.0.16(@types/node@25.9.3)(esbuild@0.27.7)(jiti@2.7.0)(tsx@4.22.4)(yaml@2.9.0)): + dependencies: + '@vitest/expect': 4.1.9 + '@vitest/mocker': 4.1.9(vite@8.0.16(@types/node@25.9.3)(esbuild@0.27.7)(jiti@2.7.0)(tsx@4.22.4)(yaml@2.9.0)) + '@vitest/pretty-format': 4.1.9 + '@vitest/runner': 4.1.9 + '@vitest/snapshot': 4.1.9 + '@vitest/spy': 4.1.9 + '@vitest/utils': 4.1.9 + es-module-lexer: 2.1.0 + expect-type: 1.3.0 + magic-string: 0.30.21 + obug: 2.1.3 + pathe: 2.0.3 + picomatch: 4.0.4 + std-env: 4.1.0 + tinybench: 2.9.0 + tinyexec: 1.2.4 + tinyglobby: 0.2.17 + tinyrainbow: 3.1.0 + vite: 8.0.16(@types/node@25.9.3)(esbuild@0.27.7)(jiti@2.7.0)(tsx@4.22.4)(yaml@2.9.0) + why-is-node-running: 2.3.0 + optionalDependencies: + '@types/node': 25.9.3 + '@vitest/coverage-v8': 4.1.9(@vitest/browser@4.1.9)(vitest@4.1.9) + happy-dom: 20.10.4 + jsdom: 24.1.3 + transitivePeerDependencies: + - msw + vitest@4.1.9(@types/node@25.9.3)(@vitest/coverage-v8@4.1.9)(happy-dom@20.10.4)(jsdom@24.1.3)(vite@8.0.16(@types/node@25.9.3)(esbuild@0.28.0)(jiti@2.7.0)(tsx@4.22.4)(yaml@2.9.0)): dependencies: '@vitest/expect': 4.1.9 @@ -15328,7 +15519,7 @@ snapshots: dependencies: chalk: 4.1.2 commander: 9.5.0 - debug: 4.4.3(supports-color@5.5.0) + debug: 4.4.3(supports-color@10.2.2) transitivePeerDependencies: - supports-color diff --git a/test/README.md b/test/README.md new file mode 100644 index 00000000..d6345591 --- /dev/null +++ b/test/README.md @@ -0,0 +1,77 @@ +# Cross-adapter verification harness + +Automates the per-adapter verification that was previously manual: run each +framework's example, open the trace, and eyeball that capture is complete and +the UI renders it. Instead: + +- **`pnpm verify`** โ€” Layer A: loads each committed golden `trace.zip` through + the real backend reader, reduces it to a capture summary, and snapshots it. + Fast, no browser, runs in CI. +- **`pnpm verify:ui`** โ€” Layer B (Phase 2): boots the player against each fixture + and visual-diffs every panel. Local, needs a browser. +- **`pnpm fixtures:regen`** โ€” the one browser step: re-runs the examples in trace + mode and refreshes the committed fixtures. + +## Layout + +``` +test/ +โ”œโ”€ capture/ +โ”‚ โ”œโ”€ matrix.ts entry table (adapters ร— runners) โ€” the only file that knows adapters +โ”‚ โ”œโ”€ summarize.ts trace โ†’ deterministic capture summary (the snapshotted value) +โ”‚ โ”œโ”€ capture-parity.test.ts expect(summarize(trace)).toMatchSnapshot() per entry +โ”‚ โ””โ”€ __snapshots__/ committed golden summaries +โ”œโ”€ support/ +โ”‚ โ”œโ”€ paths.ts repo/test roots + fixture paths +โ”‚ โ””โ”€ find-zip.ts newest-zip finder (used by regen) +โ”œโ”€ fixtures/<id>/trace.zip committed golden fixtures (one folder per entry) +โ”œโ”€ regen.ts regenerate fixtures (browser step) +โ””โ”€ vitest.config.ts standalone project (kept out of `pnpm test`) +``` + +## How it works + +`verify` runs against **committed fixtures**, not live browsers. The fixture is a +frozen known-good input: + +- **Change the app / reader / UI** โ†’ `verify` catches a regression against the + frozen fixture, no browser. +- **Change capture** (an adapter, the exporter, the collector) โ†’ run + `fixtures:regen`, then `verify:update` to accept the new snapshot (an intended + change) or read the diff as a regression. + +CI closes the "forgot to regen" gap by re-running `fixtures:regen` when a +capture-layer path changes (see `.github/workflows/`). + +## Commands + +```sh +pnpm verify # Layer A โ€” snapshot check (no browser) +pnpm verify:watch # watch mode +pnpm verify:update # seed/accept snapshots after a regen (vitest -u) +pnpm fixtures:regen # regenerate all ready fixtures (needs Chrome) +pnpm fixtures:regen wdio-mocha # regenerate one entry by id +``` + +A fixture that doesn't exist yet skips its entry, so `verify` is green on a fresh +clone until you run `fixtures:regen`. **First run:** after the first +`fixtures:regen`, run `pnpm verify:update` once to write the initial snapshots +(vitest does not auto-create them); thereafter `pnpm verify` checks against them. + +## Extending + +Add a runner by adding a row to [`capture/matrix.ts`](./capture/matrix.ts): +its `command`, `cleanDirs`, and `traceOutputGlobs`. Nothing else enumerates +adapters. Set `status: 'ready'` once the example produces a trace. + +## Status + +Five entries are `ready` with committed fixtures: WebdriverIO mocha + cucumber, +Selenium mocha + cucumber, Nightwatch BDD. + +`nightwatch-cucumber` has a working example (it runs and captures per-scenario in +trace mode) but is `planned` โ€” a **documented limitation**: Nightwatch's cucumber +runner uses a browser session per scenario, so the adapter's session-scoped trace +export (written once at run-end) has no single live session to finalize and writes +no `trace.zip`. The harness surfaced it; it's tracked in CLAUDE.md known debt. +Flip to `ready` once `@wdio/nightwatch-devtools` supports per-scenario trace writing. diff --git a/test/capture/__snapshots__/capture-parity.test.ts.snap b/test/capture/__snapshots__/capture-parity.test.ts.snap new file mode 100644 index 00000000..bfd6fcfb --- /dev/null +++ b/test/capture/__snapshots__/capture-parity.test.ts.snap @@ -0,0 +1,109 @@ +// Vitest Snapshot v1, https://vitest.dev/guide/snapshot.html + +exports[`Nightwatch ยท BDD (describe/it) [nightwatch-bdd] > captures a consistent trace shape 1`] = ` +{ + "commandVocabulary": [ + "titleContains", + "url", + ], + "groups": { + "maxDepth": 2, + "topLevel": 1, + }, + "hasConsole": true, + "hasFrames": true, + "hasMutations": true, + "hasNetwork": true, + "hasSources": true, + "hasTranscript": true, +} +`; + +exports[`Selenium ยท cucumber [selenium-cucumber] > captures a consistent trace shape 1`] = ` +{ + "commandVocabulary": [ + "assert.match", + "click", + "getText", + "getUrl", + "setValue", + "url", + ], + "groups": { + "maxDepth": 3, + "topLevel": 1, + }, + "hasConsole": true, + "hasFrames": true, + "hasMutations": true, + "hasNetwork": true, + "hasSources": true, + "hasTranscript": true, +} +`; + +exports[`Selenium ยท mocha [selenium-mocha] > captures a consistent trace shape 1`] = ` +{ + "commandVocabulary": [ + "assert.equal", + "assert.match", + "getText", + "getTitle", + "url", + ], + "groups": { + "maxDepth": 2, + "topLevel": 1, + }, + "hasConsole": true, + "hasFrames": true, + "hasMutations": true, + "hasNetwork": true, + "hasSources": true, + "hasTranscript": true, +} +`; + +exports[`WebdriverIO ยท cucumber [wdio-cucumber] > captures a consistent trace shape 1`] = ` +{ + "commandVocabulary": [ + "click", + "setValue", + "toExist", + "toHaveText", + "url", + ], + "groups": { + "maxDepth": 2, + "topLevel": 2, + }, + "hasConsole": true, + "hasFrames": true, + "hasMutations": true, + "hasNetwork": true, + "hasSources": true, + "hasTranscript": true, +} +`; + +exports[`WebdriverIO ยท mocha [wdio-mocha] > captures a consistent trace shape 1`] = ` +{ + "commandVocabulary": [ + "click", + "setValue", + "toExist", + "toHaveText", + "url", + ], + "groups": { + "maxDepth": 1, + "topLevel": 2, + }, + "hasConsole": true, + "hasFrames": true, + "hasMutations": true, + "hasNetwork": true, + "hasSources": true, + "hasTranscript": true, +} +`; diff --git a/test/capture/capture-parity.test.ts b/test/capture/capture-parity.test.ts new file mode 100644 index 00000000..9b3650e3 --- /dev/null +++ b/test/capture/capture-parity.test.ts @@ -0,0 +1,30 @@ +// Layer A โ€” cross-adapter capture parity. Loads each committed golden trace.zip +// through the real backend reader, reduces it to a capture summary, and +// snapshots it. No browser: it reads fixtures produced by `pnpm fixtures:regen`. +// +// A fixture that isn't present yet (fresh clone / CI before a regen) skips its +// entry, so the suite is green without fixtures and meaningful once they exist. +// A capture change surfaces as a snapshot diff on that adapter โ€” accept it with +// `pnpm verify:update` (an intended change) or fix it (a regression). + +import { existsSync } from 'node:fs' + +import { describe, expect, it } from 'vitest' + +import { readTraceZip } from '../../packages/backend/src/trace-reader.js' +import { fixtureTrace } from '../support/paths.js' + +import { READY_ENTRIES } from './matrix.js' +import { summarizeCapture } from './summarize.js' + +for (const entry of READY_ENTRIES) { + const fixture = fixtureTrace(entry.id) + const present = existsSync(fixture) + + describe.skipIf(!present)(`${entry.label} [${entry.id}]`, () => { + it('captures a consistent trace shape', async () => { + const data = await readTraceZip(fixture) + expect(summarizeCapture(data)).toMatchSnapshot() + }) + }) +} diff --git a/test/capture/matrix.ts b/test/capture/matrix.ts new file mode 100644 index 00000000..432766a6 --- /dev/null +++ b/test/capture/matrix.ts @@ -0,0 +1,134 @@ +// The cross-adapter verification matrix. One entry per (adapter, runner) +// example the harness drives through the same login flow. Adding a runner later +// = adding a row here; nothing else in the harness enumerates adapters. +// +// `regen.ts` uses `command` + `traceOutputGlobs` to produce and collect a golden +// fixture; `capture-parity.test.ts` loads the fixture and snapshots its capture +// summary. The two stay in sync through this file. + +export type AdapterId = 'wdio' | 'selenium' | 'nightwatch' +export type RunnerId = 'mocha' | 'cucumber' | 'bdd' + +export interface VerificationEntry { + /** Stable slug โ€” also the fixture folder name under `test/fixtures/`. */ + id: string + adapter: AdapterId + runner: RunnerId + label: string + /** `ready` entries regen + verify; `planned` are documented gaps skipped by + * both (kept here so the matrix shows the whole intended surface). */ + status: 'ready' | 'planned' + /** Why a `planned` entry can't run yet. */ + plannedReason?: string + /** Command to produce a trace, spawned from the repo root with + * `DEVTOOLS_MODE=trace` in the environment. */ + command: { cmd: string; args: string[] } + /** Directories cleaned before a regen run so "newest zip" is unambiguous. */ + cleanDirs: string[] + /** Globs (repo-root-relative) the produced `trace.zip` lands under; the + * newest match is collected as the fixture. */ + traceOutputGlobs: string[] +} + +export const ENTRIES: VerificationEntry[] = [ + { + id: 'wdio-mocha', + adapter: 'wdio', + runner: 'mocha', + label: 'WebdriverIO ยท mocha', + status: 'ready', + command: { + cmd: 'npx', + args: ['wdio', 'run', './examples/wdio/mocha/wdio.trace.conf.ts'] + }, + cleanDirs: ['examples/wdio/mocha/test-results'], + traceOutputGlobs: ['examples/wdio/mocha/test-results/**/trace*.zip'] + }, + { + id: 'wdio-cucumber', + adapter: 'wdio', + runner: 'cucumber', + label: 'WebdriverIO ยท cucumber', + status: 'ready', + command: { + cmd: 'npx', + args: ['wdio', 'run', './examples/wdio/cucumber/wdio.trace.conf.ts'] + }, + cleanDirs: ['examples/wdio/cucumber/test-results'], + traceOutputGlobs: ['examples/wdio/cucumber/test-results/**/trace*.zip'] + }, + { + id: 'selenium-mocha', + adapter: 'selenium', + runner: 'mocha', + label: 'Selenium ยท mocha', + status: 'ready', + command: { + cmd: 'pnpm', + args: ['--filter', '@wdio/selenium-devtools', 'example:mocha'] + }, + cleanDirs: [ + 'examples/selenium/mocha-test/test/test-results', + 'examples/selenium/mocha-test/allure-results' + ], + traceOutputGlobs: [ + 'examples/selenium/mocha-test/test/test-results/**/trace*.zip' + ] + }, + { + id: 'selenium-cucumber', + adapter: 'selenium', + runner: 'cucumber', + label: 'Selenium ยท cucumber', + status: 'ready', + command: { + cmd: 'pnpm', + args: ['--filter', '@wdio/selenium-devtools', 'example:cucumber'] + }, + cleanDirs: [ + 'examples/selenium/cucumber-test/features/support/test-results' + ], + traceOutputGlobs: [ + 'examples/selenium/cucumber-test/features/support/test-results/**/trace*.zip' + ] + }, + { + id: 'nightwatch-bdd', + adapter: 'nightwatch', + runner: 'bdd', + label: 'Nightwatch ยท BDD (describe/it)', + status: 'ready', + command: { + cmd: 'pnpm', + args: ['--filter', '@wdio/nightwatch-devtools', 'example'] + }, + cleanDirs: ['examples/nightwatch/tests/test-results'], + traceOutputGlobs: ['examples/nightwatch/tests/test-results/**/trace*.zip'] + }, + { + id: 'nightwatch-cucumber', + adapter: 'nightwatch', + runner: 'cucumber', + label: 'Nightwatch ยท cucumber', + // Documented limitation: Nightwatch's cucumber runner launches a browser + // session PER SCENARIO, so the adapter's session-scoped trace export (written + // once at run-end) has no single live session to finalize and produces no + // trace.zip. The example runs and captures per-scenario in trace mode; only + // the session-zip export is missing. See CLAUDE.md known debt. Supporting it + // needs per-scenario trace writing in @wdio/nightwatch-devtools. + status: 'planned', + command: { + cmd: 'pnpm', + args: ['--filter', '@wdio/nightwatch-devtools', 'example:cucumber'] + }, + cleanDirs: [ + 'examples/nightwatch/cucumber/test-results', + 'examples/nightwatch/cucumber/features/step_definitions/test-results' + ], + traceOutputGlobs: [ + 'examples/nightwatch/cucumber/**/test-results/**/trace*.zip' + ] + } +] + +export const READY_ENTRIES = ENTRIES.filter((entry) => entry.status === 'ready') diff --git a/test/capture/summarize.ts b/test/capture/summarize.ts new file mode 100644 index 00000000..892e079f --- /dev/null +++ b/test/capture/summarize.ts @@ -0,0 +1,61 @@ +// Reduces a reconstructed trace to a deterministic capture summary โ€” the value +// that gets snapshotted per adapter. Everything here must be stable across runs +// (no timestamps, ids, counts that drift, or absolute paths), so the snapshot +// only changes when capture behaviour actually changes. +// +// The command vocabulary (the sorted set of distinct command names) is the +// sharp signal, and deliberately carries assertions too: each framework names +// them differently and inconsistently โ€” WebdriverIO `toExist`/`toHaveText` +// (no result payload), Selenium `assert.*` (payload only on failure), Nightwatch +// `titleContains` (with a `passed` flag) โ€” so there is no clean cross-adapter +// predicate for "is an assertion". The vocabulary sidesteps that: if an adapter +// stops emitting its assertion (or any) rows, that name vanishes from the set +// and the snapshot diff names it. No per-framework heuristic required. + +import type { + TraceActionChild, + TracePlayerData +} from '../../packages/shared/src/trace-player.js' + +export interface CaptureSummary { + /** Sorted distinct command names captured โ€” actions AND assertions. */ + commandVocabulary: string[] + hasNetwork: boolean + hasConsole: boolean + hasMutations: boolean + hasSources: boolean + hasTranscript: boolean + hasFrames: boolean + /** Structural shape of the group tree (no titles โ€” those carry run data). */ + groups: { topLevel: number; maxDepth: number } +} + +function groupDepth(child: TraceActionChild): number { + if ('group' in child) { + const children = child.group.children ?? [] + return 1 + Math.max(0, ...children.map(groupDepth)) + } + return 0 +} + +export function summarizeCapture(data: TracePlayerData): CaptureSummary { + const trace = data.trace + const roots = data.groups ?? [] + + return { + commandVocabulary: [ + ...new Set(trace.commands.map((c) => c.command)) + ].sort(), + hasNetwork: trace.networkRequests.length > 0, + hasConsole: trace.consoleLogs.length > 0, + hasMutations: trace.mutations.length > 0, + hasSources: Object.keys(trace.sources).length > 0, + hasTranscript: + typeof data.transcript === 'string' && data.transcript.length > 0, + hasFrames: data.frames.length > 0, + groups: { + topLevel: roots.filter((child) => 'group' in child).length, + maxDepth: Math.max(0, ...roots.map(groupDepth)) + } + } +} diff --git a/test/fixtures/nightwatch-bdd/trace.zip b/test/fixtures/nightwatch-bdd/trace.zip new file mode 100644 index 00000000..344da811 Binary files /dev/null and b/test/fixtures/nightwatch-bdd/trace.zip differ diff --git a/test/fixtures/selenium-cucumber/trace.zip b/test/fixtures/selenium-cucumber/trace.zip new file mode 100644 index 00000000..f59c4160 Binary files /dev/null and b/test/fixtures/selenium-cucumber/trace.zip differ diff --git a/test/fixtures/selenium-mocha/trace.zip b/test/fixtures/selenium-mocha/trace.zip new file mode 100644 index 00000000..8b5641ac Binary files /dev/null and b/test/fixtures/selenium-mocha/trace.zip differ diff --git a/test/fixtures/wdio-cucumber/trace.zip b/test/fixtures/wdio-cucumber/trace.zip new file mode 100644 index 00000000..03323a3c Binary files /dev/null and b/test/fixtures/wdio-cucumber/trace.zip differ diff --git a/test/fixtures/wdio-mocha/trace.zip b/test/fixtures/wdio-mocha/trace.zip new file mode 100644 index 00000000..3dfb611e Binary files /dev/null and b/test/fixtures/wdio-mocha/trace.zip differ diff --git a/test/regen.ts b/test/regen.ts new file mode 100644 index 00000000..e982b9cb --- /dev/null +++ b/test/regen.ts @@ -0,0 +1,81 @@ +// Regenerates the committed golden fixtures โ€” the ONE browser-dependent step of +// the harness. Run it deliberately when you intend to change capture, then +// review the fixture diff (and `pnpm verify:update` the snapshots) and commit. +// Everyday verification runs against the committed fixtures with no browser. +// +// pnpm fixtures:regen # all ready entries +// pnpm fixtures:regen wdio-mocha # one entry by id +// +// For each entry: clean its output dirs, run the example in trace mode +// (DEVTOOLS_MODE=trace), find the newest produced trace.zip, and copy it to +// test/fixtures/<id>/trace.zip. + +import { spawnSync } from 'node:child_process' +import { copyFileSync, mkdirSync, rmSync } from 'node:fs' +import path from 'node:path' + +import { READY_ENTRIES, type VerificationEntry } from './capture/matrix.js' +import { findNewestZip } from './support/find-zip.js' +import { fixtureTrace, REPO_ROOT } from './support/paths.js' + +function regen(entry: VerificationEntry): boolean { + console.log(`\nโ”€โ”€ ${entry.label} [${entry.id}] โ”€โ”€`) + + for (const dir of entry.cleanDirs) { + rmSync(path.resolve(REPO_ROOT, dir), { recursive: true, force: true }) + } + + const { cmd, args } = entry.command + console.log(` $ DEVTOOLS_MODE=trace ${cmd} ${args.join(' ')}`) + const run = spawnSync(cmd, args, { + cwd: REPO_ROOT, + stdio: 'inherit', + env: { ...process.env, DEVTOOLS_MODE: 'trace' }, + shell: process.platform === 'win32' + }) + if (run.status !== 0) { + console.warn( + ` โš  command exited with ${run.status ?? run.signal} โ€” some examples fail on purpose; still collecting.` + ) + } + + const zip = findNewestZip(entry.traceOutputGlobs, REPO_ROOT) + if (!zip) { + console.error( + ` โœ— no trace.zip found under ${entry.traceOutputGlobs.join(', ')}` + ) + return false + } + + const dest = fixtureTrace(entry.id) + mkdirSync(path.dirname(dest), { recursive: true }) + copyFileSync(zip, dest) + console.log( + ` โœ“ ${path.relative(REPO_ROOT, zip)} โ†’ ${path.relative(REPO_ROOT, dest)}` + ) + return true +} + +const only = process.argv.slice(2) +const targets = only.length + ? READY_ENTRIES.filter((e) => only.includes(e.id)) + : READY_ENTRIES + +if (only.length && targets.length === 0) { + console.error(`No ready entry matches: ${only.join(', ')}`) + console.error(`Known: ${READY_ENTRIES.map((e) => e.id).join(', ')}`) + process.exit(1) +} + +const results = targets.map((entry) => ({ id: entry.id, ok: regen(entry) })) +const failed = results.filter((r) => !r.ok) + +console.log( + `\nRegenerated ${results.length - failed.length}/${results.length} fixtures.` +) +if (failed.length) { + console.log( + `Missing: ${failed.map((r) => r.id).join(', ')} โ€” check the run output above (Chrome present? example wired for trace?).` + ) + process.exit(1) +} diff --git a/test/support/find-zip.ts b/test/support/find-zip.ts new file mode 100644 index 00000000..ba8d8b91 --- /dev/null +++ b/test/support/find-zip.ts @@ -0,0 +1,59 @@ +import { existsSync, readdirSync, statSync } from 'node:fs' +import path from 'node:path' + +// Match a single-`*` basename glob (e.g. "trace*.zip") without a RegExp โ€” the +// matrix's globs always use one wildcard, so prefix/suffix matching suffices +// and avoids a dynamic-RegExp lint/CodeQL flag. +function basenameMatches(fileGlob: string, name: string): boolean { + const star = fileGlob.indexOf('*') + if (star === -1) { + return name === fileGlob + } + const prefix = fileGlob.slice(0, star) + const suffix = fileGlob.slice(star + 1) + return ( + name.length >= prefix.length + suffix.length && + name.startsWith(prefix) && + name.endsWith(suffix) + ) +} + +// Resolve recursive "dir/<...>/trace*.zip" globs against a base dir and return +// the newest matching file. No glob dependency โ€” a small walk covers the fixed +// shapes the matrix uses. +function matchesFor(glob: string, repoRoot: string): string[] { + const wildcardAt = glob.indexOf('**') + const baseRel = + wildcardAt === -1 ? path.dirname(glob) : glob.slice(0, wildcardAt) + const baseAbs = path.resolve(repoRoot, baseRel) + const fileGlob = path.basename(glob) + if (!existsSync(baseAbs)) { + return [] + } + + const out: string[] = [] + const walk = (dir: string): void => { + for (const dirent of readdirSync(dir, { withFileTypes: true })) { + const full = path.join(dir, dirent.name) + if (dirent.isDirectory()) { + walk(full) + } else if (basenameMatches(fileGlob, dirent.name)) { + out.push(full) + } + } + } + walk(baseAbs) + return out +} + +/** Newest file matching any of the globs (repo-root-relative), or null. */ +export function findNewestZip( + globs: string[], + repoRoot: string +): string | null { + const matches = globs.flatMap((glob) => matchesFor(glob, repoRoot)) + if (matches.length === 0) { + return null + } + return matches.sort((a, b) => statSync(b).mtimeMs - statSync(a).mtimeMs)[0] +} diff --git a/test/support/paths.ts b/test/support/paths.ts new file mode 100644 index 00000000..3273a4bb --- /dev/null +++ b/test/support/paths.ts @@ -0,0 +1,20 @@ +import path from 'node:path' +import { fileURLToPath } from 'node:url' + +// Roots resolved from this file's location so the harness works regardless of +// the process cwd (vitest, tsx regen, or a CI runner). +export const TEST_ROOT = path.resolve( + path.dirname(fileURLToPath(import.meta.url)), + '..' +) +export const REPO_ROOT = path.resolve(TEST_ROOT, '..') + +/** Committed golden trace for an entry: `test/fixtures/<id>/trace.zip`. */ +export function fixtureTrace(id: string): string { + return path.join(TEST_ROOT, 'fixtures', id, 'trace.zip') +} + +/** Recorded live event stream for an entry (Phase 1b). */ +export function fixtureLiveEvents(id: string): string { + return path.join(TEST_ROOT, 'fixtures', id, 'live-events.json') +} diff --git a/test/support/serve-fixture.ts b/test/support/serve-fixture.ts new file mode 100644 index 00000000..276a5c0e --- /dev/null +++ b/test/support/serve-fixture.ts @@ -0,0 +1,27 @@ +// Layer B support: boot the real backend in trace-serve mode against a +// committed golden fixture so the WebdriverIO visual specs can drive the trace +// player. Mirrors show-trace.ts (readTraceZip โ†’ start({ trace })), but returns +// an instance-scoped close handle instead of opening a browser. + +import { readTraceZip } from '../../packages/backend/src/trace-reader.js' +import { start } from '../../packages/backend/src/index.js' + +import { fixtureTrace } from './paths.js' + +export interface ServedFixture { + url: string + close: () => Promise<void> +} + +/** Reconstruct the fixture trace, serve it, and hand back the player URL plus a + * close handle. `start()` resolves to `{ server, port }`, so `server.close()` + * is a real per-instance shutdown (the module-level `stop()` export closes the + * last-started server, which would race parallel/sequential specs). */ +export async function serveFixture(id: string): Promise<ServedFixture> { + const trace = await readTraceZip(fixtureTrace(id)) + const { server, port } = await start({ trace }) + return { + url: `http://localhost:${port}`, + close: () => server.close() + } +} diff --git a/test/visual/.gitignore b/test/visual/.gitignore new file mode 100644 index 00000000..6634a99a --- /dev/null +++ b/test/visual/.gitignore @@ -0,0 +1,3 @@ +# Visual-service working dir โ€” per-run actual + diff images. Only the committed +# goldens under ./baseline are tracked. +.tmp/ diff --git a/test/visual/README.md b/test/visual/README.md new file mode 100644 index 00000000..3caf02c1 --- /dev/null +++ b/test/visual/README.md @@ -0,0 +1,34 @@ +# Layer B โ€” visual regression + +Boots the trace **player** against each committed golden fixture and +visual-snapshots every panel with WebdriverIO + [`@wdio/visual-service`]. + +`player.e2e.ts` walks `READY_ENTRIES` from the capture matrix. For each entry it +serves `fixtures/<id>/trace.zip` through the real backend (`serveFixture` โ†’ +`readTraceZip` + `start({ trace })`), loads the player in headless Chrome, then +snapshots the full player plus the timeline, controls, snapshot/viewport, +actions, console, network, errors, and a11y panels. A block whose fixture is +missing is skipped, so the suite is green on a fresh checkout and meaningful +once fixtures exist. + +`dashboard.e2e.ts` is a skipped scaffold for the live dashboard โ€” it awaits +Phase 1b (`fixtures/<id>/live-events.json` + a live-serve helper). + +## Running + +``` +pnpm fixtures:regen # produce the golden trace.zip fixtures first +npx wdio run test/visual/wdio.conf.ts +``` + +Requires `@wdio/visual-service` installed and a local Chrome/chromedriver. + +## Baselines + +Committed goldens live under `baseline/`. The config sets `autoSaveBaseline`, so +the **first run seeds** the baselines โ€” commit them, and later runs assert a 0% +mismatch against them. Actual/diff images from a failing run land in `.tmp/` +(not committed). An intended UI change is accepted by deleting the affected +baseline images and re-running to reseed. + +[`@wdio/visual-service`]: https://webdriver.io/docs/visual-testing diff --git a/test/visual/baseline/nightwatch-bdd-a11y-1600x900-dpr1.png b/test/visual/baseline/nightwatch-bdd-a11y-1600x900-dpr1.png new file mode 100644 index 00000000..f585740f Binary files /dev/null and b/test/visual/baseline/nightwatch-bdd-a11y-1600x900-dpr1.png differ diff --git a/test/visual/baseline/nightwatch-bdd-actions-1600x900-dpr1.png b/test/visual/baseline/nightwatch-bdd-actions-1600x900-dpr1.png new file mode 100644 index 00000000..119987fd Binary files /dev/null and b/test/visual/baseline/nightwatch-bdd-actions-1600x900-dpr1.png differ diff --git a/test/visual/baseline/nightwatch-bdd-console-1600x900-dpr1.png b/test/visual/baseline/nightwatch-bdd-console-1600x900-dpr1.png new file mode 100644 index 00000000..5f0a488e Binary files /dev/null and b/test/visual/baseline/nightwatch-bdd-console-1600x900-dpr1.png differ diff --git a/test/visual/baseline/nightwatch-bdd-controls-1600x900-dpr1.png b/test/visual/baseline/nightwatch-bdd-controls-1600x900-dpr1.png new file mode 100644 index 00000000..b9da81a1 Binary files /dev/null and b/test/visual/baseline/nightwatch-bdd-controls-1600x900-dpr1.png differ diff --git a/test/visual/baseline/nightwatch-bdd-errors-1600x900-dpr1.png b/test/visual/baseline/nightwatch-bdd-errors-1600x900-dpr1.png new file mode 100644 index 00000000..a58f35c9 Binary files /dev/null and b/test/visual/baseline/nightwatch-bdd-errors-1600x900-dpr1.png differ diff --git a/test/visual/baseline/nightwatch-bdd-network-1600x900-dpr1.png b/test/visual/baseline/nightwatch-bdd-network-1600x900-dpr1.png new file mode 100644 index 00000000..019cfa61 Binary files /dev/null and b/test/visual/baseline/nightwatch-bdd-network-1600x900-dpr1.png differ diff --git a/test/visual/baseline/nightwatch-bdd-snapshot-1600x900-dpr1.png b/test/visual/baseline/nightwatch-bdd-snapshot-1600x900-dpr1.png new file mode 100644 index 00000000..31d52c51 Binary files /dev/null and b/test/visual/baseline/nightwatch-bdd-snapshot-1600x900-dpr1.png differ diff --git a/test/visual/baseline/nightwatch-bdd-timeline-1600x900-dpr1.png b/test/visual/baseline/nightwatch-bdd-timeline-1600x900-dpr1.png new file mode 100644 index 00000000..a4de06bf Binary files /dev/null and b/test/visual/baseline/nightwatch-bdd-timeline-1600x900-dpr1.png differ diff --git a/test/visual/baseline/player-nightwatch-bdd-1600x900-dpr1.png b/test/visual/baseline/player-nightwatch-bdd-1600x900-dpr1.png new file mode 100644 index 00000000..176388c2 Binary files /dev/null and b/test/visual/baseline/player-nightwatch-bdd-1600x900-dpr1.png differ diff --git a/test/visual/baseline/player-selenium-cucumber-1600x900-dpr1.png b/test/visual/baseline/player-selenium-cucumber-1600x900-dpr1.png new file mode 100644 index 00000000..ee4602b3 Binary files /dev/null and b/test/visual/baseline/player-selenium-cucumber-1600x900-dpr1.png differ diff --git a/test/visual/baseline/player-selenium-mocha-1600x900-dpr1.png b/test/visual/baseline/player-selenium-mocha-1600x900-dpr1.png new file mode 100644 index 00000000..14033160 Binary files /dev/null and b/test/visual/baseline/player-selenium-mocha-1600x900-dpr1.png differ diff --git a/test/visual/baseline/player-wdio-cucumber-1600x900-dpr1.png b/test/visual/baseline/player-wdio-cucumber-1600x900-dpr1.png new file mode 100644 index 00000000..13c2f6ff Binary files /dev/null and b/test/visual/baseline/player-wdio-cucumber-1600x900-dpr1.png differ diff --git a/test/visual/baseline/player-wdio-mocha-1600x900-dpr1.png b/test/visual/baseline/player-wdio-mocha-1600x900-dpr1.png new file mode 100644 index 00000000..e218a510 Binary files /dev/null and b/test/visual/baseline/player-wdio-mocha-1600x900-dpr1.png differ diff --git a/test/visual/baseline/selenium-cucumber-a11y-1600x900-dpr1.png b/test/visual/baseline/selenium-cucumber-a11y-1600x900-dpr1.png new file mode 100644 index 00000000..f585740f Binary files /dev/null and b/test/visual/baseline/selenium-cucumber-a11y-1600x900-dpr1.png differ diff --git a/test/visual/baseline/selenium-cucumber-actions-1600x900-dpr1.png b/test/visual/baseline/selenium-cucumber-actions-1600x900-dpr1.png new file mode 100644 index 00000000..2120ca0a Binary files /dev/null and b/test/visual/baseline/selenium-cucumber-actions-1600x900-dpr1.png differ diff --git a/test/visual/baseline/selenium-cucumber-console-1600x900-dpr1.png b/test/visual/baseline/selenium-cucumber-console-1600x900-dpr1.png new file mode 100644 index 00000000..5b2d4cdb Binary files /dev/null and b/test/visual/baseline/selenium-cucumber-console-1600x900-dpr1.png differ diff --git a/test/visual/baseline/selenium-cucumber-controls-1600x900-dpr1.png b/test/visual/baseline/selenium-cucumber-controls-1600x900-dpr1.png new file mode 100644 index 00000000..4158729a Binary files /dev/null and b/test/visual/baseline/selenium-cucumber-controls-1600x900-dpr1.png differ diff --git a/test/visual/baseline/selenium-cucumber-errors-1600x900-dpr1.png b/test/visual/baseline/selenium-cucumber-errors-1600x900-dpr1.png new file mode 100644 index 00000000..7454fbe8 Binary files /dev/null and b/test/visual/baseline/selenium-cucumber-errors-1600x900-dpr1.png differ diff --git a/test/visual/baseline/selenium-cucumber-network-1600x900-dpr1.png b/test/visual/baseline/selenium-cucumber-network-1600x900-dpr1.png new file mode 100644 index 00000000..0b66a628 Binary files /dev/null and b/test/visual/baseline/selenium-cucumber-network-1600x900-dpr1.png differ diff --git a/test/visual/baseline/selenium-cucumber-snapshot-1600x900-dpr1.png b/test/visual/baseline/selenium-cucumber-snapshot-1600x900-dpr1.png new file mode 100644 index 00000000..6358325c Binary files /dev/null and b/test/visual/baseline/selenium-cucumber-snapshot-1600x900-dpr1.png differ diff --git a/test/visual/baseline/selenium-cucumber-timeline-1600x900-dpr1.png b/test/visual/baseline/selenium-cucumber-timeline-1600x900-dpr1.png new file mode 100644 index 00000000..6cfe65bc Binary files /dev/null and b/test/visual/baseline/selenium-cucumber-timeline-1600x900-dpr1.png differ diff --git a/test/visual/baseline/selenium-mocha-a11y-1600x900-dpr1.png b/test/visual/baseline/selenium-mocha-a11y-1600x900-dpr1.png new file mode 100644 index 00000000..f585740f Binary files /dev/null and b/test/visual/baseline/selenium-mocha-a11y-1600x900-dpr1.png differ diff --git a/test/visual/baseline/selenium-mocha-actions-1600x900-dpr1.png b/test/visual/baseline/selenium-mocha-actions-1600x900-dpr1.png new file mode 100644 index 00000000..f1201c08 Binary files /dev/null and b/test/visual/baseline/selenium-mocha-actions-1600x900-dpr1.png differ diff --git a/test/visual/baseline/selenium-mocha-console-1600x900-dpr1.png b/test/visual/baseline/selenium-mocha-console-1600x900-dpr1.png new file mode 100644 index 00000000..02747edf Binary files /dev/null and b/test/visual/baseline/selenium-mocha-console-1600x900-dpr1.png differ diff --git a/test/visual/baseline/selenium-mocha-controls-1600x900-dpr1.png b/test/visual/baseline/selenium-mocha-controls-1600x900-dpr1.png new file mode 100644 index 00000000..c6f40355 Binary files /dev/null and b/test/visual/baseline/selenium-mocha-controls-1600x900-dpr1.png differ diff --git a/test/visual/baseline/selenium-mocha-errors-1600x900-dpr1.png b/test/visual/baseline/selenium-mocha-errors-1600x900-dpr1.png new file mode 100644 index 00000000..e8c849d7 Binary files /dev/null and b/test/visual/baseline/selenium-mocha-errors-1600x900-dpr1.png differ diff --git a/test/visual/baseline/selenium-mocha-network-1600x900-dpr1.png b/test/visual/baseline/selenium-mocha-network-1600x900-dpr1.png new file mode 100644 index 00000000..3c9b0898 Binary files /dev/null and b/test/visual/baseline/selenium-mocha-network-1600x900-dpr1.png differ diff --git a/test/visual/baseline/selenium-mocha-snapshot-1600x900-dpr1.png b/test/visual/baseline/selenium-mocha-snapshot-1600x900-dpr1.png new file mode 100644 index 00000000..e8aed2a2 Binary files /dev/null and b/test/visual/baseline/selenium-mocha-snapshot-1600x900-dpr1.png differ diff --git a/test/visual/baseline/selenium-mocha-timeline-1600x900-dpr1.png b/test/visual/baseline/selenium-mocha-timeline-1600x900-dpr1.png new file mode 100644 index 00000000..07a4ab6d Binary files /dev/null and b/test/visual/baseline/selenium-mocha-timeline-1600x900-dpr1.png differ diff --git a/test/visual/baseline/wdio-cucumber-a11y-1600x900-dpr1.png b/test/visual/baseline/wdio-cucumber-a11y-1600x900-dpr1.png new file mode 100644 index 00000000..5ac6df92 Binary files /dev/null and b/test/visual/baseline/wdio-cucumber-a11y-1600x900-dpr1.png differ diff --git a/test/visual/baseline/wdio-cucumber-actions-1600x900-dpr1.png b/test/visual/baseline/wdio-cucumber-actions-1600x900-dpr1.png new file mode 100644 index 00000000..7527cb59 Binary files /dev/null and b/test/visual/baseline/wdio-cucumber-actions-1600x900-dpr1.png differ diff --git a/test/visual/baseline/wdio-cucumber-console-1600x900-dpr1.png b/test/visual/baseline/wdio-cucumber-console-1600x900-dpr1.png new file mode 100644 index 00000000..d164e1d6 Binary files /dev/null and b/test/visual/baseline/wdio-cucumber-console-1600x900-dpr1.png differ diff --git a/test/visual/baseline/wdio-cucumber-controls-1600x900-dpr1.png b/test/visual/baseline/wdio-cucumber-controls-1600x900-dpr1.png new file mode 100644 index 00000000..c7b622b7 Binary files /dev/null and b/test/visual/baseline/wdio-cucumber-controls-1600x900-dpr1.png differ diff --git a/test/visual/baseline/wdio-cucumber-errors-1600x900-dpr1.png b/test/visual/baseline/wdio-cucumber-errors-1600x900-dpr1.png new file mode 100644 index 00000000..7454fbe8 Binary files /dev/null and b/test/visual/baseline/wdio-cucumber-errors-1600x900-dpr1.png differ diff --git a/test/visual/baseline/wdio-cucumber-network-1600x900-dpr1.png b/test/visual/baseline/wdio-cucumber-network-1600x900-dpr1.png new file mode 100644 index 00000000..9027ad25 Binary files /dev/null and b/test/visual/baseline/wdio-cucumber-network-1600x900-dpr1.png differ diff --git a/test/visual/baseline/wdio-cucumber-snapshot-1600x900-dpr1.png b/test/visual/baseline/wdio-cucumber-snapshot-1600x900-dpr1.png new file mode 100644 index 00000000..a048b09a Binary files /dev/null and b/test/visual/baseline/wdio-cucumber-snapshot-1600x900-dpr1.png differ diff --git a/test/visual/baseline/wdio-cucumber-timeline-1600x900-dpr1.png b/test/visual/baseline/wdio-cucumber-timeline-1600x900-dpr1.png new file mode 100644 index 00000000..4b3fa5e2 Binary files /dev/null and b/test/visual/baseline/wdio-cucumber-timeline-1600x900-dpr1.png differ diff --git a/test/visual/baseline/wdio-mocha-a11y-1600x900-dpr1.png b/test/visual/baseline/wdio-mocha-a11y-1600x900-dpr1.png new file mode 100644 index 00000000..5ac6df92 Binary files /dev/null and b/test/visual/baseline/wdio-mocha-a11y-1600x900-dpr1.png differ diff --git a/test/visual/baseline/wdio-mocha-actions-1600x900-dpr1.png b/test/visual/baseline/wdio-mocha-actions-1600x900-dpr1.png new file mode 100644 index 00000000..1f3e13d7 Binary files /dev/null and b/test/visual/baseline/wdio-mocha-actions-1600x900-dpr1.png differ diff --git a/test/visual/baseline/wdio-mocha-console-1600x900-dpr1.png b/test/visual/baseline/wdio-mocha-console-1600x900-dpr1.png new file mode 100644 index 00000000..00e84833 Binary files /dev/null and b/test/visual/baseline/wdio-mocha-console-1600x900-dpr1.png differ diff --git a/test/visual/baseline/wdio-mocha-controls-1600x900-dpr1.png b/test/visual/baseline/wdio-mocha-controls-1600x900-dpr1.png new file mode 100644 index 00000000..eb314b4f Binary files /dev/null and b/test/visual/baseline/wdio-mocha-controls-1600x900-dpr1.png differ diff --git a/test/visual/baseline/wdio-mocha-errors-1600x900-dpr1.png b/test/visual/baseline/wdio-mocha-errors-1600x900-dpr1.png new file mode 100644 index 00000000..0edeeb6c Binary files /dev/null and b/test/visual/baseline/wdio-mocha-errors-1600x900-dpr1.png differ diff --git a/test/visual/baseline/wdio-mocha-network-1600x900-dpr1.png b/test/visual/baseline/wdio-mocha-network-1600x900-dpr1.png new file mode 100644 index 00000000..97261bd6 Binary files /dev/null and b/test/visual/baseline/wdio-mocha-network-1600x900-dpr1.png differ diff --git a/test/visual/baseline/wdio-mocha-snapshot-1600x900-dpr1.png b/test/visual/baseline/wdio-mocha-snapshot-1600x900-dpr1.png new file mode 100644 index 00000000..23206f1b Binary files /dev/null and b/test/visual/baseline/wdio-mocha-snapshot-1600x900-dpr1.png differ diff --git a/test/visual/baseline/wdio-mocha-timeline-1600x900-dpr1.png b/test/visual/baseline/wdio-mocha-timeline-1600x900-dpr1.png new file mode 100644 index 00000000..cf6f6457 Binary files /dev/null and b/test/visual/baseline/wdio-mocha-timeline-1600x900-dpr1.png differ diff --git a/test/visual/dashboard.e2e.ts b/test/visual/dashboard.e2e.ts new file mode 100644 index 00000000..b6dfe891 --- /dev/null +++ b/test/visual/dashboard.e2e.ts @@ -0,0 +1,20 @@ +// Layer B โ€” live dashboard visual regression. SCAFFOLD ONLY. +// +// Unlike player.e2e.ts (which serves a static trace.zip), the live dashboard +// needs a worker WebSocket streaming a recorded event feed into the backend. +// That depends on Phase 1b: committed `fixtures/<id>/live-events.json` streams +// plus a live-serve helper that boots the backend without `{ trace }` and +// replays those events over the worker socket. Neither exists yet. +// +// This block stays skipped until then. Do NOT implement live serving here โ€” +// it lands with Phase 1b so the harness has a real feed to replay. + +import { READY_ENTRIES } from '../capture/matrix.js' + +describe.skip('live dashboard (awaits Phase 1b live fixtures)', () => { + for (const entry of READY_ENTRIES) { + it(`matches the live dashboard for ${entry.label}`, () => { + // Pending: serveLiveFixture(entry.id) โ†’ browser.url โ†’ snapshot panels. + }) + } +}) diff --git a/test/visual/player.e2e.ts b/test/visual/player.e2e.ts new file mode 100644 index 00000000..bb32d84c --- /dev/null +++ b/test/visual/player.e2e.ts @@ -0,0 +1,98 @@ +// Layer B โ€” trace-player visual regression. For each ready matrix entry we serve +// its committed golden fixture, load the player, and snapshot the whole player +// plus every panel we have a stable selector for. A missing fixture skips the +// block (fresh clone / CI before `pnpm fixtures:regen`), matching Layer A. +// +// The player is nested Lit shadow DOM with no data-testid, so panels are +// addressed by their custom-element tag through the `>>>` deep combinator. +// Dock panels (Console/Network/Errors/A11y) live inside inactive tabs; rather +// than click through nested shadow roots we fire the same `open-dock-tab` +// window event the app itself uses to switch tabs programmatically (tabs.ts). + +import fs from 'node:fs' + +import { browser, $, expect } from '@wdio/globals' + +import { READY_ENTRIES } from '../capture/matrix.js' +import { fixtureTrace } from '../support/paths.js' +import { serveFixture, type ServedFixture } from '../support/serve-fixture.js' + +const PLAYER_ROOT = 'wdio-devtools' +// Only exists in player mode โ€” the reliable "player has mounted" signal. +const PLAYER_READY = '>>> wdio-devtools-trace-player-controls' + +// Max % of pixels allowed to differ. A small tolerance absorbs sub-pixel +// anti-aliasing / font-hinting noise (and headed-vs-headless rendering); a real +// UI change is far larger. Prevents brittle 0%-threshold failures. +const MISMATCH_TOLERANCE = 0.1 + +interface Panel { + name: string + selector: string + /** Dock-tab label to activate before snapshotting; omit for always-visible panels. */ + tab?: string +} + +const PANELS: Panel[] = [ + { name: 'timeline', selector: '>>> wdio-devtools-trace-timeline' }, + { name: 'controls', selector: '>>> wdio-devtools-trace-player-controls' }, + { name: 'snapshot', selector: '>>> wdio-devtools-browser' }, + { name: 'actions', selector: '>>> wdio-devtools-actions' }, + { + name: 'console', + selector: '>>> wdio-devtools-console-logs', + tab: 'Console' + }, + { name: 'network', selector: '>>> wdio-devtools-network', tab: 'Network' }, + { name: 'errors', selector: '>>> wdio-devtools-errors', tab: 'Errors' }, + { name: 'a11y', selector: '>>> wdio-devtools-a11y', tab: 'A11y' } +] + +async function activateDockTab(label: string): Promise<void> { + await browser.execute((tab: string) => { + window.dispatchEvent( + new CustomEvent('open-dock-tab', { detail: { label: tab } }) + ) + }, label) +} + +for (const entry of READY_ENTRIES) { + const present = fs.existsSync(fixtureTrace(entry.id)) + const suite = present ? describe : describe.skip + + suite(entry.label, () => { + let served: ServedFixture + + before(async () => { + served = await serveFixture(entry.id) + await browser.url(served.url) + await $(PLAYER_ROOT).waitForExist({ timeout: 20000 }) + await $(PLAYER_READY).waitForExist({ timeout: 20000 }) + }) + + after(async () => { + await served?.close() + }) + + it('matches the full player', async () => { + await expect(browser).toMatchScreenSnapshot( + `player-${entry.id}`, + MISMATCH_TOLERANCE + ) + }) + + for (const panel of PANELS) { + it(`matches the ${panel.name} panel`, async () => { + if (panel.tab) { + await activateDockTab(panel.tab) + } + const el = await $(panel.selector) + await el.waitForExist({ timeout: 10000 }) + await expect(el).toMatchElementSnapshot( + `${entry.id}-${panel.name}`, + MISMATCH_TOLERANCE + ) + }) + } + }) +} diff --git a/test/visual/wdio.conf.ts b/test/visual/wdio.conf.ts new file mode 100644 index 00000000..0cf6476d --- /dev/null +++ b/test/visual/wdio.conf.ts @@ -0,0 +1,75 @@ +import path from 'node:path' +import { fileURLToPath } from 'node:url' + +import type { Options } from '@wdio/types' + +// Layer B โ€” visual regression. Drives the trace player served by +// serve-fixture.ts and snapshots each panel via @wdio/visual-service. +// Capabilities mirror examples/wdio/mocha/wdio.conf.ts so the player renders at +// the same viewport the goldens were captured against. +// +// Headless is the default so baselines render deterministically across machines +// and CI. Set HEADED=1 to watch the browser drive the player live โ€” but headed +// rendering can differ from the headless goldens, so use it to observe, not to +// re-seed baselines. + +const here = path.dirname(fileURLToPath(import.meta.url)) +const headed = process.env.HEADED === '1' || process.env.HEADED === 'true' + +export const config: Options.Testrunner = { + runner: 'local', + autoCompileOpts: { + autoCompile: true, + tsNodeOpts: { + transpileOnly: true + } + }, + specs: ['./*.e2e.ts'], + exclude: [], + maxInstances: 1, + capabilities: [ + { + browserName: 'chrome', + // Opt out of BiDi: the player's nested shadow DOM is queried via the `>>>` + // deep combinator, which routes through BiDi browsingContext.locateNodes โ€” + // that intermittently hangs ~60s (esp. headed) before falling back to + // Classic. Classic resolves the same elements reliably; screenshots are + // identical either way, so committed baselines are unaffected. + 'wdio:enforceWebDriverClassic': true, + 'goog:chromeOptions': { + args: [ + ...(headed ? [] : ['--headless']), + '--disable-gpu', + '--remote-allow-origins=*', + '--window-size=1600,900' + ] + } + } + ], + logLevel: 'warn', + bail: 0, + waitforTimeout: 10000, + connectionRetryTimeout: 120000, + connectionRetryCount: 3, + framework: 'mocha', + reporters: ['spec'], + services: [ + [ + 'visual', + { + baselineFolder: path.join(here, 'baseline'), + // Actual + diff images for a failing run; the committed goldens live in + // baselineFolder, so this stays out of git (see .gitignore/README). + screenshotPath: path.join(here, '.tmp'), + // First run has no baseline โ€” seed it instead of failing, so a fresh + // checkout produces the goldens to commit. + autoSaveBaseline: true, + formatImageName: '{tag}-{width}x{height}-dpr{dpr}' + } + ] + ], + mochaOpts: { + ui: 'bdd', + timeout: 60000 + } +} diff --git a/test/vitest.config.ts b/test/vitest.config.ts new file mode 100644 index 00000000..cfe100db --- /dev/null +++ b/test/vitest.config.ts @@ -0,0 +1,22 @@ +import path from 'node:path' +import { fileURLToPath } from 'node:url' + +import { defineConfig } from 'vitest/config' + +// Standalone from the root vitest project: the verification suite reads golden +// fixtures (produced by `pnpm fixtures:regen`), so it must stay out of +// `pnpm test` and the coverage gate. Run it with `pnpm verify`. +const repoRoot = path.resolve( + path.dirname(fileURLToPath(import.meta.url)), + '..' +) + +export default defineConfig({ + root: repoRoot, + test: { + globals: true, + environment: 'node', + include: ['test/**/*.test.ts'], + exclude: ['**/node_modules/**', '**/dist/**'] + } +})