Vv/test automation

.github/workflows/verify.yml

@@ -0,0 +1,40 @@
+# Cross-adapter capture-parity gate (Layer A). Runs the committed golden
+# fixtures through the reader and snapshots them — fast, no browser. Fixture
+# regeneration (`pnpm fixtures:regen`) needs a real browser and is a deliberate
+# maintainer/nightly step, not part of this gate.
+name: Verify
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+
+jobs:
+  verify:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+      - name: Use Node.js
+        uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # v6.2.0
+        with:
+          node-version-file: .nvmrc
+      - name: Cache pnpm modules
+        uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4
+        env:
+          cache-name: cache-pnpm-modules
+        with:
+          path: ~/.pnpm-store
+          key: ${{ runner.os }}-verify-${{ env.cache-name }}-${{ hashFiles('**/pnpm-lock.yaml') }}
+          restore-keys: |
+            ${{ runner.os }}-verify-${{ env.cache-name }}-
+      - uses: pnpm/action-setup@41ff72655975bd51cab0327fa583b6e92b6d3061 # v4.2.0
+        with:
+          version: 10.26.1
+          run_install: false
+      - name: Install dependencies
+        run: pnpm install --frozen-lockfile
+      - name: 🔍 Lint harness
+        run: pnpm exec eslint test
+      - name: 🧪 Cross-adapter capture parity
+        run: pnpm verify

CLAUDE.md

@@ -248,6 +248,7 @@ Documented divergences from the conventions above. They exist today as debt to b
 - Nightwatch native asserts are buffered at call time and emitted in one batch at test-end (Nightwatch exposes no per-assertion execution/outcome hook reachable from the plugin — `client.queue.tree` / `client.reporter` aren't on `browser`). Streaming mid-run would flash every assert green before its outcome is known, so rows are held and reconciled against `results.testcases[*].assertions` at `afterEach` (the flat `results.assertions` only reflects the last testcase). The exporter re-sorts commands by timestamp (`buildActionEvents`) so the batched rows land at their real timeline positions rather than clustering at flush time. `actual` is parsed from the failure message (`but got: …`, failures only) into a collapsed `{passed, expected, actual}` result; `expected` is the assertion arg — the label still mirrors the call args, not the derived values.
 - Nightwatch's BDD `describe/it` interface fires the plugin's global `beforeEach`/`afterEach` **once per module** (with an empty `currentTest.name`), not per `it` — Nightwatch runs the individual `it`s internally with no per-testcase hook or event reachable from the plugin (the lib emits only transport-level events). So `traceGranularity:'test'` records a single slice boundary (the first test) and collapses to one session-scoped slice; retry-aware/per-test retention (`tracePolicy`) degrades to session scope for this interface. Per-test slicing works for adapters/styles that expose per-test hooks (WDIO mocha/cucumber, Selenium mocha; likely Nightwatch's exports-object style, unverified). Session-granularity trace is unaffected. **No per-test support has been built for this interface** — real per-`it` slicing needs a hook Nightwatch doesn't surface; deferred to a full-picture pass, not attempted yet.
   - **Empirically confirmed** (BDD example, `mode:'trace'` + `traceGranularity:'test'` + `tracePolicy:'retain-on-failure'` + `screenshot:'on'` + `video:'retain-on-failure'`): the one collapsed slice is keyed to the **first** test's uid (the artifacts manifest's `tests[]` still carries every testcase with correct `state`, so metadata capture is fine — it's the *slice/artifact* keying that collapses). Consequence: with a passing first test and a failing later test, the screenshot **produces** (policy `'on'` captures regardless of outcome, keyed to the first test) but the **trace zip and video are dropped** — `retain-on-failure` evaluates the first (passing) test's outcome, and the actually-failing test never gets its own slice to retain. The per-test produce-only path itself (screenshot/video write + manifest) is correct; only the BDD slice-keying limits it.
+- **Nightwatch cucumber: no `trace.zip` is written in trace mode** (surfaced by the `test/` verification harness, 2026-07-27). Root cause is hook ordering, not granularity: the plugin's cucumber `After` hook runs at `order: -1` (deliberately *after* Nightwatch's browser-quit hook, so it can read final assertion status — see `helpers/cucumberHooks.cts`). By the time `finalizeCucumberScenario` runs its capture/`flushTestSlice`, the per-scenario browser session is already torn down, so `#flushSpecTrace`/`#writeTraceIfNeeded` bail on the absent `sessionCapturer.metadata?.sessionId` and nothing is written — under **either** `traceGranularity`. (`'session'` also has no single live session at run-end because cucumber uses one session per scenario.) Per-scenario metadata capture and **live mode are unaffected**; only the trace-mode zip export is missing. A proper fix must capture/write each scenario's trace **before the browser quits** (e.g. from the last `AfterStep` or a pre-quit hook), which needs a rework + browser-iterated verification. A runnable example exists at `examples/nightwatch/cucumber/` (live-capable); the harness keeps the `nightwatch-cucumber` entry `planned` until this lands.
 - Service renders expect-webdriverio matchers as single `expect.<matcher>` rows by **folding**, not stack/depth suppression (the old `#assertionDepth`/`#matcherStarted`/self-heal machinery is gone). The matcher's value-read (`toHaveText`→`getText`, `toExist`→`isExisting`, …) is captured as a normal command; `afterAssertion` then coalesces the synthesized `expect.*` row into that read in place — inheriting its callSource, screenshot, and timeline position — and the fold replaces **by timestamp, never a public `id`**: `id` is the per-worker `commandCounter`, which resets per spec, so stamping one lets the app's id-first `replaceCommand` swap a same-id row from another spec (duplicate rows + a fold from another spec vanishing, in multi-spec live mode). `beforeAssertion` arms the pending matcher (depth-counted so aliases like `toBeChecked`→`toBeSelected` fold once); a matcher that **hard-throws** — element never resolves, so expect-webdriverio's `waitUntil` rethrows and `afterAssertion` never fires — is synthesized at `afterTest`/`afterStep` from the throwing read, so a failing assertion renders as `expect.<matcher>` whether or not the element existed. Two limits: its error is then the read's (`Can't call getText on … element wasn't found`), not an assertion-phrased message; and `MATCHER_READ_COMMANDS` is a hand-maintained allowlist, so a matcher whose read isn't listed leaves its raw read visible alongside the `expect.*` row. Plain-value jest matchers (`expect(x).toBe(y)`) don't fire the ewdio hooks, so they aren't captured as rows.
 
 ### File-size (raw line counts; soft cap is 500 logic lines)

examples/nightwatch/cucumber/features/login.feature

@@ -0,0 +1,15 @@
+Feature: Example site smoke test
+
+  Mirrors the Nightwatch BDD smoke-test: open a site, wait for the body to be
+  visible, and assert the page title. Runs as a Cucumber scenario so the
+  cross-adapter harness has a nightwatch-cucumber trace fixture.
+
+  Scenario Outline: I can open a site and read its title
+    Given I navigate to "<url>"
+    When the page body becomes visible
+    Then the page title contains "<title>"
+
+    Examples:
+      | url                    | title   |
+      | https://example.com    | Example |
+      | https://example.org    | Example |

examples/nightwatch/cucumber/features/step_definitions/login.steps.js

@@ -0,0 +1,18 @@
+// Step definitions for the example-site smoke scenario.
+//
+// Nightwatch's Cucumber integration injects the Nightwatch `browser` onto the
+// World, so steps reach it via `this.browser` — hence regular (non-arrow)
+// functions so `this` binds to the World.
+const { Given, When, Then } = require('@cucumber/cucumber')
+
+Given(/^I navigate to "([^"]*)"$/, async function (url) {
+  await this.browser.url(url)
+})
+
+When(/^the page body becomes visible$/, async function () {
+  await this.browser.waitForElementVisible('body', 5000)
+})
+
+Then(/^the page title contains "([^"]*)"$/, async function (title) {
+  await this.browser.assert.titleContains(title)
+})

examples/nightwatch/cucumber/nightwatch.cucumber.conf.cjs

@@ -0,0 +1,69 @@
+// Simple import - just require the package
+const path = require('node:path')
+const nightwatchDevtools = require('@wdio/nightwatch-devtools').default
+const { cucumberHooksPath } = require('@wdio/nightwatch-devtools')
+
+const featuresDir = path.resolve(__dirname, 'features')
+
+module.exports = {
+  // Resolve relative to this config file so the paths hold regardless of CWD.
+  src_folders: [path.resolve(featuresDir, 'step_definitions')],
+  output_folder: false, // Skip generating nightwatch reports for this example
+  custom_commands_path: [],
+  custom_assertions_path: [],
+
+  test_runner: {
+    type: 'cucumber',
+    options: {
+      feature_path: featuresDir,
+      // Registers the DevTools Before/After/Step scenario hooks so the plugin
+      // sees each scenario as a test unit.
+      require: [cucumberHooksPath]
+    }
+  },
+
+  webdriver: {
+    start_process: true,
+    // server_path: '/opt/homebrew/bin/chromedriver',
+    port: 9515
+  },
+
+  test_settings: {
+    default: {
+      // Ensure all tests run even if one fails
+      skip_testcases_on_fail: false,
+
+      desiredCapabilities: {
+        browserName: 'chrome',
+        // Required for chromedriver to expose the BiDi WebSocket channel.
+        // Without this, attachBidiHandlers silently fails and the perf-log
+        // fallback takes over.
+        webSocketUrl: true,
+        'goog:chromeOptions': {
+          args: [
+            '--headless',
+            '--no-sandbox',
+            '--disable-dev-shm-usage',
+            '--window-size=1600,900'
+          ]
+        },
+        'goog:loggingPrefs': { performance: 'ALL' }
+      },
+      // bidi: opt-in WebDriver BiDi capture for console + network. When
+      // attached, the per-command Chrome perf-log network path is gated off to
+      // avoid duplicate entries.
+      globals: nightwatchDevtools({
+        port: 3000,
+        // Cucumber exposes per-scenario hooks, so each scenario is captured as
+        // its own test unit. Trace when the harness sets DEVTOOLS_MODE=trace,
+        // otherwise stream live to the backend/UI. NOTE: trace mode does not yet
+        // emit a zip here — the cucumber After hook runs after the browser
+        // session is torn down, so there's no live session to capture/write
+        // from. See CLAUDE.md known debt. Live mode works.
+        mode: process.env.DEVTOOLS_MODE === 'trace' ? 'trace' : 'live',
+        traceGranularity: 'session',
+        bidi: true
+      })
+    }
+  }
+}

examples/selenium/cucumber-test/features/support/setup.js

@@ -6,7 +6,10 @@
 
 import { DevTools } from '@wdio/selenium-devtools'
 
+// mode defaults to 'live' (the demo default); the verification harness sets
+// DEVTOOLS_MODE=trace to produce a golden fixture from this same example.
 DevTools.configure({
+  mode: process.env.DEVTOOLS_MODE === 'trace' ? 'trace' : 'live',
   screencast: { enabled: true, quality: 70, maxWidth: 1280, maxHeight: 720 },
   headless: true
 })

examples/wdio/mocha/wdio.trace.conf.ts

@@ -0,0 +1,53 @@
+import type { Options } from '@wdio/types'
+
+// Trace-mode variant of wdio.conf.ts, used by the verification harness
+// (`pnpm fixtures:regen`) to produce a deterministic golden trace.zip. Same
+// capabilities and specs as the live config; session granularity writes one zip
+// per run. Kept separate so the demo config's live default stays untouched.
+export const config: Options.Testrunner = {
+  runner: 'local',
+  autoCompileOpts: {
+    autoCompile: true,
+    tsNodeOpts: {
+      project: './tsconfig.json',
+      transpileOnly: true
+    }
+  },
+  specs: ['./specs/**/*.e2e.ts'],
+  exclude: [],
+  maxInstances: 1,
+  capabilities: [
+    {
+      browserName: 'chrome',
+      'goog:chromeOptions': {
+        args: [
+          '--headless',
+          '--disable-gpu',
+          '--remote-allow-origins=*',
+          '--window-size=1600,900'
+        ]
+      }
+    }
+  ],
+  logLevel: 'warn',
+  bail: 0,
+  baseUrl: 'http://localhost',
+  waitforTimeout: 10000,
+  connectionRetryTimeout: 120000,
+  connectionRetryCount: 3,
+  services: [
+    [
+      'devtools',
+      {
+        mode: 'trace' as const,
+        traceGranularity: 'session' as const
+      }
+    ]
+  ],
+  framework: 'mocha',
+  reporters: ['spec'],
+  mochaOpts: {
+    ui: 'bdd',
+    timeout: 60000
+  }
+}

package.json

@@ -19,6 +19,11 @@
     "test:coverage": "vitest run --coverage",
     "test:ui": "vitest --ui",
     "test:debug": "node --inspect-brk ./node_modules/.bin/vitest run",
+    "verify": "vitest run --config test/vitest.config.ts",
+    "verify:watch": "vitest --config test/vitest.config.ts",
+    "verify:update": "vitest run --config test/vitest.config.ts -u",
+    "verify:ui": "wdio run test/visual/wdio.conf.ts",
+    "fixtures:regen": "tsx test/regen.ts",
     "lint": "pnpm --parallel lint",
     "watch": "pnpm build --watch",
     "changeset:gen": "bash .changeset/changeset-gen.sh"
@@ -43,6 +48,7 @@
     "@vitest/browser": "^4.1.8",
     "@vitest/coverage-v8": "^4.1.8",
     "@wdio/allure-reporter": "^9.29.1",
+    "@wdio/visual-service": "^10.1.0",
     "autoprefixer": "^10.5.0",
     "eslint": "^10.4.1",
     "eslint-config-prettier": "^10.1.8",

packages/nightwatch-devtools/package.json

@@ -35,6 +35,7 @@
     "lint": "eslint .",
     "example": "nightwatch -c ../../examples/nightwatch/nightwatch.conf.cjs",
     "example:retry": "nightwatch -c ../../examples/nightwatch/nightwatch.conf.cjs --retries 1",
+    "example:cucumber": "nightwatch -c ../../examples/nightwatch/cucumber/nightwatch.cucumber.conf.cjs",
     "prepublishOnly": "pnpm build"
   },
   "keywords": [
@@ -59,6 +60,7 @@
     "yazl": "^2.5.1"
   },
   "devDependencies": {
+    "@cucumber/cucumber": "^13.0.0",
     "@types/node": "25.9.3",
     "@types/ws": "^8.18.1",
     "@wdio/devtools-core": "workspace:^",