Add repeatable desktop and mobile browser QA

CHANGELOG.md

@@ -0,0 +1,15 @@
+# Changelog
+
+## 0.8.0
+
+- Add declarative saved browser QA scenarios with isolated desktop/mobile
+  sessions, literal assertions, screenshots, and bounded error/request evidence.
+- Record exact Git HEAD and dirty state plus observed changes during each run;
+  expose typed local JSON summaries for Console and PR workflows.
+- Add scenario validation, bounded timeouts and cleanup, private output, and
+  real Chromium localhost regression coverage. Existing pane actions are unchanged.
+
+## 0.7.0
+
+- Shared agent-browser sessions, local Chromium launch, CDP attach, streaming,
+  failed-request visibility, and readiness/build diagnostics.

README.md

@@ -18,6 +18,9 @@ guides to this plugin and its three siblings (Guard, Swarm, Conductor).
 
 ## Highlights
 
+- **Repeatable QA scenarios** — run saved desktop/mobile checks in fresh browser
+  sessions and collect commit-bound screenshots, assertions, and error evidence.
+  See the [QA guide](docs/qa.md) (Browser 0.8.0).
 - **Shared agent sessions** — one isolated browser session per Herdr workspace.
 - **Attach to any CDP browser** — observe a Playwright, Puppeteer, or Browser Use
   run (or any Chrome started with `--remote-debugging-port`) without owning it.
@@ -46,7 +49,7 @@ guides to this plugin and its three siblings (Guard, Swarm, Conductor).
 | --- | --- | --- |
 | Herdr | `>= 0.7.0` | Tested with Herdr 0.7.4 |
 | Node.js | `>= 20` | Node 22+ enables live WebSocket streaming, CDP attach mode, and launch mode |
-| agent-browser | Optional | Required for shared agent sessions; tested with agent-browser 0.33.x; failed-request reporting needs the `network requests` command |
+| agent-browser | Optional | Required for shared agent sessions, recording, and saved QA scenarios; QA requires 0.33.0+; tested with 0.33.x |
 | Chromium/Chrome | Optional | Any Chromium-based browser enables launch mode (`l`) and attach mode |
 | chafa | Optional | ANSI rendering and streamed JPEGs in Kitty mode |
 | carbonyl | Optional | Only required for the separate interactive Browse action |
@@ -463,8 +466,9 @@ needed.
 
 ## Development
 
-Use Node 22+ for full browser support (Node 20 supports polling only), Python
-3.11+ for manifest validation, and ShellCheck for launcher validation. The
+Use Node 22+ for the pane's full streaming, CDP attach, and launch support.
+Node 20 supports pane polling and the standalone saved QA runner. Use Python
+3.11+ for manifest validation and ShellCheck for launcher validation. The
 plugin runs its source directly; there is no bundled browser or compilation
 step.
 

docs/qa.md

@@ -0,0 +1,157 @@
+# Repeatable browser QA
+
+Browser 0.8 adds saved desktop/mobile scenarios backed by the installed
+`agent-browser` CLI. Each viewport gets a new browser session; the runner
+collects assertion outcomes, viewport screenshots, console errors, page errors,
+and failed HTTP requests in a private evidence directory. No Herdr session is
+needed for this standalone command.
+
+## First run
+
+Requirements: Node.js 20+, Git with a committed SHA-1 HEAD, agent-browser
+0.33.0+, and its installed Chromium engine. Tested with agent-browser 0.33.2.
+
+```sh
+npm install -g agent-browser
+agent-browser install
+```
+
+Copy [the example scenario](../examples/qa.scenario.json) into the application
+repository as `.herdr-browser-qa.json`. Set the expected heading and any other
+checks to match the app. Commit the scenario and app changes to produce clean
+commit-bound evidence. Start the app's development server yourself; QA does not
+execute project scripts or start servers.
+
+From the Browser checkout:
+
+```sh
+npm run qa -- check --config /path/to/app/.herdr-browser-qa.json
+npm run qa -- run --config /path/to/app/.herdr-browser-qa.json \
+  --repo /path/to/app --base-url http://localhost:3000 \
+  --output /private/tmp/my-app-qa-run --json
+```
+
+The output directory must not exist yet, its parent must exist, and it must be
+outside the tested repository. Omit `--output` to create a private temporary
+directory. Each run prints its result location. `--json` prints the result as
+one JSON object, with an additional absolute `resultPath` for local consumers.
+The saved `result.json` contains the same evidence without that absolute path.
+Exit 0 means all requested viewports passed and cleanup succeeded; exit 1 means
+a failed run (including failed engine preflight); exit 2 means invalid command,
+configuration, repository, or output setup.
+
+## Scenario contract
+
+The JSON file requires `schemaVersion: 1`, a `name`, HTTP(S) `baseUrl`,
+`viewports`, and `steps`. Unknown fields and step types are refused. The first
+step must be `navigate`. A viewport is `{ "name": "desktop", "width": 1440,
+"height": 900 }`; names must be unique lowercase slugs. All steps run in order
+for every viewport; the first failed step stops that viewport, then the runner
+attempts a final screenshot, telemetry, and cleanup before the next viewport.
+
+| Step `type` | Other fields | Check or behavior |
+| --- | --- | --- |
+| `navigate` | `path` | Open `/path` on the configured base origin |
+| `click` | `selector` | Click a CSS/engine selector |
+| `fill` | `selector`, `value` | Replace an input's text with fixture data |
+| `waitFor` | `selector` | Wait for the selected element to be visible |
+| `assertVisible` | `selector` | Require a visible element |
+| `assertText` | `selector`, `contains` | Require an element's text to contain the literal text |
+| `assertTitle` | `contains` | Require the page title to contain literal text |
+| `assertUrl` | `contains` | Require the current URL to contain literal text |
+| `screenshot` | `name` | Save the current viewport as a named PNG |
+
+The runner always attempts a final viewport screenshot, including on assertion
+failure. Screenshots use CSS viewport sizes, not full-page captures. Mobile
+means a narrow viewport; it does not emulate touch input, device scale, mobile
+user agent, or a physical device. Screenshot capture alone does not establish
+visual correctness or accessibility compliance.
+
+Limits: 64 KiB config, 1–4 viewports, widths 320–1920, heights 240–1600,
+1–40 steps, and 10 MiB per screenshot. `timeoutMs` defaults to 10000 (range
+100–30000); `runTimeoutMs` defaults to 120000 (range 1000–300000), plus bounded
+cleanup. Each command's captured output is limited to 2 MiB; each telemetry
+category is limited to 100 entries. Exceeding a telemetry bound fails evidence
+collection rather than silently dropping observations. Assertions are immediate;
+use `waitFor` after transitions before asserting their result.
+The final JSON is capped at 1 MiB. If raw telemetry would exceed that cap, the
+runner removes it, marks the run incomplete and failed, and retains counts and
+artifact references. Configuration must be a regular file; symlinks, devices,
+and pipes are rejected. Git metadata ignores inherited repository-routing
+environment variables and disables fsmonitor/external diff helpers.
+
+`failOnConsoleError`, `failOnPageError`, and `failOnFailedRequest` default to
+`true`; an explicit `false` records that category without making it fail the
+run. HTTP 4xx/5xx responses are failed requests. Some engine versions expose
+no-response requests without distinguishing a transport failure from a still
+pending request; these are separately counted as `unresolvedRequests`, and also
+fail the run when `failOnFailedRequest` is true. Long-lived requests may need an
+explicit project decision about that setting. Network capture is an observation
+window, not proof of complete network coverage.
+
+No arbitrary JavaScript, `eval`, shell commands, uploaded files, persistent
+profiles, saved authentication, or browser attach settings are accepted in the
+scenario. CLI operands cannot begin with `-`. The engine uses an explicit empty
+config and does not inherit agent-browser sessions, providers, extensions,
+restore state, init scripts, or generic credential environment variables.
+`AGENT_BROWSER_EXECUTABLE_PATH` remains available to select local Chromium.
+
+## Evidence and Console/PR use
+
+`result.json` has `schemaVersion: 1`, `kind: "herdr-browser-qa"`, unique `runId`,
+`status` (`passed` or `failed`), start/end ISO timestamps, the resolved scenario
+name and SHA-256, engine version, and:
+
+- `scenario.policy`: the three resolved `failOn*` booleans. A passing run with
+  a relaxed policy means only its configured criteria passed. Strict handoff
+  consumers should require all three booleans to be `true` and zero error counts.
+- `git.commit`, `git.branch`, and `git.dirty`: the tested repository's observed
+  start state. Dirty runs are permitted but visibly labeled.
+- `git.changedDuringRun`: a comparison of start/end HEAD, branch, status, and
+  tracked diff. A detected change fails the overall result. This comparison
+  does not detect temporary changes reverted before completion or changes to
+  the contents of already-untracked files. Dirty evidence is not clean-commit
+  verification.
+- `summary`: requested `viewports`, `passed`/`failed` viewport counts, attempted
+  `assertions`, and `consoleErrors`, `pageErrors`, `failedRequests` counts.
+- `runs[]`: viewport geometry, status, indexed step outcomes, error details,
+  bounded telemetry, unresolved request count, and artifact metadata. Each
+  artifact records a relative path, byte count, and SHA-256.
+- `cleanup.status`: `passed` or `failed`. Failed cleanup fails the result.
+
+The Git record does **not** prove the served app was built from that commit.
+Run the server from the intended checkout and review its build/deployment
+provenance separately. Evidence is ordinary editable local JSON, not signed
+attestation, an approval receipt, or permission to merge/deploy.
+
+Console consumers may read the typed summary and link local artifacts. PR
+consumers should match `git.commit` to the proposed head and require
+`dirty === false`, `changedDuringRun === false`, passed status, complete
+viewport counts, and successful cleanup before describing a clean QA run.
+Never interpolate raw browser output into executable commands.
+
+The output directory is `0700`; result and screenshots are `0600`. Console/page
+messages and screenshots can contain sensitive app data. Network URLs omit
+credentials, query strings, and fragments, but paths and messages can still
+contain secrets. Review artifacts before sharing; do not commit them by default.
+Use fixture accounts and non-sensitive form values. Browser interactions can
+submit forms or navigate away through the app; this is not an OS sandbox or
+network containment boundary.
+
+Only the newly generated session is closed, never a shared or attached browser.
+SIGINT/SIGTERM request bounded cleanup and write failed evidence. A force kill
+or host crash cannot guarantee evidence publication or cleanup; sessions have a
+short idle timeout as a backstop.
+
+## Verification
+
+```sh
+npm run validate
+npm run test:qa
+```
+
+The opt-in QA integration test runs a localhost fixture in actual Chromium at
+desktop and mobile sizes, checks PNG dimensions, exercises fill/click/assertions,
+and proves error, HTTP failure, screenshot-on-failure, and cleanup behavior.
+The ordinary suite includes schema, output bounds, isolation, Git drift,
+failure/cleanup, and subprocess timeout tests without needing a browser engine.

examples/qa.scenario.json

@@ -0,0 +1,16 @@
+{
+  "schemaVersion": 1,
+  "name": "Local app smoke",
+  "baseUrl": "http://localhost:3000",
+  "viewports": [
+    { "name": "desktop", "width": 1440, "height": 900 },
+    { "name": "mobile", "width": 390, "height": 844 }
+  ],
+  "steps": [
+    { "type": "navigate", "path": "/" },
+    { "type": "waitFor", "selector": "h1" },
+    { "type": "assertVisible", "selector": "h1" },
+    { "type": "assertText", "selector": "h1", "contains": "Your app heading" },
+    { "type": "screenshot", "name": "home" }
+  ]
+}

herdr-plugin.toml

@@ -1,6 +1,6 @@
 id = "structupath.browser"
 name = "Browser"
-version = "0.7.0"
+version = "0.8.0"
 min_herdr_version = "0.7.0"
 description = "Driveable browser pane: live screenshots, console output, localhost links; drives agent-browser sessions, attaches to any CDP browser, or launches its own Chromium"
 platforms = ["macos", "linux"]

package.json

@@ -1,12 +1,14 @@
 {
   "name": "herdr-browser",
-  "version": "0.7.0",
+  "version": "0.8.0",
   "private": true,
   "type": "module",
   "engines": { "node": ">=20" },
   "scripts": {
     "build": "node scripts/build.mjs",
     "doctor": "node bin/doctor.mjs",
+    "qa": "node bin/qa.mjs",
+    "test:qa": "HERDR_BROWSER_REQUIRE_QA=1 node --test tests/qa.test.mjs tests/qa.integration.test.mjs",
     "test": "node --test tests/*.test.mjs",
     "test:integration": "HERDR_BROWSER_REQUIRE_INTEGRATION=1 node --test --test-name-pattern='end to end|e2e:' tests/launch.integration.test.mjs tests/renderer.test.mjs",
     "validate": "npm run build && shellcheck scripts/*.sh && npm test"

tests/manifest.test.mjs

@@ -57,8 +57,8 @@ test("release version and existing action IDs remain stable", () => {
 		path.join(root, "herdr-plugin.toml"),
 		"utf8",
 	);
-	assert.equal(packageJson.version, "0.7.0");
-	assert.match(manifest, /^version = "0\.7\.0"$/m);
+	assert.equal(packageJson.version, "0.8.0");
+	assert.match(manifest, /^version = "0\.8\.0"$/m);
 	assert.deepEqual(
 		[...manifest.matchAll(/^id = "([^"]+)"$/gm)]
 			.slice(1, 6)

tests/qa.integration.test.mjs

@@ -0,0 +1,78 @@
+import assert from "node:assert/strict";
+import fs from "node:fs";
+import http from "node:http";
+import os from "node:os";
+import path from "node:path";
+import { execFileSync } from "node:child_process";
+import test from "node:test";
+import { runQa } from "../bin/qa.mjs";
+
+test("e2e: saved QA runs desktop/mobile assertions and records real browser failures", {
+	skip: process.env.HERDR_BROWSER_REQUIRE_QA !== "1" ? "opt in with npm run test:qa (agent-browser + Chromium required)" : false,
+	timeout: 180_000,
+}, async (t) => {
+	const root = fs.mkdtempSync(path.join(os.tmpdir(), "hb-qa-e2e-"));
+	const repo = path.join(root, "repo");
+	fs.mkdirSync(repo);
+	const git = (...args) => execFileSync("git", ["-C", repo, ...args], { stdio: "ignore" });
+	git("init", "-q"); git("config", "user.email", "fixture@example.test"); git("config", "user.name", "Fixture");
+	fs.writeFileSync(path.join(repo, "README.md"), "Synthetic browser QA fixture\n");
+	git("add", "README.md"); git("-c", "commit.gpgsign=false", "commit", "-qm", "fixture");
+	const server = http.createServer((req, res) => {
+		if (req.url === "/failure") { res.writeHead(503); res.end("unavailable"); return; }
+		if (req.url === "/favicon.ico") { res.writeHead(204); res.end(); return; }
+		res.setHeader("Content-Type", "text/html");
+		res.end(`<!doctype html><meta name="viewport" content="width=device-width,initial-scale=1"><title>QA fixture</title>
+<style>body{font:20px sans-serif;margin:20px}input,button{font:inherit;max-width:100%}</style>
+<h1>Ready</h1><input id="name" aria-label="Name"><button id="submit">Confirm</button><p id="result">Waiting</p>
+<p id="size"></p><script>
+document.querySelector('#size').textContent=innerWidth < 600 ? 'mobile' : 'desktop';
+document.querySelector('#submit').onclick=()=>document.querySelector('#result').textContent='Hello '+document.querySelector('#name').value;
+${req.url === "/broken" ? `
+let failureSignals = 0;
+const markFailure = () => {
+  if (++failureSignals !== 2) return;
+  const done = document.createElement('p');
+  done.id = 'failure-observed';
+  done.textContent = 'failure observed';
+  document.body.append(done);
+};
+window.addEventListener('error', markFailure, { once: true });
+console.error('fixture console error');
+setTimeout(() => { throw new Error('fixture page error'); }, 0);
+fetch('/failure').then(markFailure);
+` : ""}
+</script>`);
+	});
+	await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve));
+	t.after(async () => { await new Promise((resolve) => server.close(resolve)); fs.rmSync(root, { recursive: true, force: true }); });
+	const config = path.join(root, "scenario.json");
+	const scenario = {
+		schemaVersion: 1, name: "localhost fixture", baseUrl: `http://127.0.0.1:${server.address().port}`,
+		viewports: [{ name: "desktop", width: 1440, height: 900 }, { name: "mobile", width: 390, height: 844 }],
+		steps: [{ type: "navigate", path: "/" }, { type: "assertVisible", selector: "h1" },
+			{ type: "assertTitle", contains: "QA fixture" }, { type: "fill", selector: "#name", value: "Fixture" },
+			{ type: "click", selector: "#submit" }, { type: "assertText", selector: "#result", contains: "Hello Fixture" }],
+	};
+	fs.writeFileSync(config, JSON.stringify(scenario));
+	const passed = await runQa({ config, repo, output: path.join(root, "passed") });
+	assert.equal(passed.status, "passed", JSON.stringify(passed));
+	assert.equal(passed.git.dirty, false);
+	assert.equal(passed.summary.assertions, 6);
+	for (const run of passed.runs) {
+		const image = fs.readFileSync(path.join(root, "passed", run.artifacts[0].path));
+		assert.equal(image.readUInt32BE(16), run.viewport.width);
+		assert.equal(image.readUInt32BE(20), run.viewport.height);
+	}
+	scenario.viewports = [scenario.viewports[0]];
+	scenario.steps = [{ type: "navigate", path: "/broken" }, { type: "waitFor", selector: "#failure-observed" }, { type: "assertText", selector: "h1", contains: "Deliberately absent" }];
+	fs.writeFileSync(config, JSON.stringify(scenario));
+	const failed = await runQa({ config, repo, output: path.join(root, "failed") });
+	assert.equal(failed.status, "failed");
+	assert.equal(failed.cleanup.status, "passed");
+	assert.equal(failed.runs[0].artifacts.length, 1);
+	assert.ok(failed.runs[0].steps.some((step) => step.status === "failed"));
+	assert.ok(failed.runs[0].consoleErrors.some((entry) => entry.message.includes("fixture console error")), JSON.stringify(failed));
+	assert.ok(failed.runs[0].pageErrors.some((entry) => entry.message.includes("fixture page error")), JSON.stringify(failed));
+	assert.ok(failed.runs[0].failedRequests.some((entry) => entry.status === 503), JSON.stringify(failed));
+});