v0.7: exact per-test reporting, element screenshots, iOS goBack, faster polling

.gitignore

@@ -4,3 +4,5 @@ dist/
 e2e/mobilewright-report/
 e2e/test-results/
 .DS_Store
+e2e/live-report.json
+e2e/live-html/

CHANGELOG.md

@@ -1,5 +1,41 @@
 # Changelog
 
+## 0.7.0 — unreleased
+
+### Added
+- **Exact per-test reporting.** The driver reads Playwright's `test.info()`
+  inside `connect()`/`disconnect()` (which run inside the test), so each
+  session's test and outcome are known exactly instead of reconstructed from
+  timestamps. With `sessionPerTest` each session is named after its test as it
+  starts (`describe › title (retry N)`) and gets its verdict when the test
+  ends; pooled sessions fail if any test on them failed. Every test in the
+  Playwright report links to its TestingBot session. The timing-based join
+  remains as the fallback.
+- **Element screenshots.** `locator.screenshot()` is cropped to the element
+  (points → pixels via the screen scale) and `screenshot({ scale })`
+  downscales; previously both returned the full screen.
+- **`screen.goBack()` on iOS** performs the system left-edge back swipe
+  instead of throwing.
+- **`snapshotTuning`** option, on by default: `waitForIdleTimeout: 0` (plus
+  `animationCoolOffTimeout: 0` on iOS) once per session. Measured on
+  production: fill + assert 7.3s → 3.5s on an Android emulator, 3.4s → 2.95s
+  on an iOS simulator. `false` keeps the server defaults.
+- **`visibility: 'bounds'`** option: judge `isVisible` by frame size, like
+  local mobilecli.
+- **`allocationRetryDelay`** option: a "no device available" error while the
+  run holds no session is retried with backoff (5s doubling to 30s) until
+  the pool gives up. mobilewright's pool only retries when one of its own
+  slots is released, so such a test previously idled until its timeout.
+
+### Fixed
+- `fill()` / `clearText()` on iOS now clear the focused field through its
+  element reference. XCUITest's page source has no focused flag, and React
+  Native fields put the caret at the start, so the old backspace fallback
+  never removed anything (`fill('7')` on a field holding `0` gave `70`).
+- Rotated `sessionPerTest` sessions now correct the test's `device.model` /
+  `device.osVersion` annotations, which described the pool slot's first
+  session.
+
 ## 0.6.0 — 2026-10-28
 
 Tracks `@mobilewright/protocol` 0.0.67.

README.md

@@ -93,9 +93,11 @@ Regexes and version ranges work on both real and virtual devices: real devices r
 
 ## Sessions, naming, and reporting
 
-- By default a device slot's TestingBot session is reused across tests (fast: you pay device startup once). Sessions that hosted exactly one test are named after that test with its own pass/fail; sessions that hosted several get a run summary (e.g. `12/12 tests passed`).
-- Set **`sessionPerTest: true`** for one session, video, and named verdict **per test** — the clearest dashboard, at the cost of device startup on every test.
-- Tests are matched to sessions by **worker identity first**: the driver stamps Playwright's worker index on each session as it records it, and the run report reports the same index per test, so a test that ran in another worker process is never a candidate. Timing separates the tests within one worker, using the worker's own timestamps from the report when it is available.
+- By default a device slot's TestingBot session is reused across tests (fast: you pay device startup once). Sessions that hosted exactly one test are named after that test with its own pass/fail; sessions that hosted several pass only if every test on them passed (e.g. `1 of 3 tests failed. checkout › pays: ...`).
+- Set **`sessionPerTest: true`** for one session, video, and named verdict **per test** — the clearest dashboard, at the cost of device startup on every test. The session is named after the test as soon as it starts (`describe › title`, plus `(retry N)` on retries), and its verdict is reported the moment the test ends, while the session is still alive.
+- Each test in Playwright's HTML report gets a **`TestingBot`** annotation linking to the session (and its video) that test ran on.
+- Tests are matched to sessions **exactly**: `connect()`/`disconnect()` run inside the test, so the driver reads Playwright's `test.info()` there and records which test attempt ran on which session, with its outcome. The heuristic below is only the fallback when `@playwright/test` cannot be resolved from the project.
+- Fallback: tests are matched to sessions by **worker identity first**: the driver stamps Playwright's worker index on each session as it records it, and the run report reports the same index per test, so a test that ran in another worker process is never a candidate. Timing separates the tests within one worker, using the worker's own timestamps from the report when it is available.
 - A session is only ever given a verdict the driver can stand behind: its own test's, or the run summary when it genuinely hosted several tests (or was the run's only session). When a session cannot be tied to a test — an ambiguous overlap under heavy parallelism, or a worker that died before recording its span — it is reported with build, extra and tags but **no** pass/fail, rather than inheriting a failure from a test that ran somewhere else. `DEBUG=testingbot:observer` names the session and the reason.
 - Git metadata is reported automatically when mobilewright captures it: the commit hash, branch, author and subject land in the session's **extra** field (alongside your own `extra`, never replacing it), and the branch is added as a **tag** so the dashboard can filter by it. Enable it with `captureGitInfo: { commit: true }` in `mobilewright.config.ts` — Playwright turns it on by itself in CI.
 - Failure messages are reported to the session's status message; `build` groups sessions on the dashboard and **defaults automatically from CI environments** (GitHub Actions, GitLab, CircleCI, Buildkite, Bitrise, Travis, Azure DevOps, Jenkins, TeamCity → e.g. `owner/repo #123`), or from a `TESTINGBOT_BUILD` variable.
@@ -115,6 +117,14 @@ new TestingBotDriver({
   build: 'ci-1234',           // default: auto-detected from CI env / TESTINGBOT_BUILD
   testResults: 'on',          // 'off' disables pass/fail reporting
 
+  snapshotTuning: {},         // Appium settings applied once per session; default turns off
+                              // the wait-for-idle before each hierarchy read (much faster
+                              // assertions while the UI animates). false = server defaults
+  visibility: 'native',       // 'bounds' = any element with a non-empty frame is visible
+                              // (mobilecli-like; helps iOS toBeVisible() on overlapped views)
+  allocationRetryDelay: 5_000,// first backoff (ms) when no device is free and the run holds
+                              // no session yet; doubles up to 30s until the pool gives up
+
   allocationTimeout: 395_000, // ms to wait for a device (TestingBot queues busy devices)
   idleTimeout: 230,           // seconds before TestingBot reaps an idle session
   maxDuration: 1800,          // max session length in seconds
@@ -246,6 +256,8 @@ jobs:
 | `cannot install "..." mid-session` | TestingBot has no mid-session installs — every app in `installApps` must also be listed in the driver's `apps` option (first entry = app under test, the rest install as helper apps). |
 | Modifier chords fail on iOS | XCUITest cannot hold modifier keys; chords are Android-only (`clearText()` handles iOS differently). |
 | Sessions show the run summary instead of a test name | The session hosted several tests; `sessionPerTest: true` gives each test its own session. |
+| An iOS project never gets a device in a mixed Android + iOS run | mobilewright's pool keeps idle slots of the other platform. Run each platform in its own `mobilewright test --project ...` invocation, or raise `workers`. |
+| `toBeVisible()` fails on iOS for an element that is on screen | XCUITest reports elements drawn under a sibling as invisible. Set `visibility: 'bounds'`. |
 | A session shows no pass/fail at all | The driver could not tell which test ran in it and declined to guess. Run with `DEBUG=testingbot:observer` — it logs the session id and why. |
 | `DEBUG=testingbot:*` | Prints every allocation, hub command, upload, and report the driver performs. |
 
@@ -265,11 +277,11 @@ jobs:
 
 ## Current limitations
 
-- One TestingBot session may host several mobilewright tests unless `sessionPerTest: true`; multi-test sessions report the run-level verdict.
-- The worker process that owns a session is never told which test it is running, so the session-to-test join is reconstructed after the run from Playwright's worker index plus timing. Without the run report (`jsonReport()`) it degrades to timing alone. Either way it is deliberately conservative and leaves a session unjudged rather than risk naming it after the wrong test.
+- One TestingBot session may host several mobilewright tests unless `sessionPerTest: true`; a multi-test session keeps one name and fails if any test on it failed.
+- **Run Android and iOS projects in separate invocations** (`--project android`, then `--project ios`) unless `workers` exceeds the number of projects. mobilewright's device pool never evicts an idle slot of one platform to make room for the other, so with every slot holding an Android session an iOS test waits until it times out.
 - Modifier key chords (`pressKeys(['ctrl+a'])`) work on Android only — XCUITest cannot hold modifier keys.
-- `pressButton` on iOS supports `HOME`, `VOLUME_UP`, `VOLUME_DOWN`; `listApps()` reports the foreground app only.
-- Screenshots are always PNG, and `screenshot({ clip })` is not honoured yet — the full screen comes back uncropped. `applyDeviceSettings` turns Android animations off best-effort via `mobile: shell` (a no-op on iOS).
+- `pressButton` on iOS supports `HOME`, `VOLUME_UP`, `VOLUME_DOWN`, and `BACK` (performed as the system left-edge swipe, so `screen.goBack()` works in navigation stacks); `listApps()` reports the foreground app only.
+- Screenshots are always PNG. `locator.screenshot()` is cropped to the element and `screenshot({ scale })` downscales. `applyDeviceSettings` turns Android animations off best-effort via `mobile: shell` (a no-op on iOS).
 - `setGeolocation` needs the platform driver's location extension to be available: emulators and simulators always are, real devices depend on the OS version TestingBot's Appium build supports (iOS 17+ for XCUITest).
 - `setFoldState` (protocol 0.0.62) throws: Appium has no hinge control for TestingBot's foldables.
 - Real iOS devices need a test-signed `.ipa`; simulator builds are rejected on real devices with a clear error.
@@ -285,6 +297,14 @@ npm run build
 
 Live e2e tests against a real TestingBot account live in `e2e/` (see `e2e/mobilewright.config.ts`).
 
+`e2e/live.config.ts` runs feature checks (hierarchy speed, fill, element screenshots, iOS back navigation, per-test reporting) against TestingBot's public sample app, so it needs no local builds:
+
+```bash
+cd e2e && npm install --no-save mobilewright@latest @mobilewright/test@latest
+TESTINGBOT_KEY=... TESTINGBOT_SECRET=... npx mobilewright test -c live.config.ts --project android-emulator
+# E2E_SESSION_PER_TEST=1, E2E_FORCE_FAIL=1, E2E_RETRIES=1, E2E_VISIBILITY=bounds, E2E_NO_TUNING=1 toggle scenarios
+```
+
 ### Releasing
 
 Update `CHANGELOG.md` first — it ships inside the published tarball.

e2e/live.config.ts

@@ -0,0 +1,33 @@
+import { defineConfig } from 'mobilewright';
+import { TestingBotDriver } from '@testingbot/mobilewright-driver';
+
+/**
+ * Production feature checks against TestingBot's public sample app (a React
+ * Native calculator). Needs TESTINGBOT_KEY / TESTINGBOT_SECRET.
+ *   npx mobilewright test -c live.config.ts --project android-emulator
+ */
+const ANDROID_APP = 'https://testingbot.com/appium/sample.apk';
+const IOS_SIM_APP = 'https://testingbot.com/appium/sample.zip';
+
+export default defineConfig({
+  testDir: './live',
+  timeout: 300_000,
+  retries: Number(process.env['E2E_RETRIES'] ?? 0),
+  workers: Number(process.env['E2E_WORKERS'] ?? 1),
+  use: { actionTimeout: 30_000 },
+  expect: { timeout: 30_000 },
+  driver: new TestingBotDriver({
+    build: `mobilewright-live-${new Date().toISOString().slice(0, 10)}`,
+    sessionPerTest: process.env['E2E_SESSION_PER_TEST'] === '1',
+    apps: { android: ANDROID_APP, 'ios-simulator': IOS_SIM_APP },
+    ...(process.env['E2E_NO_TUNING'] === '1' ? { snapshotTuning: false as const } : {}),
+    ...(process.env['E2E_VISIBILITY'] === 'bounds' ? { visibility: 'bounds' as const } : {}),
+  }),
+  projects: [
+    { name: 'android-emulator', use: { platform: 'android', deviceType: 'emulator', bundleId: 'com.sampleapp' } },
+    { name: 'android-real', use: { platform: 'android', deviceType: 'real', osVersion: '>=12', bundleId: 'com.sampleapp' } },
+    { name: 'android-galaxy', use: { platform: 'android', deviceType: 'real', deviceName: /^Galaxy S2[345]$/, bundleId: 'com.sampleapp' } },
+    { name: 'ios-simulator', use: { platform: 'ios', deviceType: 'simulator', bundleId: 'org.reactjs.native.example.SampleApp' } },
+  ],
+  reporter: [['list'], ['json', { outputFile: 'live-report.json' }], ['html', { outputFolder: 'live-html', open: 'never' }]],
+});

e2e/live/features.test.ts

@@ -0,0 +1,50 @@
+import { expect, test } from '@mobilewright/test';
+
+const pngSize = (b: Buffer) => ({ width: b.readUInt32BE(16), height: b.readUInt32BE(20) });
+
+test('hierarchy polling speed', async ({ device }) => {
+  const driver = device.driver;
+  await driver.getViewHierarchy(); // warm-up
+  const times: number[] = [];
+  for (let i = 0; i < 10; i++) {
+    const t = Date.now();
+    await driver.getViewHierarchy();
+    times.push(Date.now() - t);
+  }
+  times.sort((a, b) => a - b);
+  console.log(`[timing] tuning=${process.env['E2E_NO_TUNING'] === '1' ? 'off' : 'on'} median=${times[5]}ms min=${times[0]}ms max=${times[9]}ms all=${times.join(',')}`);
+});
+
+test('fill both inputs and read the sum', async ({ screen, platform }) => {
+  await screen.getByLabel('inputA').fill('12');
+  await screen.getByLabel('inputB').fill('30');
+  if (platform === 'android') await screen.goBack(); // tall keyboards hide the sum field
+  const sum = screen.getByLabel('sum');
+  if (platform === 'ios') await expect(screen.getByType('XCUIElementTypeTextField').nth(2)).toHaveValue('42');
+  else await expect(sum).toHaveText('42');
+});
+
+test('element screenshot is cropped to the element', async ({ device, screen }) => {
+  const full = await screen.screenshot();
+  const el = await screen.getByLabel('inputA').screenshot();
+  const size = await device.driver.getScreenSize();
+  const node = (await device.driver.getViewHierarchy());
+  const find = (ns: any[]): any => ns.flatMap((n) => [n, ...[find(n.children ?? [])].flat()]);
+  const target = find(node).filter(Boolean).find((n: any) => n.label === 'inputA');
+  console.log(`[clip] full=${JSON.stringify(pngSize(full))} element=${JSON.stringify(pngSize(el))} bounds=${JSON.stringify(target?.bounds)} scale=${size.scale}`);
+  expect(pngSize(el).width).toBeLessThan(pngSize(full).width / 2);
+  expect(Math.abs(pngSize(el).width - Math.round(target.bounds.width * size.scale))).toBeLessThanOrEqual(2);
+  expect(Math.abs(pngSize(el).height - Math.round(target.bounds.height * size.scale))).toBeLessThanOrEqual(2);
+  const half = await screen.screenshot({ scale: 0.5 });
+  expect(Math.abs(pngSize(half).width - pngSize(full).width / 2)).toBeLessThanOrEqual(1);
+});
+
+test('goBack navigates back on iOS (Settings app)', async ({ device, screen, platform }) => {
+  test.skip(platform !== 'ios', 'iOS-only: Android BACK is a key event');
+  await device.launchApp('com.apple.Preferences');
+  await screen.getByText('General', { exact: true }).tap();
+  await expect(screen.getByText('About', { exact: true })).toBeVisible();
+  await screen.goBack();
+  await expect(screen.getByText('About', { exact: true })).not.toBeVisible();
+  await expect(screen.getByText('General', { exact: true })).toBeVisible();
+});

e2e/live/fill-repeat.test.ts

@@ -0,0 +1,26 @@
+import { expect, test } from '@mobilewright/test';
+for (let i = 1; i <= 4; i++) {
+  test(`fill-repeat ${i}`, async ({ device, screen, platform }) => {
+    const t = Date.now();
+    await screen.getByLabel('inputA').fill(String(i * 3));
+    await screen.getByLabel('inputB').fill(String(i * 4));
+    // Tall keyboards (Samsung) cover the sum field, and Android drops covered
+    // views from the hierarchy: dismiss the keyboard before reading the result.
+    if (platform === 'android') await screen.goBack();
+    try {
+      if (platform === 'ios') await expect(screen.getByType('XCUIElementTypeTextField').nth(2)).toHaveValue(String(i * 7));
+      else await expect(screen.getByLabel('sum')).toHaveText(String(i * 7), { timeout: 8_000 });
+    } catch (err) {
+      const d: any = device.driver;
+      const flat = (ns: any[]): any[] => ns.flatMap((n) => [n, ...flat(n.children ?? [])]);
+      const nodes = flat(await d.getViewHierarchy());
+      const hits = nodes.filter((n) => /sum|Total/.test(`${n.label ?? ''}${n.text ?? ''}`));
+      const xml: string = await d.hub.source(d.currentSessionId());
+      console.log(`[fail] model=${test.info().annotations.find((a) => a.type === 'device.model')?.description} nodes=${nodes.length} hits=${JSON.stringify(hits.map((n) => ({ t: n.type, l: n.label, x: n.text, v: n.isVisible, b: n.bounds })))}`);
+      console.log(`[fail] raw sum: ${xml.split('\n').filter((l) => /sum/.test(l)).map((l) => l.trim().slice(0, 260)).join('\n')}`);
+      console.log(`[fail] top: ${xml.split('\n').slice(0, 6).map((l) => l.trim().slice(0, 200)).join('\n')}`);
+      throw err;
+    }
+    console.log(`[fill] tuning=${process.env['E2E_NO_TUNING'] === '1' ? 'off' : 'on'} ${Date.now() - t}ms`);
+  });
+}

e2e/live/report.test.ts

@@ -0,0 +1,11 @@
+import { expect, test } from '@mobilewright/test';
+
+test.describe('reporting', () => {
+  test('passes', async ({ screen }) => {
+    await expect(screen.getByLabel('inputA').first()).toBeVisible();
+  });
+  test('fails on purpose', async ({ screen }) => {
+    test.skip(process.env['E2E_FORCE_FAIL'] !== '1', 'only with E2E_FORCE_FAIL=1');
+    await expect(screen.getByText('This text does not exist')).toBeVisible({ timeout: 3_000 });
+  });
+});

src/actions.ts

@@ -75,6 +75,21 @@ export function swipeActions(
   ])];
 }
 
+/**
+ * iOS system back gesture: a quick swipe from the left screen edge to past the
+ * middle, at mid-height. UINavigationController's interactive pop recognizer
+ * needs the touch to start within a few points of the edge.
+ */
+export function edgeSwipeBackActions(screen: ScreenSize): PointerSource[] {
+  const y = screen.height / 2;
+  return [pointerSource('finger1', [
+    move(2, y),
+    down(),
+    move(screen.width * 0.6, y, 250),
+    up(),
+  ])];
+}
+
 /** Multi-finger gesture: each pointer path becomes one W3C input source,
  *  with `time` offsets converted to per-segment move durations. */
 export function gestureActions(gestures: GestureSequence): PointerSource[] {