test(expo): small fixes to the e2e tests and their config

.changeset/expo-verify-e2e-small-fixes.md

@@ -0,0 +1,2 @@
+---
+---

.claude/skills/verify-clerk-expo/SKILL.md

@@ -110,7 +110,7 @@ $ integration/expo-native/bin/control-clerk-expo run native-auth-view native-js-
 $ integration/expo-native/bin/control-clerk-expo screen --platform ios                        # the UI tree that is on screen now; --png adds a screenshot
 ```
 
-`run --all` runs every committed spec. `run` also takes `--grep <regex>`, `--retries <n>`, `--no-video`, and `--wait <seconds>`. `--retries <n>` runs a failed test again, up to `n` more times. The default is 0, so a run of your own change shows exactly what happened. The wait covers a free lane and another `run` in this worktree that holds the device. A spec limited to one platform with `test(title, { platforms: ['ios'] }, fn)` reports as skipped on the other.
+`run --all` runs every committed spec. `run` also takes `--grep <regex>`, `--retries <n>`, `--max-failures <n>`, `--no-video`, and `--wait <seconds>`. `--retries <n>` runs a test again, up to `n` more times, when an assertion failed, the test threw an error, or the test ran out of time. A test that e2e reports as a machine failure, such as an `ENGINE_FAILURE` when an `open` or a screenshot times out in agent-device, gets one attempt whatever `n` is. A driver error e2e reports as a test error, such as a stale node or an action that timed out, is retried like any other. The default is 0, so a run of your own change shows exactly what happened. `--max-failures <n>` stops a settings group once `n` of its tests have failed. A test that the limit stopped before it started prints `FAIL` with `not run: failure-limit`, and `run` exits 1. Without the flag, a run has no limit. The wait covers a free lane and another `run` in this worktree that holds the device. A spec limited to one platform with `test(title, { platforms: ['ios'] }, fn)` reports as skipped on the other.
 
 For a JS change on a local device, edit the source and `run` the spec, with no `up` in between. For a change to a native input, the same `run` rebuilds the dev client first. On a remote device, commit and push, then `run`.
 
@@ -139,7 +139,7 @@ test('the home UserButton opens the profile of the signed-in user', async ({ hos
 - `host.seedUser({ phone?, password? })` creates a `+clerk_test` user in the worktree's application. With `password: true` the user also has a password that the run generated for that user, and `user.password` holds it. Pass it to `host.fill`, as `host.fill(field, user.password!)`, and to nothing else. It prints as `<secret:password>`, `run` redacts its value from the logs it writes, and `attach` refuses a run whose files hold it. For a sign-up through the form, `host.newEmail()` returns a new address of this run, and `host.newPhone()` returns a 555-01xx number that no user holds. `host.newEmail()` calls nothing and records nothing. After the tests, `run` finds the user by the run's prefix in the address.
 - `host.launch({ signedInAs?, authMode?, keepStorage?, landsOn? })` relaunches the test app and waits up to 60 seconds for its home. Every launch opens on the home, and a spec taps from there to the screen it needs, as a user does. `authMode` is the mode of every AuthView the home opens: `signIn`, `signUp`, or `signInOrUp`. A launch starts with fresh storage unless `keepStorage` is true, so a launch with no `signedInAs` is signed out and lands on the home's `Sign in` button. `host.launch()` with no argument is that launch. A launch with `signedInAs` lands on the home, which must show that user's email and user ID. When the home should show anything else, pass its locator as `landsOn`. A launch that keeps storage and has no `signedInAs` needs it. `host.launch` fails at once when the test app shows its error screen, and the error has the message on that screen.
 - The home lists a link to each of the test app's other screens. Signed out, it has `Sign in` (`host.app.signIn`, AuthView in a modal), `Sign in with a logo` (the same modal with a React Native logo), `Sign in full screen` (`host.app.signInFullScreen`, AuthView with no close button), `Sign in full screen with a close button`, `Custom sign-in`, `Custom sign-up`, and `Native modules`. Signed in, it has `Embedded profile` (UserProfileView inline). `Token cache` shows in both states. `homeLinks(screen)` in `specs/native.ts` has the locator of each link that `host.app` does not hold. A new screen goes in `screens/destinations.ts` and `verify/VerifyHost.tsx` of the test app and in `homeLinks`.
-- `host.expectSignedInAs(user, timeoutMs?)` waits for the home to show `Signed in as <email>`, the user's ID, and a session ID. Pass the email alone for a user that the sign-up form created. `host.expectSignedOut(timeoutMs?)` waits for the home to show `Signed out` with no user ID and no session ID. Use them for the outcome of a flow that returns to the home. When one of these waits or the wait of `host.launch` runs out, its message says what the home showed at the last reading, how many passes and screen reads the wait made, how long the slowest read took, and the error of a read that failed.
+- `host.expectSignedInAs(user, timeoutMs?)` waits for the home to show `Signed in as <email>`, the user's ID, and a session ID. Pass the email alone for a user that the sign-up form created. `host.expectSignedOut(timeoutMs?)` waits for the home to show `Signed out` with no user ID and no session ID. Use them for the outcome of a flow that returns to the home. `host.expectVisible(locator, what, timeoutMs?)` is the same wait for an element of an SDK view, and its message names the element with `what`. On iOS, use it for the first element a spec waits for after it submits a password, because iOS may cover the app with its `Save Password?` sheet then. When one of these waits or the wait of `host.launch` runs out, its message says what the home showed at the last reading, how many passes and screen reads the wait made, how long the slowest read took, and the error of a read that failed.
 - `host.app` has the locators of the home: `signIn`, `signInFullScreen`, `signedOut`, `signedIn`, `userId`, `sessionId`, `signOut`, and `error` for the error screen. `host.runId` is the id of the run, for a password that must be new in each run.
 - `host.tap(locator)` is `locator.tap()` with the assertion timeout, and `host.fill(locator, text)` taps a field and types into it. When the first tap left no field focused, it taps the field once more before it types. Fill the text fields of the native views with `host.fill`. On iOS a native text field shows no text input until it has focus, so a plain `locator.fill()` on it fails with "no text input found at the provided coordinates to clear". After it types, `host.fill` reads the focused text input back. When the screen tree marks no text input as focused, as it does on iOS for a while after a slow snapshot, it reads the value that the node the locator names shows. It sends a long text sixteen characters to a command, because the iOS runner abandons a command after 30 seconds. If the input is still empty after two seconds and the screen shows no text that it did not show before the typing, `host.fill` taps it and types once more, and it fails with "the text never reached the field" if the input is empty again. A screen that shows new text has taken the typed text and moved on, as a code screen does when its code is complete, so `host.fill` types nothing more there. For this it compares the letters of the screen's text only, so the seconds that a code screen counts down beside `Resend` are not new text. If the input holds something other than the text, `host.fill` replaces the input's contents with the text once, and it fails with "the field does not hold the typed text" if the input is still wrong. It compares letters and digits only, so the formatting a field adds does not count. A password field withholds its value, so a password is typed once and is not confirmed. When the iOS runner refuses a tap, a typing command, or a screenshot because it is still finishing an earlier command, `host.tap`, `host.fill`, and `host.screenshot` wait five seconds and try the step again, up to six times, and then fail with the refusal. After a refused typing command, `host.fill` replaces the input's contents with the text and does not type that command again, so characters that did land are not typed twice. On iOS an empty React Native field of the test app reads as its placeholder, which `host.fill` treats as any other value that is not the text.
 - `host.screenshot(label)` writes `screenshots/<label>.png` in the run directory.
@@ -270,7 +270,7 @@ For a remote device, `down` ends the runner job and waits for it to finish, and
 - `src/host.ts`, `src/fixture.ts`, `src/native-build.ts`, and `src/freshness.ts` are this repository's own: the build of the test app, the native build that CI keeps between runs, the Metro ports, and the check that Metro serves current JS. `specs/app.ts` names the test app and its entry for a dev client, and `specs/native.ts` holds the per-platform locators for the native views and the locators of the home's links. Both are this repository's own too.
 - `npm test --prefix integration/expo-native` runs the CLI's unit tests, with no network, key, or device. `npm run typecheck --prefix integration/expo-native` runs `tsc`. The `Expo Native Runner Tests` job in `.github/workflows/ci.yml` runs both on Linux when a pull request changes the package, the test app, or a package the test app links.
 - `run --github-report` hands the results of the run to `@e2e-dev/github` as one report. The reporter writes the report to the job summary. With a `GITHUB_TOKEN` that may write pull request comments, it also posts one comment on the pull request and updates that comment on later runs. The reporter never changes the exit code, and nothing is reported for a run with a tainted file.
-- `.github/workflows/verify-e2e.yml`, the `Verify end-to-end tests` workflow, runs `up --backend local`, `run --all --retries 1 --github-report`, and `down` on a runner for each platform, with a device and a Clerk application for each. It starts on a pull request to `main` that changes the package, the test app, or one of the four packages the test app links. It runs for a pull request that is not a draft, and a draft or a pull request from a fork gets a notice instead. Start it by hand with `gh workflow run verify-e2e.yml --ref <branch>`. A failing spec shows on the pull request and is not required for a merge, and a test that fails and then passes on its one retry is `flaky` and does not fail the job. The runners are GitHub-hosted, `macos-26` and `ubuntu-24.04`, and the repository variables `VERIFY_CI_RUNNER_IOS` and `VERIFY_CI_RUNNER_ANDROID` name other labels. The Platform API key comes from the `MOBILE_VERIFICATION_PLATFORM_API_KEY` repository secret. The workflow sets `VERIFY_LOCAL_BUILD=standalone` and `VERIFY_NATIVE_CACHE` ([freshness.md](references/freshness.md)). It uploads each run's `run.json`, app log, driver logs, video, screenshots, and e2e's `report.json`, `junit.xml`, summary, and failure pages for three days, and only when no secret is found in the run. `bin/boot-ios-simulators.sh wait` waits until a booted simulator is ready, and this workflow and `.github/workflows/verify-remote.yml` call it.
+- `.github/workflows/verify-e2e.yml`, the `Verify end-to-end tests` workflow, runs `up --backend local`, `run --all --retries 1 --max-failures 5 --github-report`, and `down` on a runner for each platform, with a device and a Clerk application for each. It starts on a pull request to `main` that changes the package, the test app, or one of the four packages the test app links. It runs for a pull request that is not a draft, and a draft or a pull request from a fork gets a notice instead. Start it by hand with `gh workflow run verify-e2e.yml --ref <branch>`. A failing spec shows on the pull request and is not required for a merge, and a test that fails and then passes on its one retry is `flaky` and does not fail the job. The runners are GitHub-hosted, `macos-26` and `ubuntu-24.04`, and the repository variables `VERIFY_CI_RUNNER_IOS` and `VERIFY_CI_RUNNER_ANDROID` name other labels. The Platform API key comes from the `MOBILE_VERIFICATION_PLATFORM_API_KEY` repository secret. The workflow sets `VERIFY_LOCAL_BUILD=standalone` and `VERIFY_NATIVE_CACHE` ([freshness.md](references/freshness.md)). It uploads each run's `run.json`, app log, driver logs, video, screenshots, and e2e's `report.json`, `junit.xml`, summary, and failure pages for three days, and only when no secret is found in the run. `bin/boot-ios-simulators.sh wait` waits until a booted simulator is ready, and this workflow and `.github/workflows/verify-remote.yml` call it.
 - The `Expo` workflow, `.github/workflows/expo-native-build.yml`, runs nothing on a device. It builds the test app as a Release app on Expo SDK 54, 55, and 57 for each platform, and runs the Android unit tests of `@clerk/expo-biometrics`.
 - `.github/workflows/verify-attach.yml` publishes the evidence that `attach` hands to a session's runner, with `scripts/verify-attach.mjs`. It reads the manifest that `src/core/remote/handoff.ts` writes. It and `src/core/publish.ts` each replace the block the other wrote, each applies the same rules to the comments and to the start of the line, and each lays the files out in the same tables. The script is checked out alone, so it imports nothing from the package. So a change to the manifest, the comments, those rules, or the tables is a change in both places. `scripts/verify-attach.test.mjs` has its tests.
 - A remote session runs in `.github/workflows/verify-remote.yml`. `src/platform/session-device.ts` holds the steps that the session runs on its device, and `src/core/remote/` is the code on both ends of it. The runner installs the package with `npm ci`. A session keeps the code under `src/core/` from the commit it started on, so after a change to a file that `src/core/MANIFEST` lists, commit, push, `down`, then `up`.

.claude/skills/verify-clerk-expo/features/native-js-sync.md

@@ -21,9 +21,9 @@ Preconditions:
 
 - Each spec seeds a `+clerk_test` user. The sign-out spec starts signed in through a ticket, because it tests no sign-in. The sign-in spec starts signed out and signs in through the real form.
 
-- **Native sign-out.** Run `integration/expo-native/bin/control-clerk-expo run native-js-sync/sign-out-from-native --platform ios`, and again with `--platform android`. The spec launches signed in, taps the UserButton, taps `Sign out` in the profile, and calls `host.expectSignedOut`. That waits for the home to show `Signed out` (from `useAuth`), with no `Signed in as` text and no user ID (from `useUser`) and no session ID (from `useSession`). Screenshots `before-sign-out` and `signed-out`.
-- **Native sign-in.** Run `integration/expo-native/bin/control-clerk-expo run native-js-sync/sign-in-from-native --platform ios`, and again with `--platform android`. Both tests seed a user with `host.seedUser({ password: true })`, tap `Sign in` on the home, and sign in with `signInWithPassword` from `specs/native.ts`. The first test expects the home to show the user and a session ID, relaunches with `keepStorage`, and expects the same user and the same session ID. Screenshots `signed-in` and `restored-home`. The second test signs out with the profile's `Sign out` row, signs in again, and expects the home to show the user. Screenshot `signed-in-again`.
-- **Proof.** `screenshots/signed-out.png` shows the home with `Signed out` and the `Sign in` button. The native profile closes by itself, because the UserButton that presented it leaves the home.
+- **Native sign-out.** Run `integration/expo-native/bin/control-clerk-expo run native-js-sync/sign-out-from-native --platform ios`, and again with `--platform android`. The spec launches signed in, taps the UserButton, taps `Sign out` in the profile, and calls `host.expectSignedOut`. That waits for the home to show `Signed out` (from `useAuth`), with no `Signed in as` text and no user ID (from `useUser`) and no session ID (from `useSession`). Screenshots `before-sign-out` and `signed-out-from-native`.
+- **Native sign-in.** Run `integration/expo-native/bin/control-clerk-expo run native-js-sync/sign-in-from-native --platform ios`, and again with `--platform android`. Both tests seed a user with `host.seedUser({ password: true })`, tap `Sign in` on the home, and sign in with `signInWithPassword` from `specs/native.ts`. The first test expects the home to show the user and a session ID, relaunches with `keepStorage`, and expects the same user and the same session ID. Screenshots `signed-in-from-native` and `restored-home`. The second test signs out with the profile's `Sign out` row, signs in again, and expects the home to show the user. Screenshot `signed-in-again`.
+- **Proof.** `screenshots/signed-out-from-native.png` shows the home with `Signed out` and the `Sign in` button. The native profile closes by itself, because the UserButton that presented it leaves the home.
 
 ## Gotchas
 

.claude/skills/verify-clerk-expo/features/token-cache-persistence.md

@@ -18,10 +18,10 @@ Preconditions:
 
 - The spec seeds a `+clerk_test` user and signs in with a ticket on the first launch.
 
-- **Relaunch.** Run `integration/expo-native/bin/control-clerk-expo run token-cache-persistence --platform android` (or `ios`). The spec launches signed in and reads the session ID on the home. It relaunches with `keepStorage: true` and no ticket, and expects the home to show the same user and the same session ID. Screenshot `restored-home`.
+- **Relaunch.** Run `integration/expo-native/bin/control-clerk-expo run token-cache-persistence --platform android` (or `ios`). The spec launches signed in and reads the session ID on the home. It relaunches with `keepStorage: true` and no ticket, and expects the home to show the same user and the same session ID. Screenshot `restored-home-after-relaunch`.
 - **Kept token.** On that relaunched home the test taps `Token cache` and expects `client token kept from the last launch: yes` and `user: <the seeded user ID>`. Screenshot `token-cache`.
 - **New scope.** The same test relaunches with a new storage scope, taps `Token cache` on the signed-out home, and expects `client token kept from the last launch: no` and `user: none`. Screenshot `new-scope`.
-- **Proof.** `screenshots/restored-home.png` shows the user and the session after a launch that had no ticket, and `screenshots/token-cache.png` shows the kept token.
+- **Proof.** `screenshots/restored-home-after-relaunch.png` shows the user and the session after a launch that had no ticket, and `screenshots/token-cache.png` shows the kept token.
 
 ## Gotchas
 

.github/workflows/verify-e2e.yml

@@ -239,7 +239,7 @@ jobs:
           CLERK_PLATFORM_API_KEY: ${{ secrets.MOBILE_VERIFICATION_PLATFORM_API_KEY }}
           GITHUB_TOKEN: ${{ github.token }}
         run: |
-          "$CLI" run --all --platform "$PLATFORM" --backend local --retries 1 --github-report
+          "$CLI" run --all --platform "$PLATFORM" --backend local --retries 1 --max-failures 5 --github-report
 
       - name: Release the device and delete the Clerk application
         if: always() && steps.install.outcome == 'success'

integration/expo-native/specs/features/native-js-sync/sign-in-from-native.e2e.ts

@@ -8,7 +8,7 @@ test('a native password sign-in reaches the JS hooks and survives a relaunch', a
   await signInWithPassword(host, screen, platform, user);
   await host.expectSignedInAs(user, 45_000);
   const session = (await host.app.sessionId.textContent()) ?? '';
-  await host.screenshot('signed-in');
+  await host.screenshot('signed-in-from-native');
 
   await host.launch({ keepStorage: true, landsOn: host.app.signedIn });
   await host.expectSignedInAs(user);

integration/expo-native/specs/features/native-js-sync/sign-out-from-native.e2e.ts

@@ -10,5 +10,5 @@ test('signing out in the native profile signs out the JS hooks', async ({ host,
   await host.screenshot('before-sign-out');
   await host.tap(profile.signOut);
   await host.expectSignedOut(20_000);
-  await host.screenshot('signed-out');
+  await host.screenshot('signed-out-from-native');
 });

integration/expo-native/specs/features/token-cache-persistence/relaunch.e2e.ts

@@ -15,7 +15,7 @@ test('the token cache keeps the session across a relaunch, and a new scope start
   await host.launch({ keepStorage: true, landsOn: host.app.signedIn });
   await host.expectSignedInAs(user);
   await expect(host.app.sessionId).toHaveText(session);
-  await host.screenshot('restored-home');
+  await host.screenshot('restored-home-after-relaunch');
 
   await host.tap(tokenCache);
   await expect(clientToken).toHaveText('client token kept from the last launch: yes', { timeout: 15_000 });