From 5776f922bbb02e3e8fb2a614af9e0f2c5dbbd08c Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Mon, 9 Nov 2026 20:45:55 +0000 Subject: [PATCH] Bump the e2e runner to 0.23.0 Update e2e to 0.23.0, @e2e-dev/web to 0.15.5, @e2e-dev/github to 0.4.4, and @e2e-dev/decision to 0.3.2. Carry the startup-log patch onto 0.23.0 and refresh the upstream e2e skill with e2e init. Co-authored-by: Jonathan Beckman --- .agents/skills/e2e/SKILL.md | 272 ++++++--- .agents/skills/e2e/references/agent.md | 194 +++++-- .agents/skills/e2e/references/bug-bash.md | 30 +- .agents/skills/e2e/references/debugging.md | 107 +++- .agents/skills/e2e/references/explore.md | 38 +- .agents/skills/e2e/references/mcp.md | 266 +++++++-- .agents/skills/e2e/references/running.md | 154 ++++-- .agents/skills/e2e/references/setup.md | 157 ++++-- .../skills/e2e/references/writing-tests.md | 232 ++++++-- nub.lock | 514 ++++++++++-------- package.json | 10 +- .../{e2e@0.21.3.patch => e2e@0.23.0.patch} | 0 12 files changed, 1426 insertions(+), 548 deletions(-) rename patches/{e2e@0.21.3.patch => e2e@0.23.0.patch} (100%) diff --git a/.agents/skills/e2e/SKILL.md b/.agents/skills/e2e/SKILL.md index 0f1b7027..6218d343 100644 --- a/.agents/skills/e2e/SKILL.md +++ b/.agents/skills/e2e/SKILL.md @@ -1,24 +1,57 @@ --- name: e2e -description: Agentic end-to-end tests with e2e, the e2e runner. Covers scaffolding e2e.config.ts, picking the Playwright browser engine or the agent-device mobile engine, starting the app under test from the config, driving flows with agent.act, judging with agent.assert, agent.waitFor, and agent.extract, pinning values with screen, app, browser, and expect, shaping the agent (context, system prompt, tools, personas), the replay cache, the e2e CLI, reading .e2e/report.json, and bug bashes (parallel explore runs proven with repro tests). Use when a project depends on e2e, when asked for end-to-end, browser, mobile, or agentic UI tests, to bug bash or hunt for bugs, or when an e2e run fails. +description: Writes, runs, and debugs end-to-end tests for web and mobile apps with e2e, the agentic test runner, which mixes agent.act goals with exact locator assertions on a Playwright browser or an iOS or Android device. Also drives a project's running app live over the e2e MCP server to look at a screen, check a UI change, take a screenshot or record a video, and try a locator; and runs bug bashes, parallel e2e explore runs with each bug proven by a repro test. Use when a project depends on e2e or has an e2e.config.ts, when asked for end-to-end, browser, mobile, or agentic UI tests, to check a change in the running app, to QA or bug bash an app or branch, or when an e2e run fails. +metadata: + openclaw: + homepage: https://e2e.com + requires: + bins: + - node + - npx + envVars: + - name: AI_GATEWAY_API_KEY + required: false + description: Model key for agent steps when the project's e2e config uses the Vercel AI Gateway; other providers read their own key. --- -# e2e: agentic end-to-end tests in TypeScript +# e2e: agentic end-to-end tests -SolZero uses **Nub** for install and every e2e/verify command (`nub install`, -`nub run test`, `nub run test:e2e`, `nub exec e2e ...`). Project rules live in -`AGENTS.md`, `docs/e2e.md`, and `.agents/skills/verify-solzero/`. Prefer those -wrappers over raw `npx e2e` when working in this repository. - -e2e runs UI tests with agent goals and exact assertions. `agent.act` drives +e2e runs UI tests that mix agent goals with exact steps. `agent.act` drives one goal; `agent.assert`, `agent.waitFor`, and `agent.extract` judge the -screen. `screen`, `app`, `browser`, and `expect` make exact interactions and -checks. The replay cache reruns verified actions and checks their recorded end -state without a model call; agent judgments still run live. UI targets use -`@e2e-dev/web` for browsers or `@e2e-dev/mobile` for iOS simulators, -Android emulators, and connected phones. A test that takes only `app` can check an API with `fetch` -and `expect` (topic `writing-tests`). Model sign-in commands are in -[setup](references/setup.md#subscriptions-and-api-keys). +screen; `screen`, `app`, `browser`, and `expect` act and check exactly. UI +targets use `@e2e-dev/web` for browsers or `@e2e-dev/mobile` for iOS +simulators, Android emulators, and connected phones. + +## Start here: pick the reference for the job + +Read the one reference that matches the request before doing anything else. +Each stands on its own; a reference names another topic only for a side +trip. `npx e2e guide ` prints the installed package's version of a +reference; prefer it when it differs from this copy. +The Rules and Feedback sections below still apply. + +| When you are | Topic | Read | +| --- | --- | --- | +| Adding e2e to a project, writing `e2e.config.ts`, starting the app from the config, signing in to a model, setting up mobile targets | `setup` | [references/setup.md](references/setup.md) | +| Looking at the running app, checking a UI change, taking a screenshot or video, trying a locator, running a test line, or walking a flow while you write its test, over `e2e mcp` | `mcp` | [references/mcp.md](references/mcp.md) | +| Writing or fixing a test: fixtures, locators, actions, matchers, sign-in sessions, the `browser` fixture, API-only tests | `writing-tests` | [references/writing-tests.md](references/writing-tests.md) | +| Adding `agent.*` steps, picking a model, cutting cost, budgets, the replay cache | `agent` | [references/agent.md](references/agent.md) | +| Running tests: CLI flags, CI, reporters, trace pages, `report.json` for scripts, exit codes | `running` | [references/running.md](references/running.md) | +| Handling a failed run: error codes and their fixes, `--headed`, `--debug`, `--ai-trace` | `debugging` | [references/debugging.md](references/debugging.md) | +| Exploring an app toward a goal without a test file: `e2e explore`, its budgets and verdict | `explore` | [references/explore.md](references/explore.md) | +| Asked to bug bash, QA, or hunt for bugs across an app or a branch | `bug-bash` | [references/bug-bash.md](references/bug-bash.md) | + +For anything the references do not cover, the full documentation ships in +the `docs/` directory of the installed `e2e` package +(`node_modules/e2e/docs` in a single-package project); a link such as +`/reference/cli` is `docs/reference/cli.mdx`. Complete projects for Vite, +Next.js, Astro, Angular, Expo, SwiftUI, Jetpack Compose, Kotlin Multiplatform, +Flutter, and .NET MAUI are in https://github.com/tester-army/e2e/tree/main/examples. + +## Quick start + +A config with one web target and a model, and a test that drives one goal +and pins the outcome: ```ts // e2e.config.ts @@ -32,7 +65,7 @@ export default { engine: web(), app: { url: 'http://127.0.0.1:3000', - command: { executable: 'nub', args: ['run', 'dev'], log: '.e2e/logs/app.log' }, + command: { executable: 'pnpm', args: ['dev'], log: '.e2e/logs/app.log' }, }, }, ], @@ -59,51 +92,128 @@ test('a member upgrades to Pro', async ({ app, agent, screen, browser }) => { }); ``` -## Topics +## Workflow: writing a test -Read the topic for the job before writing code. The files sit next to this -one; the installed CLI prints the same text with `npx e2e guide ` -(`e2e guide` alone prints this page). For anything the topics do not cover, -the full documentation ships in the `docs/` directory of the installed `e2e` -package (`node_modules/e2e/docs` in a single-package project); a link such as -`/reference/cli` is `docs/reference/cli.mdx`. - -| Topic | File | Read it when | -| --- | --- | --- | -| `setup` | [references/setup.md](references/setup.md) | Adding e2e to a project, writing `e2e.config.ts`, starting the app from the config, mobile targets | -| `writing-tests` | [references/writing-tests.md](references/writing-tests.md) | Writing or fixing tests: fixtures, locators, actions, matchers, sign-in sessions, the `browser` fixture | -| `agent` | [references/agent.md](references/agent.md) | Adding `agent.*` steps, picking a model, cost and budgets, the replay cache | -| `running` | [references/running.md](references/running.md) | CLI flags, reporters, `.e2e/report.json`, exit codes, CI | -| `explore` | [references/explore.md](references/explore.md) | Exploring an app toward a goal without a test file: `e2e explore`, its budgets, verdict, and `run.explore` | -| `debugging` | [references/debugging.md](references/debugging.md) | A run failed: error codes and their fixes, `--headed`, `--debug`, `--ai-trace` | -| `mcp` | [references/mcp.md](references/mcp.md) | Driving the live app from a coding agent over MCP: `e2e mcp`, its tools, and the explore-then-write loop | -| `bug-bash` | [references/bug-bash.md](references/bug-bash.md) | Asked to bug bash, QA, or hunt for bugs across an app or a branch: parallel `e2e explore` charters, merging findings, proving each with a repro test | - -## Workflow - -1. Look at what exists: `e2e.config.ts` or `e2e.config.mts`, the `tests` glob +1. Look at what exists: `e2e.config.ts`, `.mts`, `.js`, or `.mjs`, the `tests` glob (default `tests/**/*.e2e.ts`), `e2e` in `package.json`. Nothing there: - follow `setup`. -2. Learn the screens before writing a test: routes, labels, roles, button - text. Semantic locators need the accessible names the app renders, so read - the components, open the page with `--headed`, or drive the live app over - the registered `e2e mcp` server (topic `mcp`): `open_session`, `observe`, - and `locate` show exact names and check a locator before you write it. -3. Write `tests/.e2e.ts`. Drive the flow with `agent.act`, one goal - per call, and pin each outcome right after with `expect` or `agent.assert`. + follow `setup` when the user has requested e2e (see Rules). +2. With a connected `e2e mcp` server (topic `mcp`), write the test while + you walk the flow, not from memory after the walk. A test written + afterwards fails on each name or step you got wrong, one run at a time. + Before `open_session`, create `tests/.e2e.ts`, opening on the + screen it covers (step 3). Then add each step to the file as you take it + in the session. An `act` goal goes in before you walk its path, and the + path's actions stay out of the test. A `screen` line is the `Test line:` + its action printed, copied as is. When the outcome shows, run its + `expect` with `run_code` and add it once it passes; a passing statement + also joins `test_lines`. The walk ends with the test written. Without a + server, learn the accessible names the app renders from the components or + a `--headed` run. +3. A step is an `agent.act` goal or `screen` lines. Drive the flow with + `agent.act`, one atomic goal per call, and pin each outcome right after + with `expect` or `agent.assert`. A test with three goals makes three + calls. Start the test on the screen it covers, since every rerun pays + for the steps before: in a browser a saved session instead of the + sign-in form and `app.open` on that screen's path, on a device + `device.openLink`, and data from the app's API (topic `writing-tests`). Exact values go through `screen`: a sign-in form in a setup test, a field that must receive one specific string, a count that must be one number. -4. Run one file: `npx e2e run tests/.e2e.ts`. Agent steps need a - model in the config and that provider's authentication (a saved - subscription login, an API key); a local endpoint may need none. Tests - without agent steps need no model. -5. Read the failure: the reporter prints the error code, message, and a code - frame; `.e2e/report.json` has every step and artifact path. Fix the - locator, the expectation, or the app. Never add a sleep. +4. Run only the test you are working on: + `npx e2e run tests/.e2e.ts:` (over MCP, + `run_tests {files: ["tests/.e2e.ts:"]}`, the flags as + camelCase fields such as `retries: 0`), the line its `test(` opens on, + with the flags in "Rerunning a test while you fix it" below. Agent + steps need a model in the config and that provider's authentication (a + saved subscription login, an API key); a local endpoint may need none. + Tests without agent steps need no model. +5. When it fails, fix it from what the run recorded (next section). + +## Workflow: fixing a failed test + +The run already saw every screen up to the failure. Fix from what it +recorded; never walk the flow again from the start over MCP just to see a +screen. + +1. Read the failure block in the terminal: the failing line with its code + frame, the message (an assertion's `expected` and `observed`), where the + screen was (`❯ at`: a URL on the web, the app and screen title on a + device), and the nodes nearest a failed locator (`❯ on screen`). Then + read the trace page it names (`❯ trace .e2e/results//trace.md`): + every step with what it did, what the app logged, the agent's turns, and + `## Screen at failure`, the screen one node per line (topic `debugging`). +2. Fix every line that evidence covers, not only the one that failed: the + screen at failure shows the names the rest of that screen's lines need. + Fix the locator, the expectation, or the app. Never add a sleep or a + fixed delay (`setTimeout`, `waitForTimeout`): wait for the state you + need with `agent.waitFor('')` instead (topic `agent`; it + needs a model). +3. A test stops at its first failed step (an `expect.soft` lets it go on), + and every wrong line after it costs another run. Before rerunning, check + the lines after it as far as you can: against the screen at failure, the + app source, or a live session. +4. For a screen the evidence and the source leave out, look live (topic + `mcp`). `open_session {test: "tests/.e2e.ts"}` (with `:` + of its `test(` call when the file holds several) runs the test and stops + on the live screen where it failed. Act out the lines after it there and + put the `Test line:` each action prints in place of a wrong one, so the + next rerun catches every mistake. Otherwise open or reuse a session, go + straight to that screen, then `observe` and `locate`, or `run_code` for + the fixed line and the ones after it. On the web, `navigate` to the path + of the `❯ at` URL or the screen's route, and tap only to reach a state no + URL opens. A device session starts on the app's launch screen: take the + shortest path from there. +5. Rerun only that test, the way the next section says, or in a new + session with `open_session {test}` (topic `mcp` says when a session + must close first). Repeat with the next failure. + +## Rerunning a test while you fix it + +Every run of a failing test waits out its failure, and every `agent.act` +that runs live spends model calls. Keep each rerun to the one test and the +lines you changed: + +- Run by `file:line`, not the file or the suite. After an edit above the + test, look its `test(` line up again: a stale line runs another test, or + fails with `NO_TESTS` listing the lines tests start at. +- Add `--action-timeout 5000`: a locator that matches nothing fails in 5 s + instead of the default 30 s. It also bounds each action inside an agent + step, so keep it longer than the slowest screen takes to load. A `screen` + call's own `{ timeout }` still wins. An `expect` already gives up after + 5 s by default (`assertionTimeout`). +- Keep the replay cache on. A failed run keeps the recording of every goal + whose check passed, and the next run replays those with no model call. + New, changed, and failed goals run live, and so does a replay the trace + page shows as handed off. `agent.assert`, `agent.waitFor`, and + `agent.extract` always call the model. Leave `--no-cache` off: it sends every goal to the model + again. Only a `read-write` cache records: the local default, but not with + `CI` set and no cache mode in the config, nor in strict mode. A stopped + run (Ctrl-C, a command your tool's timeout kills, `--max-duration`) saves + nothing, so give the command a timeout longer than the test. +- Don't rename the test, its `describe`, or its file, and don't reword a + goal, change its params, or change the agent's `context` or the test's + `agentContext` once a goal passes. Each is part of a recording's key, so + every goal it touches records again live. Fixing a locator or an + expectation keeps the recordings. +- Add `--retries 0` when retries are on (`retries` in the config, or `CI` + set, where the default is 1); a `retries` option on the test or its + `describe` wins over the flag. Outside strict mode (`--strict-cache`, `cache.strict`), a + retry never replays: it runs every goal live and waits out the failure + again. +- Once the test passes, drop `--action-timeout` and run it one time with + `--no-cache`: every goal runs live at the config's timeouts, which shows + the agent still reaches it without its recordings. ## Rules -- In this repository run tests through Nub only: `nub run test`, `nub run test:e2e -- tests/.e2e.ts`, or `nub exec e2e run ...`. +- In a project without e2e, set it up when the user requests it. Otherwise + ask before adding a test framework, and name any existing framework. +- Live MCP tools need the project's own `e2e mcp` server registered in the + client (topic `mcp`). Tests run with the CLI, or with the server's + `run_tests`, the same run without a shell. + Keep a session open across runs while you write or fix a web test; topic + `mcp` says when a run can go beside it, and what to do on + `APP_ALREADY_RUNNING`. +- Run the CLI as `npx e2e ...` (or `pnpm exec e2e ...`). - The config is `export default { ... } satisfies E2EConfig` with `import type { E2EConfig } from 'e2e'`. `targets` is required; a UI target names an engine and declares the app beside it: `{ engine: web(), app: { url, command } }`. @@ -114,9 +224,10 @@ package (`node_modules/e2e/docs` in a single-package project); a link such as typed with `browser`. - Config and tests are ES modules whatever `package.json` sets as `type`. - Locators resolve when used. Actions wait for readiness and `expect` retries - assertions. Reads such as `textContent()` fail at once on zero matches and - `count()` answers from the current screen; nothing waits for a value to - change, so use a matcher when a value has to settle. + assertions. Reads such as `textContent()` wait for one match until the + action timeout, then answer the current value; `count()` and `isVisible()` + answer from the current screen. Nothing waits for a value to change, so use + a matcher when a value has to settle. - A locator that matches two nodes fails with `LOCATOR_AMBIGUOUS`; narrow it (topic `writing-tests`). - Secrets never appear in test code. Declare accounts under `credentials` and @@ -125,33 +236,54 @@ package (`node_modules/e2e/docs` in a single-package project); a link such as namespaces: `secrets.get` never returns a password), and hand the opaque `Secret` only to `fill()` or `agent.act` params. - Agent instructions: one goal per `act`, the wording on screen, real values - in params. Judge meaning, not phrasing: `toContain('Pro')`, not an exact - sentence a model produced. + in params. A goal is atomic: one outcome that one check confirms. Two + outcomes worth checking are two goals, so split them into two `act` calls + (topic `agent`). Bad: `'add a todo and mark it done'`. Good: `'add a + todo'`, check, then `'mark the todo as done'`, check. Judge meaning, not + phrasing: `toContain('Pro')`, not an exact sentence a model produced. - Check each agent goal's outcome. A passing `act` with a recorded check can be cached and replayed without model calls (topic `agent`). - Shape the agent for this app: `context` for vocabulary the screens use, `system` for how it works, tools for a test API, named personas under `agents`. When a step fails, tighten the goal first, then the context, then the agent. -- `.e2e/` is output (`report.json`, `artifacts/`, `cache/`, `logs/`; the +- `.e2e/` is output (`report.json`, `results/`, `cache/`, `logs/`; the config's `output` moves the report and artifacts, never `cache/` or the app's log). Read it, never edit it. ## Feedback -When e2e itself gets in your way, tell the e2e team: a command or API that -broke (`bug`), docs or this skill that misled you (`docs`), or a capability -you needed and did not find (`feature`). Send it once per problem, after you -worked around it or gave up, never for failures of the app under test. +When e2e itself gets in your way, offer to tell the e2e team: a command or +API that broke (`bug`), docs or this skill that misled you (`docs`), or a +capability you needed and did not find (`feature`). Once per problem, after +you worked around it or gave up. Never report: + +- failures of the app under test, or its own behavior; +- a problem you saw once and could not see again; +- anything on an outdated e2e: the command refuses it, so upgrade + (`npx e2e@latest upgrade`) and check the problem still happens first. + +Search [existing issues](https://github.com/tester-army/e2e/issues) first +(`gh issue list -R tester-army/e2e --search ""`); comment on a match +instead of sending a new report. + +A `bug` needs `--repro`, steps that reproduce it every time without app +content. Without such steps, or when the bug needs your app to show, omit +`--repro`: the command then prints a prefilled GitHub issue and sends +nothing; show the user the link and tell them to add a minimal reproduction +and submit it. Never file the issue yourself. + +Run it with `--dry-run` first, show the user what would be sent, and send +it only when they agree. ```bash npx e2e feedback --type bug -m "" \ - --task "" --expected "<...>" --actual "" \ + --repro "" --error-code LOCATOR_AMBIGUOUS \ + --task "" --expected "<...>" --actual "" \ --approach "" --command "" --agent "" ``` -Describe e2e's behavior only: never paste app content, page text, test files, -URLs of private apps, or credentials. Secret-named environment variables and -common token shapes are redacted, but do not rely on it. `--dry-run` prints -what would be sent. Tell the user you sent it and give them the reference id -it prints. +Describe e2e's behavior only, briefly: never paste app content, page text, +test files, URLs of private apps, or credentials. Secret-named environment +variables and common token shapes are redacted, but do not rely on it. After +sending, give the user the reference id it prints. diff --git a/.agents/skills/e2e/references/agent.md b/.agents/skills/e2e/references/agent.md index 12874049..21e3d6c3 100644 --- a/.agents/skills/e2e/references/agent.md +++ b/.agents/skills/e2e/references/agent.md @@ -4,6 +4,20 @@ Each call is one bounded invocation: fresh redacted observation, deadline, model-call budget, no shared transcript. No agent step, no model calls. +## Contents + +- Configure a model +- act: one goal +- assert, waitFor, extract: one question +- Write instructions the model can execute +- What the model sees +- Budgets and cost +- The replay cache +- Inspect what the model did +- Make the agent yours +- Beyond the built-in agent +- In CI + ## Configure a model Put an AI SDK model under `agents.default`. You can use all AI SDK providers @@ -25,9 +39,24 @@ export default { } satisfies E2EConfig; ``` -Subscription logins and local models: -[setup](setup.md#subscriptions-and-api-keys). Keep `ai@^7` installed with -any provider. +Subscription logins and local models: topic `setup`, section "Subscriptions +and API keys". Keep `ai@^7` installed with any provider. + +Decision models pick actions from choices the executor builds: install +`@e2e-dev/decision` and put `decisionExecutor({ model: +typeSafeAi.decisionModel('jev-latest'), textModel: openrouter('inception/mercury-2.5') })` +under `executor`. `openai.decisionModel('gpt-6-luna')` from `@ai-sdk/openai` +4.0.90 or later works too and takes the `vision: true` screenshot. +Tests stay plain language with no params. The text model writes field values +and upload paths. See the shipped `docs/decision-models.mdx` or +[the online guide](https://e2e.com/docs/decision-models) for setup, +gates, and limits. + +A coding agent the user already signs in to runs agent steps over the Agent Client Protocol: +install `@e2e-dev/acp` and its adapter, and put +`claudeCode({ model: 'sonnet' })` or `codex()` under +`executor`; no model provider is needed. See the shipped `docs/subscriptions.mdx` or +[the online guide](https://e2e.com/docs/subscriptions#use-your-coding-agent). - Pass a model instance, not a string (`INVALID_CONFIG`). - An agents entry is one plain object of `model`, `judge`, `system`, @@ -36,13 +65,17 @@ any provider. for a custom brain; none inherits `model` or `context` from `default`. - `model` drives `agent.act`. Judgments use `judge` when set, else `model`. - A missing model for the built-in agent raises one run-level - `MODEL_UNAVAILABLE` when the first test acquires `agent`, exit 2; auth - failures surface on the first model call as `MODEL_PROVIDER_FAILED`, or as - `CONFIG_LOAD_FAILED` for a provider that reads its key when the config loads. + `MODEL_UNAVAILABLE` when the first test acquires `agent`, exit 2, and so + does a model GitHub Copilot does not serve to the account, at the first + call; auth failures surface on the first model call as + `MODEL_PROVIDER_FAILED`, or as `CONFIG_LOAD_FAILED` for a provider that + reads its key when the config loads. - `context` is what the app calls things, sent to every model call, judges included; `system` is how the acting agent works, read only by the act loop. Both live on the agents entry; `agentContext` on a test or group - adds more. + adds more. Under `e2e run`, every prompt opens with the test's title path + (`Test: billing > upgrades the tier`), judges included, so name a test + for what it covers, not for the verdict it expects. ### Choose an agent @@ -62,9 +95,12 @@ context, so each persona records and replays its own steps. ## act: one goal ```ts -import { credentials } from 'e2e'; +import { credentials, expect } from 'e2e'; -await agent.act('add a todo named "Buy milk" and mark it done'); +await agent.act('add a todo named "Buy milk"'); +await expect(screen.getByRole('listitem', 'Buy milk')).toBeVisible(); +await agent.act('mark the "Buy milk" todo as done'); +await expect(screen.getByRole('checkbox', 'Buy milk')).toBeChecked(); await agent.act('invite {email} as an editor', { params: { email: 'ada@example.test' } }); const member = credentials.user('member'); @@ -85,7 +121,8 @@ why: `ACTION_FAILED` (product failure); `STEP_BUDGET_EXHAUSTED`, `SECRET_UNAVAILABLE`, `ENVIRONMENT_UNAVAILABLE`, `SEED_DATA_MISSING`, `TEST_SETUP_FAILED`, `AUTOMATION_UNSUPPORTED`, `POLICY_DENIED` (blocked from outside); -`MODEL_OUTPUT_INVALID` (unusable answer). Full list: topic debugging. +`MODEL_OUTPUT_INVALID` (unusable answer), `MODEL_REFUSED` (the model refused +the prompt). Full list: topic debugging. Options: @@ -113,25 +150,31 @@ The tools, one per engine action. Name the target as the screen names it - `scroll`: viewport or one scrollable node, a screen or a few. - `scroll_to`: a listed node into view, or by `text` page a list to a row. - `drag`: one node onto another. -- `upload`: project-root files to a file input; outside it or hidden (`.env`) - is `POLICY_DENIED`. +- `upload`: project-root files to a file input or a visible control that opens a + native file chooser; outside it or hidden (`.env`) is `POLICY_DENIED`. - `navigate`: a URL or app-relative path. - `back`: browser history or in-app back. +- `resize_viewport`: set the viewport size (200 to 4096 CSS px) to check a + layout at phone, tablet, or desktop width. Engines with `session.resize`, + such as web. - `type_secret`: a declared secret by name; plaintext never reaches the model. Offered only when the step declares secrets and the engine can fill them. - `screenshot`: attach viewport pixels; every later result then carries one. - `tap_at`/`hover_at`/`press_at`/`select_at`/`type_at`: a screenshot point. +- `drag_at`: from one screenshot point to another (`to_x`, `to_y`). - `dismiss_keyboard` (device): hide the on-screen keyboard. The focused field's selected text is listed as `selection="..."` (never for a secret), so a repeated `Shift+ArrowLeft` selects one word with visible feedback. Point tools serve a canvas shape, map pin, image region, or system -sheet control, hit-testing the tree first so a listed control underneath is -acted on by id; `type_at` on nothing listed taps, then types. A screen with +sheet control. `tap_at`, `hover_at`, and `drag_at` act at the point +(`drag_at` by id between two different listed controls on an engine without +a bare-point drag); `press_at`, `select_at`, and `type_at` hit-test the +tree first so a listed control underneath is acted on by id, and `type_at` +on nothing listed taps, then types. A screen with nothing to tap by id opens with a screenshot attached. A device has no -`right_click`, `select`, `upload`, or `scroll_to` by node id. Pixels are -masked and withheld after a secret fill (topic writing-tests): act on pixels -before signing in, or in a test of its own. A result says when an action +`right_click`, `select`, `upload`, or `scroll_to` by node id. Pixels mask +secure fields only (topic writing-tests). A result says when an action closed an on-screen keyboard; on a touch screen that tap was often spent closing it, so act again. @@ -159,11 +202,36 @@ expect(data.titles).toContain('Buy milk'); screen first and ask about what is visible. Judged from the tree alone, it adds `pass vision: true when the answer is in pixels`, as does `waitFor`'s timeout after an inconclusive round. -- Judgments see the assertion and the current screen only, never prior - steps or the act loop's summaries; malformed output gets one repair - round, then `MODEL_OUTPUT_INVALID`. +- Judgments see the assertion, its params, the agent's context, the test + title, and the current screen, never prior steps or the act loop's + summaries; malformed output gets one repair round, then + `MODEL_OUTPUT_INVALID`. +- `assert` and `waitFor` take `params` like `act`: + `assert('the table lists {name}', { params: { name } })`. The judge sees a + `Secret` by name only; check a secret's value with a locator assertion. +- A value shown in more than one place (a total in the summary and on the + pay button) must agree everywhere, or the judgment fails. Name the one + you mean (`'the order summary total is $42.00'`) when only it matters. - `waitFor` observes every `interval` (default 3 s), judges only when the screen changed, and is `STEP_TIMEOUT` after `timeout` (default 30 s). +- Never sleep for a fixed time to let the app settle. A sleep is too short + on a slow run and wasted time on a fast one; `waitFor` returns as soon as + the state appears: + +```ts +// BAD: guesses how long the export takes. +await new Promise((resolve) => setTimeout(resolve, 10_000)); + +// GOOD: waits for the state, up to the timeout. +await agent.waitFor('the export finished and a download link appeared', { timeout: 60_000 }); +``` + +- `waitFor` needs a model even under a custom executor and is never cached, + so each run spends judgments on it. It judges again on every screen + change, up to `maxModelCalls` (25): for a long wait on a screen that keeps + changing (a progress counter), raise `interval`. The test timeout still + caps its `timeout`. + - `extract` takes any Standard Schema validator (zod works); the model sees the schema's shape, never its value rules (`min`, `max`, lengths, patterns), which check what it read. Data the screen does not show is @@ -171,6 +239,16 @@ expect(data.titles).toContain('Buy milk'); accept absence, ask for it (`'the phone, or null when none is shown'` with `.nullable()`). +For a debug view that pairs each action with the screen it started on, give +`agent.act` an `onBeforeAction` hook that calls `app.screenshot`, and match +each capture to `onAction` by `index`, which restarts at 1 in each `act` call. +It runs before replayed actions too, does not wait for the screen to settle, +and a failed action gets a capture but no `onAction`. +Narrow `onAction`'s event on `action` for coordinates, replay included: +`point` for `tapAt`/`hoverAt`, `from`/`to` for `dragAt`. They are in the +same space as `screen.tapAt` and `boundingBox()`, rounded and clamped to the +viewport. Mobile engines have no `hoverAt` or `dragAt`. + `vision` on a judgment picks the evidence: `false` (default) the tree; `true` the tree plus a masked screenshot; `'only'` the screenshot alone, for what the screen presents (an overlay, a broken layout, a chart) where the @@ -180,6 +258,28 @@ masked pixels fail with `POLICY_DENIED`. ## Write instructions the model can execute - One goal per `act`; goal order is the test's, the path the model's. +- Keep each goal atomic: one outcome that one check confirms. When two + parts of an instruction each leave a state worth checking, they are two + goals: give each its own `act` and check. Steps toward a single outcome + are the path, not more goals: scrolling to a control, closing a banner, + opening the menu that holds it, filling the fields of one form. +- A split flow fails at the goal that broke. Each goal replays from its own + recording and gets its own `maxSteps` and `maxModelCalls`; the test + timeout still caps them all. + +```ts +// BAD: three goals in one instruction, one recording, no check between them. +await agent.act('open the Pro plan, set the seats to {seats}, and check out', { params: { seats: 3 } }); + +// GOOD: three atomic goals, each pinned before the next. +await agent.act('open the Pro plan'); +await expect(screen.getByRole('heading', 'Pro plan')).toBeVisible(); +await agent.act('set the seats to {seats}', { params: { seats: 3 } }); +await expect(screen.getByRole('status', 'Seats')).toHaveText('3 seats'); +await agent.act('check out'); +await expect(screen.getByText('Order confirmed')).toBeVisible(); +``` + - Use the words on screen: `'open the Billing tab'`, not `'upgrade'`. - Values go in `params`, never expanded: `act('rename to {name}', { params })`. - Do not describe mechanics the runner handles: waiting, scrolling, retries. @@ -203,10 +303,14 @@ values, or a `Secret`'s value; password fields masked. The first screen of a step arrives whole; later action results report what changed, keyed by node ids stable while an element exists, or the whole screen when most changed. Pixels arrive through `vision` on a judgment or the act loop's -`screenshot` and point tools, masked and withheld after a secret fill. When +`screenshot` and point tools, with secure fields masked. When the browser engine's tree capture times out, the model gets a screenshot and a warning, a judgment needs `vision: true`, and no control may be inferred -absent nor old node ids reused. Nothing the model returns runs as code or +absent nor old node ids reused. The browser engine's tree shows a frame +from another site (a hosted card field) as its `iframe` node with none of +its content, so the agent reaches it only through pixels: script steps +there with `browser.frameLocator`, and judge it with `vision`. Nothing the +model returns runs as code or selectors: the runner validates and authorizes every tool call first. ## Budgets and cost @@ -228,6 +332,9 @@ selectors: the runner validates and authorizes every tool call first. ## The replay cache +Replay reports expose the original model as `step.cache.recordedBy` when +available; older recordings omit it. Use `--no-cache` to evaluate a new model. + A passing `agent.act` saves its actions once a later check verifies the outcome; the next run replays them without model calls, and the live agent continues from the current screen when the app or final state no longer @@ -250,24 +357,40 @@ matches. Misses and hand-offs use the model; `agent.assert`, every control that appeared (with its checked or selected state) is there, every one that went away is gone, at least one of those changed during the replay, and no new alert showed; otherwise the agent takes - over mid-step. Entries are keyed per agent and per `agentContext`. `step.cache.reason` says why: `no-entry`, + over mid-step. The web engine does not observe Next.js App Router's route + announcer, so its title echo never counts. An entry's key holds the test + file and title path, the target, the instruction and params, the agent, + its `context`, and the test's `agentContext`, so renaming the test or its + file records every goal in it again, and rewording a goal records that + goal again. `step.cache.reason` says why: `no-entry`, `wrong-context`, `target-not-found`, `target-ambiguous`, `end-mismatch`, and so on. -- A step recording no actions, or changing nothing on screen or in the - route, creates no entry; one whose `unique()` value - equals, is spelled inside, or is the encoded form of another param's value - is not recorded either (`step.cache.notRecorded`: `param-collision`). +- A step changing nothing on screen or in the route creates no entry + (`step.cache.write`: `no-change`) and runs live every run, strict + included; a change only inside a frame from another site counts as + nothing, since the tree leaves it out. A step recording no actions + creates no entry either. A step whose + `unique()` value equals, is spelled inside, or is the encoded form of + another param's value is not recorded either (`step.cache.notRecorded`: + `param-collision`). - `e2e init` gitignores `.e2e/cache/`; remove that line to commit entries and share replays with CI and teammates (CI stays `read-only` unless `cache: 'read-write'` is set). -- A failing run evicts the entries it implicates; `--no-cache` rules the - cache out of a failure. A cancelled run and a failure where no model - answered (`MODEL_PROVIDER_FAILED`, `MODEL_UNAVAILABLE`) implicate nothing - and evict nothing. +- A failing run keeps the entries a check confirmed before the failure. It + evicts a replay that ran into the failure and any recording no check + confirmed. A cancelled run and a failure where no model answered + (`MODEL_PROVIDER_FAILED`, `MODEL_UNAVAILABLE`) implicate nothing and evict + nothing. +- While you fix a test, keep the cache on: each rerun replays the goals + whose check passed instead of calling the model for them (SKILL.md, + "Rerunning a test while you fix it"). `--no-cache` is for ruling out a + stale replay (topic debugging, "Tools"), and for one live run once the + test passes. - With committed recordings, `--strict-cache` in CI fails a recording that no longer replays with `REPLAY_STALE` instead of quietly spending model - calls every run; re-record locally and commit. Unrecorded steps still run - live. + calls every run; re-record with a `read-write` run without the flag and + with `cache.strict` off, then commit. Unrecorded steps still run live; + retries replay too, and a strict run never writes the cache. ## Inspect what the model did @@ -306,12 +429,13 @@ make no model calls. `defineTool(tool({ ... }), { mutates: true })` from `e2e/agent` for a test API a flow calls mid-step. - `createToolLoopExecutor` keeps the loop and replaces prompt and tool - vocabulary. + vocabulary; spread `createGrammarTools(ctx, { guard })` into its tools and + start its `system` with `BASE_RULES` to keep the built-in actions. - Any `StepExecutor` (`{ name, version?, cache?, runStep(ctx) }`) goes under `executor`; the runner still owns observations, actions, budgets, and the report, and `system` or `tools` beside `executor` is `INVALID_CONFIG`. -Reference: https://e2e.tester.army/docs/agents +Reference: https://e2e.com/docs/agents ## In CI diff --git a/.agents/skills/e2e/references/bug-bash.md b/.agents/skills/e2e/references/bug-bash.md index f774c483..a75eacd8 100644 --- a/.agents/skills/e2e/references/bug-bash.md +++ b/.agents/skills/e2e/references/bug-bash.md @@ -5,6 +5,17 @@ verification pass that turns every claimed bug into a repro test that fails for the reason reported. Plan the charters, fan them out, merge the findings, prove each bug. Report confirmed bugs only, each with its failing test. +## Contents + +- 1. Prepare +- 2. Plan charters +- 3. Fan out +- 4. Merge +- 5. Triage +- 6. Verify +- 7. Report +- Rules + ## 1. Prepare - A config with a target and an agent that holds a model (topic `setup`), @@ -26,9 +37,8 @@ prove each bug. Report confirmed bugs only, each with its failing test. Explorers that share an account report each other's edits as bugs. - Start signed-in charters from a saved session: `e2e explore --session ` runs the setup test that saves it and explores signed in (topic - `explore`). A setup that types a password withholds screenshots from then - on; sign in by cookie or API call to keep them (topic `writing-tests`). - Each charter with its own account needs its own setup and session. + `explore`). Each charter with its own account needs its own setup and + session. - Put what the local app cannot do in the agent's `context`: integrations without keys, what is seed data, what must never be clicked (paid runs, real accounts). Add the explorer's own blind spots from step 5's artifact @@ -69,7 +79,10 @@ const persona = { model: gateway('openai/gpt-6-luna-fast'), maxSteps: 40, maxMod export default { ...shared, // The project's tests, so a repro can use its setup tests' sessions, plus the repro tests from step 6. - tests: [shared.tests ?? 'tests/**/*.e2e.ts', 'tests/bugbash/**/*.e2e.ts'].flat(), + tests: [ + ...[shared.tests ?? 'tests/**/*.e2e.ts'].flat().filter((glob) => !glob.startsWith('!tests/bugbash/')), + 'tests/bugbash/**/*.e2e.ts', + ], // The app already runs: no command. targets: [{ name: 'web', engine: web(), app: { url: 'http://127.0.0.1:3000' } }], retries: 0, @@ -119,7 +132,7 @@ time budget before judging a single result. ## 3. Fan out One `e2e explore` per charter with its own output directory: -`--output .e2e/bugbash/` writes `report.json` and `artifacts/` there, +`--output .e2e/bugbash/` writes `report.json` and `results/` there, and `--reporter list,markdown` adds `summary.md`. `e2e init` gitignores specific `.e2e/` paths, not `.e2e/bugbash/`: add it to `.gitignore` or delete it when done, and delete it before a new bug bash. `charters.txt` and @@ -149,15 +162,14 @@ setup and environment problems; fix them and rerun that charter alone. Read each log's `Findings` section (topic `explore`); `summary.md` holds the same, and `report.json` has the record under `run.explore`. The video is in -the attempt's `video/` directory under `.e2e/bugbash//artifacts/`. +the attempt's `video/` directory under `.e2e/bugbash//results/`. Merge findings that describe one defect: same path, same broken behavior. Keep the clearest reproduction and every charter that hit it. Keep warnings in a separate list unless the user asked for polish. -A charter that filled a password, or whose session's setup did, prints no -`evidence` line after that point. Its video is not masked: check it for -secrets before sharing (topic `writing-tests`). +A charter's video is not masked: check it for secrets before sharing (topic +`writing-tests`). ## 5. Triage diff --git a/.agents/skills/e2e/references/debugging.md b/.agents/skills/e2e/references/debugging.md index ef97dea6..35b13b3f 100644 --- a/.agents/skills/e2e/references/debugging.md +++ b/.agents/skills/e2e/references/debugging.md @@ -3,17 +3,73 @@ Paths below sit under the configured `output` directory; `.e2e` is the default. +## Contents + +- Read the failure +- Error codes +- Tools +- Flaky tests +- Is it the app? + ## Read the failure -1. Run with `--reporter list,markdown`: `list` ends with a `Failed Tests` - section, the markdown reporter's `Failures` line names `.e2e/failures/`. -2. Open the failed test's page under `.e2e/failures/` and grep `Expected:` - / `Observed:` (an `expect`), `Asked for:` and `Waited:` (a locator), - `Look at:` (the line it unwound through), whether every attempt - failed alike (a bug, not a flake), the steps, the last model turns of a - failed agent step, and the accessibility tree at failure, one node per - line. Fix from what was there. -3. `.e2e/report.json` backs the pages: +1. Every failure in the `list` output names the failing line with a code + frame, where the screen was (`❯ at`: a URL on the web, the app and + screen title on a device), the nodes nearest a failed locator + (`❯ on screen`), and ends with `❯ trace + .e2e/results//trace.md`. Open that trace page: `Look at:` (the line it unwound + through), `Expected:` / `Observed:` (an `expect`), `Asked for:` and + `Waited:` (a locator), whether every attempt failed alike (a bug, not a + flake), then every step with what it did under it: + - the node an action landed on (`tap button "Save" (12ms)`), or `✗` and + its code; + - what an assertion read while it waited (`expect gave up after 16 reads + in 1.6s: text "0 remaining" (1 match) x16`); one value throughout + points at a wrong expectation, a broken app, or one slower than the + timeout, changing values at a race; + - what the app logged meanwhile (`✗ network error: POST /api/save 500`, + `✗ uncaught error: ...`, `⚠ console warning: ...`, `ℹ console: ...`); + `❯ app 2 errors` in the terminal quotes the first 3 errors, and the page's + `## App log` lists up to 50 with their step, errors and warnings + first, counting the rest; + - where the page went (`↪ navigated to /login`, a new tab the app + opened, a frame that loaded); + - how the screen changed since the step before (`screen: 2 changes since + step 3` with the added, removed, and changed nodes); `unchanged` after + an action means it had no visible effect; + - the cache's decision for an agent step: replayed, handed off and where + (`at action 2 of 3, tap button "Apply": ...`), what became of the + recording, and the entry file; + - the agent's last turns, loop-guard notes included, and under + `## Screen at failure` the nodes closest to what a failed locator + asked for and the screen, one node per line. + `trace` (default `retain-on-failure`) chooses which tests keep a page; + `--trace` keeps one for every test of a run. +2. Fix from that evidence, never by walking the flow again from the start + over MCP: the run already saw every screen up to the failure. + - Fix every line it covers, not only the failing one. `observed` is what + the screen showed; a `❯ on screen` node is most likely what a failed + locator meant; the screen at failure (also `screen-at-failure.txt`, + item 5) holds the names the rest of that screen's lines need. + - A test stops at its first failed step (an `expect.soft` lets it go + on), and every wrong line after it costs another run. Before + rerunning, check the lines after it as far as you can: against the + screen at failure, the app source, or a live session. + - For a screen the evidence and the source leave out, look live (topic + `mcp`): open or reuse a session, go straight to that screen, then + `observe` and `locate`, or `run_code` for the fixed line and the ones + after it. On the web, `navigate` to the path of the `❯ at` URL or the + screen's route, and tap only to reach a state no URL opens. A device + session starts on the app's launch screen: take the shortest path from + there. + - Then rerun only that test by `file:line` (SKILL.md, "Rerunning a + test while you fix it"; topic `mcp` says when a session must close + first). +3. `.e2e/report.json` holds the same facts for scripts, not for reading: + each step's `events` (`engine`, `poll`, `navigation`) and the attempt's + `appLog` (each line's `step` is a step `index`) are what the page's step + lines are rendered from, before it filters and caps them. +4. When a script needs them: ```bash jq '.run | {status, exitCode, errors}' .e2e/report.json @@ -24,11 +80,14 @@ jq '.run.results[] | select(.selected and .status != "passed") | .attempts[-1] `error.details` holds the facts, `error.source` the line, `failure` the `url`, `screen` and `screenshot` artifact ids, and `candidates`; a - failed agent step has `turns`; `selected` drops filtered-out tests + failed agent step has `turns`; any agent step has + `observationTruncated: true` when a semantic observation was incomplete; + `selected` drops filtered-out tests (recorded as `skipped`). -4. Artifacts, under `.e2e/artifacts/`: `failure/screen.txt` - and the engine's screenshot per failed attempt; a Playwright trace per - traced attempt (`nub exec playwright-core@1.63.0 show-trace ` or `npx playwright-core@1.63.0 show-trace `); downloads; with +5. Artifacts, under `.e2e/results//attempt-/` (`rerun-/attempt-/` + for a `--last-failed` rerun), beside the + trace page: `screen-at-failure.txt` + and the engine's screenshot per failed attempt; downloads; with `--video` the recording (`video/video.webm` in a local browser, each later page `video/video-part.webm` with its own `startedAt`; `video/video.mp4` on a device; a provider's file or link); with @@ -41,31 +100,33 @@ jq '.run.results[] | select(.selected and .status != "passed") | .attempts[-1] | `CONFIG_LOAD_FAILED` | The config throws while loading (a refused engine option is `INVALID_CONFIG` instead) or imports a missing package, subpath, or removed export such as `defineConfig` | Install the dependency, or fix the import or line quoted | | `INVALID_CONFIG`, `INVALID_GLOB` | Unknown or foreign key (`app`, `webServer`, `use`, `projects`, `baseURL`), `json` with `list`; a `tests` glob with braces, classes, an absolute path, or no wildcard | The message names the key or glob to write; the app is declared in the target's `app` | | `CONFIG_NOT_FOUND`, `CONFIG_AMBIGUOUS` | Wrong `--config` path; both `.ts` and `.mts` present | Fix the path; keep one | -| `NO_TESTS` | The glob, a positional, or a filter matched nothing; the message names each empty positional and each undeclared `--tag` with the nearest declared one | Check the config `tests`, the `.e2e.ts` suffix, the tag names | +| `NO_TESTS` | The glob, a positional, or a filter matched nothing; the message names each empty positional and each undeclared `--tag` with the nearest declared one | Check the config `tests`, the file suffix (`.e2e.ts` by default), the tag names | | `NO_LAST_RUN` | `--last-failed` found no `.e2e/report.json` | Run once without it | +| `INVALID_CHANGED_SINCE` | `--changed-since` names no commit in the checkout, shares no history with HEAD (a shallow clone, or unrelated histories), or git is missing | Run from a git checkout with git installed; check the ref; fetch it and more history (`git fetch --unshallow`, or `fetch-depth: 0` on `actions/checkout`); for unrelated histories, name a ref on HEAD's history | | `COLLECTION_ERROR` | `async` describe body, `test.setup` inside `describe`, an option forbidden in a serial group, registration outside collection | Restructure per `writing-tests` | | `HOOK_FAILED` | `beforeAll` or `afterAll` threw; its scope's tests skip | Fix the hook; the report carries its error | -| `UNSUPPORTED_ARTIFACT` | A test's or target's `trace` or `video` on an engine that cannot record | Drop it there, or set it at the config root or CLI (such targets skip with a notice) | -| `BROWSER_INSTALL_FAILED`, `LAUNCH_TIMEOUT` | Browser download failed; engine init or attempt start exceeded `launchTimeout` | Run `nub run test:e2e:install` or the quoted `nub exec e2e-web install ` (`--with-deps` on bare Linux); raise the root `launchTimeout` (60 s default) | +| `UNSUPPORTED_ARTIFACT` | A test's or target's `video` on an engine that cannot record | Drop it there, or set it at the config root or CLI (such targets skip with a notice) | +| `BROWSER_INSTALL_FAILED`, `LAUNCH_TIMEOUT` | Browser download failed; engine init or attempt start exceeded `launchTimeout` | Run the quoted `npx @e2e-dev/web install ` or `pnpm exec e2e-web install ` (`--with-deps` on bare Linux); raise the root `launchTimeout` (60 s default) | | `APP_UNREACHABLE` | `app.command` never answered `readyUrl` within `startupTimeout`; on a device, a message naming the iOS automation runner: the runner failed, not the app | Read the quoted log lines; check the port, `app.url`, `app.command.env`, `app.command.startupTimeout`. Runner: rerun, else `npx agent-device daemon stop` and reboot the simulator | -| `APP_ALREADY_RUNNING` | Something already serves `url` when `command` should start | Stop it, or `reuseExisting: true` locally | +| `APP_ALREADY_RUNNING` | Something already serves `url` when `command` should start, such as the app an open `e2e mcp` session started | A session's app: `close_session` before the run (topic `mcp`). Otherwise stop it if you started it, else offer the user `reuseExisting: true` (CI ignores it) | | `APP_URL_REQUIRED`, `APP_NOT_OPEN` | A navigation on a target without `app.url`; a `screen` call before `app.open()` | Add `app.url` to the target; open the app first | | `INVALID_ARGUMENT`, `INVALID_LOCATOR` | A step argument failed validation; a locator got a bad option or filter key | Fix the call the code frame names | | `LOCATOR_NOT_FOUND` | Wrong role or name, inexact text, element off screen or in an iframe, page not open | Read the markup for the accessible name; `exact: false` or a RegExp; `browser.frameLocator` for iframes; `app.open()` first; `--headed` | | `LOCATOR_AMBIGUOUS` | Two matches (hidden duplicate, repeated label) | `{ name }`, a container scope, `filter`, `first()`, or `{ visible: true }` | | `ASSERTION_FAILED` | Wrong expectation, or the state settles later than 5 s; for `agent.assert`, a false judgment (explained in the report) | Compare with the report's actual text or the screenshot; `{ timeout }` on the matcher; rewrite the question | -| `ASSERTION_INCONCLUSIVE` | `agent.assert` or `agent.extract` asked about what the screen does not show (another page, still loading, only in pixels); a failure, never a pass | `app.open()` or `agent.waitFor` the right screen first; ask about what is shown; `vision: true` when the answer is in pixels (the message says so) | +| `ASSERTION_INCONCLUSIVE` | `agent.assert` or `agent.extract` asked about what the screen does not show (another page, still loading, only in pixels, inside a frame from another site); a failure, never a pass | `app.open()` or `agent.waitFor` the right screen first; ask about what is shown; `vision: true` when the answer is in pixels (the message says so) | | `MODEL_OUTPUT_INVALID` | An `extract` or `assert` answer failed the schema after one repair round | Simplify the schema or question; pick a stronger model | | `ACTION_FAILED` | Element not actionable (covered, disabled, detached), or an operation timed out | `expect` the condition first; close overlays; check `actionTimeout` | | `TEST_TIMEOUT` | The attempt exceeded `timeout` (120 s) | Split the test, or raise `timeout` | | `STEP_NOT_AWAITED` | The body returned while a step or `expect.poll` still ran, or an `afterEach` hook or fixture teardown returned while an `expect.poll` still ran: a call without `await` | `await` the call the code frame names (every `app`, `agent`, `screen`, `expect`, `expect.poll` call) | -| `MODEL_UNAVAILABLE` | `agents..model` holds no AI SDK instance: reported once under `run.errors` at the first `agent` fixture; the run stops. A model-less custom executor fails only `waitFor` and `extract` this way | Construct one, e.g. `gateway('openai/gpt-6-luna-fast')` from `ai`, and export its key (`AI_GATEWAY_API_KEY`) | +| `MODEL_UNAVAILABLE` | `agents..model` holds no AI SDK instance: reported once under `run.errors` at the first `agent` fixture; the run stops. A model-less custom executor fails only `waitFor` and `extract` this way. Also GitHub Copilot refusing the configured model to the account (`model_not_supported`, a plan or settings limit the listing cannot show): reported once at the first call, the run stops | Construct one, e.g. `gateway('openai/gpt-6-luna-fast')` from `ai`, and export its key (`AI_GATEWAY_API_KEY`); for Copilot, pick an id the plan serves | | `MODEL_PROVIDER_FAILED` | Network, 5xx, rate limit, no credits, or no response in 120 s, after the transport retries | Check the credential (the key; for keyless `gateway()` the Vercel CLI login and `.vercel/project.json`) and quota; retry (exit 3) | +| `MODEL_REFUSED` | The model refused the prompt (finish reason `content-filter`). Exit 1 | Rephrase the prompt or instruction, or switch to a different model | | `REPLAY_STALE` | `--strict-cache` or `cache.strict`, and a committed recording no longer replays; `step.cache.reason` says why | Rerun read-write with the knob the message names off; commit the changed entry under the directory it names (a custom `cache.store` is written by that run) | | `AUTOMATION_UNSUPPORTED` | The step needs an interaction the engine's toolset lacks (drag on a device, say); blocked, exit 1 | Do that step with `screen` actions, or run the test on a target that supports it | | `STEP_TIMEOUT`, `STEP_BUDGET_EXHAUSTED` | Goal too big or ambiguous, or a slow provider | Split the goal, use on-screen wording, add vocabulary via `agents..context` or `agentContext`; raise `maxSteps` or `maxModelCalls` (budget), the config `timeout` (`act`), `judgmentTimeout` (judgments); `--debug` | | `CONTEXT_OVERFLOW` | Screen plus step history did not fit the model's context window (`act` after one shrink-and-retry; a judgment on the first overflow) | Lower `agents..maxObservationBytes`, split the step, or pick a larger-window model | -| `POLICY_DENIED` | A URL whose scheme is not `http:` or `https:` (`file:`, `view-source:`, `data:`); a password `Secret` into a non-password sink; reading a secure field, `toHaveValue`, `toHaveText`, `toContainText`, `toHaveAttribute` included; `app.screenshot()` after a secret fill | http(s) or `about:blank` only; passwords into password inputs only; assert the outcome, not the value; screenshot before filling secrets | +| `POLICY_DENIED` | A URL whose scheme is not `http:` or `https:` (`file:`, `view-source:`, `data:`); a password `Secret` into a non-password sink; reading a secure field, `toHaveValue`, `toHaveText`, `toContainText`, `toHaveAttribute` included; `vision: 'only'` with pixels withheld | http(s) or `about:blank` only; passwords into password inputs only; assert the outcome, not the value | | `UNSUPPORTED_CAPABILITY` | A fixture the engine lacks (`browser` on a device), `schema` or `vision` on `act`, an action the surface lacks | Declare `requires: ['browser']`; drop the option or action | | `SESSION_UNAVAILABLE`, `SESSION_CONTRACT` | `session: 'x'` with no setup saving `x`; a setup that skipped a declared name. `SESSION_MISMATCH`, `SESSION_EXPIRED`, `SESSION_INVALID`: the stored session is another run's or app's, expired, or corrupt | Add or fix the `test.setup`; rerun it | | `ONLY_IN_CI` | `test.only` reached CI | Remove it | @@ -78,12 +139,14 @@ jq '.run.results[] | select(.selected and .status != "passed") | .attempts[-1] | --- | --- | | `--headed` | Watch the failing step | | `--workers 1 --retries 0` | Take parallelism and retries out of the picture | -| `--no-cache` | Rule out a stale `agent.act` replay | +| `--action-timeout 5000` | While writing a test: a wrong locator fails in 5 s instead of 30 s (`--assertion-timeout` for `expect`). Keep it longer than the slowest screen takes to load | +| `--no-cache` | Rule out a stale `agent.act` replay; one run, not every rerun. A failed run already deletes the replay that ran into the failure, so rerun with the cache on first. When the failure repeats after a replay the trace page shows as kept and the `--no-cache` run passes, tighten the check after that goal: the stale replay then fails it and is deleted | | `--debug` | Each agent step's duration, model calls, cost, transcript | | `--ai-trace`, then `npx unbox-ai runs .e2e/ai-trace.json` | What the model saw and called | | `--video`, `--video=retain-on-failure` | Watch the failed attempt; `step.startedAt` minus the segment's `startedAt` is the step's offset into it | | `command.log: '.e2e/logs/app.log'` | The app's output when it never gets ready or errors mid-test | -| `await app.screenshot('before-submit')` | Evidence before a secret is filled; later calls fail with `POLICY_DENIED` | +| `await app.screenshot('before-submit')` | Evidence at a point you choose; it masks secure fields only | +| `app.startRecording()`, then `await app.stopRecording()` | One transition as a video; fails while the test's `video` records | | `CI=1 npx e2e run` | Reproduce CI-only behaviour (defaults: topic running) | ## Flaky tests diff --git a/.agents/skills/e2e/references/explore.md b/.agents/skills/e2e/references/explore.md index b3b709c5..72d48ea7 100644 --- a/.agents/skills/e2e/references/explore.md +++ b/.agents/skills/e2e/references/explore.md @@ -3,7 +3,9 @@ `e2e explore` runs the agent against the app with a goal instead of a test file: see what it can do with an app before tests exist, hunt for regressions on a branch, or find what is worth turning into a test. It needs a config with -a target and an agent that holds a model, nothing else. +a target and an agent that holds a model, nothing else. With no config, it +runs the `e2e init` wizard first in a terminal; without one (an agent's +shell, CI) it fails with `CONFIG_NOT_FOUND`, so run `npx e2e init --yes` first. ```bash npx e2e explore # goal: "Explore the app and find bugs" @@ -13,9 +15,19 @@ npx e2e explore 'Hunt for broken forms' --video npx e2e explore --session admin 'Explore the admin settings' ``` +## Contents + +- What a run does +- Start signed in +- Flags +- Reading the result +- When it does not fit + ## What a run does -1. Opens the app on the target's URL. +1. Opens the app: the target's URL, or on a device the pinned app, launched + fresh. Explore installs no build, so a local device target needs + `app.bundleId`; an `appPath` alone leaves nothing to launch. 2. Plans one step: a structured model call reads the goal, the steps and findings so far, and the current screen, and answers with a title and a concrete charter for one flow, or decides the goal is covered. @@ -41,7 +53,7 @@ secrets; otherwise the agent skips signing in and says so in the step summary. (`test.setup('...', { sessions: ['admin'] }, ...)` and `session.save('admin')`, topic `writing-tests`). The run collects the config's test files, runs exactly the setup that declares the session, then restores it into the exploration -and, on a target with a URL, opens the app, as a test with +and opens the app, as a test with `{ session: 'admin' }` does. No other test runs. The setup runs as for `e2e run`, with the configured agents, cache, and retries; only the exploration runs as the explorer. The planner and the agent are told they @@ -49,9 +61,7 @@ start signed in, so no charter is spent signing in again. A name no setup declares fails before any app process starts with `COLLECTION_ERROR`, naming the declared sessions. -A setup that filled a secret taints the restored session, so every finding -goes without a screenshot; one that signed in without a fill (a `browser.setCookies` -session cookie, say) keeps them where the engine captures pixels, and only +Findings keep their screenshots after a setup that filled a secret. Only configured secrets are redacted (topic `writing-tests`). ## Flags @@ -62,9 +72,9 @@ configured secrets are redacted (topic `writing-tests`). | `--target ` | first configured target | The one target to explore. | | `--agent ` | `default` | Build the explorer from another configured agent (`agents.`). | | `--session ` | none | Run the setup that saves this session, then explore with it restored. | -| `--max-steps ` | 8 (1 to 12) | Exploration steps at most. | -| `--timeout ` | 600000 (180000 to 900000) | Wall clock; the last minute is for the assessment. | -| `--headed`, `--reporter`, `--output`, `--debug`, `--ai-trace`, `--trace [mode]`, `--video [mode]` | as `run` | Same meaning as for `e2e run`. One attempt, so a retry mode (`on-first-retry`, `on-all-retries`, the CI trace default) records nothing (`CI=1 e2e explore` needs `--trace on`; the run's notice says so); put the goal before a bare `--trace` or `--video`. | +| `--max-steps ` | 8 (1 to 100) | Exploration steps at most. | +| `--timeout ` | 600000 (180000 to 7200000) | Wall clock; the last minute is for the assessment. | +| `--headed`, `--reporter`, `--output`, `--debug`, `--ai-trace`, `--trace [mode]`, `--video [mode]` | as `run` | Same meaning as for `e2e run`. The exploration is one attempt, so a retry mode (`on-first-retry`, `on-all-retries`) records nothing (the run's notice says to pass `--trace on`); put the goal before a bare `--trace` or `--video`. | Per-step action and model-call budgets default to 40 each; `agents..maxSteps` and `agents..maxModelCalls` in the config @@ -109,11 +119,11 @@ A step carries `errorCode` only when it did not pass; a finding carries the result whose `file` is `explore`, where its path, size, and digest are. With `--session`, `run.results` also holds the setup's result and the project's other tests as skipped (`filtered`). The attempt directory is -`.e2e/artifacts//explore--//attempt-0/`: the slug -is the goal's first words in lowercase ASCII, capped, and the digest keeps -distinct goals apart while the same goal always maps to the same directory. -For example -`.e2e/artifacts/web/explore-check-the-cart-totals-1a2b3c4d5e6f7a8b/default/attempt-0/finding-1.png`. +`.e2e/results/explore--/attempt-1/`: the slug is the goal's first +words in lowercase ASCII, capped, and the id keeps distinct goals, targets, and +agents apart while the same goal on the same target and agent always maps to +the same directory. For example +`.e2e/results/explore-check-the-cart-totals-1a2b3c4d5e6f7a8b/attempt-1/finding-1.png`. Turn a finding into a test: its `reproduction` steps are the `agent.act()` instructions or `screen.*` actions, and `expected` is the assertion. diff --git a/.agents/skills/e2e/references/mcp.md b/.agents/skills/e2e/references/mcp.md index 9875d27d..c1bec0f2 100644 --- a/.agents/skills/e2e/references/mcp.md +++ b/.agents/skills/e2e/references/mcp.md @@ -1,16 +1,41 @@ # Driving the app over MCP `e2e mcp` serves a project's live app to a coding agent over MCP (stdio). The -coding agent drives the app the way the testing agent does: look at a screen -before writing a test, check a locator before committing to it. Running tests -and reading a failed run stay on the CLI (topics `running` and `debugging`). +coding agent drives the app the way the testing agent does, writes a test +as it walks the flow, and checks its locators and runs its lines before +writing them. `run_tests` runs a test file or the whole suite as +`npx e2e run` does and returns what it printed (topics `running` and +`debugging`); to fix one failing test, a session can run it and stop on the +screen where it failed. + +## Contents + +- Setup +- Tools +- Workflow +- Fixing a failing test +- Rules ## Setup -The server ships with `e2e`. `e2e init` offers to register it; by hand: +The server ships with the project's `e2e` dependency, separately from the +skills-only plugin. `e2e init` offers to register it for Claude Code and +Cursor. Reuse an existing registration; when adding one, use the project's +config so it starts in the project it drives. + +Codex reads `.codex/config.toml` in a trusted project: + +```toml +[mcp_servers.e2e] +command = "npx" +args = ["e2e", "mcp"] +``` + +`codex mcp list`, run in the project, shows the server. Start a new session +after registering it so the tools load. Claude Code can register it with: ```bash -claude mcp add e2e -- npx e2e mcp # Claude Code +claude mcp add --scope project e2e -- npx e2e mcp ``` Or declare it in the client's project config (`.mcp.json` for Claude Code, @@ -20,36 +45,47 @@ Or declare it in the client's project config (`.mcp.json` for Claude Code, { "mcpServers": { "e2e": { "command": "npx", "args": ["e2e", "mcp"] } } } ``` -No model is needed: without the optional `ai` package, sessions open and the -deterministic tools work. +Sessions run on the project's config (`e2e.config.ts`, `.mts`, `.js`, or `.mjs`): +in a project without one, `open_session` fails with `CONFIG_NOT_FOUND`, so +set it up first (topic `setup`). No model is needed: without the optional `ai` package, sessions +open and the deterministic tools work. A test a session runs needs one only for +an agent step with no recording. Flags: `--config ` names the default config file, `--target ` -fixes the target every session opens on, `--headless` hides the browser or -simulator (sessions are headed by default outside CI), `--max-sessions ` +fixes the target every session opens on, `--headed` shows the browser or +simulator when the engine supports it in sessions that do not set `headed` +(sessions are headless by default, as in `e2e run`), `--max-sessions ` sets how many sessions may be open at once (default 4, 1 through 16). ## Tools -Four tools; everything a session can do is a catalog behind `call`. +Five tools, all on the server registered as `e2e`; everything a session can +do is a catalog behind `call`. Clients show them with the server name, such +as `mcp__e2e__open_session` in Claude Code; when another MCP server offers a +tool with the same name, use the `e2e` server's. | Tool | Does | | --- | --- | -| `open_session` | Loads the config (`config` names another file; default the nearest `e2e.config.ts`), starts the declared app command if any, boots the engine, opens the app URL, and returns the session id, the catalog, and the first observation. `target` is required when the config declares several. Each session has its own browser or device. | +| `open_session` | Loads the config (`config` names another file; default the nearest `e2e.config.ts`), starts the declared app command if any, boots the engine, opens the app (its URL, or launches a device's pinned app; it installs no build, so a local device needs `app.bundleId`), and returns the session id, the catalog, and the first observation. Its summary names the app URL, a port assigned for `:0`, and the app command's readiness URL and `env` names; e2e starts nothing else, so a database the app uses runs outside it. The config reads the MCP server's environment, not your shell's. `target` is required when the config declares several. `headed: true` shows the browser or simulator when the user wants to watch; without it the session follows the server's `--headed`, headless by default. `test` runs one test first and leaves the session on the screen where it stopped (see Fixing a failing test). Each session has its own browser or device. | | `tools` | The catalog: one line per tool with its argument names (`?` marks optional), the first sentence of its description, and `[read-only]` where it changes nothing. `tools {tool}` shows the full description and the JSON Schema of its arguments. | | `call` | Runs one catalog tool: `call {tool: "tap", args: {target: "n42"}}`. Arguments are checked against the tool's schema first; a wrong one fails with `INVALID_ARGUMENT` naming the field. | | `close_session` | Saves a recording still running, ends the attempt, disposes the engine, stops the app processes the session started once no other session uses them. | +| `run_tests` | Runs `e2e run` in its own process and returns its output: the command, the exit code, each test, the failure blocks, the summary. `files` takes `e2e run` positionals (`tests/checkout.e2e.ts:12`, a file, a directory, a glob); without it, every test runs. The other fields are `e2e run` flags in camelCase: `config`, `target`, `grep`, `grepInvert`, `tag`, `excludeTag`, `lastFailed`, `changedSince`, `headed`, `workers`, `retries`, `repeatEach`, `maxFailures`, `actionTimeout`, `noCache`, `updateSnapshots` (while fixing a test: `retries: 0, actionTimeout: 5000`). Needs no session. One run at a time (`RUNNER_BUSY`); cancelling the call interrupts the run. Exit 1 (a test failed) is a normal result; exit 2 and above is an error result. Output past 60,000 characters keeps its end; `.e2e/report.json` has every result. | The catalog, per session: | Catalog tool | Does | | --- | --- | | `observe` | A fresh observation: one node per line as `#id role "name" ...`, plus the current path. On the web engine a link's `href` is its path on the app URL's origin, else origin and path (`mailto:` and `tel:` keep their scheme, `blob:` its inner origin and path); `?…` and `#…` mark a dropped query or fragment, `data:…` and `javascript:…` an inline payload, a trailing `…` a target cut at 256 characters. | -| `tap`, `double_tap`, `long_press`, `right_click`, `hover`, `type`, `press`, `select`, `check`, `scroll`, `scroll_to`, `drag`, `upload`, `navigate`, `back`, `dismiss_keyboard` | The grammar verbs, as the testing agent gets them: `check` takes `checked`, `drag` a `to` node, `upload` project-relative `files`; `dismiss_keyboard` comes with device engines. Each reports what changed on screen; `observe` shows the whole screen. A verb the engine cannot honor is not listed and fails with `UNSUPPORTED_CAPABILITY`. | -| `type_secret` | Fills a configured secret by name: a credential's password into a password field, a `secrets` entry into any editable input; the plaintext never reaches the agent. Listed when the config declares `credentials` or `secrets` and the engine can fill secrets. | +| `tap`, `double_tap`, `long_press`, `right_click`, `hover`, `type`, `press`, `select`, `check`, `scroll`, `scroll_to`, `drag`, `upload`, `navigate`, `back`, `resize_viewport`, `dismiss_keyboard` | The grammar verbs, as the testing agent gets them: `check` takes `checked`, `drag` a `to` node, `upload` project-relative `files`; `dismiss_keyboard` comes with device engines. On an engine that declares `tapModifiers`, `tap`, `double_tap`, and `right_click` take an optional `modifiers` list (`Shift`, `Control`, `Alt`, `Meta`, `ControlOrMeta`) to hold keys for the click. Each reports what changed on screen and, after its lead, the test line that repeats it (`Test line: await screen.getByRole("button", "Save").tap();`) or `No test line:` and why; `observe` shows the whole screen. A verb the engine cannot honor is not listed and fails with `UNSUPPORTED_CAPABILITY`. | +| `type_secret` | Fills a configured secret by name: a credential's password into a password field, a `secrets` entry into any editable input; the plaintext never reaches the agent, and its test line names the credential or secret (`fill(credentials.user(name).password)`, `fill(secrets.get(name))`). Listed when the config declares `credentials` or `secrets` and the engine can fill secrets. | | `locate` | Tries a semantic locator with exactly one of `role`, `text`, `label`, `placeholder`, `testId` (`name` only with `role`; `exact` optional; anything else is `INVALID_ARGUMENT`) and returns how many nodes match, which, and the `screen.*` call to write. | -| `screenshot` | The masked pixels as an image. | -| `tap_at` | Taps a point (`x`, `y` in the latest screenshot's pixels): a listed control under it by id, otherwise the bare point, which an engine without a bare-point tap cannot do. Listed when the engine declares `tap` or a bare-point tap. | -| `hover_at` | Hovers a point the same way: a listed control under it by id, otherwise the bare point. Listed when the engine declares `hover` or a bare-point hover. | +| `run_code` | Runs test statements (`code`, as a test body writes them) on the session's `screen` and `app` with a test's waits, exactly-one rule, and matchers. Stops at the first failure and returns each statement's pass, failure (code and message), or read value, then what changed on screen. Each statement waits at most `timeout` ms (default 5000), but `timeout` doesn't stop a regex mid-match: avoid nested quantifiers such as `(\w+\s?)*`. The code is parsed, never evaluated as JavaScript: statements start with `screen`, `app`, or `expect(locator)`; arguments are literals, regexes, arrays, objects, locators, and `credentials.user(name).username`. A secret (`credentials.user(name).password`, `secrets.get(name)`) goes only into `fill`, under `type_secret`'s checks. Engine fixtures (`browser`, `device`), `app.screenshot`, and `toHaveScreenshot` are left out. Anything else is `INVALID_ARGUMENT` quoting the refused code with its line and column, and nothing runs. Reads and matchers see `` where a secret value is. When the config declares a secret or credential, a locator found by part of a text (a regex, an `exact` other than `true`, `hasText`) is `POLICY_DENIED` before anything runs: find it by its exact text. A matcher that expects `` is `INVALID_ARGUMENT`. The statements that pass, except reads, join `test_lines` as written. | +| `test_lines` | Every test line the session's actions printed and every statement `run_code` passed, in order, as a draft test body, starting with `await app.open();` for the session's own opening (after `open_session {test}`, the lines that continue that test, with no opening); an action without a line shows as a `//` comment. A line's locator matched the target alone on the live screen before the action (role and name first, test id last, narrowed to `{ visible: true }`, a container, or `.nth(i)` with a comment when twins share it); a fill names `credentials.user(name)` or `secrets.get(name)`, never a value; a navigation on the app's origin is written as its path. | +| `screenshot` | The masked pixels as an image. With `path` (a `.png` inside the project, relative to its root, not hidden or under a hidden directory), also saves a full-resolution masked capture there and returns its absolute path. | +| `tap_at` | Taps a point (`x`, `y` in the latest screenshot's pixels) where you aim. An engine without a bare-point tap taps the listed control under it by id instead, and fails on a point on nothing listed. Listed when the engine declares `tap` or a bare-point tap. | +| `hover_at` | Hovers a point the same way: the bare point, otherwise a listed control under it by id. Listed when the engine declares `hover` or a bare-point hover. | +| `drag_at` | Drags from one point (`x`, `y`) to another (`to_x`, `to_y`) where you aim. An engine without a bare-point drag drags by id instead when the points land on two different listed controls, and fails otherwise. Listed when the engine declares `dragTo` or a bare-point drag. | | `type_at` | Types `value` into the field at a point: a listed input under it is filled by id; with a keyboard, anything else is tapped to focus it and typed into, at the caret unless `replace` is set. Without a keyboard a point on nothing listed fails. Listed when the engine declares `type` or a keyboard. | | `press_at` | Sends one `key` (`Enter`, `Escape`, `Tab`) to the control at a point: a listed control gets it by id; with a keyboard, anything else is tapped to focus it and the key goes through the keyboard. Listed when the engine declares `press` or a keyboard. | | `select_at` | Picks the option whose visible label is `value` in the select-like control at a point; the point must land on a listed select. Listed when the engine declares `select`. | @@ -57,32 +93,154 @@ The catalog, per session: | `stop_recording` | Stops it and returns the absolute path of each video file, under `/videos//` (`.e2e` by default), or the URL of a provider's own recording. | | Project tools | Every `defineTool` in the agent's `tools` that applies to the target's platform, under its own name; an engine pack such as `mobileTools` adds `open_app`, `swipe`, `alert`. | -Once a secret is filled in the session, `screenshot` and the point tools -(`tap_at`, `hover_at`, `type_at`, `press_at`, `select_at`) answer with a -`PIXEL_TAINTED` line for the rest of it; act on listed nodes by id (topic -`writing-tests`). Before the session's first `screenshot`, a point tool only +Project tools use their `toModelOutput` renderer for strings as well as +structured results. Its `error-text` and `error-json` outputs set the MCP +result's `isError: true`; treat them as failed calls. + +`screenshot` masks secure fields only, after a secret fill too (topic +`writing-tests`). Before the session's first `screenshot`, a point tool +(`tap_at`, `hover_at`, `drag_at`, `type_at`, `press_at`, `select_at`) only says to take one. Resources: `e2e://guide` and `e2e://guide/` hold this skill. ## Workflow -1. `open_session`, then `call {tool: "observe"}` and act until the screen you - want to test is in front of you. Node ids are valid only for the newest - observation; an action reports what changed, so observe again before +1. Before `open_session`, create `tests/.e2e.ts` (topic + `writing-tests`) with its imports, the `test(...)`, and the call that + opens the screen it covers, such as `app.open('/settings/members')`. + Then `open_session`, which opens the app, and go to that screen: on the + web, `call {tool: "navigate", args: {url: ""}}`; + on a device, the shortest path from the launch screen. Actions that only + got you there, such as a sign-in the test's saved session replaces, stay + out of the test when its opening reaches that screen directly; otherwise + they are its first steps. Node ids are valid only for the newest + observation; an action reports what changed, so `observe` again before using new ids. -2. `call {tool: "locate", args: {...}}` for each locator you intend to write. - One match: use the printed `screen.getByRole(...)` call. Zero or several: - adjust before writing the test; the same failure would hit the test as - `LOCATOR_NOT_FOUND` or `LOCATOR_AMBIGUOUS`. -3. Write `tests/.e2e.ts` (topic `writing-tests`). Deterministic steps - where you saw exact names; `agent.act` where the flow varies. -4. Run it from the shell: `npx e2e run tests/.e2e.ts`, read the - failure (topic `debugging`), fix, repeat. -5. `close_session` when you are done; an idle session closes on its own after - 30 minutes and never outlives 4 hours. When the client exits, every session - closes and the app commands stop. To look at another project or config, - `open_session {config: "path/to/e2e.config.ts"}`; no restart needed. +2. Walk the flow one screen at a time, and write each screen's steps into + the file before you leave it. Do not walk the whole flow first and write + the test from memory afterwards. Each step is one of two kinds: + - An `agent.act` goal, the default for a path that varies: write the goal + with the words on screen and its values in `params`, then walk its + path in the session. Those actions stay out of the test, and so do the + test lines they print. + - A `screen` line, for an exact value or interaction: a sign-in form in a + setup test, a field that must receive one specific string, an + interaction that is the point of the test. Act on the control by node + id, then copy the `Test line:` the action printed into the file. Its + locator matched that control alone on the live screen. When the action + prints `No test line:` instead, the reason says why; for a control no + locator found alone, try one with `locate`, or write the line and run + it with `run_code`. + + With several `screen` steps on one screen, take their actions, then add + the lines they printed in one edit. +3. When the outcome shows, add its check before the next step: an + `agent.assert`, or an `expect` you ran first with + `call {tool: "run_code", args: {code: "await expect(screen.getByRole('status')).toHaveText('Saved');"}}`. + Build its locator from `observe`, or try it with + `call {tool: "locate", args: {...}}`. One match: use the printed + `screen.getByRole(...)` call. Zero or several: adjust before writing it; + the same failure would hit the test as `LOCATOR_NOT_FOUND` or + `LOCATOR_AMBIGUOUS`. A role query's `name` matches the accessible name + only, while `observe` shows a `status`, `alert`, or `listitem` by its + text: `status "Saved."` on an unlabelled status is its text, so match it + with `getByRole('status').filter({ hasText: 'Saved.' })` or `getByText`. + + `run_code` runs statements as a test body writes them, in order, with a + test's waits and matchers, and stops at the first failure. The + statements that pass, except reads, join `test_lines` in the order they + ran. When one fails, fix it from its code and message and the screen in + the result, then run only the fixed statement and the ones after it: + the app is where the passing statements left it, so starting over + repeats their actions. Lines on an engine fixture + (`expect(browser).toHaveURL(...)`) can't run there; write them from the + screen, whose header names the current path. In a project with + credentials or secrets, `run_code` refuses a locator found by part of a + text, `filter({ hasText })` included: use the exact text instead, + `getByText('Saved.')`, or `filter({ has: screen.getByText('Offer B') })` + to pick a row. Raise `timeout` for a slow first `app.open`. + + When the walk ends, the test is written, every line you copied from a + `Test line:` or ran with `run_code` has run once on the live app, and the + first run checks the rest. If you missed a line, `test_lines` returns + every line printed or passed so far, in order, the act goals' paths + included. One walk, in order: + + ```text + write tests/members.e2e.ts: imports, test(...), app.open('/settings/members') + call open_session, then navigate {url: "/settings/members"} its line is the app.open already written + edit + await agent.act('invite {email} as an editor', { params: { email: 'ada@example.test' } }); + call tap, type, select, tap the goal's path; its test lines stay out + call run_code {code: "await expect(screen.getByRole('status')).toContainText('Invite sent');"} passed + edit + that line + call type {target: "n40", value: "ada"} prints the test line below + Test line: await screen.getByRole("searchbox", "Search members").fill("ada"); + edit + that line, as printed + call run_code {code: "await expect(screen.getByRole('listitem')).toHaveCount(1);"} passed + edit + that line + ``` +4. Run only that test: + `run_tests {files: ["tests/.e2e.ts:"]}`, or + `npx e2e run tests/.e2e.ts:` from the shell (SKILL.md, + "Rerunning a test while you fix it"). On a failure, fix from the run's failure block + and the trace page (topic `debugging`), then rerun it the same way. To see a screen the evidence leaves out, go + straight to it instead of walking the flow again from the start: + `open_session {test}` stops on the failure screen (Fixing a failing + test); otherwise, on the web, `navigate` to its path; on a device, the + shortest path from the launch screen. Then `observe` and `locate`, or + run the fixed line with `run_code`. +5. Leave the session open through those runs, on the screen under test, + wherever Rules lets a run go beside it, so step 4's live look needs no + new session. `observe` does not reload: `navigate` even to the screen + the session is on when the run or a fix may have changed what it shows. + A live look's actions print test lines too: copy a fixed line from its + action, since `test_lines` also lists the looks after the first walk. + `close_session` once the test passes; an idle session closes on its own + after 30 minutes and never outlives 4 hours. When the client exits, every + session closes and the app commands stop. To look at another project or + config, `open_session {config: "path/to/e2e.config.ts"}` (absolute, or + relative to the server's working directory); no restart needed. + +## Fixing a failing test + +When a run's evidence leaves out a screen the fix needs (topic +`debugging`), don't walk the flow again from the start over `call`. A +session runs the test and stops where it failed: + +1. `open_session {test: "tests/.e2e.ts:"}`, where `` is + the line of the test's `test(` call, not the failing line inside it. A + file that holds one test needs no line. The test + runs the way `e2e run` runs it: `beforeAll` and `beforeEach` hooks, + `test.extend` fixtures, and the body, with the same locators, `expect`, + and timeouts. +2. Read the text above the session summary: the outcome, the error code and + message, the failing line with the code around it, the URL, the nodes on + screen nearest a locator that matched nothing, and the steps that ran. + The screen below it is the live screen where the test stopped. +3. `observe` and `locate` on that screen to find the right name or the + missing step. Then carry on from there: act out the test's next lines + with `call`, so the rerun finds no second mistake. Each action prints the + `Test line:` that repeats it; where it differs from the test's line, use + the printed one. Edit the test file. `test_lines` returns the session's lines, + which start after the test's own. +4. To rerun, `close_session`, then `open_session {test}` again. The session + loads the config and the test files fresh, so it runs the edited test + from the start and stops on the next failure, or on the final screen. +5. `close_session`, then confirm once with + `run_tests {files: ["tests/.e2e.ts:"]}`. + +- `afterEach` hooks and fixture teardown wait for `close_session`; a failure + there shows as a `Cleanup (afterEach)` line in its result. +- The replay cache treats the test as a run's first attempt: an `agent.act` + step recorded by `e2e run` or an earlier session replays, and a new one is + recorded once a later check passes. +- The name follows `e2e run`'s positional rules and must select one test on + the session's target. Several fail with `TEST_AMBIGUOUS`, which lists them + as `file:line`; none fails with `NO_TESTS`. +- `SESSION_TEST_UNSUPPORTED`: a test in a serial group, a test that starts + from a setup test's `session`, and a test that runs as several agents + can't run alone in a session. Fix those with `run_tests`. ## Rules @@ -93,11 +251,35 @@ Resources: `e2e://guide` and `e2e://guide/` hold this skill. - A failed action is an error result that leads with the tool, its target, and the code (`tap #n9 failed: LOCATOR_NOT_FOUND: ...`) and still shows the screen it re-observed: re-aim from that screen. -- Nothing a session does is recorded as a test or into the replay cache. A - session is for looking and trying; the test is what you write afterwards. -- A run from the shell and a live session share the app only if the target's - app `command` sets `reuseExisting` (ignored in CI); otherwise close the - session before running. +- Nothing a session does is recorded as a test, and only a test it runs + (`open_session {test}`) reads or writes the replay cache. A session is for + looking and trying; the test is the file you write as you walk. +- On the web, a session can stay open while `run_tests` or `e2e run` runs, unless the two + collide over the app. Read the target's `app`: + - No `command` (the app runs elsewhere): both use that app. + - A `url` on port `0`: the session and each run start their own copy of + the app. A dev server that allows one copy per directory, such as + `next dev` in Next.js 16, fails the run with `APP_UNREACHABLE`; close + the session before each run there. + - A `command` on a fixed port with `reuseExisting: true`: a run uses the + app the session started, and its data. Whichever started the app stops + it: if the session closes during a run (`close_session`, 30 idle minutes + or the 4-hour limit, since a run is not session activity), the app stops + under the run. Its next navigation fails with `APP_UNREACHABLE`; a step + already on the page fails with whatever the dead server breaks. Before a + long run, `observe` once to reset the idle clock. With `CI` set, as in + many agent sandboxes, `reuseExisting` is ignored: prefix the run with + `CI=` to unset it, which also drops the other CI defaults (topic + `running`), or close the session before each run. + - A `command` on a fixed port without it: a run beside the session fails + with `APP_ALREADY_RUNNING`. Offer the user `reuseExisting: true` (CI + ignores it); never add it unasked. Until then, close the session before + each run. +- On mobile, close the session before each run: a session and a run on one + device fight over it. It can stay open only on a target whose `device` no + run uses, with each run passing `--target` for another; an `app.command` + both start (Metro) follows the web rules above. Ask before adding a + target to the user's config (topic `setup`). - Parallel agents (subagents) share one server: each opens its own session and passes its session id to every `tools`, `call`, and `close_session`. A call may leave `session` out only while one session is open. Sessions @@ -114,4 +296,6 @@ Resources: `e2e://guide` and `e2e://guide/` hold this skill. open on theirs, or close them first. `ENGINE_IN_USE`: the target's engine comes from a package every session shares; create it in the config or a file it imports by path. `UNKNOWN_TOOL`: the name is not in this session's - catalog; the message lists what is. + catalog; the message lists what is. `TEST_AMBIGUOUS` and + `SESSION_TEST_UNSUPPORTED`: see Fixing a failing test. `RUNNER_BUSY`: + `run_tests` is already running a suite; wait for its result. diff --git a/.agents/skills/e2e/references/running.md b/.agents/skills/e2e/references/running.md index aafe162a..b7933b78 100644 --- a/.agents/skills/e2e/references/running.md +++ b/.agents/skills/e2e/references/running.md @@ -1,11 +1,21 @@ # Running tests +## Contents + +- Commands +- The replay cache +- Output +- Exit codes +- Continuous integration + ## Commands ```bash npx e2e run [files...] [options] # run tests npx e2e explore [goal] [options] # explore toward a goal without a test file (topic explore) npx e2e list [files...] [options] # print what run would select +npx e2e doctor [--check-model] # can a run start: config, engine setup, models; + # --reporter json for a host, exit 2 on a failed check npx e2e init [directory] [--yes] # scaffold a project, refresh the agent skill npx e2e guide [topic] # print this skill; topics: setup, writing-tests, agent, # running, explore, debugging, mcp, bug-bash @@ -13,6 +23,10 @@ npx e2e cache ls|clear|stats # inspect or empty the replay cache npx e2e login|logout|models [provider] # e2e/oauth subscription logins: openai, # github-copilot, opencode-console, spacexai npx e2e mcp [--target ] # MCP server for a coding agent (topic mcp) +npx e2e upgrade [version|tag] # e2e and its @e2e-dev packages to latest or a version; + # alias update, --dry-run shows the plan; updates + # this skill's copies too; an e2e without the + # command: npx e2e@latest upgrade npx e2e feedback -m [opts] # report a problem with e2e itself npx e2e telemetry [disable|enable] # anonymous usage telemetry: status or switch ``` @@ -27,27 +41,33 @@ npx e2e telemetry [disable|enable] # anonymous usage telemetry: status or switch | `--tag ` | Any of the tags, comma-separated or repeated; all of them with `--tag-mode all`. An empty `--target`, `--tag`, or `--agent` value is a usage error, exit 2. | | `--exclude-tag ` | Drop tests carrying any of these tags, however selected. | | `--grep `, `--grep-invert ` | Keep, or drop, tests whose title (describe titles and test title joined by spaces, `checkout pays`; not file or tags) matches a regular expression. Bare pattern or `'/pattern/i'`; repeat for alternatives. | -| `--last-failed` | The tests the previous run (`/report.json`) did not pass or never ran (`--max-failures` included), plus every test in a failed `beforeAll` or `afterAll` scope. A test another filter leaves out stays owed (`run.carried` in the report) until a rerun runs it. No report is `NO_LAST_RUN`, exit 2. | -| `--shard ` | One contiguous slice of the selected tests (`--shard 2/3`), cut after every other filter; serial groups stay whole, each shard brings its own setup tests. | +| `--changed-since ` | Run the configured `filters` on the change since the ref: every file that differs from the merge base of the ref and HEAD, uncommitted and untracked included (`--changed-since origin/main`). Without it no filter runs. A ref git can't measure from is `INVALID_CHANGED_SINCE`, exit 2. | +| `--last-failed` | The tests the previous run (`/report.json`) did not pass or never ran (`--max-failures` and `--max-duration` included), plus every test in a failed `beforeAll` or `afterAll` scope. A test another filter leaves out stays owed (`run.carried` in the report) until a rerun runs it. No report is `NO_LAST_RUN`, exit 2. | +| `--shard ` | One contiguous slice of the selected tests (`--shard 2/3`), cut after every other flag; serial groups stay whole, each shard brings its own setup tests. | | `--headed` | Visible browser or simulator when the engine supports it. | | `--agent ` | Run unpinned tests as these `agents.` entries (default `agents.default`), comma-separated or repeated; several names run each such test once per agent. | -| `--workers `, `--retries ` | Override the resolved values; retries 0-10, workers 1-1024. | +| `--workers `, `--retries ` | Override the resolved values; retries 0-10, workers 1-1024. A retry replays the cache only in strict mode (`--strict-cache`, `cache.strict`); otherwise it runs every agent step live. | +| `--action-timeout `, `--assertion-timeout ` | Override config `actionTimeout` (default 30000) and `assertionTimeout` (default 5000) for this run; a call's own `{ timeout }` still wins. | | `--max-failures ` | Stop after n failures: the rest skip (cause `failure-limit`), running tests end `interrupted`; exit 1. | +| `--max-duration ` | Stop after ms of running tests, over config `maxDuration`: the rest skip (cause `time-limit`), running tests end `interrupted`, `RUN_TIMEOUT`; exit 1. | | `--repeat-each ` | Run every selected test n times, each run its own result (`repeat` 0 through n-1); add `--no-cache` or later runs replay the first's recording. The `Repeats` summary row names each unstable test's failed runs. | -| `--reporter ` | `list`, `json`, `junit`, `markdown`, comma-separated; `json` cannot combine with `list`. | +| `--reporter ` | `list`, `json`, `junit`, `markdown`, comma-separated; `json` cannot combine with `list`. An empty value is a usage error, exit 2. | | `--output ` | Results directory, over the config's `output` (default `.e2e`). | | `--no-cache` | Replay cache off for this run. | -| `--strict-cache` | Fail a step whose committed recording no longer replays (`REPLAY_STALE`, exit 2) instead of handing it to the agent. | -| `--pass-with-no-tests` | Exit 0, not `NO_TESTS`, when nothing matches. | +| `--strict-cache` | Fail a step whose committed recording no longer replays (`REPLAY_STALE`, exit 2) instead of handing it to the agent; retries replay too, and the run never writes the cache. | +| `-u`, `--update-snapshots` | Write the stored screenshots `toHaveScreenshot` finds missing or different, and pass. | +| `-w`, `--watch` | Run, then run again when a test file, a source file beside the tests, or the config is saved, or a key asks: `a` or enter all, `r` the current filters, `f` failed, `u` update snapshots, `p` and `t` filter by file and title, `q` quit; a key during a run cancels it. Refused in CI and with `--reporter json`. | +| `--pass-with-no-tests` | Exit 0, not `NO_TESTS`, when nothing matches; a run with nothing to run starts no engine or app. | | `--debug` | Phase timings and an agent step table on stderr; transcripts as artifacts. | -| `--ai-trace` | Every model call, to `/ai-trace.json`. | -| `--trace [mode]`, `--video [mode]` | Which attempts record a trace, or a video (WebM on browsers, MP4 on devices), over the config and every target: bare is `on`; `--trace off` skips the cost; `retain-on-failure` (video) keeps only failed attempts; `on-first-retry` records first retries, `on-all-retries` every retry. A test's own `trace` or `video` still wins; targets whose engine cannot record are skipped with a notice. Both are greedy: write `--video=` or put test files first. The failure recap names the video. | +| `--ai-trace` | Every model call except a test filter's, to `/ai-trace.json`. | +| `--trace [mode]`, `--video [mode]` | Which tests keep a trace page (`/results//trace.md`), or which attempts record a video (WebM on browsers, MP4 on devices), over the config and every target: bare is `on` (`--trace` then pages every test, passing ones too); `--trace off` writes none; `retain-on-failure` keeps only failures; `on-first-retry` keeps first retries, `on-all-retries` every retry. A test's own `trace` or `video` still wins; targets whose engine cannot record video are skipped with a notice. Both are greedy: write `--video=` or put test files first. The failure recap names the trace page and the video. | ```bash npx e2e run tests/signup.e2e.ts npx e2e run signup.e2e.ts:12 # by bare name, the test declared at line 12 npx e2e run tests/agent --tag smoke --exclude-tag slow --grep checkout npx e2e run --last-failed # the loop after a red run +npx e2e run --changed-since origin/main # the configured filters judge the change npx e2e run --shard 2/3 # one CI job of three npx e2e run 'tests/**/*.smoke.e2e.ts' --target chromium --workers 1 --retries 0 CI=1 npx e2e run # the CI defaults, locally @@ -55,22 +75,51 @@ CI=1 npx e2e run # the CI defaults, locally `list` takes the same files and selection flags (`--config`, `--target`, `--tag`, `--tag-mode`, `--exclude-tag`, `--grep`, `--grep-invert`, -`--last-failed`, `--shard`, `--pass-with-no-tests`), prints one line per -test-target pair, `file › title [target] #tag`, skipped pairs ending in -` (skipped: )`, and starts no app, engine, or worker. -`--reporter json` prints `{ "pairs": [...] }`. +`--changed-since`, `--last-failed`, `--shard`, `--pass-with-no-tests`), +prints one line per test-target pair, `file › title [target] #tag`, skipped +pairs ending in ` (skipped: )`, and starts no app, engine, or +worker. With `--changed-since`, the configured filters run as for `run`; a +pair one left out ends in ` (left out by : )`. `--reporter json` prints +`{ "pairs": [...] }`, naming a skip or left-out reason `reason` and the +filter that left a pair out `filter`. + +`import { list } from 'e2e/runner'` returns that selection plus the pairs a +filter removed, positional arguments that matched no file, and the selected +target names. ```bash npx e2e list tests/signup.e2e.ts --tag smoke --reporter json ``` -In this repository prefer Nub wrappers: `nub run test:e2e --headed` or -`nub exec e2e run --headed`. Project scripts live in `package.json` as -`test:e2e` and related `test:e2e:*` entries; invoke them with `nub run`. +`import { run } from 'e2e/runner'` executes the suite and returns its report +without exiting the caller. Pass `cwd`, `config`, and `files` to select the +project and tests, and `changedSince` to run the configured test filters on +the change since that ref; +`onEvent` receives progress and `interruptSignal` cancels +with teardown. A runtime config uses one in-process worker. Await each run +before starting the next and close every host first; overlaps reject with +`RUNNER_BUSY`. + +`import { explore } from 'e2e/runner'` runs one exploration the same way: +`cwd`, `config`, `goal`, `target`, `agent`, `session`, `maxSteps`, and +`timeoutMs` as `e2e explore` takes them, plus `onEvent` and +`interruptSignal`. It resolves with the run's result and `explore`, the +record of the steps, findings, and why it ended. A goal or budget out of +range, an unknown target or agent, or an agent with no model rejects before +it starts (`INVALID_CONFIG`, `UNKNOWN_TARGET`, `MODEL_UNAVAILABLE`); branch +on `isE2EError(error)` and `error.code`. It shares `run`'s reservation. + +With a `package.json` script `"test:e2e": "e2e run"`, pnpm forwards `--` +literally: `pnpm test:e2e -- --headed` reaches e2e as `run -- --headed` and +exits 2. Write `pnpm test:e2e --headed` or `pnpm exec e2e run --headed`. +With npm, use `npm exec -- e2e run --headed` to pass flags to e2e. ## The replay cache Entries live under `.e2e/cache/`, one file per key, named by key digest. +`E2E_CACHE_DIR` or `--cache-dir ` (on `run` and `cache`) replaces +`cache.dir` (file store only), so worktrees can share one directory; the mode +stays as configured. `cache` commands read the same config as `run` (`--config`, `cache.dir`); with a custom `cache.store` they refuse (exit 2), inspect that store with its own tools. @@ -84,40 +133,48 @@ its own tools. ## Output `` (`.e2e` by default) holds `report.json`, `junit.xml`, -`summary.md`, `failures/`, `ai-trace.json`, `sessions/`, and `artifacts/` -(screenshots, Playwright traces, videos, `--debug` transcripts, downloads). -`artifacts/` is cleared once a run's tests start; a run stopping before +`summary.md`, `ai-trace.json`, `sessions/`, and `results/`, one directory +per test with its `trace.md` and an `attempt-/` per attempt +(screenshots, videos, `--debug` transcripts, downloads). +`results/` is cleared once a run's tests start; a run stopping before leaves the last run's files, and a `--last-failed` rerun keeps the files -the report it reruns names and writes its own under `artifacts/rerun-/`. The report records every artifact path, a +the report it reruns names and writes its own under each test's `rerun-/`. The report records every artifact path, a hosted service's video by URL. -- `list` (default): setup steps, one line per file and target, a `Failed +- `list` (default): setup steps, one line per file and target (when stdout + is not a terminal, each test also prints as it finishes), a `Failed Tests` section (error, code, failing line, code frame), then the summary rows `Test Files`, `Tests`, `AI`, `Cache` (when the replay cache ran), `Repeats` (with `--repeat-each`), `Errors`, `Start at`, `Duration`, - `Report`, `AI trace` (with `--ai-trace`). Past a minute `Duration` + `Traces` (when a page was written), `Screenshots` (the first 10 + `app.screenshot` files, then a count), `AI trace` (with `--ai-trace`), and + last `Failed`, one entry per failed test (file and line, error's first + line, trace page when written), so `| tail` keeps it. Past a minute `Duration` repeats as minutes and seconds, setup time split out as `startup` in the same parenthetical (`682.97s (11m 23s, startup 43.00s)`). -- `report.json`, written whatever the reporters, holds `run.status`, +- `report.json`, for scripts rather than reading (`jq`, CI tools; a reader + takes the terminal and the trace pages), the one record of every result, + passing ones included, written whatever the reporters, holds `run.status`, `run.exitCode`, `run.errors[]` (run-level, such as `APP_UNREACHABLE`), and `run.results[]`, one per test and target: `titlePath`, `file`, `source`, `tags` (`[]` when none), `agent`, `repeat` (0 unless `--repeat-each`), `selected`, `status`, `attempts[]` of `steps[]`, `artifacts[]`, `error`. `run.summary` counts the selected results: `passed`, `failed` (timed out included), `interrupted` (stopped by the - run: a signal, `--max-failures`, a run-level error; not a failure), `flaky`, and `skipped` add up + run: a signal, `--max-failures`, `--max-duration`, a run-level error; not a failure), `flaky`, and `skipped` add up to `selected`; `discovered - selected` were left out. - `junit`: `junit.xml` for CI summaries; `--reporter list,junit` keeps the terminal output. -- `markdown` (`--reporter list,markdown`): `summary.md` plus one page per - failed or flaky test under `failures/` (an interrupted test gets none). The summary holds counts and - spend, a block per failed test (error, facts, failing step, whether every - attempt failed alike, last model turns, screen location and closest - nodes, the line to look at, evidence paths), the flaky tests folded - alike, and every test as one folded table, a row per file (or an - exploration's findings and assessment); a failure page adds every step, - every kept turn, and the screen at failure inline. Read the page first; - paste the summary into a pull request or handoff. +- Every run that reaches its tests also writes a trace page per failed, + timed-out, interrupted, or flaky test, `results//trace.md` (the + `trace` mode, default `retain-on-failure`): every step with what + it did, the cache's decisions, what the app logged, the last model turns, + and the screen at failure. The `list` output names it under each failure + (`❯ trace `). Read it first. +- `markdown` (`--reporter list,markdown`): `summary.md` with counts and + spend, a block per failed test linked to its page, the flaky tests folded + alike, and every test as one folded table (or an exploration's findings + and assessment). Paste it into a pull request or handoff. - `json`: the report on stdout. - Custom reporters get step progress with `identity` (`attemptId`, `attemptIndex`, `stepId`, `stepIndex`: report IDs, zero-based indexes; @@ -141,7 +198,7 @@ hosted service's video by URL. | 130 | Interrupted by an external signal | The highest code present wins (`130 > 4 > 3 > 2 > 1 > 0`). 130 needs an -external signal: a `--max-failures` stop or a run-level error that +external signal: a `--max-failures` or `--max-duration` stop or a run-level error that interrupted workers keeps the failures' or the error's code. Exit 2 is deterministic, never retry it; only exit 3 is worth a job-level retry. @@ -153,7 +210,7 @@ third kills the app process groups and exits 130 at once. ## Continuous integration CI mode is on when `CI` is set (not `0` or `false`): `retries` 1, -`workers` 1, `trace` `on-first-retry`, `test.only` rejected with +`workers` 1, `test.only` rejected with `ONLY_IN_CI`, the replay cache `read-only` unless the config sets a mode explicitly (`cache: 'read-write'` or `cache.mode`), `reuseExisting` ignored. @@ -173,12 +230,14 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: pnpm/action-setup@9fd676a19091d4595eefd76e4bd31c97133911f1 # v4.2.0 - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: - node-version: 24 # SolZero pins Node 24.15 - - run: nub install --frozen-lockfile - - run: nub run test:e2e:install - - run: nub run test + node-version: 26 # Node >= 24.8, or >= 22.22.3 on 22 + cache: pnpm + - run: pnpm install --frozen-lockfile + - run: pnpm exec e2e-web install chromium --with-deps + - run: npx e2e run --reporter list,junit env: AI_GATEWAY_API_KEY: ${{ secrets.AI_GATEWAY_API_KEY }} E2E_USER_ADMIN_USERNAME: ${{ secrets.E2E_USER_ADMIN_USERNAME }} @@ -195,7 +254,7 @@ jobs: uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: e2e-artifacts - path: .e2e/artifacts + path: .e2e/results if-no-files-found: warn retention-days: 7 ``` @@ -210,3 +269,20 @@ jobs: (`AI_GATEWAY_API_KEY` for `gateway()` from `ai`) as a secret in the run step's `env`, and commit `.e2e/cache/` so recorded steps replay with no model call. +- To run only the tests a pull request can affect, install + `@e2e-dev/decision ai`, add + `filters: [decisionFilter({ model: gateway.decisionModel('typesafe-ai/jev') })]` + with `decisionFilter` from `@e2e-dev/decision` and `gateway` from `ai`, + pass `AI_GATEWAY_API_KEY` in the run step's `env`, give + `actions/checkout` `fetch-depth: 0`, and run + `npx e2e run ${{ github.event_name == 'pull_request' && format('--changed-since origin/{0}', github.base_ref) || '' }}`. + Filters run only with `--changed-since `, which that expression + passes on `pull_request` alone; never pass it on `pull_request_target`. + Every test runs without it and on a `--last-failed` rerun. The decision + filter also runs every test on a lockfile or `e2e.config` change, and the + tests whose test file, setup file, or stored screenshots changed. On GitLab, pass `--changed-since ${CI_MERGE_REQUEST_TARGET_BRANCH_SHA:-$CI_MERGE_REQUEST_DIFF_BASE_SHA}` + on merge request pipelines, not merge trains, with `GIT_DEPTH: 0`. Preview + with `npx e2e list --changed-since origin/main`, after committing: the + decision filter reads what the commits change, and a file only the + working tree changes by name alone. + Reference: https://e2e.com/docs/filters diff --git a/.agents/skills/e2e/references/setup.md b/.agents/skills/e2e/references/setup.md index 8fbf9acc..1d2c4f5f 100644 --- a/.agents/skills/e2e/references/setup.md +++ b/.agents/skills/e2e/references/setup.md @@ -1,15 +1,29 @@ # Setting up e2e +## Contents + +- Requirements +- Scaffold +- Subscriptions and API keys +- The config +- The app under test +- Environment variables the runner reads +- Mobile targets +- Done when + ## Requirements -- Node.js 22.12 or newer. +- Node.js 24.8 or newer, or 22.22.3 or newer on Node.js 22. - ES modules: `.ts` config, tests, helpers, and workspace packages exporting `.ts` source load as ESM regardless of the nearest `package.json` `type` (CommonJS packages need no change); never `require` or `module.exports`. + Imports follow TypeScript: `./x.js` or `./x` loads `x.ts`, and the nearest + `tsconfig.json` `paths` and `baseUrl` apply. Decorators need + `experimentalDecorators`; CommonJS TypeScript goes in `.cts`. - Browser tests: `@e2e-dev/web`, pinning `playwright-core` exactly; do not add `playwright` for it. Missing browsers download on first boot; in CI run - `nub run test:e2e:install` (or `nub exec e2e-web install chromium --with-deps`). - Mobile tests: `@e2e-dev/mobile`, pinning + `npx @e2e-dev/web install chromium --with-deps` (pnpm: `pnpm exec e2e-web + install chromium --with-deps`). Mobile tests: `@e2e-dev/mobile`, pinning `agent-device` exactly; each pin moves with its engine release. ## Scaffold @@ -17,18 +31,30 @@ Fresh project: ```bash -nub exec e2e init +npx e2e init # npm +pnpm dlx e2e init # pnpm ``` -With `e2e` installed, run the installed version through Nub: `nub exec e2e init`; -`nub exec e2e init my-app` scaffolds into a new directory. +With `e2e` installed, run the installed version: `npx e2e init` or `pnpm exec +e2e init`; `npx e2e init my-app` scaffolds into a new directory. -The wizard picks an engine and a model provider (None for tests without AI) +The wizard picks the engines (web, mobile, or both) and a model provider (None for tests without AI) and offers to install this skill, register the MCP server for your coding -agent, and install dependencies. `--yes` picks Playwright and Vercel AI +agent, and install dependencies. `--yes` picks the web engine (Playwright) and Vercel AI Gateway, installs the skill in `.agents/skills/` (`.claude/skills/e2e` symlinks to it), registers MCP in `.mcp.json` and `.cursor/mcp.json`, skips -installing dependencies. +installing dependencies. A flag per question overrides one default: +`--engine` (`web,mobile` for both), `--gateway`, `--endpoint`, `--skills`, +`--mcp` (comma-separated locations, or `none`), `--install`; `npx e2e init --help` lists the values. +To start a new app from one of the repo's examples instead, run `npx e2e init +my-app --example with-vite` (or `with-expo`, `with-flutter`, ...): it +downloads the example into an empty directory with its config and tests, then +asks only for the skill, MCP, and install. + +The wizard needs a terminal: without one, `init` exits with code 2. From an +agent's shell, run `npx e2e init --yes` when its defaults fit and install the +dependencies after it, or install by hand (below). Neither registers the MCP +server for Codex (topic `mcp`). Init adds dependencies and a `test:e2e` script to `package.json`, writes `e2e.config.ts` and `tests/example.e2e.ts`, updates `.gitignore`, leaves @@ -52,6 +78,7 @@ key, or a local endpoint. Authenticate: | GitHub Copilot | `npx e2e login github-copilot` (GitHub CLI signed in, or your own `--client-id`) | | OpenCode Console (OpenCode Zen and OpenCode Go) | `npx e2e login opencode-console` (approve the device code, pick the workspace) | | SuperGrok or X Premium+ | `npx e2e login spacexai` | +| Claude Max or Team | No `e2e login`: link a Claude Console organization in claude.ai for the plan's monthly API credits (Max: Settings > Billing; Team: an Owner under Organization settings > Billing), create an API key in it, set `ANTHROPIC_API_KEY`, use `anthropic('')` from `@ai-sdk/anthropic` | | Vercel AI Gateway | Set `AI_GATEWAY_API_KEY`, or sign in to the Vercel CLI and `npx vercel link`; without the key `gateway()` uses a Vercel OIDC token | | OpenRouter | Set `OPENROUTER_API_KEY` | | Local or self-hosted endpoint | Set the endpoint URL and a model it serves, plus a key if required | @@ -67,15 +94,20 @@ Switching to Copilot: install `ai`, `@ai-sdk/openai-compatible`, and over Copilot's Responses API when the plan serves that model only there, choosing per model from the plan's listing. `npx e2e models github-copilot` marks the models it cannot call at all: those served only over an API `copilot()` does not speak, and -those the plan has not enabled. +those the plan has not enabled. Whether the plan serves a model shows nowhere: on +Copilot Free, GitHub serves models through auto model selection only, so a model +called by id is usually refused, and the run stops with `MODEL_UNAVAILABLE` naming +it. Pick another id or an API provider. Switching to OpenCode Console: install `ai`, `@ai-sdk/openai-compatible`, `@ai-sdk/openai`, `@ai-sdk/anthropic`, and `@ai-sdk/google`, set `model: opencodeConsole('')` from `e2e/oauth/opencode-console`, run `npx e2e login opencode-console`. A bare id is an OpenCode Zen model; a `go/` id -(`go/deepseek-v4.1-flash`) is an OpenCode Go model and needs the workspace's Go -subscription. `npx e2e models opencode-console` lists the ids, tagged Zen or Go. In CI, -set a Console service account key as `OPENCODE_API_KEY`. +(`go/deepseek-v4.1-flash`) is an OpenCode Go model and needs an OpenCode Go +subscription: the workspace's through the login, or yours through +`OPENCODE_API_KEY`. `npx e2e models opencode-console` lists the ids, tagged +Zen or Go, through the key when it is set. In CI, set a Console service account +API key or an OpenCode Go subscription API key as `OPENCODE_API_KEY`. ## The config @@ -94,7 +126,7 @@ export default { engine: web(), app: { url: 'http://127.0.0.1:3000', - command: { executable: 'nub', args: ['run', 'dev'], log: '.e2e/logs/app.log' }, + command: { executable: 'pnpm', args: ['dev'], log: '.e2e/logs/app.log' }, }, }, ], @@ -127,21 +159,22 @@ or `E2E_USER_ADMIN_PASSWORD` first, or defer to fill time with | `retries` | `0`, `1` in CI | 0 to 10. | | `workers` | half the cores, `1` in CI | Parallel test files, capped by the `workers` the engine declares per target. | | `reporters` | `['list']` | `list`, `json`, `junit`, `markdown`, or `{ name, onEvent?, onRunFinished? }` objects; `json` excludes `list`, `--reporter` keeps the objects. | -| `cache` | `'read-write'` | `'read-write'`, `'read-only'`, `'off'`, or `{ mode, store, dir, strict }`; CI demotes only a defaulted mode to `'read-only'`. `strict` (`--strict-cache`) fails a step whose recording no longer replays (`REPLAY_STALE`) instead of handing it to the agent. | +| `cache` | `'read-write'` | `'read-write'`, `'read-only'`, `'off'`, or `{ mode, store, dir, strict }`; CI demotes only a defaulted mode to `'read-only'`. `strict` (`--strict-cache`) fails a step whose recording no longer replays (`REPLAY_STALE`) instead of handing it to the agent; retries replay too, and the run never writes the cache. | | `agents` | `{ default: built-in }` | Tests run with `default`, `e2e run --agent ` picks another, entries never inherit from `default`. Options: topic `agent`. | +| `filters` | `[]` | Test filters `{ name, filter(tests, context) }`, unique names, applied in order with `--changed-since `. Use `filters: [decisionFilter({ model: gateway.decisionModel('typesafe-ai/jev') })]`, with `decisionFilter` from `@e2e-dev/decision` and `gateway` from `ai` (install `@e2e-dev/decision ai`), to run only the tests a pull request can affect. `filter` gets candidates `{ id, file, titlePath, tags, setupFile? }` and a context `{ projectRoot, changes: { base, root, files }, configFile, signal, log, redact }` with repository-relative `files`, and returns `{ exclude: { [id]: reason }, note? }`; every test it does not name runs. Read files, the environment, or the network inside `filter`, never at config load: workers load the config too. When filters run: topic `running`. | | `credentials` | `{}` | Named `{ username, password }`; `password` is a 6+ code point string or a function returning it. | | `secrets` | `{}` | Named values the model never sees (API keys, tokens), same value rule. A separate namespace: a credential's password is `credentials.user(name).password` (named `.password`), never `secrets.get()`, so a name may be both. | | `output` | `'.e2e'` | Results directory; `--output ` for one run. Inside the project root, not the root, not a tests glob's directory, never the cache dir (`cache.dir` stays `.e2e/cache`). | | `artifacts` | none | `{ store }`: artifacts go to the host `ArtifactStore` (`{ put(artifact), putLink?(link) }`); `putLink` gets provider-hosted video links (never a passed `retain-on-failure` attempt's). Failure screenshots are always captured when the engine can. | -| `trace` | `'on'`, `'on-first-retry'` in CI | Attempts that record a Playwright trace: `'off'`, `'on'`, `'retain-on-failure'`, `'on-first-retry'`, `'on-all-retries'`; precedence and capability rule as `video`. | -| `video` | `'off'` | Same modes; `'retain-on-failure'` records all, keeps those that did not pass. Precedence: the test's `video`, `--video [mode]`, the target's (`{ engine, video }`), the config's. | +| `trace` | `'retain-on-failure'` | Tests that keep a trace page (`/results//trace.md`): `'off'`, `'on'` (every test), `'retain-on-failure'` (failed, timed-out, flaky), `'on-first-retry'`, `'on-all-retries'`; precedence as `video`. Works on every engine. | +| `video` | `'off'` | Attempts that record, same modes; `'retain-on-failure'` records all, keeps those that did not pass. Precedence: the test's `video`, `--video [mode]`, the target's (`{ engine, video }`), the config's. | | `projectId` | the package name | Report and cache identity. | - `tests` discovery enters only directories a glob can match; symlinks are not followed. -- `trace` and `video`: a config or flag mode skips targets whose engine - cannot record (one notice), a target or test mode requires it - (`UNSUPPORTED_ARTIFACT`); a retry mode with `retries: 0` prints a notice; +- `video`: a config or flag mode skips targets whose engine cannot record + (one notice), a target or test mode requires it (`UNSUPPORTED_ARTIFACT`). +- `trace` and `video`: a retry mode with `retries: 0` prints a notice; neither invalidates the replay cache. ## The app under test @@ -155,8 +188,9 @@ target's `app`: | `url` | Base URL for `app.open()` and relative navigation; required on a `web()` target, not supported on a mobile target yet. Missing scheme: `https://`, `http://` for loopback; port `0` on `127.0.0.1` or `[::1]` takes a free port. | | `bundleId` | Device targets: the bundle id, package name, or display name (`Settings`) `app.open()` launches. | | `appPath` | Device targets: the `.app` or `.apk` under test. A device target needs `bundleId` or `appPath`. | -| `launchArguments`, `permissions` | Device targets: arguments and permission states (`grant`, `deny`, `reset`) every fresh launch gets. | -| `command` | The process serving `url`: `{ executable, args, cwd, env, startupTimeout, shutdownTimeout, log, reuseExisting }`; `{port}` in `args` and `env` expands to `url`'s port. Targets declaring the same command share one process. | +| `launchArguments` | Device targets: arguments every fresh launch gets. | +| `permissions` | Device targets: states (`grant`, `deny`, `reset`) every fresh launch gets. Web targets: `grant` only, held by every page (`{ 'clipboard-write': 'grant' }` lets headless Chromium write the clipboard). | +| `command` | The process serving `url`: `{ executable, args, cwd, env, inheritEnv, startupTimeout, shutdownTimeout, log, reuseExisting }`; `{port}` in `args` and `env` expands to `url`'s port. Targets declaring the same command share one process. | | `readyUrl` | Readiness probe when it differs from `url`; `{port}` expands too. | | `environment` | `'test'`, `'staging'`, `'production'`; inferred from the host, labels the report and cache key. | | `identity` | Stable identity for cache and session keys when the origin changes per deploy (preview URLs). Defaults to the URL's origin and path, else `bundleId`, else `appPath`. | @@ -176,13 +210,14 @@ start a script that brings them up and serves the app. | Option | Meaning | | --- | --- | -| `browser` | `'chromium'` (default), `'firefox'`, `'webkit'`, or a `BrowserProvider` leasing hosted browsers over CDP (`kernel()` from `@e2e-dev/kernel`, or your own), which implies chromium and excludes `connect`. Scope `'worker'` (default): one browser per worker slot from `prepare` to `finish`; `'attempt'`: one per attempt, with `reconnectEndpoint`'s limits. | +| `browser` | `'chromium'` (default), `'firefox'`, `'webkit'`, or a `BrowserProvider` leasing hosted browsers over CDP (`kernel()` from `@e2e-dev/kernel`, `smol()` from `@e2e-dev/smol`, `lightpanda()` from `@e2e-dev/lightpanda`, `testmuBrowsers()` from `@e2e-dev/testmu/web`, or your own), which implies chromium and excludes `connect`. Scope `'worker'` (default): one browser per worker slot from `prepare` to `finish`; `'attempt'`: one per attempt, with `reconnectEndpoint`'s limits (`lightpanda()` and `testmuBrowsers()` are worker scope only). | | `viewport` | `{ width, height }`, default 1280x720; `null` follows the browser window (hosted live view, headed run). On a headed hosted browser (Kernel) use `null` and size the service's screen; a fixed size gives a smaller, unmaximized window. | -| `connect` | `{ cdpEndpoint }` attaches to a remote Chromium over CDP; both it and `reconnectEndpoint` are resolvers `(signal) => url`, not strings. With `reconnectEndpoint` it rides one persistent default context and reconnects only to the original browser and page. | +| `connect` | `{ cdpEndpoint }` attaches to a remote Chromium over CDP; `cdpEndpoint` is required, and both it and `reconnectEndpoint` are resolvers `(signal) => url`, not strings, else `INVALID_CONFIG` at load. With `reconnectEndpoint` it rides one persistent default context, needs a fresh browser from `cdpEndpoint` every attempt, and reconnects only to the original browser and page. | | `headers` | Sent to the app's site only (Vercel's `x-vercel-protection-bypass`, ngrok's `ngrok-skip-browser-warning`), `agent.act` included; disables the browser HTTP cache and service workers. | | `basicAuth` | `{ username, password }` for a `401` challenge; `password` may be `secrets.get('name')`, resolved per attempt and redacted like any secret, the base64 `Authorization` credential too. | | `userAgent` | The `User-Agent` every attempt sends and `navigator.userAgent` reports. | | `locale`, `timezoneId` | The language (`'de-DE'`: `navigator.language`, `Intl`, `Accept-Language`) and IANA time zone (`'Europe/Berlin'`) every attempt runs in. | +| `colorScheme`, `reducedMotion` | The `prefers-color-scheme` (`'light'`, `'dark'`, `'no-preference'`) and `prefers-reduced-motion` (`'reduce'`, `'no-preference'`) every page reports to its `@media` rules and `matchMedia`, from first paint. | | `initScripts` | Scripts every document runs before the page's own: source, `{ path }`, or a function with no closures. | | `testIdAttribute` | What `getByTestId` reads; default `data-testid`. | | `screencast` | `{ size?, quality? }` for the engine's own video: frame size (default the viewport's), JPEG quality 0 to 100. | @@ -219,8 +254,8 @@ engine: web(), app: { url: 'http://127.0.0.1:3000', command: { - executable: 'nub', - args: ['run', 'dev'], + executable: 'pnpm', + args: ['dev'], env: { PORT: '3000', DATABASE_URL: process.env.DATABASE_URL ?? '' }, startupTimeout: 120_000, log: '.e2e/logs/app.log', @@ -231,10 +266,14 @@ app: { - The runner spawns `command`, polls `readyUrl` (default `url`) for a 200 to 499 status within `startupTimeout` (default 60 s), and stops it when the run ends, fails, or is interrupted (`shutdownTimeout`, default 10 s). Never - ready is `APP_UNREACHABLE`; `.e2e/report.json` is still written. + ready is `APP_UNREACHABLE`; the run stops before its tests and leaves the + last run's report and results as they were (`command.log` still gets this + boot's output). - The child gets only `PATH`, `HOME`, the temp-directory variables, - `SystemRoot` and `COMSPEC` on Windows, and `command.env`; pass the rest - through `env`. Model keys and `E2E_USER_*` values are never inherited. + `SystemRoot` and `COMSPEC` on Windows, and `command.env`. Pass the rest + through `env`, or name host variables in `inheritEnv` (`['DATABASE_URL', + 'STRIPE_*']`); `env` wins over them. Model keys and `E2E_USER_*` values + are never inherited unless an `inheritEnv` name or prefix matches them. - Output is discarded unless `log` names a file under an ignored directory such as `.e2e/logs/`; without it a server dying on boot is invisible. - A `url` answering before the spawn is `APP_ALREADY_RUNNING`; @@ -272,7 +311,7 @@ For an app started elsewhere, point `app.url` at it, literally or via phones through [agent-device](https://github.com/callstack/agent-device); needs Xcode with a simulator runtime or the Android SDK with an emulator (a phone needs Xcode with Developer Mode and runner signing, or adb with USB -debugging authorized); run `npx agent-device doctor` once. Name a phone in `device` by the name +debugging authorized); run `npx e2e doctor` to check it. Name a phone in `device` by the name `npx agent-device devices` lists, not its UDID or serial. On an iPhone, device settings (permissions, `clearState`, network, location, appearance, biometrics, keychain) and the clipboard are simulator-only. @@ -312,27 +351,66 @@ export default { `app.command` (`npx expo start --port 8081`) with `readyUrl: 'http://localhost:8081/status'` starts it. - One worker per device. No `device`: every booted simulator or emulator of - the platform, and every connected phone, is the pool, up to `workers` (none booted: agent-device boots - one); one `device`: one worker whatever `workers` says; a list (`device: + the platform, and every connected phone, is the pool, up to `workers`, + minus devices another agent-device session holds (none booted: agent-device boots + one; all held: `prepare` fails); one `device`: one worker whatever `workers` says; a list (`device: ['iPhone 17', 'iPhone 17 Pro']`): an explicit pool. Devices boot in `prepare`, before the run's clock. Two sessions on one device fight over it: for parallel MCP sessions or bug-bash explorers, declare one target per device, each naming its `device`. - `device` can be a `DeviceProvider` leasing hosted devices, one per worker - slot: `easSimulators({ buildId })` from `@e2e-dev/eas` takes the project + slot: `expoSimulators({ buildId })` from `@e2e-dev/eas` takes the project from the app config's `extra.eas.projectId` (`projectId` overrides), reads - `EXPO_TOKEN`, else the `eas login` session, needs no Xcode or Android SDK, and with `buildId` EAS installs + `EXPO_TOKEN`, else the `eas login` session, needs no Xcode or Android SDK, and with `buildId` Expo installs the app (omit `app.appPath`). A run must fit one session: `maxDurationMinutes`, absent, is the account's cap (40 on a standard plan). `videoTouches: false` on the engine for video there. -- Only a control that appeared or moved with the previous action waits out - `transition` (default 500 ms); agent actions settle `settle` ms (default +- `limrun()` from `@e2e-dev/limrun` leases one device per worker and installs + `app.appPath` before tests. Set `LIMRUN_API_KEY`; install `@limrun/api` and + matching `agent-device` peers. Android needs `adb` on `PATH` for its tunnel. + iOS keyboard actions such as `locator.press('Enter')` return + `UNSUPPORTED_CAPABILITY`; tap an app button when available. + Limrun's `limrun-inc/testerarmy-e2e-limrun` repository runs the same form + and restart tests on both platforms. +- `testmu({ device, osVersion, app })` from `@e2e-dev/testmu` leases TestMu + AI (formerly LambdaTest) emulators, simulators, or real devices + (`deviceType: 'real'`); `device` and `osVersion` match its catalog + exactly, TestMu AI installs `app` (omit `app.appPath`), and it reads + `LT_USERNAME` and `LT_ACCESS_KEY`. An attempt's video links TestMu AI's + recording of the whole session. +- `browserstack({ device, osVersion, app })` from `@e2e-dev/browserstack` + leases BrowserStack App Automate real devices; `device` and `osVersion` + match its catalog exactly, BrowserStack installs `app` (a `bs://` id, an + `https` URL uploaded once per run, or a local path; omit `app.appPath`), + and it reads `BROWSERSTACK_USERNAME` and `BROWSERSTACK_ACCESS_KEY`. Set + `appiumVersion` (such as `'3.2.0'`): BrowserStack's default Appium 1.x + lacks `mobile:` commands. An attempt that records video links + BrowserStack's recording of the whole session. +- A control that appeared or moved with the previous action waits out + `transition` (default 500 ms). Opening or closing the keyboard also uses + the remaining budget. Agent actions settle `settle` ms (default 150) before the next observation, `settle: false` skips it. +- Mobile `fill` and `clear` re-read the field before typing, replay included. + Missing fields, changed security state, or incomplete captures return + `NODE_STALE` before mutation, then retry after a fresh observation. + Non-input targets return `UNSUPPORTED_CAPABILITY`; an iOS `other` that holds + a value or only text is typed into, since a custom-drawn field (Compose + Multiplatform, Flutter) can read that way. - `screen`, `expect`, `app`, `agent` work unchanged; `test` from `@e2e-dev/mobile` types the `device` fixture (`installApp`, `openLink`, `setPermission`, `setNetwork`, `setAppearance`, `clearKeychain`, `fold` - for an iPhone Duo's hinge, `locator`, more). Portable suites declare + for a foldable's hinge (iPhone Duo, Android foldable emulators), `locator`, more). Portable suites declare `requires: ['device']`. +- `device.unstable_raw` gives the session name, device id and name, and an + agent-device `client` bound to the test's session, for what `device` + lacks. Client calls are not steps: no report, trace, replay, or secret + redaction. Unstable: may change or go away in any release. +- `setNetwork('offline')` and `setAirplaneMode(true)` are Android only: on + iOS both are `UNSUPPORTED_CAPABILITY`, since a simulator shares the Mac's + network, and `setNetwork('online')` / `setAirplaneMode(false)` do nothing. + `setNetwork('offline')` turns off Wi-Fi alone and cellular stays up; an + offline test uses `setAirplaneMode(true)` (Android 11+) under + `platforms: ['android']` and turns it back off in `afterEach`. - No `state` capability: `test.setup` and `session` are unavailable; sign in per test with `screen` actions or `agent.act`, both fill a `Secret` (topic `writing-tests`, Sign-in sessions). @@ -353,3 +431,8 @@ export default { - `.gitignore` lists the `.e2e/` outputs (init adds them); drop the `.e2e/cache/` line to commit replays. - CI runs the whole suite, agent steps included, on PRs; see `running`. + +### Existing MCP registration + +Re-running `init` preserves the e2e MCP server's environment and client settings +while repairing its launch command. A current registration is left unchanged. diff --git a/.agents/skills/e2e/references/writing-tests.md b/.agents/skills/e2e/references/writing-tests.md index 9a2b719c..e968ac7c 100644 --- a/.agents/skills/e2e/references/writing-tests.md +++ b/.agents/skills/e2e/references/writing-tests.md @@ -1,5 +1,17 @@ # Writing tests +## Contents + +- A complete file +- Registration +- Fixtures +- Locators +- expect +- Sign-in sessions +- Start near the screen under test +- The browser fixture (browser only) +- Habits + ## A complete file ```ts @@ -60,12 +72,17 @@ test.skip('later', async () => {}); test.only('focus', async () => {}); // local only: CI fails with ONLY_IN_CI test('conditional', async () => { test.skip(await onlyOneOrg(), 'nothing to switch to'); }); // throws: the body stops here, reported skipped; call it before the first step test('later', async () => { test.skip('waiting on the API'); }); // bare skip from the body +test.fail('known gap', async () => {}); // expected to fail: a test failure passes it, a pass fails it with UNEXPECTED_PASS +test('gap on web', async ({ platform }) => { test.fail(platform === 'web', 'known gap'); }); // marks, does not throw; the body runs on test.setup('sign in', { sessions: ['admin'] }, async ({ app, screen, session }) => {}); // see Sign-in sessions const wsTest = test.extend<{ ws: Ws }>({ ws: async ({ browser }, use) => { await use(await seed()); await drop(); } }); wsTest('uses the workspace', async ({ ws }) => {}); // code after use() is teardown, runs after failures too ``` -A setup test cannot skip from its body (`INVALID_ARGUMENT`). +A setup test cannot skip from its body (`INVALID_ARGUMENT`), and neither a +setup test nor a serial member can call `test.fail`. `test.fail` covers test +failures and timeouts only: an infrastructure error such as `ENGINE_FAILURE` +or a blocked agent verdict still fails the test. | Option | Default | Notes | | --- | --- | --- | @@ -73,16 +90,17 @@ A setup test cannot skip from its body (`INVALID_ARGUMENT`). | `retries` | `config.retries` | 0 to 10; a serial group's value applies to its members. | | `tags` | `[]` | Distinct, non-blank, no comma or edge whitespace (`'Login Form'` is fine); union across layers. `--tag smoke` selects, `--tag-mode all` requires every tag. | | `skip` | unset | `true` or a reason string. | +| `fail` | unset | `true` or a reason string: expected to fail, as `test.fail`. Tests only, not groups or setup tests. | | `platforms` | unset | Only targets with these platforms, e.g. `['ios']`. | | `requires` | `[]` | Engine capabilities, e.g. `['browser']`; missing ones skip the test at selection rather than failing it with `UNSUPPORTED_CAPABILITY`. | | `session` | unset | Restore state saved by a setup test. | | `agentContext` | unset | Extra context for `agent.*` calls in this test or group. | | `agent` | the run's agent | A configured name (`agents.`) or a list run once per agent; `--agent` narrows the list, a setup test takes one name. Innermost wins; `agent.act(..., { agent })` names another for one call. | -| `trace`, `video` | the target's | `'off'`, `'on'`, `'retain-on-failure'`, `'on-first-retry'`, `'on-all-retries'`. Innermost wins over `--trace` / `--video`, the target, and the config; recording where the engine cannot is `UNSUPPORTED_ARTIFACT` for the run. | +| `trace`, `video` | the target's | `'off'`, `'on'`, `'retain-on-failure'`, `'on-first-retry'`, `'on-all-retries'`: whether the test keeps a trace page, which attempts record video. Innermost wins over `--trace` / `--video`, the target, and the config; video where the engine cannot record is `UNSUPPORTED_ARTIFACT` for the run. | | `serial` | `false` | Groups only. Members share app state, run in order on one worker, retry as a whole, and take the group's `trace` and `video`. | Serial members cannot set `retries`, `trace`, `video`, `session`, -`platforms`, `requires`, `skip`, or `only`, nor can a nested `describe` set +`platforms`, `requires`, `skip`, `fail`, or `only`, nor can a nested `describe` set `trace` or `video`; nesting serial groups is a `COLLECTION_ERROR`. Hooks nest: outer `beforeEach` first, inner `afterEach` first. `beforeAll` @@ -106,7 +124,8 @@ label. `browser` (`Browser`): browser targets, import `test` from `@e2e-dev/web` ### app - `baseUrl` (`string | undefined`): the target's app URL with the run's - port; `undefined` when the target declares no `app.url`. + port and no trailing slash (`${app.baseUrl}/api/users`); `undefined` when + the target declares no `app.url`. `string` with `test` from `@e2e-dev/web`. - `open(path?)`: opens the target's `app.url`, a relative path, or an absolute http(s) URL. On a device it takes no path and relaunches the pinned app. - `back()`: one history step back. @@ -115,7 +134,11 @@ label. `browser` (`Browser`): browser targets, import `test` from `@e2e-dev/web` - `clearState()`: clears cookies and storage, recreates the context, reopens the base URL. - `screenshot(label?)`: saves a redacted screenshot artifact and returns its - path. Denied after a secret fill (see Sign-in sessions). + absolute path; the `list` summary names the first 10 under `Screenshots`. It masks + secure fields only (see Sign-in sessions). +- `startRecording({ name? })`, then `stopRecording()`: records one span and + returns its videos (`path` or hosted `url`). Fails while the test's + `video` records: set `{ video: 'off' }` on the test. `clearState()` is `UNSUPPORTED_CAPABILITY` with `connect.reconnectEndpoint` or an attempt-scoped browser provider; on a device with no pinned `app`, both @@ -132,8 +155,8 @@ assertion. Every query also exists on a locator, scoped to its subtree. | `getByLabel(text, { exact?, visible? })` | Form controls by label. | | `getByPlaceholder(text, { exact?, visible? })` | Inputs by placeholder. | | `getByText(text, { exact?, visible? })` | Visible text. | -| `getByDisplayValue(value, { exact?, visible? })` | Inputs by current value; on the web it cannot scope child queries or be a `filter({ has })` target. | -| `getByTestId(id, { visible? })` | `data-testid` on the web (or `web({ testIdAttribute })`), accessibility identifier or resource id on a device; a string matches the whole id, a RegExp tests it. Last resort. | +| `getByDisplayValue(value, { exact?, visible? })` | Inputs by current value; on the web it cannot scope child queries or be a `filter({ has })` or `filter({ hasNot })` target. | +| `getByTestId(id, { visible? })` | `data-testid` on the web (or `web({ testIdAttribute })`), accessibility identifier or resource id on a device (React Native `testID`, SwiftUI `.accessibilityIdentifier`, Compose `testTag` with `testTagsAsResourceId` on Android, Flutter `Semantics(identifier:)`, .NET MAUI `AutomationId`, NativeScript `testID`); a string matches the whole id, a RegExp tests it; on Android the `:id/` prefix may be left off. Last resort. | Roles: `button`, `link`, `textbox`, `searchbox`, `combobox`, `listbox`, `option`, `checkbox`, `radio`, `radiogroup`, `switch`, `slider`, `spinbutton`, @@ -158,15 +181,21 @@ echoing its child does not count twice. `exact: false` is a case-insensitive substring; a `RegExp` matches as written. - Exactly one match per action, read, or assertion. Two fail at once with - `LOCATOR_AMBIGUOUS`; at zero, actions and assertions poll until the - timeout then `LOCATOR_NOT_FOUND`, a read fails at once. Exceptions: + `LOCATOR_AMBIGUOUS`; at zero, actions and reads poll until the action + timeout then `LOCATOR_NOT_FOUND`, assertions until the assertion timeout + then `ASSERTION_FAILED`. Exceptions: `toHaveCount`, `toBeVisible`, `toBeHidden`, `toBeAttached`, list-form `toHaveText` and `toContainText`, `isVisible()` (false at zero), `isHidden()` (true), `count()`, `all()`, `allTextContents()`. - Narrow with `filter({ hasText })` (a case-insensitive substring, unlike a - query), `filter({ has: locator })`, `first()`, `last()`, `nth(i)`, or - scoping under another locator. Any other `filter` key (`hasNot`, - `hasNotText`) or an empty `filter({})` is `INVALID_LOCATOR`. + query), `filter({ hasNotText })` to exclude matching subtree text, + `filter({ has: locator })`, `filter({ hasNot: locator })`, `first()`, + `last()`, `nth(i)`, or scoping under another locator. Any other `filter` + key (`visible`) or an empty `filter({})` is `INVALID_LOCATOR`. `a.or(b)` + matches either, still strict: add `first()` when both may match. Both + locators must come from the same screen and attempt. Web branches need + identical literal frame-selector chains; display-value branches cannot be + composed. - `visible: true` drops nodes the page hides (a closed drawer) before the exactly-one rule. Visibility is what renders, as `toBeVisible()` reads it: an `aria-hidden` spinner that paints is visible, so `toBeHidden()` waits @@ -183,14 +212,14 @@ substring; a `RegExp` matches as written. Each action resolves one node, waits up to `config.actionTimeout` (30 s, or `{ timeout }`) for it to be actionable, and does one thing: `tap()` (alias -`click()`), `doubleTap()`, `secondaryTap()` (each takes +`click()`), `doubleTap()` (alias `dblclick()`), `secondaryTap()` (each takes `{ modifiers: ['Shift'] }`), `longPress({ duration? })` (100 to 10000 ms; default 500 on the web, 1000 on a device), `fill(value | Secret)`, `pressSequentially(text, { delay? })`, `clear()`, `press(key)`, `check()`, `uncheck()`, `selectOption(label | { label } | { value } | { index })` (one option; an array is `INVALID_ARGUMENT`), `focus()`, `hover()`, `setInputFiles(paths)` -(relative to the project root), `dragTo(locator)`, `scrollIntoView()`, +(relative to the project root; `[]` clears), `dragTo(locator)`, `scrollIntoView()`, `swipe({ direction, momentum? })`. Web only, `UNSUPPORTED_CAPABILITY` on a device: `secondaryTap`, `selectOption`, `setInputFiles`, `scrollIntoView`, and `modifiers`. @@ -201,27 +230,33 @@ field and types one character per `delay`, plain string only (a `Secret` is `INVALID_ARGUMENT` and goes through `fill`). Coordinates are CSS pixels, for what the tree does not list: `tap({ position: -{ x, y } })` offsets from the node's top-left corner (with `modifiers` it is +{ x, y } })` offsets from the node's top-left corner inside its border (with `modifiers` it is `INVALID_ARGUMENT`), `screen.tapAt({ x, y })` taps a viewport point, -`screen.swipe({ from, to })` swipes along a path, +`screen.swipe({ from, to, duration? })` swipes along a path, `screen.swipe({ direction, momentum? })` swipes the viewport. Prefer a locator; a point moves with the layout. ### Reads -Reads resolve once, no retry: `textContent()`, `inputValue()`, -`getAttribute(name)`, `isVisible()`, `isHidden()`, `isEnabled()`, -`isDisabled()`, `isChecked()`, `boundingBox()`, `count()`. `all()` gives one +`textContent()`, `inputValue()`, `getAttribute(name)`, `isEnabled()`, +`isDisabled()`, `isChecked()`, and `boundingBox()` wait for one match until +the action timeout, then read once with no retry for a value. `isVisible()`, +`isHidden()`, and `count()` answer at once. `all()` gives one `nth(i)` locator per current match and `allTextContents()` every match's text, both `[]` at zero. Text is the rendered text, whitespace collapsed: on the web what `innerText` reads (`text-transform` applies, `display: none` -drops out, `
` is a space); `toHaveText` reads the same. `isChecked()` is -`false`, not an error, on a node with no checked state, so query checkable -controls by role. `waitFor({ state?: 'attached' | 'detached' | 'visible' | 'hidden', timeout? })` waits +drops out, `
` is a space); `toHaveText` reads the same. `isChecked()` on a +node that cannot be checked (a button), and `inputValue()` on one that holds +no value (a heading), throw `ACTION_FAILED`. `waitFor({ state?: 'attached' | 'detached' | 'visible' | 'hidden', timeout? })` waits within `actionTimeout`, else `LOCATOR_NOT_FOUND`. For a value that has to settle use `expect`, not a read. Reading a password field's value or attributes is `POLICY_DENIED`, as is `toHaveAttribute` on one, negated too. +`screen.tree()` returns the whole screen as the agent reads it (one node +per line, secrets redacted) and `screen.debug()` prints it to the test's +output. Both read once without waiting. A locator does not have them, and +a frame throws `INVALID_ARGUMENT`. + ## expect `expect(locator)` polls up to `config.assertionTimeout` (5 s) or @@ -235,11 +270,11 @@ attempt); a throwing read keeps polling, and it is not a report step. A poll that the body or a hook returns without awaiting is cancelled and fails that phase with `STEP_NOT_AWAITED`. `expect.soft(x)` keeps a failure instead of throwing; the attempt fails -after the body with every soft failure listed. If the body calls `test.skip`, -the test stays skipped and the failures remain in `secondaryErrors`. The CLI -shows `Skipped After Failure`, also when a retry skips after an earlier -failure. Set `failOnSkippedFailure: true` in the config to fail the run with -exit code 1 in either case. The default is `false`; clean skips stay green. +after the body with every soft failure listed. If the body then calls +`test.skip`, the attempt still fails, with the skip reason in its `skip`. +A retry that skips after an earlier failed attempt stays skipped; the CLI shows +it under `Skipped After Failure`, and `failOnSkippedFailure: true` in the +config fails the run with exit code 1. The default is `false`. `expect.any(Class)`, `expect.anything()`, `expect.objectContaining(obj)`, `expect.arrayContaining(arr)`, `expect.stringContaining(s)`, and `expect.stringMatching(s | RegExp)` stand in for values inside `toEqual`, @@ -259,7 +294,7 @@ typed; prefer it when the app already has a schema for the response. | Locator matchers | Browser matchers | Value matchers | | --- | --- | --- | -| `toBeVisible`, `toBeHidden`, `toBeAttached`, `toBeEnabled`, `toBeDisabled`, `toBeChecked`, `toBeSelected`, `toBeExpanded`, `toBeFocused`, `toHaveText`, `toContainText`, `toHaveValue`, `toHaveAttribute`, `toHaveCount`, `toHaveAccessibleName` | `toHaveURL`, `toHaveTitle`, `toHaveClass(locator, expected)` | `toBe`, `toEqual`, `toMatchObject`, `toBeTruthy`, `toBeFalsy`, `toBeNull`, `toBeUndefined`, `toBeDefined`, `toHaveLength`, `toHaveProperty`, `toContain`, `toMatch`, `toBeGreaterThan`, `toBeGreaterThanOrEqual`, `toBeLessThan`, `toBeLessThanOrEqual`, `toBeCloseTo`, `toMatchSchema` | +| `toBeVisible`, `toBeHidden`, `toBeAttached`, `toBeEnabled`, `toBeDisabled`, `toBeChecked`, `toBeSelected`, `toBeExpanded`, `toBeFocused`, `toHaveText`, `toContainText`, `toHaveValue`, `toHaveAttribute`, `toHaveCount`, `toHaveAccessibleName`, `toHaveScreenshot` (also on `screen`) | `toHaveURL`, `toHaveTitle`, `toHaveClass(locator, expected)` | `toBe`, `toEqual`, `toMatchObject`, `toBeTruthy`, `toBeFalsy`, `toBeNull`, `toBeUndefined`, `toBeDefined`, `toHaveLength`, `toHaveProperty`, `toContain`, `toMatch`, `toBeGreaterThan`, `toBeGreaterThanOrEqual`, `toBeLessThan`, `toBeLessThanOrEqual`, `toBeCloseTo`, `toMatchSchema` | `toHaveText` compares the whole normalized text, `toContainText` a substring or RegExp, `toHaveValue` a form control's value as is, whitespace included @@ -278,6 +313,19 @@ works on `toHaveText`, `toContainText`, `toHaveAccessibleName`, `toHaveAttribute(name, value)`, and `expect(browser).toHaveURL`. Any other option, `indeterminate` or `useInnerText` included, is `INVALID_ARGUMENT`, in JavaScript too. +Screenshots: `await expect(screen).toHaveScreenshot('home.png')` compares the +screen, `expect(locator).toHaveScreenshot()` one element's box, against +`-snapshots/--.png`, on web, iOS, and Android +alike. Commit the files. The first run writes the file and fails: look at it, +commit it, rerun. A CI run writes nothing into the project and attaches a +missing one at `snapshots/` in the test's results. `--update-snapshots` +(`-u`) writes missing and different files and passes, after an intended UI +change. Options: `threshold`, `maxDiffPixels`, `maxDiffPixelRatio`, `mask` +(locators painted over, for clocks and other changing content), `maskColor`, +`timeout`. A failure attaches `-expected`, `-actual`, and `-diff` images to the +step. Files are per target and OS (`-web-linux.png`): screenshots written on a +Mac never serve Linux CI, so write CI's from a CI job with `-u`. + ## Sign-in sessions Sign in once in a setup test, save the state under a name, and let other @@ -314,11 +362,10 @@ test('the dashboard opens directly', { session: 'admin' }, async ({ app, screen match `[A-Za-z0-9_.-]{1,128}`. - A session holds cookies, local storage, and IndexedDB for one run, encrypted and deleted at cleanup; server state is not part of it. -- Once a secret is filled, model pixels and assertion screenshots are - withheld for the rest of that session (later serial members included) and - `app.screenshot()` is `POLICY_DENIED`. A restored session keeps its setup's - taint; a setup that signs in without a fill (`browser.setCookies`, say) leaves - screenshots available. +- Screenshots, the model's and the artifacts', mask secure fields only, + after a secret fill too. A generic secret in a plain input, or a value the + app shows anywhere else (a show-password toggle, a toast), is in the image + and reaches the model provider: use test credentials. - Credentials live in the config, values in the environment: ```ts @@ -329,7 +376,8 @@ credentials: { - A static password or secret needs 6 or more code points, else `INVALID_CONFIG` at config load, so an unset variable fails every command. - A function is read at fill time and redacted only from that fill on. + A function `password` is read at fill time and redacted only from that fill + on; `username` is always a string. - `E2E_USER__USERNAME` and `E2E_USER__PASSWORD` override either field per run, even over a function; `` is the credential name uppercased, every character outside `[A-Z0-9]` as `_`. Two credentials (or @@ -351,6 +399,72 @@ credentials: { `` (and so does the test id, for a value in a title), so the agent cannot type it. Pass the handle. +## Start near the screen under test + +A test pays for every step before the screen it checks, on every run and on +every rerun while you fix it. A wrong locator at the end of a long flow +reruns the whole flow, then waits out the 30 s action timeout. Begin each +test as close to its screen as the app allows: + +- Sign in once in a setup test and declare `session` on the test (Sign-in + sessions). Only the setup fills the sign-in form. A session lasts one run, + so the setup still runs on every run, once per target for all the selected + tests that declare its session. +- Open the screen by its path, `app.open('/projects/42/settings')`, not the + home page and three taps. `observe` over MCP prints the path of the screen + you reached. +- Create the data the screen needs through the app's API in a `test.extend` + fixture. Click through a creation flow only in the test that covers it. + A declared session is restored before fixtures run, so when the app signs + in with a cookie, `browser.cookies()` signs the fixture's requests in. + Check the response in the fixture: a + failed seed then fails there, not 30 s later on a locator. While exploring + over MCP, a project tool that seeds data (topic `agent`) gets there in one + `call`. + +```ts +import { test } from '@e2e-dev/web'; +import { expect } from 'e2e'; +import { z } from 'zod'; + +const projectTest = test.extend<{ project: { id: string } }>({ + project: async ({ app, browser }, use) => { + const cookie = (await browser.cookies()).map((c) => `${c.name}=${c.value}`).join('; '); + const api = (path: string, init: RequestInit = {}) => + fetch(`${app.baseUrl}${path}`, { ...init, headers: { cookie, 'content-type': 'application/json' } }); + const created = await api('/api/projects', { method: 'POST', body: JSON.stringify({ name: `Project ${Date.now()}` }) }); + const project = expect(await created.json()).toMatchSchema(z.object({ id: z.string() })); + await use(project); + await api(`/api/projects/${project.id}`, { method: 'DELETE' }); // runs after a failure too + }, +}); + +projectTest('renames a project', { session: 'admin' }, async ({ app, screen, project }) => { + await app.open(`/projects/${project.id}/settings`); // straight to the screen under test + await screen.getByLabel('Name').fill('Atlas'); + await screen.getByRole('button', 'Save').tap(); + await expect(screen.getByRole('status')).toHaveText('Saved'); +}); +``` + +On a device there are no sessions and `app.open()` takes no path: +`device.openLink('myapp://projects/42')` opens a route, or a sign-in link, +in the app. + +Grow a test that spans several screens one screen at a time when you have +not checked its steps live: write up to the next screen you have not seen +pass, run it, then add the next. A passing locator step takes milliseconds, +and an `agent.act` that a later check confirmed replays from the cache with +no model call. The cache keeps those entries when a later line fails, so +each run pays model calls only for its new part. That holds for a first +attempt with a read-write cache: a retry never replays, and with `CI` set +the cache records nothing by default, so grow the test outside `CI` and +pass `--retries 0` when the config retries. Keep the title and the +goals already written as they are while the test grows: a cache entry +belongs to the test's file and title path and the goal's instruction and +params. A fixture's run-unique value in a goal's params goes in as +`unique(value)` (topic `agent`), or every run misses. + ## The browser fixture (browser only) Prefer `app` and `screen`; `browser` is for what only a browser has, and a @@ -367,7 +481,8 @@ the app opens itself (`target="_blank"`, `window.open`) is not followed: - `locator(css)`: raw CSS or XPath; not portable, a last resort. - `frameLocator(css)`: a `Screen` scoped to one iframe (`browser.frameLocator('#payment').getByLabel('Card number')`); keeps - `locator(css)` and `frameLocator(css)` for nesting. + `locator(css)` and `frameLocator(css)` for nesting. The agent's tree + leaves out a frame from another site, so script steps there with it. - `evaluate(fn | source, arg?)`: runs a function or source string in the page, JSON in and out, no closures; a throw in the page is `EVALUATE_FAILED`. @@ -375,27 +490,46 @@ the app opens itself (`target="_blank"`, `window.open`) is not followed: before the page's own scripts; call it before `app.open`. `arg` is JSON. - `route(pattern, handler)`, `unroute(pattern)`: intercept requests, newest route first; `route.request` has `url`, `method`, `headers`, `postData`. + `route.fetch()` reads upstream without deciding the route and returns + `{ url, status, headers, body(), text(), json() }`; redirects are not followed. + Hand binary bodies on with `body()`, never `text()`. The handler calls exactly one of - `fulfill({ status?, headers?, contentType?, json | body | path })`, + `fulfill({ status?, headers?, contentType?, json | body | path })` (`body` + a string or bytes; several cookies as one `\n`-joined `set-cookie`), `continue({ url?, method?, headers?, postData? })` (straight to the - network), `fallback()` (the route registered before it), or `abort()`; + network; a header set to `undefined` is dropped, configured site headers + still added), `fallback()` (the route registered before it), or `abort()`; none or two fails the next step with `ACTION_FAILED`, an unsupported option with `INVALID_ARGUMENT`. +- `unrouteAll()` removes every route; configured site headers stay. - `waitForResponse(pattern, { timeout? })`: resolves once the headers - arrive, with `{ url, status, headers, json(), text() }`; `text()` and - `json()` wait for the body (up to the action timeout) and reject with - `ACTION_FAILED` when it could not be read. + arrive, with `{ url, status, headers, body(), json(), text() }`; the body + readers wait for the body (up to the action timeout) and reject with + `ACTION_FAILED` when it could not be read. Start the wait before the + request, including the first `browser.goto` or `app.open`. - `cookies()`, `setCookies([...])`: a target is an http(s) URL or a domain. +- `clearCookies({ name?, domain?, path? })`: remove cookies matching every + string or RegExp filter; no options removes all cookies, other storage stays. - `setViewport({ width, height })`: resize. +- `emulateMedia({ colorScheme?, reducedMotion? })`: what the page's `@media` + rules and `matchMedia` read, for the rest of the test. Call it before + `app.open` to load the page with it; a feature left out keeps its value. - `onDialog('accept' | 'dismiss' | handler)`: awaited; resolves to an async unsubscribe. Register it before the tap that opens the dialog. A handler - gets `{ message, accept(text?), dismiss() }`, `accept` taking the prompt + gets `{ type, message, accept(text?), dismiss() }`, where `type` is + `alert`, `confirm`, `prompt`, or `beforeunload`, `accept` taking the prompt text; no handler, or one that neither accepts nor dismisses, fails the - next step with `INVALID_STATE`. + next step with `INVALID_STATE`. A throwing handler's unanswered dialog is + dismissed too, and its error fails the next step. - `waitForDownload(() => trigger, { timeout? })`: returns - `{ path, suggestedFilename }`. -- `keyboard.press(key)`, `keyboard.type(text)`, `mouse.*`: unfocused input; - prefer `locator.press` and `locator.fill`. + `{ path, absolutePath, suggestedFilename }`; `absolutePath` names the saved + runner-side file for reading its contents, including with a hosted browser. +- `waitForDownloads(() => trigger, { count, timeout? })`: calls the trigger + once and returns `count` of those results in start order. +- `keyboard.press(key)`, `keyboard.type(text)`, `keyboard.down(key)`, + `keyboard.up(key)`, `mouse.*`: unfocused input. Only `down(key)` holds a + key or modifier until `up(key)`. `press(key)` presses and releases within + the call. Prefer `locator.press` and `locator.fill`. A `route`, `unroute`, or `waitForResponse` pattern is a glob string or `RegExp` matched against the full URL (`*` stays within one path segment, @@ -409,9 +543,13 @@ A `route`, `unroute`, or `waitForResponse` pattern is a glob string or `browser.locator('.btn-primary')`. - Test data gets a run-unique name (`Invoice ${Date.now()}`) and `afterEach` cleanup, so replays and retries never trip over leftovers. -- APIs live in the same suite: `fetch(new URL('/api/users', app.baseUrl))` +- APIs live in the same suite: ``fetch(`${app.baseUrl}/api/users`)`` plus value matchers, in a `test.extend` fixture that reads `browser.cookies()` when the API needs the session. -- No sleeps or polling loops; a matcher with a longer `timeout` instead. +- No sleeps, fixed delays, or polling loops. Wait for the state, not a + duration: `agent.waitFor('the import finished')` with a `timeout` long + enough for the slowest case (topic `agent` lists its model and budget + limits), or a matcher with a longer `timeout` when a locator names the + state exactly. - `await` every step call and `expect.poll`, else `STEP_NOT_AWAITED` at the line of the call. - Assert the fact a model produced with `toContain`, not its exact sentence. diff --git a/nub.lock b/nub.lock index 1a078082..713cf43d 100644 --- a/nub.lock +++ b/nub.lock @@ -25,9 +25,9 @@ patchedDependencies: '@cloudflare/containers@0.3.7': hash: 3ba0e745e0dcdddc06705fa0c7ee010aaa7259d52fd075bcf84ee0c5990ba44f path: patches/@cloudflare__containers@0.3.7.patch - e2e@0.21.3: + e2e@0.23.0: hash: 1f55d7623fea3c16a58e22464d1a4919a5d805ba9da209e0b33068891295ba82 - path: patches/e2e@0.21.3.patch + path: patches/e2e@0.23.0.patch importers: @@ -62,14 +62,14 @@ importers: specifier: ^4.20260702.1 version: 4.20260702.1 '@e2e-dev/decision': - specifier: 0.3.0 - version: 0.3.0(ai@7.0.152(zod@4.6.1))(e2e@0.21.3(@ai-sdk/anthropic@4.0.87(zod@4.6.1))(@ai-sdk/openai@4.0.101(zod@4.6.1))(@ai-sdk/openai-compatible@3.0.44(zod@4.6.1))(ai@7.0.152(zod@4.6.1))) + specifier: 0.3.2 + version: 0.3.2(ai@7.0.152(zod@4.6.1))(e2e@0.23.0(@ai-sdk/anthropic@4.0.87(zod@4.6.1))(@ai-sdk/openai@4.0.101(zod@4.6.1))(ai@7.0.152(zod@4.6.1))) '@e2e-dev/github': - specifier: 0.4.3 - version: 0.4.3(e2e@0.21.3(@ai-sdk/anthropic@4.0.87(zod@4.6.1))(@ai-sdk/openai@4.0.101(zod@4.6.1))(@ai-sdk/openai-compatible@3.0.44(zod@4.6.1))(ai@7.0.152(zod@4.6.1))) + specifier: 0.4.4 + version: 0.4.4(e2e@0.23.0(@ai-sdk/anthropic@4.0.87(zod@4.6.1))(@ai-sdk/openai@4.0.101(zod@4.6.1))(ai@7.0.152(zod@4.6.1))) '@e2e-dev/web': - specifier: 0.15.4 - version: 0.15.4(e2e@0.21.3(@ai-sdk/anthropic@4.0.87(zod@4.6.1))(@ai-sdk/openai@4.0.101(zod@4.6.1))(@ai-sdk/openai-compatible@3.0.44(zod@4.6.1))(ai@7.0.152(zod@4.6.1))) + specifier: 0.15.5 + version: 0.15.5(e2e@0.23.0(@ai-sdk/anthropic@4.0.87(zod@4.6.1))(@ai-sdk/openai@4.0.101(zod@4.6.1))(ai@7.0.152(zod@4.6.1))) '@electric-sql/pglite': specifier: 0.4.5 version: 0.4.5 @@ -98,8 +98,8 @@ importers: specifier: 1.0.0-rc.5-ab785fc version: 1.0.0-rc.5-ab785fc(@electric-sql/pglite@0.4.5)(@libsql/client@0.17.4)(pg@8.23.1)(postgres@3.4.9)(zod@4.6.1) e2e: - specifier: 0.21.3 - version: 0.21.3(patch_hash=1f55d7623fea3c16a58e22464d1a4919a5d805ba9da209e0b33068891295ba82)(@ai-sdk/anthropic@4.0.87(zod@4.6.1))(@ai-sdk/openai@4.0.101(zod@4.6.1))(@ai-sdk/openai-compatible@3.0.44(zod@4.6.1))(ai@7.0.152(zod@4.6.1)) + specifier: 0.23.0 + version: 0.23.0(patch_hash=1f55d7623fea3c16a58e22464d1a4919a5d805ba9da209e0b33068891295ba82)(@ai-sdk/anthropic@4.0.87(zod@4.6.1))(@ai-sdk/openai@4.0.101(zod@4.6.1))(ai@7.0.152(zod@4.6.1)) esbuild: specifier: 0.28.2 version: 0.28.2 @@ -585,10 +585,6 @@ packages: resolution: {integrity: sha512-fX2ENAc7iDpZ+Wp4+Rk06Usn/Ys7dI9uAkGv0jlF6XVrW13NkRWx5Ou+U6lIM2E1fTLkCs16GGrUAaVvroag7A==} engines: {node: '>=22'} - '@ai-sdk/provider@4.0.28': - resolution: {integrity: sha512-6ThcmkJQXSMlBrf6mirIjUUS04Q5/kApbYTOzd5lZfYXLa1XPpnDLxP/G96/DKc00fDIUOW9S2ls0mSzb0+l9w==} - engines: {node: '>=22'} - '@ai-sdk/provider@4.0.3': resolution: {integrity: sha512-e0CpNWJUY7OxAFAnCZkw+ri9QOHWwTs1tXP42782KFGCU07qt8NiXCrCVowyCB5dP2r5/Uls+g2oPd8kOJn9dw==} engines: {node: '>=22'} @@ -597,6 +593,10 @@ packages: resolution: {integrity: sha512-C6ylMuAXjgqKJllp6ODIJwUltK/tm3ZmIj4V2fqENP3Y8gif2DyeJSsTcDwFrXppgEKcrBisi/BWx2tYeoeCAw==} engines: {node: '>=22'} + '@ai-sdk/provider@4.0.36': + resolution: {integrity: sha512-GqvWZSZdBelCGughlBdOoupT258rgM3YMbaLAS9WSZJ0vlt8c9vm3My+ZT31mYhiReMCMCF2qF2axnGbILpk1A==} + engines: {node: '>=22'} + '@ai-sdk/typesafe-ai@3.0.26': resolution: {integrity: sha512-l8iSYnlro5Bp7OBFBGUIHhjH6Zt6Oi3iw+thlEk2C2iTHUXH/BNr+UmTtM7OKN5TNJFUeiAiNRPF56xaBFjbnw==} engines: {node: '>=22'} @@ -1189,25 +1189,25 @@ packages: '@drizzle-team/brocli@0.12.1': resolution: {integrity: sha512-yHwomvolVafvUXhy5TdiJVkuNCxXn+bKOs2aWiWEQh/5YaV97Oa/abJz3dmx7lxgIUsUlS1Jl4mEfvJPo3YsVw==} - '@e2e-dev/decision@0.3.0': - resolution: {integrity: sha512-YrgNcVak2wINlEJo9NCDXxlvhzmIW/wulCnF3sx9vbTe/06QMtZx+IUOiXmHmX/CG5or9QuOHSGglJud13B1vA==} + '@e2e-dev/decision@0.3.2': + resolution: {integrity: sha512-HTdsaL/ICAnceRchPngmsqbTP52U8Yll9pM6CQcyBcJr99DP//xqggph2sJ4XUBKqD0YJUQ8mlNul41dmBPp6g==} engines: {node: ^22.22.3 || >=24.8.0} peerDependencies: ai: ^7.0.134 e2e: '>=0.21.3 <1' - '@e2e-dev/github@0.4.3': - resolution: {integrity: sha512-wFnbp/YlhloyBek0lsnOZTR9m8SQmXAUhG/r5V03T47vcrBsxCswbbxLCTd2R85ck1/2ZNvMATwoWWtklj2otg==} + '@e2e-dev/github@0.4.4': + resolution: {integrity: sha512-r//1cz8akcnxKHk72bf5YHZwQK6wDm+m2ex2xBKLEPCDWJrsaUQgkJlXwkVkl4a3sjoghqsC39Obu8LA+qfP9Q==} engines: {node: ^22.22.3 || >=24.8.0} peerDependencies: e2e: '>=0.15.0 <1' - '@e2e-dev/web@0.15.4': - resolution: {integrity: sha512-j4bDSP6wV310h9LwrMVfsUHv7uAIIzK516NboLco6Ah92B32uShIHne5V3AI1lvKtLip+PWlpJ+aw848vogBwQ==} + '@e2e-dev/web@0.15.5': + resolution: {integrity: sha512-fUXrSgfii6w/q7OyX9/kEIDLpIMHpEkTH81hrR8WLnqk8li3bRyXJQBGWqtq9NoWpS2UANnhMI45n6Fv1ErCnw==} engines: {node: ^22.22.3 || >=24.8.0} hasBin: true peerDependencies: - e2e: '>=0.18.0 <1' + e2e: '>=0.22.1 <1' '@effect/platform-node-shared@4.0.5': resolution: {integrity: sha512-a8rCdEkfZBGe9C6bsIjxVZuRp73svIyK5IHwHdDAVsvZXjrbkxusfI+rZ8WlX31RP1mKizwI/E72GQTNfdMWHw==} @@ -1865,6 +1865,10 @@ packages: resolution: {integrity: sha512-laVmIhPGpWi7rG9q+0UigblRN0Hws/Yjlb6qLFtVqryXVLkc4h+yN8z731POD2LnAGQlFe7/tXhUm7Bi0YB91g==} engines: {node: '>=20'} + '@modelcontextprotocol/core@2.4.0': + resolution: {integrity: sha512-z8QBKqdfXLPzkLQuG4xp8hrVEMm1PZ1IgxMN2sMbJiFiAKOWXnAzogXn2W4aqIBlP1ea6U5yiqVquvUNM1Cj6Q==} + engines: {node: '>=20'} + '@modelcontextprotocol/sdk@1.32.0': resolution: {integrity: sha512-8BviX/hK4Gd2eL1KdTwm0gfl4d3wdoErW0Jd00h6nomUAtk6IJk5vwQJc6kfWtJTzN5OEV/lgIiJKI8fiDGAlA==} engines: {node: '>=18'} @@ -1879,6 +1883,10 @@ packages: resolution: {integrity: sha512-e59MfWuSssj6DoQ96JCTqdW1AXT5/XtnTl0WWd6MU1Ka434fFUGGosCLJbGNySDhHGvxQWoRWK137FXYFkYNWw==} engines: {node: '>=20'} + '@modelcontextprotocol/server@2.4.0': + resolution: {integrity: sha512-rgR9yxSaAJ+bcsi/XroT/S2AfPiIWloXAnrdfYSCms9ivVju0ishu/wW0J21Ho6+ZOR1r7RyeRbjiRuF87ix5Q==} + engines: {node: '>=20'} + '@mongodb-js/zstd@7.0.0': resolution: {integrity: sha512-mQ2s0pYYiav+tzCDR05Zptem8Ey2v8s11lri5RKGhTtL4COVCvVCk5vtyRYNT+9L8qSfyOqqefF9UtnW8mC5jA==} engines: {node: '>= 20.19.0'} @@ -1920,259 +1928,259 @@ packages: resolution: {integrity: sha512-eSYWTm620tTk45EKSedaUL8MFYI8hW164hIXsgIHyxu3VobUB3fFCu5t0hQby6OoWRPsG1KkKUG2M5UadiLiVg==} engines: {node: '>=14'} - '@oxc-parser/binding-android-arm-eabi@0.154.0': - resolution: {integrity: sha512-sDdEtDl7vKdCvf54j72RJbowE3jCmyRz/+oOB6IAwQxAHZ0eI+Dcj4hvVnm0b9Dy31plUkdmfds9C+/Zt895Tg==} + '@oxc-parser/binding-android-arm-eabi@0.157.0': + resolution: {integrity: sha512-zFtSEal1K/17LATxVs0VcT/Itt+fxUQ9Ni8HtG+pPcEqy/TenRO74XffkY2dBiy3Pd+w0FuebwCZ6PJx1z7scQ==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm] os: [android] - '@oxc-parser/binding-android-arm64@0.154.0': - resolution: {integrity: sha512-UHrDPELzvWZoVkIDdSycj4bzcTMCJG3IbN++mSUgVBaUtNtWHjQ98U6Vz5bpF/1Ic3FYJyYhMr74huO5tVvpZw==} + '@oxc-parser/binding-android-arm64@0.157.0': + resolution: {integrity: sha512-WSlkFUnjamuv+2+TXW5F1rp2I24KVEXj4S+H7J7Y8T/KLR2xXbHj8lgxbZZtjSKi3Z/aGGrlFEOy7EmCWpIUlg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [android] - '@oxc-parser/binding-darwin-arm64@0.154.0': - resolution: {integrity: sha512-mokYzajOSXxtSRDWCdHARGdntrKH+rZZ24Jzo8AtafyNzz0NARmpMm26Tc5B6qiGrr6UliWibHCOif1qRN2vaA==} + '@oxc-parser/binding-darwin-arm64@0.157.0': + resolution: {integrity: sha512-Usk8UoURmtG2qBLwrO+il+kf6CahEJJriRQACuAwF2CqcWqGsvl9xj9JIMDxP9VTinOt/BZUuDZxcpn68QKsAA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [darwin] - '@oxc-parser/binding-darwin-x64@0.154.0': - resolution: {integrity: sha512-WbtUBrclOVItWfLRSvs+Z/WLtXxJ2Ctv50PyuX894HzDN5/e0qzxVpTNPIVMzOerBhgKtFsSMcfxlYIAbvdVgg==} + '@oxc-parser/binding-darwin-x64@0.157.0': + resolution: {integrity: sha512-yOvjPixSD2sYJxtQXOzprXdV1xbr2RZyEnbkwFnnxz332Q4+gwkQgCGXZkFlQpc95Fk4XFITACfbFzze4XXktA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [darwin] - '@oxc-parser/binding-freebsd-x64@0.154.0': - resolution: {integrity: sha512-YooHZSLawOe2fqm+lDdG8oAaQh2NYV/RMYTYzWRLy72stm2BgE9O/8Al6kFENjkmzyyBccDLLBirGxVtpzQfrw==} + '@oxc-parser/binding-freebsd-x64@0.157.0': + resolution: {integrity: sha512-BLTsP8TWvmV53jgGBynwv2DR3hKtjpdpL6fXnQs2Fg5P6HsFTShFjrgQrHMSM1cA4lGmlcj7tdTx96RoJfYuuA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [freebsd] - '@oxc-parser/binding-linux-arm-gnueabihf@0.154.0': - resolution: {integrity: sha512-hKOwhZbrXdTwc4bGVfbLCXl9f2swl6zWdISocLQSNX9MgPmwHE3qGzOUkCrkuGpwdCVHPnkfOwsqTuYsrP5uNg==} + '@oxc-parser/binding-linux-arm-gnueabihf@0.157.0': + resolution: {integrity: sha512-wGMDInIrhayQpNxwpXIWqa2QiKk+/6RxOi4GxPVoqmwrCdn2xj3xXyDUm3RujxQaraZjeWfh0+h9nedoivnL1w==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm] os: [linux] - '@oxc-parser/binding-linux-arm-musleabihf@0.154.0': - resolution: {integrity: sha512-tnjiooXEU013349O/yxKbF5Cusp1CfHWQ8kYR8tGeRzR/P4sOpQGQo21fsS71M0fMD6ESLq+lbvXFWi43PuKeQ==} + '@oxc-parser/binding-linux-arm-musleabihf@0.157.0': + resolution: {integrity: sha512-N6OLfOnku4otlU6eF+HLU+1fH2XAaItc+c7dE+S09SzRZHvkaltU9aGTlMhmAIee95hOqkOCVFHNzQYJ1KS+gQ==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm] os: [linux] - '@oxc-parser/binding-linux-arm64-gnu@0.154.0': - resolution: {integrity: sha512-Cao2kj+8d7BT6XJU1gV9eVQjV+Cr+e8vQm3tddDRZiM2hHrMDxwy3j6t+ofaSl0idyuPPCZ4AlpmQoj8yOT1QA==} + '@oxc-parser/binding-linux-arm64-gnu@0.157.0': + resolution: {integrity: sha512-YfY+qZ8d9tHJEvIq+zXyCcSve/jWgTNDiDtr1W4WVMh5+4fsOjYtH2JDIXIXUkILZzPKgJYbQ08Y/mwhr1gQLQ==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [linux] libc: [glibc] - '@oxc-parser/binding-linux-arm64-musl@0.154.0': - resolution: {integrity: sha512-uiwA0V3q8llWA7/8r+a9BBjk4j5YMDmi6wfOsyQfGOJMQSLHEiZ3HI8ulea766ykUa9wJFWk9XWEl4uJrZLQqw==} + '@oxc-parser/binding-linux-arm64-musl@0.157.0': + resolution: {integrity: sha512-ytZeO8BKkezwkLoWH3hK0R9Jd+S1BQhcQaWiSAWhXE7Dncq1XiJKb4w5hCe4lwl2ytgoSBgB0XEABhwD2/IhJg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [linux] libc: [musl] - '@oxc-parser/binding-linux-ppc64-gnu@0.154.0': - resolution: {integrity: sha512-LICfN0Xs4Zdko+bJy6eOAEQojDuIR26suF2JtOsMaXjgbCrpRSH7/NCvJhY13YvYBx12V6hnwrL8n0lk0rbNJQ==} + '@oxc-parser/binding-linux-ppc64-gnu@0.157.0': + resolution: {integrity: sha512-54w3SexApRhsw3DOAXBZ85rEMR2y29NDOUr18t69fcsnemS/N1YU+zQddHCol/HDhBPyLHg5fpxChRCSzxHwqw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [ppc64] os: [linux] libc: [glibc] - '@oxc-parser/binding-linux-riscv64-gnu@0.154.0': - resolution: {integrity: sha512-gbbSYxLF8ssRbQ2qQPy+2EhGNwNH1bAoRdkk6cnWaKHqmCikpSIPtpXjFbCCtjTi0uojqOM3ghFQwQPbDQsU0Q==} + '@oxc-parser/binding-linux-riscv64-gnu@0.157.0': + resolution: {integrity: sha512-byg0y1OHdjjhll0qWJQyowKdCxM/zCKysIIOqkWgCid/cR+RU5nTz/63OeAZvIb1z/VB1CyypDRdLn2Ov3r+HQ==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [riscv64] os: [linux] libc: [glibc] - '@oxc-parser/binding-linux-riscv64-musl@0.154.0': - resolution: {integrity: sha512-nSnRYYDLCcRVjSQzF3g6rKqCNYwo5o+aQgnn8BBXfe+ofCgMGlGqkfRyvXm6ptZvoxNkkWtLlqM8W7Xc3j6ZnQ==} + '@oxc-parser/binding-linux-riscv64-musl@0.157.0': + resolution: {integrity: sha512-CiQrt8esQTeL17kscbsfpdx5cywItx3y95YejsihET68oYnxHjbEgj8JaqlIWmhpL2h98piqPBLhWmZbp67nQQ==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [riscv64] os: [linux] libc: [musl] - '@oxc-parser/binding-linux-s390x-gnu@0.154.0': - resolution: {integrity: sha512-BxBQoJKtzRQgZEKQIt6CDUm8RGKC5T8Qo4576yDa5/xgc10X3fBDItMbRabzLtbUBfTqKLl+8AVr/H6CF7W22g==} + '@oxc-parser/binding-linux-s390x-gnu@0.157.0': + resolution: {integrity: sha512-wSzuUshJVxcrx0c9UJYUOuUf0XHThC1+f8CEfeFbosg7FzTkjCdt89JKKU7BH8C9eTPrnJ5ir4QM0pWpVSmHnA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [s390x] os: [linux] libc: [glibc] - '@oxc-parser/binding-linux-x64-gnu@0.154.0': - resolution: {integrity: sha512-/whkust2vYy+GjRUVL5cWaJmEu6jkdFfo5mkxgckyeOX61eMwo4iBoiwzeJZeMWAdzb/XOtQyLApHhFht3sOWA==} + '@oxc-parser/binding-linux-x64-gnu@0.157.0': + resolution: {integrity: sha512-C4BY4QSVzN6R03Ner3r6EtIvLNR4VW4RAb63m+L0tiF/zD7grkBHMn1svyrOBFR4i32+GWJHu05xC4gGQYBhtA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [linux] libc: [glibc] - '@oxc-parser/binding-linux-x64-musl@0.154.0': - resolution: {integrity: sha512-EXURTyW9riw4B/kZnnE+58emAtw6+3VEszQfXUZkRqGaKIcmWDYkZtKB+xY/blHvuZ+Qy/fDclc31nmUCgtkSA==} + '@oxc-parser/binding-linux-x64-musl@0.157.0': + resolution: {integrity: sha512-kN3LdZ2xp9z+TLuzYUWPR/mz/HrKd20vMCGWyby1d8T8K5t5NPemcqfSv3y0oMwXK8OF3lW7MNJnc3uWphvwvw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [linux] libc: [musl] - '@oxc-parser/binding-openharmony-arm64@0.154.0': - resolution: {integrity: sha512-ZRFooyH3qsR6vuR4jFGlG3XM36490SvvXtNaUahBK4zHpooMdTHNaOMtGGJI0EW38m5hi4rOco9kw7ZwCIp28w==} + '@oxc-parser/binding-openharmony-arm64@0.157.0': + resolution: {integrity: sha512-cQKUrgBAFUaGcqp9Umb3GToFkvow92YeJxJuzbk8YPnButv9aY5HLyKUpxxh4WqJvP6POiNR5fUDDB5wI8Wiaw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [openharmony] - '@oxc-parser/binding-win32-arm64-msvc@0.154.0': - resolution: {integrity: sha512-U0Vp+4lvwUa/PnOCJTZOjydj1rX2+VLHz4Y2vSFqlN8/7LTiURgD7z4uQ9yNv9O/NlU3mELBnPDlu7aaSYG1zQ==} + '@oxc-parser/binding-win32-arm64-msvc@0.157.0': + resolution: {integrity: sha512-qZsymd8L67o5bpzZ7gVO7qJUIF9Uf0/mXcoG0MDchBvdL7Q2tAF4aZBZFqc/YRd0WP3jTvqjs3CxlFhHgi48dA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [win32] - '@oxc-parser/binding-win32-ia32-msvc@0.154.0': - resolution: {integrity: sha512-Hc8QgsFY5eUUP25we8TISUruibG2cDt0H4IcPG9RKdabb6odkCzIqjV6Gk9beGWGuOwMVCr81H/Su1q363LCCg==} + '@oxc-parser/binding-win32-ia32-msvc@0.157.0': + resolution: {integrity: sha512-Ro9PcgwvRglGKI9fEYkbTKvIgh/hzWjv+yPyCTNuxp173i79YbTkkYeOvAEC9SFQPZXYr4+SuIrpCIaWN6sbUA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [ia32] os: [win32] - '@oxc-parser/binding-win32-x64-msvc@0.154.0': - resolution: {integrity: sha512-hD7judkUMA/hZPoamvKADs6+an1smp8UTK0aMYRijkvi00K/1vl7vEtmyn2JyQ9S+vibtnfcNzxqTLGyJUZCRQ==} + '@oxc-parser/binding-win32-x64-msvc@0.157.0': + resolution: {integrity: sha512-Ea0fqR+HLzzCt9tYhKbb6aJlM6wBXKWfuDwLhcG519B8HGtxthDzLGfbqMQdlX2Ii5gAZt/Hgxig/iApdxZSzQ==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [win32] - '@oxc-project/runtime@0.154.0': - resolution: {integrity: sha512-cBQNGoh659BNgvA8OC/e4BNDtFkv03dJaYdkp2adgOQNtISNAPQ+nB4S8XvWtNk7Xl+0g+7aa+iRGR25Xycd1Q==} + '@oxc-project/runtime@0.157.0': + resolution: {integrity: sha512-srnUknwVoji5JdmU664pA4XyjBJXmgxP8rWlnZhFovzw6XQxaICGWAuScZdlLmIHI+ZRc7WgeiC2CURF4tAszQ==} engines: {node: ^20.19.0 || >=22.12.0} '@oxc-project/types@0.153.0': resolution: {integrity: sha512-jSNrRIyvPVLL5E1z/tS5zLAprosGz/FuI1PWwEOz59j3E4TTTzSpqKb/RDGqJhhW0QHYISGXxq582wGx6T11Tw==} - '@oxc-project/types@0.154.0': - resolution: {integrity: sha512-KYOUGQFlLWR+s4QMV2Q9ko38TZm7EdO87Qm7rvZLqk/webse/cDjtyxKhg/47l0byCbs/k46LT8qc4PKjAikzA==} - '@oxc-project/types@0.155.0': resolution: {integrity: sha512-SeE8JPwW1xs1tHeNycjUtUBdhorRa8o4jlvjwk49R7/Z291i+4DQa7JJIgOm/TaYQZx+QrPQbVJ/9c8dMIZN8Q==} - '@oxc-transform/binding-android-arm-eabi@0.154.0': - resolution: {integrity: sha512-Wqd1sJDGwaDPlO2BFdLD7+OMQR4T3Y9PaqS1D+kjjHPEJZ2lotY8cm67d6Otp0IaS7uKGU76S8aa5mHJwTvwXQ==} + '@oxc-project/types@0.157.0': + resolution: {integrity: sha512-95iTox1qdt6WGWce3xwNOVayByOfLaYAq+qGPpub6NWbBh9sv0GG3+zyfh9ZBCO/jVu5A0EWI+qQI7g6fevjxg==} + + '@oxc-transform/binding-android-arm-eabi@0.157.0': + resolution: {integrity: sha512-uRHbSeooFBkwRx9DkedM9YPMAvnwPd9ds6iu+ki9yerAGgTe7dhQAgvrbxrbnhQF3L43QR2LMEkV1zobNpI2cQ==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm] os: [android] - '@oxc-transform/binding-android-arm64@0.154.0': - resolution: {integrity: sha512-X236HnhL/WzibySNF8UA5Mk9sLtx8OWSKOhzpHDfmR0gLNK9Rhnr6bCO35+bdgiUWpBZEdxxe9/a5brRZzuMww==} + '@oxc-transform/binding-android-arm64@0.157.0': + resolution: {integrity: sha512-sZAV2CvG3hqJPG6O3OYF5hXfbWkb+kJQvYdfa0lC788dpQQI+zBR7RpFDpAY+3u4WpV2+v1H8SfSOdsw7aJ1ug==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [android] - '@oxc-transform/binding-darwin-arm64@0.154.0': - resolution: {integrity: sha512-CRewaU+ThBKqDthUOOrmwEu8XfPsnKRuwiO6a/OXMJbQ7l9/247N1WTxMMWZWBkJObK5pjd/oDAaGxvmJjTcWQ==} + '@oxc-transform/binding-darwin-arm64@0.157.0': + resolution: {integrity: sha512-/vxwVl483evp5RmWHoz999TL3RW3ghgOlgyd8VNGkBeFgxm2lJRKa0MKzbP7l0v6uIF6789yNe8b7B8BgaTWwA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [darwin] - '@oxc-transform/binding-darwin-x64@0.154.0': - resolution: {integrity: sha512-dQhIDr8+AtgIimzFX1XNEsnQfFGVH/aBGoR4u5HgCXu1B9ISAgbDr91N72ehN769eNrNnRUP18weNuNVzHL5kw==} + '@oxc-transform/binding-darwin-x64@0.157.0': + resolution: {integrity: sha512-Uoqdq4CaoMh2hyeOl5L2f8cnXOW36KZR2/MzY4E4zAkO2xKYaWHa/XWNPkFeTfnaOi9lmMIvZAYZ8R7ELxRhPg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [darwin] - '@oxc-transform/binding-freebsd-x64@0.154.0': - resolution: {integrity: sha512-QWEPzmEPPIzZ/y6k0vsiwcUqCtrOHuRNovmW6VeG4y07fBSfVp35uweGQrxoJL/XebkR19M4sIpnFTwcniV6pA==} + '@oxc-transform/binding-freebsd-x64@0.157.0': + resolution: {integrity: sha512-QFjfMWa/oPK40lyWBCQR8/PlnpgVdPN6yOZqU5HJIzWyo0FwPd7tLpig7rtQgiBEhj+XoyxexXHUVFlwA9Oa/A==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [freebsd] - '@oxc-transform/binding-linux-arm-gnueabihf@0.154.0': - resolution: {integrity: sha512-C+tHdgh6dDMgJMkxYSv+5pC8XtHySJx5G1POi5GmADZITPXhvZT6NDUkzToOs9Ja8BMYgejhOeK6G3++AwoF+A==} + '@oxc-transform/binding-linux-arm-gnueabihf@0.157.0': + resolution: {integrity: sha512-aQHBtkX/75l8/hvBTggNLpfzPzHZ1vN/qxK9h+r0Gas5XaI4yeW9Qd/VPxGskBgtjmN3BfHXQvAFaj+R9vyBGA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm] os: [linux] - '@oxc-transform/binding-linux-arm-musleabihf@0.154.0': - resolution: {integrity: sha512-odNOaCqjIJUPM7h3y3+ZgBVZTbPWzFyuMSG2uRRIFB3If5q3d5d7hCK3/CMIeVy7g4JT8IRPoBDttYW/lX6cAg==} + '@oxc-transform/binding-linux-arm-musleabihf@0.157.0': + resolution: {integrity: sha512-dJVOATtEl0LxQuYXTsSAWRay19J5uq7bs1PBHBktEwoCdbXUa8Tm2lPZaX2QuEqC98xk7yAWMOlesv40x6m1rA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm] os: [linux] - '@oxc-transform/binding-linux-arm64-gnu@0.154.0': - resolution: {integrity: sha512-clfcZaasfgkzqkrT2zwMDtLIUDoiu7f3tyoV5QFnlYKFQvJ5zA5tWhjacPZWKnRh7W9UjfILGo4W/Z/4zOXoCA==} + '@oxc-transform/binding-linux-arm64-gnu@0.157.0': + resolution: {integrity: sha512-Jdt0PpTwGWEqFkDqkRrR6yJNMevQebaQFnFZIxxnNXPVDze6mNxpEaPvJEmH0Dxzk1vQQm/I/SM08PAAb2qjpg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [linux] libc: [glibc] - '@oxc-transform/binding-linux-arm64-musl@0.154.0': - resolution: {integrity: sha512-60PLk9egLY1QyZhuC8fEI5yuDSSch9BzQn8B7YYEwRtXtHnvk9U3Rjk7x9TvN+QeDLwPSDQm+nOGyKyuYFusUQ==} + '@oxc-transform/binding-linux-arm64-musl@0.157.0': + resolution: {integrity: sha512-qCLNQRA3KLaiCG9uXQmFbhjoetlCyfFwFdSXqbVVPCUPi1VFhuBGFltb2ZOMrxE2zMyh9RD27xzVULm+SRB1/A==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [linux] libc: [musl] - '@oxc-transform/binding-linux-ppc64-gnu@0.154.0': - resolution: {integrity: sha512-Tvh6NamG/Qkg1V+X+o9zO38GGnbImAvtLR5eappfdImOh6cHHq9Lp7HLTY3LO9OTaiLEPejFo/hpAPMPxQ3SYw==} + '@oxc-transform/binding-linux-ppc64-gnu@0.157.0': + resolution: {integrity: sha512-B5bEXT0mG88o2tIoKK/x2KOoXhbab5SjjBm5WfWlcpKLfr8ioYeZPPNT2gZN6FMq+pYuJA3f+i9dFIVlaqFz6w==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [ppc64] os: [linux] libc: [glibc] - '@oxc-transform/binding-linux-riscv64-gnu@0.154.0': - resolution: {integrity: sha512-RF0+/kQzlnvN0PPbWP1E2Zn6ayYztkPj97K/N9yLa1t+zWdfppBve8ww0lAszkd60J1akYW4TfUhQJiIYXVWGQ==} + '@oxc-transform/binding-linux-riscv64-gnu@0.157.0': + resolution: {integrity: sha512-dDYzPGj3/SrcCwWK78FDXg5UXjJRxhVXADxIHvxiCFAeOAUGBFqBS1ykxw/VcsMueDr2QVgsYySPp+VB6fUNSg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [riscv64] os: [linux] libc: [glibc] - '@oxc-transform/binding-linux-riscv64-musl@0.154.0': - resolution: {integrity: sha512-vU0UF00U21tcIINr/3BKSNJCyrUigHjo/8bF5msZnZ3R61Yyh8sTHrunbw6sQIDYZykb9N9tHKlrt35N5iEhRQ==} + '@oxc-transform/binding-linux-riscv64-musl@0.157.0': + resolution: {integrity: sha512-zfplsQ80jaRXnD/l9d1w4awRanMkEiR+rlTJbxOkB+FnuypwHU3JYVJlrQLr/dph3XIrQSkvxsitY2XpyHdLpw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [riscv64] os: [linux] libc: [musl] - '@oxc-transform/binding-linux-s390x-gnu@0.154.0': - resolution: {integrity: sha512-1e8afsXnXzsgirRRhJx4pV7e2a9xfFMGT9wSNeCEuBCY7pFXE6t81Wp3K6gbIMkrHbdZLPzjvvudvrBcSUpZKA==} + '@oxc-transform/binding-linux-s390x-gnu@0.157.0': + resolution: {integrity: sha512-X7rmxwvgEQrg2qs0goelrFU5X1m5yaTNdnNBT94bK0cQB5q+E7HCxV8eX9wK3qMP8p2bae6N8+ck4gW9XGKBBQ==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [s390x] os: [linux] libc: [glibc] - '@oxc-transform/binding-linux-x64-gnu@0.154.0': - resolution: {integrity: sha512-+k8nMWTqEPPwTt3pdNaH1Ek/HY5Cf7idDOUhoLkWErWiUBCGGpYND47xad+mbN1v0j7CaPP2qHWTsRhN8LReKw==} + '@oxc-transform/binding-linux-x64-gnu@0.157.0': + resolution: {integrity: sha512-wi5EV43pRIiodHVttWD/785k78IMEyKUaSN1o9JN3Yu1GFBCXJEy9wU4J2V6//4iCeEFywdzQiMbHizG4cl0eQ==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [linux] libc: [glibc] - '@oxc-transform/binding-linux-x64-musl@0.154.0': - resolution: {integrity: sha512-wSgC5avEHzEzdSNAO4E7Sq3JppVJCLYgSK1IOuS4f6VOU5eui+HtMdQIeRYsKTc+vANG5YaQ+0c0dHuEo4/kZQ==} + '@oxc-transform/binding-linux-x64-musl@0.157.0': + resolution: {integrity: sha512-ZK6gmsC1rGiqov3mRltKCwJiRxAhEv7l72TdfClGlLy1kH97GiPsnhLnIiCfqRwahN3W9DX98xMoijhtI/taTg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [linux] libc: [musl] - '@oxc-transform/binding-openharmony-arm64@0.154.0': - resolution: {integrity: sha512-4kyIYf1EUdMa2x1PebyHNQF+BVkSWBd2fniMubBZZsPCUwtKv89wOzIjQ7EJLrvWFYWLIa4KdUNoKbuvunIJLQ==} + '@oxc-transform/binding-openharmony-arm64@0.157.0': + resolution: {integrity: sha512-gz2syX2odmf8qzbiEVy+y+giea43tXnTavzy44/JsopYvUodDWOx4aZzu+ZrQZsbfmae1ntDv3NZWyTkbip2yQ==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [openharmony] - '@oxc-transform/binding-win32-arm64-msvc@0.154.0': - resolution: {integrity: sha512-ftqAPU2DcYT9bonTogmNIOFDoILnCehdL2DFz8OCY9+z0LZaPY1F4v/Es0VE22m8++C/hbuNPVHpTYaUPNMrNw==} + '@oxc-transform/binding-win32-arm64-msvc@0.157.0': + resolution: {integrity: sha512-44/lSyHppcduYVtPpWURyUTB8IINPqmwwDFbZKxClGARBhDcPgcvcgI6X4niSUWYJTBHf7vT+OcxmWygRc17dQ==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [win32] - '@oxc-transform/binding-win32-ia32-msvc@0.154.0': - resolution: {integrity: sha512-NdX4YzkjY6zkq/61WZYfdTBu5d+LsuQLmJf/P3wNPgTmfHeO9V7K7CHrfBrU7VbwtA26qM28+zr29LXPhEbGEw==} + '@oxc-transform/binding-win32-ia32-msvc@0.157.0': + resolution: {integrity: sha512-S7a7wqrwQAMTUCocS4weusiQLNY2WP9kJCGCOD1in6PRq1vQFkPt52mh1GLQCzCH+HMBEMx+wK9XotVZBRIVuA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [ia32] os: [win32] - '@oxc-transform/binding-win32-x64-msvc@0.154.0': - resolution: {integrity: sha512-gdkSxCHFdO3w/QPmooRORWAr/HxIwAKSnnN4QLPnf2cR2156Ux5oYLU63u2PnE64402KtX2pBhr+Psgq/peDxg==} + '@oxc-transform/binding-win32-x64-msvc@0.157.0': + resolution: {integrity: sha512-lADvsdq/qHDlkdgzaaJbcwQZOEdNcjrLFW/MdJDG3AhUQSIINibJly2U9Q/yMsoUx/tAnR31gRvWWkLjHq5lxA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [win32] @@ -3607,17 +3615,17 @@ packages: oxc-transform-react: optional: true - '@vitest/expect@5.0.5': - resolution: {integrity: sha512-mDBbYrXKpc+4TcSDwBW6adCMtPnide6Z1g8bI6gL0tY+nhdBxtGoOFW/kMrLDt6lJg3xjEUBkrY9yIPlgzpuig==} + '@vitest/expect@5.1.0': + resolution: {integrity: sha512-Okx71SHCjnfax7YLaxcJzZINi71BF2S5FezzQipheEQsiBAB0dhRBwuQ7uCJUH0/eGAT6HgAtmVVzrEZo6Qmcw==} - '@vitest/pretty-format@5.0.5': - resolution: {integrity: sha512-24sSdIlMA9DVPXnYiB9siwU0bTd887Y9jX3+vabztCSsvHwo7ipvJ/a2L3Rljw8+2rK1f5fCj3h5U9I/Jsqgbw==} + '@vitest/pretty-format@5.1.0': + resolution: {integrity: sha512-IRx5mc30CrFzyMNni6SZKQ1r53kv049Mcvvk0WbWcD6PUz/bFjG4zYv0KosQAeonPpGp5WxqYBMqdSOfJfvVWw==} - '@vitest/spy@5.0.5': - resolution: {integrity: sha512-lwJRExuH417257Q5wv9QYugN9IMJkoxcrKrYxGPQkBzl7wlNpTUxD/5yNX4ESKVUIShKPgkGchYPFMWOOk0wzw==} + '@vitest/spy@5.1.0': + resolution: {integrity: sha512-2DPO5OMHWKrBngC+fCmTQqemNF0gidRhp3lxbBmRhW+M1XJrfgPpqNqWVevXtn2f2ufD5ZaR1B6B641aAlcX5A==} - '@vitest/utils@5.0.5': - resolution: {integrity: sha512-/rBu7gzLDQI/SC79Tn98b5DwOgaHU/Gt4I9OVeP3YofzNaYtbImRCOJQmT1S6vOJg9gEIlWC+r3QxKkjuSa1nQ==} + '@vitest/utils@5.1.0': + resolution: {integrity: sha512-V+qgCwkvVw+s1t1S5TaoBHsLi5Sz/YmohCMoOu0mZ8ZQst2bKhRYARAQucWpVg8D0mv0x5MHcWXY5QdOsoiGBQ==} '@workflow/serde@4.1.0': resolution: {integrity: sha512-pav4F2BoirECWR7Nf1TKt+2eETcBj7jj4cBefQ8VXQCA6NPkaKeLfj/zMgi+3zYV5ZIBT4GuUiphsj0/b9hPQQ==} @@ -4409,8 +4417,8 @@ packages: resolution: {integrity: sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==} engines: {node: '>= 0.4'} - e2e@0.21.3: - resolution: {integrity: sha512-XFnnpJnZVnUqZdxvAsf4fj620hFIYkGRP0klAR1dsPsIfQSwr3/RIUzYWfrkKFX6pXnjdqafd7ROsDauWceTaw==} + e2e@0.23.0: + resolution: {integrity: sha512-x4wbcLDbwtmB9avyruqQX2j77i8HynAGG74vxA2SVSCJwnD7CaioyfBOxKU3L7J7K7cWIDU0LNz3yR5j29UjNQ==} engines: {node: ^22.22.3 || >=24.8.0} hasBin: true peerDependencies: @@ -5434,8 +5442,8 @@ packages: oniguruma-to-es@4.3.6: resolution: {integrity: sha512-csuQ9x3Yr0cEIs/Zgx/OEt9iBw9vqIunAPQkx19R/fiMq2oGVTgcMqO/V3Ybqefr1TBvosI6jU539ksaBULJyA==} - oxc-parser@0.154.0: - resolution: {integrity: sha512-Gr/6fkjZwi937c4Tsz8y0iO4U0FLPkZI6JSVucLAIvQyqh379px7RS8ORIc0HOORbFjXZ6zyqWLl53qxhSeofw==} + oxc-parser@0.157.0: + resolution: {integrity: sha512-2MJ9tZGXUbEIhQT/O2OPLlYCgur75ES26t2Pb1sYrvDQcRjrE7GmubkALgPEB6LzOY6sM81AAHKfl5M/FgnjJg==} engines: {node: ^20.19.0 || >=22.12.0} peerDependencies: '@oxc-parser/binding-wasm32-wasi': '*' @@ -5443,8 +5451,8 @@ packages: '@oxc-parser/binding-wasm32-wasi': optional: true - oxc-transform@0.154.0: - resolution: {integrity: sha512-f1i5S4wm+a5t6baY5mw/+LFLt95VpOwwcU56TlSGfd23c5DstR2k9RKj0vv16/7/ra+ptU09D83sNOL4fza1NA==} + oxc-transform@0.157.0: + resolution: {integrity: sha512-2H8XPFEbZQv/uoJ863a8YHdNrWWfETcKJ/fVutW7X1ryA5UhhNqaJsV1ib9y2pHOoHfofBIMzfIBCQmzLOVnTA==} engines: {node: ^20.19.0 || >=22.12.0} peerDependencies: '@oxc-transform/binding-wasm32-wasi': '*' @@ -6441,6 +6449,9 @@ packages: zod@4.6.1: resolution: {integrity: sha512-341aRWQsve0rvronKNTqZpjmzdbUDlFuzHaI/XLg/Ej82qffDJRRfBTCuv7+9q/rMjB6LSLyEBnW4InJeMtt/Q==} + zod@4.6.7: + resolution: {integrity: sha512-g8W2D2kU6a8untc053fX1uVBKzM7yqAxW/3zwk83+6RyF2+SKOuV+c0wBDhUwzV1PaGapmZH+rmcUzPf+3CGwg==} + zustand@4.5.7: resolution: {integrity: sha512-CHOUy7mu3lbD6o6LJLfllpjkzhHXSBlX8B9+qPddUsIfeF5S/UZ5q0kmCsnRqT1UHFQZchNFDDzMbQsuesHWlw==} engines: {node: '>=12.7.0'} @@ -6473,6 +6484,12 @@ snapshots: '@ai-sdk/provider-utils': 5.0.67(zod@4.6.1) zod: 4.6.1 + '@ai-sdk/anthropic@4.0.87(zod@4.6.7)': + dependencies: + '@ai-sdk/provider': 4.0.33 + '@ai-sdk/provider-utils': 5.0.67(zod@4.6.7) + zod: 4.6.7 + '@ai-sdk/gateway@4.0.125(zod@4.6.1)': dependencies: '@ai-sdk/provider': 4.0.33 @@ -6480,6 +6497,13 @@ snapshots: '@vercel/oidc': 3.2.0 zod: 4.6.1 + '@ai-sdk/gateway@4.0.125(zod@4.6.7)': + dependencies: + '@ai-sdk/provider': 4.0.33 + '@ai-sdk/provider-utils': 5.0.67(zod@4.6.7) + '@vercel/oidc': 3.2.0 + zod: 4.6.7 + '@ai-sdk/harness-claude-code@1.0.42(zod@4.6.1)': dependencies: '@ai-sdk/harness': 1.0.41(ws@8.21.0)(zod@4.6.1) @@ -6530,6 +6554,12 @@ snapshots: '@ai-sdk/provider-utils': 5.0.67(zod@4.6.1) zod: 4.6.1 + '@ai-sdk/openai@4.0.101(zod@4.6.7)': + dependencies: + '@ai-sdk/provider': 4.0.33 + '@ai-sdk/provider-utils': 5.0.67(zod@4.6.7) + zod: 4.6.7 + '@ai-sdk/openai@4.0.60(zod@4.6.1)': dependencies: '@ai-sdk/provider': 4.0.10 @@ -6562,11 +6592,16 @@ snapshots: undici: 7.30.1 zod: 4.6.1 - '@ai-sdk/provider@4.0.10': + '@ai-sdk/provider-utils@5.0.67(zod@4.6.7)': dependencies: - json-schema: 0.4.0 + '@ai-sdk/provider': 4.0.33 + '@standard-schema/spec': 1.1.0 + '@workflow/serde': 4.1.0 + eventsource-parser: 3.1.1 + undici: 7.30.1 + zod: 4.6.7 - '@ai-sdk/provider@4.0.28': + '@ai-sdk/provider@4.0.10': dependencies: json-schema: 0.4.0 @@ -6578,6 +6613,10 @@ snapshots: dependencies: json-schema: 0.4.0 + '@ai-sdk/provider@4.0.36': + dependencies: + json-schema: 0.4.0 + '@ai-sdk/typesafe-ai@3.0.26(zod@4.6.1)': dependencies: '@ai-sdk/provider': 4.0.33 @@ -7208,20 +7247,20 @@ snapshots: '@drizzle-team/brocli@0.12.1': {} - '@e2e-dev/decision@0.3.0(ai@7.0.152(zod@4.6.1))(e2e@0.21.3(@ai-sdk/anthropic@4.0.87(zod@4.6.1))(@ai-sdk/openai@4.0.101(zod@4.6.1))(@ai-sdk/openai-compatible@3.0.44(zod@4.6.1))(ai@7.0.152(zod@4.6.1)))': + '@e2e-dev/decision@0.3.2(ai@7.0.152(zod@4.6.1))(e2e@0.23.0(@ai-sdk/anthropic@4.0.87(zod@4.6.1))(@ai-sdk/openai@4.0.101(zod@4.6.1))(ai@7.0.152(zod@4.6.1)))': dependencies: - ai: 7.0.152(zod@4.6.1) - e2e: 0.21.3(patch_hash=1f55d7623fea3c16a58e22464d1a4919a5d805ba9da209e0b33068891295ba82)(@ai-sdk/anthropic@4.0.87(zod@4.6.1))(@ai-sdk/openai@4.0.101(zod@4.6.1))(@ai-sdk/openai-compatible@3.0.44(zod@4.6.1))(ai@7.0.152(zod@4.6.1)) + ai: 7.0.152(zod@4.6.7) + e2e: 0.23.0(patch_hash=1f55d7623fea3c16a58e22464d1a4919a5d805ba9da209e0b33068891295ba82)(@ai-sdk/anthropic@4.0.87(zod@4.6.1))(@ai-sdk/openai@4.0.101(zod@4.6.1))(ai@7.0.152(zod@4.6.1)) pngjs: 7.0.0 - zod: 4.6.1 + zod: 4.6.7 - '@e2e-dev/github@0.4.3(e2e@0.21.3(@ai-sdk/anthropic@4.0.87(zod@4.6.1))(@ai-sdk/openai@4.0.101(zod@4.6.1))(@ai-sdk/openai-compatible@3.0.44(zod@4.6.1))(ai@7.0.152(zod@4.6.1)))': + '@e2e-dev/github@0.4.4(e2e@0.23.0(@ai-sdk/anthropic@4.0.87(zod@4.6.1))(@ai-sdk/openai@4.0.101(zod@4.6.1))(ai@7.0.152(zod@4.6.1)))': dependencies: - e2e: 0.21.3(patch_hash=1f55d7623fea3c16a58e22464d1a4919a5d805ba9da209e0b33068891295ba82)(@ai-sdk/anthropic@4.0.87(zod@4.6.1))(@ai-sdk/openai@4.0.101(zod@4.6.1))(@ai-sdk/openai-compatible@3.0.44(zod@4.6.1))(ai@7.0.152(zod@4.6.1)) + e2e: 0.23.0(patch_hash=1f55d7623fea3c16a58e22464d1a4919a5d805ba9da209e0b33068891295ba82)(@ai-sdk/anthropic@4.0.87(zod@4.6.1))(@ai-sdk/openai@4.0.101(zod@4.6.1))(ai@7.0.152(zod@4.6.1)) - '@e2e-dev/web@0.15.4(e2e@0.21.3(@ai-sdk/anthropic@4.0.87(zod@4.6.1))(@ai-sdk/openai@4.0.101(zod@4.6.1))(@ai-sdk/openai-compatible@3.0.44(zod@4.6.1))(ai@7.0.152(zod@4.6.1)))': + '@e2e-dev/web@0.15.5(e2e@0.23.0(@ai-sdk/anthropic@4.0.87(zod@4.6.1))(@ai-sdk/openai@4.0.101(zod@4.6.1))(ai@7.0.152(zod@4.6.1)))': dependencies: - e2e: 0.21.3(patch_hash=1f55d7623fea3c16a58e22464d1a4919a5d805ba9da209e0b33068891295ba82)(@ai-sdk/anthropic@4.0.87(zod@4.6.1))(@ai-sdk/openai@4.0.101(zod@4.6.1))(@ai-sdk/openai-compatible@3.0.44(zod@4.6.1))(ai@7.0.152(zod@4.6.1)) + e2e: 0.23.0(patch_hash=1f55d7623fea3c16a58e22464d1a4919a5d805ba9da209e0b33068891295ba82)(@ai-sdk/anthropic@4.0.87(zod@4.6.1))(@ai-sdk/openai@4.0.101(zod@4.6.1))(ai@7.0.152(zod@4.6.1)) playwright-core: 1.64.1(zod@4.6.1) transitivePeerDependencies: - bufferutil @@ -7690,6 +7729,10 @@ snapshots: dependencies: zod: 4.6.1 + '@modelcontextprotocol/core@2.4.0': + dependencies: + zod: 4.6.1 + '@modelcontextprotocol/sdk@1.32.0(@cfworker/json-schema@4.1.1)(zod@4.6.1)': dependencies: '@cfworker/json-schema': 4.1.1 @@ -7718,6 +7761,11 @@ snapshots: '@modelcontextprotocol/core': 2.3.1 zod: 4.6.1 + '@modelcontextprotocol/server@2.4.0': + dependencies: + '@modelcontextprotocol/core': 2.4.0 + zod: 4.6.1 + '@mongodb-js/zstd@7.0.0': dependencies: node-addon-api: 8.9.2 @@ -7757,126 +7805,126 @@ snapshots: '@opentelemetry/semantic-conventions@1.43.0': {} - '@oxc-parser/binding-android-arm-eabi@0.154.0': + '@oxc-parser/binding-android-arm-eabi@0.157.0': optional: true - '@oxc-parser/binding-android-arm64@0.154.0': + '@oxc-parser/binding-android-arm64@0.157.0': optional: true - '@oxc-parser/binding-darwin-arm64@0.154.0': + '@oxc-parser/binding-darwin-arm64@0.157.0': optional: true - '@oxc-parser/binding-darwin-x64@0.154.0': + '@oxc-parser/binding-darwin-x64@0.157.0': optional: true - '@oxc-parser/binding-freebsd-x64@0.154.0': + '@oxc-parser/binding-freebsd-x64@0.157.0': optional: true - '@oxc-parser/binding-linux-arm-gnueabihf@0.154.0': + '@oxc-parser/binding-linux-arm-gnueabihf@0.157.0': optional: true - '@oxc-parser/binding-linux-arm-musleabihf@0.154.0': + '@oxc-parser/binding-linux-arm-musleabihf@0.157.0': optional: true - '@oxc-parser/binding-linux-arm64-gnu@0.154.0': + '@oxc-parser/binding-linux-arm64-gnu@0.157.0': optional: true - '@oxc-parser/binding-linux-arm64-musl@0.154.0': + '@oxc-parser/binding-linux-arm64-musl@0.157.0': optional: true - '@oxc-parser/binding-linux-ppc64-gnu@0.154.0': + '@oxc-parser/binding-linux-ppc64-gnu@0.157.0': optional: true - '@oxc-parser/binding-linux-riscv64-gnu@0.154.0': + '@oxc-parser/binding-linux-riscv64-gnu@0.157.0': optional: true - '@oxc-parser/binding-linux-riscv64-musl@0.154.0': + '@oxc-parser/binding-linux-riscv64-musl@0.157.0': optional: true - '@oxc-parser/binding-linux-s390x-gnu@0.154.0': + '@oxc-parser/binding-linux-s390x-gnu@0.157.0': optional: true - '@oxc-parser/binding-linux-x64-gnu@0.154.0': + '@oxc-parser/binding-linux-x64-gnu@0.157.0': optional: true - '@oxc-parser/binding-linux-x64-musl@0.154.0': + '@oxc-parser/binding-linux-x64-musl@0.157.0': optional: true - '@oxc-parser/binding-openharmony-arm64@0.154.0': + '@oxc-parser/binding-openharmony-arm64@0.157.0': optional: true - '@oxc-parser/binding-win32-arm64-msvc@0.154.0': + '@oxc-parser/binding-win32-arm64-msvc@0.157.0': optional: true - '@oxc-parser/binding-win32-ia32-msvc@0.154.0': + '@oxc-parser/binding-win32-ia32-msvc@0.157.0': optional: true - '@oxc-parser/binding-win32-x64-msvc@0.154.0': + '@oxc-parser/binding-win32-x64-msvc@0.157.0': optional: true - '@oxc-project/runtime@0.154.0': {} + '@oxc-project/runtime@0.157.0': {} '@oxc-project/types@0.153.0': {} - '@oxc-project/types@0.154.0': {} - '@oxc-project/types@0.155.0': {} - '@oxc-transform/binding-android-arm-eabi@0.154.0': + '@oxc-project/types@0.157.0': {} + + '@oxc-transform/binding-android-arm-eabi@0.157.0': optional: true - '@oxc-transform/binding-android-arm64@0.154.0': + '@oxc-transform/binding-android-arm64@0.157.0': optional: true - '@oxc-transform/binding-darwin-arm64@0.154.0': + '@oxc-transform/binding-darwin-arm64@0.157.0': optional: true - '@oxc-transform/binding-darwin-x64@0.154.0': + '@oxc-transform/binding-darwin-x64@0.157.0': optional: true - '@oxc-transform/binding-freebsd-x64@0.154.0': + '@oxc-transform/binding-freebsd-x64@0.157.0': optional: true - '@oxc-transform/binding-linux-arm-gnueabihf@0.154.0': + '@oxc-transform/binding-linux-arm-gnueabihf@0.157.0': optional: true - '@oxc-transform/binding-linux-arm-musleabihf@0.154.0': + '@oxc-transform/binding-linux-arm-musleabihf@0.157.0': optional: true - '@oxc-transform/binding-linux-arm64-gnu@0.154.0': + '@oxc-transform/binding-linux-arm64-gnu@0.157.0': optional: true - '@oxc-transform/binding-linux-arm64-musl@0.154.0': + '@oxc-transform/binding-linux-arm64-musl@0.157.0': optional: true - '@oxc-transform/binding-linux-ppc64-gnu@0.154.0': + '@oxc-transform/binding-linux-ppc64-gnu@0.157.0': optional: true - '@oxc-transform/binding-linux-riscv64-gnu@0.154.0': + '@oxc-transform/binding-linux-riscv64-gnu@0.157.0': optional: true - '@oxc-transform/binding-linux-riscv64-musl@0.154.0': + '@oxc-transform/binding-linux-riscv64-musl@0.157.0': optional: true - '@oxc-transform/binding-linux-s390x-gnu@0.154.0': + '@oxc-transform/binding-linux-s390x-gnu@0.157.0': optional: true - '@oxc-transform/binding-linux-x64-gnu@0.154.0': + '@oxc-transform/binding-linux-x64-gnu@0.157.0': optional: true - '@oxc-transform/binding-linux-x64-musl@0.154.0': + '@oxc-transform/binding-linux-x64-musl@0.157.0': optional: true - '@oxc-transform/binding-openharmony-arm64@0.154.0': + '@oxc-transform/binding-openharmony-arm64@0.157.0': optional: true - '@oxc-transform/binding-win32-arm64-msvc@0.154.0': + '@oxc-transform/binding-win32-arm64-msvc@0.157.0': optional: true - '@oxc-transform/binding-win32-ia32-msvc@0.154.0': + '@oxc-transform/binding-win32-ia32-msvc@0.157.0': optional: true - '@oxc-transform/binding-win32-x64-msvc@0.154.0': + '@oxc-transform/binding-win32-x64-msvc@0.157.0': optional: true '@oxfmt/binding-android-arm-eabi@0.74.0': @@ -8876,24 +8924,24 @@ snapshots: '@rolldown/pluginutils': 1.0.1 vite: 8.3.5(@types/node@24.19.2)(esbuild@0.28.2)(tsx@4.23.15) - '@vitest/expect@5.0.5': + '@vitest/expect@5.1.0': dependencies: '@standard-schema/spec': 1.1.0 '@types/chai': 5.2.3 - '@vitest/spy': 5.0.5 - '@vitest/utils': 5.0.5 + '@vitest/spy': 5.1.0 + '@vitest/utils': 5.1.0 chai: 6.3.0 tinyrainbow: 3.2.0 - '@vitest/pretty-format@5.0.5': + '@vitest/pretty-format@5.1.0': dependencies: tinyrainbow: 3.2.0 - '@vitest/spy@5.0.5': {} + '@vitest/spy@5.1.0': {} - '@vitest/utils@5.0.5': + '@vitest/utils@5.1.0': dependencies: - '@vitest/pretty-format': 5.0.5 + '@vitest/pretty-format': 5.1.0 convert-source-map: 2.0.0 tinyrainbow: 3.2.0 @@ -8995,6 +9043,13 @@ snapshots: '@ai-sdk/provider-utils': 5.0.67(zod@4.6.1) zod: 4.6.1 + ai@7.0.152(zod@4.6.7): + dependencies: + '@ai-sdk/gateway': 4.0.125(zod@4.6.7) + '@ai-sdk/provider': 4.0.33 + '@ai-sdk/provider-utils': 5.0.67(zod@4.6.7) + zod: 4.6.7 + ajv-formats@3.0.1(ajv@8.20.0): dependencies: ajv: 8.20.0 @@ -9397,26 +9452,25 @@ snapshots: es-errors: 1.3.0 gopd: 1.2.0 - e2e@0.21.3(patch_hash=1f55d7623fea3c16a58e22464d1a4919a5d805ba9da209e0b33068891295ba82)(@ai-sdk/anthropic@4.0.87(zod@4.6.1))(@ai-sdk/openai@4.0.101(zod@4.6.1))(@ai-sdk/openai-compatible@3.0.44(zod@4.6.1))(ai@7.0.152(zod@4.6.1)): + e2e@0.23.0(patch_hash=1f55d7623fea3c16a58e22464d1a4919a5d805ba9da209e0b33068891295ba82)(@ai-sdk/anthropic@4.0.87(zod@4.6.1))(@ai-sdk/openai@4.0.101(zod@4.6.1))(ai@7.0.152(zod@4.6.1)): dependencies: - '@ai-sdk/anthropic': 4.0.87(zod@4.6.1) - '@ai-sdk/openai': 4.0.101(zod@4.6.1) - '@ai-sdk/openai-compatible': 3.0.44(zod@4.6.1) - '@ai-sdk/provider': 4.0.28 + '@ai-sdk/anthropic': 4.0.87(zod@4.6.7) + '@ai-sdk/openai': 4.0.101(zod@4.6.7) + '@ai-sdk/provider': 4.0.36 '@clack/prompts': 1.8.1 '@jridgewell/remapping': 2.3.5 - '@modelcontextprotocol/server': 2.3.1 - '@oxc-project/runtime': 0.154.0 - '@vitest/expect': 5.0.5 - ai: 7.0.152(zod@4.6.1) + '@modelcontextprotocol/server': 2.4.0 + '@oxc-project/runtime': 0.157.0 + '@vitest/expect': 5.1.0 + ai: 7.0.152(zod@4.6.7) commander: 15.0.0 fast-png: 8.0.0 get-tsconfig: 4.14.3 - oxc-parser: 0.154.0 - oxc-transform: 0.154.0 + oxc-parser: 0.157.0 + oxc-transform: 0.157.0 picocolors: 1.1.1 pixelmatch: 8.0.0 - zod: 4.6.1 + zod: 4.6.7 transitivePeerDependencies: - '@oxc-parser/binding-wasm32-wasi' - '@oxc-transform/binding-wasm32-wasi' @@ -10583,51 +10637,51 @@ snapshots: regex: 6.1.0 regex-recursion: 6.0.2 - oxc-parser@0.154.0: + oxc-parser@0.157.0: dependencies: - '@oxc-project/types': 0.154.0 + '@oxc-project/types': 0.157.0 optionalDependencies: - '@oxc-parser/binding-android-arm-eabi': 0.154.0 - '@oxc-parser/binding-android-arm64': 0.154.0 - '@oxc-parser/binding-darwin-arm64': 0.154.0 - '@oxc-parser/binding-darwin-x64': 0.154.0 - '@oxc-parser/binding-freebsd-x64': 0.154.0 - '@oxc-parser/binding-linux-arm-gnueabihf': 0.154.0 - '@oxc-parser/binding-linux-arm-musleabihf': 0.154.0 - '@oxc-parser/binding-linux-arm64-gnu': 0.154.0 - '@oxc-parser/binding-linux-arm64-musl': 0.154.0 - '@oxc-parser/binding-linux-ppc64-gnu': 0.154.0 - '@oxc-parser/binding-linux-riscv64-gnu': 0.154.0 - '@oxc-parser/binding-linux-riscv64-musl': 0.154.0 - '@oxc-parser/binding-linux-s390x-gnu': 0.154.0 - '@oxc-parser/binding-linux-x64-gnu': 0.154.0 - '@oxc-parser/binding-linux-x64-musl': 0.154.0 - '@oxc-parser/binding-openharmony-arm64': 0.154.0 - '@oxc-parser/binding-win32-arm64-msvc': 0.154.0 - '@oxc-parser/binding-win32-ia32-msvc': 0.154.0 - '@oxc-parser/binding-win32-x64-msvc': 0.154.0 - - oxc-transform@0.154.0: + '@oxc-parser/binding-android-arm-eabi': 0.157.0 + '@oxc-parser/binding-android-arm64': 0.157.0 + '@oxc-parser/binding-darwin-arm64': 0.157.0 + '@oxc-parser/binding-darwin-x64': 0.157.0 + '@oxc-parser/binding-freebsd-x64': 0.157.0 + '@oxc-parser/binding-linux-arm-gnueabihf': 0.157.0 + '@oxc-parser/binding-linux-arm-musleabihf': 0.157.0 + '@oxc-parser/binding-linux-arm64-gnu': 0.157.0 + '@oxc-parser/binding-linux-arm64-musl': 0.157.0 + '@oxc-parser/binding-linux-ppc64-gnu': 0.157.0 + '@oxc-parser/binding-linux-riscv64-gnu': 0.157.0 + '@oxc-parser/binding-linux-riscv64-musl': 0.157.0 + '@oxc-parser/binding-linux-s390x-gnu': 0.157.0 + '@oxc-parser/binding-linux-x64-gnu': 0.157.0 + '@oxc-parser/binding-linux-x64-musl': 0.157.0 + '@oxc-parser/binding-openharmony-arm64': 0.157.0 + '@oxc-parser/binding-win32-arm64-msvc': 0.157.0 + '@oxc-parser/binding-win32-ia32-msvc': 0.157.0 + '@oxc-parser/binding-win32-x64-msvc': 0.157.0 + + oxc-transform@0.157.0: optionalDependencies: - '@oxc-transform/binding-android-arm-eabi': 0.154.0 - '@oxc-transform/binding-android-arm64': 0.154.0 - '@oxc-transform/binding-darwin-arm64': 0.154.0 - '@oxc-transform/binding-darwin-x64': 0.154.0 - '@oxc-transform/binding-freebsd-x64': 0.154.0 - '@oxc-transform/binding-linux-arm-gnueabihf': 0.154.0 - '@oxc-transform/binding-linux-arm-musleabihf': 0.154.0 - '@oxc-transform/binding-linux-arm64-gnu': 0.154.0 - '@oxc-transform/binding-linux-arm64-musl': 0.154.0 - '@oxc-transform/binding-linux-ppc64-gnu': 0.154.0 - '@oxc-transform/binding-linux-riscv64-gnu': 0.154.0 - '@oxc-transform/binding-linux-riscv64-musl': 0.154.0 - '@oxc-transform/binding-linux-s390x-gnu': 0.154.0 - '@oxc-transform/binding-linux-x64-gnu': 0.154.0 - '@oxc-transform/binding-linux-x64-musl': 0.154.0 - '@oxc-transform/binding-openharmony-arm64': 0.154.0 - '@oxc-transform/binding-win32-arm64-msvc': 0.154.0 - '@oxc-transform/binding-win32-ia32-msvc': 0.154.0 - '@oxc-transform/binding-win32-x64-msvc': 0.154.0 + '@oxc-transform/binding-android-arm-eabi': 0.157.0 + '@oxc-transform/binding-android-arm64': 0.157.0 + '@oxc-transform/binding-darwin-arm64': 0.157.0 + '@oxc-transform/binding-darwin-x64': 0.157.0 + '@oxc-transform/binding-freebsd-x64': 0.157.0 + '@oxc-transform/binding-linux-arm-gnueabihf': 0.157.0 + '@oxc-transform/binding-linux-arm-musleabihf': 0.157.0 + '@oxc-transform/binding-linux-arm64-gnu': 0.157.0 + '@oxc-transform/binding-linux-arm64-musl': 0.157.0 + '@oxc-transform/binding-linux-ppc64-gnu': 0.157.0 + '@oxc-transform/binding-linux-riscv64-gnu': 0.157.0 + '@oxc-transform/binding-linux-riscv64-musl': 0.157.0 + '@oxc-transform/binding-linux-s390x-gnu': 0.157.0 + '@oxc-transform/binding-linux-x64-gnu': 0.157.0 + '@oxc-transform/binding-linux-x64-musl': 0.157.0 + '@oxc-transform/binding-openharmony-arm64': 0.157.0 + '@oxc-transform/binding-win32-arm64-msvc': 0.157.0 + '@oxc-transform/binding-win32-ia32-msvc': 0.157.0 + '@oxc-transform/binding-win32-x64-msvc': 0.157.0 oxfmt@0.74.0: dependencies: @@ -11662,6 +11716,8 @@ snapshots: zod@4.6.1: {} + zod@4.6.7: {} + zustand@4.5.7(@types/react@19.2.18)(react@19.2.8): dependencies: '@types/react': 19.2.18 diff --git a/package.json b/package.json index 6758f2a7..4b917066 100644 --- a/package.json +++ b/package.json @@ -85,9 +85,9 @@ "@ai-sdk/openai": "4.0.101", "@ai-sdk/typesafe-ai": "3.0.26", "@cloudflare/workers-types": "^4.20260702.1", - "@e2e-dev/decision": "0.3.0", - "@e2e-dev/github": "0.4.3", - "@e2e-dev/web": "0.15.4", + "@e2e-dev/decision": "0.3.2", + "@e2e-dev/github": "0.4.4", + "@e2e-dev/web": "0.15.5", "@electric-sql/pglite": "0.4.5", "@electric-sql/pglite-socket": "0.1.5", "@modelcontextprotocol/sdk": "1.32.0", @@ -97,7 +97,7 @@ "@yielded/effect-tsrs": "0.2.2", "ai": "7.0.152", "drizzle-kit": "1.0.0-rc.5-ab785fc", - "e2e": "0.21.3", + "e2e": "0.23.0", "esbuild": "0.28.2", "jsonc-parser": "^3.3.1", "lefthook": "^2.2.2", @@ -156,6 +156,6 @@ "patchedDependencies": { "@cloudflare/containers@0.3.7": "patches/@cloudflare__containers@0.3.7.patch", "@ai-sdk/harness-codex@1.0.43": "patches/@ai-sdk+harness-codex@1.0.43.patch", - "e2e@0.21.3": "patches/e2e@0.21.3.patch" + "e2e@0.23.0": "patches/e2e@0.23.0.patch" } } diff --git a/patches/e2e@0.21.3.patch b/patches/e2e@0.23.0.patch similarity index 100% rename from patches/e2e@0.21.3.patch rename to patches/e2e@0.23.0.patch