diff --git a/.device-evidence/CHECKLIST-runner-failures.md b/.device-evidence/CHECKLIST-runner-failures.md deleted file mode 100644 index b69fcb13c4..0000000000 --- a/.device-evidence/CHECKLIST-runner-failures.md +++ /dev/null @@ -1,310 +0,0 @@ -# Device evidence checklist - -Live evidence the coordinator runs serially on the connected iPhone. Each item names the exact -command, the environment it needs, and the rendered error that proves the change. Do not paraphrase -the JSON: paste it. Record the commit SHA the build under test was made from. - -## #2680 — typed build-failure reasons - -Build the CLI first, then stop any warm daemon so the run is on this commit: - -```sh -pnpm build && pnpm clean:daemon -node --experimental-strip-types src/bin.ts daemon stop --all || true -``` - -### 1. Signing with no team configured -> `signing_no_development_team` - -```sh -env -u AGENT_DEVICE_IOS_TEAM_ID -u AGENT_DEVICE_IOS_PROVISIONING_PROFILE \ - node --experimental-strip-types src/bin.ts --json \ - prepare ios-runner --platform ios --device "" -``` - -Expected: exit non-zero, one error object with - -```json -{ - "code": "COMMAND_FAILED", - "message": "xcodebuild build-for-testing failed", - "hint": "Configure signing in Xcode or set AGENT_DEVICE_IOS_TEAM_ID for physical-device runs.", - "details": { "reason": "signing_no_development_team" } -} -``` - -`hint`, `logPath` and `diagnosticId` are top-level, never inside `details`. Also record the Xcode -version (`xcodebuild -version`) so the `signing_no_development_team` fixture in -`packages/platform-apple/src/runner/__tests__/runner-startup-failure-fixtures.ts` can move from -`shipped-sniff-trigger` to `captured` and its `xcodeVersion` from `unobserved` to that version. -Paste the whole error so the fixture's `output` can become the capture and its `command` can be -recorded. - -### 2. A bundle identifier somebody else already owns -> `bundle_identifier_already_registered` - -```sh -env AGENT_DEVICE_IOS_TEAM_ID="" \ - AGENT_DEVICE_IOS_BUNDLE_ID="com.apple.TestFlight" \ - node --experimental-strip-types src/bin.ts --json \ - prepare ios-runner --platform ios --device "" -``` - -Expected: same envelope shape with -`details.reason: "bundle_identifier_already_registered"` and a hint naming -`AGENT_DEVICE_IOS_BUNDLE_ID`. A registered-but-foreign identifier may surface the `Failed registering -bundle identifier` line or the `App Identifier ... is not available` line; both rules produce this -one reason, so record which line xcodebuild printed. - -### 3. Reasons with no device exposure (host-side or configuration-only) - -These do not need the iPhone, but do need a real xcodebuild run; record output with the Xcode -version so the matching fixture's `provenance` can be upgraded: - -```sh -# devtools_security_developer_mode_disabled (macOS admin state, no device work) -DevToolsSecurity -status - -# signing_provisioning_profile_missing: point the runner build at a profile that is not installed, -# then read the reason off the same prepare command as above. -env AGENT_DEVICE_IOS_TEAM_ID="" \ - AGENT_DEVICE_IOS_PROVISIONING_PROFILE="no-such-profile-installed" \ - node --experimental-strip-types src/bin.ts --json \ - prepare ios-runner --platform ios --device "" -``` - -Expected: `details.reason` is `signing_provisioning_profile_missing`, and the profile rows only fire -when xcodebuild says what is wrong with the profile **on the same line as the profile**: its -`IDEProvisioningErrorDomain` diagnostic naming the profile, `doesn't include ...`, or `has expired` -(#2688 review). Paste the whole error and keep the line breaks — which line carried which phrase is -what promotes the `profile-xcode-signing-error`, `profile-does-not-cover-app-id` and `profile-expired` -fixtures from `invented-shape` to `captured`, and a capture that splits the two phrases across lines -belongs to `profile-note-above-an-expired-certificate` instead. A different reason is worth recording -just as much: say which one and treat the fixtures as unconfirmed rather than editing the rules to -fit. - -### 4. The line that claims no reason yet -> `build_failed_unclassified` - -`xcodebuild` reports a settings mismatch with a line that names a profile ("has conflicting -provisioning settings"). #2680 deliberately publishes `build_failed_unclassified` for it, because no -capture has proved which lever clears it. To reach it, pin a profile while leaving automatic signing -on: - -```sh -env AGENT_DEVICE_IOS_TEAM_ID="" \ - AGENT_DEVICE_IOS_PROVISIONING_PROFILE="match-development" \ - node --experimental-strip-types src/bin.ts --json \ - prepare ios-runner --platform ios --device "" -``` - -Expected: either `signing_provisioning_profile_missing` (xcodebuild complained about the profile and -said what was wrong with it) or `build_failed_unclassified` — which is also what a run that merely -mentions the profile it used gets, since a name is not a complaint (#2688 review). Paste the error and the `xcodebuild -version` either way: a -capture of the conflicting-settings line is what would let a follow-up name the cause, and the -capture must show which build setting disagrees before any hint naming a lever is written. -## #2683 — device-readiness facts from `devicectl device info details` - -Build the CLI first, and stop any warm daemon so the run is on this commit (same preamble as the -#2680 section above). - -### 1. Both facts are readable, and they are two facts - -```sh -xcrun devicectl device info details --device "" --json-output /tmp/device-details.json -node -e 'const d=require("/tmp/device-details.json").result.deviceProperties;console.log(JSON.stringify({developerModeStatus:d.developerModeStatus,ddiServicesAvailable:d.ddiServicesAvailable}))' -xcodebuild -version -``` - -Expected: `{"developerModeStatus":"enabled","ddiServicesAvailable":true}` on a healthy device, plus -the Xcode version. Also record `bootState` and `tunnelState` from the same file: those two are what -make a `ddiServicesAvailable: false` an answer at all rather than a device that was not listening. - -`packages/platform-apple/src/core/__tests__/fixtures/ios-device-info-details.json` holds this payload -and `packages/platform-apple/src/runner/__tests__/runner-startup-failure-fixtures.ts` records the two -state pairings, both at `invented-shape` until this capture moves them to `captured`. Paste the raw -values rather than a summary. - -The committed payload is masked, and the mask is a rule with a test, not a one-time edit: -`hardwareProperties.serialNumber` and `deviceProperties.bootedSnapshotName` carry `MASKED`, -`hardwareProperties.ecid` is `0`, and `connectionProperties.tunnelIPAddress` is a documentation-only -`fd00:` address. Everything the reader consumes — the toggle, the image, `bootState`, `tunnelState`, -OS build — stays verbatim. If you refresh this capture, apply the same mask in both the `result` -block and the mirrored `properties` block, and leave the states alone. - -### 2. A healthy device is left alone - -```sh -node --experimental-strip-types src/bin.ts --json prepare ios-runner --platform ios --device "" -``` - -Expected: success as before, and a `ios_runner_session_startup` diagnostic whose timings include -`verify_device_readiness`. No reason may appear on a healthy device: the probe reads, it does not -guess. Record the `verify_device_readiness` duration next to `verify_host_dev_tools_security`. - -### 3. Developer Mode off on the device -> `device_developer_mode_disabled` - -Turn the toggle off on a device you are willing to re-pair (Settings > Privacy & Security > -Developer Mode, then restart), and rerun the `prepare ios-runner` command above. - -Expected: exit non-zero with - -```json -{ - "code": "COMMAND_FAILED", - "message": "The iOS device reports that Developer Mode is turned off", - "details": { - "reason": "device_developer_mode_disabled", - "developerMode": "disabled", - "developerDiskImage": "unavailable" - } -} -``` - -`hint` is top-level and names `Settings > Privacy & Security > Developer Mode`. Record what -`developerDiskImage` says; either value is acceptable as long as the toggle stays the reason. - -### 4. Developer disk image down with the toggle on -> carried onto the failure, not a refusal - -This is the pairing the old hint got wrong, so it is the evidence that matters. Reach it with a -device whose iOS build is newer than the installed Xcode supports, or before Xcode finishes -installing device support for a freshly paired phone, with Developer Mode on. - -This state does NOT stop the run before the build (#2683 review): since iOS 17 CoreDevice mounts the -personalized disk image on demand during build and launch, a phone that has just been rebooted reports -the image down while the very next build clears it. Run the `prepare ios-runner` command above and -record which of the two outcomes you get — both are evidence, and the second one is the reason the -pre-build refusal was removed: - -- **The build clears it**: the command succeeds. Paste the `ios_runner_session_startup` timings showing - `verify_device_readiness` ran and `ensure_xctestrun` followed it. Nothing may name device support. -- **The build fails and names no cause of its own**: `details.reason` is - `device_developer_disk_image_unavailable` and `details.developerDiskImage` is `unavailable`, and the - hint names device support WITHOUT mentioning `Settings > Privacy & Security`. -- **The build fails and names its own cause** (e.g. `signing_no_development_team`): that reason wins and - `details.developerDiskImage` is still `unavailable`. The device state never overwrites xcodebuild's - sentence. - -If the toggle reason appears here instead, that is the bug this issue exists to fix: paste the whole -error and the `/tmp/device-details.json` payload rather than adjusting a rule. - -Capture the details payload at the same moment as the run, and check it says `tunnelState: -"connected"` and `bootState: "booted"`. A `ddiServicesAvailable: false` read any other way is not this -case: an asleep or unreachable device has the same field and no obstacle, and the run must not name -device support for it. The hint you get here is the one `core/devicectl.ts` owns and the device report -carries, so it is worded identically to the hint `devicectl` output produces for the same complaint — -paste both strings and confirm they match character for character. - -### 5. A device that cannot be read claims nothing - -Unplug the iPhone (or shut it down) after a session exists, then rerun the `prepare ios-runner` -command. - -Expected: the failure names whatever the transport could not reach, and no `details.reason` of -`device_developer_mode_disabled` or `device_developer_disk_image_unavailable` appears anywhere in the -error. An unreadable device is never diagnosed. This is also the case that catches a sleeping phone -reporting `ddiServicesAvailable: false`: if the hint here says `Let Xcode finish preparing this -device`, the corroboration rule has been lost, and pasting the payload with its `tunnelState` and -`bootState` is the evidence — do not adjust the rule to make the run pass. - -### 6. A device and a Mac that are both wrong publish the device's reason - -With the iPhone's Developer Mode toggle off AND `sudo DevToolsSecurity -status` reporting the -developer-tools setting disabled, run the `prepare ios-runner` command. - -Expected: one error, whose `details.reason` is `device_developer_mode_disabled`. The Mac's reason is -the other one (`devtools_security_developer_mode_disabled`) and must not be what you get: the phone's -fix needs no admin rights on the Mac, so the probe that can be acted on has to be the one that speaks. -Record which of the two appears; a captured pairing here is what would let a follow-up drop the -ordering argument. - -### 7. Log evidence belongs to the command that wrote it (optional, needs a crash repro) - -Run any command that crashes the app under test, then a second command that fails for its own -reason (for example a selector that no longer exists). - -Expected: the second error carries no `details.runnerFailureReason`. Before this change it inherited -`target_app_axruntime_coretext_crash` from the first command's lines in `runner.log`. If the repro -is not reachable, say so; the pairing is covered by -`packages/platform-apple/src/runner/__tests__/runner-failure-diagnostics.test.ts`. - -## Results — coordinator run, 2026-09-20 (both families) - -Built from `15808ae228` on `thymikee-iphone` (iPhone 17 Pro, iOS 27.0, build 24A437), cabled. - -``` -$ xcodebuild -version -Xcode 26.2 -Build version 17C52 -``` - -#2680 §3 is captured. A device build pointed at a team with no certificate, on a fresh derived -path so no cached artifact short-circuits it, reaches signing and fails with one long `error:` line -per target: - -``` -.../AgentDeviceRunner.xcodeproj: error: No Accounts: Add a new account in Accounts settings. (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner') -.../AgentDeviceRunner.xcodeproj: error: No profiles for 'com.callstack.agentdevice.runner' were found: Xcode couldn't find any iOS App Development provisioning profiles matching 'com.callstack.agentdevice.runner'. (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner') -``` - -`details.reason` is `signing_provisioning_profile_missing` with the profile hint, and the fixture -`no-profiles-for-bundle-id` is now `captured` with this transcript verbatim. This is the run that -answers the wrapping question the rows were held on: the matched phrase arrives inside one `error:` -line, so the sibling rows in the same provisioning family do not split the way a wrapped line would. - -The #2680 sections 1 and 2 are blocked on this account, and the mechanism is worth recording because it is the -same for all of them: against a signed-in account with a valid identity, `xcodebuild` is invoked with -`-allowProvisioningUpdates`, so the build either signs successfully or dies earlier than the -diagnostic a row keys on. - -- Unsetting `AGENT_DEVICE_IOS_TEAM_ID` **succeeds** — automatic signing resolves the team from the - installed identity and reuses an installed team profile. So section 1's `signing_no_development_team` - cannot be induced here; it needs an account signed in with no development team. -- `AGENT_DEVICE_IOS_BUNDLE_ID=com.apple.TestFlight` **succeeds** for the same reason, and a bogus - `AGENT_DEVICE_IOS_PROVISIONING_PROFILE` is repaired rather than honoured. So §2's - `bundle_identifier_already_registered` needs an app id owned by a different team that automatic - signing cannot register. -- The same gating applies to `bundle_identifier_unavailable` (`App Identifier` + `not available`), - `profile-does-not-cover-app-id` (`Provisioning profile` + `doesn't include`) and `profile-expired` - (`Provisioning profile` + `has expired`): each needs a profile or app id already claimed elsewhere, - which this account will not produce. Recorded beside the fixtures in - `runner-startup-failure-fixtures.ts` so the rows read as host-gated, not unexamined. - -#2680 §4 needs a build that fails for an unrelated reason while naming no signing fact; the -classifier's behaviour there is pinned by `runner-xctestrun-startup-failure-reasons.test.ts` and needs no -device claim to hold. - -### #2683 — what the phone actually reported - -Developer Mode off, same device, `prepare ios-runner --platform ios --json`: - -``` -"deviceProperties": { "developerModeStatus": "disabled", "ddiServicesAvailable": true } - -details.reason device_developer_mode_disabled -details.deviceReadiness { developerMode: "disabled", developerDiskImage: "available" } -hint Enable Developer Mode on the iOS device (Settings > Privacy & Security > - Developer Mode), restart it when prompted, unlock it, then retry. -``` - -The recognised spelling is the device's own lowercase `"disabled"`, so the one refusal -`preflightIosRunnerDeviceReadiness` raises is proven rather than inferred, and an unrecognised -spelling is no longer a live risk for this state. - -Image-down was reached by rebooting and holding the phone locked, watching `devicectl` until -`ddiServicesAvailable` read `false` while `bootState` was `booted`. Two findings from it: - -- The state is only reachable **while locked**. `ddiServicesAvailable` flips back to `true` within - seconds of unlock, so an unlocked image-down device does not exist on iOS 27 and the capture this - checklist asked for cannot be produced on it. -- Inside that window `prepare` fails at the connect stage — `Runner did not accept connection - (xcodebuild exited early)`, exit 70 — and publishes no `developerDiskImage`. A rerun at `eaf411e`, - where both connect-stage failures carry the device states, gave the same error, exit 70 and - `IOS_RUNNER_CONNECT_TIMEOUT` with no `developerDiskImage`. The cause is the observability rule, not - the call site. Before the first unlock the tunnel never comes up: every `devicectl device info - details` read from 10:56 to 10:58 UTC, before and after that run, said `tunnelState: "unavailable"`, - `bootState: "booted"`, `ddiServicesAvailable: false`. So `readDeviceReadiness` returns - `available: false`, the session has no device states, and there is nothing to carry. On iOS 27 a - locked phone does not publish its image state at all. The connect-stage enrichment carries the - fact only when the preflight could observe it (tunnel connected, device booted). - -The host-deadline invariant also held under a real device fault: on a 25s budget the same -image-down state produced `details.reason: prepare_deadline_expired` with **no** device reason and no -readiness facts, across every sample of two separate windows. diff --git a/.device-evidence/CHECKLIST.md b/.device-evidence/CHECKLIST.md deleted file mode 100644 index 63f61f0c85..0000000000 --- a/.device-evidence/CHECKLIST.md +++ /dev/null @@ -1,131 +0,0 @@ -# #2682 physical-device checklist — `thymikee-iphone` (coordinator-owned) - -Simulator evidence is recorded in `DECISION.md`. This is the same scenario on the physical lane. -Use a purpose-specific session name; close it afterwards. Do not reuse another lane's session. - -```sh -UDID= -S="--platform ios --udid $UDID --session apex2682-physical" -bin/agent-device.mjs open com.callstack.agentdevicelab $S --foreground -``` - -The fixture must show its own surface (Metro/dev-client per `examples/test-app/README.md`); an -already-installed bundle id is not evidence. - -## Sequence - -1. `bin/agent-device.mjs screenshot --out /tmp/2682-1-app.png $S` → fixture app. -2. Hand off inside the app (prefer a real in-app handoff; the AccessorySetupKit/permission or an - external link surface both qualify). Fallback that exercises the identical runner path: - `xcrun devicectl handle URL`/a user tap that opens Safari or Settings. -3. `bin/agent-device.mjs screenshot --out /tmp/2682-2-foreign.png $S` -4. `bin/agent-device.mjs snapshot -i $S | tee /tmp/2682-3-snapshot.txt` -5. `bin/agent-device.mjs snapshot -i $S` (again, app already foreground) -6. `bin/agent-device.mjs close $S` - -## Expected BEFORE (`main`, no disclosure) - -- Step 3 shows the other app; step 4 shows the fixture app's tree. -- Nothing in either output says why. The only record is in the session's `runner.log`: - `AGENT_DEVICE_RUNNER_ACTIVATE bundle=com.callstack.agentdevicelab state=<2|3> reason=`. -- A run where step 3 still shows the fixture app is not evidence — the handoff must be visible. - -## Expected AFTER (PR1 head) - -- Step 4 carries, as a warning line and as `data.targetActivation` from PR2: - `The session app was not foreground when this command arrived (prior state runningBackground), so - the runner activated it before answering (reason - ). Any capture taken - earlier in this session described the only app other than the session app with an active - accessibility session (pid ), not the session app. Re-capture now that the session app answers, - or drive the other app in its own session.` -- Step 5 carries no disclosure — the fact belongs to the command that activated. -- `` is a **liveness** claim and the sentence is written to claim no more: `otherActiveApplicationPid` - is present only when exactly one application other than the session app held an active accessibility - session. `activeApplications` exposes no ordering, so nothing here proves which app owned the screen. - A pid that is not an AX-active application of the session is a bug; an absent pid is not. - -## Proving the disclosure comes from activation, not a later state read - -`priorState` is read at `RunnerTests+Lifecycle.swift` inside `activateTarget` **before** -`XCUIApplication.activate()` runs, and the fact is stamped only in the branch where `activate()` is -actually called. Assert the pair on the same device run: - -- `runner.log`: `AGENT_DEVICE_RUNNER_ACTIVATE_FACT bundle=... reason=... priorState=<2|3> - otherActiveApplicationPid=` — the same line's `priorState` is what the response must carry, named - by the raw values the SDK declares: 2 is `runningBackgroundSuspended`, 3 is `runningBackground` - (#2726 — the decoder had these two reversed). `state=` on the preceding - `AGENT_DEVICE_RUNNER_ACTIVATE` must be non-foreground. -- The response can therefore never report `priorState: runningForeground`. Both directions are pinned - in the runner unit lane (`UnitTests/RunnerTests+LifecycleCacheTests.swift`): the already-foreground - call asserts `pendingTargetActivation == nil` and the backgrounded call asserts the stamped - `priorState` equals `XCUIApplication.State.runningBackground.rawValue`. -- Correlate `` with the app from step 3: `xcrun devicectl list processes --device $UDID` (or the - physical-lane equivalent of the simulator's `ps`) should find that pid alive. Confirming it is - AX-active is what the runner probed; confirming it *owned the screen* is not what this fact claims. - On a physical device `proc_pidpath`-style resolution is deliberately NOT attempted. - -## Still to prove on this lane (PR1 risk list) - -- Whether `activeApplications` reports more than the session app while a foreign app is foreground - on real hardware (decides how often `otherActiveApplicationPid` is present at all). -- That screenshot stays lifecycle, #2438 in-place system-host serving still applies (Apple Pay sheet - from **Automation lab → Open Apple Pay**), and ADR 0005 `targetReset` on external relaunch is - untouched. - -## Not covered by the disclosure shipped in PR2 - -`press ` and `press @ref` answered from a live ref frame consume no capture, so they carry no -disclosure even though the runner may have re-activated the app to serve them. Treat a silent -interaction as "unknown", never as "no repair". Follow-up: callstack/agent-device#2694. - -## PR2 simulator lane — daemon disclosure (apex-2682-proto, iOS 26.2) - -Driven with `bin/agent-device.mjs` after `pnpm build && pnpm clean:daemon`, session `apex2682-pr3`, -off-app handoff via `xcrun simctl openurl https://example.com` before each command. Every -command below re-activated the session app, and each answered with `targetActivation` plus the -appended warning: - -| Command | Result | -| --- | --- | -| `snapshot -i` | warning first in `warnings`, ahead of the AX-backend warning; `targetActivation` carried | -| `snapshot -i --level digest` | `targetActivation` survives the digest projection | -| `find "Tab Bar" --first` | disclosed on the matched response | -| `press 'label="INFO"'` | disclosed on the interaction response | -| `wait 'label="INFO"' 8000` | disclosed on the satisfied response | - -`otherActiveApplicationPid` on the simulator was the Safari process, consistent across commands (pid 33878). - -### Proof the review asked for on this lane - -The false-attribution class is covered by unit tests rather than a device run, because the device -cannot show a cache hit on command: `packages/daemon/src/execution/__tests__/selector-capture-runtime.test.ts` -drives a session-snapshot cache hit and pins that the consumed tree earns no repair proof and -discloses nothing, `packages/daemon/src/core/__tests__/capture-disclosure-target-activation.test.ts` -pins that the response builder never borrows a repair the stored snapshot happens to carry, and -`packages/daemon/src/execution/interaction/internal/interaction-capture-disclosure.test.ts` pins that -a press which consumed no capture is not disclosed against an older tree. - -## Why the live activation lane is manual - -`test/integration/smoke-ios-target-activation.test.ts` is env-gated and deliberately absent from the -`ios.yml` / `replays-manual.yml` test lists. On GitHub-hosted simulators the runner reads its own -target as `.runningForeground` while a foreign app is provably on screen (uploaded artifact of -https://github.com/callstack/agent-device/actions/runs/35460535722/job/105943468492: rendered app -screenshot, Settings screenshot, 31 captures in ~450 ms each, no repair). The runner is then correct -to activate nothing, so a CI assertion on the disclosure measures `XCUIApplication.state` on that -host rather than this feature. Tracked as callstack/agent-device#2696. - -Manual evidence stands in for it, and both are reproducible: -- local simulator (`apex-2682-proto`), first capture after the handoff: the shared sentence in - `data.warnings`, printed under the snapshot node count and naming a non-foreground prior state and the - pid probe — `The session app was not foreground when this command - arrived (prior state runningBackground), so the runner activated it before answering (reason - stale_target). Any capture taken earlier in this session described the only app other than the - session app with an active accessibility session (pid 33878), not the session app. Re-capture now - that the session app answers, or drive the other app in its own session.` — and, on this head, the - typed fact beside it as `data.targetActivation`: - `{"reason":"stale_target","priorState":"runningBackground","otherActiveApplicationPid":33878}`. - PR1 alone carries the sentence only; the typed field is this PR's daemon seam. Either way the - following capture reports neither. -- the physical device sequence in this file, which the coordinator runs. diff --git a/.device-evidence/DECISION.md b/.device-evidence/DECISION.md deleted file mode 100644 index b3b834b0de..0000000000 --- a/.device-evidence/DECISION.md +++ /dev/null @@ -1,57 +0,0 @@ -# #2682 — target activation disclosure: settled policy - -Prototype branch `apex/2682-proto` (throwaway, not shipped). Lane: dedicated iPhone 17 Pro -simulator `apex-2682-proto` (`C2748D97-B92D-4B3B-9C65-AF86494AD904`), iOS 26.2, fixture -`com.callstack.agentdevicelab`. Physical `thymikee-iphone` untouched. - -## Policy - -**Ship (a): the runner keeps re-activating and discloses it. The (a) prior is CONFIRMED.** - -Reproduced on the simulator: an off-app handoff (`xcrun simctl openurl -https://example.com` — the LaunchServices path an in-app external link takes) put MobileSafari in -the foreground. `screenshot` returned Safari's `Example Domain` page, status bar reading -"◀ Agent Device Tes…". The next `snapshot -i` returned `@e1 [application] "Agent Device Tester" … -@e13 [button] "Home"` — the session app — and said nothing about why. The only record anywhere was -`AGENT_DEVICE_RUNNER_ACTIVATE … state=2 reason=bundle_changed` in `runner.log`. The agent's own two -observations contradicted each other silently. - -Latency is not the reason to prefer (a) — disclosure is free there. The state is already read before -`activate()` runs (`RunnerTests+Lifecycle.swift:205`, for the existing NSLog), so the emit sits -inside the branch that already pays `activate()` (measured 36–48 ms). The happy path never enters -that branch: five already-foreground `snapshot -i` runs measured 221–240 ms against 245–262 ms -before the change, and carry no field. (b) would break the screenshot→snapshot flow callers already -use and needs per-caller recovery; (a) preserves behavior and PR1 ships alone. - -## Foreground identity: ship `priorState` + `otherActiveApplicationPid`, not a foreground owner - -`XCAXClient_iOS` on this runtime answers `activeApplications` and `systemApplication` only; -`frontmostApplication` and `focusedApplication` do not exist there. Each element exposes -`processIdentifier` and nothing else — no `bundleIdentifier`, `executablePath`, `active`, or `state`. -The bundle id is reachable only by escalating: `proc_pidpath(pid)` (no iOS SDK header; symbol -declared by hand) → `NSBundle` → `com.apple.mobilesafari`, cross-checked against host `ps` for the -same pid. That is a new private surface with unknown physical-device behavior, and -`activeApplications` has no proven ordering to identify which element is foreground. So PR1 states a -pid, and only when exactly one foreign application is AX-active; it never guesses a bundle id. - -Adversarial review (PR1/PR2) caught the naming over-claiming what that probe proves: with no ordering -in `activeApplications`, the pid establishes only that exactly one other application held an active -accessibility session at that moment — a liveness claim, not a foreground owner. The field is therefore -`otherActiveApplicationPid` and the sentence says the same thing the value proves. `priorState` is -unaffected: it is `XCUIApplication.state` read before `activate()` ran, which is a real fact about the -session app. - -## Disclosure as it lands (same simulator, PR1 head) - -``` -The session app was not foreground when this command arrived (prior state runningBackground), so -the runner activated it before answering (reason stale_target). Any capture taken earlier in this -session described the only app other than the session app with an active accessibility session -(pid 33878), not the session app. Re-capture now that the session app answers, or drive the other -app in its own session. -``` - -pid 33878 resolved to `…/MobileSafari.app/MobileSafari` by host `ps`. The immediately following -`snapshot -i`, with the app already foreground, carries no disclosure — the fact belongs to the -command that paid for it. `test/integration/smoke-ios-target-activation.test.ts` runs the whole -sequence as an env-gated live lane and passes on this simulator. diff --git a/AGENTS.md b/AGENTS.md index 6e12cd3380..b830dd8b1e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,61 +1,62 @@ # AGENTS.md -`agent-device` is a CLI and daemon for automating devices and web targets. +`agent-device` is a CLI and daemon that automates devices and web targets. The daemon owns sessions. Registry declarations route commands to platform runtimes. ## Task routing -Read only the row for your current task. Follow further links only when they apply. +Read only the row for your current task. Follow other links only when they apply. | When the task involves | Read | | --- | --- | -| Domain vocabulary | `CONTEXT.md`, `docs/agents/domain.md` | +| Domain vocabulary | `CONTEXT.md` | | Architecture decisions | `docs/adr/README.md` | | Source changes, package boundaries, routing, diagnostics | `docs/agents/code-ownership.md` | -| Running checks or diagnosing a failed gate | `docs/agents/testing.md` | -| Writing or changing tests | `docs/agents/test-authoring.md` | -| Auditing or deleting tests | `docs/agents/test-audit.md` | +| Checks or a failed gate | `docs/agents/testing.md` | +| New or changed tests | `docs/agents/test-authoring.md` | +| Test audits or deletions | `docs/agents/test-audit.md` | | Mutation experiments or base comparisons | `docs/agents/test-experiments.md` | -| Changing gate selection, fuzzing, or mutation harnesses | `docs/agents/gates.md` | +| Gate selection, fuzzing, or mutation harnesses | `docs/agents/gates.md` | | Selector capture, polling, interaction fast paths, replay | `docs/agents/selector-capture.md` | -| Adding or changing a CLI flag | `docs/agents/cli-flags.md` | -| Opening or updating a PR | `docs/agents/pull-requests.md` | -| Reviewing a change | `docs/agents/review.md` | +| A new or changed CLI flag | `docs/agents/cli-flags.md` | +| A PR to open or update | `docs/agents/pull-requests.md` | +| A change to review | `docs/agents/review.md` | | Device-facing changes or manual device verification | `docs/agents/device-verification.md` | -| Writing issues or PRDs, and triage labels | `docs/agents/issue-tracker.md`, `docs/agents/triage-labels.md` | +| Issues, PRDs, or triage labels | `docs/agents/issue-tracker.md`, `docs/agents/triage-labels.md` | | Web backend setup or diagnostics | `docs/agents/web-backend.md` | -Device-facing changes affect live device behavior: dispatch, capture, interaction, app lifecycle, or native execution. -A daemon edit that changes none of these needs no device run. +A device-facing change affects live device behavior: dispatch, capture, interaction, app lifecycle, or native execution. +If a daemon edit changes none of these, a device run is not required. -Versioned CLI help owns command semantics. For app-driving startup, use the skill's routing card. +The versioned CLI help owns command semantics. To drive an app, start with the skill's routing card. ## Working rules -- Keep edits within the requested command family or module group. Inspect other platforms only when the task requires it. -- Fix recurring failures at the owning type, registry, or construction path. Do not reconstruct another owner's state. -- Use typed reasons and details for behavior. Never match error text. -- Add compatibility or fallback behavior only when explicitly requested or approved. Remove superseded paths during migrations. -- Encode invariants in names, types, and boundaries. Comments explain public APIs, tool directives, or constraints code cannot express. -- Resolve gate failures at the reported invariant. Do not suppress them or add an allowlist to get a pass. -- Use `@agent-device/selectors` for selector parsing and matching. See `docs/agents/code-ownership.md` for other shared owners. -- Keep skills as routing cards. Keep `CONTEXT.md` glossary-only. Update owning help and user docs when behavior changes. +- Keep edits in the requested command family or module group. Examine other platforms only when the task requires it. +- Fix a recurring failure at the owning type, registry, or construction path. Do not rebuild another owner's state. +- Use typed reasons and details for behavior. Do not match error text. +- Add compatibility or fallback behavior only on explicit request or approval. During a migration, remove superseded paths. +- Encode invariants in names, types, and boundaries. Write comments only for public APIs, tool directives, or constraints that code cannot show. +- Resolve a gate failure at the reported invariant. Do not suppress it. Do not add an allowlist to get a pass. +- Use `@agent-device/selectors` to parse and match selectors. For other shared owners, see `docs/agents/code-ownership.md`. +- Keep skills as routing cards. Keep `CONTEXT.md` as a glossary only. When behavior changes, update the owning help and user docs. ## Worktree setup -- Use `pnpm`. Use OXC for lint and format. Run `pnpm format` repository-wide. -- In a fresh worktree, run `pnpm install --frozen-lockfile && pnpm build` before testing. -- Preserve other worktrees, devices, sessions, dirty files, and experiments. -- Never copy `.env` into a worktree. For host credentials, source the main checkout's file by absolute path. -- Give parallel workers disjoint files and distinct devices. `pnpm check:affected --run` and `pnpm check` serialize automatically. -- The layering scan reads tracked files. Stage new modules before trusting its result. -- Move a path's Fallow baseline entry when renaming its file. Do not regenerate unrelated baselines. +- Use `pnpm`. Use OXC to lint and format. Run `pnpm format` on the full repository. +- In a new worktree, run `pnpm install --frozen-lockfile && pnpm build` before you test. +- Do not change other worktrees, devices, sessions, dirty files, or experiments. Do not use `git stash`; all worktrees share it. +- Do not copy `.env` into a worktree. For host credentials, source the main checkout's file by its absolute path. +- Give each parallel worker different files and a different device. `pnpm check:affected --run` and `pnpm check` run one at a time automatically. +- The layering scan reads only tracked files. Stage new modules before you trust its result. +- When you rename a file, move its Fallow baseline entry. Do not regenerate unrelated baselines. -The validation lock reclaims provably dead owners automatically. Unknown owners remain blocking. -If waiting times out, report the printed owner and lock state; do not remove it by hand. +The validation lock automatically reclaims owners that are provably dead. An unknown owner continues to block. +If the wait times out, report the printed owner and the lock state. Do not remove the lock manually. ## Agent docs -Write short instructions with one action per sentence. Put the condition before its action. -Use the same term for the same concept. Keep commands and domain terms exact. -Route specialized rules to their task document. Delete duplicate or enforced rules before adding reminders. +Write agent docs in ASD-STE100. Write one instruction in each sentence. +Put the condition before the instruction. Use one term for one concept. +Keep commands and domain terms exact. Put specialized rules and new shared constraints in the owning task document, not in agent memory. +Before you add a reminder, delete duplicate or enforced rules. diff --git a/CONTEXT.md b/CONTEXT.md index 45a63d0004..e397d03f5f 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -1,14 +1,13 @@ # Agent Device Domain Language -Canonical vocabulary for the automation domain. Use these names in code, tests, issues, and -architecture notes; implementation decisions and procedures belong in ADRs and task guidance. +Use these canonical names in code, tests, issues, and ADRs. ## Language ### Sessions, targets, and devices **Platform family**: -An internal ownership group for related automation platforms: Apple, Android, HarmonyOS, Vega, +An internal ownership group of related automation platforms: Apple, Android, HarmonyOS, Vega, Linux, web. **Platform leaf**: @@ -26,8 +25,8 @@ The platform-neutral boundary that reports runtime facts and binds an admitted d runtime owner. **Runtime owner**: -The one local platform module or provider runtime selected to execute -behavior for an ownership-qualified device. +The one local platform module or provider runtime that executes behavior for an +ownership-qualified device. **Request binding**: A request-lived attachment of cancellation, diagnostics, progress, and admitted context to a @@ -54,7 +53,7 @@ assets. The selected automation destination, such as mobile, TV, or desktop. **Session**: -Daemon-owned state for one selected target and its opened app or surface. +Daemon-owned state for one selected target and its open app or surface. **Device key**: A stable provider-scoped identity for device ownership and contention. @@ -121,8 +120,8 @@ _Avoid_: New or migrated command behavior One route an interaction command takes from a resolved target to device execution. **Coordinate-first resolved element activation**: -An Apple interaction that resolves a semantic element and activates its resolved center point -without a second lookup. +An Apple interaction that resolves a semantic element and activates its center point without +a second lookup. **Parent-owned touch point**: A point that keeps the selected parent's identity while avoiding independently interactive @@ -166,8 +165,7 @@ An optional post-action observation that waits for a quiet UI and diffs against tree. **Resolution disclosure**: -Bounded response evidence describing how an interaction target resolved, issuing no new actionable -refs. +Bounded response evidence of how an interaction target resolved; it issues no new actionable refs. ### Gestures and touch @@ -208,8 +206,8 @@ traversal depth kept apart from regular presented depth, and narrowing only wher prove it complete. **Regular presented-depth frontier**: -The acquisition boundary for an unscoped regular snapshot, measured against regular presented depth -after structural wrappers collapse. +The acquisition boundary for an unscoped regular snapshot, measured in regular presented depth after +structural wrappers collapse. **Snapshot eligibility**: Membership in a presented snapshot projection, independent of current hittability. @@ -220,7 +218,7 @@ backend: viewport and scroll-container clipping, ancestor projection, scroll hin depth. Platform differences enter as a fold policy, never as a backend exception. **Presented node**: -A wire-facing snapshot value produced at the presentation boundary. +A wire-facing snapshot value made at the presentation boundary. **Snapshot capture plan**: An ordered set of capture backends under one shared wall-clock budget. @@ -289,7 +287,22 @@ A provider runtime mapping a cloud-owned Appium or WebDriver session into agent- leases, runtime behavior, artifacts, and release. **Cloud artifact**: -Provider-hosted session output: video, automation logs, device logs, or dashboard links. +Provider-hosted session output: video, automation or device logs, or dashboard links. **Daemon artifact type**: An optional semantic category from the owner of a daemon-managed downloadable artifact. + +**Provider-backed integration scenario**: +A device-free test on the real daemon request path that fakes only external device and host tools. + +**Provider transcript**: +An exact record of external provider calls, to verify command translation. + +**Scenario transcript**: +A command-level flow of user-visible behavior through daemon commands. + +**In-process provider scenario harness**: +A runner that calls the daemon request handler with no HTTP listener. + +**HTTP contract test**: +A narrow test of JSON-RPC transport, authentication, and response finalization. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0fd0eacb6f..f794d0ca29 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -201,31 +201,21 @@ pnpm test:maestro-compat # example of a focused family suite pnpm exec vitest run path/to/file.test.ts ``` -Before pushing a normal code change, let the repository derive the required gates: +Before pushing, commit the change and validate that exact commit. The repository derives the +required gates from the diff: ```bash -pnpm check:affected --run +pnpm check:affected --run --head "$(git rev-parse HEAD)" ``` -The selector combines the committed diff with staged, unstaged, and untracked files. Unknown, -workflow, lockfile, and selector-owning changes fail open to the full local set. It reports -device/toolchain checks that remain GitHub-authoritative instead of trying to run them implicitly. - -For broad refactors or when explicitly requested, run the deterministic core aggregate: - -```bash -pnpm check -``` - -Serialized validation (`pnpm check` and `check:affected --run`) requires macOS or Linux. -On Windows, use the affected plan and direct focused checks. - -`pnpm check` takes the same host lock as `check:affected --run`, prints a receipt path, and writes -logs under `.tmp/validation`. It covers formatting, lint, typechecking, layering, dependency-graph parity, production -exports, MCP metadata, the distributable build, bundle ownership, Fallow, unit tests, and local smoke -tests. It is intentionally not a simulation of every CI job: coverage, provider integration, -history-backed compatibility, specialized toolchains, and live device/browser lanes remain separate. -GitHub CI is authoritative. +Only a passed receipt with `exactCommit: true` proves validation at that commit. Without `--head`, +the run also includes staged, unstaged, and untracked files; treat that result as local feedback. +Unknown, workflow, lockfile, and selector-owning changes fail open to the full local set. For broad +refactors or when explicitly requested, run `pnpm check`: formatting, lint, typechecking, layering, +production exports, MCP metadata, the build, Fallow, unit tests, and local smoke tests. Both commands +share one host lock and need macOS or Linux; on Windows, use the affected plan and direct focused +checks. GitHub CI stays authoritative for coverage, provider integration, specialized toolchains, +and live device lanes. Useful direct entry points: @@ -333,7 +323,8 @@ Read the release notes, inspect the affected-check plan, and treat a green updat candidate rather than an automatic merge: ```bash -pnpm check:affected --run +pnpm check:affected --verbose +pnpm check:affected --run --head "$(git rev-parse HEAD)" ``` ## Issues diff --git a/docs/agents/cli-flags.md b/docs/agents/cli-flags.md index 1b253fef28..b4efd82299 100644 --- a/docs/agents/cli-flags.md +++ b/docs/agents/cli-flags.md @@ -1,64 +1,57 @@ # Adding a CLI Flag -Thread a flag only through the layers that consume it: +Pass a flag only through layers that consume it: -1. `packages/contracts/src/cli-flags.ts`: add to `CliFlags`; add the definition to the matching - `packages/command-registry/src/flag-definitions-*.ts` owner and the relevant group in - `packages/command-registry/src/flag-groups.ts` - (for example `SNAPSHOT_FLAGS`). Then update the command family metadata/schema that exposes the - flag; find the owner with - `rg -n "|supportedFlags|allowedFlags" src/commands src/cli/parser`. For - schema-only CLI commands, the owner is `SCHEMA_ONLY_CLI_COMMAND_SCHEMAS` in - `src/commands/schema/command-overrides.ts`. Every flag declaration states `projectConfig` - (may be set from a project `agent-device.json`) and `recorded` (the session recorder - copies it into `SessionAction.flags`). Both are required, so a new declaration that - omits either does not compile — that, not an allowlist, is the completeness gate. Set - `projectConfig: true` only when repository control is safe; a new flag is otherwise - operator-only. Set `recorded: true` only when a `.ad` recording must carry the flag. -2. `src/commands/cli-grammar/*`: read the CLI flag into command input. -3. `src/commands/command-projection.ts` and command-family projection helpers: write the input into - the daemon request only if the flag affects daemon execution. -4. `src/commands/*-command-contracts.ts`: add to the command input schema only if the option should - be available through Node.js or MCP as structured input. An input key that names a credential, - an endpoint a credential is sent to, or operator infrastructure declares `operatorField(...)` - (`src/commands/command-input.ts`), which is what keeps the MCP and AI SDK tool schemas from - offering the model a parameter to write it into. One of the shared common keys declares the same - audience in its `src/commands/common-input-fields.ts` row instead. -5. `src/client/client-types.ts`: update the public typed client option only when the Node.js - interface exposes it. -6. `src/client/client-normalizers.ts`: update daemon flag normalization only when the request still - needs a public-to-internal translation. -7. `packages/daemon/src/core/context.ts` and `packages/daemon/src/core/dispatch-context.ts`: add the field only when it flows into - platform dispatch. -8. Handler/platform modules: thread the option only after the command surface, grammar, and - projection prove it belongs there. -9. `scripts/integration-progress-model.ts`: classify the flag (device-observable vs - intentionally-outside). The architecture-progress gate fails CI on unclassified public flags. -10. If the flag changes interaction semantics, revisit the affected cells in - `packages/contracts/src/interaction-guarantees.ts` (scope with `appliesTo` when the flag exists only on - some commands). +1. Add the flag to `CliFlags` in `packages/contracts/src/cli-flags.ts`. + Add its definition to the owner in `packages/command-registry/src/flag-definitions-*.ts`. + Add it to the applicable group in `packages/command-registry/src/flag-groups.ts`, for example `SNAPSHOT_FLAGS`. + Update the command family metadata/schema that exposes the flag. + To find the owner, run `rg -n "|supportedFlags|allowedFlags" src/commands src/cli/parser`. + For schema-only CLI commands, use `SCHEMA_ONLY_CLI_COMMAND_SCHEMAS` in `src/commands/schema/command-overrides.ts`. + Each flag declaration requires `projectConfig` and `recorded`. + `projectConfig` permits a project `agent-device.json` to set the flag. + `recorded` makes the session recorder copy it into `SessionAction.flags`. + Compilation enforces completeness, without an allowlist. A declaration without either field does not compile. + Only when repository control is safe, set `projectConfig: true`. Otherwise, a new flag is operator-only. + Only when a `.ad` recording must carry the flag, set `recorded: true`. +2. In `src/commands/cli-grammar/*`, read the CLI flag into command input. +3. If the flag affects daemon execution, write the input into the daemon request. + Use `src/commands/command-projection.ts` and the command-family projection helpers. +4. If Node.js or MCP should expose structured input, add a field to the command's field map. `defineFieldCommandMetadata` generates `inputSchema` and the audience. + For a credential, its destination endpoint, or operator infrastructure, declare the input key with `operatorField(...)` (`src/commands/command-input.ts`). + MCP and AI SDK tool schemas thus exclude parameters for the model to write that input. + For a shared common key, declare the same audience in its `src/commands/common-input-fields.ts` row instead. +5. If Node.js exposes the option, update the public typed client option in `src/client/client-types.ts`. +6. If the request still needs public-to-internal translation, update daemon flag normalization in `src/client/client-normalizers.ts`. +7. If the field enters platform dispatch, add it to `packages/daemon/src/core/context.ts` and `packages/daemon/src/core/dispatch-context.ts`. +8. Only after the command surface, grammar, and projection prove applicability, pass the option through handler/platform modules. +9. In `scripts/integration-progress-model.ts`, classify the flag as device-observable or intentionally-outside. + In CI, the architecture-progress gate fails on unclassified public flags. +10. If the flag changes interaction semantics, examine the affected cells in `packages/contracts/src/interaction-guarantees.ts` again. + If only some commands have the flag, use `appliesTo` to limit its scope. -Command-only flags (like `find --first`) that never reach the platform layer usually stop at -steps 1-3, plus step 9. +Command-only flags that never reach the platform layer usually stop at steps 1–3, plus step 9. +An example is `find --first`. ## Where CLI help and schema live -- Long help prose: `src/commands/schema/cli-help.ts`. Flag definitions: `src/commands/cli-grammar/`. -- Synopsis: `src/commands/schema/usage.ts` generates the `[label]` flag tail from `allowedFlags`, so a - new option reaches `--help` without any synopsis edit. Declare `usageFlags` on the command only - when its synopsis names fewer options: `[]` for a synopsis that is pure grammar (or writes its own - mutually-exclusive brackets), otherwise the subset it names. `Command flags:` always lists - everything in `allowedFlags`. Keep a cross-cutting opt-in out of every synopsis with - `usageHidden: true` on its flag definition. `src/commands/schema/usage.test.ts` fails a tail that names - an option the command does not accept, or one the hand-written grammar already wrote. -- A command whose first positional picks an action declares `flagsByAction` on its CLI schema, mapping - each action to the options it reads, and derives `allowedFlags` from that table. The parser then - refuses a typed option its action never reads, off the keys the caller provided: config, env, and - remote-config defaults fill `flags` without becoming a request, so `AGENT_DEVICE_FPS=30` never breaks - `record stop`. Keep an option every action reads out of the table — a row list is the only set the - parser refuses, and refusing `--json` would refuse the command. -- Command-specific usage/flag metadata lives with the command family metadata that owns the command. -- Parser/help *rendering* stays in `src/cli/parser/`; command schema metadata is derived from command - metadata, family declarations, and the schema-only merge path in - `src/commands/schema/command-overrides.ts`. Keep the two separate. -- Locating an owner: `rg -n "helpDescription|summary|supportedFlags|allowedFlags" src/commands src/cli/parser`. +- Long help prose is in `src/commands/schema/cli-help.ts`. + Flag definitions are in `packages/command-registry/src/flag-definitions-*.ts`. + CLI input readers are in `src/commands/cli-grammar/`. +- `src/commands/schema/usage.ts` generates the synopsis `[label]` flag tail from `allowedFlags`. + A new option reaches `--help` without a synopsis edit. + Only when the command synopsis names fewer options, declare `usageFlags`. + For pure grammar or grammar with its own mutually-exclusive brackets, use `[]`. + Otherwise, use the subset it names. `Command flags:` always lists all of `allowedFlags`. + For an opt-in across commands, set `usageHidden: true` on its flag definition. This excludes it from every synopsis. + `src/commands/schema/usage.test.ts` rejects tail options that the command does not accept or the handwritten grammar already includes. +- If the first positional selects an action, declare `flagsByAction` on the CLI schema. + Map each action to the options it reads. Derive `allowedFlags` from that table. + From caller-supplied keys, the parser rejects typed options that the action never reads. + Config, env, and remote-config defaults populate `flags` without becoming a request. Thus, `AGENT_DEVICE_FPS=30` never breaks `record stop`. + Exclude options that every action reads from the table. + The parser rejects only options in rows. If it rejected `--json`, it would reject the command. +- Keep command-specific usage/flag metadata with the command family metadata that owns the command. +- Keep argument parsing in `src/cli/parser/`. Keep help rendering in `src/commands/schema/`. + Command schema metadata derives from command metadata, family declarations, and the schema-only merge in `src/commands/schema/command-overrides.ts`. +- To find an owner, run `rg -n "helpDescription|summary|supportedFlags|allowedFlags" src/commands src/cli/parser`. diff --git a/docs/agents/code-ownership.md b/docs/agents/code-ownership.md index b321116bec..4d8bb4d306 100644 --- a/docs/agents/code-ownership.md +++ b/docs/agents/code-ownership.md @@ -1,7 +1,7 @@ # Code ownership Read this for shared contracts, package boundaries, request routing, or diagnostics. -Before editing a high-fan-in module, run `pnpm depgraph affected ` to scope callers and guarantees. +Before you edit a module with many callers, run `pnpm depgraph affected ` to identify callers and guarantees. ## Declarations @@ -10,33 +10,58 @@ Read the owner. Do not copy its policy into consumers. | Question | Owner | | --- | --- | | Command names | `packages/command-registry/src/catalog.ts` | -| Surface, runtime, batch, timeout traits | `packages/command-registry/src/registry.ts` | -| Daemon routes and request policy | `packages/daemon/src/core/daemon-command-registry.ts` | +| Surface, runtime, batch, timeout, daemon route and policy traits | `packages/command-registry/src/registry.ts` | | Interaction guarantees | `packages/contracts/src/interaction-guarantees.ts` | | Runtime-fact admission | `packages/daemon/src/core/runtime-admission.ts`, `src/platform-runtime-gateway.ts` | | Input fields and audiences | `src/commands/common-input-fields.ts`, `src/commands/input-audience.ts` | -Shared selectors belong in `@agent-device/selectors`. Request cancellation and progress belong in -`@agent-device/host-kit/request`. Per-call abort guards belong beside transports in -`src/daemon-client/daemon-client-transport.ts`. Cross-layer contracts belong in `packages/contracts/src`. -CLI flags belong in `src/commands/cli-grammar`; schema composition belongs in `src/commands/schema`. -Build interaction data through `buildInteractionResponseData`. Change cross-language rules through `contracts/fixtures/` golden tables. +Shared selectors belong in `@agent-device/selectors`. +Request cancellation and progress belong in `@agent-device/host-kit/request`. +Per-call abort guards belong beside transports in `src/daemon-client/daemon-client-transport.ts`. +Contracts shared across layers belong in `packages/contracts/src`. +For CLI flag owners, see `cli-flags.md`. +Schema composition belongs in `src/commands/schema`. +Build interaction data through `buildInteractionResponseData`. +Change cross-language rules through `contracts/fixtures/` golden tables. ## Modules and tests -Name modules for the domain question they answer. Colocate claims with their enforcement. -Use barrels only at package boundaries. Split files over 1,000 lines before adding behavior, except generated or fixture data. -Split a source module and its tests together. Do not extend legacy interaction or platform index test aggregations. +Name modules for the domain question they answer. Keep claims with the code that enforces them. +Use barrels only at package boundaries. +Before you add behavior, split files over 1,000 lines, except generated or fixture data. +Split a source module and its tests together. +Do not extend legacy interaction or platform index test aggregations. Keep tests beside their named module. Put repository artifact guards in `test/repo-guards/`. -Keep a single-package fixture local. Shared helpers belong in `@agent-device/test-kit`, a devDependency. -Production code never imports test-kit. Test-kit never depends on platforms, providers, the daemon, or root. +Shared helpers belong in `@agent-device/test-kit`, a devDependency. +Production code never imports test-kit. +Test-kit never depends on: + +- Platforms. +- Providers. +- The daemon. +- Root. + Do not export test support from production packages. ## Diagnostics Use `@agent-device/host-kit/diagnostics`. Keep request diagnostics in the session request log. Get session artifact paths from `packages/daemon-contracts/src/session-artifact-paths.ts`. -Keep app/device output in `app.log`; keep Apple runner and xcodebuild output in `runner.log`. -Normalize errors with `normalizeError`. Preserve `hint`, `diagnosticId`, `logPath`, and typed `details`. -For interactions taking at least five seconds, inspect daemon and runner recovery, retries, accessibility failures, and timeouts. -Use best-effort probes or caches only when the feature contract permits them. Keep their budgets below the required operation's budget. +Keep app/device output in `app.log`. +Keep Apple runner and xcodebuild output in `runner.log`. +Normalize errors with `normalizeError`. Preserve these fields: + +- `hint`. +- `diagnosticId`. +- `logPath`. +- Typed `details`. + +For interactions that take at least five seconds, examine: + +- Daemon and runner recovery. +- Retries. +- Accessibility failures. +- Timeouts. + +Only when the feature contract permits them, use best-effort probes or caches. +Keep their budgets below the required operation's budget. diff --git a/docs/agents/device-foldable.md b/docs/agents/device-foldable.md index 5052023b28..e8286081af 100644 --- a/docs/agents/device-foldable.md +++ b/docs/agents/device-foldable.md @@ -1,24 +1,28 @@ # Foldable Apple verification -Read ADR 0025 before changing capture behavior on a multi-panel device. The iOS 27.1 runtime ships -only with the Xcode that carries it, and `xcode-select` may point at an older one, so pin the -toolchain per command: -`DEVELOPER_DIR=/Contents/Developer xcrun devicectl device info displays --device `. +Before you change capture behavior on a multi-panel device, read ADR 0025. +To list the panels, run `xcrun devicectl device info displays --device `. -- Panels: that command lists each integrated panel with `backlightState`. Only the lit panel is - capturable — a capture of the dark panel exits 0 and writes an all-black PNG. -- Input routing: verify a fresh control after each pose change. A successful synthesis acknowledgement - does not prove a hit. Inspect the runner and simulator `testmanagerd`/BackBoard logs for display - identity and delivery; the resolved app window owns gesture coordinates and its target screen. -- Pose: `agent-device fold closed|half-open|open` sends private HID inside the simulator and reads - the hinge back through `devicectl device motion hinge-angle`. Device Hub and host Accessibility - permission are not required. Re-snapshot afterwards: refs and coordinates do not survive folds. - Verify locally with Device Hub stopped and the simulator booted through `simctl boot`; test all - three poses, active panel capture, and an app interaction. Duo coverage remains local until GHA - supports the runtime. To inspect the angle independently: - `DEVELOPER_DIR=/Contents/Developer xcrun devicectl device motion hinge-angle --device --session-timeout 1 --timeout 5`. -- Touch overlays export at the captured track size. The compositor checks its output (size, - non-black) against the raw capture; on failure `record stop` keeps the raw file and reports - `overlayWarning`. `record start --hide-touches` is only for the fastest raw capture. -- iOS 27 SDK apps need UIScene lifecycle to launch on iOS 27.0/27.1. -- The 27.1 beta accepts only the iPhone Duo device type; use another runtime for a non-foldable control. +- Panels: the command lists each integrated panel with `backlightState`. Only the lit panel permits capture. + A capture of the dark panel exits 0 and writes an all-black PNG. +- Input routing: after each pose change, verify a fresh control. + A successful synthesis acknowledgement does not prove a hit. + Examine the runner and simulator `testmanagerd`/BackBoard logs for display identity and delivery. + The resolved app window owns gesture coordinates and its target screen. +- Pose: `agent-device fold closed|half-open|open` sends private HID inside the simulator. + It reads the hinge back through `devicectl device motion hinge-angle`. + Device Hub and host Accessibility permission are not required. + After a fold, take another snapshot. Folds invalidate refs and coordinates. + Verify locally with Device Hub stopped and the simulator booted through `simctl boot`. + Test these items: + - All three poses. + - Capture of the active panel. + - An app interaction. + Until GHA supports the runtime, Duo coverage remains local. + devicectl 651.13.5 has no `device motion`. There, `fold` reports `fold-pose-dispatched-unverified`. + To examine the pose, compare the lit panel from the panel list. +- Touch overlays export at the captured track size. + The compositor checks its output size and non-black content against the raw capture. + If that check fails, `record stop` keeps the raw file. It reports `overlayWarning`. + `record start --hide-touches` is only for the fastest raw capture. +- To launch on iOS 27.0/27.1, iOS 27 SDK apps need UIScene lifecycle. diff --git a/docs/agents/device-verification.md b/docs/agents/device-verification.md index 75f6a4b96b..f1c59f4b87 100644 --- a/docs/agents/device-verification.md +++ b/docs/agents/device-verification.md @@ -4,59 +4,61 @@ Use this for Apple runner changes or manual device runs. For foldables, also rea ## Prepare -1. Build runtime changes with `pnpm build`. +1. To build runtime changes, run `pnpm build`. 2. For Android source verification, run `pnpm build:android`. 3. For Swift runner changes, run `pnpm build:xcuitest`. - Graceful shutdown can hand off an active runner; a new daemon can adopt the old binary. -4. Run `pnpm clean:daemon` after the builds. The daemon does not reload code. -5. Use distinct devices for parallel worktrees. Inspect worktree-scoped state with `pnpm daemon:state-dir`. + A graceful shutdown can hand off an active runner. A new daemon can adopt the old binary. +4. After the builds, run `pnpm clean:daemon`. The daemon does not reload code. +5. To examine the worktree-scoped state, run `pnpm daemon:state-dir`. 6. For `Agent Device Tester`, follow `examples/test-app/README.md` for the build, Metro, and expected app surface. - An installed `com.callstack.agentdevicelab` alone is insufficient. -7. For Android apps using Metro, run `adb reverse tcp: tcp:` before opening the app or URL. + An installed `com.callstack.agentdevicelab` alone is not sufficient. +7. For Android apps that use Metro, run `adb reverse tcp: tcp:` before you open the app or URL. ## Prove the path -Run the exact changed route. Record commands, device, source SHA, and observable results. -For Android, require `snapshot -i --json` to show `androidSnapshot.backend: android-helper` -and `helperVersion` matching `package.json`. UIAutomator fallback proves only that fallback. +Run the exact route that changed. Record the commands, device, source SHA, and observable results. +For Android, `snapshot -i --json` must show `androidSnapshot.backend: android-helper`. +Its `helperVersion` must match `package.json`. A UIAutomator fallback proves only that fallback. For Apple connection failures, start at `packages/platform-apple/src/runner/runner-startup-transport.ts`. -Retry policy belongs in `runner-error-classification.ts`; command typing belongs in `runner-contract.ts`. -Keep transport below session/client behavior. Keep xctestrun preparation outside request execution. -After signing, warm Node with `node -e 0` before measuring Gatekeeper-sensitive startup. +Retry policy belongs in `runner-error-classification.ts`. Command typing belongs in `runner-contract.ts`. +Keep the transport below session and client behavior. Keep xctestrun preparation outside request execution. +After signing, run `node -e 0` before you measure Gatekeeper-sensitive startup. +Before you measure on a physical iOS device, set Auto-Lock to Never. +A failed command can look like a fast success. Examine each result before you record its time. ## Ownership and cleanup -Use purpose-specific sessions. For extra isolation, use a separate `--state-dir` under `/private/tmp`. -Close every manually opened session, including failed attempts, with its original session, platform, UDID, and state directory. +Use one session for each purpose. For more isolation, use a separate `--state-dir` under `/private/tmp`. +Close each session that you opened manually, also failed attempts, with its original session, platform, UDID, and state directory. For `DEVICE_IN_USE`: -- "already in use by session X": follow this daemon's `close --session` hint. -- "owned by session X in workspace Y": inspect `device status`; follow `device release --stale` only for provably dead owners. - A live or uncertain owner blocks `open` unless the device rebooted after the claim. - With reboot evidence, `open` settles old resources, takes the claim, and reports release in `warnings`. - Do not hunt PIDs or steal the claim. +- "already in use by session X": follow the `close --session` hint of this daemon. +- "owned by session X in workspace Y": examine `device status`. Use `device release --stale` only for provably dead owners. + A live or uncertain owner blocks `open`, unless the device rebooted after the claim. + With reboot evidence, `open` settles old resources, takes the claim, and reports the release in `warnings`. + Do not search for PIDs. Do not steal the claim. -For stuck cleanup, run `agent-device device status --stale`. -Stop the owning daemon with `agent-device daemon stop --state-dir `; add `--clean` for retained runners. -Release dead owners with `agent-device device release --stale`. -If blocked, report the session, state directory, and status output. Never hide incomplete cleanup. -The physical runner handoff checklist lives in `docs/evidence/ios-physical-runner-handoff-2026-09-19.md`. +If cleanup is stuck, run `agent-device device status --stale`. +To stop the owning daemon, run `agent-device daemon stop --state-dir `. For retained runners, add `--clean`. +To release dead owners, run `agent-device device release --stale`. +If you are blocked, report the session, state directory, and status output. Always report incomplete cleanup. +The physical runner handoff checklist is in `docs/evidence/ios-physical-runner-handoff-2026-09-19.md`. ## Platform policy -Apple runner changes run `pnpm check:xctest-selection` and build the affected target. -Swift `#if` guards own XCTest classification. Do not maintain a second test-name list. -Use macOS host lanes for pure decisions and simulator lanes for iOS/XCTest semantics. +For Apple runner changes, run `pnpm check:xctest-selection` and build the affected target. +Swift `#if` guards own the XCTest classification. Do not keep a second list of test names. +Use macOS host lanes for pure decisions. Use simulator lanes for iOS and XCTest semantics. If system policy rejects an unsigned host bundle, rebuild with `CODE_SIGN_IDENTITY="Apple Development"` -or select an identity from `security find-identity -v -p codesigning`. -First runs need XCUITest automation permission. +or an identity from `security find-identity -v -p codesigning`. +The first run needs XCUITest automation permission. -Keep Apple target changes synchronized across the kernel model, runtime-fact admission, dispatch, +When you change Apple targets, synchronize the kernel model, runtime-fact admission, dispatch, discovery, and xctestrun preparation. Simulator-set scoping must not hide the host macOS target. HarmonyOS has no provisioned CI device or HDC. Mocked tests do not replace local hardware evidence. -Read live smoke harnesses before running them; their environment contracts live in their entry files under `test/integration/`. +Before you run a live smoke harness, read it. Its environment contract is in its entry file under `test/integration/`. -If a sandbox rejects localhost with `listen EPERM`, use host access when permitted. -Inspect the underlying error first. A generic startup or cleanup failure does not prove a sandbox cause. +If a sandbox rejects localhost with `listen EPERM`, use host access, if permitted. +First, examine the underlying error. A generic startup or cleanup failure does not prove a sandbox cause. diff --git a/docs/agents/domain.md b/docs/agents/domain.md deleted file mode 100644 index e9fac6e360..0000000000 --- a/docs/agents/domain.md +++ /dev/null @@ -1,22 +0,0 @@ -# Domain Docs - -Use `CONTEXT.md` for domain vocabulary and `docs/adr/README.md` to locate decisions relevant to the -change. A proposed ADR contradiction requires an explicit decision update. - -## Test-harness vocabulary - -**Provider-backed integration scenario**: -A device-free test through the real daemon request path that replaces only external device or host -tool execution. - -**Provider transcript**: -An exact record of external provider calls used to verify command translation. - -**Scenario transcript**: -A command-level integration flow describing user-visible behavior through daemon commands. - -**In-process provider scenario harness**: -An integration runner that invokes the daemon request handler without opening an HTTP listener. - -**HTTP contract test**: -A narrow test of JSON-RPC transport, authentication, and response finalization. diff --git a/docs/agents/gates.md b/docs/agents/gates.md index 801b4fb734..9c62da462b 100644 --- a/docs/agents/gates.md +++ b/docs/agents/gates.md @@ -1,23 +1,31 @@ # Maintain gates -Use this when changing gate selection or harnesses. For execution, read [validation](testing.md). +For execution, read [validation](testing.md). `CHECK_CATALOG` in `scripts/check-affected/checks.ts` owns checks and commands. CI declares ownership through the shared `run-gate` action with a literal gate id. -Update the catalog and executable model for a new check. Run `pnpm check:gate-manifest` to verify ownership and reachability. -Plant a violation to prove a new gate or selector detects it. Use [isolated experiments](test-experiments.md). +For a new check, update the catalog and executable model. +To verify ownership and reachability, run `pnpm check:gate-manifest`. +Insert a violation to prove a new gate or selector detects it. +Use [isolated experiments](test-experiments.md). For parser and geometry changes, extend shared fast-check arbitraries and properties. Use `PROPERTY_RUNS` budgets. Keep examples for past bugs or named decisions. -Parser fuzz targets live in `scripts/fuzz/targets.ts`. Generators carry planted invalid outcomes. -Promote discoveries with the harness command; do not hand-copy unshrunk inputs. +Parser fuzz targets are in `scripts/fuzz/targets.ts`. Generators include deliberately invalid outcomes. +Use the harness command to promote discoveries. Do not manually copy inputs that the harness has not shrunk. -Mutation declarations live in `scripts/mutation/modules.ts`; test ownership lives in `scripts/mutation/ownership.ts`. +Mutation declarations are in `scripts/mutation/modules.ts`. +Test ownership is in `scripts/mutation/ownership.ts`. `scripts/mutation/run.ts` uses `KERNEL_MODULES` weekly and `LANE_SOURCES` for PR canaries. Dispatch with `--with-dependencies` adds `LANE_DEPENDENCIES`. -Scores are advisory; incomplete sweeps fail. Read `pnpm mutation:run --help` before a pilot. -Concurrency torture uses production lock decisions. Run `pnpm test:concurrency-torture`; replay with the printed `TORTURE_SEED`. +Scores are advisory. Incomplete sweeps fail. +Before a pilot, read `pnpm mutation:run --help`. +Concurrency torture uses production lock decisions. +Run `pnpm test:concurrency-torture`. +To replay, use the printed `TORTURE_SEED`. -Do not commit run- or commit-stamped output under `scripts/`. Generate it or fetch evidence artifacts. +Do not commit output stamped with a run or commit under `scripts/`. +Generate it or fetch evidence artifacts. Versioned corpora, schemas, fixtures, and golden tables remain source inputs. -Plain `.mjs` packaging fixtures keep child-process use local when TypeScript helpers cannot load; prefer `execFile`. +If TypeScript helpers cannot load, plain `.mjs` packaging fixtures keep child-process use local. +Prefer `execFile`. diff --git a/docs/agents/issue-tracker.md b/docs/agents/issue-tracker.md index 01ab4b20af..59983c38c4 100644 --- a/docs/agents/issue-tracker.md +++ b/docs/agents/issue-tracker.md @@ -1,31 +1,34 @@ # Issue Tracker: GitHub -Issues and PRDs for this repo live as GitHub issues in `callstack/agent-device`. Use the `gh` CLI for issue operations. +Issues and PRDs for this repo are GitHub issues in `callstack/agent-device`. Use the `gh` CLI for issue operations. ## Triage scope -External PRs are not task requests in the issue-triage queue. Reading linked PRs to verify an issue's -dependencies is allowed; it does not expand the task into PR maintenance. +An external PR is not a task request in the issue-triage queue. To verify issue dependencies, you can read linked PRs. +Do not maintain those PRs. ## Conventions -Write issues as implementation contracts: purpose, required behavior, observable completion -conditions, and dependencies. Include an API, data shape, or CLI example for boundary changes. +Write each issue as an implementation contract: purpose, required behavior, observable completion +conditions, and dependencies. For a boundary change, include an API, data shape, or CLI example. -Sandboxed `gh` may lack host login credentials. Retry the focused command with host authentication -access when permitted before treating it as an authentication failure. Do not reconfigure -credentials; report the command and remaining blocker if the retry fails. +If sandboxed `gh` fails with an authentication error, run the same command with host access, if permitted. +Do not reconfigure credentials. If it fails again, report the command and blocker. For label meanings and state flow, see `docs/agents/triage-labels.md`. +## Current state + +When you change a decision or the scope of your issue, post a comment that starts with `Decision:`. +After a PR merges on your umbrella issue, post a `Current state:` comment: merged PRs, open PRs, and the next step. +Before you continue work, read all `Decision:` comments (newest wins per topic) and the newest `Current state:` comment. + ## Dependencies and sequencing -- Treat `Blocked by: ...` lines, linked prerequisite issues, and branch-base notes as part of the issue contract. -- Before scheduling or reviewing work, check blockers and decide whether the work should wait, stack on a prerequisite branch, or explicitly rescope. -- Do not mark an issue ready for implementation while prerequisite semantics are unresolved. - A dependent PR can be published with its base and blockers explicit; merge-readiness follows - `docs/agents/pull-requests.md`. -- When closing an umbrella issue, verify child issue states and the key implementation PRs instead of relying only on checked boxes. +- `Blocked by: ...` lines, linked prerequisite issues, and branch-base notes are part of the issue contract. +- Before you schedule or review work, examine the blockers. If one is open, decide: wait, stack on its branch, or rescope explicitly. +- If the prerequisite semantics are not resolved, do not mark the issue as ready for implementation. + You can publish a dependent PR with its base and blockers explicit. +- Before you close an umbrella issue, verify child issue states and key implementation PRs, not only checked boxes. -For authorized issue publication, use GitHub issues. A skill workflow alone is not authorization -to publish. +A skill workflow alone does not authorize you to publish an issue. diff --git a/docs/agents/pull-requests.md b/docs/agents/pull-requests.md index f63b7b609b..dd0cdeef9a 100644 --- a/docs/agents/pull-requests.md +++ b/docs/agents/pull-requests.md @@ -1,55 +1,91 @@ # Open or update a PR -A request to open or ship a PR authorizes branch creation, commits, push, and PR creation after validation. +A request to open or ship a PR authorizes these steps after validation: + +- Branch creation. +- Commits. +- Push. +- PR creation. + Apply authorization already given. Merging and releasing need separate authorization. ## Validate -Follow [validation](testing.md). Before pushing, run `pnpm check:affected --run --head `. -Record its receipt in the PR body. A serialized gate stage can validate the pushed head instead. -Record that result before reporting publication. -For unresolved failures, follow the blocked-validation table. An authorized incomplete draft can record a blocker; it is not a pass. +Follow [validation](testing.md). +Record the `check:affected --run --head` receipt in the PR body. A serialized gate stage can validate the pushed head instead. +Before you report publication, record that result. +For unresolved failures, follow the blocked-validation table. +An authorized incomplete draft can record a blocker. This is not a pass. Push once per review round. Each push restarts CI. Drafts skip device lanes. For device-facing changes, follow [device verification](device-verification.md). -Prove the changed route ran live. Fixtures and neighbouring routes do not replace this evidence. -Report missing devices or commands and incomplete session cleanup. +Prove the changed route ran live. Fixtures and adjacent routes do not replace this evidence. +Report missing devices or commands. -- Published: branch pushed, PR opened, validation status recorded at a named commit. +- Published: the branch is pushed, the PR is open, and validation status is recorded at a named commit. - Merge-ready: required checks pass at the actual head, with live evidence for device-facing changes. -Do not wait for CI merely to report publication. Do not claim skipped integration or coverage passed. +Do not wait for CI merely to report publication. +Do not claim that skipped integration or coverage passed. ## Self-review -Give a fresh reviewer the worktree, head, base ref, and this prompt. Do not supply an implementation summary. +Give a fresh reviewer these items: -> You did not write this change. Assume it has bugs and find them. Read AGENTS.md and docs/agents/review.md. -> Read the full diff against the merge-base and untracked files. Open callers and siblings for each hunk. -> Keep this checkout read-only. Run experiments only in a separate disposable checkout. -> Report only findings with a triggering scenario: file:line, severity, what breaks, and the fix. If none, say so. +- The worktree. +- The head. +- The base ref. +- This prompt. -Fix accepted findings and rerun affected checks. Record a reason for rejected findings. -If no independent reviewer is available, reread the full diff cold and report that limitation. -An independent reviewer is required at the growth thresholds in [review guidance](review.md). +Do not supply an implementation summary. + +> You did not write this change. Assume it has bugs. Find them. +> Read AGENTS.md and docs/agents/review.md. +> Read the full diff against the merge-base. Read untracked files. +> For each hunk, open callers and siblings. +> Keep this checkout read-only. Run experiments only in a separate disposable checkout. +> Report only findings with a triggering scenario. Include: +> +> - file:line. +> - Severity. +> - What breaks. +> - The fix. +> +> If there are no findings, say so. + +Fix accepted findings. Rerun affected checks. +For rejected findings, record a reason. +If no independent reviewer is available, review the full diff anew. Report that limitation. +At the growth thresholds in [review guidance](review.md), an independent reviewer is required. ## Scope and commits Keep gross diffs below 1,000 lines, or 2,500 for one coherent change in self-contained commits. -Measure with `git diff --stat origin/main...HEAD`. Stack only independently landable layers. +Measure with `git diff --stat origin/main...HEAD`. +Stack only layers that can merge independently. Rename-only `refactor(move)` PRs are exempt when `git diff -M90% --stat origin/main...HEAD` proves no material content change. -Use conventional commit prefixes, without agent tags. Put enforcement edits in a final `chore(gates): ` commit. +Use conventional commit prefixes, without agent tags. +Put enforcement edits in a final `chore(gates): ` commit. -Rebase for conflicts or relevant upstream changes. Disjoint changes alone do not require rebase. -After rebase, validate the new head. Keep old evidence attributed to its original SHA. +For conflicts or relevant upstream changes, rebase. Disjoint changes alone do not require rebase. +After a rebase, validate the new head. Keep old evidence attributed to its original SHA. ## PR body -Use ready-for-review by default. Use draft when requested or intentionally incomplete. +Use ready-for-review by default. When requested or intentionally incomplete, use draft. Keep the body below 250 words: -- `## Summary`: behavior and reason; touched-file count; scope expansion; linked issue. - Add 1–3 CLI/Node/MCP examples for public API changes. -- `## Validation`: tested SHA, command results, CI obligations, and unresolved risks. - Use remote-accessible evidence. For fixes, show base/head SHAs and the same command's results. +- `## Summary`: include: + - The behavior and reason. + - The touched-file count. + - Scope expansion. + - The linked issue. + For public API changes, add 1–3 CLI/Node/MCP examples. +- `## Validation`: include: + - The tested SHA. + - Command results. + - CI obligations. + - Unresolved risks. + Use evidence accessible remotely. + For fixes, show base/head SHAs and the same command's results. For docs-only changes, explain why runtime tests do not apply. diff --git a/docs/agents/review.md b/docs/agents/review.md index 3a7e8e4a59..cf38097bd9 100644 --- a/docs/agents/review.md +++ b/docs/agents/review.md @@ -1,28 +1,50 @@ # Review a change -Keep reviewed source read-only. Run checks in an isolated checkout if they can modify it. Use [isolated experiments](test-experiments.md) for mutations or base comparisons. +Keep reviewed source read-only. +If checks can modify it, run them in an isolated checkout. +For mutations or base comparisons, use [isolated experiments](test-experiments.md). ## Reviewing -- Read linked dependencies and relevant ADRs. Resolve prerequisite conflicts first. Update or supersede conflicting decisions explicitly. -- Trace routing and surface changes through daemon and backend. Helper tests bypassing the router do not prove dispatch. - Check affected CLI, Node, daemon, MCP, help, and docs surfaces. -- Trace typed errors from producers through normalization, transport, and sibling consumers. Do not replace lost reason codes with text matching. -- Classify interaction paths in `packages/contracts/src/interaction-guarantees.ts`. Verify each claim against native code. - Error-path delegation does not prove success-path parity. Prove both paths reachable before comparing performance. +- Read linked dependencies. + For each relevant ADR, read its Status and "Rules at a glance" first. + If it has no "Rules at a glance", read its Decision section. + Read other sections only for the rules that the change touches. + Resolve prerequisite conflicts first. + Update or supersede conflicting decisions explicitly. +- Trace routing and surface changes through daemon and backend. + Helper tests that bypass the router do not prove dispatch. + Check the affected surfaces: CLI, Node, daemon, MCP, help, and docs. +- Trace typed errors from producers through normalization, transport, and sibling consumers. + Do not replace lost reason codes with text matching. +- Classify interaction paths in `packages/contracts/src/interaction-guarantees.ts`. + Verify each claim against native code. + Error-path delegation does not prove success-path parity. + Before you compare performance, prove both paths are reachable. - A platform-family claim covers every backend, including Appium and ADB, or names the covered backends. -- Keep snapshot-wide metadata at response level. Append warnings through the shared builder; never replace prior warnings. +- Keep snapshot-wide metadata at response level. + Append warnings through the shared builder. Never replace prior warnings. Check compact defaults, bounded arrays, and artifact paths for large evidence. -- Before retaining compatibility, run `git tag --contains `. Unreleased shapes have no external compatibility obligation. -- Require each new test to detect a credible production failure. Test doubles must fail at the same seam as production. -- Find dead code, redundant guards, casts, narrating comments, and competing owners. Fix the owning rule, not only named sites. +- Before you retain compatibility, run `git tag --contains `. + Unreleased shapes have no external compatibility obligation. +- Require each new test to detect a credible production failure. + Test doubles must fail at the same seam as production. +- Find dead code, redundant guards, casts, comments that repeat the code, and competing owners. + Fix the owning rule, not only named sites. +- For a bulk rewrite of prose (docs, ADRs, help), get a review from another model provider. + The reviewer compares each changed sentence with the base. A self-review is not evidence. -Use CI Size evidence. At about 700 net production lines or 3 kB npm growth, require an independent reviewer to assess a smaller design. -Also assess any new abstraction, dependency, seam, or ownership rule. Weigh gross churn for moves. Explain justified growth. +Use CI Size evidence. +At about 700 net production lines or 3 kB npm growth, require an independent reviewer to assess a smaller design. +Also assess each new abstraction, dependency, seam, and ownership rule. + +For moves, weigh gross churn. Explain justified growth. ## Report findings -Report only triggering scenarios: `file:line`, severity, what breaks, and the fix. -If none, say so. Reject incorrect findings with evidence. +Report only triggering scenarios. Include `file:line`, severity, what breaks, and the fix. + +If there are no findings, say so. Reject incorrect findings with evidence. Prove claims such as "one reader", "pure move", or "no shim left" with search or `git diff -M90%`. -Encode repeated findings as types, structure, checks, or tests. Remove prose reminders once enforced. +Encode repeated findings as types, structure, checks, or tests. +Once enforced, remove prose reminders of the rule. diff --git a/docs/agents/selector-capture.md b/docs/agents/selector-capture.md index c82a958f09..8031134621 100644 --- a/docs/agents/selector-capture.md +++ b/docs/agents/selector-capture.md @@ -1,47 +1,62 @@ # Selector Capture Reliability -Read this before changing selector capture, polling, snapshot caching, or interaction fast paths. -These are cross-route behavior requirements; their rationale and owning decisions live in ADRs -0002, 0004, 0005, 0011, 0014, and 0015. - -- Direct iOS selectors are a narrow fast path for simple one-term selectors and are disabled while - post-gesture stabilization is pending. A structured selector miss may return to snapshot - resolution; ambiguity and other runner failures remain failures. -- Regular selector reads are capture-backed. `@ref` resolves against its authorized ref frame, - while `get`, `is`, `find`, and `wait` selectors capture through the backend. Polling bypasses the - snapshot cache, as do active freshness recovery and stabilization. The cache serves a stored tree - only while it is the newest observation: a side-effect seam or a native read such as `wait - text`'s owner text reading retires it. -- Sparse capture verdicts are observable failures and never replace the session snapshot. Only a - user-facing snapshot may publish a fallback screenshot; internal polling must not create one - artifact per attempt. -- Apple accessibility failures are not proof of an empty UI. Recovery plans may serve regular - snapshots, while raw and strict paths preserve the failure. Fatal runner evidence invalidates the - cached target. -- A recorded XCTest failure after a tap is an ambiguous outcome. Only a usable same-presentation - capture with a changed accessibility digest may corroborate success; missing, sparse, mismatched, - or unchanged evidence remains failure. -- Android helper reuse is not snapshot-result caching. Freshness is short-lived, action-triggered, - and learned only from route-safe complete observations. -- Post-gesture stabilization runs first and Android freshness recovery composes afterward when - required. Gesture-like mutations mark stabilization and disable direct iOS selector shortcuts - while it is pending. -- Session snapshot writes go through the shared snapshot mutation boundary. Sparse observations and - empty ref-scoped projections do not overwrite stored evidence. -- Maestro matching remains snapshot- and policy-owned. Coordinate dispatch uses fresh geometry; an - exact same-generation iOS target may use atomic selector activation, and a structured live miss - returns to fresh Maestro resolution. Never add fuzzy matching, synthetic geometry, hierarchy - heuristics, or error-message classification that changes authored selector meaning. - -When changing one of these rules, update the relevant ADR, guarantee-matrix cell, contract scenario, -and TypeScript/Swift parity table rather than describing a new exception here. +Before changes to selector capture, polling, snapshot caching, or interaction fast paths, read this. +These requirements apply across routes. +ADRs 0002, 0004, 0005, 0011, 0014, and 0015 own their rationale and decisions. + +- Direct iOS selectors are a narrow fast path for simple one-term selectors. + While post-gesture stabilization is pending, they are disabled. + A structured selector miss may return to snapshot resolution. + Ambiguity and other runner failures remain failures. +- Regular selector reads use capture. + `@ref` resolves against its authorized ref frame. + Selectors for `get`, `is`, `find`, and `wait` capture through the backend. + Polling, active freshness recovery, and stabilization bypass the snapshot cache. + The cache serves a stored tree only while it is the newest observation. + A side-effect seam or a native read retires it. + An example is `wait text`'s owner text read. +- Sparse capture verdicts are observable failures. They never replace the session snapshot. + Only a user-facing snapshot may publish a fallback screenshot. + Internal polling must not create one artifact per attempt. +- Apple accessibility failures do not prove an empty UI. + Recovery plans may serve regular snapshots. Raw and strict paths preserve the failure. + Fatal runner evidence invalidates the cached target. +- After a tap, a recorded XCTest failure is an ambiguous outcome. + Only a usable same-presentation capture with a changed accessibility digest may corroborate success. + Missing, sparse, mismatched, or unchanged evidence remains failure. +- Android helper reuse is not snapshot-result caching. + Freshness is short-lived and action-triggered. + Only route-safe complete observations establish freshness. +- Post-gesture stabilization runs first. + When required, Android freshness recovery composes afterward. + Gesture-like mutations mark stabilization. + While stabilization is pending, these mutations disable direct iOS selector shortcuts. +- Write session snapshots through the shared snapshot mutation boundary. + Sparse observations and empty ref-scoped projections do not overwrite stored evidence. +- Maestro matching remains snapshot- and policy-owned. + Coordinate dispatch uses fresh geometry. + An exact same-generation iOS target may use atomic selector activation. + A structured live miss returns to fresh Maestro resolution. + Never add fuzzy matching, synthetic geometry, hierarchy heuristics, or error-message classification that changes authored selector meaning. + +For changes to these rules, update: + +- The relevant ADR. +- The guarantee-matrix cell. +- The contract scenario. +- The TypeScript/Swift parity table. + +Do not describe a new exception here instead. ## Replay and coordinates -Element interactions support selector and `@ref`, then record a selector chain after resolution. -Replay failures may re-resolve that chain only to produce a ranked divergence suggestion; ADR 0012 -retired silent script rewriting. +Element interactions support selector and `@ref`. +After resolution, they record a selector chain. +Replay failures may re-resolve that chain only to produce a ranked divergence suggestion. +ADR 0012 retired silent script rewriting. -Selector keys stay centralized in `@agent-device/selectors`; `is` predicates use -`evaluateIsPredicate`. On macOS, snapshot rectangles are absolute in window space and point actions -translate through the interaction root frame. Prefer selector or ref tests over raw coordinates. +Keep selector keys centralized in `@agent-device/selectors`. +The `is` predicates use `evaluateIsPredicate`. +On macOS, snapshot rectangles are absolute in window space. +On macOS, point actions translate through the interaction root frame. +Prefer selector or ref tests over raw coordinates. diff --git a/docs/agents/test-audit.md b/docs/agents/test-audit.md index 325268c8c5..7cb7152ad4 100644 --- a/docs/agents/test-audit.md +++ b/docs/agents/test-audit.md @@ -1,35 +1,53 @@ # Audit tests -Use this to audit or delete tests. Optimize for confidence, not deletion counts. -Use [test authoring](test-authoring.md) for test quality and [validation](testing.md) for checks. +Optimize for confidence, not deletion counts. +For test quality, use [test authoring](test-authoring.md). +For checks, use [validation](testing.md). ## Retain independent contracts Do not delete completeness guards for registries, gates, layering, or DI seams. -Retain golden tables, cross-language inspection, `@ts-expect-error` pins, identity checks, -exact-array checks that detect duplicates, and vocabulary checks surviving a production/fixture co-update. -A slow or implementation-like test is not redundant without measured evidence. +Retain these contracts: + +- Golden tables. +- Cross-language inspection. +- `@ts-expect-error` pins. +- Identity checks. +- Exact-array checks that detect duplicates. +- Vocabulary checks that survive a simultaneous production/fixture update. + +Without measured evidence, slow or implementation-like tests are not redundant. ## Candidate evidence -Before editing, record: +Before you edit, record: -- Location, name, and detectable failure. +- The location, name, and detectable failure. - Production callers and the stronger remaining test's assertions, with locations. - Candidate and sibling skip guards, including workflow environment flags. -- Actual runner: Vitest project, Node script, XCTest lane, or gate id. Only unreachable tests are never-run findings. -- What deletion unlocks and its risk. +- The actual runner: Vitest project, Node script, XCTest lane, or gate id. +- What deletion permits and its risk. -Use [isolated experiments](test-experiments.md) for mutations. -For a deletion, prove the retained test detects the mutant after removing the candidate. +Only unreachable tests are never-run findings. +For mutations, use [isolated experiments](test-experiments.md). +For a deletion, remove the candidate. Prove the retained test still detects the mutant. Report baseline and mutant counts. Source inspection alone does not prove detection. ## Delegated audits -Assign disjoint read scopes to read-only reviewers. Do not assign edit ownership to a reviewer. -Each brief names the reviewed SHA and requires production-owner inspection, skip/runner verification, -retention criteria, and concrete failure scenarios. Experiments use separate disposable checkouts. -Subsystem review, duplicate-pattern search, and runner inventory are separate scopes when needed. +Assign disjoint read scopes to read-only reviewers. +Do not assign edit ownership to a reviewer. +Each brief names the reviewed SHA. Each brief requires: + +- Inspection of the production owner. +- Skip/runner verification. +- Retention criteria. +- Concrete failure scenarios. + +Experiments use separate disposable checkouts. +When needed, use separate scopes for subsystem review, duplicate-pattern search, and runner inventory. -Land one owner-boundary batch per PR. Remove unused test-only seams with the tests. -Report retained false positives. Rediscover after a relevant rebase. +Put one batch for one owner boundary in each PR. +Remove unused test-only seams with the tests. +Report retained false positives. +After a relevant rebase, repeat discovery. diff --git a/docs/agents/test-authoring.md b/docs/agents/test-authoring.md index ce6776c2b1..19800f795f 100644 --- a/docs/agents/test-authoring.md +++ b/docs/agents/test-authoring.md @@ -1,47 +1,68 @@ # Write tests -Use this when adding or changing tests. Read [validation](testing.md) for checks and [test audits](test-audit.md) for deletions. +For checks, read [validation](testing.md). +For deletions, read [test audits](test-audit.md). ## Choose the boundary -Before adding a test, name: +Before you add a test, name: 1. The observable behavior, invariant, or independent contract. 2. The production edit that would break it. -3. Why existing coverage does not catch that edit. +3. Why existing coverage does not detect that edit. -Test through the strongest production boundary. Do not add test-only production exports or dependency injection. -A regression test must fail without the fix for the intended reason. Use [isolated experiments](test-experiments.md). -Pure moves retain their tests; verify discovery at the new path. -Pair classifier positives with the closest negative. Prove identical error text without the typed reason cannot trigger recovery. +Test through the strongest production boundary. +Do not add test-only production exports or dependency injection. +Without the fix, a regression test must fail for the intended reason. +Use [isolated experiments](test-experiments.md). +Pure moves retain their tests. Verify discovery at the new path. +Pair classifier positives with the closest negative. +Prove identical error text without the typed reason cannot trigger recovery. For callback canaries, assert semantic success, not lifecycle completion. Automatic replay coverage requires a PR or scheduled lane that runs the scenario at the actual head. -Reject assertion-free probes, self-comparisons, expectations computed by the subject, duplicate cases, and mocks asserting their own inputs. -Reject tests whose names promise more than their inputs exercise. Type expressions erased at runtime do not prove runtime behavior. -Retain compile-time `@ts-expect-error` pins; an unused directive fails typechecking. +Reject these tests: + +- Assertion-free probes. +- Self-comparisons. +- Expectations computed by the subject. +- Duplicate cases. +- Mocks that assert their own inputs. + +Reject tests whose names promise more than their inputs exercise. +Type expressions erased at runtime do not prove runtime behavior. +Retain compile-time `@ts-expect-error` pins. An unused directive fails typechecking. ## Fixtures and seams -Look in `@agent-device/test-kit` and `packages/daemon/src/__tests__/support/` before adding helpers. -Keep single-package fixtures beside their module. Packages that test-kit depends on keep helpers local. -Do not import test-kit in a `vi.mock` factory for a module its entry loads. Spy on that seam in place. -Use `mkdtempForTest` or `mkdtempForTestSync`. Global setup removes the run directory after workers exit. -Mock the seam the subject consumes. Daemon handlers receive fake runtime facts and facets, not generic dispatch mocks. -When migrating an ADR 0019 command, move its tests to the runtime seam. - -Vitest workers signal only themselves and direct children. Mock process-signal helpers for fabricated PIDs. -Do not disable isolation, switch the fork pool, or add unit retries to hide failures. -Use existing clocks or tool seams instead of waiting production time. -Slow-test pins only shrink; a new pin needs measured justification. -Split oversized test families with their source. Do not extend test files over 1,000 lines. +Before you add helpers, examine `@agent-device/test-kit` and `packages/daemon/src/__tests__/support/`. +Keep single-package fixtures beside their module. +Packages that test-kit depends on keep helpers local. +Do not import test-kit in a `vi.mock` factory for a module its entry loads. +Spy on that seam in place. +Use `mkdtempForTest` or `mkdtempForTestSync`. +After workers exit, global setup removes the run directory. +Mock the seam the subject consumes. +Daemon handlers receive fake runtime facts and facets, not generic dispatch mocks. +When you migrate an ADR 0019 command, move its tests to the runtime seam. + +Vitest workers signal only themselves and direct children. +For fabricated PIDs, mock process-signal helpers. +Do not disable isolation to hide failures. +Do not switch the fork pool to hide failures. +Do not add unit retries to hide failures. +Use existing clocks or tool seams instead of production-time waits. +Slow-test pins only shrink. A new pin needs measured justification. +Do not extend test files over 1,000 lines. Split a test file only with its source module. Import the subject module, not a platform barrel. ## Run a focused test Use `pnpm exec vitest run --project unit-core `. For `packages/platform-apple/src/runner/**`, use `--project apple-runner`. -Local runs default to four workers. Set `AGENT_DEVICE_VITEST_MAX_WORKERS=` for a different host share; CI ignores it. -Scripts tests are outside root TypeScript includes. Check new helpers with their runner and review their types. +Local runs default to four workers. +For a different host share, set `AGENT_DEVICE_VITEST_MAX_WORKERS=`. CI ignores it. +Scripts tests are outside root TypeScript includes. +Check new helpers with their runner. Review their types. -Wire changes emitting `platform` or `appleOs` need provider integration and coverage evidence. +Wire changes that emit `platform` or `appleOs` need provider integration and coverage evidence. diff --git a/docs/agents/test-experiments.md b/docs/agents/test-experiments.md index 65e8f3df37..0cf7e0f95c 100644 --- a/docs/agents/test-experiments.md +++ b/docs/agents/test-experiments.md @@ -1,11 +1,11 @@ # Isolated test experiments -Use this for manual mutations, regression proof, or base comparisons. Scheduled mutation sweeps follow [gate maintenance](gates.md). -Keep the reviewed and implementation checkouts unchanged. Reviewers have read scopes, not edit ownership. +Use this for manual mutations, regression proof, or base comparisons. For scheduled mutation sweeps, follow [gate maintenance](gates.md). +Do not change the reviewed or implementation checkout. ## Prepare -1. Record the reviewed SHA, `git status --porcelain=v1`, and any existing diff before the experiment. +1. Before the experiment, record the reviewed SHA, `git status --porcelain=v1`, and any existing diff. 2. Create a disposable checkout outside app-managed worktrees: ```sh @@ -15,30 +15,29 @@ git worktree add --detach "$experiment_dir" "$reviewed_sha" ``` 3. In that checkout, run `pnpm install --frozen-lockfile && pnpm build`. - Do not reuse another checkout's `node_modules` or copy `.env`. -4. For base comparison, create it at the named base SHA instead. Record that SHA before running. - Use the same command, environment, and worker limit. Run sequentially without competing test processes. -5. If the change is uncommitted, apply only its relevant patch and required untracked files to the disposable checkout. + Do not use the `node_modules` of a different checkout. +4. For a base comparison, create the checkout at the named base SHA. Record that SHA before you run. + Use the same command, environment, and worker limit. Run in sequence, with no competing test processes. +5. If the change is uncommitted, apply only its patch and the necessary untracked files to the disposable checkout. Record the patch input. It is not exact-commit evidence. ## Prove detection -- Regression: run the new test against code without the fix. It must fail for the claimed bug. - Run it with the fix. It must pass. If it passes without the fix, improve the test. -- Mutation: run the unmodified focused suite. It must pass. Plant the production edit representing the claimed bug. - Run the same suite. It must fail for that bug. Record counts, for example `8 passed` then `1 failed, 7 passed`. -- Test deletion: perform the mutation after removing the candidate. Prove the retained test detects it. -- Structural gate or selection rule: plant a violation or failing selected test. Verify the intended gate reports it. -- Unchanged-base failure: do not copy head changes into the base checkout. Preserve both logs and SHAs. +- Regression: run the new test on the code without the fix. It must fail for the claimed bug. + Run it with the fix. It must pass. If it passes without the fix, make the test better. +- Mutation: run the unmodified focused suite. It must pass. Plant the production edit that represents the claimed bug. + Run the same suite. It must fail for that bug. Record the counts, for example `8 passed`, then `1 failed, 7 passed`. +- Test deletion: remove the candidate, then do the mutation. Prove that a retained test detects it. +- Structural gate or selection rule: plant a violation or a failing selected test. Verify that the intended gate reports it. +- Unchanged-base failure: do not copy head changes into the base checkout. Keep both logs and SHAs. A setup failure is not evidence. Measure detection. -Use distinct checkouts and devices for parallel experiments. ## Finish 1. Remove only the disposable worktree you created: `git worktree remove --force "$experiment_dir"`. -2. Verify the reviewed SHA, status, and existing diff still match the recorded originals. -3. Report before/after counts, commands, SHAs or patch inputs, failure reason, and cleanup status. +2. Verify that the reviewed SHA, status, and existing diff match the recorded originals. +3. Report before and after counts, commands, SHAs or patch inputs, failure reason, and cleanup status. -Stop and report if the reviewed checkout changes, setup fails, or detection remains unproven. -Never restore files in the reviewed checkout to hide an experiment's edits. +If the reviewed checkout changes, setup fails, or detection is not proven, stop and report. +Do not hide that change by restoring files. diff --git a/docs/agents/testing.md b/docs/agents/testing.md index 6f5ae58238..4f85d704a0 100644 --- a/docs/agents/testing.md +++ b/docs/agents/testing.md @@ -4,52 +4,87 @@ For mutations or base comparisons, use [isolated experiments](test-experiments.m ## Run checks -1. While editing, run the focused test or `pnpm check:quick`. -2. Before pushing, commit the change. Run `pnpm check:affected --run --head `. +1. During edits, run the focused test or `pnpm check:quick`. +2. Before you push, commit the change. + Run `pnpm check:affected --run --head `. 3. For a broad refactor or an explicitly requested full gate, run `pnpm check`. -Serialized validation requires macOS or Linux. On Windows, use plan mode or focused checks. -Both broad commands acquire the same host lock. Wait for the owner to finish, or cancel your waiter with Ctrl-C. -Do not add lock wrappers or delete lock files. A wait ends after two hours. Nested validation fails immediately. +Serialized validation requires macOS or Linux. +On Windows, use plan mode or focused checks. +Both broad commands acquire the same host lock. +Wait for the owner to finish, or cancel your waiter with Ctrl-C. +Do not add lock wrappers. Do not delete lock files. +A wait ends after two hours. Nested validation fails immediately. Focused tests do not lock. Avoid subprocess-heavy tests beside a broad gate. -The runner prints progress and a receipt path with refs, checkout state, command logs, exits, skips, and Vitest results. +The runner prints progress and a receipt path. The receipt contains: + +- Refs. +- Checkout state. +- Command logs. +- Exits. +- Skips. +- Vitest results. + Execution stops at the first failure. Unexecuted checks remain `not-run`. - `pnpm check:affected`: compact plan. - `pnpm check:affected --verbose`: plan with selection reasons. - `pnpm --silent check:affected --json`: JSON plan. - `pnpm --silent check:affected --run --json`: JSON execution receipt on stdout; progress on stderr. -- `pnpm check:affected --run`: include staged, unstaged, and untracked changes for local feedback. +- `pnpm check:affected --run`: includes staged, unstaged, and untracked changes for local feedback. Only a passed receipt with `exactCommit: true` proves validation at the named commit. A dirty-tree pass is local feedback. Detected checkout changes invalidate the run. -Initial/final checks detect persistent changes; they do not sandbox the checkout. Stop all edits during validation. +Initial/final checks detect persistent changes. They do not sandbox the checkout. +During validation, stop all edits. ## Interpret selection -A package manifest or TypeScript config can affect every consumer. A full selection for these changes is intentional. -`FAIL-OPEN` means the selector could not safely narrow the set. Read `failOpenReasons` in plan JSON or `plan.failOpenReasons` in a receipt. +A package manifest or TypeScript config can affect every consumer. +A full selection for these changes is intentional. +`FAIL-OPEN` means the selector could not safely narrow the set. +Read `failOpenReasons` in plan JSON or `plan.failOpenReasons` in a receipt. Do not reduce selection to get a pass. -Read required-check status on the PR. The receipt lists skipped checks and owners in `results[].reason`. -GitHub owns full coverage, native builds, device lanes, provider integration, and history-backed compatibility. -Related tests do not replace those obligations. Check the Integration Tests and Coverage jobs at the actual PR head. -For docs-only changes, run the selected documentation checks. Do not add tests asserting prose. +Read required-check status on the PR. +The receipt lists skipped checks and owners in `results[].reason`. +GitHub owns: + +- Full coverage. +- Native builds. +- Device lanes. +- Provider integration. +- Compatibility that uses history. + +Related tests do not replace those obligations. +Check the Integration Tests and Coverage jobs at the actual PR head. +For docs-only changes, run the selected documentation checks. +Do not add tests that assert prose. ## A gate failed -Keep failure attribution separate from validation status. Never call a failing gate passed. +Keep failure attribution separate from validation status. +Never call a failing gate passed. | Evidence | Next action | Outcome | | --- | --- | --- | -| New failure caused by this change | Fix its owner. Rerun after the fix. | Validation failed until a new run passes. | -| Suspected host contention or infrastructure fault | Save the log. Run the same failing command on unchanged base in an isolated checkout, without competing tests. | Cause unproven until compared. A solo pass does not prove contention. | +| New failure caused by this change | Fix its owner. After the fix, rerun. | Validation failed until a new run passes. | +| Suspected host contention or infrastructure fault | Save the log. In an isolated checkout, run the same failing command on unchanged base, without competing tests. | Until compared, the cause is unproven. A solo pass does not prove contention. | | Same failure reproduced on unchanged base | Record SHAs, command, logs, and matching failure. Assign the blocker to a named person or linked issue. | Base-blocked. If publication is authorized, publish a draft with the blocker and remaining checks. Do not mark merge-ready. | | Base comparison unavailable or inconclusive | Record the missing tool/device/evidence and next action. | Blocked, cause unproven. An authorized draft can record this. | -| Failure source fixed, environment repaired, or new upstream commit contains the fix | Run the failing check, then the affected gate on the current head. | Replace blocked status only with new evidence. | +| Failure source fixed, environment repaired, or new upstream commit contains the fix | Run the failing check. Then run the affected gate on the current head. | Replace blocked status only with new evidence. | + +Match failures by: + +- Check. +- Location. +- Diagnostic code. +- Assertion. -Match failures by check, location, diagnostic code, and assertion. -The implementing agent owns diagnosis until a named owner or issue accepts the blocker. -Do not loop on an unchanged failure. Do not widen timeouts, mask exit codes, or invoke CI authority as a local pass. -Rerun failed CI once only when its log shows an infrastructure fault. Treat the same failure twice as a finding. +Until a named owner or issue accepts the blocker, the implementing agent owns diagnosis. +Do not loop on an unchanged failure. +Do not widen timeouts. Do not mask exit codes. +Do not claim CI authority as a local pass. +Only if its log shows an infrastructure fault, rerun failed CI once. +If the same failure occurs twice, treat it as a finding. diff --git a/docs/agents/triage-labels.md b/docs/agents/triage-labels.md index 20febeaf29..e54b751248 100644 --- a/docs/agents/triage-labels.md +++ b/docs/agents/triage-labels.md @@ -1,13 +1,13 @@ # Issue Label Workflow -This file defines five triage roles, each a GitHub label of the same name. +Each of these five triage roles has a GitHub label with the same name. | Label | Apply when | | --- | --- | -| `needs-triage` | Every new issue; remove once it is understood and valid | -| `needs-info` | Reporter or external input is required | -| `ready-for-agent` | Fully specified; an AFK agent can pick it up with no extra human context | -| `ready-for-human` | Only the Chief of Staff agent applies it | -| `wontfix` | Only after an explicit maintainer decision not to act | +| `needs-triage` | Apply to every new issue. Once the issue is understood and valid, remove it. | +| `needs-info` | Reporter or external input is required. | +| `ready-for-agent` | Fully specified. An AFK agent can start without extra human context. | +| `ready-for-human` | Only the Chief of Staff agent applies it. | +| `wontfix` | Only after an explicit maintainer decision not to act. | -No other agent adds or removes `ready-for-human` on an issue or PR, even when a skill's triage step names it. +Even if a skill's triage step names `ready-for-human`, no other agent adds or removes it on an issue or PR. diff --git a/docs/agents/web-backend.md b/docs/agents/web-backend.md index f8efb009db..d9f83d8a4f 100644 --- a/docs/agents/web-backend.md +++ b/docs/agents/web-backend.md @@ -3,17 +3,22 @@ Web automation uses a managed `agent-browser` backend as an implementation detail. - Runtime web commands resolve the backend only from the state-dir managed install at `tools/agent-browser/`. -- Normal `--platform web` commands do not install the managed backend on first use. If the backend is missing, they fail with a setup hint. -- Use `agent-device web setup` before first web automation and in CI/sandbox bootstrap steps. -- Use `agent-device web doctor` to run the backend health check. +- Normal `--platform web` commands do not install the managed backend on first use. + If the backend is missing, they fail with a setup hint. +- Before first web automation or in CI/sandbox bootstrap steps, use `agent-device web setup`. +- To run the backend health check, use `agent-device web doctor`. - The managed install respects `--state-dir` / `AGENT_DEVICE_STATE_DIR`. -- Web automation requires Node 24+ while the rest of agent-device keeps its Node 22 baseline. -- Every backend call spawns the package's declared `bin` entry with the current Node runtime, - never the `node_modules/.bin` console shim: Windows ships that shim as `.cmd`, which - `child_process.spawn` refuses without a shell (CVE-2024-27980 hardening), and a shell would - reintroduce argument-quoting hazards. `runManagedAgentBrowser` is the only path that executes - the backend. Setup spawns `npm` from PATH as before, except on Windows, where a bare `npm` is - not spawnable and npm's own `npm-cli.js` runs under the current Node instead. +- Web automation requires Node 24+. + The rest of agent-device keeps its Node 22 baseline. +- Each backend call spawns the package's declared `bin` entry with the current Node runtime. + It never spawns the `node_modules/.bin` console shim. + Windows supplies that shim as `.cmd`. + Without a shell, `child_process.spawn` refuses it because of CVE-2024-27980 hardening. + A shell would reintroduce hazards from argument quoting. + Only `runManagedAgentBrowser` executes the backend. + Except on Windows, setup spawns `npm` from PATH as before. + On Windows, bare `npm` is not spawnable. + Instead, npm's own `npm-cli.js` runs under the current Node. Default first-run flow: @@ -27,8 +32,15 @@ agent-device network dump 25 --platform web agent-device close --platform web ``` -Do not document direct `agent-browser` commands as agent-device features. Web `network dump` is the -narrow exception: it adapts `agent-browser network requests` to the existing agent-device network -evidence shape. Browser-specific network routing/interception/HAR, CDP, React web, tabs, downloads, -auth vaults, and profiling stay out of the minimal web surface until there is an explicit -agent-device command design for them. +Do not document direct `agent-browser` commands as agent-device features. +Web `network dump` is the narrow exception. +It adapts `agent-browser network requests` to the existing agent-device network evidence shape. +Until an explicit agent-device command design exists for them, exclude these features from the minimal web surface: + +- Browser-specific network routing/interception/HAR. +- CDP. +- React web. +- Tabs. +- Downloads. +- Auth vaults. +- Profiling. diff --git a/docs/ios-runner-protocol-optimizations.md b/docs/ios-runner-protocol-optimizations.md deleted file mode 100644 index d09c3ecc31..0000000000 --- a/docs/ios-runner-protocol-optimizations.md +++ /dev/null @@ -1,159 +0,0 @@ -# iOS runner protocol optimization plan - -Issue #656 is now split into protocol infrastructure plus follow-up optimizations. The lifecycle -protocol makes commands identifiable, but the performance wins come from changing when the daemon -uses `uptime`, retries, invalidates sessions, and asks the runner for lifecycle status. - -## Work slices - -### 1. Status-before-invalidate recovery - -Status: in progress on `codex/ios-runner-status-recovery`. - -Goal: when a command has been sent and the HTTP response is lost, ask the runner for -`status(statusCommandId)` before invalidating the session or surfacing an ambiguous transport -failure. - -Acceptance criteria: - -- Post-send retryable transport failures issue one bounded `status` probe with the original - `commandId` before session invalidation. -- `completed` with retained small response JSON returns the recovered command result without - invalidating or resending the command. -- `failed` returns the runner failure code/message/hint instead of a generic transport failure. -- `notAccepted`, status timeout, or status transport failure preserves the existing invalidation - behavior. -- Read-only commands whose response was not retained keep the existing retry behavior. -- Status recovery probes are short-budget and do not consume the full command timeout. - -iOS simulator validation: - -- Unit: `pnpm exec vitest run packages/platform-apple/src/runner/__tests__/runner-client-command-retry.test.ts`. -- Unit bundle: `pnpm exec vitest run packages/platform-apple/src/runner/__tests__/runner-client.test.ts packages/platform-apple/src/runner/__tests__/runner-session.test.ts packages/platform-apple/src/runner/__tests__/runner-client-command-retry.test.ts packages/platform-apple/src/runner/__tests__/runner-provider.test.ts`. -- Build: `pnpm build:xcuitest`. -- Manual sim smoke after build: - - `pnpm build` - - `pnpm clean:daemon` - - run a simple iOS simulator session against Settings with `open`, `snapshot -i`, one selector - interaction, and `close`. - - confirm there is no visible behavior change and diagnostics show no unexpected session - invalidation. - -### 2. Adaptive `uptime` preflight policy - -Status: implemented with guardrails (see ADR 0005). The earlier blanket "recent success" cache was -shipped and then reverted in #702 because XCTest could return a successful tap response and then fail -the runner while re-resolving a navigation-disappeared element, and because sparse AX-fallback -snapshots were cached as healthy state. #702's coordinate-first activation removed that teardown -trigger, so the skip is reintroduced as a structurally narrower "healthy mutation recency" signal. - -Goal: skip the per-command `uptime` for hot allowlisted interaction loops only when a proven healthy -mutating response makes the runner's liveness already known, while every uncertain path keeps -preflighting. - -Acceptance criteria (as shipped): - -- First-command/startup, no-record, stale-record, app-activation-uncertain, and non-allowlisted - (conservative) commands still preflight; readiness probes and read-only startup commands keep - their existing skips. -- Recency is derived only from healthy (parsed ok, non-`runnerFatal`) responses of an explicit - mutating allowlist (`tap`, `longPress`, `drag`, `swipe`, `scroll`, `sequence`) for the same - `appBundleId`, within a 5s freshness window, and lives only on the session object so it dies with - every invalidation/restart. Snapshots and read-only responses never refresh it. -- A transport failure after a skipped preflight clears the recency record and marks the error with - the skip context (`runnerReadinessPreflightSkipped`, distinct from the restart predicate's - `runnerReadinessPreflightFailed`). Connection-shaped failures run status recovery before - invalidation — never a replay; timeout-shaped failures propagate with the skip context, matching - the existing classification for preflighted sends. -- Diagnostics expose whether a command used, skipped, or recovered from a readiness preflight, - including command type, skip reason, and recency age. -- Measured threshold: 1 runner request per hot allowlisted command after the first, with no increase - in invalidation or failure rate. - -iOS simulator validation: - -- Start a fresh simulator session and run one interaction: verify the first mutating command still - preflights. -- Run a hot loop of repeated selector interactions against the same visible control: verify the - runner remains healthy and diagnostics explain any readiness probe that was skipped. -- Compare median command latency for a hot interaction loop before and after the change. A useful - threshold is at least one fewer runner request per hot command and no increase in failure rate. - -### 3. Status-visible transport path - -Goal: make `accepted` and `started` states practically observable while a command is still running. -The Swift journal already records these states, but the runner currently serializes connection -handling, so a concurrent status request can be blocked behind the command it is querying. - -Acceptance criteria: - -- `status` can be answered while another runner command is waiting on main-thread XCTest work. -- The status path remains journal-only and does not touch app activation, XCTest dispatch, or - command retry logic. -- Long-running command status can report `accepted` or `started` before the command reaches a - terminal state. -- Existing command execution remains serial where mutation ordering matters. - -iOS simulator validation: - -- Run a deliberately long runner command in one request. -- While it is in flight, query `status(statusCommandId)` from another request. -- Verify status returns before the long command completes and reports `accepted` or `started`. -- Verify normal command ordering is unchanged for back-to-back mutating commands. - -### 4. Session invalidation reduction - -Goal: avoid tearing down otherwise healthy runner sessions when lifecycle status proves the command -completed or failed cleanly. - -Acceptance criteria: - -- Completed/failed lifecycle status suppresses invalidation for ambiguous post-send transport - errors when the runner remains reachable. -- Unknown status states still invalidate to preserve current safety. -- Diagnostics record why invalidation was skipped or retained. -- No command is replayed after an observed mutating `accepted`, `started`, `completed`, or `failed` - state. - -iOS simulator validation: - -- Inject or simulate a lost response after a command completes. -- Verify status recovery prevents runner restart. -- Run the next command in the same session and verify it succeeds without re-launching xcodebuild. - -### 5. Response retention tuning - -Goal: retain enough small command results for useful recovery without making the runner retain large -snapshots or binary-like payloads. - -Acceptance criteria: - -- Small scalar responses can be recovered from `lifecycleResponseJson`. -- Snapshot node trees and screenshots are not serialized or retained in the journal. -- The journal memory cap remains bounded by entry count and response JSON size. -- Retention policy is documented in tests or runner fixtures so future commands do not accidentally - store large payloads. - -iOS simulator validation: - -- Run small-result commands and verify status can recover retained JSON. -- Run snapshot-heavy commands and verify status reports terminal state without retained response JSON. -- Confirm the runner remains responsive after repeated snapshots. - -## Suggested ordering - -1. Land status-before-invalidate recovery first. It is the safety net needed before reducing - defensive preflights. -2. Add diagnostics/metrics for preflight use, skipped preflights, status recovery, and invalidation - reason. This can happen alongside slice 1 or 2. -3. Reduce `uptime` for hot interaction loops with a conservative command allowlist. -4. Make the status transport path observable during long-running commands. -5. Broaden the preflight policy only after simulator measurements show stable behavior. - -## Side-by-side work - -- Status recovery and diagnostics can be developed together or separately. -- Transport status visibility can proceed independently once the protocol is on `main`. -- Adaptive `uptime` should wait for status recovery, because it relies on the same recovery path for - ambiguous post-send failures. -- Response retention tuning can proceed independently as long as it preserves the current caps. diff --git a/docs/node-client-result-types.md b/docs/node-client-result-types.md deleted file mode 100644 index 0eb4a0c99c..0000000000 --- a/docs/node-client-result-types.md +++ /dev/null @@ -1,31 +0,0 @@ -# Node client result types - -The `0.20` minor line intentionally narrows public TypeScript return types when a command has an -accurate, closed daemon result contract. Runtime payloads are unchanged. This is source-compatible -for callers that only read real response fields, but TypeScript code that indexed arbitrary keys -from the former `Record` result must switch to the declared fields or explicitly -narrow an external payload at its own trust boundary. - -The closed-result batch covers: - -- `command.doctor` -- `capture.diff` (`kind: 'snapshot'`) -- `replay.run` -- `replay.test` -- `recording.record` -- `recording.trace` - -Their canonical result types live in `src/contracts`, feed `CommandResultMap`, and have matching MCP -output schemas. - -The remaining broad methods are deliberate: - -- `command.alert` and `command.reactNative` spread platform/interaction-specific data. -- The interaction family (`click` through `find`) needs one public response projection reconciled - with settle/evidence and fast-path additions before its existing runtime contracts are safe as - public return types. -- Observability (`perf`, `logs`, `events`, `network`, `audio`) is action- and backend-dependent. -- `settings.update` spreads backend-specific setting data. - -Those methods remain `CommandRequestResult` until their producers expose accurate closed public -contracts. Do not narrow them with casts, compatibility aliases, or invented partial shapes. diff --git a/docs/usbmux-runner-transport-experiment.md b/docs/usbmux-runner-transport-experiment.md deleted file mode 100644 index 87b656c299..0000000000 --- a/docs/usbmux-runner-transport-experiment.md +++ /dev/null @@ -1,100 +0,0 @@ -# #1403 experiment: usbmux as the primary physical-iOS runner transport - -Live results from 2026-07-30, thymikee-iphone (iPhone 17 Pro, iOS 26.5.2, UDID -00008150-001849640CF8401C), macOS host with Xcode 26.2. Method: forced-route -env override `AGENT_DEVICE_IOS_RUNNER_ROUTE=usbmux` in `runner-command-route.ts` -(committed with this doc), isolated `--state-dir` daemons per leg, timings -from per-request `--debug` ndjson plus wall-clock. The override is -**daemon-scoped**: the daemon captures its environment at launch, so setting -the variable on a later CLI invocation against a running daemon silently -keeps the previous route — every route leg needs a fresh isolated -state-dir/daemon launched with the desired value. Every timing sample was -validated against real command output after an early instrument artifact (see -Pitfalls). - -## Numbers (same build, same cable state, verified output) - -| Scenario | network (tunnel-IP) route | usbmux route | -|---|---|---| -| Steady-state snapshot, cable in | 350–450 ms | 440 ms | -| Snapshot after 40 s idle | **4.5–4.6 s** (tunnel re-probe + session re-establish) | **440 ms** (unchanged) | -| Runner killed, next snapshot | ~4.0 s self-heal | 4.7 s self-heal | -| Steady-state snapshot, Wi-Fi only (no cable) | 425–495 ms | fails (see below) | -| Full lifecycle open/snapshot/tap/screenshot/close | works | works | - -Older-build measurements (first pass) agreed in shape: first open pays one-time -xctestrun prep (~13 s cold) and runner startup; fresh device resolution costs -~6.7 s of devicectl listing either way. - -## Findings - -1. **The idle tax is the payoff, and it is bigger than expected.** Past the - 30 s tunnel-IP cache TTL, the network route re-probes the tunnel and - re-establishes runner readiness: ~4.5 s per idle gap, every time. Over - usbmux the session stays hot indefinitely (40 s gaps measured at steady - 440 ms). Agent workflows are exactly the idle-gappy workload that hits this - tax on almost every step. -2. **Steady state is a wash.** The per-command usbmuxd handshake (fresh unix - socket, ListDevices, Connect) roughly cancels the network route's pooled- - connection advantage; ~±90 ms either way. -3. **Wi-Fi deletion is off the table.** A CoreDevice-paired, Wi-Fi-reachable - iPhone (devicectl `available (paired)`, live tunnel) does NOT appear in - usbmuxd at all — modern CoreDevice Wi-Fi runs over `remoted`, invisible to - usbmuxd. usbmuxd listed the device only while cabled, `ConnectionType: USB` - only, and the entry disappears on unplug. The CoreDevice network route must - stay as the Wi-Fi fallback → this is a "usbmux-primary + network-fallback" - reshuffle, not a full tunnel-code deletion. -4. **Failure semantics gap (also affects today's xctest backend).** With the - cable out, forced-usbmux commands hang for 2×45 s of retries and surface a - generic "Runner did not accept connection" — the usbmux client's precise - `DEVICE_NOT_FOUND` ("Connect the device by cable, trust this Mac…") is - swallowed by `shouldRetryRunnerConnectError`. An implementation should make - usbmux DEVICE_NOT_FOUND non-retryable (or fall back to the network route - immediately) and preserve the hint. -5. **Lock behavior is transport-independent.** A locked phone wedges AX - capture/runner relaunch identically on both routes (SBMainWorkspace - "Locked" denial; xcodebuild "Unlock thymikee-iphone to Continue"). -6. **usbmux UDID matching is exact.** usbmuxd `SerialNumber` equals the - dashed hardware UDID that `DeviceInfo.id` already uses for devicectl-listed - devices; DeviceID (10) is the mux handle. Multi-device disambiguation is - structural. (Multi-attached-device matrix not exercised: one device only.) - -## Shipped design (matches the issue's desired outcome, minus full deletion) - -Implemented on top of this evidence; the experiment's -`AGENT_DEVICE_IOS_RUNNER_ROUTE` override is gone because usbmux-first is now -the real behaviour rather than something to opt into. - -- One private route resolver: physical devices resolve to usbmux first. When - usbmuxd answers that the device is not attached, the resolver is marked for - the rest of the request and re-resolves to the CoreDevice tunnel route. - Callers never choose a transport; `waitForRunner` and `sendRunnerCommandOnce` - both go through the same resolver. -- The tunnel lookup, its 30 s cache, and the cache invalidation now only run on - the fallback path, so a cabled device never pays the probe/TTL tax that cost - ~4.5 s per idle gap. -- Failure semantics: the usbmux "device not attached" verdict carries - `usbmuxDeviceAttached: false`, and `isUsbmuxDeviceUnattachedError` is the only - thing that triggers fallback. It is answered inside the same connect attempt - rather than by burning a retry, so a Wi-Fi-only device is not slowed down. -- The xctest backend stays usbmux-only (it has no CoreDevice tunnel by - definition), so its verdict is terminal and surfaces the cable/trust/unlock - hint instead of being retried for the full budget. - -## Pitfalls for whoever implements/re-measures - -- `bin/agent-device.mjs` runs `dist/`, not `src/`. A stale dist silently ran - the first "usbmux leg" on stock network code while the env override sat - inert in the daemon (numbers looked plausible; only the cable-pull test - exposed it). Rebuild and verify the daemon process (`ps eww `, dist - mtime, and a behavioral probe) before trusting route attribution. -- `/usr/bin/time` on a failing command still prints a small `real` — a fast - error latency is indistinguishable from a fast success unless output is - asserted. Runner-lease contention ("already owned by another daemon") - produced exactly this artifact. -- The route override is daemon-scoped (env captured at daemon launch). Never - flip `AGENT_DEVICE_IOS_RUNNER_ROUTE` between commands against the same - daemon and expect the route to change — start a fresh isolated - state-dir/daemon per leg, then verify with `ps eww `. -- Sessions are cwd-scoped; run every command of a leg from the same cwd. -- Auto-Lock must be Never during measurement or the runner dies mid-series. diff --git a/packages/command-registry/src/command-result.ts b/packages/command-registry/src/command-result.ts index 11f9d271e9..2406b7bc95 100644 --- a/packages/command-registry/src/command-result.ts +++ b/packages/command-registry/src/command-result.ts @@ -96,6 +96,9 @@ export interface CommandResultMap { * command falls back to the untyped `Record` bag. That default * branch is what keeps the mapping total over every command name, so consumers * can switch to `CommandResult` without first migrating every command. + * Add a command to the map only when its producer exposes an accurate closed + * contract. Do not narrow an unmigrated result with casts, compatibility + * aliases, or invented partial shapes. */ export type CommandResult = N extends keyof CommandResultMap ? CommandResultMap[N] diff --git a/packages/platform-apple/src/runner/__tests__/runner-startup-failure-fixtures.ts b/packages/platform-apple/src/runner/__tests__/runner-startup-failure-fixtures.ts index 8c520eceba..e2f63d4928 100644 --- a/packages/platform-apple/src/runner/__tests__/runner-startup-failure-fixtures.ts +++ b/packages/platform-apple/src/runner/__tests__/runner-startup-failure-fixtures.ts @@ -14,18 +14,19 @@ import { IOS_SIMULATOR } from './device-fixtures.ts'; * Provenance is the point of this file, so it is stated per entry and never as a blanket claim: * * - `captured` — `output` was pasted from a run, and `command` plus `xcodeVersion` (from - * `xcodebuild -version`) were recorded with it by `.device-evidence/CHECKLIST-runner-failures.md`. + * `xcodebuild -version`) were recorded with it in the same run. * - `shipped-sniff-trigger` — the substrings a rule matches are the ones shipped in * `resolveSigningFailureHint` before #2680, which is evidence xcodebuild can emit them. The * sentence around them is ours, so `command` and `xcodeVersion` stay unrecorded. * - `invented-shape` — no shipped trigger and no capture. The entry exists to exercise a rule and * makes no claim about wording xcodebuild prints. * - * Until Phase B captures the real runs, every entry is `unobserved` for `xcodeVersion` and carries - * no `command`: an invocation we did not run is not provenance. Nothing in the classifier reads + * An entry that is not `captured` is `unobserved` for `xcodeVersion` and carries no `command`: an + * invocation we did not run is not provenance. The capture commands for the uncaptured rows are in + * `git show 59fdf008b9:.device-evidence/CHECKLIST-runner-failures.md`. Nothing in the classifier reads * these fields; they exist so a reason can be traced to an observation instead of to a guess. * - * Blocked on the host, not unexamined. One Phase B run on `thymikee-iphone` / Xcode 26.2 settled the + * Blocked on the host, not unexamined. One run on a physical iPhone with Xcode 26.2 settled the * shape question these rows were held on — `No profiles for '' were found` arrives as one long * `error:` line, not a wrapped one — and `no-profiles-for-bundle-id` below is now `captured`. The * rest are gated on an Apple account this machine does not have, and they need it for the same @@ -280,7 +281,7 @@ export const RUNNER_STARTUP_FAILURE_FIXTURES: readonly RunnerStartupFailureFixtu provenance: 'invented-shape', output: "error: Provisioning profile \"match-development\" is not a valid provisioning profile (in target 'AgentDeviceRunner' from project 'AgentDeviceRunner')\nError Domain=IDEProvisioningErrorDomain Code=17 \"Provisioning profile 'match-development' is not a valid provisioning profile.\"\n** TEST BUILD FAILED **\n", - note: "Xcode repeats the profile inside the same line as its IDEProvisioningErrorDomain diagnostics, which is what the row reads: domain on one line and profile on another is two facts, not one complaint. Sentence and domain code are our reconstruction; Phase B capture has to record the real wording and this entry's xcodeVersion.", + note: "Xcode repeats the profile inside the same line as its IDEProvisioningErrorDomain diagnostics, which is what the row reads: domain on one line and profile on another is two facts, not one complaint. Sentence and domain code are our reconstruction; a capture run has to record the real wording and this entry's xcodeVersion.", }, { id: 'profile-does-not-cover-app-id',