diff --git a/CHANGELOG.md b/CHANGELOG.md index 89500c54b4..3f7886fdbc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,8 @@ - Add `settings mobile-data on|off` for adb-backed Android targets. It switches mobile data through `svc data` and reports the `mobileData` state stored in the global `mobile_data` row; multi-SIM read-back is not covered yet. Every other target refuses it before anything runs, and `--app` is refused because the switch is device-wide. - On iOS, a `fill` that runs under a degraded XCTest snapshot channel now aims at a point, including ref, selector and replay fills. When the runner identifies no text input under that point, it refuses with `NO_TEXT_INPUT_AT_POINT` (`error.details.reason: no_text_input_at_point`, `dispatched: no`) before dispatching its focus tap, and no field changes; when the input lookup itself cannot answer, it refuses with `TEXT_INPUT_PROBE_UNAVAILABLE` (`error.details.reason: text_input_probe_unavailable`). Previously such a fill cleared and typed through the app-level first responder, so it could rewrite a field the caller never named, or report a clear failure after emptying it. A ref or selector fill whose rect centre misses the real field (an outer container, a non-input node) now needs a point inside the input's own rect. - Opening a macOS app by bundle identifier now waits briefly when the app is already running but not yet registered with LaunchServices — the case when a supervisor or dev script started it directly — and if registration arrives within that bounded window the open attaches to the running copy, so the session drives the app the caller started, with its arguments. A copy still unlisted when the window closes gets the plain `open` as before, which can still duplicate; the wait only bounds the race, it does not close it. Launching an app that is not running costs one extra LaunchServices lookup and one process-table read, plus one bundle-identifier read when app-shaped processes exist. +- Limrun iOS sessions now serve `alert get/wait/accept/dismiss` instead of refusing with `unsupported-provider-mode`. System alerts are read from the same element-tree capture snapshots ride — a blank-label, process-exclusive application root hosting a text-and-buttons sheet — and accepted or dismissed by tapping the button whose English label states the asked-for action; a capture that is alert-shaped but unresolvable (degraded buttons, competing sheets, an unverifiable owner) is reported as undecided and never pressed or counted as dismissed. The one shape no capture fact separates is an app dialog under a blanked root shared by no other process; the label-only acting legs answer it with a refusal, never a guess. +- The tree-based alert backends (Android and Limrun iOS) now share one alert driver and one button vocabulary. The vocabulary is a per-role preference table over English labels — `alert accept/dismiss` classify the same label the same way on both platforms, so Android now recognizes iOS variant grants (`Allow Once`, `Allow All the Time`, `Allow While Using App`, `Allow Notifications`) and `Ask App Not to Track` as acting choices — and Android layers its own grant-id rank on top of it: the permissioncontroller ids order the sheet's grants (foreground-only, then always, then one-time) wherever an id exists, so a localized Android 11+ sheet ("Mientras la app está en uso"/"Solo esta vez") and the Android 10 "Allow only while using the app" resolve to the lasting grant instead of refusing. A choice refuses only when the ranks name no single physical button for the action — the same label or the same grant id at two points, or a candidate no rank reaches — and each backend's own policy answers then (Android's neutral-button/Back fallback, Limrun's typed refusal). `alert not found` and the no-matching-button refusal now carry the same typed `details.reason` on both platforms. `alert get` reports its outcome as a typed `status` (`visible`/`none`/`undecided`) so clients switch on states instead of parsing the message. An alert budget bounds only how long a leg polls: every capture that started inside the budget finishes and its answer counts, and only the caller's cancellation stops a capture or press from being acted on (Android's adb child is killed; Limrun's client exposes no abort, so a call already in flight runs to its answer while the leg itself ends at the next seam the caller's cancellation can reach). A quiet-screen `alert accept|dismiss` therefore always answers `alert-not-found` with `dispatched: no`, never a reason that depends on which capture crossed the deadline. An `alert wait` budget above 2147483647 ms is refused as `INVALID_ARGS` rather than misfiring a timer. - Limrun iOS snapshots no longer fail when the instance reports a numeric or boolean accessibility value (the system photo picker's tabs report `1`/`0`): the provider seam normalizes variant scalars to text, the way the Apple tree does. Limrun nodes now also publish `selected: true` for nodes carrying the `Selected` trait, so snapshot output and `selected:` selectors can tell a picked photo from an unpicked one. - `settings` refuses a setting the target does not serve before anything changes, with `UNSUPPORTED_OPERATION`, `settings is not supported on this device`, and the target's `error.details.reason` (such as `unsupported-platform-leaf`). This replaces the macOS `INVALID_ARGS` refusal and the Apple `setting-unsupported-on-leaf` reason, and now also covers settings that used to reach the device first, such as Android `reset-keychain` or Apple `animations`. - `settings` refuses, before touching the device, a state its setting does not accept (`Invalid state: …`), a permission action or mode its target does not take (such as `permission deny accessibility`), and a stray extra argument (such as `clear-app-state `). The CLI and the daemon accept the same state spellings, in any casing or surrounding whitespace. diff --git a/packages/capture-kit/package.json b/packages/capture-kit/package.json index 93b2b54e02..50eb31dafd 100644 --- a/packages/capture-kit/package.json +++ b/packages/capture-kit/package.json @@ -18,6 +18,10 @@ "types": "./src/snapshot/android-replacement-surface-occlusion.ts", "default": "./src/snapshot/android-replacement-surface-occlusion.ts" }, + "./alert-tree-driver": { + "types": "./src/alert-tree-driver.ts", + "default": "./src/alert-tree-driver.ts" + }, "./app-log-durable-resource": { "types": "./src/capture-admission/app-log-durable-resource.ts", "default": "./src/capture-admission/app-log-durable-resource.ts" diff --git a/packages/capture-kit/src/alert-tree-driver.ts b/packages/capture-kit/src/alert-tree-driver.ts new file mode 100644 index 0000000000..ef4451d95f --- /dev/null +++ b/packages/capture-kit/src/alert-tree-driver.ts @@ -0,0 +1,447 @@ +import { + type AlertAction, + ALERT_ACTION_RETRY_MS, + ALERT_BUTTON_UNDECIDED_REASON, + ALERT_CAPTURE_UNDECIDED_REASON, + type AlertCapture, + type AlertInfo, + type AlertPlatform, + ALERT_NOT_FOUND_REASON, + ALERT_POLL_INTERVAL_MS, + ALERT_WINDOW_EXPIRED_REASON, + type AlertStatusKind, + DEFAULT_ALERT_TIMEOUT_MS, + MAX_ALERT_WINDOW_MS, +} from '@agent-device/contracts/alert-contract'; +import { Deadline, type DeadlineClock, pollUntilSettled } from '@agent-device/host-kit/retry'; +import { AppError, createRequestCanceledError } from '@agent-device/kernel/errors'; +import { successText } from '@agent-device/kernel/success-text'; + +/** + * THE tree-based alert driver: every backend that reads alerts out of a capture tree — the + * Android ADB legs and the Limrun iOS legs — reaches poll, confirm, same-alert, refusal, and + * response building through this one neutral module, and its button roles through the shared + * label vocabulary in `contracts/alert-contract`. A backend supplies only its own observation + * (`capture` + `findCandidate`), its platform's acting policy (`choose`), and its press seam + * (`press`). It lives in the capture kit, not in a platform package: an iOS provider must not + * depend on the Android package to poll a tree. + * + * The budget rule is the whole cancellation story, and `pollUntilSettled` owns it: every probe + * that started inside the budget is awaited to its answer and that answer counts — a started + * capture finishes — and the deadline only ends the LOOP, never a capture in flight. Only the + * caller's cancellation stops a transport. A quiet screen therefore answers `alert-not-found` + * on every backend instead of a timing-dependent window failure. The cost of the rule is stated + * in the docs: a transport with no abort (Limrun's client) can overrun the budget by one call, + * and a capture that never returns keeps the leg open until the caller cancels it. + */ + +/** The point a press seam aims at: a button's center in the backend's own coordinate space. */ +export type TreeAlertButton = Readonly<{ label: string; x: number; y: number }>; + +/** + * What one capture says about alerts; the one three-way observation lives in the alert + * contract next to the shared vocabulary, and every tree backend's reader answers in it. + */ +export type TreeAlertObservation = AlertCapture; + +/** One acting choice: press a named button, or press a platform fallback key (Android Back). */ +export type TreeAlertPress = + | { kind: 'press'; button: TButton } + | { kind: 'fallback'; label: string }; + +export type TreeAlertCandidate< + TAlert extends AlertInfo & { buttons: string[] }, + TButton extends TreeAlertButton, +> = Readonly<{ + alert: TAlert; + buttons: TButton[]; +}>; + +/** + * The seams one tree-based backend owns. `signal` is always the daemon request's own + * cancellation, never a budget timer: it is what may stop a transport in flight. A transport + * with no abort (Limrun's client) cannot be stopped this way — its in-flight call keeps running + * and the leg waits for the answer it started (the overrun named above). The caller meanwhile + * learns of its cancellation from its own client transport; the daemon's own answer lands once + * the leg settles on the started probe's answer. + */ +export type TreeAlertSeam = Readonly<{ signal: AbortSignal }>; + +export type TreeAlertDriver< + TCapture, + TAlert extends AlertInfo & { buttons: string[] }, + TButton extends TreeAlertButton, + TCandidate extends TreeAlertCandidate, + TPlatform extends AlertPlatform, +> = Readonly<{ + platform: TPlatform; + timeoutMs?: number; + /** The daemon request's cancellation, when the caller rides one; the only seam stop signal. */ + signal?: AbortSignal; + capture: (seam: TreeAlertSeam) => Promise; + findCandidate: (capture: TCapture) => TreeAlertObservation; + choose: (candidate: TCandidate, action: 'accept' | 'dismiss') => TreeAlertPress | null; + press: (choice: TreeAlertPress, seam: TreeAlertSeam) => Promise; +}>; + +/** The records the daemon composes CLI and SDK output around, identical across tree backends. */ +export type TreeAlertResult< + TAlert extends AlertInfo & { buttons: string[] }, + TPlatform extends AlertPlatform, +> = + | { + kind: 'alertStatus'; + platform: TPlatform; + action: 'get'; + alert: TAlert | null; + /** Typed absence vs. an unresolvable capture, so clients switch on states, not messages. */ + status: AlertStatusKind; + message?: string; + } + | { + kind: 'alertWait'; + platform: TPlatform; + action: 'wait'; + alert: TAlert; + waitedMs: number; + message?: string; + } + | { + kind: 'alertHandled'; + platform: TPlatform; + action: 'accept' | 'dismiss'; + handled: true; + alert: TAlert; + button: string; + coordinates?: { x: number; y: number }; + message?: string; + }; + +const ALERT_NOT_FOUND_HINT = + 'If a sheet is visible in snapshot but alert reports no alert, it is likely app-owned UI. Use snapshot -i and press the visible label/ref.'; + +/** + * Runs one alert leg. `get` is a single observation with no budget to bound. `wait` polls one + * budget for an alert to appear. `accept`/`dismiss` look, press exactly the one choice the + * action's role names, and confirm the alert is gone. In every leg a started capture finishes + * on the caller's cancellation alone, and the deadline only ends the polling loop. + */ +export async function runTreeAlertAction< + TCapture, + TAlert extends AlertInfo & { buttons: string[] }, + TButton extends TreeAlertButton, + TCandidate extends TreeAlertCandidate, + TPlatform extends AlertPlatform, +>( + action: AlertAction, + driver: TreeAlertDriver, +): Promise> { + if (action === 'get') { + // One immediate observation with no budget to expire: only the caller's cancellation binds it. + return buildTreeAlertStatusResponse(driver.platform, await observeTreeAlert(driver)); + } + if (action === 'wait') { + const start = Date.now(); + const polled = ( + await pollTreeAlert(driver, Deadline.fromTimeoutMs(alertWindowMs(driver.timeoutMs))) + ).value; + if (polled.kind !== 'alert') { + // Both a quiet capture and one that stayed alert-shaped yet unresolvable mean no + // actionable alert ever resolved within the window; the timeout phrasing is the honest + // answer for either. + throw new AppError('COMMAND_FAILED', 'alert wait timed out'); + } + return { + kind: 'alertWait', + platform: driver.platform, + action: 'wait', + alert: polled.candidate.alert, + waitedMs: Date.now() - start, + ...successText('Alert visible'), + }; + } + return await actOnTreeAlert(action, driver); +} + +/** + * One capture read through the driver's seams, on the caller's cancellation alone. A capture + * whose seam transport died with that cancellation reports the shared cancellation error, never + * a raw transport failure: an aborted request must not read as a broken backend. + */ +async function observeTreeAlert( + driver: TreeAlertObserverDriver, +): Promise> { + if (driver.signal?.aborted) throw createRequestCanceledError(); + let capture: TCapture; + try { + capture = await driver.capture({ signal: driver.signal ?? neverSignal }); + } catch (error) { + if (driver.signal?.aborted) { + // A capture that dies with the cancellation dispatches nothing on the get/wait/lookup + // routes; post-press deaths rejoin `actOnTreeAlert`'s catch, which owns the typed 'unknown'. + throw createRequestCanceledError({ dispatched: 'no' }, error); + } + throw error; + } + return driver.findCandidate(capture); +} + +type TreeAlertObserverDriver = Readonly<{ + signal?: AbortSignal; + capture: (seam: TreeAlertSeam) => Promise; + findCandidate: (capture: TCapture) => TreeAlertObservation; +}>; + +/** + * Polls the driver's observation until `settled` accepts an answer or the budget ends the loop. + * The budget never cuts a capture: `pollUntilSettled` awaits every probe it starts and clamps + * each sleep to the deadline, so the terminal answer is what the probes answered, never how the + * clock fell. With no `settled` given, the loop ends on an alert. `value` is the last answer + * either way and `sawUndecided` states the window saw an unresolvable capture at some point. The + * acting lookup is what that fact binds: it decides absence vs undecided there, and a degraded + * sheet that never resolved is not an absence. The wait leg deliberately ignores it — its + * timeout phrasing ("no alert resolved within the window") is the honest answer for a quiet + * screen and for an unresolved sheet alike, and `alert get` remains the undecided reporter. + */ +async function pollTreeAlert( + driver: TreeAlertObserverDriver, + deadline: DeadlineClock, + settled: (observation: TreeAlertObservation) => boolean = (observation) => + observation.kind === 'alert', +): Promise<{ settled: boolean; value: TreeAlertObservation; sawUndecided: boolean }> { + let sawUndecided = false; + const { settled: loopSettled, value } = await pollUntilSettled({ + probe: () => observeTreeAlert(driver), + settled: (observation) => { + if (observation.kind === 'undecided') sawUndecided = true; + return settled(observation); + }, + deadline, + intervalMs: ALERT_POLL_INTERVAL_MS, + signal: driver.signal, + }); + return { settled: loopSettled, value, sawUndecided }; +} + +/** + * Accept and dismiss share one contract on every tree backend: look within the action budget, + * press exactly the one choice the action's role names, then confirm the alert is gone. + * `dismiss` never authorizes anything — an alert whose denying choice cannot be named is + * reported as the ambiguity it is instead of pressing something. + */ +async function actOnTreeAlert< + TCapture, + TAlert extends AlertInfo & { buttons: string[] }, + TButton extends TreeAlertButton, + TCandidate extends TreeAlertCandidate, + TPlatform extends AlertPlatform, +>( + action: 'accept' | 'dismiss', + driver: TreeAlertDriver, +): Promise> { + const polled = await lookupTreeAlert(action, driver); + // A cancel that arrived while the lookup's last probe was in flight is honored here, before + // anything is dispatched: the caller withdrew the action, so the press never starts and the + // exit is the clean pre-press cancellation (`dispatched: no`), not the press-time unknown. + if (driver.signal?.aborted) { + throw createRequestCanceledError({ dispatched: 'no' }); + } + const candidate = polled.candidate; + const choice = driver.choose(candidate, action); + if (!choice) { + throw new AppError( + 'COMMAND_FAILED', + `alert ${action} found an alert but no button that clearly matches the action`, + { + reason: ALERT_BUTTON_UNDECIDED_REASON, + dispatched: 'no', + alert: candidate.alert, + hint: 'Inspect alert get --json for the visible buttons, then press the choice you want by label/ref.', + }, + ); + } + // The press rides only the caller's cancellation: once the loop has chosen, the press is the + // action itself, not a probe a budget may end. From the moment the press starts, every + // caller-cancelled exit is typed `dispatched: unknown` — the device may already have received + // the tap — whether the transport dies with the cancel (the catch), answers it AFTER the + // cancel (a transport with no abort resolves normally on an aborted request: the post-press + // check), or the cancel arrives during the post-press confirm window. None of them may read + // as a clean 'no'. + try { + await driver.press(choice, { signal: driver.signal ?? neverSignal }); + await confirmTreeAlertGone(driver, candidate.alert, action); + } catch (error) { + if (driver.signal?.aborted) { + throw createRequestCanceledError({ dispatched: 'unknown' }, error); + } + throw error; + } + return { + kind: 'alertHandled', + platform: driver.platform, + action, + handled: true, + alert: candidate.alert, + button: choice.kind === 'press' ? choice.button.label : choice.label, + ...(choice.kind === 'press' ? { coordinates: { x: choice.button.x, y: choice.button.y } } : {}), + ...successText(`Alert ${action}ed`), + }; +} + +/** + * The acting leg's lookup phase: the action budget, ended either by a candidate or by the + * whole window's answer. An unsettled lookup is aggregated over the window, not just its last + * probe: one undecided capture anywhere in it says the screen held an alert-shaped something + * the reader could not resolve, which no later quiet read may rewrite into proven absence. + */ +async function lookupTreeAlert< + TCapture, + TAlert extends AlertInfo & { buttons: string[] }, + TButton extends TreeAlertButton, + TCandidate extends TreeAlertCandidate, + TPlatform extends AlertPlatform, +>( + action: 'accept' | 'dismiss', + driver: TreeAlertDriver, +): Promise<{ kind: 'alert'; candidate: TCandidate }> { + const lookup = await pollTreeAlert(driver, Deadline.fromTimeoutMs(ALERT_ACTION_RETRY_MS)); + const answer = lookup.settled + ? lookup.value + : lookup.sawUndecided + ? ({ kind: 'undecided' } as const) + : ({ kind: 'none' } as const); + if (answer.kind !== 'alert') throw lookupRefusal(action, answer); + return answer; +} + +/** + * The acting leg's confirm phase: `alert accept|dismiss` means the dialog is gone, not that a + * press was sent, so only an observed absence or a different alert in its place settles the + * leg. The confirm window is its own budget, started the moment the press lands; like the + * lookup's, it ends its loop at the deadline and consumes the answer of every probe that + * started inside it. Neither happened, so the "still visible" claim may only be made by the + * latest capture that ANSWERED, and it must name this very dialog again — the freshest answer + * said something else (or nothing), and all the leg may then report is the press + * unconfirmed: the dialog may already be gone. + */ +async function confirmTreeAlertGone< + TCapture, + TAlert extends AlertInfo & { buttons: string[] }, + TButton extends TreeAlertButton, + TCandidate extends TreeAlertCandidate, + TPlatform extends AlertPlatform, +>( + driver: TreeAlertDriver, + dismissed: TAlert, + action: 'accept' | 'dismiss', +): Promise { + const confirm = await pollTreeAlert( + driver, + Deadline.fromTimeoutMs(ALERT_ACTION_RETRY_MS), + (observation) => + observation.kind === 'none' || + (observation.kind === 'alert' && !sameTreeAlert(observation.candidate.alert, dismissed)), + ); + if (confirm.settled) return; + if (confirm.value.kind === 'alert' && sameTreeAlert(confirm.value.candidate.alert, dismissed)) { + throw visibleAfterPressFailure(action, dismissed); + } + throw new AppError( + 'COMMAND_FAILED', + `alert ${action} pressed the button but the dismissal could not be confirmed`, + { + alert: dismissed, + reason: ALERT_WINDOW_EXPIRED_REASON, + hint: 'The alert button was pressed; the post-press checks never settled the dialog as gone, so it may already be gone. Run alert get --json before retrying.', + }, + ); +} + +/** + * What an unsettled lookup answers with: absence, or an alert-shaped capture no reader could + * resolve. Both pressed nothing, and neither may be reported as the other. + */ +function lookupRefusal( + action: 'accept' | 'dismiss', + answer: { kind: 'none' } | { kind: 'undecided' }, +): AppError { + if (answer.kind === 'undecided') { + return new AppError( + 'COMMAND_FAILED', + `alert ${action} refused: the capture is alert-shaped but cannot be resolved to one alert`, + { + reason: ALERT_CAPTURE_UNDECIDED_REASON, + dispatched: 'no', + hint: 'Inspect alert get --json and snapshot -i, then press the visible choice by label/ref.', + }, + ); + } + return new AppError('COMMAND_FAILED', 'alert not found', { + reason: ALERT_NOT_FOUND_REASON, + dispatched: 'no', + hint: ALERT_NOT_FOUND_HINT, + }); +} + +/** + * The stuck-alert failure: a post-press capture that answered named the same dialog again, so + * the press did not dismiss what the latest completed read still saw on screen. + */ +function visibleAfterPressFailure( + action: 'accept' | 'dismiss', + dismissed: TAlert, +): AppError { + return new AppError('COMMAND_FAILED', `alert ${action} did not dismiss the visible alert`, { + alert: dismissed, + reason: ALERT_WINDOW_EXPIRED_REASON, + hint: 'The alert button was pressed but the dialog was visible on the last check that answered. Inspect alert get --json, then press the visible button by label/ref or retry.', + }); +} + +/** Alert identity across a dismissal re-check: the same title and the same button row, in order. */ +function sameTreeAlert( + left: AlertInfo & { buttons: string[] }, + right: AlertInfo & { buttons: string[] }, +): boolean { + return left.title === right.title && left.buttons.join('\u0000') === right.buttons.join('\u0000'); +} + +function buildTreeAlertStatusResponse< + TAlert extends AlertInfo & { buttons: string[] }, + TButton extends TreeAlertButton, + TCandidate extends TreeAlertCandidate, + TPlatform extends AlertPlatform, +>( + platform: TPlatform, + observation: TreeAlertObservation, +): TreeAlertResult { + return { + kind: 'alertStatus', + platform, + action: 'get', + alert: observation.kind === 'alert' ? observation.candidate.alert : null, + status: + observation.kind === 'alert' + ? 'visible' + : observation.kind === 'undecided' + ? 'undecided' + : 'none', + ...(observation.kind === 'alert' + ? successText('Alert visible') + : observation.kind === 'undecided' + ? successText('Cannot decide the alert: the capture is alert-shaped but unresolvable') + : successText('No alert visible')), + }; +} + +/** The seam signal for a leg with no caller cancellation: nothing can cancel it. */ +const neverSignal = new AbortController().signal; + +function alertWindowMs(timeoutMs: number | undefined): number { + return typeof timeoutMs === 'number' && + Number.isFinite(timeoutMs) && + timeoutMs >= 0 && + timeoutMs <= MAX_ALERT_WINDOW_MS + ? timeoutMs + : DEFAULT_ALERT_TIMEOUT_MS; +} diff --git a/packages/contracts/src/alert-contract.test.ts b/packages/contracts/src/alert-contract.test.ts new file mode 100644 index 0000000000..c179f65a84 --- /dev/null +++ b/packages/contracts/src/alert-contract.test.ts @@ -0,0 +1,107 @@ +import { describe, expect, it } from 'vitest'; +import { classifyAlertButtonRole, selectRankedAlertButtonByRole } from './alert-contract.ts'; + +type Button = { label: string; role: ReturnType }; +const button = (label: string): Button => ({ label, role: classifyAlertButtonRole(label) }); + +describe('selectRankedAlertButtonByRole', () => { + it('resolves distinct grants by the declared preference order', () => { + const buttons = [button("Don't Allow"), button('Allow Once'), button('Allow While Using App')]; + expect(selectRankedAlertButtonByRole(buttons, 'accept')?.label).toBe('Allow While Using App'); + expect(selectRankedAlertButtonByRole(buttons, 'dismiss')?.label).toBe("Don't Allow"); + }); + + it('resolves the Android grant pair by the same table', () => { + const buttons = [button('While using the app'), button('Only this time'), button('Deny')]; + expect(selectRankedAlertButtonByRole(buttons, 'accept')?.label).toBe('While using the app'); + }); + + it('refuses on equal-rank duplicates: the capture names no single physical button', () => { + const buttons = [button('OK'), button('OK'), button('Cancel')]; + expect(selectRankedAlertButtonByRole(buttons, 'accept')).toBeNull(); + }); + + it('refuses when multiple candidates share a role the vocabulary cannot rank', () => { + // Roles can arrive from a platform id classifier; labels outside the vocabulary then tie. + const buttons: Button[] = [ + { label: 'While on the go', role: 'accept' }, + { label: 'Just this run', role: 'accept' }, + ]; + expect(selectRankedAlertButtonByRole(buttons, 'accept')).toBeNull(); + }); + + it('refuses a mix of ranked and unranked candidates rather than preferring the known label', () => { + // The unranked one's role came from a platform id classifier, so it is a real grant the + // table cannot order. Letting the one known English label win would press an arbitrary + // grant of an unordered pair. + const buttons: Button[] = [ + { label: 'OK', role: 'accept' }, + { label: 'Unrecognized grant', role: 'accept' }, + ]; + expect(selectRankedAlertButtonByRole(buttons, 'accept')).toBeNull(); + }); + + it('keeps the declared order over mixed English variants of distinct grants', () => { + // The closest positive: both candidates are ranked, so this is a preference, not ambiguity. + const buttons = [button('Allow Once'), button('Allow')]; + expect(selectRankedAlertButtonByRole(buttons, 'accept')?.label).toBe('Allow'); + }); + + it('returns null when no button carries the role', () => { + expect(selectRankedAlertButtonByRole([button('Settings')], 'accept')).toBeNull(); + }); + + it('orders candidates the label table cannot read by the backend rank it supplies', () => { + // The localized-sheet shape at the shared boundary: both labels are outside the vocabulary, + // and the backend's own fact (here, a grant-id rank) is the only order the pair has. The + // boundary must use it instead of refusing a pair the owning backend can order. + type RankedButton = Button & { id: string }; + const idRanks: Record = { 'id/fg': 0, 'id/once': 1 }; + const rankById = (button: RankedButton): number | null => idRanks[button.id] ?? null; + const buttons: RankedButton[] = [ + { label: 'Solo esta vez', role: 'accept', id: 'id/once' }, + { label: 'Mientras la app est\u00e1 en uso', role: 'accept', id: 'id/fg' }, + ]; + expect(selectRankedAlertButtonByRole(buttons, 'accept', rankById)?.id).toBe('id/fg'); + }); + + it('still refuses when the backend rank reaches neither of a multi-candidate pair', () => { + // The closest negative: the ranker exists but names nothing for these buttons, so the + // label fallback leaves an unranked candidate and the refusal stands. + type RankedButton = Button & { id: string }; + const buttons: RankedButton[] = [ + { label: 'Mientras la app est\u00e1 en uso', role: 'accept', id: 'vendor/a' }, + { label: 'Solo esta vez', role: 'accept', id: 'vendor/b' }, + ]; + expect(selectRankedAlertButtonByRole(buttons, 'accept', () => null)).toBeNull(); + }); + + it('lets the backend rank decide a pair the label table would order the other way', () => { + // Both sources know these labels, and they conflict: the label table prefers + // "While using the app" (position 1) over "Only this time" (position 15), while the + // backend ranks them 2 and 0. The owning backend's fact is the authority — the label + // table ranks only where the backend declares nothing. + type RankedButton = Button & { id: string }; + const idRanks: Record = { 'id/once': 0, 'id/fg': 2 }; + const rankById = (button: RankedButton): number | null => idRanks[button.id] ?? null; + const buttons: RankedButton[] = [ + { label: 'While using the app', role: 'accept', id: 'id/fg' }, + { label: 'Only this time', role: 'accept', id: 'id/once' }, + ]; + expect(selectRankedAlertButtonByRole(buttons, 'accept', rankById)?.id).toBe('id/once'); + }); + + it('refuses a pair whose ranks come from different sources rather than comparing scales', () => { + // A backend id-rank and a label-table position are numbers on unrelated scales: id-rank 2 + // (a one-time grant) says nothing against label position 3 ("Always allow"). The pair is + // not orderable by any table, so it is refused, never resolved by whichever number is + // smaller. + type RankedButton = Button & { id: string }; + const rankById = (button: RankedButton): number | null => (button.id === 'id/once' ? 2 : null); + const buttons: RankedButton[] = [ + { label: 'Only this time', role: 'accept', id: 'id/once' }, + { label: 'Always allow', role: 'accept', id: 'no/rank' }, + ]; + expect(selectRankedAlertButtonByRole(buttons, 'accept', rankById)).toBeNull(); + }); +}); diff --git a/packages/contracts/src/alert-contract.ts b/packages/contracts/src/alert-contract.ts index b7af81b93a..9a8c29d227 100644 --- a/packages/contracts/src/alert-contract.ts +++ b/packages/contracts/src/alert-contract.ts @@ -2,12 +2,23 @@ export const ALERT_POLL_INTERVAL_MS = 300; export const DEFAULT_ALERT_TIMEOUT_MS = 10_000; export const ALERT_ACTION_RETRY_MS = 2_000; +/** + * The largest alert budget the legs accept. The window signal rides `AbortSignal.timeout`, whose + * Node ceiling is 2^31-1 ms: larger values are rejected outright there, and its underlying timer + * overflows to a 1 ms firing when merely exceeded. The daemon refuses a larger `alert wait` + * budget as `INVALID_ARGS`, and a tree driver reads one as the bounded default — never a + * silently shortened window. + */ +export const MAX_ALERT_WINDOW_MS = 2_147_483_647; + /** * The one alert failure the family retries on: the backend looked and there was no alert *yet*. * Both Apple backends state it in typed form — the XCTest runner as this `ErrorPayload.code` * (surfacing as `details.runnerErrorCode`, like `RUNNER_BUSY`, without changing the wire error - * code), the macOS helper as `details.reason`. Retry and the fallback hint key on these, never on - * the message text: a transport, runner or helper failure must never read as an absent alert. + * code), the macOS helper as details.reason — and so do the tree-based acting legs (Android and + * Limrun iOS, through the shared driver), as this `details.reason` paired with `dispatched: + * 'no'`. Retry and the fallback hint key on these, never on the message text: a transport, + * runner or helper failure must never read as an absent alert. */ export const ALERT_NOT_FOUND_RUNNER_CODE = 'ALERT_NOT_FOUND'; export const ALERT_NOT_FOUND_REASON = 'alert-not-found'; @@ -20,6 +31,152 @@ export const ALERT_NOT_FOUND_REASON = 'alert-not-found'; */ export const ALERT_BUTTON_NOT_FOUND_RUNNER_CODE = 'ALERT_BUTTON_NOT_FOUND'; +/** + * The acting leg's confirm budget ran out after a press landed without settling whether the + * alert is gone: the dialog was seen standing again after the press ("did not dismiss"), or no + * post-press read ever resolved to an answer ("could not be confirmed"). Typed so a caller can + * tell "a press went out and the screen never proved it worked" from absence or an + * unresolvable capture. The lookup never answers with it — a budget ends its loop, it does not + * cancel a started capture — so a quiet screen is always `alert-not-found`. + */ +export const ALERT_WINDOW_EXPIRED_REASON = 'alert-window-expired'; + +/** What an `alert get` capture resolved to, typed so clients switch on states, not messages. */ +export type AlertStatusKind = 'visible' | 'none' | 'undecided'; + +export type AlertButtonRole = 'accept' | 'dismiss' | 'neutral'; + +/** + * What one capture from a tree-based alert backend says about alerts. `none` states absence; + * `undecided` states the capture is alert-shaped yet unresolvable (a degraded or shared root, + * competing sheets) and proves nothing — confirming a dismissal may accept `none`, never + * `undecided`. Backends whose capture cannot reach `undecided` report a quiet capture as `none`. + */ +export type AlertCapture = + | { kind: 'alert'; candidate: TCandidate } + | { kind: 'none' } + | { kind: 'undecided' }; + +/** + * The one alert button label vocabulary every tree-based alert backend classifies with, + * declared as a per-role preference table: each entry names one English label (regex only for + * spelling variants of the *same* label), earlier entries outrank later ones, so a sheet + * offering several grants resolves by declared preference ("while using the app" over "only + * this time") instead of tree order. The table ranks labels, not buttons it cannot read: a + * backend owning a fact no label carries — Android's permission-grant ids — supplies its own + * rank per candidate, and the label table ranks only where the backend declares nothing. + * Buttons neither the vocabulary nor a backend rank reaches stay neutral, and acting legs + * refuse on the button-undecided reason rather than guessing at a translation. + */ +const ACCEPT_LABEL_PREFERENCES: readonly RegExp[] = [ + /^allow while using (?:the )?app$/i, + /^while using the app$/i, + /^allow all$/i, + /^always allow$/i, + /^allow all the time$/i, + /^allow notifications$/i, + /^allow$/i, + /^ok$/i, + /^yes$/i, + /^continue$/i, + /^save$/i, + /^confirm$/i, + /^turn on$/i, + /^open settings$/i, + // One-time grants last: a sheet offering a lasting grant should not spend the tap here. + /^allow once$/i, + /^only this time$/i, +]; +// `don.?t allow` covers the straight, missing, and typographic apostrophe in one alternate. +const DISMISS_LABEL_PREFERENCES: readonly RegExp[] = [ + /^cancel$/i, + /^deny$/i, + /^don.?t allow$/i, + /^not now$/i, + /^no$/i, + /^dismiss$/i, + /^close$/i, + /^close app$/i, + /^later$/i, + /^skip$/i, + /^ask app not to track$/i, +]; + +const LABEL_PREFERENCES: Record<'accept' | 'dismiss', readonly RegExp[]> = { + accept: ACCEPT_LABEL_PREFERENCES, + dismiss: DISMISS_LABEL_PREFERENCES, +}; + +/** One button's action role under the shared vocabulary; anything unnamed is neutral. */ +export function classifyAlertButtonRole(label: string): AlertButtonRole { + const trimmed = label.trim(); + if (ACCEPT_LABEL_PREFERENCES.some((pattern) => pattern.test(trimmed))) return 'accept'; + if (DISMISS_LABEL_PREFERENCES.some((pattern) => pattern.test(trimmed))) return 'dismiss'; + return 'neutral'; +} + +/** + * A role-named button's place in the declared preference table, or null when its label carries + * no vocabulary entry (localized grants, unnamed choices). + */ +function alertButtonPreferenceRank(label: string, role: 'accept' | 'dismiss'): number | null { + const index = LABEL_PREFERENCES[role].findIndex((pattern) => pattern.test(label.trim())); + return index === -1 ? null : index; +} + +/** + * The one choice boundary for a shared choice: pick the action's role candidates by declared + * preference. Each candidate's rank comes from the owning backend first — a backend may own + * facts a label cannot carry, like Android's permission-grant ids — and from the label + * vocabulary only where the backend declares nothing. A single candidate wins outright. + * Several resolve only when every one of them carries a rank AND all ranks come from the same + * source: backend ids and label positions are different scales, and an id-rank 2 (a one-time + * grant) says nothing against a label-position 3 ("Always allow"). A mix of sources cannot be + * ordered by any table, so it names no single physical button and is refused rather than + * resolved by tree order or by one side's number happening to be smaller. + */ +export function selectRankedAlertButtonByRole< + TButton extends Readonly<{ role: AlertButtonRole; label: string }>, +>( + buttons: readonly TButton[], + role: 'accept' | 'dismiss', + backendRank?: (button: TButton) => number | null, +): TButton | null { + const matching = buttons.filter((button) => button.role === role); + if (matching.length === 1) return matching[0] ?? null; + if (matching.length === 0) return null; + const ranked = matching.map((button) => { + const fromBackend = backendRank?.(button) ?? null; + return { + button, + rank: fromBackend ?? alertButtonPreferenceRank(button.label, role), + source: fromBackend !== null ? ('backend' as const) : ('label' as const), + }; + }); + if (ranked.some((entry) => entry.rank === null)) return null; + if (ranked.some((entry) => entry.source !== ranked[0]?.source)) return null; + ranked.sort((a, b) => (a.rank ?? 0) - (b.rank ?? 0)); + const [first, second] = ranked; + if (!first || (second && second.rank === first.rank)) return null; + return first.button; +} + +/** + * The reason a tree-based acting leg (`alert accept|dismiss`) refuses after seeing an alert: the + * visible buttons name no single choice for the asked-for action. Callers branch on + * `error.details.reason` instead of the prose; `dispatched: 'no'` states the refusal ran before + * any press. + */ +export const ALERT_BUTTON_UNDECIDED_REASON = 'alert_button_undecided'; + +/** + * The reason a tree-based acting leg refuses before it has a candidate at all: the capture was + * alert-shaped but unresolvable (degraded buttons, competing sheets, an unverifiable owner). + * This is not absence — reporting `alert-not-found` over it would state more than the capture + * supports — and confirming a dismissal must never accept such a capture as proof of absence. + */ +export const ALERT_CAPTURE_UNDECIDED_REASON = 'alert_capture_undecided'; + export const ALERT_ACTIONS = ['get', 'accept', 'dismiss', 'wait'] as const; export type AlertAction = (typeof ALERT_ACTIONS)[number]; diff --git a/packages/contracts/src/alert-runtime.test.ts b/packages/contracts/src/alert-runtime.test.ts index d0a5dab34d..87629b4d7e 100644 --- a/packages/contracts/src/alert-runtime.test.ts +++ b/packages/contracts/src/alert-runtime.test.ts @@ -58,6 +58,9 @@ test('each local leg forwards the window and the session target to its own owner timeoutMs: 37, appBundleId: 'com.example.app', surface: 'app', + // The request's cancellation rides along: owners whose transport cannot abort a call (the + // Limrun tree legs) still refuse to consume a capture once the request is canceled. + signal, }; expect(legs.readAlert).toHaveBeenCalledWith(expectedOptions); expect(legs.awaitAlert).toHaveBeenCalledWith(expectedOptions); @@ -76,12 +79,13 @@ test('an absent target field never reaches the owner as an explicit undefined', const readAlert = vi.fn(async (_options?: AlertInteractorOptions) => ({})); const resolveInteractor = async () => ({ readAlert }) as unknown as Interactor; const source = localInteractorSource({ device, resolveInteractor }); - const operations = bindAlertLeg('readAlert', new AbortController().signal, source); + const signal = new AbortController().signal; + const operations = bindAlertLeg('readAlert', signal, source); await operations.readAlert({ surface: 'frontmost-app' }); - expect(readAlert).toHaveBeenCalledWith({ surface: 'frontmost-app' }); - expect(Object.keys(readAlert.mock.calls[0]?.[0] ?? {})).toEqual(['surface']); + expect(readAlert).toHaveBeenCalledWith({ surface: 'frontmost-app', signal }); + expect(Object.keys(readAlert.mock.calls[0]?.[0] ?? {})).toEqual(['surface', 'signal']); }); // Facts admitted the leg, so an interactor without it is an ownership bug the caller must see diff --git a/packages/contracts/src/alert-runtime.ts b/packages/contracts/src/alert-runtime.ts index b3879cd205..c0e69cc660 100644 --- a/packages/contracts/src/alert-runtime.ts +++ b/packages/contracts/src/alert-runtime.ts @@ -94,11 +94,15 @@ async function resolveAlertInteractor( }); } -function alertInteractorOptions(input: AlertRuntimeInput): AlertInteractorOptions { +function alertInteractorOptions( + input: AlertRuntimeInput, + signal: AbortSignal | undefined, +): AlertInteractorOptions { return { ...(input.timeoutMs === undefined ? {} : { timeoutMs: input.timeoutMs }), ...(input.appBundleId === undefined ? {} : { appBundleId: input.appBundleId }), ...(input.surface === undefined ? {} : { surface: input.surface }), + ...(signal ? { signal } : {}), }; } @@ -124,7 +128,7 @@ export function bindAlertLeg( // held at its merge-base size (`eager-closure-budgets`); a static edge would grow it. const { requireInteractorMethod } = await import('./interactor-operation-binding.ts'); const method = requireInteractorMethod(interactor[leg], ALERT_LEG_LABELS[leg]); - return await method.call(interactor, alertInteractorOptions(input)); + return await method.call(interactor, alertInteractorOptions(input, signal)); }, }) as Readonly Promise>>>; } diff --git a/packages/contracts/src/interactor-types.ts b/packages/contracts/src/interactor-types.ts index c4b6381db4..d5cf0a7343 100644 --- a/packages/contracts/src/interactor-types.ts +++ b/packages/contracts/src/interactor-types.ts @@ -474,6 +474,8 @@ export type Interactor = { */ export type AlertInteractorOptions = { timeoutMs?: number; + /** The daemon request's cancellation; owners whose transport can abort thread it into the legs. */ + signal?: AbortSignal; appBundleId?: string; surface?: SessionSurface; }; diff --git a/packages/daemon/src/execution/handlers/__tests__/snapshot-alert.test.ts b/packages/daemon/src/execution/handlers/__tests__/snapshot-alert.test.ts index f30c9788e7..8cef94a0fb 100644 --- a/packages/daemon/src/execution/handlers/__tests__/snapshot-alert.test.ts +++ b/packages/daemon/src/execution/handlers/__tests__/snapshot-alert.test.ts @@ -140,3 +140,34 @@ for (const action of ['accept', 'dismiss']) { expect(successor?.actions).toEqual([]); }); } + +test('alert wait forwards its timeout budget to the runtime leg', async () => { + const sessionStore = makeSessionStore(); + const sessionName = 'ios-sim'; + sessionStore.publish(sessionName, makeSession(sessionName)); + + await handleSnapshotCommands(sessionName, sessionStore, ['wait', '1500']); + + // The ceiling test below proves the refusal; this proves a legal budget crosses the runtime + // seam whole. (The Apple runner leg reads the state as `get` and waits on its own deadline.) + expect(mockRunnerCommand.mock.calls[0]?.[1]).toMatchObject({ + command: 'alert', + timeoutMs: 1_500, + }); +}); + +test('alert wait refuses a budget past the window ceiling instead of arming a misfired timer', async () => { + // Above 2^31-1 ms Node's timer overflows: `AbortSignal.timeout` throws outright and a bare + // setTimeout fires after 1 ms. The refusal must be the typed argument error, never either. + const sessionStore = makeSessionStore(); + const sessionName = 'ios-sim'; + sessionStore.publish(sessionName, makeSession(sessionName)); + + await expect( + handleSnapshotCommands(sessionName, sessionStore, ['wait', '4294967296']), + ).rejects.toThrow(/exceeds the maximum/); + await expect( + handleSnapshotCommands(sessionName, sessionStore, ['wait', '2147483648']), + ).rejects.toThrow(/exceeds the maximum/); + expect(mockRunnerCommand).not.toHaveBeenCalled(); +}); diff --git a/packages/daemon/src/execution/handlers/snapshot-alert.ts b/packages/daemon/src/execution/handlers/snapshot-alert.ts index 3c67413983..0e9e4f87bf 100644 --- a/packages/daemon/src/execution/handlers/snapshot-alert.ts +++ b/packages/daemon/src/execution/handlers/snapshot-alert.ts @@ -3,7 +3,9 @@ import type { AlertRuntimeInput } from '@agent-device/contracts/alert-runtime'; import { type AlertAction, DEFAULT_ALERT_TIMEOUT_MS as DEFAULT_TIMEOUT_MS, + MAX_ALERT_WINDOW_MS, } from '@agent-device/contracts/alert-contract'; +import { AppError } from '@agent-device/kernel/errors'; import type { BoundDeviceRuntime } from '@agent-device/contracts/platform-runtime'; import { alertAcceptUse, @@ -151,7 +153,7 @@ export async function handleAlertCommand( session?.trace?.outPath, ); const data = await bound.execute({ - timeoutMs: parseTimeout(req.positionals?.[1]) ?? DEFAULT_TIMEOUT_MS, + timeoutMs: alertWaitTimeoutMs(action, req.positionals?.[1]), ...alertTarget(session), execution: runtimeExecutionFromContext(context), }); @@ -179,3 +181,21 @@ function normalizeAlertAction(action: string | undefined): AlertAction { if (action === 'accept' || action === 'dismiss' || action === 'wait') return action; return 'get'; } + +/** + * The `alert wait` budget before it reaches any leg. Past the contract's window ceiling the + * budget cannot be armed as a timer at all, so the command refuses the argument instead of + * starting a wait that would misfire; the driver's own default remains the last line for + * budgets that arrive by other routes. + */ +function alertWaitTimeoutMs(action: AlertAction, rawTimeout: string | undefined): number { + const timeoutMs = parseTimeout(rawTimeout) ?? DEFAULT_TIMEOUT_MS; + if (action === 'wait' && timeoutMs > MAX_ALERT_WINDOW_MS) { + throw new AppError( + 'INVALID_ARGS', + `alert wait timeoutMs exceeds the maximum of ${MAX_ALERT_WINDOW_MS} ms`, + { hint: 'Pass a shorter timeout, or run repeated alert waits.' }, + ); + } + return timeoutMs; +} diff --git a/packages/platform-android/src/__tests__/alert-detection.test.ts b/packages/platform-android/src/__tests__/alert-detection.test.ts index 126cdc8a8d..ef7f8f7465 100644 --- a/packages/platform-android/src/__tests__/alert-detection.test.ts +++ b/packages/platform-android/src/__tests__/alert-detection.test.ts @@ -58,3 +58,214 @@ test('chooseAndroidAlertButton accepts a single neutral button', () => { assert.equal(chooseAndroidAlertButton(candidate?.buttons ?? [], 'accept')?.label, 'Later'); }); + +test('the shared vocabulary classifies iOS variant grants on Android trees too', () => { + // One label vocabulary: the same label states the same role on every tree backend, so + // `alert accept` can act on the same button on both platforms. Non-button ids keep the + // classification on the labels rather than the platform id map. + const candidate = findAndroidAlertCandidate([ + node(0, 'android.app.AlertDialog'), + text(1, 'Location access', 'android:id/alertTitle'), + button(2, "Don't allow", 'com.example:id/deny', { x: 52, y: 612 }), + button(3, 'Allow while using the app', 'com.example:id/grant', { x: 210, y: 612 }), + ]); + + assert.equal( + chooseAndroidAlertButton(candidate?.buttons ?? [], 'accept')?.label, + 'Allow while using the app', + ); + assert.equal(chooseAndroidAlertButton(candidate?.buttons ?? [], 'dismiss')?.label, "Don't allow"); +}); + +test('the Android 11+ permission sheet resolves its two grants by the preference table', () => { + // The shipped Android 11+ shape: permission_allow_foreground_only_button and + // permission_allow_one_time_button both classify accept by id, so the shared ranked choice + // (not uniqueness) must pick the lasting grant; main pressed it too, so a refusal here + // regresses a working route. + const candidate = findAndroidAlertCandidate([ + node(0, 'android.widget.FrameLayout', { + bundleId: 'com.google.android.permissioncontroller', + }), + text( + 1, + 'Allow Probe to access location in the background?', + 'com.android.permissioncontroller:id/permission_message', + ), + button( + 2, + 'While using the app', + 'permission_allow_foreground_only_button', + { x: 32, y: 1420 }, + true, + ), + button(3, 'Only this time', 'permission_allow_one_time_button', { x: 32, y: 1490 }, true), + button(4, 'Deny', 'permission_deny_button', { x: 32, y: 1560 }, true), + ]); + + assert.equal( + chooseAndroidAlertButton(candidate?.buttons ?? [], 'accept')?.label, + 'While using the app', + ); + assert.equal(chooseAndroidAlertButton(candidate?.buttons ?? [], 'dismiss')?.label, 'Deny'); +}); + +test('a localized Android 11+ sheet resolves its grants by the id-derived rank', () => { + // The maintainer's localized-sheet regression: on a Spanish device neither label is in the + // English vocabulary, so the label table alone refuses — where main pressed the first + // permission_allow id. The ids carry the rank: the foreground-only grant outranks the + // one-time grant whatever the display strings say, and the deny id still answers `dismiss`. + const candidate = findAndroidAlertCandidate([ + node(0, 'android.widget.FrameLayout', { + bundleId: 'com.google.android.permissioncontroller', + }), + text( + 1, + '¿Permitir que Probe acceda a la ubicación?', + 'com.android.permissioncontroller:id/permission_message', + ), + button( + 2, + 'Mientras la app está en uso', + 'permission_allow_foreground_only_button', + { x: 32, y: 1420 }, + true, + ), + button(3, 'Solo esta vez', 'permission_allow_one_time_button', { x: 32, y: 1490 }, true), + button(4, 'No permitir', 'permission_deny_button', { x: 32, y: 1560 }, true), + ]); + + assert.equal( + chooseAndroidAlertButton(candidate?.buttons ?? [], 'accept')?.label, + 'Mientras la app está en uso', + ); + assert.equal(chooseAndroidAlertButton(candidate?.buttons ?? [], 'dismiss')?.label, 'No permitir'); +}); + +test('the Android 10 sheet grants foreground-only by its id', () => { + // Android 10's copy "Allow only while using the app" is outside the vocabulary table; its + // resource id is the same foreground-only grant id the id-classifier already reads, so the + // backend rank resolves the pair the labels cannot order. + const candidate = findAndroidAlertCandidate([ + node(0, 'android.widget.FrameLayout', { + bundleId: 'com.android.permissioncontroller', + }), + text( + 1, + 'Allow Probe to access this device\u2019s location?', + 'com.android.permissioncontroller:id/permission_message', + ), + button( + 2, + 'Allow only while using the app', + 'permission_allow_foreground_only_button', + { x: 32, y: 1420 }, + true, + ), + button(3, 'Allow', 'permission_allow_one_time_button', { x: 32, y: 1490 }, true), + button(4, "Don't allow", 'permission_deny_button', { x: 32, y: 1560 }, true), + ]); + + assert.equal( + chooseAndroidAlertButton(candidate?.buttons ?? [], 'accept')?.label, + 'Allow only while using the app', + ); + assert.equal(chooseAndroidAlertButton(candidate?.buttons ?? [], 'dismiss')?.label, "Don't allow"); +}); + +test('a grant id outside the declared ranks keeps the refusal against a known label', () => { + // The closest negative to the id-rank positives: a vendor fork's own allow id is a real + // grant (the id classifier reads `permission_allow*`), yet no declared rank reaches it. + // Letting the one English label win would press an arbitrary grant of an unordered pair — + // and the neutral-button fallback must not aim the tap at it either — so both acting + // choices refuse while the deny id still answers `dismiss`. + const candidate = findAndroidAlertCandidate([ + node(0, 'android.widget.FrameLayout', { + bundleId: 'com.google.android.permissioncontroller', + }), + text( + 1, + 'Allow Probe to access contacts?', + 'com.android.permissioncontroller:id/permission_message', + ), + button(2, 'Allow', 'permission_allow_button', { x: 32, y: 1420 }, true), + // A vendor fork's own id, written as the vendor ships it: the fixture does not prefix it + // with the platform controller's id home (that prefix is what makes a real controller id), + // and the identifier pattern still reads it as a permission node. + button( + 3, + 'Always allow and sync', + 'com.vendor.permission:id/permission_allow_always_and_sync_button', + { x: 32, y: 1490 }, + false, + ), + button(4, 'Deny', 'permission_deny_button', { x: 32, y: 1560 }, true), + ]); + + assert.equal(chooseAndroidAlertButton(candidate?.buttons ?? [], 'accept'), null); + assert.equal(chooseAndroidAlertButton(candidate?.buttons ?? [], 'dismiss')?.label, 'Deny'); +}); + +test('two points carrying one grant id refuse instead of resolving by tree order', () => { + // Same physical rank twice — the id-rank twin of the same-label tie: the capture names the + // grant at two centers and no single one of them. + const candidate = findAndroidAlertCandidate([ + node(0, 'android.widget.FrameLayout', { + bundleId: 'com.google.android.permissioncontroller', + }), + text( + 1, + 'Allow Probe to access contacts?', + 'com.android.permissioncontroller:id/permission_message', + ), + button( + 2, + 'While using the app', + 'permission_allow_foreground_only_button', + { x: 32, y: 1420 }, + true, + ), + button( + 3, + 'While using the app', + 'permission_allow_foreground_only_button', + { x: 32, y: 1640 }, + true, + ), + button(4, 'Deny', 'permission_deny_button', { x: 32, y: 1560 }, true), + ]); + + assert.equal(chooseAndroidAlertButton(candidate?.buttons ?? [], 'accept'), null); + assert.equal(chooseAndroidAlertButton(candidate?.buttons ?? [], 'dismiss')?.label, 'Deny'); +}); + +test('nested nodes over one button center deduplicate to one candidate', () => { + // uiautomator commonly nests the clickable wrapper and the label node at the same center. + // They are one physical button, and collapsing them must not read as ambiguity. + const candidate = findAndroidAlertCandidate([ + node(0, 'android.app.AlertDialog'), + text(1, 'Discard draft?', 'android:id/alertTitle'), + button(2, 'OK', 'android:id/button1', { x: 52, y: 612 }), + node(3, 'android.widget.TextView', { + label: 'OK', + parentIndex: 2, + rect: { x: 52, y: 612, width: 128, height: 52 }, + }), + ]); + + assert.deepEqual(candidate?.alert.buttons, ['OK']); + assert.equal(chooseAndroidAlertButton(candidate?.buttons ?? [], 'accept')?.label, 'OK'); +}); + +test('the same label at two distinct centers keeps two candidates so the choice refuses', () => { + // Two spatially distinct OKs are two physical buttons; the capture names no single one, so + // the shared ranked boundary refuses rather than taking the tree-first candidate. + const candidate = findAndroidAlertCandidate([ + node(0, 'android.app.AlertDialog'), + text(1, 'Which OK?', 'android:id/alertTitle'), + button(2, 'OK', 'android:id/button1', { x: 52, y: 612 }), + button(3, 'OK', 'android:id/button1_copy', { x: 240, y: 612 }), + ]); + + assert.deepEqual(candidate?.alert.buttons, ['OK', 'OK']); + assert.equal(chooseAndroidAlertButton(candidate?.buttons ?? [], 'accept'), null); +}); diff --git a/packages/platform-android/src/__tests__/alert.test.ts b/packages/platform-android/src/__tests__/alert.test.ts index eb6a5aa95c..8f70388091 100644 --- a/packages/platform-android/src/__tests__/alert.test.ts +++ b/packages/platform-android/src/__tests__/alert.test.ts @@ -2,16 +2,21 @@ import { test, vi } from 'vitest'; import assert from 'node:assert/strict'; import type { DeviceInfo } from '@agent-device/kernel/device'; import { deviceShellArgv, type ShellWord } from '@agent-device/kernel/device-shell'; +import { isRequestCanceledError } from '@agent-device/kernel/errors'; +import type { RawSnapshotNode } from '@agent-device/kernel/snapshot'; import { button, node, text } from './alert-fixtures.ts'; -const runAndroidAdb = vi.fn(async (_device: DeviceInfo, _args: readonly string[]) => ({ - exitCode: 0, - stdout: '', - stderr: '', -})); +type FakeShellOptions = { signal?: AbortSignal }; +const runAndroidAdb = vi.fn( + async (_device: DeviceInfo, _args: readonly string[], _options?: FakeShellOptions) => ({ + exitCode: 0, + stdout: '', + stderr: '', + }), +); const runAndroidShell = vi.fn( - async (device: DeviceInfo, words: ShellWord[]) => - await runAndroidAdb(device, deviceShellArgv('adb', 'shell', words)), + async (device: DeviceInfo, words: ShellWord[], options?: FakeShellOptions) => + await runAndroidAdb(device, deviceShellArgv('adb', 'shell', words), options), ); vi.mock('../adb.ts', () => ({ runAndroidAdb, runAndroidShell })); @@ -88,6 +93,39 @@ test('accepting a button alert records the tapped button and its coordinates', a assert.deepEqual('coordinates' in result ? result.coordinates : undefined, { x: 116, y: 638 }); }); +test('accepting the Android 11+ permission sheet taps the preferred grant once', async () => { + runAndroidAdb.mockClear(); + // The regression the maintainer asked for at the action seam: both permission_allow_* ids + // classify accept, and the shared preference table must land one tap on the lasting grant. + const sheet = [ + node(0, 'android.widget.FrameLayout', { + bundleId: 'com.google.android.permissioncontroller', + }), + text( + 1, + 'Allow Probe to access location in the background?', + 'com.android.permissioncontroller:id/permission_message', + ), + button( + 2, + 'While using the app', + 'permission_allow_foreground_only_button', + { x: 32, y: 1420 }, + true, + ), + button(3, 'Only this time', 'permission_allow_one_time_button', { x: 32, y: 1490 }, true), + button(4, 'Deny', 'permission_deny_button', { x: 32, y: 1560 }, true), + ]; + const result = await handleAndroidAlert(device, 'accept', { + captureNodes: dialogUntilPressed(sheet), + }); + + assert.equal(result.kind, 'alertHandled'); + assert.equal(result.kind === 'alertHandled' ? result.button : undefined, 'While using the app'); + assert.deepEqual('coordinates' in result ? result.coordinates : undefined, { x: 96, y: 1446 }); + assert.equal(runAndroidAdb.mock.calls.filter(([, args]) => args.includes('tap')).length, 1); +}); + test('a fallback Back dismissal (no matching button) carries no coordinates', async () => { runAndroidAdb.mockClear(); const result = await handleAndroidAlert(device, 'dismiss', { @@ -147,7 +185,8 @@ test('dismiss fails when the dialog is still visible after the action budget', a (error: unknown) => error instanceof Error && error.message === 'alert dismiss did not dismiss the visible alert' && - (error as { code?: string }).code === 'COMMAND_FAILED', + (error as { code?: string }).code === 'COMMAND_FAILED' && + (error as { details?: { reason?: unknown } }).details?.reason === 'alert-window-expired', ); assert.equal(runAndroidAdb.mock.calls.length, 1); }); @@ -161,3 +200,142 @@ test('wait with a spent budget still reports an alert that is already visible', assert.equal(result.kind, 'alertWait'); assert.equal(result.alert?.title, 'Automation confirmation'); }); + +test('a caller cancel during the press discloses that the tap may or may not have landed', async () => { + // The one dispatched-unknown exit: the budget never stops a press, so the only way a press + // dies mid-flight is the caller's own cancellation. The adb child is killed between the tap's + // dispatch and its rejection, so the device may already have received it — the cancellation + // carries that typed, never a clean 'no'. The fake reacts to its seam signal the way the adb + // executor does — a child killed on abort — and the caller speaks only after the tap is + // verifiably in flight, so the test exercises the mid-press cut, not a pre-aborted shortcut. + runAndroidAdb.mockClear(); + const caller = new AbortController(); + const tapSignals: (AbortSignal | undefined)[] = []; + let markTapStarted!: () => void; + const tapStarted = new Promise((resolve) => { + markTapStarted = resolve; + }); + runAndroidAdb.mockImplementationOnce(async (_device, args, options) => { + if (args.includes('tap')) { + tapSignals.push(options?.signal); + await new Promise((_resolve, reject) => { + markTapStarted(); + options?.signal?.addEventListener('abort', () => reject(new Error('adb killed')), { + once: true, + }); + }); + } + return { exitCode: 0, stdout: '', stderr: '' }; + }); + + const result = handleAndroidAlert(device, 'dismiss', { + signal: caller.signal, + captureNodes: dialogUntilPressed(), + }); + result.catch(() => {}); + await tapStarted; + caller.abort(); + + await assert.rejects( + result, + (error: unknown) => + isRequestCanceledError(error) && + (error as { details?: { dispatched?: unknown } }).details?.dispatched === 'unknown', + ); + assert.equal(tapSignals.length, 1); + assert.equal(tapSignals[0]?.aborted, true); +}); + +test('a quiet screen with captures straddling the lookup answers alert-not-found', async () => { + // The maintainer's budget-rule pin: ~900 ms captures against the 2 s lookup, so the probe + // that starts at 1.2 s cannot answer before the deadline. The deadline ends the loop, it + // does not cancel the started capture, and a window that only ever answered quiet is absence + // — never the old timing-dependent `alert-window-expired` this timeline used to throw at + // the straddling probe. + runAndroidAdb.mockClear(); + const quietWithSlowCaptures = async () => { + await new Promise((resolve) => setTimeout(resolve, 900)); + return []; + }; + + await assert.rejects( + underFakeTimers( + async () => + await handleAndroidAlert(device, 'dismiss', { captureNodes: quietWithSlowCaptures }), + ), + (error: unknown) => + error instanceof Error && + error.message === 'alert not found' && + (error as { details?: { reason?: unknown; dispatched?: unknown } }).details?.reason === + 'alert-not-found' && + (error as { details?: { dispatched?: unknown } }).details?.dispatched === 'no', + ); + assert.equal(runAndroidAdb.mock.calls.length, 0); +}); + +test('the read-first capture rides the caller cancellation and dies with it', async () => { + // Android's transport is cancellable, so the invariant here is the actual child-process cut: + // the capture seam receives the leg's live AbortSignal, and aborting the caller cuts the + // in-flight capture before any press exists. The caller speaks only after the capture is + // verifiably started, so the seam is on the in-flight path rather than a pre-aborted short + // circuit. + runAndroidAdb.mockClear(); + const caller = new AbortController(); + const seamSignals: AbortSignal[] = []; + let rejectCapture: ((error: Error) => void) | undefined; + let markCaptureStarted!: () => void; + const captureStarted = new Promise((resolve) => { + markCaptureStarted = resolve; + }); + const result = handleAndroidAlert(device, 'dismiss', { + signal: caller.signal, + captureNodes: (seam) => + new Promise((_resolve, reject) => { + seamSignals.push(seam.signal); + rejectCapture = reject; + markCaptureStarted(); + seam.signal.addEventListener('abort', () => reject(new Error('adb killed')), { + once: true, + }); + }), + }); + result.catch(() => {}); + await captureStarted; + caller.abort(); + rejectCapture?.(new Error('adb killed')); + + await assert.rejects( + result, + (error: unknown) => + isRequestCanceledError(error) && + (error as { details?: { dispatched?: unknown } }).details?.dispatched === 'no', + ); + assert.equal(seamSignals.length, 1); + assert.equal(seamSignals[0]?.aborted, true); + assert.equal(runAndroidAdb.mock.calls.length, 0); +}); + +test('a slow first read is never cut off by its own action budget', async () => { + // The Android smoke timeline: the session's first capture pays the snapshot-helper cold + // start (install + session bring-up), landing well past the 2s action budget. Under the + // budget rule the deadline only ends the polling loop — every probe that started inside the + // window finishes and its answer counts — so this read resolves the alert and the press + // rides only the caller's cancellation. Before the rule, this was the executed CI failure: + // `alert-window-expired, dispatched: no` with the dialog plainly on screen. + runAndroidAdb.mockClear(); + let reads = 0; + const slowWarmRead = async () => { + reads += 1; + if (reads > 1) return runAndroidAdb.mock.calls.length === 0 ? dialog : []; + // 2.5 driven seconds: past the 2s lookup window the old guarded first probe enforced. + await new Promise((resolve) => setTimeout(resolve, 2_500)); + return dialog; + }; + + const result = await underFakeTimers( + async () => await handleAndroidAlert(device, 'dismiss', { captureNodes: slowWarmRead }), + ); + + assert.equal(result.kind, 'alertHandled'); + assert.equal(runAndroidAdb.mock.calls.length, 1); +}); diff --git a/packages/platform-android/src/__tests__/input-actions.test.ts b/packages/platform-android/src/__tests__/input-actions.test.ts index 35261477c9..1d9494bfb5 100644 --- a/packages/platform-android/src/__tests__/input-actions.test.ts +++ b/packages/platform-android/src/__tests__/input-actions.test.ts @@ -4,12 +4,14 @@ import { backAndroid, homeAndroid, longPressAndroid, + pressAndroid, pressAndroidEnter, pressAndroidTvRemote, probeAndroidDisplayRotation, scrollAndroid, setAndroidOrientation, } from '../input-actions.ts'; +import { AppError } from '@agent-device/kernel/errors'; import { ANDROID_EMULATOR } from './test-utils/device-fixtures.ts'; import { withFakeAdb } from './test-utils/fake-adb.ts'; import { withAndroidAdbProvider } from '../adb-executor.ts'; @@ -371,6 +373,41 @@ test('pressAndroidEnter presses the ENTER keyevent', async () => { ); }); +test('pressAndroid surfaces a failed tap as a typed dispatch error, not a false success', async () => { + // The device route raises the failed `adb shell input tap` as its own AppError; the input + // seam must hand that through with dispatch evidence, never as a tap that happened. + await withFakeAdb( + (args) => + args[1] === 'input' + ? { stdout: '', stderr: 'INJECT_EVENTS security exception', exitCode: 1 } + : undefined, + async ({ device }) => { + await assert.rejects(pressAndroid(device, 100, 200), (error: unknown) => { + assert.ok(error instanceof AppError); + assert.equal(error.code, 'COMMAND_FAILED'); + assert.equal(error.details?.dispatched, 'unknown'); + assert.equal(error.details?.exitCode, 1); + return true; + }); + }, + ); +}); + +test('pressAndroid dispatches no failure suppression even when one is smuggled past the type', async () => { + // The option type is the interface; an untyped caller cannot widen it back. The tap still + // fails typed: `allowFailure` never reaches the transport. + await withFakeAdb( + (args) => + args[1] === 'input' ? { stdout: '', stderr: 'adb: device offline', exitCode: 1 } : undefined, + async ({ device }) => { + await assert.rejects( + pressAndroid(device, 100, 200, { allowFailure: true } as { signal?: AbortSignal }), + (error: unknown) => error instanceof AppError && error.code === 'COMMAND_FAILED', + ); + }, + ); +}); + // The orientation settle polls at its own interval; the clock is the assertion, not the wait. vi.mock('@agent-device/host-kit/retry', () => ({ sleep: async () => {} })); diff --git a/packages/platform-android/src/alert-detection.ts b/packages/platform-android/src/alert-detection.ts index 89f66a3925..9d99be531a 100644 --- a/packages/platform-android/src/alert-detection.ts +++ b/packages/platform-android/src/alert-detection.ts @@ -1,14 +1,20 @@ import { centerOfRect, type RawSnapshotNode } from '@agent-device/kernel/snapshot'; -import type { AlertInfo, AlertSource } from '@agent-device/contracts/alert-contract'; +import { + type AlertButtonRole, + classifyAlertButtonRole, + type AlertInfo, + type AlertSource, + selectRankedAlertButtonByRole, +} from '@agent-device/contracts/alert-contract'; import { isSnapshotNodeInteractionBlocked } from '@agent-device/capture-kit/snapshot-occlusion'; -type AndroidAlertButtonRole = 'accept' | 'dismiss' | 'neutral'; - export type AndroidAlertButton = { label: string; x: number; y: number; - role: AndroidAlertButtonRole; + role: AlertButtonRole; + /** The node's resource id — the backend-owned fact that can rank a grant's role. */ + identifier: string; }; type AndroidAlertSource = Extract; @@ -43,10 +49,6 @@ const ANDROID_ALERT_BUTTON_ID_PATTERN = /^android:id\/button[123]$/i; const ANDROID_PERMISSION_ID_PATTERN = /(?:^|:)id\/permission_/i; const ANDROID_BLOCKING_DIALOG_PATTERN = /\b(?:is(?:n't| not) responding|keeps stopping|has stopped|close app|app info)\b/i; -const ACCEPT_LABEL_PATTERN = - /^(?:ok|allow|allow all|while using the app|only this time|yes|continue|save|confirm|turn on|open settings)$/i; -const DISMISS_LABEL_PATTERN = - /^(?:cancel|deny|don.t allow|don’t allow|not now|no|dismiss|close|close app|later|skip)$/i; export function classifyAndroidAlertIdentifier( identifier: string | null, @@ -85,12 +87,22 @@ function isReachableAlertNode(node: RawSnapshotNode): boolean { return !isSnapshotNodeInteractionBlocked(node); } +/** + * Android's acting policy on the shared choice boundary: the action's role-named button chosen + * by declared preference — the permissioncontroller grant ids first (foreground-only over + * always over one-time, whatever language their labels are in), the shared label vocabulary + * where no grant id exists; else, for dismiss, the first neutral button (Android dialogs + * commonly name the leaving path only neutrally); else, for accept, a single-button dialog's + * only affirmative path. Two choices that tie on one rank (the same label or the same grant id + * at two points) or a candidate no rank source reaches refuse at the shared boundary rather + * than resolving by tree order — the driver turns that refusal into a Back key for dismiss, or + * a typed refusal. + */ export function chooseAndroidAlertButton( buttons: AndroidAlertButton[], action: 'accept' | 'dismiss', ): AndroidAlertButton | null { - const role = action === 'accept' ? 'accept' : 'dismiss'; - const exact = buttons.find((button) => button.role === role); + const exact = selectRankedAlertButtonByRole(buttons, action, rankAndroidAlertButtonById); if (exact) return exact; if (action === 'dismiss') { return buttons.find((button) => button.role === 'neutral') ?? null; @@ -99,6 +111,29 @@ export function chooseAndroidAlertButton( return buttons.length === 1 ? (buttons[0] ?? null) : null; } +/** + * The permissioncontroller grant ids in the order Android's own dialog prefers them: a lasting + * foreground grant over an always grant over a one-time grant. Earlier entries outrank later + * ones; a lower number wins. These ids reach production on every grant sheet: they are the same + * `permission_allow` ids the platform-id classifier already reads (and the ids main tapped), + * stable across locales while their labels are not. + */ +const ANDROID_GRANT_ID_RANKS: readonly RegExp[] = [ + /(?:^|:)id\/permission_allow_foreground_only(?:_button)?$/i, + /(?:^|:)id\/permission_allow(?:_all_time|_always)?(?:_button)?$/i, + /(?:^|:)id\/permission_allow_one_time(?:_button)?$/i, +]; + +/** + * The backend-owned rank of a button named by a grant id — the label-independent preference the + * resource id states. Deny ids carry no declared order (their labels rank them), and any other + * id ranks nothing here. + */ +function rankAndroidAlertButtonById(button: AndroidAlertButton): number | null { + const index = ANDROID_GRANT_ID_RANKS.findIndex((pattern) => pattern.test(button.identifier)); + return index === -1 ? null : index; +} + function findAndroidAlertNodes(nodes: RawSnapshotNode[]): { nodes: RawSnapshotNode[]; source: AndroidAlertInfo['source']; @@ -153,35 +188,39 @@ function findSystemDialogNodes(nodes: RawSnapshotNode[]): RawSnapshotNode[] { } function findAlertButtons(nodes: RawSnapshotNode[]): AndroidAlertButton[] { - const seen = new Set(); + const seenPhysicalButtons = new Set(); const buttons: AndroidAlertButton[] = []; for (const node of nodes) { const label = readNodeText(node); if (!label || !node.rect || !isButtonLike(node)) continue; const normalized = label.trim().toLowerCase(); - if (!normalized || seen.has(normalized)) continue; - seen.add(normalized); + if (!normalized) continue; const point = centerOfRect(node.rect); + // Two nodes over one center are the one physical button (uiautomator commonly nests it); + // the same label at two centers is two buttons, and the shared choice boundary refuses — + // collapsing them here would hand the first one back as if the capture had named it. + const physicalButton = `${normalized}|${point.x}|${point.y}`; + if (seenPhysicalButtons.has(physicalButton)) continue; + seenPhysicalButtons.add(physicalButton); buttons.push({ label, x: point.x, y: point.y, role: classifyAndroidAlertButton(node, label), + identifier: node.identifier ?? '', }); } return buttons; } -function classifyAndroidAlertButton(node: RawSnapshotNode, label: string): AndroidAlertButtonRole { +function classifyAndroidAlertButton(node: RawSnapshotNode, label: string): AlertButtonRole { const identifier = node.identifier ?? ''; const roleFromId = classifyAndroidAlertButtonById(identifier); if (roleFromId) return roleFromId; - if (ACCEPT_LABEL_PATTERN.test(label.trim())) return 'accept'; - if (DISMISS_LABEL_PATTERN.test(label.trim())) return 'dismiss'; - return 'neutral'; + return classifyAlertButtonRole(label); } -function classifyAndroidAlertButtonById(identifier: string): AndroidAlertButtonRole | null { +function classifyAndroidAlertButtonById(identifier: string): AlertButtonRole | null { if (/(?:^|:)id\/button1$/i.test(identifier)) return 'accept'; if (/(?:^|:)id\/button2$/i.test(identifier)) return 'dismiss'; if (/(?:^|:)id\/button3$/i.test(identifier)) return 'neutral'; diff --git a/packages/platform-android/src/alert.ts b/packages/platform-android/src/alert.ts index 711ba999be..fb23d08cb3 100644 --- a/packages/platform-android/src/alert.ts +++ b/packages/platform-android/src/alert.ts @@ -1,202 +1,83 @@ -import { - type AlertAction, - ALERT_ACTION_RETRY_MS, - ALERT_POLL_INTERVAL_MS, - DEFAULT_ALERT_TIMEOUT_MS, -} from '@agent-device/contracts/alert-contract'; -import { AppError } from '@agent-device/kernel/errors'; import { withDiagnosticTimer } from '@agent-device/host-kit/diagnostics'; -import { Deadline, pollUntilSettled } from '@agent-device/host-kit/retry'; -import { successText } from '@agent-device/kernel/success-text'; - +import type { + TreeAlertDriver, + TreeAlertResult, + TreeAlertSeam, +} from '@agent-device/capture-kit/alert-tree-driver'; import type { DeviceInfo } from '@agent-device/kernel/device'; import type { RawSnapshotNode } from '@agent-device/kernel/snapshot'; +import type { AlertAction } from '@agent-device/contracts/alert-contract'; import { chooseAndroidAlertButton, findAndroidAlertCandidate, + type AndroidAlertButton, type AndroidAlertCandidate, type AndroidAlertInfo, } from './alert-detection.ts'; import { backAndroid, pressAndroid } from './input-actions.ts'; +/** + * The Android alert legs: the shared tree-alert driver (`capture-kit`'s `runTreeAlertAction`, + * the one poll/confirm/same-alert/response owner every tree-based backend rides) plus this + * platform's own observation (`captureNodes` + the candidate reader) and acting policy (the + * id-derived grant rank over the shared label vocabulary, the neutral-button/Back fallback). + * The driver import is function-scoped: a static one would add a new eager module to the + * mechanics façade and break its merge-base closure budget (ADR-0019). + */ + type AndroidAlertOptions = { timeoutMs?: number; - captureNodes: () => Promise; + /** The daemon request's cancellation; the driver hands it to every capture and press seam. */ + signal?: AbortSignal; + captureNodes: (seam: TreeAlertSeam) => Promise; }; -export type AndroidAlertResult = - | { - kind: 'alertStatus'; - platform: 'android'; - action: 'get'; - alert: AndroidAlertInfo | null; - message?: string; - } - | { - kind: 'alertWait'; - platform: 'android'; - action: 'wait'; - alert: AndroidAlertInfo; - waitedMs: number; - message?: string; - } - | { - kind: 'alertHandled'; - platform: 'android'; - action: 'accept' | 'dismiss'; - handled: true; - alert: AndroidAlertInfo; - button: string; - coordinates?: { x: number; y: number }; - message?: string; - }; +export type AndroidAlertResult = TreeAlertResult; export async function handleAndroidAlert( device: DeviceInfo, action: AlertAction, options: AndroidAlertOptions, ): Promise { - if (action === 'wait') { - return await waitForAndroidAlert( - options.captureNodes, - options.timeoutMs ?? DEFAULT_ALERT_TIMEOUT_MS, - ); - } - if (action === 'get') { - const candidate = await readAndroidAlertCandidate(options.captureNodes); - return buildAndroidAlertStatusResponse(candidate?.alert ?? null); - } - return await handleAndroidAlertAction(device, action, options.captureNodes); -} - -async function waitForAndroidAlert( - captureNodes: AndroidAlertOptions['captureNodes'], - timeoutMs: number, -): Promise { - const start = Date.now(); - const candidate = await pollAndroidAlertCandidate(captureNodes, timeoutMs); - if (!candidate) { - throw new AppError('COMMAND_FAILED', 'alert wait timed out'); - } - return { - kind: 'alertWait', - platform: 'android', - action: 'wait', - alert: candidate.alert, - waitedMs: Date.now() - start, - ...successText('Alert visible'), - }; -} - -async function handleAndroidAlertAction( - device: DeviceInfo, - action: 'accept' | 'dismiss', - captureNodes: AndroidAlertOptions['captureNodes'], -): Promise { - const candidate = await pollAndroidAlertCandidate(captureNodes, ALERT_ACTION_RETRY_MS); - if (!candidate) { - throw new AppError('COMMAND_FAILED', 'alert not found', { - hint: 'If a sheet is visible in snapshot but alert reports no alert, it is likely app-owned UI. Use snapshot -i and press the visible label/ref.', - }); - } - - const button = chooseAndroidAlertButton(candidate.buttons, action); - if (button) { - await pressAndroid(device, button.x, button.y); - await confirmAndroidAlertDismissed(candidate.alert, action, captureNodes); - return buildAndroidAlertHandledResponse(action, candidate.alert, button.label, { - x: button.x, - y: button.y, - }); - } - - if (action === 'dismiss') { - await backAndroid(device); - await confirmAndroidAlertDismissed(candidate.alert, action, captureNodes); - return buildAndroidAlertHandledResponse(action, candidate.alert, 'Back'); - } - - throw new AppError('COMMAND_FAILED', 'alert accept found an alert but no accept button', { - alert: candidate.alert, - hint: 'Inspect alert get --json for visible buttons, then use press by visible label/ref if needed.', - }); -} - -/** - * `alert accept|dismiss` means the dialog is gone, not that a button was pressed: the next - * command reads the app, and a capture that lands while the dialog window is still up sees - * only the dialog. A different alert taking its place counts as dismissed. Bounded by the - * same budget the pre-press lookup uses; iOS's runner applies the same re-check. - */ -async function confirmAndroidAlertDismissed( - dismissed: AndroidAlertInfo, - action: 'accept' | 'dismiss', - captureNodes: AndroidAlertOptions['captureNodes'], -): Promise { - const { settled } = await pollUntilSettled({ - probe: () => readAndroidAlertCandidate(captureNodes), - settled: (current) => !current || !sameAndroidAlert(current.alert, dismissed), - deadline: Deadline.fromTimeoutMs(ALERT_ACTION_RETRY_MS), - intervalMs: ALERT_POLL_INTERVAL_MS, - }); - if (settled) return; - throw new AppError('COMMAND_FAILED', `alert ${action} did not dismiss the visible alert`, { - alert: dismissed, - hint: 'The alert button was pressed but the dialog is still visible. Inspect alert get --json, then press the visible button by label/ref or retry.', - }); -} - -function sameAndroidAlert(left: AndroidAlertInfo, right: AndroidAlertInfo): boolean { - return left.title === right.title && left.buttons.join('\u0000') === right.buttons.join('\u0000'); -} - -async function pollAndroidAlertCandidate( - captureNodes: AndroidAlertOptions['captureNodes'], - timeoutMs: number, -): Promise { - const { value } = await pollUntilSettled({ - probe: () => readAndroidAlertCandidate(captureNodes), - settled: (candidate) => candidate !== null, - deadline: Deadline.fromTimeoutMs(timeoutMs), - intervalMs: ALERT_POLL_INTERVAL_MS, - }); - return value; -} - -async function readAndroidAlertCandidate( - captureNodes: AndroidAlertOptions['captureNodes'], -): Promise { - const result = await withDiagnosticTimer('snapshot_capture', captureNodes, { - backend: 'android', - purpose: 'alert', - }); - return findAndroidAlertCandidate(result); -} - -function buildAndroidAlertStatusResponse(alert: AndroidAlertInfo | null): AndroidAlertResult { - return { - kind: 'alertStatus', - platform: 'android', - action: 'get', - alert, - ...(alert ? successText('Alert visible') : successText('No alert visible')), - }; -} - -function buildAndroidAlertHandledResponse( - action: 'accept' | 'dismiss', - alert: AndroidAlertInfo, - button: string, - coordinates?: { x: number; y: number }, -): AndroidAlertResult { - return { - kind: 'alertHandled', + const driver: TreeAlertDriver< + RawSnapshotNode[], + AndroidAlertInfo, + AndroidAlertButton, + AndroidAlertCandidate, + 'android' + > = { platform: 'android', - action, - handled: true, - alert, - button, - ...(coordinates ? { coordinates } : {}), - ...successText(`Alert ${action}ed`), + timeoutMs: options.timeoutMs, + ...(options.signal ? { signal: options.signal } : {}), + capture: (seam) => + withDiagnosticTimer('snapshot_capture', () => options.captureNodes(seam), { + backend: 'android', + purpose: 'alert', + }), + // Android's dump either names a candidate or names none; it has no capture state that can + // reach `undecided`. + findCandidate: (nodes) => { + const candidate = findAndroidAlertCandidate(nodes); + return candidate ? { kind: 'alert', candidate } : { kind: 'none' }; + }, + choose: (candidate, act) => { + const button = chooseAndroidAlertButton(candidate.buttons, act); + if (button) return { kind: 'press', button }; + // No button states any leaving choice: dismissing an Android dialog is what the Back + // key means. The refusal message and typed reason come from the driver, not from here. + return act === 'dismiss' ? { kind: 'fallback', label: 'Back' } : null; + }, + press: async (choice, seam) => { + // The adb executor kills the child on abort, so the press transport stops with the + // caller's cancellation; whether a canceled tap still reached the device is typed + // upstream. The budget never stops a press — a chosen press is the action itself. + if (choice.kind === 'press') { + await pressAndroid(device, choice.button.x, choice.button.y, { signal: seam.signal }); + return; + } + await backAndroid(device, { signal: seam.signal }); + }, }; + const { runTreeAlertAction } = await import('@agent-device/capture-kit/alert-tree-driver'); + return await runTreeAlertAction(action, driver); } diff --git a/packages/platform-android/src/input-actions.ts b/packages/platform-android/src/input-actions.ts index 49fbc0fbbc..b0cebe77e9 100644 --- a/packages/platform-android/src/input-actions.ts +++ b/packages/platform-android/src/input-actions.ts @@ -29,19 +29,40 @@ import { androidAdbResultError, discloseAdbInputDispatch } from './adb-failure.t import { executeAndroidTouchPlan, readAndroidGestureViewportReading } from './touch-executor.ts'; import type { AndroidHelperSessionOptions } from './snapshot-helper-types.ts'; -export async function pressAndroid(device: DeviceInfo, x: number, y: number): Promise { +/** + * What a caller may steer on a dispatched input command: cancellation only. The wider executor + * options stay out on purpose — `allowFailure` would let a dispatch claim success over a failed + * adb command, which no input dispatch may do. + */ +export type AndroidInputDispatchOptions = Readonly<{ signal?: AbortSignal }>; + +export async function pressAndroid( + device: DeviceInfo, + x: number, + y: number, + options?: AndroidInputDispatchOptions, +): Promise { try { - await runAndroidShell(device, ['input', 'tap', x, y]); + // Picked field by field so no runtime caller can smuggle `allowFailure: true` past the type. + await runAndroidShell(device, ['input', 'tap', x, y], { + allowFailure: false, + ...(options?.signal ? { signal: options.signal } : {}), + }); } catch (error) { throw discloseAdbInputDispatch(error); } } /** Two `input tap` sends; a failure of the second follows a tap that already landed. */ -export async function doubleTapAndroid(device: DeviceInfo, x: number, y: number): Promise { - await pressAndroid(device, x, y); +export async function doubleTapAndroid( + device: DeviceInfo, + x: number, + y: number, + options?: AndroidInputDispatchOptions, +): Promise { + await pressAndroid(device, x, y, options); try { - await pressAndroid(device, x, y); + await pressAndroid(device, x, y, options); } catch (error) { throw discloseDispatchAfterSteps(error, 1); } @@ -57,8 +78,15 @@ export async function pressAndroidTvRemote( await runAndroidShell(device, ['input', ...keyeventArgs, keyevent]); } -export async function backAndroid(device: DeviceInfo): Promise { - await runAndroidShell(device, ['input', 'keyevent', '4']); +export async function backAndroid( + device: DeviceInfo, + options?: AndroidInputDispatchOptions, +): Promise { + await runAndroidShell( + device, + ['input', 'keyevent', '4'], + options?.signal ? { signal: options.signal } : undefined, + ); } export async function homeAndroid(device: DeviceInfo): Promise { diff --git a/packages/provider-limrun/src/alert-detection.fixtures.ts b/packages/provider-limrun/src/alert-detection.fixtures.ts new file mode 100644 index 0000000000..ac94c09f67 --- /dev/null +++ b/packages/provider-limrun/src/alert-detection.fixtures.ts @@ -0,0 +1,354 @@ +import type { IosTreeNode } from './snapshot.ts'; + +/** + * Trees shaped from live Limrun iOS instances (issue #3349 evidence). The shapes are what the + * detection rules are actually tested against, so each keeps the provider's real fields: + * `role`/`type` pairs, `AXLabel`, per-node `pid`, and a `frame` dictionary. + * + * The captured discriminator: a system alert arrives as its own `AXApplication` root whose + * `AXLabel` is blank and whose `pid` (and every child's `pid`) is the hosting process, not the + * app's. The app's own `UIAlertController` arrives inside the app's root with the app's `pid`. + */ + +/** The app under test, as the app's own root reports it. */ +export const LIMRUN_APP_PID = 59_012; +/** The process hosting system alerts (SpringBoard's alert agent on the instance). */ +export const LIMRUN_ALERT_HOST_PID = 24_261; +/** SpringBoard, which contributes the status-bar nodes that ride along on app screens. */ +export const LIMRUN_SPRINGBOARD_PID = 33_887; + +export const LIMRUN_ALERT_SCREEN: Readonly<{ width: number; height: number }> = Object.freeze({ + width: 402, + height: 874, +}); + +function alertRoot(children: IosTreeNode[]): IosTreeNode { + return { + type: 'Application', + role: 'AXApplication', + AXLabel: ' ', + pid: LIMRUN_ALERT_HOST_PID, + frame: { x: 0, y: 0, ...LIMRUN_ALERT_SCREEN }, + traits: ['None'], + children, + }; +} + +/** Positive control: the live ATT prompt tree — its own root, host pid, two texts, two buttons. */ +export function limrunAttAlertTree(): IosTreeNode { + return alertRoot([ + { + type: 'StaticText', + role: 'AXStaticText', + AXLabel: 'Allow “ATTProbe” to track your activity across other companies’ apps and websites?', + pid: LIMRUN_ALERT_HOST_PID, + frame: { x: 71, y: 381.33, width: 260, height: 64.33 }, + traits: ['Scrollable', 'StaticText'], + }, + { + type: 'StaticText', + role: 'AXStaticText', + AXLabel: 'This build asks for tracking authorization once, for automation testing.', + pid: LIMRUN_ALERT_HOST_PID, + frame: { x: 71, y: 453.33, width: 260, height: 38 }, + traits: ['Scrollable', 'StaticText'], + }, + { + type: 'Button', + role: 'AXButton', + AXLabel: 'Ask App Not to Track', + pid: LIMRUN_ALERT_HOST_PID, + enabled: true, + frame: { x: 57, y: 511.67, width: 288, height: 48 }, + traits: ['Button', 'Scrollable'], + }, + { + type: 'Button', + role: 'AXButton', + AXLabel: 'Allow', + pid: LIMRUN_ALERT_HOST_PID, + enabled: true, + frame: { x: 57, y: 567.67, width: 288, height: 48 }, + traits: ['Button', 'Scrollable'], + }, + ]); +} + +/** The same alert with its buttons removed: a text sheet is not an actionable alert. */ +export function limrunAlertTreeWithoutButtons(): IosTreeNode { + const root = limrunAttAlertTree(); + return { ...root, children: (root.children ?? []).filter((n) => n.role !== 'AXButton') }; +} + +/** + * The standard iOS location prompt shape: three choices, two of which grant ("Allow Once" and + * "Allow While Using App"). The capture itself never marks a preferred grant; the shared + * preference table does, so `alert accept` presses the lasting grant rather than refusing. + */ +export function limrunLocationAlertTree(): IosTreeNode { + return alertRoot([ + { + type: 'StaticText', + role: 'AXStaticText', + AXLabel: '“Probe” Would Like to Use Your Current Location', + pid: LIMRUN_ALERT_HOST_PID, + frame: { x: 71, y: 381.33, width: 260, height: 64.33 }, + traits: ['Scrollable', 'StaticText'], + }, + { + type: 'Button', + role: 'AXButton', + AXLabel: "Don't Allow", + pid: LIMRUN_ALERT_HOST_PID, + enabled: true, + frame: { x: 57, y: 511.67, width: 288, height: 48 }, + traits: ['Button', 'Scrollable'], + }, + { + type: 'Button', + role: 'AXButton', + AXLabel: 'Allow Once', + pid: LIMRUN_ALERT_HOST_PID, + enabled: true, + frame: { x: 57, y: 567.67, width: 288, height: 48 }, + traits: ['Button', 'Scrollable'], + }, + { + type: 'Button', + role: 'AXButton', + AXLabel: 'Allow While Using App', + pid: LIMRUN_ALERT_HOST_PID, + enabled: true, + frame: { x: 57, y: 623.67, width: 288, height: 48 }, + traits: ['Button', 'Scrollable'], + }, + ]); +} + +/** + * The live ATT prompt with every choice degraded out of action: one disabled, one explicitly + * unhittable, one label-less and frameless. This is what a capture looks like when the tap was + * ignored or the sheet is mid-transition — the dialog is still visibly up, yet nothing on it + * can be pressed. + */ +export function limrunAlertTreeWithUnpressableButtons(): IosTreeNode { + const root = limrunAttAlertTree(); + return { + ...root, + children: [ + ...(root.children ?? []).slice(0, 2), + { + type: 'Button', + role: 'AXButton', + AXLabel: 'Allow', + pid: root.pid, + enabled: false, + frame: { x: 57, y: 511.67, width: 288, height: 48 }, + traits: ['Button'], + }, + { + type: 'Button', + role: 'AXButton', + AXLabel: 'Ask App Not to Track', + pid: root.pid, + enabled: true, + hittable: false, + frame: { x: 57, y: 567.67, width: 288, height: 48 }, + traits: ['Button'], + }, + { + type: 'Button', + role: 'AXButton', + AXLabel: 'Later', + pid: root.pid, + enabled: true, + traits: ['Button'], + }, + ], + }; +} + +/** An alert whose only choices are ambiguous: no label classifies as accept or dismiss. */ +export function limrunAlertAmbiguousButtonsTree(): IosTreeNode { + const root = limrunAttAlertTree(); + return { + ...root, + children: [ + ...(root.children ?? []).slice(0, 2), + { + type: 'Button', + role: 'AXButton', + AXLabel: 'Maybe Later', + pid: LIMRUN_ALERT_HOST_PID, + enabled: true, + frame: { x: 57, y: 511.67, width: 288, height: 48 }, + traits: ['Button', 'Scrollable'], + }, + { + type: 'Button', + role: 'AXButton', + AXLabel: 'More Info', + pid: LIMRUN_ALERT_HOST_PID, + enabled: true, + frame: { x: 57, y: 567.67, width: 288, height: 48 }, + traits: ['Button', 'Scrollable'], + }, + ], + }; +} + +/** The live app home screen: app root with the app pid, plus SpringBoard status-bar nodes. */ +export function limrunAppScreenTree(): IosTreeNode { + return { + type: 'Application', + role: 'AXApplication', + AXLabel: 'ATTProbe', + pid: LIMRUN_APP_PID, + frame: { x: 0, y: 0, ...LIMRUN_ALERT_SCREEN }, + traits: ['None'], + children: [ + { + type: 'StaticText', + role: 'AXStaticText', + AXLabel: 'ATT Probe', + pid: LIMRUN_APP_PID, + frame: { x: 161, y: 418.5, width: 80, height: 20.33 }, + }, + { + type: 'Button', + role: 'AXButton', + AXLabel: 'Request tracking', + AXUniqueId: 'request-tracking', + pid: LIMRUN_APP_PID, + enabled: true, + frame: { x: 136.67, y: 441, width: 129, height: 20.33 }, + }, + { + type: 'Button', + role: 'AXButton', + AXLabel: 'Present app alert', + AXUniqueId: 'present-app-alert', + pid: LIMRUN_APP_PID, + enabled: true, + frame: { x: 136, y: 463.33, width: 130.33, height: 20.33 }, + }, + { + type: 'StaticText', + role: 'AXStaticText', + AXLabel: '21:15', + pid: LIMRUN_SPRINGBOARD_PID, + frame: { x: 53, y: 22.33, width: 42, height: 20.33 }, + }, + { + type: 'GenericElement', + role: 'AXGenericElement', + AXLabel: 'Cellular', + pid: LIMRUN_SPRINGBOARD_PID, + frame: { x: 288.33, y: 26.33, width: 19.33, height: 12.33 }, + }, + { + type: 'GenericElement', + role: 'AXGenericElement', + AXLabel: '100% battery power', + pid: LIMRUN_SPRINGBOARD_PID, + frame: { x: 339.33, y: 26, width: 27.33, height: 13 }, + }, + ], + }; +} + +/** + * Negative control: the live app-owned `UIAlertController` ("Unsaved changes" / Cancel / + * Discard). It shares the flat text+button shape of a system alert but rides the app's root + * with the app's pid, so the alert legs must refuse to treat it as one. + */ +export function limrunAppAlertDialogTree(): IosTreeNode { + const home = limrunAppScreenTree(); + return { + ...home, + children: [ + { + type: 'StaticText', + role: 'AXStaticText', + AXLabel: 'Unsaved changes', + pid: LIMRUN_APP_PID, + frame: { x: 82, y: 408.67, width: 238, height: 20.33 }, + }, + { + type: 'StaticText', + role: 'AXStaticText', + AXLabel: 'Leave without saving?', + pid: LIMRUN_APP_PID, + frame: { x: 82, y: 432.67, width: 238, height: 15.67 }, + }, + { + type: 'Button', + role: 'AXButton', + AXLabel: 'Cancel', + pid: LIMRUN_APP_PID, + enabled: true, + frame: { x: 66, y: 469.33, width: 135, height: 44 }, + traits: ['Button', 'Scrollable'], + }, + { + type: 'Button', + role: 'AXButton', + AXLabel: 'Discard', + pid: LIMRUN_APP_PID, + enabled: true, + frame: { x: 201.33, y: 469.33, width: 134.67, height: 44 }, + traits: ['Button', 'Scrollable'], + }, + ...(home.children ?? []).slice(3), + ], + }; +} + +/** SpringBoard's home screen: a blank-label root, but full of launch icons rather than a dialog. + * Root and icons share SpringBoard's pid, as the live home capture reports it. */ +export function limrunHomeScreenTree(): IosTreeNode { + return { + ...alertRoot([ + { + type: 'Button', + role: 'AXButton', + AXLabel: 'Maps', + pid: LIMRUN_SPRINGBOARD_PID, + enabled: true, + frame: { x: 24.33, y: 88, width: 168.33, height: 191 }, + traits: ['Scrollable', 'UpdatesFrequently', 'LaunchIcon'], + }, + { + type: 'Button', + role: 'AXButton', + AXLabel: 'Calendar', + pid: LIMRUN_SPRINGBOARD_PID, + enabled: true, + frame: { x: 209.67, y: 88, width: 168.33, height: 191 }, + traits: ['Scrollable', 'UpdatesFrequently', 'LaunchIcon'], + }, + ]), + pid: LIMRUN_SPRINGBOARD_PID, + }; +} + +/** + * The app screen and the ATT alert as sibling roots — what a capture with a live system alert + * over an app looks like. The alert is the actionable one here; the app screen is background. + */ +export function limrunAlertOverlayOnAppScreenTree(): IosTreeNode[] { + return [limrunAppScreenTree(), limrunAttAlertTree()]; +} + +/** + * Lookalike (not a live capture): the live app dialog of {@link limrunAppAlertDialogTree} with + * its root's label blanked, content pids untouched. The captures reject it through process + * exclusivity — every live app capture carries foreign-pid status-bar nodes on the app root, + * while the system alert root is process-exclusive. The residual the detector honestly cannot + * separate is an app dialog under a blanked root in a capture with no foreign-pid node at all; + * that is what the label-strict acting legs backstop. + */ +export function limrunBlankRootAppDialogLookalikeTree(): IosTreeNode { + const dialog = limrunAppAlertDialogTree(); + return { ...dialog, AXLabel: ' ' }; +} diff --git a/packages/provider-limrun/src/alert-detection.test.ts b/packages/provider-limrun/src/alert-detection.test.ts new file mode 100644 index 0000000000..c778438dab --- /dev/null +++ b/packages/provider-limrun/src/alert-detection.test.ts @@ -0,0 +1,249 @@ +import { describe, expect, it } from 'vitest'; +import { + chooseLimrunAlertButton, + isLimrunAlertHostRoot, + readLimrunAlertCapture, +} from './alert-detection.ts'; +import { + limrunAlertAmbiguousButtonsTree, + limrunAlertOverlayOnAppScreenTree, + limrunAlertTreeWithUnpressableButtons, + limrunAlertTreeWithoutButtons, + limrunAppAlertDialogTree, + limrunAppScreenTree, + limrunAttAlertTree, + limrunBlankRootAppDialogLookalikeTree, + limrunHomeScreenTree, +} from './alert-detection.fixtures.ts'; + +function alertTreeWithButtons(...labels: string[]) { + const root = limrunAttAlertTree(); + return { + ...root, + children: [ + ...(root.children ?? []).slice(0, 2), + ...labels.map((label, index) => ({ + type: 'Button', + role: 'AXButton', + AXLabel: label, + pid: root.pid, + enabled: true, + frame: { x: 57, y: 511.67 + index * 56, width: 288, height: 48 }, + traits: ['Button', 'Scrollable'], + })), + ], + }; +} + +describe('isLimrunAlertHostRoot', () => { + it('names the blank-label application root a system alert arrives on', () => { + expect(isLimrunAlertHostRoot(limrunAttAlertTree())).toBe(true); + }); + + it('refuses an app-owned root: the app names its own application', () => { + expect(isLimrunAlertHostRoot(limrunAppScreenTree())).toBe(false); + expect(isLimrunAlertHostRoot(limrunAppAlertDialogTree())).toBe(false); + }); + + it('refuses roots that are not applications', () => { + expect(isLimrunAlertHostRoot({ role: 'AXGroup', AXLabel: ' ' })).toBe(false); + expect(isLimrunAlertHostRoot({ role: 'AXButton', AXLabel: ' ' })).toBe(false); + }); +}); + +describe('readLimrunAlertCapture', () => { + it('reads the live ATT alert: title, message, and both buttons in tree order', () => { + const capture = readLimrunAlertCapture(limrunAttAlertTree()); + expect(capture.kind).toBe('alert'); + if (capture.kind !== 'alert') return; + expect(capture.candidate.alert.title).toBe( + 'Allow “ATTProbe” to track your activity across other companies’ apps and websites?', + ); + expect(capture.candidate.alert.message).toBe( + 'This build asks for tracking authorization once, for automation testing.', + ); + expect(capture.candidate.alert.buttons).toEqual(['Ask App Not to Track', 'Allow']); + expect(capture.candidate.alert.platform).toBe('ios'); + expect(capture.candidate.alert.source).toBe('permission'); + expect(capture.candidate.buttons.map((button) => button.role)).toEqual(['dismiss', 'accept']); + // The tap point is the live button frame's center, rounded the way `centerOfRect` rounds. + expect(capture.candidate.buttons[1]).toEqual({ + label: 'Allow', + x: 201, + y: 592, + role: 'accept', + }); + }); + + it('finds the alert when the app screen is the first root of the capture', () => { + const capture = readLimrunAlertCapture(limrunAlertOverlayOnAppScreenTree()); + expect(capture.kind === 'alert' && capture.candidate.alert.buttons).toEqual([ + 'Ask App Not to Track', + 'Allow', + ]); + }); + + it('refuses the live app-owned UIAlertController: a named root is app UI, not a host sheet', () => { + expect(readLimrunAlertCapture(limrunAppAlertDialogTree()).kind).toBe('none'); + }); + + it('refuses a plain app screen', () => { + expect(readLimrunAlertCapture(limrunAppScreenTree()).kind).toBe('none'); + }); + + it("refuses SpringBoard's home screen: launch icons are not a dialog", () => { + expect(readLimrunAlertCapture(limrunHomeScreenTree()).kind).toBe('none'); + }); + + it('refuses a widget-bearing home screen: LaunchIcon buttons never qualify', () => { + const home = limrunHomeScreenTree(); + const withWidgets = { + ...home, + children: [ + { + type: 'StaticText', + role: 'AXStaticText', + AXLabel: 'Mostly Sunny', + pid: home.pid, + frame: { x: 24, y: 300, width: 160, height: 20 }, + }, + ...(home.children ?? []), + ], + }; + expect(readLimrunAlertCapture(withWidgets).kind).toBe('none'); + }); + + it('reports a text-only sheet on a blank root as undecided, not absence', () => { + // Buttons can drop out between captures mid-transition; a message on a host-shaped root + // is alert-shaped content the reader cannot resolve, and absence is a stronger claim. + expect(readLimrunAlertCapture(limrunAlertTreeWithoutButtons()).kind).toBe('undecided'); + }); + + it('reports a one-choice sheet as undecided: alert-shaped, not resolvable to an action', () => { + expect(readLimrunAlertCapture(alertTreeWithButtons('Allow')).kind).toBe('undecided'); + expect(readLimrunAlertCapture(alertTreeWithButtons('OK')).kind).toBe('undecided'); + }); + + it('reports a capture mixing an app screen and a text-only sheet as undecided', () => { + expect( + readLimrunAlertCapture([limrunAppScreenTree(), limrunAlertTreeWithoutButtons()]).kind, + ).toBe('undecided'); + }); + + it('reports a capture whose host sheets disagree as undecided, not as no alert', () => { + const second = { + ...limrunAttAlertTree(), + pid: 99, + children: (limrunAttAlertTree().children ?? []).map((child) => ({ ...child, pid: 99 })), + }; + expect(readLimrunAlertCapture([limrunAttAlertTree(), second]).kind).toBe('undecided'); + }); + + it('reports a blank-label root that reports no pid as undecided: unverifiable, not absent', () => { + const root = limrunAttAlertTree(); + expect(readLimrunAlertCapture({ ...root, pid: undefined }).kind).toBe('undecided'); + }); + + it('reports a blank-label root whose content belongs to another process as undecided', () => { + const root = limrunAttAlertTree(); + const foreignContent = { + ...root, + children: (root.children ?? []).map((child) => ({ ...child, pid: 4242 })), + }; + expect(readLimrunAlertCapture(foreignContent).kind).toBe('undecided'); + }); + + it('reports the blank-root app-dialog lookalike as undecided, never an actionable alert', () => { + // The live app captures ride SpringBoard status nodes (a foreign pid) on the app root; the + // system alert root is process-exclusive. Blanking the app dialog's label keeps that + // discriminator intact: the capture is alert-shaped yet never resolves to a candidate some + // action could press into. + expect(readLimrunAlertCapture(limrunBlankRootAppDialogLookalikeTree()).kind).toBe('undecided'); + }); + + it('reports a blank-label sheet sharing its root with another process as undecided', () => { + const root = limrunAttAlertTree(); + const shared = { + ...root, + children: [ + ...(root.children ?? []), + { + type: 'GenericElement', + role: 'AXGenericElement', + AXLabel: 'Cellular', + pid: 4242, + frame: { x: 288, y: 26, width: 19, height: 12 }, + }, + ], + }; + expect(readLimrunAlertCapture(shared).kind).toBe('undecided'); + }); + + it('reports a sheet whose every button is unpressable as undecided, not absence', () => { + // Disabled, explicitly unhittable, and frameless buttons press nothing; the dialog they + // belong to is still visibly up, so the capture must not claim absence. + expect(readLimrunAlertCapture(limrunAlertTreeWithUnpressableButtons()).kind).toBe('undecided'); + }); + + it('reports a permission alert from its button vocabulary and a plain one as a system dialog', () => { + const att = readLimrunAlertCapture(limrunAttAlertTree()); + expect(att.kind === 'alert' && att.candidate.alert.source).toBe('permission'); + const plain = readLimrunAlertCapture(alertTreeWithButtons('No', 'Yes')); + expect(plain.kind === 'alert' && plain.candidate.alert.source).toBe('system-dialog'); + }); +}); + +describe('chooseLimrunAlertButton', () => { + function buttonsOf(tree: Parameters[0]) { + const capture = readLimrunAlertCapture(tree); + if (capture.kind !== 'alert') throw new Error('expected an alert capture'); + return capture.candidate.buttons; + } + + it('accepts the affirmative and dismisses the denying choice of the ATT prompt', () => { + const buttons = buttonsOf(limrunAttAlertTree()); + expect(chooseLimrunAlertButton(buttons, 'accept')?.label).toBe('Allow'); + expect(chooseLimrunAlertButton(buttons, 'dismiss')?.label).toBe('Ask App Not to Track'); + }); + + it('fails an ambiguous accept closed when no button classifies as affirmative', () => { + const buttons = buttonsOf(limrunAlertAmbiguousButtonsTree()); + expect(chooseLimrunAlertButton(buttons, 'accept')).toBeNull(); + }); + + it('fails an ambiguous dismiss closed when no button classifies as denying', () => { + const buttons = buttonsOf(limrunAlertAmbiguousButtonsTree()); + expect(chooseLimrunAlertButton(buttons, 'dismiss')).toBeNull(); + }); + + it('never picks a choice of the wrong role, whatever else the alert offers', () => { + const allowOnly = buttonsOf(alertTreeWithButtons('Deny', 'Allow')); + expect(chooseLimrunAlertButton(allowOnly, 'accept')?.label).toBe('Allow'); + expect(chooseLimrunAlertButton(allowOnly, 'dismiss')?.label).toBe('Deny'); + const noAccept = buttonsOf(alertTreeWithButtons('Cancel', 'Maybe Later')); + expect(chooseLimrunAlertButton(noAccept, 'accept')).toBeNull(); + const noDismiss = buttonsOf(alertTreeWithButtons('Open Settings', 'Continue')); + expect(chooseLimrunAlertButton(noDismiss, 'dismiss')).toBeNull(); + }); + + it('classifies the classic iOS pair, the typographic apostrophe, and iOS variant grants', () => { + const classic = buttonsOf(alertTreeWithButtons("Don't Allow", 'OK')); + expect(chooseLimrunAlertButton(classic, 'accept')?.label).toBe('OK'); + expect(chooseLimrunAlertButton(classic, 'dismiss')?.label).toBe("Don't Allow"); + const curly = buttonsOf(alertTreeWithButtons('Don’t Allow', 'Allow')); + expect(chooseLimrunAlertButton(curly, 'dismiss')?.label).toBe('Don’t Allow'); + const settings = buttonsOf(alertTreeWithButtons('Not Now', 'Open Settings')); + expect(chooseLimrunAlertButton(settings, 'accept')?.label).toBe('Open Settings'); + expect(chooseLimrunAlertButton(settings, 'dismiss')?.label).toBe('Not Now'); + for (const label of ['Allow Once', 'Allow All the Time', 'Allow While Using App']) { + const granted = buttonsOf(alertTreeWithButtons('Ask App Not to Track', label)); + expect(chooseLimrunAlertButton(granted, 'accept')?.label).toBe(label); + } + }); + + it('refuses a choice two physical buttons share: pressing one of them would guess', () => { + const doubled = buttonsOf(alertTreeWithButtons('Allow', 'Deny', 'Allow')); + expect(chooseLimrunAlertButton(doubled, 'accept')).toBeNull(); + expect(chooseLimrunAlertButton(doubled, 'dismiss')?.label).toBe('Deny'); + }); +}); diff --git a/packages/provider-limrun/src/alert-detection.ts b/packages/provider-limrun/src/alert-detection.ts new file mode 100644 index 0000000000..422dc8317e --- /dev/null +++ b/packages/provider-limrun/src/alert-detection.ts @@ -0,0 +1,226 @@ +import { + type AlertButtonRole, + classifyAlertButtonRole, + type AlertInfo, + type AlertSource, + selectRankedAlertButtonByRole, +} from '@agent-device/contracts/alert-contract'; +import { centerOfRect, type Rect } from '@agent-device/kernel/snapshot'; +import { isPositiveFiniteRect } from '@agent-device/kernel/rect'; +import type { TreeAlertObservation } from '@agent-device/capture-kit/alert-tree-driver'; +import { readIosNodeChildren, readIosNodeRect, type IosTreeNode } from './snapshot.ts'; + +/** + * System-alert detection over a Limrun iOS element tree (issue #3349). A Limrun iOS direct + * session has no XCUITest runner, but its element tree carries the system alert: on a live + * instance the ATT prompt arrived as its own `AXApplication` root with a blank `AXLabel` and one + * `pid` shared by its whole subtree — never the app's. The app's own `UIAlertController` rode + * inside the app's named root under the app's pid. Nothing in a capture names an owner, so the + * host rule is a stated heuristic: a host sheet qualifies only on a blank-label root reporting a + * `pid` whose entire subtree reports that same one. The named residual these captures cannot + * exclude: an app blanking its own root label while showing an unshared dialog — the acting legs + * press only label-unambiguous buttons as the backstop. Button eligibility never asserts + * hitability: captures omit `hittable` entirely, so an explicit `false` refuses and an omission + * carries no signal; enabled, label, and complete-rect rules admit a button. + */ + +/** Only the two host-sheet sources exist on this surface: no node names a package to attribute. */ +type LimrunAlertSource = Extract; + +export type LimrunAlertButton = Readonly<{ + label: string; + x: number; + y: number; + role: AlertButtonRole; +}>; + +export type LimrunAlertInfo = AlertInfo & { + buttons: string[]; + platform: 'ios'; + source: LimrunAlertSource; +}; + +export type LimrunAlertCandidate = Readonly<{ + alert: LimrunAlertInfo; + buttons: LimrunAlertButton[]; +}>; + +// The button roles come from the shared tree-alert label vocabulary in contracts/alert-contract: +// the same label states the same role on Android and here, so `alert dismiss` cannot act +// differently on the two platforms for one label. English-only, matching the untranslated +// device locale these instances run; a localized alert resolves as a candidate whose buttons +// all classify neutral, and the acting legs refuse on the button-undecided reason. +/** The subset that states a grant/deny choice; a plain Yes/No dialog is not a permission prompt. */ +const PERMISSION_LABEL_PATTERN = + /^(?:allow.*|deny|don.?t allow|ask app not to track|turn on|open settings)$/i; +/** SpringBoard marks its home-screen icons with this trait; an alert never carries it. */ +const LAUNCH_ICON_TRAIT = 'LaunchIcon'; + +function nodeLabel(node: IosTreeNode): string { + const label = node.AXLabel ?? node.label; + return typeof label === 'string' ? label.trim() : ''; +} + +/** + * The application root a system alert arrives on: the blank-label `AXApplication`. An app names + * its own root, so a labeled application is the app's scene — including when it hosts an + * in-app `UIAlertController` — and never a system host. SpringBoard also reports a blank label; + * its screens are refused downstream because they carry no dialog-shaped content. + */ +export function isLimrunAlertHostRoot(node: IosTreeNode): boolean { + if (node.role !== 'AXApplication' && node.type !== 'Application') return false; + return nodeLabel(node).length === 0; +} + +/** + * The one capture reader: what this element tree says about system alerts, in the shared + * tree-alert observation the driver confirms against. `none` states the capture carries no + * alert-shaped content on any host root; `undecided` states it carries alert-shaped content this + * reader cannot resolve into one actionable alert — more than one alert-shaped host sheet, or a + * sheet whose choices are visible yet unactionable (a degraded or shared root). Only `alert` and + * `none` carry information; `undecided` is not absence, and confirming a dismissal may accept + * `none` but never `undecided`. + */ +export type LimrunAlertCapture = TreeAlertObservation; + +/** + * The actionable alert on this capture, or why the capture answers nothing. A host sheet is + * actionable only on a blank-label, process-exclusive root (it reports a `pid` and its whole + * subtree reports that same one) offering at least two pressable choices (enabled, not reported + * unhittable, with a complete rect and a label) under at least one piece of text, with no + * SpringBoard launch icons. A root that shows the same dialog-shaped content but fails one of + * those tests — a button dropping to unhittable, disabled, frameless, or label-less between + * captures, a root sharing its subtree with another pid — still shows a dialog, so the capture + * is undecided, not absent. A named root is app UI, a home screen offers launch icons, and a + * blank root with neither a message nor a pair of button-shaped nodes carries nothing a dialog + * could have degraded from: those are absence. What these captures cannot exclude is an app + * dialog under a blanked root no other process rides; the label-strict acting legs are the + * backstop named above. + */ +export function readLimrunAlertCapture(tree: IosTreeNode | IosTreeNode[]): LimrunAlertCapture { + let candidate: LimrunAlertCandidate | null = null; + let alertShapedRoots = 0; + for (const root of Array.isArray(tree) ? tree : [tree]) { + if (!isLimrunAlertHostRoot(root)) continue; + const observed = readAlertHostRoot(root); + if (observed === 'absent') continue; + alertShapedRoots += 1; + if (observed !== 'undecided') candidate = observed; + } + // One actionable sheet among nothing else alert-shaped is the answer. Two sheets, or one that + // is alert-shaped yet unactionable, is content this reader must not resolve by guessing — + // pressing into it would pick a side the observation does not support. + if (alertShapedRoots === 0) return { kind: 'none' }; + return alertShapedRoots === 1 && candidate ? { kind: 'alert', candidate } : { kind: 'undecided' }; +} + +type LimrunHostRootReading = LimrunAlertCandidate | 'undecided' | 'absent'; + +function readAlertHostRoot(root: IosTreeNode): LimrunHostRootReading { + const descendants = collectDescendants(root); + // A home screen is a home screen: SpringBoard marks its icons, and widgets can carry text, + // so the trait answers absence before any dialog question is asked. + if (descendants.some(isLaunchIcon)) return 'absent'; + const texts = descendants + .filter(isStaticTextNode) + .map(nodeLabel) + .filter((text) => text.length > 0); + const choices = descendants.filter((node) => isButtonNode(node) && !isLaunchIcon(node)); + // Absence means nothing a dialog could have degraded from: no message and no pair of + // button-shaped nodes. + if (texts.length === 0 && choices.length < 2) return 'absent'; + // Process exclusivity is the capture-side host heuristic named above: the live system alert + // root was process-exclusive (one pid on every node) while the app root rode SpringBoard's + // status nodes under a second pid. It settles actionability, never absence. + const exclusive = + typeof root.pid === 'number' && descendants.every((node) => node.pid === root.pid); + if (!exclusive) return 'undecided'; + const buttons = findAlertButtons(descendants); + // Dialog-shaped content that is not actionable — a sheet rendered mid-transition, one whose + // buttons just dropped to disabled, unhittable, or frameless, a message-less button pair — + // still shows something alert-shaped. That is undecided, never absence, so no caller can + // report it as a confirmed no-alert. + if (texts.length === 0 || buttons.length < 2) return 'undecided'; + const [title, ...rest] = texts; + const alert: LimrunAlertInfo = { + ...(title ? { title } : {}), + ...(rest.length > 0 ? { message: [...new Set(rest)].join('\n') } : {}), + buttons: buttons.map((button) => button.label), + platform: 'ios', + source: classifyAlertSource(buttons), + }; + return { alert, buttons }; +} + +function collectDescendants(root: IosTreeNode): IosTreeNode[] { + const nodes: IosTreeNode[] = []; + const visit = (node: IosTreeNode) => { + for (const child of readIosNodeChildren(node)) { + nodes.push(child); + visit(child); + } + }; + visit(root); + return nodes; +} + +function isButtonNode(node: IosTreeNode): boolean { + return node.role === 'AXButton' || node.type === 'Button'; +} + +function isLaunchIcon(node: IosTreeNode): boolean { + return node.traits?.includes(LAUNCH_ICON_TRAIT) === true; +} + +function isStaticTextNode(node: IosTreeNode): boolean { + return node.role === 'AXStaticText' || node.type === 'StaticText'; +} + +/** + * Pressable alert choices only: a disabled button is not a choice the alert offers right now, a + * button without a complete rect has no point the touch seam could aim at, and a button the + * provider explicitly reports as `hittable: false` cannot be pressed; an omitted `hittable` + * carries no signal either way. Process membership was settled by the caller, so every physical + * node stays in the list: two buttons sharing a label are two points a tap could land on, and + * the chooser refuses rather than taking the first. + */ +function findAlertButtons(nodes: IosTreeNode[]): LimrunAlertButton[] { + const buttons: LimrunAlertButton[] = []; + for (const node of nodes) { + if (!isButtonNode(node) || node.enabled === false || node.hittable === false) continue; + const label = nodeLabel(node); + if (!label) continue; + const rect = readIosNodeRect(node); + if (!isPressableRect(rect)) continue; + const point = centerOfRect(rect); + buttons.push({ label, x: point.x, y: point.y, role: classifyAlertButtonRole(label) }); + } + return buttons; +} + +function isPressableRect(rect: Rect | undefined): rect is Rect { + return isPositiveFiniteRect(rect); +} + +/** A button vocabulary naming a grant/deny choice makes this a permission prompt. */ +function classifyAlertSource(buttons: readonly LimrunAlertButton[]): LimrunAlertSource { + return buttons.some((button) => PERMISSION_LABEL_PATTERN.test(button.label.trim())) + ? 'permission' + : 'system-dialog'; +} + +/** + * The button one action may press, or null when pressing any visible choice would guess. + * Deliberately stricter than Android's: with no `android:id/button1`-style identity to fall back + * on, only a button whose own label states the asked-for choice may be pressed, resolved through + * the shared preference table — a multi-grant sheet ("Allow Once" / "Allow While Using App" / + * "Don't Allow") resolves to the preferred grant, while equal-rank labels or a vocabulary-free + * multiplicity refuse. An accept must never press a denial, and an undecided action is refused + * rather than resolved by tree position. + */ +export function chooseLimrunAlertButton( + buttons: readonly LimrunAlertButton[], + action: 'accept' | 'dismiss', +): LimrunAlertButton | null { + return selectRankedAlertButtonByRole(buttons, action); +} diff --git a/packages/provider-limrun/src/alert.test.ts b/packages/provider-limrun/src/alert.test.ts new file mode 100644 index 0000000000..fb1c4e6820 --- /dev/null +++ b/packages/provider-limrun/src/alert.test.ts @@ -0,0 +1,505 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { + ALERT_BUTTON_UNDECIDED_REASON, + ALERT_CAPTURE_UNDECIDED_REASON, +} from '@agent-device/contracts/alert-contract'; +import { AppError } from '@agent-device/kernel/errors'; +import { + limrunAlertAmbiguousButtonsTree, + limrunAlertTreeWithUnpressableButtons, + limrunAppScreenTree, + limrunAttAlertTree, + limrunLocationAlertTree, +} from './alert-detection.fixtures.ts'; +import { handleLimrunIosAlert, type LimrunIosAlertClient } from './alert.ts'; +import type { IosTreeNode } from './snapshot.ts'; + +/** + * The legs share the tree-alert driver, whose polls sleep between captures on the shared + * deadline primitive; the clock is the assertion, not the wait (`docs/agents/testing.md`). + */ +async function underFakeTimers(run: () => Promise): Promise { + vi.useFakeTimers(); + try { + const outcome = run(); + outcome.catch(() => {}); + await vi.runAllTimersAsync(); + return await outcome; + } finally { + vi.useRealTimers(); + } +} + +function clientWithTrees(trees: Array): LimrunIosAlertClient { + const elementTree = vi.fn(async () => { + const next = trees.length > 1 ? trees.shift() : trees[0]; + // The real client answers an array for a multi-root capture and a single node otherwise. + return (Array.isArray(next) ? next : [next]) as unknown as ReturnType< + LimrunIosAlertClient['elementTree'] + >; + }); + return { elementTree, tap: vi.fn(async () => {}) }; +} + +afterEach(() => { + vi.useRealTimers(); +}); + +describe('wait budget', () => { + it('spends exactly the caller window and reports the timeout', async () => { + const client = clientWithTrees([limrunAppScreenTree()]); + + await expect( + underFakeTimers(() => handleLimrunIosAlert(client, 'wait', { timeoutMs: 5_000 })), + ).rejects.toThrow('alert wait timed out'); + // One poll per interval over the window, plus the final sample the polling primitive + // lands exactly on the deadline: a probe that starts at the boundary is a started probe, + // and the budget rule lets it answer — the deadline ends the loop, it does not unstart it. + expect(client.elementTree).toHaveBeenCalledTimes(Math.ceil(5_000 / 300) + 1); + }); + + it('treats a malformed window as the bounded default rather than an unbound loop', async () => { + const client = clientWithTrees([limrunAppScreenTree()]); + + await expect( + underFakeTimers(() => handleLimrunIosAlert(client, 'wait', { timeoutMs: Number.NaN })), + ).rejects.toThrow('alert wait timed out'); + // The fallback is the contract default, spent on the driven clock — this returned, so the + // loop had a bound, which a NaN comparison would not have given it. + expect(client.elementTree).toHaveBeenCalledTimes(Math.ceil(10_000 / 300) + 1); + }); + + it('reads a finite budget past the timer ceiling as the bounded default, never a misfire', async () => { + // 2^32 reaches `AbortSignal.timeout` as a RangeError and 2^31 as a 1 ms timer; both must + // arrive here as the contract default, spent on the driven clock. + for (const oversized of [2 ** 31, 2 ** 32]) { + const client = clientWithTrees([limrunAppScreenTree()]); + await expect( + underFakeTimers(() => handleLimrunIosAlert(client, 'wait', { timeoutMs: oversized })), + ).rejects.toThrow('alert wait timed out'); + expect(client.elementTree).toHaveBeenCalledTimes(Math.ceil(10_000 / 300) + 1); + } + }); + + it('consumes the boundary sample: an alert answering exactly at the deadline is still reported', async () => { + // The budget rule applied to `wait`: the poll that the sleep lands on the deadline is a + // started probe — it finishes and its answer counts. The old contract refused it as a late + // verdict; under the new one a sheet visible as the budget closes is exactly what `wait` + // is for. No probe ever starts once the budget has passed: the boundary sample is the last. + const client = { + elementTree: vi + .fn() + .mockResolvedValueOnce([limrunAppScreenTree()]) + .mockResolvedValueOnce([limrunAttAlertTree()]), + tap: vi.fn(async () => {}), + } satisfies LimrunIosAlertClient; + + const result = await underFakeTimers(() => + handleLimrunIosAlert(client, 'wait', { timeoutMs: 300 }), + ); + expect(result.kind).toBe('alertWait'); + expect(client.elementTree).toHaveBeenCalledTimes(2); + }); + + it('presses a candidate whose started lookup probe answered after the deadline', async () => { + // The maintainer's whole-rule pin on the acting leg: a bounded lookup probe stalling past + // its 2s budget still finishes, and the alert it names is the alert on screen — the press + // acts on it rather than refusing. The old code threw `alert-window-expired, dispatched: + // no` here even though the sheet was plainly visible to the probe it had started. + const client = { + elementTree: vi + .fn() + .mockResolvedValueOnce([limrunAppScreenTree()]) + .mockImplementationOnce( + async () => + new Promise((resolve) => { + setTimeout(() => resolve([limrunAttAlertTree()]), 3_000); + }), + ) + // The post-press confirm read sees the app screen: the press did land. + .mockResolvedValue([limrunAppScreenTree()]), + tap: vi.fn(async () => {}), + } satisfies LimrunIosAlertClient; + + const handled = await underFakeTimers(() => handleLimrunIosAlert(client, 'accept')); + expect(handled.kind).toBe('alertHandled'); + expect(client.tap).toHaveBeenCalledTimes(1); + }); + + it('surfaces a failed capture as itself, never as a timeout', async () => { + const elementTree = vi + .fn() + .mockRejectedValue( + new Error('websocket closed'), + ) as unknown as LimrunIosAlertClient['elementTree']; + const client = { elementTree, tap: vi.fn(async () => {}) } satisfies LimrunIosAlertClient; + + await expect(handleLimrunIosAlert(client, 'wait')).rejects.toThrow('websocket closed'); + }); + + it('ends the leg on cancellation only when the capture transport dies with it', async () => { + // The accepted capability limit, tested rather than assumed. The Limrun client exposes no + // abort on `elementTree`, so a caller cancel cannot unstart or cut a capture already in + // flight: awaiting it keeps the leg open (the caller's own request ends at the daemon; this + // promise is detached work no seam may fake-abort). What does end the leg is the transport + // route the SDK does have — an instance teardown that fails the pending call — and the + // driver reports that as the shared cancellation, never as a broken backend. + const caller = new AbortController(); + let settled = 'pending'; + let rejectCapture: ((error: unknown) => void) | undefined; + let markCaptureStarted!: () => void; + const captureStarted = new Promise((resolve) => { + markCaptureStarted = resolve; + }); + const client = { + elementTree: vi.fn(() => { + markCaptureStarted(); + return new Promise((_resolve, reject) => { + rejectCapture = reject; + }); + }) as unknown as LimrunIosAlertClient['elementTree'], + tap: vi.fn(async () => {}), + } satisfies LimrunIosAlertClient; + const leg = handleLimrunIosAlert(client, 'get', { signal: caller.signal }); + leg.then( + () => { + settled = 'resolved'; + }, + () => { + settled = 'rejected'; + }, + ); + await captureStarted; + // Canceled, and the in-flight capture with no abort still holds the leg open. + caller.abort(); + await Promise.resolve(); + expect(settled).toBe('pending'); + + // The instance/session teardown route: the pending call fails, the leg ends, and it ends as + // the caller's cancellation — not as a transport failure. It is a pre-press exit (acting + // legs never press before their lookup returns; get has no press), so it carries the clean + // dispatched: no like the other pre-press refusal. + rejectCapture?.(new Error('websocket closed')); + await expect(leg).rejects.toMatchObject({ + details: { reason: 'request_canceled', dispatched: 'no' }, + }); + }); + + it('reports a press-cancelled acting leg as dispatched: unknown even when the tap answers after the cancel', async () => { + // The tap transport has no abort either: the cancel can arrive while `tap()` is pending and + // the promise simply RESOLVE anyway. A resolved press after a cancel still says "the device + // may have the tap" — the leg must answer `dispatched: unknown`, never a confirmation or a + // clean cancellation. + let resolveTap: (() => void) | undefined; + let markTapStarted!: () => void; + const tapStarted = new Promise((resolve) => { + markTapStarted = resolve; + }); + const trees: Array> = [limrunAttAlertTree()]; + const client = { + elementTree: vi.fn(async () => { + // Lookup resolves the alert; a confirm read would see it gone — the cancel must end + // the leg before that read is ever started. + const next = + trees.length > 0 ? (trees.shift() ?? limrunAppScreenTree()) : limrunAppScreenTree(); + return [next] as unknown as ReturnType; + }), + tap: vi.fn(() => { + markTapStarted(); + return new Promise((resolve) => { + resolveTap = resolve; + }); + }), + } satisfies LimrunIosAlertClient; + + const caller = new AbortController(); + const leg = handleLimrunIosAlert(client, 'accept', { signal: caller.signal }); + leg.catch(() => {}); + await tapStarted; + caller.abort(); + resolveTap?.(); + + await expect(leg).rejects.toMatchObject({ + details: { reason: 'request_canceled', dispatched: 'unknown' }, + }); + // Exactly one dispatch attempt, never a replay: the leg died with the cancel, not with a + // second press or a confirm that contradicts what the caller already withdrew. + expect(client.tap).toHaveBeenCalledTimes(1); + // And the closest witness that the leg ended BEFORE any confirm read: the only capture is + // the lookup probe that resolved the alert — the confirm loop's top-of-loop cancellation + // check stops it before it starts a probe. + expect(client.elementTree).toHaveBeenCalledTimes(1); + }); + + it('honors a cancel that arrives while a straddling lookup probe is in flight: no press starts', async () => { + // The probe the cancel rode finished anyway (the budget rule) — but its alert answer is NOT + // acted on once the caller withdrew the action: the leg refuses at the pre-press boundary + // with the clean cancellation and dispatched: no, and no tap is ever attempted. + let resolveCapture: ((value: unknown) => void) | undefined; + let markCaptureStarted!: () => void; + const captureStarted = new Promise((resolve) => { + markCaptureStarted = resolve; + }); + const held = { + elementTree: vi.fn(() => { + markCaptureStarted(); + return new Promise((resolve) => { + resolveCapture = resolve; + }); + }) as unknown as LimrunIosAlertClient['elementTree'], + tap: vi.fn(async () => {}), + } satisfies LimrunIosAlertClient; + + const caller = new AbortController(); + const leg = handleLimrunIosAlert(held, 'accept', { signal: caller.signal }); + leg.catch(() => {}); + await captureStarted; + caller.abort(); + resolveCapture?.([limrunAttAlertTree()]); + + await expect(leg).rejects.toMatchObject({ + details: { reason: 'request_canceled', dispatched: 'no' }, + }); + expect(held.tap).toHaveBeenCalledTimes(0); + }); + + it('consumes a confirm capture that answers past its window deadline', async () => { + // The same rule after the press: the first confirm probe started inside the confirm budget, + // so it finishes and its answer decides. The Limrun WebSocket client cannot abort an + // in-flight `elementTree` — under the old contract the leg refused the late answer and + // reported the press unconfirmed; under the budget rule the capture that saw the app screen + // IS the confirmation, and the leg handled. The cost is the documented one-call overrun: + // the leg waits for the started transport instead of answering at 2 s. + const client = { + elementTree: vi + .fn() + .mockResolvedValueOnce([limrunAttAlertTree()]) // lookup probe resolves the alert + // The confirm probe stalls 3s on the driven clock — past its 2s budget — and answers + // "gone": under the rule, that answer counts. + .mockImplementationOnce( + async () => + new Promise((resolve) => { + setTimeout(() => resolve([limrunAppScreenTree()]), 3_000); + }), + ), + tap: vi.fn(async () => {}), + } satisfies LimrunIosAlertClient; + + const handled = await underFakeTimers(() => handleLimrunIosAlert(client, 'accept')); + expect(client.tap).toHaveBeenCalledTimes(1); + expect(handled.kind).toBe('alertHandled'); + }); +}); + +describe('get on an undecided capture', () => { + it('says the capture cannot decide, rather than claiming no alert', async () => { + const client = clientWithTrees([[limrunAttAlertTree(), limrunAttAlertTree()]]); + + const result = await handleLimrunIosAlert(client, 'get'); + + expect(result.kind === 'alertStatus' && result.alert).toBeNull(); + // The states are typed on the result, not just in the message: `undecided` and `none` + // arrive with the same null alert and must stay switchable apart. + expect(result.kind === 'alertStatus' && result.status).toBe('undecided'); + expect(result.kind === 'alertStatus' && result.message).toBe( + 'Cannot decide the alert: the capture is alert-shaped but unresolvable', + ); + }); +}); + +describe('acting budgets', () => { + it('spends the confirm budget and reports a stuck alert', async () => { + const client = clientWithTrees([limrunAttAlertTree(), limrunAttAlertTree()]); + + const thrown = await underFakeTimers(() => handleLimrunIosAlert(client, 'dismiss')).then( + () => undefined, + (error: unknown) => error, + ); + expect(thrown).toBeInstanceOf(AppError); + const failure = thrown as AppError; + expect(failure.message).toBe('alert dismiss did not dismiss the visible alert'); + // Every spent-budget exit of an acting leg branches on this reason, per commands.md. + expect(failure.details?.reason).toBe('alert-window-expired'); + expect(client.tap).toHaveBeenCalledTimes(1); + }); + + it('reports an unconfirmed dismissal when an undecided re-read overrules the last match', async () => { + // Confirm probe 1 re-names the pressed alert; probe 2 is alert-shaped yet unresolvable and + // stays that way to the end of the budget. The undecided read is the fresher fact, so the + // leg may not report the dialog as standing on the older match — it reports the press as + // unconfirmed, which is all a spent budget with no settling read may say. + const client = clientWithTrees([ + limrunAttAlertTree(), + limrunAttAlertTree(), + [limrunAttAlertTree(), limrunAttAlertTree()], + ]); + + const thrown = await underFakeTimers(() => handleLimrunIosAlert(client, 'dismiss')).then( + () => undefined, + (error: unknown) => error, + ); + expect(thrown).toBeInstanceOf(AppError); + const failure = thrown as AppError; + expect(failure.message).toBe( + 'alert dismiss pressed the button but the dismissal could not be confirmed', + ); + expect(failure.details?.reason).toBe('alert-window-expired'); + }); + + it('confirms on the capture that answers after a straddle, even one that first re-named the alert', async () => { + // Budget rule over the older sighting rule: confirm probe 1 re-named the pressed alert, + // probe 2 started inside the budget and answers past its deadline that the screen is now + // the app. Both are started-probe answers, so both count, and the freshest one settles the + // dismissal — the dialog the latest capture that answered saw is gone. The old code kept + // the earlier sighting as a still-visible failure precisely because this straddling capture + // was refused; the maintainer's whole-rule supersedes that. + const client = { + elementTree: vi + .fn() + .mockResolvedValueOnce(limrunAttAlertTree()) + .mockResolvedValueOnce(limrunAttAlertTree()) + .mockImplementationOnce( + async () => + new Promise((resolve) => { + setTimeout(() => resolve([limrunAppScreenTree()]), 3_000); + }), + ), + tap: vi.fn(async () => {}), + } satisfies LimrunIosAlertClient; + + const handled = await underFakeTimers(() => handleLimrunIosAlert(client, 'accept')); + expect(handled.kind).toBe('alertHandled'); + expect(client.tap).toHaveBeenCalledTimes(1); + }); + + it('refuses an ambiguous alert without tapping, on a typed reason', async () => { + const client = clientWithTrees([limrunAlertAmbiguousButtonsTree()]); + const thrown = await handleLimrunIosAlert(client, 'accept').then( + () => undefined, + (error: unknown) => error, + ); + expect(thrown).toBeInstanceOf(AppError); + const refusal = thrown as AppError; + expect(refusal.code).toBe('COMMAND_FAILED'); + expect(refusal.details?.reason).toBe(ALERT_BUTTON_UNDECIDED_REASON); + expect(refusal.details?.dispatched).toBe('no'); + expect(client.tap).not.toHaveBeenCalled(); + }); + + it('reports an unconfirmed dismissal when the post-tap capture stays unresolvable to the end', async () => { + // The first probe resolves the single alert; every post-press probe carries two disagreeing + // alert-shaped sheets. An unresolvable capture proves absence for nothing, so the confirm + // loop keeps its budget and fails — and, having never named this alert on screen again, it + // reports the press as unconfirmed rather than claiming a dialog it never re-observed. + const client = clientWithTrees([ + limrunAttAlertTree(), + [limrunAttAlertTree(), limrunAttAlertTree()], + ]); + + await expect(underFakeTimers(() => handleLimrunIosAlert(client, 'dismiss'))).rejects.toThrow( + 'alert dismiss pressed the button but the dismissal could not be confirmed', + ); + }); + + it('reports an unconfirmed dismissal when the same root survives with degraded buttons', async () => { + // The tap was ignored: the post-press capture is the same alert root with its choices dropped + // to disabled, explicitly unhittable, and frameless — an unresolvable shape, not a re-named + // alert. Degraded-but-present is not absence; the leg must fail, not report handled, and it + // may not claim "still visible" beyond what the undecided capture proved. + const client = clientWithTrees([limrunAttAlertTree(), limrunAlertTreeWithUnpressableButtons()]); + + await expect(underFakeTimers(() => handleLimrunIosAlert(client, 'accept'))).rejects.toThrow( + 'alert accept pressed the button but the dismissal could not be confirmed', + ); + expect(client.tap).toHaveBeenCalledTimes(1); + }); + + it('confirms handled when the post-tap capture shows the app screen instead', async () => { + // The closest genuine no-alert negative to the case above: the same seam and budgets, and + // the app's own named root after the tap — observed absence, which does confirm. + const client = clientWithTrees([limrunAttAlertTree(), limrunAppScreenTree()]); + const result = await handleLimrunIosAlert(client, 'accept'); + expect(result.kind === 'alertHandled' && result.button).toBe('Allow'); + }); + + it('taps exactly once on accept, and the next get reports no alert', async () => { + // The maintainer's live-validated timeline at the unit seam: one tap for the action, the + // post-tap confirm observes the app screen, and a following read sees that same absence. + const client = clientWithTrees([limrunAttAlertTree(), limrunAppScreenTree()]); + + const handled = await handleLimrunIosAlert(client, 'accept'); + expect(handled.kind).toBe('alertHandled'); + expect(client.tap).toHaveBeenCalledTimes(1); + + const status = await handleLimrunIosAlert(client, 'get'); + expect(status.kind === 'alertStatus' && status.alert).toBeNull(); + expect(status.kind === 'alertStatus' && status.message).toBe('No alert visible'); + }); + + it('accepts the three-button location sheet on the preferred grant, once', async () => { + // The multi-grant regression at the action seam: "Allow Once" and "Allow While Using App" + // both carry the accept role, and the shared preference table must land one tap on the + // lasting grant rather than refusing the sheet. + const client = clientWithTrees([limrunLocationAlertTree(), limrunAppScreenTree()]); + + const handled = await handleLimrunIosAlert(client, 'accept'); + expect(handled.kind).toBe('alertHandled'); + expect(handled.kind === 'alertHandled' && handled.button).toBe('Allow While Using App'); + expect(client.tap).toHaveBeenCalledTimes(1); + // Grant center: x 57 + 288/2, y 623.67 + 48/2, rounded by the press seam to whole points. + expect(client.tap).toHaveBeenCalledWith(201, 648); + }); + + it('refuses to act on an alert-shaped but unresolvable capture, without tapping', async () => { + const client = clientWithTrees([limrunAlertTreeWithUnpressableButtons()]); + + const thrown = await underFakeTimers(() => handleLimrunIosAlert(client, 'accept')).then( + () => undefined, + (error: unknown) => error, + ); + expect(thrown).toBeInstanceOf(AppError); + const refusal = thrown as AppError; + expect(refusal.code).toBe('COMMAND_FAILED'); + expect(refusal.details?.reason).toBe(ALERT_CAPTURE_UNDECIDED_REASON); + expect(refusal.details?.dispatched).toBe('no'); + expect(client.tap).not.toHaveBeenCalled(); + }); +}); + +describe('button vocabulary', () => { + function alertWith(left: string, right: string): IosTreeNode { + const root = limrunAttAlertTree(); + return { + ...root, + children: [ + ...(root.children ?? []).slice(0, 2), + ...[left, right].map((label, index) => ({ + type: 'Button', + role: 'AXButton', + AXLabel: label, + pid: root.pid, + enabled: true, + frame: { x: 57, y: 511.67 + index * 56, width: 288, height: 48 }, + traits: ['Button', 'Scrollable'], + })), + ], + }; + } + + it('accepts the iOS variant grants, not just the bare Allow', async () => { + for (const label of ['Allow While Using App', 'Allow Once', 'Allow All the Time']) { + const client = clientWithTrees([ + alertWith('Ask App Not to Track', label), + limrunAppScreenTree(), + ]); + const result = await handleLimrunIosAlert(client, 'accept'); + expect(result.kind === 'alertHandled' && result.button).toBe(label); + } + }); + + it('dismisses Close App, the ANRB denial', async () => { + const client = clientWithTrees([alertWith('Close App', 'Wait'), limrunAppScreenTree()]); + const result = await handleLimrunIosAlert(client, 'dismiss'); + expect(result.kind === 'alertHandled' && result.button).toBe('Close App'); + }); +}); diff --git a/packages/provider-limrun/src/alert.ts b/packages/provider-limrun/src/alert.ts new file mode 100644 index 0000000000..4c88c89259 --- /dev/null +++ b/packages/provider-limrun/src/alert.ts @@ -0,0 +1,76 @@ +import type { + TreeAlertDriver, + TreeAlertObservation, + TreeAlertPress, + TreeAlertResult, +} from '@agent-device/capture-kit/alert-tree-driver'; +import { runTreeAlertAction } from '@agent-device/capture-kit/alert-tree-driver'; +import type { InstanceClient as LimrunIosClient } from '@limrun/api/ios-client'; +import { + chooseLimrunAlertButton, + type LimrunAlertCandidate, + type LimrunAlertInfo, + readLimrunAlertCapture, +} from './alert-detection.ts'; + +/** + * The four alert legs on a Limrun iOS direct session (issue #3349). There is no XCUITest runner + * to ask, so this backend supplies the shared tree-alert driver (`capture-kit`'s + * `runTreeAlertAction`, the one poll/confirm/same-alert/response owner every tree-based backend + * rides) with its own observation and press seams only: observation through the session's own + * element-tree capture and its host-root reader, action through the same point-touch seam every + * other touch operation rides. + */ + +/** The client methods the legs use; declared here so the legs are testable without a live instance. */ +export type LimrunIosAlertClient = Readonly<{ + elementTree: LimrunIosClient['elementTree']; + tap: LimrunIosClient['tap']; +}>; + +export type LimrunIosAlertResult = TreeAlertResult; + +export type LimrunIosAlertAction = 'get' | 'wait' | 'accept' | 'dismiss'; + +export type LimrunIosAlertOptions = Readonly<{ + timeoutMs?: number; + /** The daemon request's cancellation, refused at (never canceled inside) the transport. */ + signal?: AbortSignal; +}>; + +export async function handleLimrunIosAlert( + client: LimrunIosAlertClient, + action: LimrunIosAlertAction, + options?: LimrunIosAlertOptions, +): Promise { + const driver: TreeAlertDriver< + Awaited>, + LimrunAlertInfo, + LimrunAlertCandidate['buttons'][number], + LimrunAlertCandidate, + 'ios' + > = { + platform: 'ios', + timeoutMs: options?.timeoutMs, + ...(options?.signal ? { signal: options.signal } : {}), + // The Limrun WebSocket client exposes no abort on `elementTree`/`tap`, so the seam signal + // cannot cancel its transport: a started call always runs to its answer, and the leg waits + // for it (the documented one-call overrun) rather than dropping the answer. + capture: () => client.elementTree(), + findCandidate: (tree): TreeAlertObservation => + readLimrunAlertCapture(tree as Parameters[0]), + // Stricter than Android's by design: only the button whose own label states the asked-for + // choice may be pressed; there is no platform fallback key to invent on iOS. The shared + // driver names the refusal when this returns null. + choose: (candidate, act): TreeAlertPress | null => { + const button = chooseLimrunAlertButton(candidate.buttons, act); + return button ? { kind: 'press', button } : null; + }, + press: async (choice) => { + // `dismiss` never authorizes tracking: the chooser above presses only the one button the + // action's own label names, and a tap that falls back to a guess is never dispatched. + if (choice.kind === 'press') await client.tap(choice.button.x, choice.button.y); + }, + }; + return await runTreeAlertAction(action, driver); +} diff --git a/packages/provider-limrun/src/interaction-operations.test.ts b/packages/provider-limrun/src/interaction-operations.test.ts index 02b387bb65..5f88785bea 100644 --- a/packages/provider-limrun/src/interaction-operations.test.ts +++ b/packages/provider-limrun/src/interaction-operations.test.ts @@ -125,19 +125,20 @@ test('the iOS leg serves the simctl settings only and reads nothing back', () => ).toEqual(liveSessionUnavailable); }); -// R59: same split again — the Android leg reuses the local family's interactor, and the iOS -// direct session has no XCUITest runner to read a sheet from. -test('alert legs ride the Android interactor and are refused on the iOS leg', () => { +// R59 split, then #3349 closed the iOS half: the Android leg reuses the local family's +// interactor, and the iOS direct session reads system alerts from its element-tree capture and +// presses their buttons through its touch seam, so a live session serves all four legs on both +// platforms. +test('alert legs are available on both legs of a live session and refused without one', () => { for (const leg of ['readAlert', 'awaitAlert', 'acceptAlert', 'dismissAlert'] as const) { expect(limrunAlertOperationFacts(androidMobileDevice)[leg]).toEqual({ available: true }); - expect(limrunAlertOperationFacts(iosDevice)[leg]).toEqual({ - available: false, - reason: 'unsupported-provider-mode', - hint: 'Limrun iOS direct sessions do not expose alert inspection yet.', - }); + expect(limrunAlertOperationFacts(iosDevice)[leg]).toEqual({ available: true }); expect(limrunAlertOperationFacts(androidMobileDevice, liveSessionUnavailable)[leg]).toEqual( liveSessionUnavailable, ); + expect(limrunAlertOperationFacts(iosDevice, liveSessionUnavailable)[leg]).toEqual( + liveSessionUnavailable, + ); } }); diff --git a/packages/provider-limrun/src/interaction-operations.ts b/packages/provider-limrun/src/interaction-operations.ts index 48b5c66519..0faa394905 100644 --- a/packages/provider-limrun/src/interaction-operations.ts +++ b/packages/provider-limrun/src/interaction-operations.ts @@ -268,22 +268,18 @@ export function limrunClipboardOperationFacts( }); } -const alertUnavailableIos = unavailableFact( - 'unsupported-provider-mode', - 'Limrun iOS direct sessions do not expose alert inspection yet.', -); - /** * `alert` splits like every other interaction leaf: the Android leg rides the local family's own * interactor factory, which reads the same accessibility dump it always did; the iOS direct - * session has no XCUITest runner to read a sheet from, so all four legs refuse together. + * session reads system alerts out of the same element-tree capture its snapshot rides and presses + * their buttons through the same touch seam (issue #3349), so all four legs are available on + * both legs of a live session. */ export function limrunAlertOperationFacts( - device: DeviceInfo, + _device: DeviceInfo, liveSessionUnavailable?: RuntimeOperationUnavailability, ) { - const cell = - liveSessionUnavailable ?? (device.platform === 'android' ? available : alertUnavailableIos); + const cell = liveSessionUnavailable ?? available; return Object.freeze({ ...alertRuntimeOperationFacts({ read: cell, wait: cell, accept: cell, dismiss: cell }), }); diff --git a/packages/provider-limrun/src/ios-interactor-alert.test.ts b/packages/provider-limrun/src/ios-interactor-alert.test.ts new file mode 100644 index 0000000000..d3cb09b660 --- /dev/null +++ b/packages/provider-limrun/src/ios-interactor-alert.test.ts @@ -0,0 +1,174 @@ +import { describe, expect, it, vi } from 'vitest'; +import { ALERT_BUTTON_UNDECIDED_REASON } from '@agent-device/contracts/alert-contract'; +import { AppError } from '@agent-device/kernel/errors'; +import { + limrunAlertTreeWithoutButtons, + limrunAppAlertDialogTree, + limrunAppScreenTree, + limrunAttAlertTree, +} from './alert-detection.fixtures.ts'; +import { createLimrunIosInteractor, type LimrunIosSession } from './ios.ts'; +import type { IosTreeNode } from './snapshot.ts'; + +// The legs load `./alert.ts` per call (ADR-0019 laziness inside the provider), and that module +// carries the shared tree-alert driver from platform-android's mechanics facet. Warm it at +// collection, like `platform-android`'s own alert test does, so the first timed leg measures +// the legs, not the module graph's first transform. +await import('./alert.ts'); + +/** + * The four legs as the interactor exposes them: observation is one `elementTree()` read per + * probe, action is a `tap()` at the chosen button's center. Trees are the live Limrun captures. + * These cases resolve on the first probe each; budget loops are pinned with driven clocks in + * `alert.test.ts`, so nothing here waits production time. + */ +function sessionWithTrees(trees: Array) { + const elementTree = vi.fn(async () => { + const next = trees.length > 1 ? trees.shift() : trees[0]; + return [next]; + }); + const client = { elementTree, tap: vi.fn(async () => {}) }; + const session = { + platform: 'ios', + instanceId: 'limrun-alert-instance', + client, + } as unknown as LimrunIosSession; + return { interactor: createLimrunIosInteractor(session), client }; +} + +describe('readAlert', () => { + it('reports the live ATT alert with its buttons', async () => { + const { interactor } = sessionWithTrees([limrunAttAlertTree()]); + + const result = await interactor.readAlert!(); + + expect(result.kind).toBe('alertStatus'); + expect(result.status).toBe('visible'); + expect(result.alert).toMatchObject({ + platform: 'ios', + source: 'permission', + buttons: ['Ask App Not to Track', 'Allow'], + }); + }); + + it('reports no alert for the live app-owned dialog rather than claiming it', async () => { + const { interactor } = sessionWithTrees([limrunAppAlertDialogTree()]); + + const result = await interactor.readAlert!(); + + expect(result.kind).toBe('alertStatus'); + expect(result.alert).toBeNull(); + expect(result.status).toBe('none'); + }); +}); + +describe('awaitAlert', () => { + it('returns as soon as a capture shows the alert', async () => { + // The fixture serves the same alert on every read, so the read count is the assertion that + // proves this leg returns on the FIRST capture rather than polling past it. + const { interactor, client } = sessionWithTrees([limrunAttAlertTree(), limrunAttAlertTree()]); + + const result = await interactor.awaitAlert!({ timeoutMs: 5_000 }); + + expect(result.kind).toBe('alertWait'); + expect(result.alert).toMatchObject({ buttons: ['Ask App Not to Track', 'Allow'] }); + expect(client.elementTree).toHaveBeenCalledTimes(1); + }); +}); + +describe('acceptAlert', () => { + it('taps the affirmative choice and reports it, then re-reads to confirm', async () => { + const { interactor, client } = sessionWithTrees([ + limrunAttAlertTree(), + limrunAppScreenTree(), + limrunAppScreenTree(), + ]); + + const result = await interactor.acceptAlert!(); + + expect(result).toMatchObject({ + kind: 'alertHandled', + action: 'accept', + handled: true, + button: 'Allow', + // `centerOfRect` rounds: the live Allow frame centers at (201, 592). + coordinates: { x: 201, y: 592 }, + }); + expect(client.tap).toHaveBeenCalledTimes(1); + expect(client.tap).toHaveBeenCalledWith(201, 592); + }); + + it('refuses an alert whose buttons name no affirmative choice and taps nothing', async () => { + const { interactor, client } = sessionWithTrees([ + { + ...limrunAlertTreeWithoutButtons(), + children: [ + ...(limrunAlertTreeWithoutButtons().children ?? []), + { + type: 'Button', + role: 'AXButton', + AXLabel: 'Maybe Later', + pid: limrunAlertTreeWithoutButtons().pid, + enabled: true, + frame: { x: 57, y: 511.67, width: 288, height: 48 }, + }, + { + type: 'Button', + role: 'AXButton', + AXLabel: 'More Info', + pid: limrunAlertTreeWithoutButtons().pid, + enabled: true, + frame: { x: 57, y: 567.67, width: 288, height: 48 }, + }, + ], + }, + ]); + + const error = await interactor.acceptAlert!().then( + () => undefined, + (error: unknown) => error, + ); + expect(error).toBeInstanceOf(AppError); + const refusal = error as AppError; + expect(refusal.code).toBe('COMMAND_FAILED'); + expect(refusal.details?.reason).toBe(ALERT_BUTTON_UNDECIDED_REASON); + expect(refusal.details?.dispatched).toBe('no'); + expect(client.tap).not.toHaveBeenCalled(); + }); +}); + +describe('dismissAlert', () => { + it('taps the denying choice, never the affirmative one', async () => { + const { interactor, client } = sessionWithTrees([ + limrunAttAlertTree(), + limrunAppScreenTree(), + limrunAppScreenTree(), + ]); + + const result = await interactor.dismissAlert!(); + + expect(result).toMatchObject({ + kind: 'alertHandled', + action: 'dismiss', + button: 'Ask App Not to Track', + }); + // Exactly one tap, at the denying button: proving the affirmative one was never pressed. + expect(client.tap).toHaveBeenCalledTimes(1); + expect(client.tap).toHaveBeenCalledWith(201, 536); + }); + + it('accepts a replacement alert as a dismissal of the first', async () => { + const replacement = { + ...limrunAttAlertTree(), + children: (limrunAttAlertTree().children ?? []).map((child, index) => + index >= 2 ? { ...child, AXLabel: index === 2 ? 'No' : 'Yes' } : child, + ), + }; + const { interactor } = sessionWithTrees([limrunAttAlertTree(), replacement, replacement]); + + await expect(interactor.dismissAlert!()).resolves.toMatchObject({ + kind: 'alertHandled', + button: 'Ask App Not to Track', + }); + }); +}); diff --git a/packages/provider-limrun/src/ios.ts b/packages/provider-limrun/src/ios.ts index de9b69cbd9..9c38d6ca4d 100644 --- a/packages/provider-limrun/src/ios.ts +++ b/packages/provider-limrun/src/ios.ts @@ -406,6 +406,35 @@ class LimrunIosInteractor implements Interactor { const { setLimrunIosSetting } = await import('./ios-settings.ts'); return await setLimrunIosSetting(this.session, setting, state, appId, options); } + + /** + * The four alert legs observe through the same element-tree capture the snapshot rides and act + * through the same point tap every touch operation rides; `alert.ts` holds the host-root rule + * that separates a system alert from the app's own dialog. Loaded on the alert path like the + * other seam modules to keep this provider's declared import-time closure budget. + */ + async readAlert(options?: { signal?: AbortSignal }): Promise> { + const { handleLimrunIosAlert } = await import('./alert.ts'); + return await handleLimrunIosAlert(this.session.client, 'get', options); + } + + async awaitAlert(options?: { + timeoutMs?: number; + signal?: AbortSignal; + }): Promise> { + const { handleLimrunIosAlert } = await import('./alert.ts'); + return await handleLimrunIosAlert(this.session.client, 'wait', options); + } + + async acceptAlert(options?: { signal?: AbortSignal }): Promise> { + const { handleLimrunIosAlert } = await import('./alert.ts'); + return await handleLimrunIosAlert(this.session.client, 'accept', options); + } + + async dismissAlert(options?: { signal?: AbortSignal }): Promise> { + const { handleLimrunIosAlert } = await import('./alert.ts'); + return await handleLimrunIosAlert(this.session.client, 'dismiss', options); + } } async function prepareLimrunIosAsset( diff --git a/scripts/layering/package-boundaries.test.ts b/scripts/layering/package-boundaries.test.ts index bc3c73aac1..9a7b9eefbc 100644 --- a/scripts/layering/package-boundaries.test.ts +++ b/scripts/layering/package-boundaries.test.ts @@ -473,6 +473,7 @@ test('the real tree parses, declares, and passes R11', () => { ); assert.deepEqual([...captureKitPackage.exportTargets.keys()].sort(), [ '@agent-device/capture-kit', + '@agent-device/capture-kit/alert-tree-driver', '@agent-device/capture-kit/android-replacement-surface-occlusion', '@agent-device/capture-kit/app-log-durable-resource', '@agent-device/capture-kit/app-log-files', diff --git a/src/core/interactors/android.test.ts b/src/core/interactors/android.test.ts index b403e84c29..21bb580ab6 100644 --- a/src/core/interactors/android.test.ts +++ b/src/core/interactors/android.test.ts @@ -98,6 +98,22 @@ test('a device-only session releases the helper after fill and scroll', async () }); }); +test('alert reads ride the session-scope helper, not a per-probe cold start', async () => { + // The executed Android smoke failure: an alert probe without the session scope cold-started + // the snapshot helper inside the window that bounds the probe, so the post-press confirm read + // was aborted before the device answered. Alert captures scope like every other read. + snapshotAndroidMock.mockResolvedValue(makeAndroidSnapshotCapture([])); + const interactor = createAndroidInteractor(device, undefined, { + appBundleId: 'com.example.app', + }); + + if (!interactor.readAlert) throw new Error('Android interactor must expose readAlert'); + await interactor.readAlert(); + + const options = snapshotAndroidMock.mock.lastCall?.[1]; + expect(options?.helperSessionScope).toBe('daemon-session'); +}); + test('a transient snapshot reaches the Android capture with its settle deadline', async () => { snapshotAndroidMock.mockResolvedValue(makeAndroidSnapshotCapture([])); diff --git a/src/core/interactors/android.ts b/src/core/interactors/android.ts index e61b5f94af..b9bb6da9d1 100644 --- a/src/core/interactors/android.ts +++ b/src/core/interactors/android.ts @@ -55,8 +55,15 @@ export function createAndroidInteractor( const helperSessionScope = androidHelperSessionScope(runnerContext?.appBundleId); const alertOptions = (timeoutMs?: number) => ({ ...(timeoutMs === undefined ? {} : { timeoutMs }), - captureNodes: async () => { - const capture = await snapshotAndroid(device, { includeHiddenContentHints: false }); + ...(runnerContext?.signal ? { signal: runnerContext.signal } : {}), + captureNodes: async (seam: { signal: AbortSignal }) => { + const capture = await snapshotAndroid(device, { + includeHiddenContentHints: false, + signal: seam.signal, + // App-backed sessions keep the snapshot helper warm; a command-scoped probe would + // force-stop and cold-restart it inside the window that bounds the probe. + helperSessionScope, + }); return buildSnapshotState(androidSnapshotPublicationInput(capture), undefined).nodes; }, }); diff --git a/website/docs/docs/commands.md b/website/docs/docs/commands.md index e0014faa7f..5c7486ab93 100644 --- a/website/docs/docs/commands.md +++ b/website/docs/docs/commands.md @@ -397,7 +397,11 @@ agent-device alert dismiss - `wait_stable_timeout`: the UI did not settle. - Branch on `readableCaptures`, `waitedMs`, `matches`, and `firstMatch` instead of parsing error text. `firstMatch` carries identity and text only; absence failures make no visibility or rect claim. - Polling wait timeouts, and a replayed wait refused with `wait_landmark_identity_mismatch`, also carry `captures`, `readableCaptures`, and `polls` so you can see where the budget went. Each poll has `startedMs`, `durationMs`, and an `outcome`: `readable`, `unreadable`, `retriable`, `deadline`, `runner-restart`, or `readiness`. Long waits keep the first 5 and last 25 polls. A wait that never saw a readable capture reports the error its polls hit, with that error's code and retry details, instead of a generic timeout; `wait stable` and a never-readable strict absence keep their own diagnostics. `logPath` links the full request log. -- `alert` (same as `alert get`) inspects or handles system alerts on iOS simulator, macOS desktop, and Android permission dialogs. On iOS it also covers app-owned modal popups with native blocking markers. +- `alert` (same as `alert get`) inspects or handles system alerts on iOS simulator, macOS desktop, and Android permission dialogs. On iOS it also covers app-owned modal popups with native blocking markers. The `[timeout]` argument applies to `alert wait` only, on every backend: `alert wait ` takes a budget up to 2147483647 ms (~24.8 days), larger is refused as `INVALID_ARGS` before anything runs, while `get`/`accept`/`dismiss` run their own fixed per-backend budgets and ignore it. +- On the snapshot-derived backends (Android, Limrun iOS) `alert get --json` answers with a typed `status`: `visible`, `none`, or `undecided` (an alert-shaped capture that resolves to no single alert). Branch on `status`, not on the message. +- On those backends, `alert accept` presses the lasting grant when a permission sheet offers several: Android orders its grants by the platform's own ids (foreground-only, then always, then one-time) whatever the button language, and elsewhere the English label table decides ("while using the app" over "only this time"). A sheet whose choices no rank orders — or that names the same choice at two points — is refused with `alert_button_undecided` and `dispatched: no`, never pressed by position; read the labels with `alert get` and press your choice by label/ref. +- On those backends, when `alert accept`/`dismiss` fails before pressing anything, `error.details.reason` names why (a capture or transport failure that is not a driver refusal may instead surface as its own error without a `reason`). A budget never cancels a running capture or press: it only ends the polling loop, so every check that started inside the budget finishes and counts. Before any press, `dispatched: no` and the reason is `alert-not-found` (the action budget ended with no alert on screen), `alert_capture_undecided`, or `alert_button_undecided` (the sheet is too ambiguous to act on). After a press, `dispatched` is absent and the reason is `alert-window-expired`; “did not dismiss the visible alert” means the latest post-press capture that answered named the same dialog again, “could not be confirmed” means the press went out but the latest read was unresolvable or never answered, so the dialog may already be gone. If you cancel the request at any point from the press onward — the tap in flight, or the post-press checks — the cancellation reports `dispatched: unknown`: the device may already have received the tap. Run `alert get` before acting again. +- On those backends a slow capture is never lost to its budget: a check that straddles the deadline still answers, which is why a quiet `alert accept`/`dismiss` always reports `alert-not-found` and never a timing-dependent failure (`alert get` answers `status: none` and `alert wait` times out on a quiet screen). The cost is a wait: the command stays open until the capture it started returns. On Android your cancellation stops the capture and the tap (the adb child is killed). On Limrun a request already sent to the instance cannot be stopped: the command ends when that last request answers (the Limrun client gives a request that stops answering its own ~30-second timeout) or the connection drops, and cancelling does not free it early. If a Limrun alert command stalls, restarting the instance session is what clears it. - Use `alert get` for a quick check, and `alert wait ` only when a prompt may appear after async work. Don't use `settings permission` to answer a dialog already on screen; it is for setup before a flow. - `accept` and `dismiss` are sent once on every platform and never replayed, even if the response is lost. On iOS the runner activates the button once, then only observes until the alert disappears, changes, or the deadline expires. A changed alert does not prove a permission was granted; verify the app outcome. - On the Apple XCUITest runner, `dismiss` activates only a button its decline list recognizes (`Cancel`, `Don't Allow`, `Ask App Not to Track`, and similar English labels), or the close button of a popover whose native marker reads "dismiss popup". It never taps anything else: unknown, reworded, or fully localized decline copy fails with `error.code: ALERT_BUTTON_NOT_FOUND` and `details.dispatched: no`, tapping nothing, rather than guessing at an affirmative ([#3627](https://github.com/callstack/agent-device/issues/3627)). A button choice that finds no button to name answers with that code and `dispatched: no` for `accept` too. To answer a prompt with unrecognized wording, read the labels with `alert get`, then press the button by label through `snapshot` + `press @ref`. The macOS helper's `dismiss` keeps its own rules (the app's cancel-button attribute, its label list, and a last-button fallback). diff --git a/website/docs/docs/limrun.md b/website/docs/docs/limrun.md index d95345efa1..b5e259a400 100644 --- a/website/docs/docs/limrun.md +++ b/website/docs/docs/limrun.md @@ -71,7 +71,7 @@ agent-device disconnect On Android, agent-device connects to Limrun over an ADB tunnel. Snapshots use the Android snapshot helper, installs upload the app through Limrun asset storage, and port reverse runs over ADB, including the usual reverse setup for a local Metro server. -On iOS, agent-device supports app lifecycle commands, snapshots, screenshots, taps, text input, scrolling, the home button, clipboard read and write, and app installation. `open --launch-args` relaunches the app with those arguments through simctl. iOS cannot reverse a remote device port to a local host port, so for Metro use a publicly reachable HTTPS endpoint or bridge URL instead of a local-only address. +On iOS, agent-device supports app lifecycle commands, snapshots, screenshots, taps, text input, scrolling, the home button, clipboard read and write, app installation, and system alerts. `alert get/wait/accept/dismiss` read the alert from the same element tree as snapshots and dispatch the selected button; a choice is pressed only when its English label states the asked-for action (`Allow`, `Don't Allow`, `Ask App Not to Track`, and similar), so a localized alert is reported but never pressed. A sheet offering two grants (`Allow Once` / `Allow While Using App`) presses the preferred lasting grant from the declared table, never a coin flip. App-owned dialogs are not system alerts there, so use `snapshot -i` and then `press @ref` or a label selector for the visible control. `open --launch-args` relaunches the app with those arguments through simctl. iOS cannot reverse a remote device port to a local host port, so for Metro use a publicly reachable HTTPS endpoint or bridge URL instead of a local-only address. On iOS, `settings` supports `appearance`, `permission`, `location`, and `clear-app-state`, with the same arguments as a local simulator. `clear-app-state` resets the app's data container through Limrun's soft reset. That reset relaunches the app once before agent-device stops it, so the app may run once after its data is cleared. `reset-keychain`, `wifi`, `airplane`, `mobile-data`, `faceid`, `touchid`, and `text-size` are unsupported, and you cannot read back any iOS setting. On Android, `settings mobile-data on|off` runs over the ADB tunnel like on a local device.