Conversation
iOS permission automation now drives agent-device's maintained XCUITest runner through its Node client instead of a Harness-owned Swift agent, an ad-hoc HTTP transport and a bespoke xcodebuild cache. The user-facing contract is unchanged: with `permissions: true` Harness watches for system permission prompts and taps the first known positive button. The watchdog is now a Node polling loop that runs only while the app under test is running, so SpringBoard's home screen is never walked. Removes the Xcode project, the seven XCTest modules, the `harness xctest build` command and the build/boot overlap strategy. Adds an optional `codeSign.runnerBundleId` for personal teams on physical devices, and a vitest config so this package's tests run in CI at all.
`harness ci plan-ios-runner-cache` prints a cache key derived from the installed agent-device version and `xcodebuild -version`, and action.yml uses it to cache the runner build in place of the removed XCTest agent cache. The action's required CLI protocol moves to 2 because the action now invokes a subcommand older CLIs do not have.
…ssion agent-device's `interactions.press` refuses to run without an open session (`SESSION_NOT_FOUND`), unlike `command.alert`, which attaches an implicit runner session per call. Open one session bound to SpringBoard during prepare so taps land; the app under test is never opened or otherwise touched by agent-device.
- prepare() races agent-device commands against the init abort signal and stops the daemon when it loses, so no daemon, xcodebuild or runner survives holding the host-global device claim; the prepare budget is derived from platformReadyTimeout instead of a fixed 600 s. - dispose() stops the daemon first and then drains the watchdog with a bound, instead of waiting out an in-flight 20 s poll. - The daemon stop runs through runCommand (SIGKILL-bounded) and reports failures at warn with the recovery command; the session is closed first. - Unclassified command failures are reported at warn with code and hint, and the watchdog disables itself after three consecutive ones. - Positive labels are tapped in trimmed form, and a label carrying a quote or backslash is refused rather than producing a broken selector. - User-provided AGENT_DEVICE_IOS_* values are never overwritten. Pins agent-device to 0.21.0 exactly, bounds `xcodebuild -version` in the runner cache planner and annotates CI when the cache is disabled, and adds the ADR merge-gate step asserting no agent-device process survives an iOS e2e job.
agent-device refuses to clean a derived-data path supplied through AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH, which Harness always sets, so a runner cache left over from a different agent-device, Xcode or SDK version failed every run with an error naming an environment variable the user never set. The cache directory belongs to Harness, so clear it and retry the prepare once; a path the user provided is still left untouched and the original error surfaces.
Each watchdog tick now calls `alert accept` directly instead of `alert get` followed by `press`, so a two-button prompt costs one runner round trip rather than two. The runner's accept can only activate a button from its accept list, or the sole button of a one-button alert when that label is not a dismiss label, so it can never press deny or cancel. Prompts it refuses -- above all the three-button location sheet -- report "alert accept button not found" and fall back to the previous `alert get` plus label press. A dismissed prompt skips the poll gap so a following prompt is handled straight away, bounded so a prompt that never clears cannot turn the loop into a busy wait.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is this?
iOS permission automation (
permissions: true) no longer runs on a Harness-maintained XCUITest target. It runs on agent-device, which already owns a maintained runner, and the Harness side shrinks to one TypeScript module.The decision and its trade-offs are recorded in
docs/internal/adr/0001-replace-xctest-agent-with-agent-device.md, including three dated amendments for facts that turned out differently during implementation.The user-facing contract is unchanged: with
permissions: true, Harness watches for system permission prompts while the app under test is running and answers them, and physical devices still needdevice.codeSign.Removed:
packages/platform-ios/xctest-agent/(an Xcode project and ~470 lines of Swift), seven TypeScript modules (xctest-agent*.ts,xctest-command.ts,startup-strategy.ts) and their tests, theharness xctest buildCLI command, thebuildXCTestAgentexport and its four types, the build/boot overlap strategy, and thexctest-agent-simulatorcache handling in the action. Harness now contains no Swift, no Xcode project, noxcodebuildinvocation and no HTTP transport for permission automation.Added:
packages/platform-ios/src/permission-agent.ts, aharness ci plan-ios-runner-cachehelper for the action's cache key, an optionaldevice.codeSign.runnerBundleId,engines.node >= 22.12on every package, and avite.config.tsforplatform-apple, which had no test target at all — its existing tests had never run in CI.Breaking paths
com.callstack.agentdevice.runner(plus its.uitestscompanion). Provisioning profiles created forHarnessXCTestAgentno longer apply. Personal teams can point the runner at their own identifier with the newdevice.codeSign.runnerBundleId..xctestrunartifacts.HARNESS_IOS_XCTESTRUN_FILEandHARNESS_IOS_XCTEST_DERIVED_DATA_PATHare kept, but the artifact must be anAgentDeviceRunnerbuild produced byagent-device prepare ios-runner --json.harness xctest buildis removed. The docs point toagent-device prepare ios-runner.engineson every package. This is agent-device's floor; Harness previously declared none.harnessActionProtocolmoves to 2, because the action now calls aharness cisubcommand older CLIs do not have.How does it work?
After the simulator is booted and the app installed, Harness calls
command.prepare({ action: 'ios-runner' })against its own daemon, whose state dir is~/.agent-device/harness/<projectRootHash>so it never replaces or inherits environment from a developer's own daemon. Device claims are host-global, so arbitration between the two stays correct, and a claimed device surfacesDEVICE_IN_USEverbatim with its recovery hint and is never retried.A Node loop then answers prompts. Each tick calls
alert accept, which detects and presses in one round trip; when agent-device refuses the prompt — the three-button location sheet is the usual case — it falls back toalert getplus apressby label from the Harness positive-label list. The loop is serialised (one command in flight), abort-driven, and runs only while the app session reports the app running: polling with SpringBoard's home screen foregrounded walks the whole icon grid and trips the runner's watchdog.ALERT_NOT_FOUND,RUNNER_BUSYand timeouts are transient; anything else is reported and disables the watchdog after three consecutive failures rather than failing the run.Harness stays the sole owner of the app lifecycle —
simctl/devicectllaunch, kill and relaunch it exactly as before. agent-device is given one session bound tocom.apple.springboard, becauseinteractions.pressrefuses to run session-less; it never opens the app under test.Teardown stops the daemon with
--cleanand copies the session'srunner.loganddaemon.loginto.harness/logs, so the xcodebuild output the action used to upload is still there.The runner build is cached in
.harness/cache/agent-device-runner;action.ymlkeys that cache on the installed agent-device version plusxcodebuild -version, with no restore-key prefixes, since a near-miss entry cannot be reused.ADR merge gates
E2E iOS job with
permissions: true— pending; it runs on this PR.Signed physical iPhone with
device.codeSign— outstanding. No signed device was available to me. A maintainer with a provisioned iPhone needs to run the playgroundiphone-16-prorunner and confirm the runner signs and attaches under the new bundle ids, including thecodeSign.runnerBundleIdpath for personal teams.Startup benchmark — measured locally on iPhone 17 Pro / iOS 26.4, playground permissions test:
prepare→ session readyA forced cold runner rebuild cost 42 s locally. A true CI cold build was not reproducible on this machine and should be read from the first job run on this branch.
No process survives a run — a new
if: always()step on both iOS jobs fails whenpgrep -f 'AgentDeviceRunner|xcodebuild test-without-building|agent-device/dist/src/internal/daemon'matches anything.Things a reviewer should know
The native runtime disconnected during test executionmoments after the prompt was accepted. The immediately repeated run passed, and the runner log for the failed one shows leftover app-switcher state on the simulator. I could not reproduce it and have not explained it.Yes/Done/Open, or the sole button of a one-button system alert. It can never press a deny or dismiss button (verified against the runner'schooseAlertButton). Documented.AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH, which Harness always sets, so a cache from a different agent-device or Xcode version used to fail every run. Harness now clears its own cache directory and retries prepare once; a path the user provided is left untouched.isAppInstalledletsxcrun simctl appinfo's exit code 3 escape as an unhandledSubprocessErrorinstead of falling through to the install path, so a run against a simulator without the app installed dies during platform setup. Reproducible onmain.Verification
pnpm install;pnpm exec nx run-many -t lint,typecheck,build,test --projects="packages/*"(20 projects green;platform-applenow contributes 9 files / 66 tests);pnpm exec nx release plan:check; and the two local simulator runs above, after which no agent-device daemon,xcodebuildorAgentDeviceRunnerprocess from the run remained.Why is this useful?
The Xcode/SDK cache keys, physical-device signing, external
.xctestruninjection, graceful-shutdown races (#139, #164, #176) and CPU tuning (#163) that this layer existed to manage are now someone else's maintained problem, and every Xcode release stops being a potential break in this repository. What is left is a small, tested Node module with explicit error classification and bounded teardown.Along the way the replacement closes gaps the old agent had: the watchdog disables itself with a clear message instead of failing a run, teardown is bounded rather than waiting out an in-flight command, a stale runner cache heals itself, and
platform-apple's test suite runs in CI for the first time.