fix: rotate synthesized iOS taps into native screen space

[ysamlan]

Summary

Synthesized iOS taps/drags land in the wrong place once the device is rotated — in landscape, taps on a sidebar row, keyboard key, or any element-by-ref miss their target. XCUICoordinate taps are unaffected.

XCSynthesizedEventRecord pointer paths are consumed in device-native (portrait) screen coordinates, but synthesizedTapAt/synthesizedDragAt pass the element frame's interface-oriented coordinates through unchanged (they coincide in portrait but differ by a 90° rotation in landscape). The synthesized path already passes the app's interfaceOrientation to the event record, but the path points still need to be in native space. From bisecting it looks like this bug was introduced by #702 and the fix should be in the scope of ADR 0005. The pre-#702 XCUICoordinate.tap() was fine because XCTest rotates internally.

Fix: rotate the point into native space using the app's interfaceOrientation before synthesizing (identity fallback when the orientation can't be read, so a missing reading is never worse than today). Adds testNativeSynthesizedPointRotatesByInterfaceOrientation as a regression test.

Test plan

See testNativeSynthesizedPointRotatesByInterfaceOrientation.

Deterministic unit test (no device) — passes with the fix, fails against the pre-fix pass-through.

To reproduce the original bug and its fix end-to-end on a simulator you can drive a rotation-supported app with a click target through:

agent-device --udid <SIM> open <your.test.app>
agent-device rotate landscape-left
REF=$(agent-device snapshot | grep '<a tappable row/button>' | grep -oE '@e[0-9]+' | head -1)
agent-device click "$REF"     # this branch or before #702: tap lands on target; current main: misses

Integration tests would require adding a rotation aware device test -- I have one locally to prove this out but maybe a bit heavy for this one case. Happy to add as a follow-up if useful.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR adjusts synthesized tap/drag coordinates to account for interface orientation by rotating “interface-oriented” points into the device-native (portrait) coordinate space used by synthesized events.

Changes:

• Add nativeSynthesizedPoint(...) to rotate points based on UIInterfaceOrientation values.
• Update synthesized tap and swipe to use rotated coordinates.
• Expose an Objective-C helper to read the app’s interfaceOrientation, and add a unit test for the rotation logic.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Interaction.swift	Adds point-rotation helper + updates synthesized gestures to use it; adds a rotation test.
ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerSynthesizedGesture.m	Adds interfaceOrientationForApplication: via dynamic selector lookup.
ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerSynthesizedGesture.h	Declares orientation helper and documents return values.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[thymikee]

Pushed a follow-up commit that applies the same native-coordinate conversion to synthesized transform gestures, not just tap/drag. This covers the pinch, rotate-gesture, and transform-gesture paths that all go through the private XCTest synthesis API.

Also made the test app Gesture Lab rotation-capable and surfaced its gesture status in the visible landscape area so this can be verified against our own fixture.

Validation I ran locally:

• pnpm build
• pnpm build:xcuitest
• test-app tsc --noEmit
• iOS simulator, Agent Device Tester via Expo Go, landscape-left: Gesture Lab ended at pan changed yes, pinch changed yes, rotate changed yes.