Skip to content

refactor(apple-runner): classify AX snapshot failure once at acquisition #2972

Description

@thymikee

Classify native AX snapshot failure once at acquisition

Deletion-first acceptance

This is part of #2803's simplify-and-shrink initiative. Before production simplification, name the production mechanism, duplicated decision, state, fallback, forwarding layer, or public/internal interface this change removes. Compare deletion and inlining against extraction. Moving the same state into another file is not a deletion case.

Every PR reports baseline/final SHAs; added, deleted and net production lines; test/fixture/documentation changes separately; and the exact mechanisms removed. Use a rename-aware diff (git diff --numstat -M BASE...HEAD) and account for all affected production paths, including destination packages. Moving tests out of production files, generated output, formatting churn, and moving code elsewhere are not production-code deletion. Preserve readable code; no compressed formatting to hit a number.

Default for simplification work: net production reduction. A required behavior-preserving size split may be line-neutral. Growth needs an explicit, quantified explanation of the correctness or type guarantee it buys, why deletion/inlining cannot achieve it, and what old mechanism is removed; it is not automatically acceptable because ownership looks cleaner. If the only benefit is relocating fields or adding a wrapper, defer the change. Required regression tests and a minimal correctness fix are allowed to grow without inventing unrelated deletion to offset them.

Priority and scope gate

Early small deletion: remove downstream AX message classification while retaining necessary native-edge normalization. Prefer deleting the redundant branch/helper over introducing a new error hierarchy or classifier abstraction. Report its production delta separately from added negative regression cases. Serialize shared runner Swift edits; #2911 is already merged.

Problem and evidence

The snapshot acquisition edge already turns a recognized AX serialization failure into SnapshotCaptureFailure(code: axSnapshotErrorCode, ...). Downstream policy nevertheless classifies it again from the human message. That allows wording on an unrelated failure to affect recovery, the quality reason, and terminal target invalidation.

Verified against origin/main at 5712a3552e209c9c853bc1f42cb3e007f1213139: apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotAcquisition.swift:116–134 performs native classification, :159–181 constructs the typed failure and then permits message-based reclassification. RunnerTests+SnapshotCapturePlan.swift:361 retains AX failure for terminal policy, and RunnerTests+SnapshotPresentationInvariant.swift:17 derives its quality reason. The latest placeholder changes do not change these sites.

Intended design

Keep unavoidable private-API text interpretation at the native acquisition edge. Above that edge, a snapshot failure's machine code is the sole AX classification authority. Preserve the current recognized native messages, public code, diagnostic message, hint, raw-snapshot failure behavior, and interactive fail-closed behavior.

The intended interface is deliberately small:

static func isAxSnapshotFailure(_ failure: SnapshotCaptureFailure) -> Bool {
  failure.code == axSnapshotErrorCode
}

Use the existing failure carrier; do not add a second isAXFailure flag that can disagree with its code. Keep or name a private acquisition-edge classifier for actual native error/exception evidence. If NSError or NSException exposes reliable structured evidence, characterize it first and prefer it only where proven. This issue does not depend on discovering a new native code or eliminating all native-edge text parsing.

Implementation and acceptance

  1. Inventory every constructor of SnapshotCaptureFailure and every consumer of isAxSnapshotFailure, including private-AX and presentation error mapping. Prove each genuine AX path reaches the same machine code before tightening consumers.
  2. Add paired behavioral regressions through existing product seams: a typed AX failure with neutral wording remains AX; an unrelated typed failure containing identical AX-looking prose remains unrelated. Assert both quality classification and the terminal capture-plan consequence.
  3. Make downstream classification code-only. If a producer currently relies on downstream text classification, move that normalization to its actual acquisition edge while retaining its observed classification and public error details.
  4. Retain native exception handling, source diagnostic text, and the capture-plan ordering. Keep tests with the owning modules; do not add test-only production exports.

Before/after measures: there is no downstream AX message sniff after normalization; all previously recognized native AX failures keep their code and recovery result; changing diagnostic prose cannot alter quality or terminal policy. Plant the old downstream text fallback and observe the unrelated-code negative fail. A native-edge fixture still recognizes each currently supported signature, including the closest negative that resembles an AX message but lacks the full recognized signature.

Validation

Extend the owning snapshot acquisition/quality/terminal tests, including UnitTests/RunnerTests+SnapshotTests.swift, RunnerTests+SnapshotCapturePlanTests.swift, and RunnerTests+SnapshotPresentationInvariantTests.swift as appropriate to the actual call sites. Pure classification belongs in the macOS host XCTest lane; native exception capture semantics belong in the iOS simulator lane. Preserve RunnerTests+SnapshotCapturePlanOccupancyTests.swift as integration proof of recovery while abandoned native work remains outstanding.

Run pnpm check:xctest-selection, pnpm build:xcuitest:ios, and pnpm build:xcuitest:macos; run the selected host/simulator XCTest cases through the repository's existing harness and record the selected cases. Finish with pnpm check:affected --run and all reported exact-head obligations. Do not substitute compilation for XCTest evidence or claim a real-device AX failure was reproduced unless measured.

Scope and readiness

Ready for implementation after the constructor inventory; unexpected producer classification must be characterized before changing it. Independent child of umbrella #2803, with no dependency on broader capture-plan extraction. No retry, timeout, fallback order, error-message cleanup, or wire-schema changes. Expected effort: S–M; risk: medium because AX classification controls fail-closed behavior.

Source paths in this issue are rooted at apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/; extension names abbreviate the RunnerTests+ prefix where stated. Tests live in its UnitTests/ directory. For a fresh worktree, run pnpm install --frozen-lockfile && pnpm build before the focused checks. Read docs/agents/testing.md and docs/agents/device-verification.md for native lane selection and build freshness.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions