Skip to content

test(apple-runner): characterize target transitions and cache lifetimes #2968

Description

@thymikee

Characterize runner target transitions and cache lifetimes before changing ownership

Deletion-first acceptance

This is part of #2803's simplify-and-shrink initiative. Before production simplification, name the production mechanism, duplicated decision, state, fallback, forwarding layer, or public/internal interface this change removes. Compare deletion and inlining against extraction. Moving the same state into another file is not a deletion case.

Every PR reports baseline/final SHAs; added, deleted and net production lines; test/fixture/documentation changes separately; and the exact mechanisms removed. Use a rename-aware diff (git diff --numstat -M BASE...HEAD) and account for all affected production paths, including destination packages. Moving tests out of production files, generated output, formatting churn, and moving code elsewhere are not production-code deletion. Preserve readable code; no compressed formatting to hit a number.

Default for simplification work: net production reduction. A required behavior-preserving size split may be line-neutral. Growth needs an explicit, quantified explanation of the correctness or type guarantee it buys, why deletion/inlining cannot achieve it, and what old mechanism is removed; it is not automatically acceptable because ownership looks cleaner. If the only benefit is relocating fields or adding a wrapper, defer the change. Required regression tests and a minimal correctness fix are allowed to grow without inventing unrelated deletion to offset them.

For this test/characterization task, name the uncertainty or regression gap resolved instead of a production deletion. Expected production behavior delta is zero; report test/harness growth separately.

Priority and scope gate

Scheduling: later characterization, after the active correctness/small-deletion and host-cleanup work; retain needs-triage until scheduled. Its output must include a go/no-go decision for #2969: either demonstrate a concrete reset/identity correctness gap or name repeated transition logic/state that can be removed. If neither exists, retain the characterization evidence and defer #2969. Tests may grow; no production refactor is required to close this characterization task. Serialize with all other shared runner Swift edits.

Parent: #2803. Readiness: queued characterization, not authorization to redesign reset policy. Evidence baseline: 5712a3552e209c9c853bc1f42cb3e007f1213139; recheck current main. This is the prerequisite for the target-transition owner issue. #2911's RunnerMainOwnedState and #2882's actor source guards remain separate work.

Problem and evidence

Main-actor isolation is now enforced, but target state has intentionally different lifetimes. RunnerTests+Lifecycle.swift:195–255,385–389 separately binds, invalidates, refreshes and resets the target. The existing tests cover several individual cases without presenting one explicit transition/lifetime contract. An ownership extraction must not assume that all target-related state resets together.

Publish the current contract in a focused runner architecture note beside the owning documentation, linked from the runner README. It should cite executable owners rather than duplicate their implementations. Add characterization tests to the existing source-matched test files; do not change production behavior to make a more attractive matrix.

Required matrix and implementation contract

Record, for every transition, the before/after identity, text witness, warmup exemption, log deduplication, stabilization deadline, accessibility health, post-snapshot delay, channel penalty and private-AX depth memory. Separate stored state from whether a lookup can still match it.

Starting observations to verify against execution:

Transition Current effects that must be characterized
Host-app binding Bind host handle with nil bundle/PID; clear witness, warmup and log markers.
Ordinary target activation Bind requested handle/bundle/PID; clear the same short-lived state; start stabilization. It does not explicitly clear penalty/depth memory.
Target invalidation, including missing bundle Clear handle/bundle/PID and short-lived state; do not infer that all other caches were cleared.
Observed positive PID replacement Rebind handle/PID, clear short-lived state and both capture memories, arm one warmup exemption and start stabilization.
External relaunch reset Invalidate target, explicitly clear both capture memories and start stabilization; warmup is not armed here.
Successful/sparse/fatal capture Record health and post-snapshot behavior through the existing main-state writers; deferred writes retain their current ordering.

AXSnapshotFallback.swift:109–157 additionally requires bundle plus positive PID and unexpired time to reuse accepted depth; no PID means no stored hint, and first-rung success must not renew it. SnapshotCapturePlan.swift:133–162 holds one bundle-scoped penalty, with nil bundle matching any target. SnapshotTiming.swift:60–85 independently owns the one-shot warmup exemption, consumed by an actual capture plan rather than a modal-only observation. Do not combine these owners.

Completion and validation

Exercise A→B→A, same-bundle changed/unchanged/missing PID, AX invalidation, explicit external relaunch, missing bundle, modal-only capture and deferred write followed by the next command. Reuse LifecycleCacheTests, CommandDispatchTests, SnapshotCaptureTargetTests, AXSnapshotFallbackTests, AXRecoveryConformanceTests and SnapshotTimingTests. Tests must invoke production transition/lookup behavior; seeded preconditions alone are not evidence that a transition works.

Before: separate examples and implicit lifetime assumptions. After: every transition effect that the dependent ownership refactor will move has executable coverage linked from the matrix, with surprising existing behavior preserved. An unexercised effect blocks the dependent refactor; alternatively explicitly exclude that effect from its scope. Do not close this prerequisite as complete while a required row is unproved. Distinguish proven rows from deferred rows in the completion report. Plant omitted-reset and incorrectly-renewed-memory mutations to prove the relevant assertions distinguish them. Suspected defects become separately scoped issues, not silent policy corrections.

Run pnpm check:xctest-selection, unit-enabled iOS and macOS runner builds, the corresponding selected XCTest cases, and pnpm check:affected --run. Use source guards for lane selection, with simulator cases for actual XCTest identity/activation behavior. Record exact-head evidence and remaining physical-device limitations. No new generation cache, live-policy rewrite, timing changes or cosmetic source reorganization is included.

Source paths in this issue are rooted at apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/; extension names abbreviate the RunnerTests+ prefix where stated. Tests live in its UnitTests/ directory. For a fresh worktree, run pnpm install --frozen-lockfile && pnpm build before the focused checks. Read docs/agents/testing.md and docs/agents/device-verification.md for native lane selection and build freshness.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions