diff --git a/docs/adr/0011-interaction-guarantee-contract.md b/docs/adr/0011-interaction-guarantee-contract.md index 8d8e5baf86..2dbb3582b0 100644 --- a/docs/adr/0011-interaction-guarantee-contract.md +++ b/docs/adr/0011-interaction-guarantee-contract.md @@ -15,6 +15,7 @@ trade-offs independently: | --- | --- | --- | | `runtime-selector` | daemon tree capture → `resolveSelectorChain` → guards → coordinate tap | full semantics | | `runtime-ref` | session snapshot → ref lookup → guards → coordinate tap | full semantics | +| `target-drag` | independently resolve source + destination → guards → one pointer plan | dual-endpoint drag semantics | | `direct-ios-selector` | selector sent to the XCTest runner, which queries and taps natively | saves a full snapshot round trip | | `native-ref` | `backend.tapTarget`/`fillTarget` for `click @ref` / `fill @ref` | saves resolution round trips | | `coordinate` | raw x/y tap | escape hatch; semantics intentionally minimal | @@ -266,6 +267,10 @@ dimensions as their only frame source; cheaper frame sources such as intentionally excluded because they can diverge from full-screen screenshot coordinates on affected simulators. +Target-authored `gesture drag` is classified as the `target-drag` path above because it resolves two +elements and therefore inherits the element guarantees and ADR 0012 identity requirements for both. +Coordinate-authored drag remains outside this matrix. + Two-contact pan/pinch/rotate/transform planning is now owned by the typed gesture-plan contract in [ADR 0013](0013-unified-gesture-plans.md). It remains a sibling of this element-targeting matrix: coordinate gestures do not acquire diff --git a/docs/adr/0012-interactive-replay.md b/docs/adr/0012-interactive-replay.md index 8ddeafef2b..0a608d6425 100644 --- a/docs/adr/0012-interactive-replay.md +++ b/docs/adr/0012-interactive-replay.md @@ -132,6 +132,15 @@ the corresponding provider contract cases in the same change. ### 3. Versioned `.ad` target-binding evidence +> **Amendment (#1567): dual-endpoint target evidence.** An action that resolves two elements records +> one `# agent-device:targets-v1 {"source":{...},"destination":{...}}` annotation instead of a +> single `target-v1`. Each nested value is a complete `TargetAnnotationV1` under the same identity, +> normalization, field-size, and verification rules. The wrapper is capped at 8,320 bytes. Replay +> verifies both selectors and carries two post-resolution guards before dispatch; either endpoint's +> selector miss, identity mismatch, unverifiable evidence, or dispatch-time guard mismatch refuses +> the entire action before pointer-down. Unknown future `targets-vN` annotations remain ordinary +> comments to an older reader, matching the single-target versioning rule. + Recording writes evidence for every action that resolves an element target. The plain-text format is a versioned comment immediately before the action it annotates: diff --git a/docs/adr/0013-unified-gesture-plans.md b/docs/adr/0013-unified-gesture-plans.md index e945db0562..76539957e8 100644 --- a/docs/adr/0013-unified-gesture-plans.md +++ b/docs/adr/0013-unified-gesture-plans.md @@ -33,7 +33,8 @@ while still routing through the canonical `pan` input. The runtime plans canonical intent in `packages/contracts/src/gesture-plan.ts`. Contact topology is separate from motion: -- one contact: pan or fling with a complete pointer trajectory and an explicit execution profile; +- one contact: pan, fling, or target-authored drag with a complete pointer trajectory and an explicit + execution profile; - two contacts: pan, pinch, rotate, or transform with two complete, synchronized trajectories. `swipe` without a duration remains public sugar for a fixed-duration fling. Timed public forms @@ -133,8 +134,22 @@ Public two-finger pan is additive: `pointerCount?: 1 | 2` on pan and CLI `kind`, `durationMs`, `pointerCount`, `from`, and `to` fields, followed by backend evidence. Recording/replay keeps its existing public command identity and session semantics. -ADR 0011's element dispatch-path matrix remains unchanged: coordinate gestures do not resolve -selectors or refs and therefore cannot claim element-targeting guarantees. +Target-authored drag is additive: `gesture drag ` accepts a selector or pinned +snapshot ref at each endpoint, resolves both endpoints before device injection, then lowers their center +points to one uninterrupted single-pointer plan. The plan holds at the source, moves over the authored +duration, optionally holds at the destination, and releases. Both endpoint-resolution disclosures and +portable selector chains are returned to the caller. Recordings replace session-local refs at both +endpoints with those selector chains and attach one `targets-v1` annotation containing independent source +and destination identity evidence. Replay verifies both identities before pointer-down and guards both +independent endpoint resolutions against the verified elements. Ref admission happens for both endpoints +before either is dispatched, and the usual mutation boundary expires the frame after the gesture. Because +this contract includes the source hold, timed movement, and destination hold phases, target-authored drag +is admitted only on Android touch devices and iOS/iPadOS, whose adapters preserve the full plan. + +ADR 0011's coordinate path remains unchanged: coordinate-authored gestures do not resolve selectors or +refs and therefore cannot claim element-targeting guarantees. Target-authored drag explicitly runs the +shared selector/ref resolution preflight before it enters the coordinate gesture executor; its endpoint +resolution and recording contracts are tested at that composition seam. ## Consequences diff --git a/examples/test-app/replays/drag-android.ad b/examples/test-app/replays/drag-android.ad new file mode 100644 index 0000000000..3bfa4405aa --- /dev/null +++ b/examples/test-app/replays/drag-android.ad @@ -0,0 +1,15 @@ +context platform=android kind=emulator timeout=60000 + +env APP_TARGET="com.callstack.agentdevicelab" +env APP_URL="" + +open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}" +react-native dismiss-overlay +wait "Gesture lab" 30000 +wait "gesture canary ready" 5000 +wait "drag completed no" 5000 + +gesture drag id="drag-source" id="drag-destination" 600 500 0 +wait "drag completed yes" 5000 + +close diff --git a/examples/test-app/replays/drag.ad b/examples/test-app/replays/drag.ad new file mode 100644 index 0000000000..3c1199a2ea --- /dev/null +++ b/examples/test-app/replays/drag.ad @@ -0,0 +1,14 @@ +context platform=ios kind=simulator timeout=60000 + +env APP_TARGET="Agent Device Tester" +env APP_URL="" + +open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}" +wait "Gesture lab" 30000 +wait "gesture canary ready" 5000 +wait "drag completed no" 5000 + +gesture drag id="drag-source" id="drag-destination" 600 500 0 +wait "drag completed yes" 5000 + +close diff --git a/examples/test-app/src/screens/GestureLab.tsx b/examples/test-app/src/screens/GestureLab.tsx index 7554616fe2..3e890492da 100644 --- a/examples/test-app/src/screens/GestureLab.tsx +++ b/examples/test-app/src/screens/GestureLab.tsx @@ -64,6 +64,7 @@ export function GestureLab() { const [canaryReady, setCanaryReady] = useState(false); const [transform, setTransform] = useState(initialTransform); const [counts, setCounts] = useState(initialCounts); + const [dragCompleted, setDragCompleted] = useState(false); const transformRef = useRef(initialTransform); const gestureStartRef = useRef(initialTransform); const androidTouchStartRef = useRef(undefined); @@ -191,6 +192,15 @@ export function GestureLab() { .minDistance(4) .runOnJS(true) .onStart(handleTwoPointerPan); + const holdDragGesture = Gesture.Pan() + .activateAfterLongPress(500) + .minDistance(10) + .runOnJS(true) + .onEnd((event, completed) => { + if (completed && Math.hypot(event.translationX, event.translationY) > 60) { + setDragCompleted(true); + } + }); const legacyFlingRefs = [ legacyFlingLeftRef, legacyFlingRightRef, @@ -337,6 +347,30 @@ export function GestureLab() { {changeStatusLabel} + + + + + Source + + + + Destination + + + + drag completed {dragCompleted ? 'yes' : 'no'} + ); } diff --git a/examples/test-app/src/screens/gesture-lab-styles.ts b/examples/test-app/src/screens/gesture-lab-styles.ts index 121a2649c0..99b3a7d971 100644 --- a/examples/test-app/src/screens/gesture-lab-styles.ts +++ b/examples/test-app/src/screens/gesture-lab-styles.ts @@ -44,5 +44,25 @@ export function createGestureLabStyles(colors: AppColors) { fontWeight: '600', lineHeight: 18, }, + dragRow: { + flexDirection: 'row', + gap: 20, + justifyContent: 'space-between', + }, + dragEndpoint: { + alignItems: 'center', + backgroundColor: colors.cardStrong, + borderColor: colors.line, + borderRadius: 8, + borderWidth: 1, + flex: 1, + height: 56, + justifyContent: 'center', + }, + dragEndpointLabel: { + color: colors.text, + fontSize: 13, + fontWeight: '700', + }, }); } diff --git a/packages/ad-replay/src/internal/__tests__/plan-digest.test.ts b/packages/ad-replay/src/internal/__tests__/plan-digest.test.ts index a543927e47..b1139ad767 100644 --- a/packages/ad-replay/src/internal/__tests__/plan-digest.test.ts +++ b/packages/ad-replay/src/internal/__tests__/plan-digest.test.ts @@ -118,6 +118,32 @@ test('computeReplayPlanDigest changes when target evidence consumed before actio } }); +test('computeReplayPlanDigest binds both entries of multi-target evidence', () => { + const evidence = { + role: 'view', + ancestry: [], + sibling: 0, + viewportOrder: 0, + verification: 'verified' as const, + }; + const targetEvidences = { + source: { ...evidence, id: 'source' }, + destination: { ...evidence, id: 'destination' }, + }; + const original = digestFor([action({ targetEvidences })]); + assert.notEqual( + original, + digestFor([ + action({ + targetEvidences: { + ...targetEvidences, + destination: { ...targetEvidences.destination, id: 'different-destination' }, + }, + }), + ]), + ); +}); + test('computeReplayPlanDigest never changes based on unsubstituted ${VAR} text (variable VALUES never affect the digest)', () => { // The digest is computed over the still-unsubstituted action text; --env // values are resolved later, at invocation time, so two runs with diff --git a/packages/ad-replay/src/internal/plan-digest.ts b/packages/ad-replay/src/internal/plan-digest.ts index dc86f02f4b..1eafd390b3 100644 --- a/packages/ad-replay/src/internal/plan-digest.ts +++ b/packages/ad-replay/src/internal/plan-digest.ts @@ -59,6 +59,7 @@ function canonicalizeAction( flags: action.flags ?? {}, runtime: action.runtime ?? null, targetEvidence: action.targetEvidence ?? null, + targetEvidences: action.targetEvidences ?? null, source: { path: sourcePath, line }, }; } diff --git a/packages/ad-replay/src/internal/runtime-port-types.ts b/packages/ad-replay/src/internal/runtime-port-types.ts index 66e561cfae..3d21be54bb 100644 --- a/packages/ad-replay/src/internal/runtime-port-types.ts +++ b/packages/ad-replay/src/internal/runtime-port-types.ts @@ -150,6 +150,13 @@ export type AdReplayTargetBindingEvidence = Readonly<{ /** The pre-action guard `dispatchStep` threads to the interaction layer's own resolution. */ export type AdReplayDispatchGuard = Readonly< | { readonly kind: 'target'; readonly guard: AdReplayVerifiedTargetGuard } + | { + readonly kind: 'targets'; + readonly guards: Readonly<{ + source: AdReplayVerifiedTargetGuard; + destination: AdReplayVerifiedTargetGuard; + }>; + } | { readonly kind: 'landmark'; readonly landmark: TargetAnnotationV1 } >; @@ -210,6 +217,7 @@ export type AdReplayStepRuntime = Readonly<{ action: SessionAction, resolvedAction: SessionAction, index: number, + targetRole?: 'source' | 'destination', ): AdReplayVerificationEntry; /** * Captures a fresh snapshot for classification or for a divergence's diff --git a/packages/ad-replay/src/internal/target-verification.ts b/packages/ad-replay/src/internal/target-verification.ts index d982209fde..970dab1273 100644 --- a/packages/ad-replay/src/internal/target-verification.ts +++ b/packages/ad-replay/src/internal/target-verification.ts @@ -71,6 +71,7 @@ export type AdReplayTargetStructuralDenotation = Readonly<{ * sees the untyped bag itself. */ export type AdReplayGuardMismatchEvidence = Readonly<{ + targetRole?: 'source' | 'destination'; observed: LocalIdentity | undefined; expectedStructural: AdReplayTargetStructuralDenotation | undefined; observedStructural: AdReplayTargetStructuralDenotation | undefined; diff --git a/packages/ad-replay/src/internal/verify-dispatch.ts b/packages/ad-replay/src/internal/verify-dispatch.ts index b0dbaee259..99e6ad7b75 100644 --- a/packages/ad-replay/src/internal/verify-dispatch.ts +++ b/packages/ad-replay/src/internal/verify-dispatch.ts @@ -7,9 +7,12 @@ import { } from './target-verification.ts'; import type { AdReplayDispatchGuard, + AdReplayDispatchOutcome, + AdReplayObservation, AdReplayScrubValue, AdReplayStepOutcome, AdReplayStepRuntime, + AdReplayVerifiedTargetGuard, } from './runtime-port-types.ts'; /** @@ -48,6 +51,16 @@ export async function verifyAndDispatchStep( index: number, artifactPaths: readonly string[], ): Promise { + if (action.targetEvidences) { + return verifyAndDispatchMultiTargetStep( + runtime, + scrubVars, + action, + resolvedAction, + index, + artifactPaths, + ); + } const recorded = action.targetEvidence; if (!recorded) return dispatchNoGuard(runtime, action, resolvedAction, index, artifactPaths); @@ -118,16 +131,7 @@ export async function verifyAndDispatchStep( failure: await runtime.buildTargetBindingFailure( action, index, - { - kind: 'identity-unverifiable', - matchCount: undefined, - observed: undefined, - candidateNodes: [], - mismatches: [], - causeCode: 'IDENTITY_UNVERIFIABLE', - causeMessage: `Could not capture a fresh snapshot to verify the recorded target before acting (${observation.reason}).`, - ...(observation.hint !== undefined ? { causeHint: observation.hint } : {}), - }, + captureUnavailableEvidence(observation), artifactPaths, scrubVars, ), @@ -161,6 +165,123 @@ export async function verifyAndDispatchStep( }; } +/** Verifies both bindings of a dual-target action before dispatching either endpoint. */ +async function verifyAndDispatchMultiTargetStep( + runtime: AdReplayStepRuntime, + scrubVars: readonly AdReplayScrubValue[], + action: SessionAction, + resolvedAction: SessionAction, + index: number, + artifactPaths: readonly string[], +): Promise { + const recorded = action.targetEvidences!; + const guards: { + source?: AdReplayVerifiedTargetGuard; + destination?: AdReplayVerifiedTargetGuard; + } = {}; + + for (const targetRole of ['source', 'destination'] as const) { + const endpointAction: SessionAction = { + ...action, + targetEvidence: recorded[targetRole], + }; + const entry = runtime.beginTargetVerification( + endpointAction, + resolvedAction, + index, + targetRole, + ); + if (entry.kind !== 'pre-dispatch') { + return dispatchNoGuard(runtime, action, resolvedAction, index, artifactPaths); + } + const plan = planPreDispatchTargetVerification({ + recorded: recorded[targetRole], + token: entry.token, + }); + if (plan.kind === 'skip') { + return dispatchNoGuard(runtime, action, resolvedAction, index, artifactPaths); + } + if (plan.kind === 'recorded-unverifiable') { + return { + status: 'failed', + failure: await runtime.buildRecordedUnverifiableFailure( + endpointAction, + index, + artifactPaths, + scrubVars, + ), + }; + } + + const observation = await runtime.captureObservation(endpointAction, index, { + retryLaunchRace: true, + }); + if (observation.state !== 'available') { + return { + status: 'failed', + failure: await runtime.buildTargetBindingFailure( + endpointAction, + index, + captureUnavailableEvidence(observation), + artifactPaths, + scrubVars, + ), + }; + } + + const classification = runtime.classifyTarget({ + action: endpointAction, + index, + token: plan.token, + nodes: observation.nodes, + }); + if (!classification.verified) { + return { + status: 'failed', + failure: await runtime.buildTargetBindingFailure( + endpointAction, + index, + { + kind: classification.kind, + matchCount: classification.matchCount, + observed: classification.observed, + candidateNodes: classification.candidateNodes, + mismatches: classification.mismatches, + causeCode: classification.causeCode, + causeMessage: classification.causeMessage, + }, + artifactPaths, + scrubVars, + ), + }; + } + guards[targetRole] = classification.guard; + } + + return dispatchWithGuard(runtime, scrubVars, action, resolvedAction, index, artifactPaths, { + kind: 'targets', + guards: { + source: guards.source!, + destination: guards.destination!, + }, + }); +} + +function captureUnavailableEvidence( + observation: Extract, +) { + return { + kind: 'identity-unverifiable' as const, + matchCount: undefined, + observed: undefined, + candidateNodes: [], + mismatches: [], + causeCode: 'IDENTITY_UNVERIFIABLE', + causeMessage: `Could not capture a fresh snapshot to verify the recorded target before acting (${observation.reason}).`, + ...(observation.hint !== undefined ? { causeHint: observation.hint } : {}), + }; +} + /** Dispatches with no pre-action guard — nothing to cross-check, so a mismatch marker can never legitimately fire. */ async function dispatchNoGuard( runtime: AdReplayStepRuntime, @@ -211,22 +332,14 @@ async function dispatchWithGuard( // The refusal markers are only ever attached to an annotated action; fall // back to the plain dispatch failure if the invariant is somehow violated. - const recorded = action.targetEvidence; - if (!recorded) return { status: 'failed', failure: outcome.plainFailure }; - - const evidence = - outcome.status === 'guard-mismatch' - ? deriveReplayTargetGuardMismatchEvidence( - recorded, - outcome.evidence, - guard.kind === 'target' ? guard.guard.matchCount : 0, - ) - : deriveWaitLandmarkMismatchEvidence(recorded, outcome.evidence); + const binding = resolvePostDispatchBinding(action, guard, outcome); + if (!binding) return { status: 'failed', failure: outcome.plainFailure }; + const evidence = derivePostDispatchEvidence(binding.recorded, binding.matchCount, outcome); return { status: 'failed', failure: await runtime.buildPostDispatchTargetBindingFailure( - action, + binding.reportAction, index, { kind: 'identity-mismatch', @@ -242,3 +355,38 @@ async function dispatchWithGuard( ), }; } + +function resolvePostDispatchBinding( + action: SessionAction, + guard: AdReplayDispatchGuard, + outcome: Extract, +) { + if (outcome.status !== 'guard-mismatch' || guard.kind !== 'targets') { + return action.targetEvidence + ? { + recorded: action.targetEvidence, + reportAction: action, + matchCount: guard.kind === 'target' ? guard.guard.matchCount : 0, + } + : undefined; + } + const targetRole = outcome.evidence.targetRole ?? 'source'; + const recorded = action.targetEvidences?.[targetRole]; + return recorded + ? { + recorded, + reportAction: { ...action, targetEvidence: recorded }, + matchCount: guard.guards[targetRole].matchCount, + } + : undefined; +} + +function derivePostDispatchEvidence( + recorded: NonNullable, + matchCount: number, + outcome: Extract, +) { + return outcome.status === 'guard-mismatch' + ? deriveReplayTargetGuardMismatchEvidence(recorded, outcome.evidence, matchCount) + : deriveWaitLandmarkMismatchEvidence(recorded, outcome.evidence); +} diff --git a/packages/ad-script/src/internal/__tests__/script.test.ts b/packages/ad-script/src/internal/__tests__/script.test.ts index 120be8ccc2..15b56a7216 100644 --- a/packages/ad-script/src/internal/__tests__/script.test.ts +++ b/packages/ad-script/src/internal/__tests__/script.test.ts @@ -495,6 +495,18 @@ test('a target-v1 annotation immediately preceding an action line attaches to th assert.deepEqual(actions[0]?.targetEvidence, SAVE_EVIDENCE); }); +test('a targets-v1 annotation binds both drag endpoints to one action', () => { + const source = { ...SAVE_EVIDENCE, id: 'source', label: 'Source' }; + const destination = { ...SAVE_EVIDENCE, id: 'destination', label: 'Destination' }; + const script = [ + `# agent-device:targets-v1 ${JSON.stringify({ source, destination })}`, + 'gesture drag id="source" id="destination" 800 500 0', + ].join('\n'); + const action = parseReplayScriptDetailed(script).actions[0]; + assert.deepEqual(action?.targetEvidences, { source, destination }); + assert.equal(action?.targetEvidence, undefined); +}); + test('a target-v1 annotation followed by a blank line before the action is rejected as INVALID_ARGS', () => { const script = [SAVE_EVIDENCE_LINE, '', 'click @e12 "Save"'].join('\n'); assert.throws( diff --git a/packages/ad-script/src/internal/__tests__/target-annotation-serde.test.ts b/packages/ad-script/src/internal/__tests__/target-annotation-serde.test.ts index 6d52f15fee..04a9a2c325 100644 --- a/packages/ad-script/src/internal/__tests__/target-annotation-serde.test.ts +++ b/packages/ad-script/src/internal/__tests__/target-annotation-serde.test.ts @@ -2,16 +2,21 @@ import { test } from 'vitest'; import assert from 'node:assert/strict'; import { AppError } from '@agent-device/kernel/errors'; import { + formatMultiTargetAnnotationCommentLine, formatTargetAnnotationCommentLine, normalizeLabelField, + parseMultiTargetAnnotationCommentLine, + parseMultiTargetAnnotationV1Payload, parseTargetAnnotationCommentLine, parseTargetAnnotationV1Payload, + serializeMultiTargetAnnotationV1, serializeTargetAnnotationV1, truncateToUtf8Bytes, TARGET_ANNOTATION_MAX_ANCESTRY, TARGET_ANNOTATION_MAX_FIELD_BYTES, TARGET_ANNOTATION_MAX_PAYLOAD_BYTES, TARGET_ANNOTATION_LINE_RE, + MULTI_TARGET_ANNOTATION_MAX_PAYLOAD_BYTES, } from '../target-annotation-serde.ts'; import type { TargetAnnotationV1 } from '@agent-device/contracts/replay'; @@ -66,6 +71,36 @@ test('parse(serialize(evidence)) round trips to a semantically equal object', () assert.deepEqual(parsedBack, evidence); }); +test('targets-v1 canonically round-trips source and destination evidence', () => { + const evidence = { + source: baseEvidence({ id: 'source', label: 'Source' }), + destination: baseEvidence({ id: 'destination', label: 'Destination' }), + }; + const serialized = serializeMultiTargetAnnotationV1(evidence); + assert.deepEqual(parseMultiTargetAnnotationV1Payload(serialized), evidence); + const parsed = parseMultiTargetAnnotationCommentLine( + formatMultiTargetAnnotationCommentLine(evidence), + ); + assert.deepEqual(parsed, { kind: 'v1', evidence }); +}); + +test('targets-v1 rejects a payload missing either endpoint', () => { + assertInvalidArgs( + () => parseMultiTargetAnnotationV1Payload('{"source":{"role":"view"}}'), + /requires source and destination/, + ); +}); + +test('targets-v1 rejects a wrapper above its bounded two-target payload cap', () => { + assertInvalidArgs( + () => + parseMultiTargetAnnotationV1Payload( + JSON.stringify({ padding: 'x'.repeat(MULTI_TARGET_ANNOTATION_MAX_PAYLOAD_BYTES) }), + ), + /exceeds the .*byte payload cap/, + ); +}); + test('parseTargetAnnotationCommentLine accepts known fields in any JSON key order', () => { const line = '# agent-device:target-v1 {"verification":"verified","sibling":0,"role":"button","viewportOrder":2,"ancestry":[],"id":"save"}'; diff --git a/packages/ad-script/src/internal/script-formatting.ts b/packages/ad-script/src/internal/script-formatting.ts index e5a37a1270..fe6931f83f 100644 --- a/packages/ad-script/src/internal/script-formatting.ts +++ b/packages/ad-script/src/internal/script-formatting.ts @@ -6,7 +6,10 @@ import { appendScreenshotActionScriptArgs, appendSnapshotActionScriptArgs, } from './script-utils.ts'; -import { formatTargetAnnotationCommentLine } from './target-annotation-serde.ts'; +import { + formatMultiTargetAnnotationCommentLine, + formatTargetAnnotationCommentLine, +} from './target-annotation-serde.ts'; import type { SessionAction } from '@agent-device/contracts/session'; export function formatPortableActionLine( @@ -38,5 +41,8 @@ export function formatPortableActionLine( * target evidence. Shared by both script writers for one canonical form. */ export function formatTargetAnnotationLines(action: SessionAction): string[] { + if (action.targetEvidences) { + return [formatMultiTargetAnnotationCommentLine(action.targetEvidences)]; + } return action.targetEvidence ? [formatTargetAnnotationCommentLine(action.targetEvidence)] : []; } diff --git a/packages/ad-script/src/internal/script.ts b/packages/ad-script/src/internal/script.ts index 7186630479..da224eee84 100644 --- a/packages/ad-script/src/internal/script.ts +++ b/packages/ad-script/src/internal/script.ts @@ -12,7 +12,10 @@ import { parseReplayRuntimeFlags, stripRecordedRefGeneration, } from './script-utils.ts'; -import { parseTargetAnnotationCommentLine } from './target-annotation-serde.ts'; +import { + parseMultiTargetAnnotationCommentLine, + parseTargetAnnotationCommentLine, +} from './target-annotation-serde.ts'; /** * The `.ad` script env/var key shape: uppercase letters, digits, and @@ -55,7 +58,9 @@ export type ParsedReplayScript = { actionSourcePaths?: (string | undefined)[]; }; -type PendingTargetAnnotation = { evidence: SessionAction['targetEvidence']; line: number }; +type PendingTargetAnnotation = + | { kind: 'single'; evidence: NonNullable; line: number } + | { kind: 'multiple'; evidence: NonNullable; line: number }; // fallow-ignore-next-line complexity export function parseReplayScriptDetailed(script: string): ParsedReplayScript { @@ -70,9 +75,10 @@ export function parseReplayScriptDetailed(script: string): ParsedReplayScript { index: number, why: string, ): never => { + const tag = annotation.kind === 'multiple' ? 'targets-v1' : 'target-v1'; throw new AppError( 'INVALID_ARGS', - `target-v1 annotation on line ${annotation.line} must be immediately followed by its action line (line ${index + 1} ${why}).`, + `${tag} annotation on line ${annotation.line} must be immediately followed by its action line (line ${index + 1} ${why}).`, ); }; @@ -83,10 +89,16 @@ export function parseReplayScriptDetailed(script: string): ParsedReplayScript { continue; } if (trimmed.startsWith('#')) { + const multiAnnotation = parseMultiTargetAnnotationCommentLine(trimmed); + if (multiAnnotation.kind === 'v1') { + if (pending) rejectUnbound(pending, index, 'is another target annotation'); + pending = { kind: 'multiple', evidence: multiAnnotation.evidence, line: index + 1 }; + continue; + } const annotation = parseTargetAnnotationCommentLine(trimmed); if (annotation.kind === 'v1') { - if (pending) rejectUnbound(pending, index, 'is another target-v1 annotation'); - pending = { evidence: annotation.evidence, line: index + 1 }; + if (pending) rejectUnbound(pending, index, 'is another target annotation'); + pending = { kind: 'single', evidence: annotation.evidence, line: index + 1 }; continue; } // An ordinary or future-target-vN comment still counts as an @@ -116,7 +128,8 @@ export function parseReplayScriptDetailed(script: string): ParsedReplayScript { ); if (gestureArityError) throw new AppError('INVALID_ARGS', gestureArityError); if (pending) { - parsed.targetEvidence = pending.evidence; + if (pending.kind === 'single') parsed.targetEvidence = pending.evidence; + else parsed.targetEvidences = pending.evidence; pending = undefined; } actions.push(parsed); @@ -124,9 +137,10 @@ export function parseReplayScriptDetailed(script: string): ParsedReplayScript { sawAction = true; } if (pending) { + const tag = pending.kind === 'multiple' ? 'targets-v1' : 'target-v1'; throw new AppError( 'INVALID_ARGS', - `target-v1 annotation on line ${pending.line} must be immediately followed by its action line (end of script reached).`, + `${tag} annotation on line ${pending.line} must be immediately followed by its action line (end of script reached).`, ); } return { actions, actionLines }; diff --git a/packages/ad-script/src/internal/target-annotation-serde.ts b/packages/ad-script/src/internal/target-annotation-serde.ts index 07360f9e83..2fdd050df5 100644 --- a/packages/ad-script/src/internal/target-annotation-serde.ts +++ b/packages/ad-script/src/internal/target-annotation-serde.ts @@ -20,6 +20,7 @@ import { AppError } from '@agent-device/kernel/errors'; const TARGET_ANNOTATION_TAG = 'agent-device:target-v1'; +const MULTI_TARGET_ANNOTATION_TAG = 'agent-device:targets-v1'; // Captures the rest of the line verbatim: a line claiming the tag with a // garbage payload is a malformed v1 annotation, never an ordinary comment. // @@ -35,15 +36,19 @@ const TARGET_ANNOTATION_TAG = 'agent-device:target-v1'; // would reject (all-whitespace) can never reach this regex in practice. /** Internal-test visibility only: the pattern itself is the ReDoS regression surface. */ export const TARGET_ANNOTATION_LINE_RE = /^#\s*agent-device:target-v(\d+)(?:\s+(\S.*))?$/; +const MULTI_TARGET_ANNOTATION_LINE_RE = /^#\s*agent-device:targets-v(\d+)(?:\s+(\S.*))?$/; export const TARGET_ANNOTATION_MAX_FIELD_BYTES = 256; export const TARGET_ANNOTATION_MAX_PAYLOAD_BYTES = 4096; +export const MULTI_TARGET_ANNOTATION_MAX_PAYLOAD_BYTES = + TARGET_ANNOTATION_MAX_PAYLOAD_BYTES * 2 + 128; export const TARGET_ANNOTATION_MAX_ANCESTRY = 8; // The annotation SHAPE lives in contracts/ so the recorded-action type can be stated without // depending on this zone; every consumer imports it from there directly. import type { TargetAncestryEntry, + MultiTargetAnnotationV1, TargetAnnotationV1, TargetRect, TargetScrollRegion, @@ -165,6 +170,17 @@ export function formatTargetAnnotationCommentLine(evidence: TargetAnnotationV1): return `# ${TARGET_ANNOTATION_TAG} ${serializeTargetAnnotationV1(evidence)}`; } +export function serializeMultiTargetAnnotationV1(evidence: MultiTargetAnnotationV1): string { + return JSON.stringify({ + source: buildCanonicalTargetAnnotationObject(evidence.source), + destination: buildCanonicalTargetAnnotationObject(evidence.destination), + }); +} + +export function formatMultiTargetAnnotationCommentLine(evidence: MultiTargetAnnotationV1): string { + return `# ${MULTI_TARGET_ANNOTATION_TAG} ${serializeMultiTargetAnnotationV1(evidence)}`; +} + // --------------------------------------------------------------------------- // Parsing (decision 3's parser bullet + "Replay-time verification" intro). // --------------------------------------------------------------------------- @@ -174,6 +190,50 @@ export type TargetAnnotationLineParseResult = | { kind: 'future-version' } | { kind: 'v1'; evidence: TargetAnnotationV1 }; +export type MultiTargetAnnotationLineParseResult = + | { kind: 'none' } + | { kind: 'future-version' } + | { kind: 'v1'; evidence: MultiTargetAnnotationV1 }; + +export function parseMultiTargetAnnotationCommentLine( + rawLine: string, +): MultiTargetAnnotationLineParseResult { + const trimmed = rawLine.trim(); + const match = MULTI_TARGET_ANNOTATION_LINE_RE.exec(trimmed); + if (!match) return { kind: 'none' }; + if (Number(match[1]) !== 1) return { kind: 'future-version' }; + return { kind: 'v1', evidence: parseMultiTargetAnnotationV1Payload((match[2] ?? '').trim()) }; +} + +export function parseMultiTargetAnnotationV1Payload(jsonText: string): MultiTargetAnnotationV1 { + if (utf8ByteLength(jsonText) > MULTI_TARGET_ANNOTATION_MAX_PAYLOAD_BYTES) { + throw new AppError( + 'INVALID_ARGS', + `targets-v1 annotation exceeds the ${MULTI_TARGET_ANNOTATION_MAX_PAYLOAD_BYTES}-byte payload cap.`, + ); + } + let parsed: unknown; + try { + parsed = JSON.parse(jsonText); + } catch { + throw new AppError('INVALID_ARGS', 'targets-v1 annotation is not valid JSON.'); + } + if (typeof parsed !== 'object' || parsed === null || Array.isArray(parsed)) { + throw new AppError('INVALID_ARGS', 'targets-v1 annotation must be a JSON object.'); + } + const record = parsed as Record; + if (record.source === undefined || record.destination === undefined) { + throw new AppError( + 'INVALID_ARGS', + 'targets-v1 annotation requires source and destination evidence.', + ); + } + return { + source: parseTargetAnnotationV1Payload(JSON.stringify(record.source)), + destination: parseTargetAnnotationV1Payload(JSON.stringify(record.destination)), + }; +} + /** * Recognizes a `# agent-device:target-vN {...}` comment line. `N !== 1` is an * ordinary comment to this (v1) reader, per decision 3: "An unknown future diff --git a/packages/contracts/src/client-gesture.ts b/packages/contracts/src/client-gesture.ts index 998f547143..823f318e4f 100644 --- a/packages/contracts/src/client-gesture.ts +++ b/packages/contracts/src/client-gesture.ts @@ -81,6 +81,14 @@ export type PanOptions = DeviceCommandBaseOptions & { durationMs?: number; }; +export type DragOptions = DeviceCommandBaseOptions & { + source: string; + destination: string; + sourceHoldMs?: number; + moveMs?: number; + destinationHoldMs?: number; +}; + export type FlingOptions = DeviceCommandBaseOptions & { direction: ScrollDirection; x: number; diff --git a/packages/contracts/src/gesture-input.test.ts b/packages/contracts/src/gesture-input.test.ts index c59b3f4bca..48d6997e17 100644 --- a/packages/contracts/src/gesture-input.test.ts +++ b/packages/contracts/src/gesture-input.test.ts @@ -47,3 +47,65 @@ test('rotate does not accept velocity', () => { code: 'INVALID_ARGS', }); }); + +test('drag requires two non-empty targets and validates each timing phase', () => { + for (const input of [ + { kind: 'drag', source: '', destination: 'id="drop-target"' }, + { kind: 'drag', source: 'id="drag-source"', destination: ' ' }, + { kind: 'drag', source: 'id="drag-source"', destination: 'id="drop-target"', sourceHoldMs: 0 }, + { kind: 'drag', source: 'id="drag-source"', destination: 'id="drop-target"', moveMs: 15 }, + { + kind: 'drag', + source: 'id="drag-source"', + destination: 'id="drop-target"', + destinationHoldMs: -1, + }, + ]) { + assert.throws(() => readGesturePayload(input), { code: 'INVALID_ARGS' }); + } + + assert.deepEqual( + readGesturePayload({ + kind: 'drag', + source: 'id="drag-source"', + destination: '@e2~s42', + destinationHoldMs: 0, + }), + { + kind: 'drag', + source: 'id="drag-source"', + destination: '@e2~s42', + sourceHoldMs: undefined, + moveMs: undefined, + destinationHoldMs: 0, + }, + ); + + assert.deepEqual( + readGesturePayload({ + kind: 'drag', + source: ' id="drag-source" ', + destination: ' id="drop-target" ', + }), + { + kind: 'drag', + source: 'id="drag-source"', + destination: 'id="drop-target"', + sourceHoldMs: undefined, + moveMs: undefined, + destinationHoldMs: undefined, + }, + ); + assert.throws( + () => + readGesturePayload({ + kind: 'drag', + source: 'id="drag-source"', + destination: 'id="drop-target"', + sourceHoldMs: 4_000, + moveMs: 4_000, + destinationHoldMs: 4_000, + }), + { code: 'INVALID_ARGS', message: 'gesture drag total duration must be at most 10000' }, + ); +}); diff --git a/packages/contracts/src/gesture-input.ts b/packages/contracts/src/gesture-input.ts index ad9fb62244..d8e2ffde30 100644 --- a/packages/contracts/src/gesture-input.ts +++ b/packages/contracts/src/gesture-input.ts @@ -8,12 +8,32 @@ import { type SwipePreset, } from './scroll-gesture.ts'; import { + DEFAULT_DRAG_DESTINATION_HOLD_MS, + DEFAULT_DRAG_MOVE_MS, + DEFAULT_DRAG_SOURCE_HOLD_MS, GESTURE_DURATION_MAX_MS, GESTURE_DURATION_MIN_MS, type GesturePointerCount, } from './gesture-plan-types.ts'; -export const GESTURE_KINDS = ['pan', 'fling', 'swipe', 'pinch', 'rotate', 'transform'] as const; +export const COORDINATE_GESTURE_KINDS = [ + 'pan', + 'fling', + 'swipe', + 'pinch', + 'rotate', + 'transform', +] as const; +export const GESTURE_KINDS = [...COORDINATE_GESTURE_KINDS, 'drag'] as const; + +export type DragGesturePayload = { + kind: 'drag'; + source: string; + destination: string; + sourceHoldMs?: number; + moveMs?: number; + destinationHoldMs?: number; +}; export type PanGesturePayload = { kind: 'pan'; @@ -62,7 +82,10 @@ export type GesturePayload = | SwipeGesturePayload | PinchGesturePayload | RotateGesturePayload - | TransformGesturePayload; + | TransformGesturePayload + | DragGesturePayload; + +export type CoordinateGesturePayload = Exclude; export function readGesturePayload(input: unknown): GesturePayload { const record = readRecord(input); @@ -78,6 +101,9 @@ export function readGesturePayload(input: unknown): GesturePayload { durationMs: readOptionalGestureDuration(record), }; } + if (kind === 'drag') { + return readDragGesturePayload(record); + } if (record.pointerCount !== undefined) { throw new AppError('INVALID_ARGS', 'pointerCount is supported only for gesture pan'); } @@ -137,6 +163,35 @@ export function readGesturePayload(input: unknown): GesturePayload { }; } +function readDragGesturePayload(record: Record): DragGesturePayload { + const sourceHoldMs = readOptionalInteger(record, 'sourceHoldMs', { min: 1, max: 10_000 }); + const moveMs = readOptionalInteger(record, 'moveMs', { min: 16, max: 10_000 }); + const destinationHoldMs = readOptionalInteger(record, 'destinationHoldMs', { + // Releasing immediately at the destination is valid; activating a drag + // at the source requires a positive hold. + min: 0, + max: 10_000, + }); + const totalDurationMs = + (sourceHoldMs ?? DEFAULT_DRAG_SOURCE_HOLD_MS) + + (moveMs ?? DEFAULT_DRAG_MOVE_MS) + + (destinationHoldMs ?? DEFAULT_DRAG_DESTINATION_HOLD_MS); + if (totalDurationMs > GESTURE_DURATION_MAX_MS) { + throw new AppError( + 'INVALID_ARGS', + `gesture drag total duration must be at most ${GESTURE_DURATION_MAX_MS}`, + ); + } + return { + kind: 'drag', + source: readNonEmptyString(record, 'source'), + destination: readNonEmptyString(record, 'destination'), + sourceHoldMs, + moveMs, + destinationHoldMs, + }; +} + function readOptionalGestureDuration(record: Record): number | undefined { const durationMs = readOptionalInteger(record, 'durationMs'); if (durationMs === undefined) return undefined; @@ -173,6 +228,14 @@ function readNumber(record: Record, key: string): number { return value; } +function readNonEmptyString(record: Record, key: string): string { + const value = record[key]; + if (typeof value !== 'string' || value.trim().length === 0) { + throw new AppError('INVALID_ARGS', `Expected ${key} to be a non-empty string.`); + } + return value.trim(); +} + function readEnum( record: Record, key: string, diff --git a/packages/contracts/src/gesture-normalization.test.ts b/packages/contracts/src/gesture-normalization.test.ts index 1ac2bfb136..e5b525c625 100644 --- a/packages/contracts/src/gesture-normalization.test.ts +++ b/packages/contracts/src/gesture-normalization.test.ts @@ -4,6 +4,7 @@ import { describeReplayGestureArityError, gesturePayloadFromPositionals, gesturePayloadToPositionals, + normalizeGestureCommandInput, normalizePublicGesture, normalizePublicSwipeMotion, swipePayloadFromPositionals, @@ -58,6 +59,81 @@ test('gesture recording codec round-trips fling with distance', () => { assert.deepEqual(gesturePayloadFromPositionals(gesturePayloadToPositionals(payload)), payload); }); +test('gesture recording codec round-trips selector-authored drag timings', () => { + const payload = { + kind: 'drag' as const, + source: 'id="drag-source"', + destination: '@e2~s42', + sourceHoldMs: 800, + moveMs: 700, + destinationHoldMs: 250, + }; + assert.deepEqual(gesturePayloadFromPositionals(gesturePayloadToPositionals(payload)), payload); +}); + +test('drag recording materializes all timing slots for every sparse option combination', () => { + for (let mask = 0; mask < 8; mask += 1) { + const payload = { + kind: 'drag' as const, + source: 'id="source"', + destination: 'id="destination"', + ...(mask & 1 ? { sourceHoldMs: 900 } : {}), + ...(mask & 2 ? { moveMs: 700 } : {}), + ...(mask & 4 ? { destinationHoldMs: 200 } : {}), + }; + assert.deepEqual(gesturePayloadFromPositionals(gesturePayloadToPositionals(payload)), { + ...payload, + sourceHoldMs: payload.sourceHoldMs ?? 800, + moveMs: payload.moveMs ?? 500, + destinationHoldMs: payload.destinationHoldMs ?? 0, + }); + } +}); + +test('drag payloads normalize to one target-authored runtime command', () => { + assert.deepEqual( + normalizeGestureCommandInput({ + kind: 'drag', + source: 'id="drag-source"', + destination: 'id="drop-target"', + sourceHoldMs: 700, + moveMs: 450, + destinationHoldMs: 100, + }), + { + intent: 'drag', + source: 'id="drag-source"', + destination: 'id="drop-target"', + sourceHoldMs: 700, + moveMs: 450, + destinationHoldMs: 100, + }, + ); +}); + +test('gesture drag positional syntax requires both endpoints and rejects trailing arguments', () => { + assert.throws(() => gesturePayloadFromPositionals(['drag', 'id="drag-source"']), { + code: 'INVALID_ARGS', + }); + assert.throws( + () => + gesturePayloadFromPositionals([ + 'drag', + 'id="drag-source"', + 'id="drop-target"', + '800', + '500', + '0', + 'extra', + ]), + { + code: 'INVALID_ARGS', + message: + 'gesture drag accepts at most 5 arguments: source destination [sourceHoldMs] [moveMs] [destinationHoldMs].', + }, + ); +}); + test('a retired trailing positional reports its migration, not a bare usage line', () => { assert.throws(() => swipePayloadFromPositionals(['197', '650', '197', '300', '300']), { code: 'INVALID_ARGS', diff --git a/packages/contracts/src/gesture-normalization.ts b/packages/contracts/src/gesture-normalization.ts index 73e26f96cb..655ea390e4 100644 --- a/packages/contracts/src/gesture-normalization.ts +++ b/packages/contracts/src/gesture-normalization.ts @@ -1,7 +1,12 @@ import type { Point } from '@agent-device/kernel/snapshot'; import { AppError } from '@agent-device/kernel/errors'; import { readGesturePayload, type GESTURE_KINDS, type GesturePayload } from './gesture-input.ts'; -import type { GestureSemanticInput } from './gesture-plan-types.ts'; +import type { GestureCommandInput, GestureSemanticInput } from './gesture-plan-types.ts'; +import { + DEFAULT_DRAG_DESTINATION_HOLD_MS, + DEFAULT_DRAG_MOVE_MS, + DEFAULT_DRAG_SOURCE_HOLD_MS, +} from './gesture-plan-types.ts'; export type NormalizedPublicGesture = { gesture: GestureSemanticInput; @@ -81,6 +86,11 @@ const PUBLIC_GESTURE_SYNTAX: Record = { max: 7, usage: 'gesture transform accepts at most 7 arguments: x y dx dy scale degrees [durationMs]', }, + 'gesture drag': { + max: 5, + usage: + 'gesture drag accepts at most 5 arguments: source destination [sourceHoldMs] [moveMs] [destinationHoldMs]', + }, }; /** `swipe x1 y1 x2 y2 durationMs` translates to the equivalent timed pan. */ @@ -195,7 +205,7 @@ function readOriginDelta(args: readonly string[]): { origin: Point; delta: Point /** The explicit parser for the public CLI and `.ad` gesture syntax. */ // fallow-ignore-next-line complexity export function gesturePayloadFromPositionals( - positionals: string[], + positionals: readonly string[], pointerCount?: number, ): GesturePayload { const kind = positionals[0]; @@ -256,11 +266,31 @@ export function gesturePayloadFromPositionals( durationMs: optionalPositionNumber(args[6]), }); } + case 'drag': { + assertGestureArity('gesture drag', args); + return readGesturePayload({ + kind, + source: args[0], + destination: args[1], + sourceHoldMs: optionalPositionNumber(args[2]), + moveMs: optionalPositionNumber(args[3]), + destinationHoldMs: optionalPositionNumber(args[4]), + }); + } default: return readGesturePayload({ kind }); } } +/** Reads the selector-authored drag grammar without exposing its positional layout. */ +export function dragGesturePayloadFromPositionals( + positionals: readonly string[], +): Extract | undefined { + if (positionals[0] !== 'drag') return undefined; + const payload = gesturePayloadFromPositionals(positionals); + return payload.kind === 'drag' ? payload : undefined; +} + /** Serializes structured gesture input for `.ad` recordings. */ export function gesturePayloadToPositionals(input: GesturePayload): string[] { switch (input.kind) { @@ -294,6 +324,17 @@ export function gesturePayloadToPositionals(input: GesturePayload): string[] { input.degrees, input.durationMs, ]); + case 'drag': + // Materialize the independent timing slots. Omitting an interior value + // would shift every following positional and silently change replay. + return [ + input.kind, + input.source, + input.destination, + String(input.sourceHoldMs ?? DEFAULT_DRAG_SOURCE_HOLD_MS), + String(input.moveMs ?? DEFAULT_DRAG_MOVE_MS), + String(input.destinationHoldMs ?? DEFAULT_DRAG_DESTINATION_HOLD_MS), + ]; } } @@ -355,9 +396,27 @@ export function normalizePublicGesture(input: GesturePayload): NormalizedPublicG durationMs: input.durationMs, }, }; + case 'drag': + throw new AppError( + 'INVALID_ARGS', + 'gesture drag targets must be resolved before coordinate normalization', + ); } } +/** Converts every public gesture payload into the runtime's semantic command shape. */ +export function normalizeGestureCommandInput(input: GesturePayload): GestureCommandInput { + if (input.kind !== 'drag') return normalizePublicGesture(input).gesture; + return { + intent: 'drag', + source: input.source, + destination: input.destination, + sourceHoldMs: input.sourceHoldMs, + moveMs: input.moveMs, + destinationHoldMs: input.destinationHoldMs, + }; +} + export function normalizePublicSwipeMotion(input: { from: Point; to: Point; diff --git a/packages/contracts/src/gesture-plan-types.ts b/packages/contracts/src/gesture-plan-types.ts index 11eee2d055..52e446c220 100644 --- a/packages/contracts/src/gesture-plan-types.ts +++ b/packages/contracts/src/gesture-plan-types.ts @@ -5,9 +5,21 @@ export type GesturePointerCount = 1 | 2; export const GESTURE_DURATION_MIN_MS = 16; export const GESTURE_DURATION_MAX_MS = 10_000; +export const DEFAULT_DRAG_SOURCE_HOLD_MS = 800; +export const DEFAULT_DRAG_MOVE_MS = 500; +export const DEFAULT_DRAG_DESTINATION_HOLD_MS = 0; export type GestureIntent = 'fling' | 'pan' | 'pinch' | 'rotate' | 'transform'; +export type DragGestureInput = { + intent: 'drag'; + source: string; + destination: string; + sourceHoldMs?: number; + moveMs?: number; + destinationHoldMs?: number; +}; + /** Selects one-pointer release timing without changing semantic gesture intent. */ export type GestureExecutionProfile = 'endpoint-hold' | 'timed-pan'; @@ -39,6 +51,8 @@ export type GestureSemanticInput = durationMs?: number; }; +export type GestureCommandInput = GestureSemanticInput | DragGestureInput; + export type PointerTrajectorySample = { offsetMs: number; point: Point }; export type PointerTrajectory = { @@ -48,7 +62,11 @@ export type PointerTrajectory = { export type SinglePointerTrajectory = { pointerId: 0; - samples: readonly [PointerTrajectorySample, PointerTrajectorySample]; + samples: readonly [ + PointerTrajectorySample, + PointerTrajectorySample, + ...PointerTrajectorySample[], + ]; }; export type SinglePointerGesturePlan = { diff --git a/packages/contracts/src/gesture-plan.test.ts b/packages/contracts/src/gesture-plan.test.ts new file mode 100644 index 0000000000..bf734e8ef2 --- /dev/null +++ b/packages/contracts/src/gesture-plan.test.ts @@ -0,0 +1,81 @@ +import assert from 'node:assert/strict'; +import { test } from 'vitest'; +import { buildDragGesturePlan, singlePointerPlanEndpoints } from './gesture-plan.ts'; + +const DRAG_PROPERTY_RUNS = 100; + +test('drag keeps one pointer down through source hold, movement, and destination hold', () => { + const plan = buildDragGesturePlan( + { + from: { x: 20, y: 30 }, + to: { x: 120, y: 230 }, + sourceHoldMs: 800, + moveMs: 700, + destinationHoldMs: 250, + }, + { x: 0, y: 0, width: 400, height: 800 }, + ); + + assert.equal(plan.topology, 'single'); + assert.equal(plan.durationMs, 1_750); + assert.deepEqual(plan.pointers[0]?.samples[0], { offsetMs: 0, point: { x: 20, y: 30 } }); + assert.deepEqual(plan.pointers[0]?.samples[1], { offsetMs: 800, point: { x: 20, y: 30 } }); + assert.deepEqual(plan.pointers[0]?.samples.at(-2), { + offsetMs: 1_500, + point: { x: 120, y: 230 }, + }); + assert.deepEqual(plan.pointers[0]?.samples.at(-1), { + offsetMs: 1_750, + point: { x: 120, y: 230 }, + }); + assert.deepEqual(singlePointerPlanEndpoints(plan), { + start: { x: 20, y: 30 }, + end: { x: 120, y: 230 }, + }); +}); + +test('drag rejects a combined duration above the backend ceiling', () => { + assert.throws( + () => + buildDragGesturePlan( + { + from: { x: 20, y: 30 }, + to: { x: 120, y: 230 }, + sourceHoldMs: 5_000, + moveMs: 5_000, + destinationHoldMs: 1, + }, + { x: 0, y: 0, width: 400, height: 800 }, + ), + { code: 'INVALID_ARGS', message: 'gesture drag total duration must be at most 10000' }, + ); +}); + +test('every drag plan preserves ordered samples and exact hold/move endpoints', () => { + let state = 0x6d2b79f5; + const next = (minimum: number, maximum: number): number => { + state = (Math.imul(state, 1_664_525) + 1_013_904_223) >>> 0; + return minimum + (state % (maximum - minimum + 1)); + }; + for (let run = 0; run < DRAG_PROPERTY_RUNS; run += 1) { + const from = { x: next(2, 398), y: next(2, 798) }; + const to = { x: next(2, 398), y: next(2, 798) }; + const sourceHoldMs = next(1, 3_000); + const moveMs = next(16, 3_000); + const destinationHoldMs = next(0, 3_000); + const plan = buildDragGesturePlan( + { from, to, sourceHoldMs, moveMs, destinationHoldMs }, + { x: 0, y: 0, width: 400, height: 800 }, + ); + const samples = plan.pointers[0].samples; + assert.equal(samples[0]?.offsetMs, 0); + assert.deepEqual(samples[0]?.point, from); + assert.equal(samples[1]?.offsetMs, sourceHoldMs); + assert.deepEqual(samples[1]?.point, from); + assert.equal(samples.at(-1)?.offsetMs, sourceHoldMs + moveMs + destinationHoldMs); + assert.deepEqual(samples.at(-1)?.point, to); + for (let index = 1; index < samples.length; index += 1) { + assert.ok(samples[index]!.offsetMs >= samples[index - 1]!.offsetMs); + } + } +}); diff --git a/packages/contracts/src/gesture-plan.ts b/packages/contracts/src/gesture-plan.ts index 4535d4060b..d20c478811 100644 --- a/packages/contracts/src/gesture-plan.ts +++ b/packages/contracts/src/gesture-plan.ts @@ -6,7 +6,13 @@ import { gestureDirectionDelta, type SwipePreset, } from './scroll-gesture.ts'; -import { GESTURE_DURATION_MAX_MS, GESTURE_DURATION_MIN_MS } from './gesture-plan-types.ts'; +import { + DEFAULT_DRAG_DESTINATION_HOLD_MS, + DEFAULT_DRAG_MOVE_MS, + DEFAULT_DRAG_SOURCE_HOLD_MS, + GESTURE_DURATION_MAX_MS, + GESTURE_DURATION_MIN_MS, +} from './gesture-plan-types.ts'; import type { GesturePlan, GestureExecutionProfile, @@ -124,14 +130,76 @@ export function singlePointerPlanEndpoints(plan: SinglePointerGesturePlan): { start: Point; end: Point; } { - const [ - { - samples: [start, end], - }, - ] = plan.pointers; + const [{ samples }] = plan.pointers; + const start = samples[0]; + const end = samples.at(-1)!; return { start: start.point, end: end.point }; } +/** + * Plans one uninterrupted pointer contact: activate at the source, move to the + * destination, optionally hold there, then release. Element targets are + * resolved by the interaction runtime before this portable planning seam. + */ +export function buildDragGesturePlan( + input: { + from: Point; + to: Point; + sourceHoldMs?: number; + moveMs?: number; + destinationHoldMs?: number; + }, + viewport: Rect, +): SinglePointerGesturePlan { + const frame = normalizeViewport(viewport); + const sourceHoldMs = normalizePositiveDuration( + input.sourceHoldMs, + DEFAULT_DRAG_SOURCE_HOLD_MS, + 'gesture drag sourceHoldMs', + ); + const moveMs = normalizeDuration(input.moveMs, DEFAULT_DRAG_MOVE_MS, 'gesture drag moveMs'); + const destinationHoldMs = normalizeNonNegativeDuration( + input.destinationHoldMs, + DEFAULT_DRAG_DESTINATION_HOLD_MS, + 'gesture drag destinationHoldMs', + ); + const durationMs = sourceHoldMs + moveMs + destinationHoldMs; + if (durationMs > GESTURE_DURATION_MAX_MS) { + throw new AppError( + 'INVALID_ARGS', + `gesture drag total duration must be at most ${GESTURE_DURATION_MAX_MS}`, + ); + } + const move = buildSinglePointerPlan('pan', input.from, input.to, moveMs, frame, 'timed-pan'); + return withContactHolds(move, sourceHoldMs, destinationHoldMs); +} + +function withContactHolds( + plan: SinglePointerGesturePlan, + sourceHoldMs: number, + destinationHoldMs: number, +): SinglePointerGesturePlan { + const pointer = plan.pointers[0]; + const [moveStart, moveEnd] = pointer.samples; + const durationMs = sourceHoldMs + plan.durationMs + destinationHoldMs; + const samples: SinglePointerTrajectory['samples'] = [ + { offsetMs: 0, point: moveStart.point }, + { ...moveStart, offsetMs: moveStart.offsetMs + sourceHoldMs }, + { ...moveEnd, offsetMs: moveEnd.offsetMs + sourceHoldMs }, + ...(destinationHoldMs > 0 ? [{ offsetMs: durationMs, point: moveEnd.point }] : []), + ]; + return { + ...plan, + durationMs, + pointers: [ + { + ...pointer, + samples, + }, + ], + }; +} + function buildFlingPlan( input: Extract, viewport: Rect, @@ -409,6 +477,28 @@ function normalizeDuration(value: number | undefined, fallback: number, field: s return durationMs; } +function normalizePositiveDuration(value: number | undefined, fallback: number, field: string) { + const durationMs = value ?? fallback; + if (!Number.isInteger(durationMs) || durationMs < 1 || durationMs > GESTURE_DURATION_MAX_MS) { + throw new AppError( + 'INVALID_ARGS', + `${field} must be an integer between 1 and ${GESTURE_DURATION_MAX_MS}`, + ); + } + return durationMs; +} + +function normalizeNonNegativeDuration(value: number | undefined, fallback: number, field: string) { + const durationMs = value ?? fallback; + if (!Number.isInteger(durationMs) || durationMs < 0 || durationMs > GESTURE_DURATION_MAX_MS) { + throw new AppError( + 'INVALID_ARGS', + `${field} must be an integer between 0 and ${GESTURE_DURATION_MAX_MS}`, + ); + } + return durationMs; +} + function defaultTransformDuration(rotationDegrees: number): number { const rotationDuration = Math.ceil(Math.abs(rotationDegrees) / MAX_ROTATION_DEGREES_PER_SAMPLE) * diff --git a/packages/contracts/src/interaction-guarantees.ts b/packages/contracts/src/interaction-guarantees.ts index 4122f1d950..c73a589057 100644 --- a/packages/contracts/src/interaction-guarantees.ts +++ b/packages/contracts/src/interaction-guarantees.ts @@ -72,6 +72,7 @@ export type InteractionGuarantee = (typeof INTERACTION_GUARANTEES)[number]; export const INTERACTION_PATH_IDS = [ 'runtime-selector', 'runtime-ref', + 'target-drag', 'direct-ios-selector', 'native-ref', 'coordinate', @@ -224,6 +225,55 @@ export const INTERACTION_DISPATCH_PATHS: Record { + const setup = createTransport(async () => ({ ok: true, data: { message: 'Dragged' } })); + const client = createAgentDeviceClient(setup.config, { transport: setup.transport }); + + await client.interactions.drag({ + source: 'id="drag-source"', + destination: '@e2~s42', + sourceHoldMs: 700, + moveMs: 600, + destinationHoldMs: 200, + }); + + assert.deepEqual( + setup.calls.map(({ command, positionals, input }) => ({ command, positionals, input })), + [ + { + command: 'gesture', + positionals: [], + input: { + kind: 'drag', + source: 'id="drag-source"', + destination: '@e2~s42', + sourceHoldMs: 700, + moveMs: 600, + destinationHoldMs: 200, + }, + }, + ], + ); +}); + // fallow-ignore-next-line complexity test('replay.run serializes client-collected AD_VAR shell env into daemon request', async () => { const previousAppId = process.env.AD_VAR_APP_ID; diff --git a/src/__tests__/test-utils/property-arbitraries.ts b/src/__tests__/test-utils/property-arbitraries.ts index 5aa10c54fd..fe0cbc5761 100644 --- a/src/__tests__/test-utils/property-arbitraries.ts +++ b/src/__tests__/test-utils/property-arbitraries.ts @@ -1,5 +1,5 @@ import { - GESTURE_KINDS, + COORDINATE_GESTURE_KINDS, SCROLL_DIRECTIONS, SWIPE_PRESETS, type GesturePayload, @@ -91,7 +91,7 @@ function pointInViewportArb(viewport: Rect): fc.Arbitrary { */ function gesturePayloadArbByKind( viewport: Rect, -): Record<(typeof GESTURE_KINDS)[number], fc.Arbitrary> { +): Record<(typeof COORDINATE_GESTURE_KINDS)[number], fc.Arbitrary> { const origin = pointInViewportArb(viewport); const delta = fc.record({ x: fc.integer({ min: -viewport.width, max: viewport.width }), @@ -144,7 +144,7 @@ export const gestureInViewportArb: fc.Arbitrary<{ viewport: Rect; gesture: Gestu const byKind = gesturePayloadArbByKind(viewport); return fc.record({ viewport: fc.constant(viewport), - gesture: fc.oneof(...GESTURE_KINDS.map((kind) => byKind[kind])), + gesture: fc.oneof(...COORDINATE_GESTURE_KINDS.map((kind) => byKind[kind])), }); }); diff --git a/src/agent-device-client.ts b/src/agent-device-client.ts index 261cec24ab..705b637160 100644 --- a/src/agent-device-client.ts +++ b/src/agent-device-client.ts @@ -15,6 +15,7 @@ import type { CaptureScreenshotResult, CaptureSnapshotOptions, CaptureSnapshotResult, + DragOptions, FlingOptions, InternalRequestOptions, Lease, @@ -401,6 +402,7 @@ export function createAgentDeviceClient( longPress: async (options) => await executeCommand('longpress', options), swipe: async (options) => await executeCommand('swipe', options), pan: async (options) => await executeCommand('gesture', panGestureInput(options)), + drag: async (options) => await executeCommand('gesture', dragGestureInput(options)), fling: async (options) => await executeCommand('gesture', flingGestureInput(options)), swipeGesture: async (options) => await executeCommand('gesture', swipePresetGestureInput(options)), @@ -453,6 +455,10 @@ function panGestureInput(options: PanOptions): InternalRequestOptions & Record { + return { ...options, kind: 'drag' }; +} + function flingGestureInput( options: FlingOptions, ): InternalRequestOptions & Record { diff --git a/src/cli/parser/__tests__/cli-help-topics.test.ts b/src/cli/parser/__tests__/cli-help-topics.test.ts index 8e7b0ddb62..aaa04c8e5b 100644 --- a/src/cli/parser/__tests__/cli-help-topics.test.ts +++ b/src/cli/parser/__tests__/cli-help-topics.test.ts @@ -19,7 +19,7 @@ test('usage includes concise top-level commands', async () => { assert.match(usageText, /clipboard read \| clipboard write /); assert.match(usageText, /keyboard \[action\]/); assert.match(usageText, /trigger-app-event\s{2,}Invoke app-defined automation\/test events/); - assert.match(usageText, /gesture \.\.\./); + assert.match(usageText, /gesture \.\.\./); assert.doesNotMatch( usageText, /install-from-source \| install-from-source --github-actions-artifact/, @@ -37,6 +37,12 @@ test('usage includes concise top-level commands', async () => { assert.match(usageText, /trace start \| trace stop /); }); +test('gesture help documents selectors and pinned refs for both drag endpoints', async () => { + const help = await usageForCommand('gesture'); + assert.ok(help); + assert.match(help, /drag /); +}); + test('usage includes only global flags in the top-level global flags section', async () => { const usageText = await usage(); const flagsSection = usageText.slice( diff --git a/src/cli/parser/cli-help.ts b/src/cli/parser/cli-help.ts index 28c87ada44..4bfce39080 100644 --- a/src/cli/parser/cli-help.ts +++ b/src/cli/parser/cli-help.ts @@ -224,7 +224,7 @@ Command shape: Snapshot refs look like @e12. After snapshot -i, use the exact @eN ref from that output. If the exact ref is not known yet, first output snapshot -i, then use a concrete example shape like press @e12 in the next command; do not write @, @ref, @Label_Name, or @eN placeholders. Close means agent-device close. App-owned back means back; system back means back --system. - Taps are press or click; tap is an alias for press. On Android TV, tvOS, and Vega OS, read help tv and use tv-remote press up|down|left|right|select to move D-pad/remote focus before activating controls; use tv-remote longpress