Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions android/snapshot-helper/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,10 +51,19 @@ invocations can pass `0` when immediate capture during ongoing animation is pref
acquisition has a separate 500 ms stabilization bound that is used only when no root is available or
an active/focused window is temporarily missing its root; complete captures pay no additional wait.

Child reads run in batches of up to eight on a shared worker pool. This lets Android serve
multiple accessibility requests between animation frames on busy apps. XML retains depth-first
child order and the same depth/node limits. Each capture still clears the accessibility cache;
results are not reused across captures.

## One-Shot Modes

Passing `-e mode snapshot|viewport|gesture` selects what a single instrumentation run does; `snapshot`
is the default and matches the `Run` section above.
The `viewport` mode reads application and keyboard bounds after the same best-effort idle wait
used by snapshots: a 100 ms quiet interval with a 500 ms maximum wait. This allows short window
transitions, including keyboard appearance, to settle without waiting indefinitely on live content.
Comment on lines +63 to +65

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This says viewport uses the same idle wait as snapshots, but viewport reads always call prepare(automation) with the 100 ms/500 ms defaults while snapshots honor the supplied wait arguments. Describe the viewport values as fixed defaults to avoid implying that -e waitForIdleTimeoutMs or -e waitForIdleQuietMs configures both modes.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At android/snapshot-helper/README.md, line 63:

<comment>This says viewport uses the same idle wait as snapshots, but viewport reads always call `prepare(automation)` with the 100 ms/500 ms defaults while snapshots honor the supplied wait arguments. Describe the viewport values as fixed defaults to avoid implying that `-e waitForIdleTimeoutMs` or `-e waitForIdleQuietMs` configures both modes.</comment>

<file context>
@@ -60,9 +60,10 @@ results are not reused across captures.
-The `viewport` mode reads application and keyboard bounds while content updates continue; it does
-not wait for the accessibility event stream to become idle. Snapshot mode uses the idle wait
-configured above.
+The `viewport` mode reads application and keyboard bounds after the same best-effort idle wait
+used by snapshots: a 100 ms quiet interval with a 500 ms maximum wait. This allows short window
+transitions, including keyboard appearance, to settle without waiting indefinitely on live content.
</file context>
Suggested change
The `viewport` mode reads application and keyboard bounds after the same best-effort idle wait
used by snapshots: a 100 ms quiet interval with a 500 ms maximum wait. This allows short window
transitions, including keyboard appearance, to settle without waiting indefinitely on live content.
The `viewport` mode reads application and keyboard bounds after a best-effort idle wait of 100 ms quiet and 500 ms maximum. These viewport values are fixed; snapshot mode uses the idle wait configured above.

Snapshot mode uses the idle wait configured above.

```sh
# Read the interactive-window viewport without capturing a snapshot.
Expand Down Expand Up @@ -96,6 +105,10 @@ closes that connection; the process stays alive for the next connection:
the XML body after the header block, as described below. The response protocol literal is always
`android-snapshot-helper-v1`, regardless of session or one-shot transport.

When an input method window overlaps the application's vertical swipe path, `viewport` responses
also include `keyboardX`, `keyboardY`, `keyboardWidth`, and `keyboardHeight`. All four fields are
omitted when no such window is reported; their absence does not establish that the keyboard is hidden.

## Output Contract

The APK emits instrumentation status records using
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
package com.callstack.agentdevice.snapshothelper;

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.FutureTask;

/**
* Reads a bounded group of children concurrently and retains their traversal order.
*
* <p>Accessibility cache misses are served on the app's UI thread. Queuing siblings together lets
* the app serve multiple reads between animation frames. Workers are reused because Android keeps
* an AccessibilityInteractionClient for each calling thread for the lifetime of the process.
* Closing waits for every read before releasing nodes, including unread results after truncation
* or failure, so no worker outlives the parent node it is reading.
*/
final class AccessibilityReadBatch<T> implements AutoCloseable {
interface Reader<T> {
T read(int index);
}

interface Releaser<T> {
void release(T value);
}

static final int MAX_READS = 8;
private static final ExecutorService READERS =
Executors.newFixedThreadPool(
MAX_READS,
task -> {
Thread thread = new Thread(task, "agent-device-accessibility-read");
thread.setDaemon(true);
return thread;
});

private final List<FutureTask<T>> reads = new ArrayList<>();
private final Releaser<T> release;

AccessibilityReadBatch(int start, int count, Reader<T> read, Releaser<T> release) {
if (count < 1 || count > MAX_READS) {
throw new IllegalArgumentException("Accessibility read batch must contain 1 to 8 reads");
}
this.release = release;
for (int offset = 0; offset < count; offset += 1) {
int index = start + offset;
FutureTask<T> task = new FutureTask<>(() -> read.read(index));
reads.add(task);
READERS.execute(task);
}
}

T get(int offset) {
return await(reads.get(offset));
}

@Override
public void close() {
for (FutureTask<T> read : reads) {
T node;
try {
node = await(read);
} catch (RuntimeException | Error ignored) {
// get() propagates read failures. Other reads must finish before the parent is released.
continue;
}
if (node != null) {
release.release(node);
}
}
}

private static <T> T await(FutureTask<T> read) {
boolean interrupted = false;
try {
while (true) {
try {
return read.get();
} catch (InterruptedException error) {
interrupted = true;
} catch (ExecutionException error) {
Throwable cause = error.getCause();
if (cause instanceof RuntimeException) {
throw (RuntimeException) cause;
}
if (cause instanceof Error) {
throw (Error) cause;
}
throw new IllegalStateException("Android accessibility child read failed", cause);
}
}
} finally {
if (interrupted) {
Thread.currentThread().interrupt();
}
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,32 @@
import android.view.accessibility.AccessibilityWindowInfo;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.TimeoutException;

/** Captures and serializes the Android accessibility window tree. */
final class AccessibilityTreeCapture {
static final long DEFAULT_WAIT_FOR_IDLE_TIMEOUT_MS = 500;
static final long DEFAULT_WAIT_FOR_IDLE_QUIET_MS = 100;
private static final long WINDOW_LIST_SETTLE_TIMEOUT_MS = 1_000;

private AccessibilityTreeCapture() {}

static void prepare(UiAutomation automation) throws InterruptedException {
prepare(automation, DEFAULT_WAIT_FOR_IDLE_QUIET_MS, DEFAULT_WAIT_FOR_IDLE_TIMEOUT_MS);
}

static void prepare(UiAutomation automation, long quietMs, long timeoutMs)
throws InterruptedException {
enableInteractiveWindowRetrieval(automation);
if (timeoutMs > 0) {
try {
automation.waitForIdle(Math.min(quietMs, timeoutMs), timeoutMs);
} catch (TimeoutException ignored) {
// A busy accessibility stream can still expose usable windows and nodes.
}
}
}

static Result capture(
UiAutomation automation, int maxDepth, int maxNodes, long stabilizationTimeoutMs)
throws InterruptedException, AccessibilityCaptureStabilizer.IncompleteCaptureException {
Expand Down Expand Up @@ -105,7 +124,7 @@ private static void clearAccessibilityCacheViaServiceInfo(UiAutomation automatio
* the window list to name an active window. The platform fills the list after the flag is set, so
* a read straight after it can list only the status bar and none of the app's windows.
*/
static void enableInteractiveWindowRetrieval(UiAutomation automation)
private static void enableInteractiveWindowRetrieval(UiAutomation automation)
throws InterruptedException {
if (!turnOnInteractiveWindowRetrieval(automation)) {
return;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@
import android.view.accessibility.AccessibilityNodeInfo.AccessibilityAction;
import android.view.accessibility.AccessibilityWindowInfo;
import java.util.List;
import java.util.Locale;

/** Serializes accessibility nodes using the helper's host-consumed XML contract. */
final class AccessibilityTreeXml {
Expand Down Expand Up @@ -91,16 +90,7 @@ static void appendNode(
hasAccessibilityAction(node, AccessibilityAction.ACTION_SCROLL_BACKWARD)));
}
appendAttribute(xml, "password", Boolean.toString(node.isPassword()));
appendAttribute(
xml,
"bounds",
String.format(
Locale.ROOT,
"[%d,%d][%d,%d]",
bounds.left,
bounds.top,
bounds.right,
bounds.bottom));
appendBoundsAttribute(xml, "bounds", bounds.left, bounds.top, bounds.right, bounds.bottom);

int childCount = depth >= maxDepth ? 0 : node.getChildCount();
if (depth >= maxDepth && node.getChildCount() > 0) {
Expand All @@ -112,20 +102,26 @@ static void appendNode(
}

xml.append(">");
for (int index = 0; index < childCount; index += 1) {
if (stats.nodeCount >= maxNodes) {
stats.truncated = true;
break;
}
AccessibilityNodeInfo child = AccessibilityNodeReads.child(node, index);
if (child == null) {
continue;
}
try {
appendNode(xml, child, index, depth + 1, maxDepth, maxNodes, stats, null);
} finally {
child.recycle();
int index = 0;
while (index < childCount && stats.nodeCount < maxNodes) {
int count = Math.min(
AccessibilityReadBatch.MAX_READS, Math.min(childCount - index, maxNodes - stats.nodeCount));
int offset = 0;
try (AccessibilityReadBatch<AccessibilityNodeInfo> children =
new AccessibilityReadBatch<>(
index, count, childIndex -> AccessibilityNodeReads.child(node, childIndex),
AccessibilityNodeInfo::recycle)) {
for (; offset < count && stats.nodeCount < maxNodes; offset += 1) {
AccessibilityNodeInfo child = children.get(offset);
if (child != null) {
appendNode(xml, child, index + offset, depth + 1, maxDepth, maxNodes, stats, null);
}
}
}
index += offset;
}
if (index < childCount) {
stats.truncated = true;
}
xml.append("</node>");
}
Expand Down Expand Up @@ -206,16 +202,9 @@ private static void appendWindowMetadata(StringBuilder xml, WindowMetadata metad
appendAttribute(xml, "window-layer", Integer.toString(metadata.layer));
appendAttribute(xml, "window-active", Boolean.toString(metadata.active));
appendAttribute(xml, "window-focused", Boolean.toString(metadata.focused));
appendAttribute(
xml,
"window-bounds",
String.format(
Locale.ROOT,
"[%d,%d][%d,%d]",
metadata.bounds.left,
metadata.bounds.top,
metadata.bounds.right,
metadata.bounds.bottom));
appendBoundsAttribute(
xml, "window-bounds", metadata.bounds.left, metadata.bounds.top,
metadata.bounds.right, metadata.bounds.bottom);
if (metadata.regionRect != null) {
appendAttribute(xml, "window-region-rect", Boolean.toString(metadata.regionRect));
}
Expand All @@ -230,6 +219,13 @@ private static void appendAttribute(StringBuilder xml, String name, CharSequence
xml.append('"');
}

static void appendBoundsAttribute(
StringBuilder xml, String name, int left, int top, int right, int bottom) {
xml.append(' ').append(name).append("=\"[")
.append(left).append(',').append(top).append("][")
.append(right).append(',').append(bottom).append("]\"");
}

private static boolean hasAccessibilityAction(
AccessibilityNodeInfo node, AccessibilityAction action) {
List<AccessibilityAction> actions = node.getActionList();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
import android.view.accessibility.AccessibilityWindowInfo;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.TimeoutException;

/**
* Resolves the active application window bounds used to validate planned gestures, plus the input
Expand Down Expand Up @@ -59,18 +58,13 @@ Rect toRect() {

@SuppressWarnings("deprecation")
static Reading readReading(UiAutomation automation) throws InterruptedException {
try {
automation.waitForIdle(100, 2_000);
} catch (TimeoutException ignored) {
// Window/root state can still be usable when the app is animating continuously.
}
// UiAutomation.getWindows() transfers recyclable AccessibilityWindowInfo instances, and this
// read runs repeatedly inside the persistent helper session: copy the bounds the precedence
// below needs, then recycle every window before resolving.
// UiAutomation.getWindows() answers with an empty list until interactive retrieval is on, which
// is the same seam the tree capture already uses. Without it this read sees no windows at all and
// the keyboard below is invisible to it.
AccessibilityTreeCapture.enableInteractiveWindowRetrieval(automation);
AccessibilityTreeCapture.prepare(automation);
Rect activeBounds = null;
Rect fallbackBounds = null;
List<WindowEdges> inputMethodWindows = new ArrayList<>();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,6 @@
public final class SnapshotInstrumentation extends Instrumentation {
private static final long GESTURE_UI_AUTOMATION_CONNECT_TIMEOUT_MS = 8_000;
private static final int CHUNK_SIZE = 2 * 1024;
// Match the host default: bounded wait for microinteraction reliability without the stock
// uiautomator idle tax. Direct callers can pass 0 when immediate capture is preferred.
private static final long DEFAULT_WAIT_FOR_IDLE_TIMEOUT_MS = 500;
private static final long DEFAULT_WAIT_FOR_IDLE_QUIET_MS = 100;
private static final long DEFAULT_TIMEOUT_MS = 8_000;
private static final long ROOT_CAPTURE_STABILIZATION_TIMEOUT_MS = 500;
private static final int DEFAULT_MAX_DEPTH = 128;
Expand All @@ -45,9 +41,11 @@ public void onCreate(Bundle arguments) {
public void onStart() {
super.onStart();
long waitForIdleTimeoutMs =
readLongArgument(arguments, "waitForIdleTimeoutMs", DEFAULT_WAIT_FOR_IDLE_TIMEOUT_MS);
readLongArgument(
arguments, "waitForIdleTimeoutMs", AccessibilityTreeCapture.DEFAULT_WAIT_FOR_IDLE_TIMEOUT_MS);
long waitForIdleQuietMs =
readLongArgument(arguments, "waitForIdleQuietMs", DEFAULT_WAIT_FOR_IDLE_QUIET_MS);
readLongArgument(
arguments, "waitForIdleQuietMs", AccessibilityTreeCapture.DEFAULT_WAIT_FOR_IDLE_QUIET_MS);
long timeoutMs = readLongArgument(arguments, "timeoutMs", DEFAULT_TIMEOUT_MS);
int maxDepth = readIntArgument(arguments, "maxDepth", DEFAULT_MAX_DEPTH);
int maxNodes = readIntArgument(arguments, "maxNodes", DEFAULT_MAX_NODES);
Expand Down Expand Up @@ -357,18 +355,7 @@ private AccessibilityTreeCapture.Result captureXml(
InterruptedException,
AccessibilityCaptureStabilizer.IncompleteCaptureException {
UiAutomation automation = getConnectedUiAutomation(timeoutMs);
AccessibilityTreeCapture.enableInteractiveWindowRetrieval(automation);
if (waitForIdleTimeoutMs > 0) {
try {
// Best-effort settle: wait for the accessibility stream to become idle, but require only
// a short quiet window once it does. Using the full timeout as the quiet window made every
// stable snapshot pay a fixed 500 ms tax.
long quietMs = Math.min(waitForIdleQuietMs, waitForIdleTimeoutMs);
automation.waitForIdle(quietMs, waitForIdleTimeoutMs);
} catch (TimeoutException ignored) {
// Busy or animated apps can still expose a usable root; capture whatever is available.
}
}
AccessibilityTreeCapture.prepare(automation, waitForIdleQuietMs, waitForIdleTimeoutMs);
return AccessibilityTreeCapture.capture(
automation, maxDepth, maxNodes, ROOT_CAPTURE_STABILIZATION_TIMEOUT_MS);
}
Expand Down
Loading
Loading