Skip to content

fix(android-e2e): wait for the Home/Recents surface change before pixel assertions - #4716

Open
thymikee wants to merge 2 commits into
mainfrom
t3/android-transition-wait
Open

thymikee wants to merge 2 commits into
mainfrom
t3/android-transition-wait

Conversation

@thymikee

@thymikee thymikee commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Summary

Closes #4710.

The Android emulator smoke scenario (smoke:automation-system) took one screenshot immediately after home and after app-switcher and asserted it differed from the previous surface. When the launcher or the Recents transition had not drawn yet, the capture still showed the previous surface, so the identical-image assertion failed intermittently on main and on unrelated PRs (e.g. #4297).

assertHomeAndRecentsRestoration now waits at that observation seam with the bounded waitForSurfaceChange helper (test/integration/android-emulator-e2e/live-surface-transition.ts, same contract as the existing waitForImeWindow): re-capture every 250 ms until the screenshot differs from the baseline, within a 10 s deadline. Each attempt removes and overwrites the artifact, so the final observed image is kept and a failed capture cannot leave a stale one; captures are bounded by the remaining deadline (5 s cap, failure evidence deferred to the deadline verdict); a capture resolving after the deadline is rejected as late. The Home foreground-package proof and both assertFilesDiffer pixel comparisons are retained unchanged; the wait fails at the deadline if navigation never produces the change. No fixed blanket sleep, no smoke-test retry, no weakened assertion, no production change, no workflow/runner change (#4712 untouched).

10 files touched (2 new test files + the new helper). Regression coverage is at both levels: live-surface-transition.test.ts pins the helper contract under fake timers (delayed transition, transient capture failures, timeout negative, no capture outlives the remaining deadline, late observations rejected), and live-automation-scenario.test.ts drives the real scenario through the harness with a scripted device (new LiveDeviceContext.runCli seam) whose first Home/Recents captures are stale — a single-capture route fails those tests, verified by temporarily restoring the old route.

Validation

  • Final head: 8516300c956fa37f99a2a9a27a35fdbe9a28f9a3 after pnpm install --frozen-lockfile && pnpm build. pnpm check:affected --run --head 8516300c956fa37f99a2a9a27a35fdbe9a28f9a3 (receipt .tmp/validation/run-6RCKeL, exactCommit=true): every gate passed except the unit suite's host-load flake in scripts/__tests__/release-publish.test.ts (5 s timeout; passes solo and also fails on unchanged origin/main in an isolated worktree under the same load); 20616/20627 unit tests passed including all 9 new ones. Focused: 9/9 in the two new files plus the neighboring harness suites (27/27 in the android-emulator-e2e + runtime suites); pnpm typecheck, pnpm format, oxlint clean; node --test test/integration/smoke-android-emulator-coverage.test.ts 10/10.
  • Hosted Android validation: workflow run 44388138478 at head_sha 8516300c956fa37f99a2a9a27a35fdbe9a28f9a3 — the Smoke Tests job passed on the initial run plus ten consecutive reruns at the unchanged head (11 executions, all success; job attempts 135098193369, 135103408040, 135106651054, 135111121813, 135115570823, 135119719184, 135123602783, 135127181860, 135130796666, 135133950196, 135137594890). Per-attempt logs and android-artifacts are retained (proof per attempt: ✔ live Android emulator fixture E2E, step-history capture Android Home/Recents surface (attempt N) steps, distinct md5s for foreground/home/recents PNGs proving both pixel comparisons, and home-recents-restoration in coverage-report.json proving fixture restoration). This satisfies Android emulator smoke: Home and Recents screenshot comparisons fail intermittently #4710's ten-consecutive-passes criterion at the final head.

…el assertions

The Android smoke scenario captured one screenshot immediately after home and
app-switcher and asserted it differed from the previous surface. When the
launcher or Recents transition had not drawn yet, the capture still showed the
previous surface and the identical-image assertion failed intermittently on
main and unrelated PRs (#4710).

assertHomeAndRecentsRestoration now polls screenshots through the new
waitForSurfaceChange helper at that seam: each attempt overwrites the artifact
so the final observed image is kept, the Home foreground-package proof and both
assertFilesDiffer pixel comparisons are retained, and the wait fails at its
10s deadline if navigation never produces the change. No fixed sleep, no
retry of the smoke test, no weakened assertion.

live-surface-transition.test.ts pins the contract deterministically under
fake timers: a delayed transition satisfies the wait only once a capture
differs (keeping that final image), transient capture failures are retried,
and a surface that never changes (or captures that keep failing) fails at the
deadline with the last observation.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 6.39 MB 6.39 MB 0 B
Package (unpacked) 6.39 MB 6.39 MB 0 B
Package (download) 1.94 MB 1.94 MB +7 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 46.7 ms 38.1 ms -8.7 ms
CLI --help 83.7 ms 72.4 ms -11.3 ms

@thymikee

Copy link
Copy Markdown
Member Author

The review of 13b6f71 found two issues that need attention before merge.

Could the regression exercise both Home and Recents through the scenario harness with delayed captures? The new cases call the helper directly, so restoring the old single captures in the scenario leaves them passing while the race returns. Please show that the regression fails with the old route. Test

The changed Android smoke path has no completed hosted run at the final head, and issue #4710 requires ten consecutive passes at an unchanged head. Please run the hosted scenario through Home, Recents, and fixture restoration ten times at the final head, retaining logs and artifacts that identify the head and prove both screenshot comparisons and restoration. Scenario

Not blocking: You can take or leave a change to pass the remaining deadline through screenshot capture, bound or defer failure evidence, reject late observations, and cover slow capture and failure paths.

The Android Smoke Tests job, which runs this scenario, was still running when I posted. No conflicts are known. Before merge, protect the scenario wiring, address the deadline overrun, and obtain the exact-head Android smoke evidence.

@apex-sentinel-staging

This comment has been minimized.

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 1 new finding on 13b6f71. The verdict and the full list are in the summary comment.

Comment on lines +19 to +46
export async function waitForSurfaceChange(
context: LiveContext,
baselinePath: string,
outputPath: string,
captureStep: string,
failure: string,
capture: SurfaceCapture = (output, attempt) =>
capturePng(context, `${captureStep} (attempt ${attempt})`, output),
): Promise<void> {
const deadline = Date.now() + SURFACE_TRANSITION_TIMEOUT_MS;
let lastObservation = 'no capture finished';
let attempt = 0;
for (
let remaining = SURFACE_TRANSITION_TIMEOUT_MS;
remaining > 0;
remaining = deadline - Date.now()
) {
attempt += 1;
try {
await capture(outputPath, attempt);
if (filesDiffer(baselinePath, outputPath)) return;
lastObservation = `capture ${attempt} still matched ${baselinePath}`;
} catch (error) {
lastObservation = `capture ${attempt} failed: ${error instanceof Error ? error.message : String(error)}`;
}
const pause = Math.min(SURFACE_TRANSITION_POLL_MS, deadline - Date.now());
if (pause > 0) await new Promise((resolve) => setTimeout(resolve, pause));
}

@apex-sentinel-staging apex-sentinel-staging Bot Nov 18, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer raised on 8516300: the code it pointed at changed.

🟡 Minor · waitForSurfaceChange does not bound a capture attempt against its 10 s deadline

If a single screenshot attempt stalls (wedged adb/daemon), await capture(...) blocks for up to 120 s because capturePng → runStep passes no timeoutMs, so runBuiltCliJson falls back to CLI_TIMEOUT_MS. The 10 s deadline is only re-checked after that subprocess returns, so the advertised wait overshoots by up to two minutes before assert.fail fires. The sibling waitForImeWindow avoids this by bounding each read with Math.min(remaining, IME_WINDOW_READ_TIMEOUT_MS).

Details

persona: javascript, senior-developer

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if it is wrong — I will re-check the cited code.

…d bound captures by the wait deadline

Review of #4716: the helper-level tests alone would still pass if the scenario
went back to single captures, so the regression now also runs
assertHomeAndRecentsRestoration through the real harness wiring (context runCli
override) against a scripted device whose first Home capture still shows the
fixture and whose first Recents capture still shows Home. The scenario passes
only via the polling wait, and fails at the 10s deadline when either surface
never changes — a single-capture route fails both pixel comparisons on the
first identical capture.

waitForSurfaceChange now passes the remaining deadline into each capture
(capped at 5s, allowFailure so per-attempt failure evidence stays deferred to
the deadline verdict), removes the stale artifact before each attempt, and
rejects a capture that resolves after the deadline as a late observation.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
@apex-sentinel-staging

Copy link
Copy Markdown

Apex Sentinel · ✅ Approve · review of 8516300

Since the review of 13b6f71: 1 new · 0 still open · 1 no longer raised

1 finding(s): 1 minor.

🟡 Minor (1)

Where Finding
test/integration/android-emulator-e2e/live-automation-scenario.ts:224 🆕 assertHomeAndRecentsRestoration exported only so a test can reach it, with no test-side harness entry

1 finding below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (6e18ea334968).

💬 Talking to Sentinel

In a finding's thread, reply with:

Command What it does
@apex-sentinel-staging note <context> Adds context; Sentinel re-checks the cited code
@apex-sentinel-staging pivot <why> Says the finding is wrong; Sentinel re-checks the cited code
@apex-sentinel-staging like Marks the finding as useful
@apex-sentinel-staging dislike <why> Marks it as not useful, so it is not raised again

On this conversation, comment @apex-sentinel-staging review to run a full review once again.

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 1 new finding on 8516300. The verdict and the full list are in the summary comment.

}

async function assertHomeAndRecentsRestoration(context: LiveContext): Promise<void> {
export async function assertHomeAndRecentsRestoration(context: LiveContext): Promise<void> {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🟡 Minor · assertHomeAndRecentsRestoration exported only so a test can reach it, with no test-side harness entry

A harness refactor that changes how steps are picked and run keeps this test green while the lane it guards goes unexercised. The test imports assertHomeAndRecentsRestoration directly from the scenario module and manually sets context.currentScenario = 'smoke:automation-system' to fake the route, bypassing the harness entry entirely.

Suggested change

Drive the scenario from the live-device harness entry (register it in the harness and run through createLiveDeviceHarness) instead of exporting the function directly, or document on the export why it is a test seam.

Details

persona: maintainability

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if it is wrong — I will re-check the cited code.

@thymikee

Copy link
Copy Markdown
Member Author

At 8516300, the default capture now forwards the remaining timeout, fixing the issue raised in the earlier review, but the regression coverage and hosted validation still need work. No conflicts are known.

Could the deadline test exercise the default capture through the harness with a CLI double that observes and enforces the forwarded timeout? The current test injects a capture that rejects itself, so removing the timeout from the default capture would leave it passing and allow the long screenshot wait to return. Please verify that removing the timeout makes the test fail.

The hosted run was still in progress, so ten consecutive hosted Android Smoke Tests passes at this head are not yet established. Please run the hosted Android smoke lane ten consecutive times at the unchanged head and retain links and output showing that smoke:automation-system reaches both Home and Recents transition waits and passes their pixel comparisons.

Could the test use the harness-owned runCli injection to run the registered scenario with a scripted device, keeping timeout handling at the screenshot observation boundary? That would require exposing or constructing the Android harness with its existing runner injection, then deleting the context override and scenario-internal export.

The open scenario runner thread still applies; no lower-priority threads still apply. The capture timeout thread no longer applies because the default capture forwards the bounded timeout.

Smoke Tests, which runs both changed transitions, was still running when I posted. Before merge, add a regression that detects removal of the default capture timeout and obtain ten consecutive hosted Android smoke passes at the exact head.

@thymikee

Copy link
Copy Markdown
Member Author

Both blocking findings are addressed at 8516300c956fa37f99a2a9a27a35fdbe9a28f9a3.

1. Scenario-level regression. New test/integration/android-emulator-e2e/live-automation-scenario.test.ts drives the real assertHomeAndRecentsRestoration through the scenario harness (shared createLiveDeviceHarness wiring: commonFlags, step history, command/behavior evidence) using a new per-context LiveDeviceContext.runCli override that scripts the device: the first Home capture still returns the fixture image and the first Recents capture still returns Home. The scenario then passes only through the polling wait (step history shows capture Android Home surface (attempt 2) / capture Android Recents surface (attempt 2)), and two negatives fail at the 10s deadline when either surface never changes. Red-on-old-route proof: with the two wait calls temporarily replaced by single capturePng calls (the pre-#4716 route), all three tests in that file fail with Android Home should replace fixture pixels / Android Recents should differ from Home pixels; with the wait restored, 9/9 pass across the two new test files.

2. Deadline overrun (taken). waitForSurfaceChange now passes the remaining deadline into each capture (capped at 5s per attempt, allowFailure so per-attempt failure evidence stays deferred to the deadline verdict), removes the stale artifact before each attempt, and rejects a capture that resolves after the deadline as a late observation. New deterministic tests: no capture outlives the remaining deadline and a capture that resolves after the deadline is rejected as a late observation.

3. Ten consecutive hosted Android smoke passes at the final head. Workflow run 44388138478 (android.yml, event pull_request, head_sha 8516300c956fa37f99a2a9a27a35fdbe9a28f9a3): the initial Smoke Tests job plus ten reruns of the same job at the unchanged head, all success — 11 consecutive executions, 10 of them reruns:

# job attempt conclusion
1 https://github.com/callstack/agent-device/actions/runs/44388138478/job/135098193369 success
2 https://github.com/callstack/agent-device/actions/runs/44388138478/job/135103408040 success
3 https://github.com/callstack/agent-device/actions/runs/44388138478/job/135106651054 success
4 https://github.com/callstack/agent-device/actions/runs/44388138478/job/135111121813 success
5 https://github.com/callstack/agent-device/actions/runs/44388138478/job/135115570823 success
6 https://github.com/callstack/agent-device/actions/runs/44388138478/job/135119719184 success
7 https://github.com/callstack/agent-device/actions/runs/44388138478/job/135123602783 success
8 https://github.com/callstack/agent-device/actions/runs/44388138478/job/135127181860 success
9 https://github.com/callstack/agent-device/actions/runs/44388138478/job/135130796666 success
10 https://github.com/callstack/agent-device/actions/runs/44388138478/job/135133950196 success
11 https://github.com/callstack/agent-device/actions/runs/44388138478/job/135137594890 success

Each attempt's full job log and android-artifacts download are retained (per-attempt proof.txt plus logs/artifacts, kept locally at /tmp/ad-smoke-4716/01..11). Every proof shows: ✔ live Android emulator fixture E2E with pass 1 / fail 0; step-history screenshot steps capture Android fixture foreground, capture Android Home surface (attempt N), capture Android Recents surface (attempt N) (the attempt suffix proves the new wait route is live in CI); three distinct md5s for foreground.png/home.png/recents.png (both pixel comparisons passed); and home-recents-restoration present in coverage-report.json (fixture restoration after Home/Recents).

Local gates at 8516300c95: pnpm check:affected --run --head 8516300c956fa37f99a2a9a27a35fdbe9a28f9a3 (receipt .tmp/validation/run-6RCKeL, exactCommit=true) — every gate passed except the unit suite's known host-load flake in scripts/__tests__/release-publish.test.ts (5s timeout; passes solo and also fails on unchanged origin/main under the same load); 20616/20627 unit tests passed, including all 9 new tests. Focused runs: 9/9 across the two new files plus the neighboring harness suites; pnpm typecheck, pnpm format, oxlint clean.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Android emulator smoke: Home and Recents screenshot comparisons fail intermittently

1 participant