Repository navigation
Conversation
…el assertions The Android smoke scenario captured one screenshot immediately after home and app-switcher and asserted it differed from the previous surface. When the launcher or Recents transition had not drawn yet, the capture still showed the previous surface and the identical-image assertion failed intermittently on main and unrelated PRs (#4710). assertHomeAndRecentsRestoration now polls screenshots through the new waitForSurfaceChange helper at that seam: each attempt overwrites the artifact so the final observed image is kept, the Home foreground-package proof and both assertFilesDiffer pixel comparisons are retained, and the wait fails at its 10s deadline if navigation never produces the change. No fixed sleep, no retry of the smoke test, no weakened assertion. live-surface-transition.test.ts pins the contract deterministically under fake timers: a delayed transition satisfies the wait only once a capture differs (keeping that final image), transient capture failures are retried, and a surface that never changes (or captures that keep failing) fails at the deadline with the last observation. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
Size Report
Startup median (7 runs, lower is better):
|
|
The review of 13b6f71 found two issues that need attention before merge. Could the regression exercise both Home and Recents through the scenario harness with delayed captures? The new cases call the helper directly, so restoring the old single captures in the scenario leaves them passing while the race returns. Please show that the regression fails with the old route. Test The changed Android smoke path has no completed hosted run at the final head, and issue #4710 requires ten consecutive passes at an unchanged head. Please run the hosted scenario through Home, Recents, and fixture restoration ten times at the final head, retaining logs and artifacts that identify the head and prove both screenshot comparisons and restoration. Scenario Not blocking: You can take or leave a change to pass the remaining deadline through screenshot capture, bound or defer failure evidence, reject late observations, and cover slow capture and failure paths. The Android Smoke Tests job, which runs this scenario, was still running when I posted. No conflicts are known. Before merge, protect the scenario wiring, address the deadline overrun, and obtain the exact-head Android smoke evidence. |
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Apex Sentinel: 1 new finding on 13b6f71. The verdict and the full list are in the summary comment.
| export async function waitForSurfaceChange( | ||
| context: LiveContext, | ||
| baselinePath: string, | ||
| outputPath: string, | ||
| captureStep: string, | ||
| failure: string, | ||
| capture: SurfaceCapture = (output, attempt) => | ||
| capturePng(context, `${captureStep} (attempt ${attempt})`, output), | ||
| ): Promise<void> { | ||
| const deadline = Date.now() + SURFACE_TRANSITION_TIMEOUT_MS; | ||
| let lastObservation = 'no capture finished'; | ||
| let attempt = 0; | ||
| for ( | ||
| let remaining = SURFACE_TRANSITION_TIMEOUT_MS; | ||
| remaining > 0; | ||
| remaining = deadline - Date.now() | ||
| ) { | ||
| attempt += 1; | ||
| try { | ||
| await capture(outputPath, attempt); | ||
| if (filesDiffer(baselinePath, outputPath)) return; | ||
| lastObservation = `capture ${attempt} still matched ${baselinePath}`; | ||
| } catch (error) { | ||
| lastObservation = `capture ${attempt} failed: ${error instanceof Error ? error.message : String(error)}`; | ||
| } | ||
| const pause = Math.min(SURFACE_TRANSITION_POLL_MS, deadline - Date.now()); | ||
| if (pause > 0) await new Promise((resolve) => setTimeout(resolve, pause)); | ||
| } |
There was a problem hiding this comment.
No longer raised on 8516300: the code it pointed at changed.
🟡 Minor · waitForSurfaceChange does not bound a capture attempt against its 10 s deadline
If a single screenshot attempt stalls (wedged adb/daemon), await capture(...) blocks for up to 120 s because capturePng → runStep passes no timeoutMs, so runBuiltCliJson falls back to CLI_TIMEOUT_MS. The 10 s deadline is only re-checked after that subprocess returns, so the advertised wait overshoots by up to two minutes before assert.fail fires. The sibling waitForImeWindow avoids this by bounding each read with Math.min(remaining, IME_WINDOW_READ_TIMEOUT_MS).
Details
persona: javascript, senior-developer
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if it is wrong — I will re-check the cited code.
…d bound captures by the wait deadline Review of #4716: the helper-level tests alone would still pass if the scenario went back to single captures, so the regression now also runs assertHomeAndRecentsRestoration through the real harness wiring (context runCli override) against a scripted device whose first Home capture still shows the fixture and whose first Recents capture still shows Home. The scenario passes only via the polling wait, and fails at the 10s deadline when either surface never changes — a single-capture route fails both pixel comparisons on the first identical capture. waitForSurfaceChange now passes the remaining deadline into each capture (capped at 5s, allowFailure so per-attempt failure evidence stays deferred to the deadline verdict), removes the stale artifact before each attempt, and rejects a capture that resolves after the deadline as a late observation. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
Apex Sentinel · ✅ Approve · review of
|
| Where | Finding |
|---|---|
test/integration/android-emulator-e2e/live-automation-scenario.ts:224 |
🆕 assertHomeAndRecentsRestoration exported only so a test can reach it, with no test-side harness entry |
1 finding below minor hidden (user_settings.min_severity).
Reviewed with this repository's own Sentinel configuration (6e18ea334968).
💬 Talking to Sentinel
In a finding's thread, reply with:
| Command | What it does |
|---|---|
@apex-sentinel-staging note <context> |
Adds context; Sentinel re-checks the cited code |
@apex-sentinel-staging pivot <why> |
Says the finding is wrong; Sentinel re-checks the cited code |
@apex-sentinel-staging like |
Marks the finding as useful |
@apex-sentinel-staging dislike <why> |
Marks it as not useful, so it is not raised again |
On this conversation, comment @apex-sentinel-staging review to run a full review once again.
There was a problem hiding this comment.
Apex Sentinel: 1 new finding on 8516300. The verdict and the full list are in the summary comment.
| } | ||
|
|
||
| async function assertHomeAndRecentsRestoration(context: LiveContext): Promise<void> { | ||
| export async function assertHomeAndRecentsRestoration(context: LiveContext): Promise<void> { |
There was a problem hiding this comment.
🆕 🟡 Minor · assertHomeAndRecentsRestoration exported only so a test can reach it, with no test-side harness entry
A harness refactor that changes how steps are picked and run keeps this test green while the lane it guards goes unexercised. The test imports assertHomeAndRecentsRestoration directly from the scenario module and manually sets context.currentScenario = 'smoke:automation-system' to fake the route, bypassing the harness entry entirely.
Suggested change
Drive the scenario from the live-device harness entry (register it in the harness and run through createLiveDeviceHarness) instead of exporting the function directly, or document on the export why it is a test seam.
Details
persona: maintainability
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if it is wrong — I will re-check the cited code.
|
At 8516300, the default capture now forwards the remaining timeout, fixing the issue raised in the earlier review, but the regression coverage and hosted validation still need work. No conflicts are known. Could the deadline test exercise the default capture through the harness with a CLI double that observes and enforces the forwarded timeout? The current test injects a capture that rejects itself, so removing the timeout from the default capture would leave it passing and allow the long screenshot wait to return. Please verify that removing the timeout makes the test fail. The hosted run was still in progress, so ten consecutive hosted Android Smoke Tests passes at this head are not yet established. Please run the hosted Android smoke lane ten consecutive times at the unchanged head and retain links and output showing that smoke:automation-system reaches both Home and Recents transition waits and passes their pixel comparisons. Could the test use the harness-owned runCli injection to run the registered scenario with a scripted device, keeping timeout handling at the screenshot observation boundary? That would require exposing or constructing the Android harness with its existing runner injection, then deleting the context override and scenario-internal export. The open scenario runner thread still applies; no lower-priority threads still apply. The capture timeout thread no longer applies because the default capture forwards the bounded timeout. Smoke Tests, which runs both changed transitions, was still running when I posted. Before merge, add a regression that detects removal of the default capture timeout and obtain ten consecutive hosted Android smoke passes at the exact head. |
|
Both blocking findings are addressed at 1. Scenario-level regression. New 2. Deadline overrun (taken). 3. Ten consecutive hosted Android smoke passes at the final head. Workflow run 44388138478 ( Each attempt's full job log and Local gates at |
Summary
Closes #4710.
The Android emulator smoke scenario (
smoke:automation-system) took one screenshot immediately afterhomeand afterapp-switcherand asserted it differed from the previous surface. When the launcher or the Recents transition had not drawn yet, the capture still showed the previous surface, so the identical-image assertion failed intermittently onmainand on unrelated PRs (e.g. #4297).assertHomeAndRecentsRestorationnow waits at that observation seam with the boundedwaitForSurfaceChangehelper (test/integration/android-emulator-e2e/live-surface-transition.ts, same contract as the existingwaitForImeWindow): re-capture every 250 ms until the screenshot differs from the baseline, within a 10 s deadline. Each attempt removes and overwrites the artifact, so the final observed image is kept and a failed capture cannot leave a stale one; captures are bounded by the remaining deadline (5 s cap, failure evidence deferred to the deadline verdict); a capture resolving after the deadline is rejected as late. The Home foreground-package proof and bothassertFilesDifferpixel comparisons are retained unchanged; the wait fails at the deadline if navigation never produces the change. No fixed blanket sleep, no smoke-test retry, no weakened assertion, no production change, no workflow/runner change (#4712 untouched).10 files touched (2 new test files + the new helper). Regression coverage is at both levels:
live-surface-transition.test.tspins the helper contract under fake timers (delayed transition, transient capture failures, timeout negative, no capture outlives the remaining deadline, late observations rejected), andlive-automation-scenario.test.tsdrives the real scenario through the harness with a scripted device (newLiveDeviceContext.runCliseam) whose first Home/Recents captures are stale — a single-capture route fails those tests, verified by temporarily restoring the old route.Validation
8516300c956fa37f99a2a9a27a35fdbe9a28f9a3afterpnpm install --frozen-lockfile && pnpm build.pnpm check:affected --run --head 8516300c956fa37f99a2a9a27a35fdbe9a28f9a3(receipt.tmp/validation/run-6RCKeL,exactCommit=true): every gate passed except theunitsuite's host-load flake inscripts/__tests__/release-publish.test.ts(5 s timeout; passes solo and also fails on unchangedorigin/mainin an isolated worktree under the same load); 20616/20627 unit tests passed including all 9 new ones. Focused: 9/9 in the two new files plus the neighboring harness suites (27/27 in the android-emulator-e2e + runtime suites);pnpm typecheck,pnpm format,oxlintclean;node --test test/integration/smoke-android-emulator-coverage.test.ts10/10.head_sha 8516300c956fa37f99a2a9a27a35fdbe9a28f9a3— the Smoke Tests job passed on the initial run plus ten consecutive reruns at the unchanged head (11 executions, all success; job attempts 135098193369, 135103408040, 135106651054, 135111121813, 135115570823, 135119719184, 135123602783, 135127181860, 135130796666, 135133950196, 135137594890). Per-attempt logs andandroid-artifactsare retained (proof per attempt:✔ live Android emulator fixture E2E, step-historycapture Android Home/Recents surface (attempt N)steps, distinct md5s for foreground/home/recents PNGs proving both pixel comparisons, andhome-recents-restorationin coverage-report.json proving fixture restoration). This satisfies Android emulator smoke: Home and Recents screenshot comparisons fail intermittently #4710's ten-consecutive-passes criterion at the final head.