[flaky-ci] Recover MAUI R2R Helix device failures - #12715
simonrozsival wants to merge 5 commits into
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
simonrozsival
added
the
flaky-ci
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
/review |
|
✅ Android PR Reviewer completed successfully!
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
The device selection, package allow-listing, bounded retries, and process-level regression coverage are well structured, and all 44 reported checks are green. The recovery path still needs an end-to-end budget below Helix’s 10-minute work-item timeout; otherwise the outer timeout can preempt the explicit diagnostics and cleanup this change is intended to provide. I also noted that repeated device polls overwrite their earlier snapshots.
Generated by Android PR Reviewer for #12715 · gpt56 · 123.4 AIC · ⌖ 8.58 AIC · ⊞ 26.3K
Comment /review to run again
Cap the complete MAUI R2R work item at nine minutes so repository recovery and final error handling finish before Helix's ten-minute timeout. Preserve every repeated ADB device snapshot and retain the command transcript that consumes the overall deadline. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
/review |
|
✅ Android PR Reviewer completed successfully!
|
There was a problem hiding this comment.
⚠️ Needs Changes
Found 1 warning: the new runner re-encodes ADB diagnostics as ASCII, which can corrupt non-ASCII crash and log output. The bounded recovery flow, package allow-list, targeted uninstall behavior, and fake-ADB coverage are otherwise well structured.
CI is currently red: the macOS and Linux build checks have failed, while the Windows build is still in progress. Those failures were not duplicated as review comments because their detailed cause was not available from the check results.
Generated by Android PR Reviewer for #12715 · gpt56 · 115.3 AIC · ⌖ 11.3 AIC · ⊞ 25.7K
Comment /review to run again
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The new adb device parsing can mis-handle real-world adb devices output noise and should be made robust before relying on it for CI recovery decisions.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Lite
Findings: 1
New issues introduced by this change (2)
| Severity | Finding |
|---|---|
 |
build-tools/automation/scripts/RunMauiR2RHelix.ps1 — ❌ bug: Get-AdbDeviceSnapshot currently treats any non-empty line as a device row, so adb daemon… |
 |
build-tools/automation/scripts/RunMauiR2RHelix.ps1 — |
What changed in this PR
This PR hardens the MAUI R2R Helix device work-item execution by replacing the previous inline batch logic with a dedicated, tested PowerShell runner that performs bounded device recovery (ADB restart, UID-exhaustion reboot/install retry), reduces package churn, and uploads diagnostics for infrastructure failures.
Changes:
- Replace embedded
adbbatch workflow in the Helix payload withRunMauiR2RHelix.ps1and ship it in the work-item payload. - Add a PowerShell test suite (including process exit-code validation) and wire it into the YAML template.
- Implement bounded recovery paths (single ADB restart; single reboot+install retry for UID exhaustion; single uninstall+retry for update-incompatible).
| File | Description |
|---|---|
| build-tools/automation/yaml-templates/run-maui-r2r-helix-matrix.yaml | Copies the new runner into the Helix payload and adds test execution steps in the pipeline template. |
| build-tools/automation/scripts/RunMauiR2RHelix.ps1 | New Helix-side device runner implementing bounded recovery + diagnostics capture. |
| build-tools/automation/scripts/RunMauiR2RHelix.Tests.ps1 | New unit-style PowerShell tests using a fake adb to validate recovery boundaries and no-uninstall behavior. |
| build-tools/automation/scripts/RunMauiR2RHelix.Tests.Process.ps1 | New process-level wrapper to validate exit-code behavior when running the test suite under pwsh. |
Ignore ADB daemon startup banners when selecting the attached device and keep ADB output UTF-8 through capture and uploaded diagnostics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
/review |
|
✅ Android PR Reviewer completed successfully!
|
There was a problem hiding this comment.
⚠️ Needs Changes
Found 1 error: the new no-uninstall flow can false-pass by launching an allowed package left from a previous work item when the newly built APK installs under an unexpected application ID. The inline comment describes how to bind validation to the APK under test.
The bounded ADB recovery, allow-listed cleanup, diagnostic preservation, and focused fake-ADB coverage are otherwise well structured. All 44 reported checks, including the Azure DevOps dotnet-android build, are green.
Generated by Android PR Reviewer for #12715 · copilot · gpt56 · 99.8 AIC · ⌖ 11.2 AIC · ⊞ 21K
Comment /review to run again
| $result = Invoke-Adb $arguments -TimeoutSeconds 180 | ||
| Write-CommandResult "install-$ConfigurationName-attempt-$attempt" $arguments $result | ||
| if ($result.ExitCode -eq 0) { | ||
| return |
There was a problem hiding this comment.
🤖 ❌ Testing — A successful adb install -r does not prove that this APK updated $PackageName: if the built APK unexpectedly has a different application ID, Android installs it as a second package, and the allowed package retained from a previous work item still makes Verify-MauiR2RPackage and the launch check pass against stale code. Please verify the APK application ID before installation or otherwise prove that the selected package was replaced, and cover the stale-package case in the fake-ADB suite.
Rule: Do not validate stale artifacts.
| [switch] $SkipExtendedDiagnostics | ||
| ) | ||
|
|
||
| $ErrorActionPreference = 'Stop' |
There was a problem hiding this comment.
We are using both bash and yaml for CI, and this introduces powershell (which we might already have a small amount of).
Is there a way this could be done without introducing more scripts (more code)? I'm wondering if inline bash in yaml is what we should standardize on.
Summary
INSTALL_FAILED_INSUFFICIENT_STORAGEwithcould not be assigned a valid UIDRecovery boundaries
The runner allow-lists the four MAUI R2R package IDs and never uninstalls unrelated packages. The outer Helix work-item timeout remains unchanged; all internal command and recovery waits are bounded.
Service-owner boundary
The repository cannot authorize or physically attach a device, quarantine a Helix host, remove unrelated applications, or factory-reset/reprovision a device. Persistent unauthorized/no-device states and UID exhaustion that survives one reboot remain explicit failures for the Helix/device-lab owners tracked by dotnet/arcade#17483.
Validation
RunMauiR2RHelix.Tests.ps1, covering healthy devices, ADB restart recovery, missing/unauthorized/multiple devices, UID reboot recovery, persistent UID exhaustion, signature cleanup, unknown install failures, emptypidofoutput, and no-uninstall cleanupConfigureMauiR2RHelixEnvironment.Tests.ps1git diff --checkAddresses #12658.
Flaky-CI inventory: #12704