[flaky-ci] Isolate Gradle state in CI - #12707
simonrozsival wants to merge 9 commits into
Conversation
Prevent concurrent single-use Gradle daemons from sharing the manifestmerger/java-source-utils registry, keep pipeline Gradle state job-local, and persist only dependency downloads and wrapper distributions. Preserve high-importance Gradle output for the previously opaque binding fixture failure. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
simonrozsival
added
the
flaky-ci
Use missing-safe cache state recovery and inspect the current Azure job timeline so only explicitly recovered optional cache failures remain non-gating. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Poll the current-job timeline within a wall-clock bound before applying recovered optional-task exemptions, so delayed task results cannot be masked. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use HttpClient full-response cancellation against the remaining wall-clock budget and reject overdue or unknown timeline results. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
/review |
|
✅ Android PR Reviewer completed successfully!
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
❌ Reject
Found 2 errors:
- Nested issue gates do not preserve the reusable templates’ custom
xaSourcePath, causing script-not-found failures for non-default checkouts. - The new HTTP integration test expects Azure log masking in raw loopback request bytes and therefore fails when run.
CI is green for Azure DevOps build 1587136. The Gradle cache isolation and fail-closed timeline logic are otherwise thoughtfully defensive, with good coverage of delayed and malformed timeline responses.
Generated by Android PR Reviewer for #12707 · gpt56 · 200.8 AIC · ⌖ 19.5 AIC · ⊞ 25.7K
Comment /review to run again
Preserve custom checkout paths in nested test templates and parse the actual authorization header in the loopback fixture. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
/review |
|
✅ Android PR Reviewer completed successfully!
|
There was a problem hiding this comment.
⚠️ Needs Changes
Findings: 0 errors, 0 warnings, 1 suggestion.
The Gradle state isolation, cache-recovery allowlist, timeline gate, and project-reference serialization are coherent with the surrounding build graph. I left one inline suggestion to cover the explicit non-success HTTP response path.
CI is currently red: build 1587818 failed in the Linux Build, macOS Build, and Windows Build & Smoke Test lanes. The available check metadata does not expose enough Azure log detail to attribute those failures to this change, but the PR is not merge-ready until those lanes are green.
Generated by Android PR Reviewer for #12707 · gpt56 · 240.9 AIC · ⌖ 8.84 AIC · ⊞ 25.7K
Comment /review to run again
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Exercise the explicit non-success response path in the fail-closed Azure timeline gate fixture. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The newly added PowerShell test fixtures for the gate/recovery logic are not currently wired to run in CI, reducing confidence that future edits won’t reintroduce failures.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Lite
Findings: 1
New issues introduced by this change (3)
| Severity | Finding |
|---|---|
 |
build-tools/automation/scripts/FailOnUnexpectedIssues.Tests.ps1 — |
 |
build-tools/automation/yaml-templates/cache-gradle.yaml — 💡 Similar to fail-on-issue.yaml, this direct path concatenation can be fragile if xaSourcePath… |
|
build-tools/automation/yaml-templates/fail-on-issue.yaml — 💡 Using string concatenation with .../${{ parameters.xaSourcePath }}/... is a bit brittle across… |
What changed in this PR
This PR aims to reduce CI flakiness by isolating Gradle state per job (while still caching the “download-only” bits) and by adding a stricter, timeline-based gate that only allows known-recovered optional cache failures to remain non-gating.
Changes:
- Isolates
GRADLE_USER_HOMEunder$(Agent.TempDirectory)and caches onlycaches/modules-2plus wrapper distributions, with recovery when cache restore is incomplete. - Serializes Gradle-wrapper–invoking projects by extending the project reference chain (preventing concurrent single-use daemons from racing).
- Improves diagnostics by preserving high-importance Gradle stdout/stderr for the binding zip test target and by replacing the simple “SucceededWithIssues => fail” gate with a timeline-aware allowlist gate.
| File | Description |
|---|---|
| tests/CodeGen-Binding/Xamarin.Android.LibraryProjectZip-LibBinding/Xamarin.Android.LibraryProjectZip-LibBinding.targets | Elevates Gradle stdout/stderr importance to improve failure diagnostics. |
| src/manifestmerger/manifestmerger.csproj | Adds a project reference to serialize Gradle wrapper usage with java-source-utils. |
| build-tools/automation/yaml-templates/variables.yaml | Sets job-level GRADLE_USER_HOME default under $(Agent.TempDirectory). |
| build-tools/automation/yaml-templates/stage-msbuild-emulator-tests.yaml | Plumbs xaSourcePath through to the issue gate template. |
| build-tools/automation/yaml-templates/run-msbuild-tests.yaml | Plumbs xaSourcePath through to the issue gate template. |
| build-tools/automation/yaml-templates/fail-on-issue.yaml | Replaces inline gating with FailOnUnexpectedIssues.ps1 and provides OAuth token env. |
| build-tools/automation/yaml-templates/cache-gradle.yaml | Creates isolated Gradle home, caches only download artifacts, and invokes recovery validation. |
| build-tools/automation/yaml-templates/build-linux-steps.yaml | Updates cleanup commentary to reflect new Gradle home location. |
| build-tools/automation/scripts/RecoverGradleCache.ps1 | Resets cache dirs to a verified-empty state when cache restore state is missing/invalid. |
| build-tools/automation/scripts/RecoverGradleCache.Tests.ps1 | Adds fixtures validating cache-reset and allowlist emission behavior. |
| build-tools/automation/scripts/FailOnUnexpectedIssues.ps1 | Implements timeline-based gate to fail closed unless only allowlisted recovered tasks are in issue state. |
| build-tools/automation/scripts/FailOnUnexpectedIssues.Tests.ps1 | Adds fixtures validating timeline polling, allowlisting, and fail-closed behavior. |
| build-tools/automation/azure-pipelines.yaml | Passes xaSourcePath to the gate template for correct script resolution. |
| build-tools/automation/azure-pipelines-public.yaml | Passes xaSourcePath to the gate template for correct script resolution. |
| build-tools/automation/azure-pipelines-internal.yaml | Passes xaSourcePath to the gate template for correct script resolution. |
Execute the PowerShell regression fixtures once in the Linux build and resolve script paths with Join-Path across platforms. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Summary
java-source-utilsbeforemanifestmerger, extending the existing Gradle project chain so their single-use daemons cannot race over one registryGRADLE_USER_HOMEunder$(Agent.TempDirectory)and persist onlycaches/modules-2plus wrapper distributionsXamarin.Android.LibraryProjectZip-LibBindingfailureRoot cause
The nine Windows incidents in builds 1574227, 1574359, 1575505, 1576382, 1576423, 1576487, 1576567, 1578693, and 1578694 all failed while
manifestmergerandjava-source-utilsran concurrently. Although both use--no-daemon, Gradle starts a single-use daemon and both processes sharedC:\Users\cloudtest\.gradle\daemon\9.5.0. The failing stack ends in daemon registry removal and file-lock contention handling withTimeout waiting to connect to the Gradle daemon.Build 1575255 has a separate macOS Kotlin
ClasspathEntrySnapshotTransform/KotlinCompilerVersionfailure and is intentionally not covered by a generic retry. Build 1576677 did not retain Gradle stdout/stderr, so this change improves that target's diagnostics without assuming the same cause.References #12704.
Optional cache recovery contract
Cache tasks have stable task reference names. Recovery reads cache state through missing-safe environment variables and adds a task reference to the semicolon-delimited
RECOVERED_OPTIONAL_TASK_REFSvariable only after the affected cache path has been reset to a verified empty directory.The final issue gate has the stable
failOnUnexpectedIssuestask reference and polls the current Azure build timeline within a wall-clock bound until its own record is visible and every preceding direct task underSYSTEM_JOBIDis terminal with a documented result. Timeline HTTP requests useHttpClient.SendAsync(..., ResponseContentRead, cancellationToken)with the remaining budget, so response-body publication time is included and no overdue response—including the first—is accepted. The gate then permits only failed orSucceededWithIssuesrecords whose task reference/name appears in the runtime allowlist; the gate and later/post-job records are excluded. Missing tokens, missing or malformed timeline data, API failures, unsupported task results, incomplete publication at timeout, absent allowlists, and unrelated issue tasks all fail closed.Validation
Mono.Androidstatic graph reuses onejava-source-utilsproject instance through the newmanifestmergerdependencyjava-source-utilswith a cold isolated Gradle home and reran it warmmanifestmergertwice with the same isolated Gradle home; both Gradle invocations completed successfully