[flaky-ci] Bound hosted dependency acquisition - #12705
Open
simonrozsival wants to merge 5 commits into
Open
simonrozsival wants to merge 5 commits into
simonrozsival wants to merge 5 commits into
Conversation
Fall back to verified source downloads when the Azure archive cache is unavailable, and bound workload-pack acquisition with Retry-After-aware NuGet limits, clean retry state, and a per-attempt timeout. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
simonrozsival
added
the
flaky-ci
Classify only dynamically recovered optional cache failures from a complete Azure timeline, and bound workload installation retries inside the child process. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Member
Author
|
/review |
Contributor
|
✅ Android PR Reviewer completed successfully!
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
✅ LGTM — 0 errors, 0 warnings, 1 testing suggestion.
The recovery gate fails closed for malformed/incomplete timelines, limits the exception to explicitly recovered cache tasks, and the workload retry remains bounded at both process and task levels. All 44 reported CI checks are green.
Generated by Android PR Reviewer for #12705 · gpt56 · 260.6 AIC · ⌖ 3.54 AIC · ⊞ 26.1K
Comment /review to run again
Cover the retry path where the first process exits nonzero and the second attempt succeeds. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…hosted-dependency-ci
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
run-dotnet-preview.yaml invokes Invoke-ProcessWithRetry.ps1 in-process even though it uses exit, which will terminate the calling script block and skip subsequent error-handling logic.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Lite
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
 |
build-tools/automation/yaml-templates/run-dotnet-preview.yaml — ❌ error: Invoke-ProcessWithRetry.ps1 uses exit to return an exit code, but this template… |
What changed in this PR
This PR hardens the Azure DevOps PR pipeline against flaky dependency acquisition (Cache@2 rate limits, workload pack download stalls) by treating certain cache failures as recoverable and by bounding retry/timeout behavior for workload extraction, while failing closed on unexpected or malformed timeline results.
Changes:
- Make Android archive caching non-fatal on Cache@2 failure, then explicitly recover by resetting the archive cache directory and allowlisting the cache task as “recovered optional”.
- Replace the “SucceededWithIssues” gate with a timeline-based validator that only ignores known recovered optional tasks and otherwise fails closed.
- Add bounded per-process retry/timeout handling for
ExtractWorkloadPacks, plus NuGet 429/Retry-After limits and disabled HTTP-cache reuse between attempts.
| File | Description |
|---|---|
| build-tools/create-packs/Directory.Build.targets | Cleans stale workload temp dirs and adds --no-http-cache to workload install to reduce stuck/stale downloads. |
| build-tools/automation/yaml-templates/setup-test-environment-steps.yaml | Switches workload extraction to bounded task/process timeouts and process-level retries; configures NuGet retry env vars. |
| build-tools/automation/yaml-templates/run-dotnet-preview.yaml | Adds optional process-timeout + retry wrapper support for dotnet invocations (currently with a call pattern issue noted in comments). |
| build-tools/automation/yaml-templates/fail-on-issue.yaml | Replaces inline job-status check with FailOnUnexpectedIssues.ps1 and passes OAuth token to enable timeline queries. |
| build-tools/automation/yaml-templates/cache-android-archives.yaml | Names the Cache@2 task, marks it continueOnError, and runs recovery script to reset cache path + record recovered task refs. |
| build-tools/automation/scripts/RecoverAndroidArchiveCache.Tests.ps1 | Adds tests for cache recovery behavior and recovered-task allowlist updates. |
| build-tools/automation/scripts/RecoverAndroidArchiveCache.ps1 | Implements cache failure recovery by resetting cache directory and setting RECOVERED_OPTIONAL_TASK_REFS. |
| build-tools/automation/scripts/Invoke-ProcessWithRetry.Tests.ps1 | Adds tests for bounded timeout behavior and retry-on-nonzero-exit behavior. |
| build-tools/automation/scripts/Invoke-ProcessWithRetry.ps1 | Implements process execution with timeout, process-tree termination, and bounded retries. |
| build-tools/automation/scripts/FailOnUnexpectedIssues.Tests.ps1 | Adds comprehensive tests for timeline polling, fail-closed behavior, and recovered-only allowlisting. |
| build-tools/automation/scripts/FailOnUnexpectedIssues.ps1 | Implements timeline-based validation to only ignore known recovered optional task failures under SucceededWithIssues. |
| build-tools/automation/azure-pipelines.yaml | Wires xaSourcePath into the new fail-on-issue template usage. |
| build-tools/automation/azure-pipelines-public.yaml | Wires xaSourcePath into the new fail-on-issue template usage (public pipeline). |
| build-tools/automation/azure-pipelines-internal.yaml | Wires xaSourcePath into the new fail-on-issue template usage (internal pipeline). |
Cover in-process call-operator invocation for success, failure, and timeout. Record results from a separate caller to detect premature exits without changing the working template behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cache@2failures only after resetting the disposable cache path, then dynamically allowlist the stableandroidArchivesCachetask referenceRetry-AfterlimitsFailure analysis
Build 1578771 failed when
Cache@2returned HTTP 429 before Android archive source downloads could run. Build 1579664 spent 220 minutes inextract workload packs: several 60-second NuGet download stalls recovered, butdotnet workload installthen stopped making progress until the macOS job reached its 240-minute limit.This change retries only dependency acquisition; it does not add retries around builds or tests.
Validation
FailOnUnexpectedIssues.Tests.ps1— recovered-only pass, unrelated issue failure, eventual timeline completion, missing gate, malformed/API failures, and delayed response-body deadlineRecoverAndroidArchiveCache.Tests.ps1— valid, unset/failed, appended allowlist, and failed reset casesInvoke-ProcessWithRetry.Tests.ps1— two deterministic hanging attempts fail within the total boundbuild-tools/create-packs/Microsoft.Android.Sdk.projReferences #12704