[CI] Retry flaky-test artifact downloads - #4153
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/4153
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 New Failures, 1 Unrelated FailureAs of commit 9900f33 with merge base 236e29a ( NEW FAILURES - The following jobs have failed:
BROKEN TRUNK - The following job failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks! |
|
| Prefix | Label Applied | Example |
|---|---|---|
[Algorithm] |
new algo | [Algorithm] Add new RL objective |
[BE] |
BE | [BE] Improve error messages |
[Benchmark] or [Benchmarks] |
Benchmarks | [Benchmark] Add collector benchmark |
[BugFix] |
BugFix | [BugFix] Fix memory leak in collector |
[Example] or [Examples] |
Examples | [Example] Add training script |
[Feature] |
Feature | [Feature] Add new optimizer |
[Doc] or [Docs] |
Documentation | [Doc] Update installation guide |
[Refactor] |
Refactoring | [Refactor] Clean up module imports |
[CI] |
CI | [CI] Fix workflow permissions |
[Test] or [Tests] |
Tests | [Tests] Add unit tests for buffer |
[Trainer] or [Trainers] |
Trainers | [Trainer] Add trainer config |
[Environment] or [Environments] |
Environments | [Environments] Add Gymnasium support |
[Data] |
Data | [Data] Fix replay buffer sampling |
[LLM] |
llm/ | [LLM] Add reward model integration |
[Minor] |
small change | [Minor] Fix typo in error message |
[Performance] or [Perf] |
Performance | [Performance] Optimize tensor ops |
[BC-Breaking] |
bc breaking | [BC-Breaking] Remove deprecated API |
[Deprecation] |
Deprecation | [Deprecation] Mark old function |
[Algorithm] or [Algorithms] |
new algo | [Algorithm] Add new objective |
[Quality] |
Quality | [Quality] Fix typos and add codespell |
[Versioning] |
versioning | [Versioning] Bump release version |
[WIP] |
WIP | [WIP] Draft implementation |
Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks! |
|
Thanks for the follow-up commit. The latest CI still shows one new Dreamer test failure and the known Gym trunk failure. I don’t have permission to rerun failed jobs (GitHub returns HTTP 403). Could a maintainer please rerun the failed jobs when convenient? |
| delay = ARTIFACT_DOWNLOAD_BACKOFF_SECONDS * 2**attempt | ||
| log( | ||
| "Warning: Artifact download returned " | ||
| f"{response.status_code}; retrying in {delay}s" | ||
| ) | ||
| time.sleep(delay) |
There was a problem hiding this comment.
Retry-After: 120 slept for 2 seconds in my probe. every retry can land inside the same cooldown. can a 429 honor Retry-After before local backoff?
|
Thanks for catching this. Commit 9900f33 now honors the Retry-After header for 429 responses, using the larger of the server-provided delay and the local exponential backoff. It supports both delay-seconds and HTTP-date values and adds regression coverage. Focused local tests pass: 4 passed. |
Description
Fixes #4054. The flaky-test tracker currently fails the whole analysis job when a GitHub Actions artifact download encounters a transient connection reset, TLS error, or retryable 5xx/429 response.
Artifact downloads now retry request exceptions and transient HTTP statuses with bounded exponential backoff, while permanent HTTP errors still fail fast.
Motivation and Context
The failed tracker runs in #4054 show
ConnectionResetErrorandSSLCertVerificationErrorwhile downloading artifact archives from GitHub. The API requests succeed, but the redirected artifact request is not retried.Types of changes
Checklist
Validation
python -m pytest -q .github/scripts/test_analyze_flaky_tests.pypython -m ruff check --ignore EXE001,PERF102 .github/scripts/analyze_flaky_tests.py .github/scripts/test_analyze_flaky_tests.pyDisclosure
AI assistance was used for repository exploration and drafting. I reviewed the implementation and ran the validation commands above.