Skip to content

[CI] Retry flaky-test artifact downloads - #4153

Open
CoffeeDrivenCoder wants to merge 4 commits into
pytorch:mainfrom
CoffeeDrivenCoder:fix/flaky-tracker-artifact-download
Open

[CI] Retry flaky-test artifact downloads#4153
CoffeeDrivenCoder wants to merge 4 commits into
pytorch:mainfrom
CoffeeDrivenCoder:fix/flaky-tracker-artifact-download

Conversation

@CoffeeDrivenCoder

Copy link
Copy Markdown

Description

Fixes #4054. The flaky-test tracker currently fails the whole analysis job when a GitHub Actions artifact download encounters a transient connection reset, TLS error, or retryable 5xx/429 response.

Artifact downloads now retry request exceptions and transient HTTP statuses with bounded exponential backoff, while permanent HTTP errors still fail fast.

Motivation and Context

The failed tracker runs in #4054 show ConnectionResetError and SSLCertVerificationError while downloading artifact archives from GitHub. The API requests succeed, but the redirected artifact request is not retried.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)

Checklist

  • I have read the CONTRIBUTION guide.
  • I have updated the tests accordingly.

Validation

  • python -m pytest -q .github/scripts/test_analyze_flaky_tests.py
  • python -m ruff check --ignore EXE001,PERF102 .github/scripts/analyze_flaky_tests.py .github/scripts/test_analyze_flaky_tests.py
  • Existing TorchRL CPU/GPU regression tests for the tracked flaky-test entries pass locally.

Disclosure

AI assistance was used for repository exploration and drafting. I reviewed the implementation and ran the validation commands above.

@pytorch-bot

pytorch-bot Bot commented Aug 24, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/4153

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 1 Unrelated Failure

As of commit 9900f33 with merge base 236e29a (image):

NEW FAILURES - The following jobs have failed:

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla

meta-cla Bot commented Aug 24, 2026

Copy link
Copy Markdown

Hi @CoffeeDrivenCoder!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ PR Title Label Error

PR title must start with a label prefix in brackets (e.g., [BugFix]).

Current title: CI: retry flaky-test artifact downloads

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix Label Applied Example
[Algorithm] new algo [Algorithm] Add new RL objective
[BE] BE [BE] Improve error messages
[Benchmark] or [Benchmarks] Benchmarks [Benchmark] Add collector benchmark
[BugFix] BugFix [BugFix] Fix memory leak in collector
[Example] or [Examples] Examples [Example] Add training script
[Feature] Feature [Feature] Add new optimizer
[Doc] or [Docs] Documentation [Doc] Update installation guide
[Refactor] Refactoring [Refactor] Clean up module imports
[CI] CI [CI] Fix workflow permissions
[Test] or [Tests] Tests [Tests] Add unit tests for buffer
[Trainer] or [Trainers] Trainers [Trainer] Add trainer config
[Environment] or [Environments] Environments [Environments] Add Gymnasium support
[Data] Data [Data] Fix replay buffer sampling
[LLM] llm/ [LLM] Add reward model integration
[Minor] small change [Minor] Fix typo in error message
[Performance] or [Perf] Performance [Performance] Optimize tensor ops
[BC-Breaking] bc breaking [BC-Breaking] Remove deprecated API
[Deprecation] Deprecation [Deprecation] Mark old function
[Algorithm] or [Algorithms] new algo [Algorithm] Add new objective
[Quality] Quality [Quality] Fix typos and add codespell
[Versioning] versioning [Versioning] Bump release version
[WIP] WIP [WIP] Draft implementation

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

@github-actions github-actions Bot added the CI Has to do with CI setup (e.g. wheels & builds, tests...) label Aug 24, 2026
@CoffeeDrivenCoder CoffeeDrivenCoder changed the title CI: retry flaky-test artifact downloads [CI] Retry flaky-test artifact downloads Aug 24, 2026
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 25, 2026
@meta-cla

meta-cla Bot commented Aug 25, 2026

Copy link
Copy Markdown

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

@CoffeeDrivenCoder

Copy link
Copy Markdown
Author

Thanks for the follow-up commit. The latest CI still shows one new Dreamer test failure and the known Gym trunk failure. I don’t have permission to rerun failed jobs (GitHub returns HTTP 403). Could a maintainer please rerun the failed jobs when convenient?

Comment on lines +147 to +152
delay = ARTIFACT_DOWNLOAD_BACKOFF_SECONDS * 2**attempt
log(
"Warning: Artifact download returned "
f"{response.status_code}; retrying in {delay}s"
)
time.sleep(delay)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retry-After: 120 slept for 2 seconds in my probe. every retry can land inside the same cooldown. can a 429 honor Retry-After before local backoff?

@CoffeeDrivenCoder

Copy link
Copy Markdown
Author

Thanks for catching this. Commit 9900f33 now honors the Retry-After header for 429 responses, using the larger of the server-provided delay and the local exponential backoff. It supports both delay-seconds and HTTP-date values and adds regression coverage. Focused local tests pass: 4 passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI Has to do with CI setup (e.g. wheels & builds, tests...) CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CI] Flaky Test Tracker workflow is failing

3 participants