Repository navigation
DOI-3830: fix: rebase and retry push with jitter on non-fast-forward - #4
Open
yashkalwaniDW wants to merge 1 commit into
Open
yashkalwaniDW wants to merge 1 commit into
yashkalwaniDW wants to merge 1 commit into
Conversation
Parallel preview destroy/deploy jobs race on the deploy repo's main branch; a fixed 10s whole-step retry made colliding jobs retry in lockstep and lose. Retry the push in place: pull --rebase, jittered sleep, up to 6 tries. Raise default commit-timeout 60 -> 180s to fit the inner loop.
Author
|
CI status: failing checks are pre-existing and unrelated to this change The failing checks (
Passing on this PR: lint, context, CodeQL. The change touches only the "Push to Github" step and the Verification plan (instead of the broken tests): a local concurrent-push race test of the new loop, and one real preview PR close against a multi-app repo pointed at this branch. Results will be added here. Context: DOI-3830, companion dw-github-workflows#337. |
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Jira: DOI-3830
Problem
Preview destroy/deploy jobs (one per app, across PRs) push to the same branch of the argocd deploy repo concurrently. The "Push to Github" step retried the whole reset/apply/push sequence with a fixed 10s wait and 4 attempts, so colliding jobs retried in lockstep and most lost every round. Example: dw-cc-posting-service PR 59, 13 destroy jobs, 5 pushes landed (run).
Change
git pull --rebase origin <ref>, jittered sleep (1-8s plus attempt number), push again, up to 6 tries; exit non-zero if all fail. App paths are disjoint so the rebase is clean.commit-timeout60s -> 180s to fit the inner loop.Follow-ups
dw-github-workflowspins this action by SHA (argocd-deploy/action.yml); bump the pin after this merges.Testing
Not run. Needs a multi-app PR closed against a workflow pinned to this branch.