Skip to content

DOI-3830: fix: rebase and retry push with jitter on non-fast-forward - #4

Open
yashkalwaniDW wants to merge 1 commit into
mainfrom
fix/push-retry-rebase-jitter
Open

yashkalwaniDW wants to merge 1 commit into
mainfrom
fix/push-retry-rebase-jitter

Conversation

@yashkalwaniDW

@yashkalwaniDW yashkalwaniDW commented Sep 30, 2026 •

Copy link
Copy Markdown

Jira: DOI-3830

Problem

Preview destroy/deploy jobs (one per app, across PRs) push to the same branch of the argocd deploy repo concurrently. The "Push to Github" step retried the whole reset/apply/push sequence with a fixed 10s wait and 4 attempts, so colliding jobs retried in lockstep and most lost every round. Example: dw-cc-posting-service PR 59, 13 destroy jobs, 5 pushes landed (run).

Change

  • On a rejected push: git pull --rebase origin <ref>, jittered sleep (1-8s plus attempt number), push again, up to 6 tries; exit non-zero if all fail. App paths are disjoint so the rebase is clean.
  • Default commit-timeout 60s -> 180s to fit the inner loop.

Follow-ups

  • dw-github-workflows pins this action by SHA (argocd-deploy/action.yml); bump the pin after this merges.
  • Companion: dw-github-workflows#337 serializes the matrix and fails the run when destroy fails.

Testing

Not run. Needs a multi-app PR closed against a workflow pinned to this branch.

Parallel preview destroy/deploy jobs race on the deploy repo's main branch;
a fixed 10s whole-step retry made colliding jobs retry in lockstep and lose.
Retry the push in place: pull --rebase, jittered sleep, up to 6 tries.
Raise default commit-timeout 60 -> 180s to fit the inner loop.
@yashkalwaniDW

yashkalwaniDW commented Sep 30, 2026 •

Copy link
Copy Markdown
Author

CI status: failing checks are pre-existing and unrelated to this change

The failing checks (test-destroy, test-helm-raw, test-helm-raw-default-kube-version, test-helmfile-raw, test-helmfile-raw-default-kube-version, and the ci wrapper that propagates them) fail because the repo is missing secrets, not because of this diff:

  • DOCKERHUB_USERNAME / DOCKERHUB_PASSWORD resolve to empty, so the localstack service-container credentials block is invalid and the jobs fail at setup: The template is not valid ... Unexpected value '' (e.g. run 36738429556).
  • One job fails in actions/create-github-app-token: The 'client-id' (or deprecated 'app-id') input must be set to a non-empty string.
  • The same tests fail identically on main (run 33877535712, 2026-09-04), so main was already red for these.

Passing on this PR: lint, context, CodeQL. The change touches only the "Push to Github" step and the commit-timeout default, which these tests fail before reaching.

Verification plan (instead of the broken tests): a local concurrent-push race test of the new loop, and one real preview PR close against a multi-app repo pointed at this branch. Results will be added here.

Context: DOI-3830, companion dw-github-workflows#337.

@yashkalwaniDW yashkalwaniDW changed the title DOI-3829: fix: rebase and retry push with jitter on non-fast-forward DOI-3830: fix: rebase and retry push with jitter on non-fast-forward Sep 30, 2026

This branch had an error being deployed

1 failed deployment
terratest — 5d73ec76 Deployed Sep 30, 2026 by github-actions[bot] via assert #7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant