kubernetes: jitter transient dial retries - #4039
Draft
crazy-max wants to merge 2 commits into
Draft
Conversation
…n lockstep calculateBackoff was a pure function of the attempt number, so every builder retrying the same condition waited exactly the same durations. That matters here because CSR approval lagging node readiness is a cluster-wide event: concurrent builds scheduled onto newly-ready nodes hit the transient TLS error at the same moment, then retry in unison against an API server already working through the approval backlog. Add jitter drawn from [d, 2d], capped by maxDelay, where d is the exponential value for the attempt. The exponential value is the floor rather than the midpoint, so a retry is never issued sooner than the schedule would have on its own. Centring it would let the first retry fire at baseDelay/2, which undercuts a configured minimum at exactly the wrong moment. With maxRetries=5 and baseDelay=500ms the delays used are 500ms through 4s, so the 10s cap is never reached in practice. Marked the math/rand call with the same #nosec pattern podchooser uses. Signed-off-by: Santhosh Kumar Somarapu <somarapu.santhosh91@gmail.com>
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
carry and closes #3995
Kubernetes dial retries now add jitter to the exponential backoff used for transient connection errors. This keeps concurrent builders from retrying in lockstep when node readiness gets ahead of CSR approval and many builds hit the same temporary TLS failure at once.
The jitter is additive, so retries never happen sooner than the existing exponential schedule would allow, and the delay remains capped by the configured maximum.