Batch GetIamPolicy calls for all google_*_iam_* resources - #18892
Open
joscha-alisch wants to merge 2 commits into
Open
Batch GetIamPolicy calls for all google_*_iam_* resources#18892joscha-alisch wants to merge 2 commits into
joscha-alisch wants to merge 2 commits into
Conversation
The code here used readIamPolicyWithRetry which is (1) doing retries within retries (2) trying to aquire a lock that is already held by this function. Additionally, the error check was inverted and would actually cause a panic if triggered.
modular-magician
added
the
awaiting-approval
|
Googlers: For automatic test runs see go/terraform-auto-test-runs. @roaks3, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look. You can help make sure that review is quick by doing a self-review and by running impacted tests locally. |
Author
|
This PR also relates to #18890. Together they speed up our organizations GCP project management pipelines by a factor of 10 through this efficient batching and also completely eliminate any rate limiting on read requests per minute. |
joscha-alisch
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Currently, only the write path for IAM policies is batched. This makes the write path quick, but the refresh / read path for all
google_*_iam_*resources makes the same redundant API requests for each of those.In case of non-authoritative management of IAM policies via
google_project_iam_memberthis can result in hundreds of unnecessary API calls.This PR introduces request batching for the read path of all
google_*_iam_*resources, similiar to how it's already being done forgoogle_project_service.In my test project with 175 individual
google_project_iam_member, the number of IAM policy API calls goes from 175 down to 1, reducing the time to refresh/plan from 73 seconds to 14.Contributes to
While introducing this, I also noticed an inverted error condition causing a panic and also another deadlock in one of the call sites. Given I had to touch that call site anyway, I also fixed those two issues (guided by tests showing both deadlock and panic). Let me know if you'd rather want this as a separate PR.
Release Note Template for Downstream PRs (will be copied)
See Write release notes for guidance.