Repository navigation
vmware: fetch each vCenter tag and category once per run - #621
Open
tyderian1978 wants to merge 1 commit into
Open
tyderian1978 wants to merge 1 commit into
tyderian1978 wants to merge 1 commit into
Conversation
get_vmware_object_tags() fetched every tag attached to an object, and its category when tag_name_include_category is set, from the vSphere tagging API, for every object. Most objects share a few hundred tags, so a run fetched the same tags thousands of times, one HTTPS request each. Tags and category names are now kept by id for the run, per source (tag ids are only unique within one vCenter). The tags returned are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Disclaimer: AI generated, based off the work we're doing with Netbox.
What happens
get_vmware_object_tags()asks the vSphere tagging API for every tag attached to an object, and for the tag's category whentag_name_include_categoryis set, for every cluster, host and VM. Most objects share a small set of tags, so a run fetches the same tags thousands of times, one HTTPS request each. On the two vCenters we measured, that was about 21,000 tag and category lookups per run, for about 700 distinct ones.Fix
Tags and category names are kept by id for the run, per source (tag ids are only unique within one vCenter). The tags returned are unchanged. The caches are created on first use rather than in
__init__, so a handler built without__init__, astests/test_vmware_tag_category.pydoes, works too.Tests
tests/test_vmware_tag_cache.py, with a fake tagging API that counts lookups: five objects sharing two tags cost two tag lookups (and two category lookups withtag_name_include_category) and get the same tags as before; two sources keep separate caches. The lookup counts fail ondevelopmentwithout the change. Full suite: 610 passed, 8 skipped.Together with the IP address index (separate PR), a dry run against our production NetBox and vCenters went from 38 min 45 s to 27 min 42 s, with identical planned changes.