Use ThreadPool Executor to fetch packages concurrently#143
Open
atom-andrew wants to merge 1 commit intoFHPythonUtils:masterfrom
Open
Use ThreadPool Executor to fetch packages concurrently#143atom-andrew wants to merge 1 commit intoFHPythonUtils:masterfrom
atom-andrew wants to merge 1 commit intoFHPythonUtils:masterfrom
Conversation
atom-andrew
changed the title
atom-andrew
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose of This Pull Request
Speed up checks using a ThreadPool to do work concurrently. This particularly important for use a pre-commit hook, where speed is critical.
Please check the relevant option by placing an "X" inside the brackets:
Overview of Changes
Fetch data concurrently.
Related Issue
Reviewer Focus
Additional Notes
Possibly this will cause us to hit rate limits faster. The python gil presumably would prevent this from overloading all the cores on the server so it shouldn't have a system usability impact.
Our pre-commit pipes the output of
uv exportto this tool. It skips internal packages (they would fail anyway because they all go through a gitlab pypi repo that doesn't support the legacy pypi json api, but that's another story), so there is no need to handle indirection. This would be very fast if requests could be made concurrently, but currently they are not.