gvfs-helper: parallelize POST requests - #980
Open
Derrick Stolee (derrickstolee) wants to merge 3 commits into
Open
gvfs-helper: parallelize POST requests#980Derrick Stolee (derrickstolee) wants to merge 3 commits into
Derrick Stolee (derrickstolee) wants to merge 3 commits into
Conversation
tyrielv
reviewed
Aug 24, 2026
| installation of multiple prefetch packs. Values less than `1` are | ||
| treated as `1`. | ||
|
|
||
| gvfs.postThreads:: |
There was a problem hiding this comment.
Does this feature ever tend to bottleneck on CPU (ie when indexing pack files) or only on network/remote?
I'm wondering if there'd be any benefit or downside to supporting "values less than 1 are treated as NUMBER_OF_PROCESSORS" like checkout.workers does.
There was a problem hiding this comment.
Also, GVFS uses Environment.ProcessorCount as the default parallelism value for its analogous workflow (gvfs prefetch --files or --folders)
There was a problem hiding this comment.
We could consider the "values less than one" option. I worry that the network will saturate at a lower parallelism than the CPU doing pack-indexing.
Large object requests currently fetch batches sequentially, leaving network capacity unused when server latency dominates the transfer. Allow callers to select multiple POST workers through a new gvfs.postThreads configuration value. Default the value to one so existing users retain the sequential path, and treat values below one as one. Document how higher values enable concurrent HTTP requests that stream into separate index-pack processes. Signed-off-by: Derrick Stolee <stolee@gmail.com> Signed-off-by: Neil Kainga <t-neilkainga@microsoft.com> Co-authored-by: Neil Kainga <t-neilkainga@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fetching a large set of missing objects through gvfs-helper performs each HTTP POST and index-pack operation sequentially. This leaves the client waiting on individual network transfers even when the server and local machine can support concurrent work. Use a mutex-protected queue to distribute full object batches across worker threads. Each worker owns an HTTP-subsystem-configured curl handle and streams each response into a fresh index-pack process, including cache-server fallback attempts. Initialize curl handles before starting threads to retain configured proxy, TLS, user-agent, and timeout behavior. Serialize child startup while marking pipe descriptors close-on-exec so concurrent index-pack children cannot keep sibling pipes open. Keep OID formatting and result collection thread-local, propagate transfer and pack installation failures, and avoid touching temporary packs owned by other processes. Partition work into batches containing at least two objects because a single non-commit object can be returned loose instead of as a pack. Continue using the existing sequential path when threading is unavailable or the configured block size cannot satisfy that constraint. Signed-off-by: Derrick Stolee <stolee@gmail.com> Signed-off-by: Neil Kainga <t-neilkainga@microsoft.com> Co-authored-by: Neil Kainga <t-neilkainga@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Exercise gvfs-helper POST requests with both one and four configured workers so the sequential and parallel paths must fetch identical object sets. Cover multiple batches, a final single-OID remainder, and duplicate requests while checking both installed objects and packfile counts. Register the new script in the Meson integration test list so Meson builds run the same coverage as the default test harness. Signed-off-by: Derrick Stolee <stolee@gmail.com> Signed-off-by: Neil Kainga <t-neilkainga@microsoft.com> Co-authored-by: Neil Kainga <t-neilkainga@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Derrick Stolee (derrickstolee)
force-pushed
the
parallel-post-threads
branch
from
August 24, 2026 17:16
f05b1d5 to
6b263d8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Improve full-clone performance by allowing
gvfs-helper postto download object batches concurrently. The newgvfs.postThreadsconfiguration defaults to 1, preserving the existing sequential behavior unless explicitly enabled.The parallel implementation uses independent curl handles and
index-pack --stdinchildren with a mutex-protected work queue. It preserves Git HTTP configuration, supports threadless builds through the sequential fallback, and avoids sending singleton loose-object responses toindex-pack.Dogfood testing exposed a deadlock caused by sibling
index-packprocesses inheriting one another's pipe descriptors. The series serializes child creation and marks parent pipe descriptors close-on-exec. It also removes the sharedoid_to_hex()buffer race, protects SIGPIPE handling, isolates retry child lifecycles, closes error-path descriptors, and avoids deleting temporary packs belonging to concurrent processes using the shared ODB.Neil Kainga diagnosed and tested the pipe inheritance fix on a 1JS full clone with
gvfs.postThreads=8and is credited in the fix commits.The focused
t5798-gvfs-helper-post-threads.shcoverage exercises sequential and parallel requests, small batches, singleton remainders, duplicate downloads, and object integrity.