Repository navigation
Docker proxies: release a streamed layer's staging space once it is stored - #829
Merged
Merged
Conversation
…tored Since #679 a proxy serves an uncached layer while it downloads it, and every response held the shared download, which held a reader of the staged file: its unlinked temp file and its whole-size reservation in server.max_staging_bytes. A client that read slowly, or stopped reading with its TCP window at zero, kept both after the layer was verified and stored, so a few such pulls of large layers could use up the staging budget, and every push and first proxy download answered 503 until those clients went away. The staged file is now the download task's. Responses take a handle on it for each read only, and the task drops the download's own handle when it ends, after reporting the layer stored (with its S3 key) or the download failed. A response still on the layer goes on from the stored blob, through the local blob cache like a pull that joins after the commit, or ends short when the download failed (a verified layer that could not be stored included, as in 2.1). So no response keeps staging space past the download, and a stalled one needs no idle timeout. store_fetched returns the key it stored the blob under. A unit test stalls a reader after verification and checks the reservation is back once the layer is stored; an integration test stalls a real pull of a 32 MiB layer, which fails on the code before with all 32 MiB still held, and then reads the rest of the layer from the store. #679's changelog fragment, docs/docker.md and docs/deployment.md say when the space is given back. Fixes #763 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012e6DJbjxnhsmaqc5gaMAcx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #763
Problem
Since #679 a proxy serves an uncached layer while it downloads it. Every response held the shared
Download, which held aStageReaderof the staged file. That reader kept the unlinked temp file open and kept the layer's whole-size reservation inserver.max_staging_bytes. A client that read slowly, or stopped reading with its TCP window at zero, kept both after the layer was verified and stored. A few such pulls of large layers could use up the 4 GiB default, and then every push and every first proxy download answered 503 until those clients disconnected.Change
crates/server/src/images/download.rs).Downloadkeeps itsStageReaderin aMutex<Option<_>>. A response takes a handle for one read at a time, and never across waits. When the task ends (Download::end), it first reports the layer stored (Progress::Stored, with its S3 key) or the download failed, then drops the download's handle. The reservation and the temp file go with it.BlobStore::openat the offset they had reached. That goes through the local blob cache, whichcommitwrites through, just as a pull that joins after the commit does. If the download failed, they end short. This includes a layer that was verified but could not be stored (object store down, proxy repointed), which a 2.1 pull would also have failed on.Progressis now an enum:Staging { written },Verified,Stored.store_fetchedreturns the S3 key it stored the blob under.Because no response keeps staging space past the download anymore, streamed responses don't need the idle or write timeout the issue offers as an alternative. A stalled client now holds what any 2.1 pull of a cached blob holds.
Changelog
The bug was never released: #679 is new in 2.2.0. So rather than adding a
763.fixed.mdthat 2.2.0's notes would list as a fix, this amendsdocs/changes/2.2.0/679.feature.mdto say when the space is given back. That follows #787, which amended703.fixed.md.docs/docker.mdand the Temp space paragraph ofdocs/deployment.mdsay the same.Tests
images::download):a_reader_that_stalls_after_verification_holds_no_staging_space_once_storeda_verified_download_that_is_not_stored_ends_its_readers_shorttests/registry.rs): newa_stalled_pull_of_a_downloading_layer_releases_its_staging_space_once_stored.33554432 bytes held.cargo test -p forge-server --test registry: 50 passedcargo test -p forge-server --lib: 284 passedcargo clippy -p forge-server --all-targets -- -D warnings,cargo fmt --all --check,cargo xtask changelog --check,cargo xtask docs --check: all cleanGenerated by Claude Code