Skip to content

Docker proxies: release a streamed layer's staging space once it is stored - #829

Merged
schmitch merged 1 commit into
mainfrom
claude/gifted-edison-lanjeo
Nov 3, 2026
Merged

schmitch merged 1 commit into
mainfrom
claude/gifted-edison-lanjeo

Conversation

@schmitch

@schmitch schmitch commented Nov 3, 2026

Copy link
Copy Markdown
Member

Fixes #763

Problem

Since #679 a proxy serves an uncached layer while it downloads it. Every response held the shared Download, which held a StageReader of the staged file. That reader kept the unlinked temp file open and kept the layer's whole-size reservation in server.max_staging_bytes. A client that read slowly, or stopped reading with its TCP window at zero, kept both after the layer was verified and stored. A few such pulls of large layers could use up the 4 GiB default, and then every push and every first proxy download answered 503 until those clients disconnected.

Change

  • The download task owns the staged file (crates/server/src/images/download.rs). Download keeps its StageReader in a Mutex<Option<_>>. A response takes a handle for one read at a time, and never across waits. When the task ends (Download::end), it first reports the layer stored (Progress::Stored, with its S3 key) or the download failed, then drops the download's handle. The reservation and the temp file go with it.
  • Responses still on the layer go on from the stored blob. They read it through BlobStore::open at the offset they had reached. That goes through the local blob cache, which commit writes through, just as a pull that joins after the commit does. If the download failed, they end short. This includes a layer that was verified but could not be stored (object store down, proxy repointed), which a 2.1 pull would also have failed on.
  • Progress is now an enum: Staging { written }, Verified, Stored.
  • store_fetched returns the S3 key it stored the blob under.

Because no response keeps staging space past the download anymore, streamed responses don't need the idle or write timeout the issue offers as an alternative. A stalled client now holds what any 2.1 pull of a cached blob holds.

Changelog

The bug was never released: #679 is new in 2.2.0. So rather than adding a 763.fixed.md that 2.2.0's notes would list as a fix, this amends docs/changes/2.2.0/679.feature.md to say when the space is given back. That follows #787, which amended 703.fixed.md. docs/docker.md and the Temp space paragraph of docs/deployment.md say the same.

Tests

  • Unit (images::download):
    • New: a_reader_that_stalls_after_verification_holds_no_staging_space_once_stored
    • New: a_verified_download_that_is_not_stored_ends_its_readers_short
    • The existing two are adapted to the new progress states.
  • Integration (tests/registry.rs): new a_stalled_pull_of_a_downloading_layer_releases_its_staging_space_once_stored.
    • A real client reads 1 MiB of a 32 MiB layer and stops. That is more than the sockets can buffer, so the server's response can't finish. The test then waits for the layer to be linked, checks that staging usage drops to 0, and checks that the client's eventual read of the rest returns the upstream's layer.
    • On the code before this change it fails with 33554432 bytes held.
  • Run locally:
    • cargo test -p forge-server --test registry: 50 passed
    • cargo test -p forge-server --lib: 284 passed
    • cargo clippy -p forge-server --all-targets -- -D warnings, cargo fmt --all --check, cargo xtask changelog --check, cargo xtask docs --check: all clean

Generated by Claude Code

…tored

Since #679 a proxy serves an uncached layer while it downloads it, and
every response held the shared download, which held a reader of the
staged file: its unlinked temp file and its whole-size reservation in
server.max_staging_bytes. A client that read slowly, or stopped reading
with its TCP window at zero, kept both after the layer was verified and
stored, so a few such pulls of large layers could use up the staging
budget, and every push and first proxy download answered 503 until those
clients went away.

The staged file is now the download task's. Responses take a handle on
it for each read only, and the task drops the download's own handle when
it ends, after reporting the layer stored (with its S3 key) or the
download failed. A response still on the layer goes on from the stored
blob, through the local blob cache like a pull that joins after the
commit, or ends short when the download failed (a verified layer that
could not be stored included, as in 2.1). So no response keeps staging
space past the download, and a stalled one needs no idle timeout.

store_fetched returns the key it stored the blob under. A unit test
stalls a reader after verification and checks the reservation is back
once the layer is stored; an integration test stalls a real pull of a
32 MiB layer, which fails on the code before with all 32 MiB still held,
and then reads the rest of the layer from the store. #679's changelog
fragment, docs/docker.md and docs/deployment.md say when the space is
given back.

Fixes #763

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012e6DJbjxnhsmaqc5gaMAcx
@schmitch
schmitch added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit 953d11f Nov 3, 2026
19 checks passed
@schmitch
schmitch deleted the claude/gifted-edison-lanjeo branch November 3, 2026 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Docker proxies: a streamed layer keeps its staging space and temp file until the slowest client has read it

2 participants