feat: serve world downloads as pre-signed S3 links - #521
Merged
Conversation
Adds world.download.storage, choosing between the built-in HTTP server (local, the default) and pre-signed links from the bucket the backups already use (s3). The S3 path needs no open port and imposes no size limit: archives upload in parts, streamed off disk, so neither the heap nor the five-gigabyte ceiling on a single PUT bounds a world. Where the archive goes is now a DownloadDelivery, since the mode was otherwise branched on in four places. WorldDownloadService keeps the export orchestration; the deliveries own their own lifecycle, budget and cleanup. Progress carries a phase, so the bar tracks the upload instead of sitting full while it runs. Deliveries reserve budget rather than sample it, links start their life when handed out rather than when the command was typed, exports are bounded by the staging disk's free space, and the purge runs off the main thread. Exports get their own thread so they cannot starve backups. behind-proxy keys pinning and rate limiting on the last X-Forwarded-For hop; off by default, since trusting the header on a directly reachable port would let anyone claim any address. Also stops /worlds setItem wiping a configured icon texture when the held head's profile carries none.
The S3 and SFTP credentials sat under world.backup.storage only because backups needed them first. World downloads read the same bucket, and a second feature reading a key path named after the first is a lie that only gets more expensive to keep. They move to a root storage section, and each feature names the backend it wants: world.backup.storage and world.download.storage are now plain type names. Where files land within a backend belongs to the feature, not the credentials, so backups keep their prefix in world.backup.path and downloads keep theirs. Downloads no longer require backups to be on S3, since neither feature owns the bucket any more. A backend is only selected once its credentials are present, so a half-configured one falls back to local with the missing key named, rather than being handed out and failing at connection time. Existing configs migrate on first start.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #520.
Pre-signed S3 delivery
Adds
world.download.storage, choosing between the built-in HTTP server (local, the default) and pre-signed links from the configured bucket (s3). The S3 path needs no open port and imposes no size limit: archives upload in parts, streamed off disk, so neither the heap nor the 5 GB ceiling on a singlePUTbounds a world.Where the archive goes is now a
DownloadDelivery, since the mode was otherwise branched on in four places.WorldDownloadServicekeeps the export orchestration; the deliveries own their own lifecycle, budget and cleanup.Progress carries a phase, so the bar tracks the upload instead of sitting full while it runs.
Storage credentials move to the root
The S3 and SFTP credentials sat under
world.backup.storageonly because backups needed them first. Downloads read the same bucket, and a second feature reading a key path named after the first is a lie that gets more expensive to keep.They move to a root
storagesection, and each feature names the backend it wants. Where files land within a backend belongs to the feature (world.backup.path), so two features sharing a bucket do not share a prefix. Downloads no longer require backups to be on S3. Existing configs migrate on first start (MigrationV4ToV5).Fixes found while building this
publishnow takes a lifetime, and the delivery starts the clock when it hands the link out.stop()wiped the staging folder under in-flight exports, which then published to a closed delivery.DELETEs on the Bukkit main thread./worlds setItemwiped a configured icon texture when the held head's profile carried none.Notes
behind-proxy(default off) keys pinning and rate limiting on the lastX-Forwarded-Forhop. Off by default because trusting the header on a directly reachable port would let anyone claim any address.expiration-minutesis the lever.saveAndFlushmain-thread stalls were left alone deliberately. Paper exposes no async flush andsave(boolean)is main-thread-only; dropping the flush would reintroduce the half-written region files fixed in feat: download worlds as single-player saves #520.Testing
Unit tests only — the pre-signing is pinned against AWS's published test vector, and the multipart split is pinned for part count and minimum part size. The S3 path has never run against a live bucket; worth a MinIO smoke test before release.