Skip to content

feat: serve world downloads as pre-signed S3 links - #521

Merged
thomasmny merged 2 commits into
masterfrom
feat/world-downloads-s3
Aug 1, 2026
Merged

feat: serve world downloads as pre-signed S3 links#521
thomasmny merged 2 commits into
masterfrom
feat/world-downloads-s3

Conversation

@thomasmny

@thomasmny thomasmny commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Follow-up to #520.

Pre-signed S3 delivery

Adds world.download.storage, choosing between the built-in HTTP server (local, the default) and pre-signed links from the configured bucket (s3). The S3 path needs no open port and imposes no size limit: archives upload in parts, streamed off disk, so neither the heap nor the 5 GB ceiling on a single PUT bounds a world.

Where the archive goes is now a DownloadDelivery, since the mode was otherwise branched on in four places. WorldDownloadService keeps the export orchestration; the deliveries own their own lifecycle, budget and cleanup.

Progress carries a phase, so the bar tracks the upload instead of sitting full while it runs.

Storage credentials move to the root

The S3 and SFTP credentials sat under world.backup.storage only because backups needed them first. Downloads read the same bucket, and a second feature reading a key path named after the first is a lie that gets more expensive to keep.

They move to a root storage section, and each feature names the backend it wants. Where files land within a backend belongs to the feature (world.backup.path), so two features sharing a bucket do not share a prefix. Downloads no longer require backups to be on S3. Existing configs migrate on first start (MigrationV4ToV5).

Fixes found while building this

  • Link expiry was stamped when the command was typed, so packing and uploading ate into it — a slow export produced a link that was already dead. publish now takes a lifetime, and the delivery starts the clock when it hands the link out.
  • The storage budget was sampled but never reserved, so concurrent exports were each promised the whole remainder.
  • stop() wiped the staging folder under in-flight exports, which then published to a closed delivery.
  • The expiry purge ran blocking S3 DELETEs on the Bukkit main thread.
  • Exports ran on the shared background pool with no cap, starving backups.
  • /worlds setItem wiped a configured icon texture when the held head's profile carried none.

Notes

  • behind-proxy (default off) keys pinning and rate limiting on the last X-Forwarded-For hop. Off by default because trusting the header on a directly reachable port would let anyone claim any address.
  • A pre-signed URL is a bearer token: unlike a local link it cannot be pinned to one client or rate limited, and it is replayable until it expires. Documented; expiration-minutes is the lever.
  • Two saveAndFlush main-thread stalls were left alone deliberately. Paper exposes no async flush and save(boolean) is main-thread-only; dropping the flush would reintroduce the half-written region files fixed in feat: download worlds as single-player saves #520.

Testing

Unit tests only — the pre-signing is pinned against AWS's published test vector, and the multipart split is pinned for part count and minimum part size. The S3 path has never run against a live bucket; worth a MinIO smoke test before release.

Adds world.download.storage, choosing between the built-in HTTP server
(local, the default) and pre-signed links from the bucket the backups
already use (s3). The S3 path needs no open port and imposes no size
limit: archives upload in parts, streamed off disk, so neither the heap
nor the five-gigabyte ceiling on a single PUT bounds a world.

Where the archive goes is now a DownloadDelivery, since the mode was
otherwise branched on in four places. WorldDownloadService keeps the
export orchestration; the deliveries own their own lifecycle, budget and
cleanup.

Progress carries a phase, so the bar tracks the upload instead of sitting
full while it runs.

Deliveries reserve budget rather than sample it, links start their life
when handed out rather than when the command was typed, exports are
bounded by the staging disk's free space, and the purge runs off the main
thread. Exports get their own thread so they cannot starve backups.

behind-proxy keys pinning and rate limiting on the last X-Forwarded-For
hop; off by default, since trusting the header on a directly reachable
port would let anyone claim any address.

Also stops /worlds setItem wiping a configured icon texture when the held
head's profile carries none.
The S3 and SFTP credentials sat under world.backup.storage only because
backups needed them first. World downloads read the same bucket, and a
second feature reading a key path named after the first is a lie that
only gets more expensive to keep.

They move to a root storage section, and each feature names the backend
it wants: world.backup.storage and world.download.storage are now plain
type names. Where files land within a backend belongs to the feature, not
the credentials, so backups keep their prefix in world.backup.path and
downloads keep theirs.

Downloads no longer require backups to be on S3, since neither feature
owns the bucket any more.

A backend is only selected once its credentials are present, so a
half-configured one falls back to local with the missing key named,
rather than being handed out and failing at connection time. Existing
configs migrate on first start.
@thomasmny
thomasmny merged commit fb8be25 into master Aug 1, 2026
2 checks passed
@thomasmny
thomasmny deleted the feat/world-downloads-s3 branch August 1, 2026 13:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant