fix: avoid buffering large client uploads into memory - #17856
Open
paulpopus wants to merge 4 commits into
Open
Conversation
Contributor
📦 esbuild Bundle Analysis for payloadThis analysis was generated by esbuild-bundle-analyzer. 🤖
Largest pathsThese visualization shows top 20 largest paths in the bundle.Meta file: packages/next/meta_index.json, Out file: esbuild/index.js
Meta file: packages/payload/meta_index.json, Out file: esbuild/index.js
Meta file: packages/payload/meta_shared.json, Out file: esbuild/exports/shared.js
Meta file: packages/richtext-lexical/meta_client.json, Out file: esbuild/exports/client_optimized/index.js
Meta file: packages/ui/meta_client.json, Out file: esbuild/exports/client_optimized/index.js
Meta file: packages/ui/meta_shared.json, Out file: esbuild/exports/shared_optimized/index.js
DetailsNext to the size is how much the size has increased or decreased compared with the base branch of this PR.
|
paulpopus
changed the title
paulpopus
marked this pull request as ready for review
August 20, 2026 12:51
nathanlentz
requested changes
Aug 20, 2026
| throw new APIError('uploadConfig.handlers is not present for ' + collectionSlug) | ||
| } | ||
|
|
||
| const contentRequirement = getFileContentRequirement({ mimeType: file.mimeType, uploadConfig }) |
Collaborator
There was a problem hiding this comment.
I might be missing where this gets accounted for, but does the content requirement here know about request-level uploadEdits? I am wondering about a scenario where a collection which should enter the header only path is followed up by a request that includes a crop. Could we end up passing only the header bytes into cropImage? Would crop/resize edits need to promote this to a full fetch?
I'm worried about a timing problem here where the decision for how much content to fetch only takes into account the collection configuration and MIME type which results in Sharp later processing truncated bytes.
| bufferToSave = fileBuffer.data | ||
| } else if (file.tempFilePath) { | ||
| bufferToSave = await fs.readFile(file.tempFilePath) | ||
| bufferToSave = skipTempFileBuffer ? Buffer.alloc(0) : await fs.readFile(file.tempFilePath) |
Collaborator
There was a problem hiding this comment.
Our storage adapters set disableLocalStorage: true, so I think their upload paths avoid this read. Is this branch mainly supporting custom upload handlers that keep local storage enabled?
If so, it looks like we stream the upload to a temp file and then load the whole thing back into memory here before saving it locally which walks us back into the same issue we are trying to avoid I think.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Client uploads send a file straight from the browser to storage, bypassing the Payload server. The server still reads the file afterward, to save it locally or to read metadata such as width and height. It previously always downloaded the whole file into one in-memory buffer for that step, with no size limit. This is a real risk for the multi-gigabyte files that client uploads exist to support. The server now reads only the bytes each save operation actually needs, and streams any full-file read straight to disk instead of memory.
Why
Azure's client-upload support removes the old five-gigabyte upload ceiling, so a single upload can now be far larger than the server's available memory. Re-downloading that whole file into a buffer, just to check its size or resave it unchanged, does not scale with that change.
How
disableLocalStorageoption. Collections that leave it unset, the most common case, could take the reduced-fetch path meant only for disabled local storage. That risked saving a truncated or empty file.Testing
Added tests for the new content-requirement decision and the streaming behavior. Added integration tests that complete a real client upload against the S3 and Azure storage adapters for a file needing only its dimensions, the exact path the request-clone fix above corrects. No integration test exercised that path before, which is how the bug shipped unnoticed.
Alternatives considered
Tried splitting a full-file read into repeated smaller range requests, on the idea that shorter-lived reads would free memory sooner. Measured memory during the change and found no improvement, so it was not kept.
Related work
Related to #17318 and #17319, which enabled Azure client uploads larger than 5 GB.