fix: avoid buffering large client uploads into memory

[paulpopus]

Backport of #17856

Summary

Client uploads send a file straight from the browser to storage, bypassing the Payload server. The server still reads the file afterward, to save it locally or to read metadata such as width and height. It previously always downloaded the whole file into one in-memory buffer for that step, with no size limit. This is a real risk for the multi-gigabyte files that client uploads exist to support. The server now reads only the bytes each save operation actually needs, and streams any full-file read straight to disk instead of memory.

Why

Azure's client-upload support removes the old five-gigabyte upload ceiling, so a single upload can now be far larger than the server's available memory. Re-downloading that whole file into a buffer, just to check its size or resave it unchanged, does not scale with that change.

How

• A new content-requirement check decides how much of a file the server needs before it fetches anything: nothing, when the client-reported metadata is enough; a small byte range, when only the image dimensions are needed; or the full file, when local storage needs the real bytes, the file will be resized or reformatted, or a configured mime-type allow list needs to inspect the content.
• A full-file read now streams straight to a temporary file on disk, replacing the single in-memory buffer.
• An unmodified temporary file that only needs saving to disk-disabled storage is left untouched, instead of being read into memory and written back unchanged.
• Fixed a check for the disableLocalStorage option. Collections that leave it unset, the most common case, could take the reduced-fetch path meant only for disabled local storage. That risked saving a truncated or empty file.
• Fixed the byte-range probe's request clone. It broke native request properties that real storage handlers read, such as an abort signal. Every adapter in this repository that reads that property, including S3 and Azure, crashed against a client upload needing only its image dimensions.
• Corrected the list of image formats treated as animated. It skipped multi-page TIFF files and wrongly flagged every AVIF file as animated, even single-frame ones.

Testing

Added tests for the new content-requirement decision and the streaming behavior. Added integration tests that complete a real client upload against the S3 and Azure storage adapters for a file needing only its dimensions, the exact path the request-clone fix above corrects. No integration test exercised that path before, which is how the bug shipped unnoticed.

Alternatives considered

Tried splitting a full-file read into repeated smaller range requests, on the idea that shorter-lived reads would free memory sooner. Measured memory during the change and found no improvement, so it was not kept.

Related work

Related to #17318 and #17319, which enabled Azure client uploads larger than 5 GB.

[github-actions]

📦 esbuild Bundle Analysis for payload

This analysis was generated by esbuild-bundle-analyzer. 🤖

Meta File	Out File	Size (raw)	Note
packages/next/meta_index.json	esbuild/index.js	985.32 KB	🆕 Added
packages/payload/meta_index.json	esbuild/index.js	1.40 MB	🆕 Added
packages/payload/meta_shared.json	esbuild/exports/shared.js	192.84 KB	🆕 Added
packages/richtext-lexical/meta_client.json	esbuild/exports/client_optimized/index.js	290.56 KB	🆕 Added
packages/ui/meta_client.json	esbuild/exports/client_optimized/index.js	1.18 MB	🆕 Added
packages/ui/meta_shared.json	esbuild/exports/shared_optimized/index.js	16.32 KB	🆕 Added

Largest paths

These visualization shows top 20 largest paths in the bundle.

Meta file: packages/next/meta_index.json, Out file: esbuild/index.js

Path	Size
../../node_modules	${{\color{Goldenrod}{ ████████████████████▌ }}}$ 82.4%, 807.66 KB
dist/views/Version	${{\color{Goldenrod}{ █▎ }}}$ 5.3%, 51.49 KB
dist/views/Dashboard	${{\color{Goldenrod}{ ▌ }}}$ 2.2%, 21.37 KB
dist/views/Document	${{\color{Goldenrod}{ ▍ }}}$ 1.7%, 16.64 KB
dist/views/List	${{\color{Goldenrod}{ ▎ }}}$ 1.2%, 11.40 KB
dist/views/Root	${{\color{Goldenrod}{ ▎ }}}$ 1.0%, 9.90 KB
dist/views/Versions	${{\color{Goldenrod}{ ▏ }}}$ 0.6%, 6.17 KB
dist/views/API	${{\color{Goldenrod}{ ▏ }}}$ 0.6%, 6.13 KB
dist/elements/Nav	${{\color{Goldenrod}{ ▏ }}}$ 0.6%, 5.96 KB
dist/views/Account	${{\color{Goldenrod}{ ▏ }}}$ 0.6%, 5.55 KB
dist/elements/DocumentHeader	${{\color{Goldenrod}{ ▏ }}}$ 0.5%, 4.81 KB
dist/views/Login	${{\color{Goldenrod}{ }}}$ 0.4%, 4.40 KB
dist/layouts/Root	${{\color{Goldenrod}{ }}}$ 0.3%, 3.20 KB
dist/views/ForgotPassword	${{\color{Goldenrod}{ }}}$ 0.3%, 3.13 KB
dist/views/CreateFirstUser	${{\color{Goldenrod}{ }}}$ 0.3%, 2.81 KB
dist/templates/Default	${{\color{Goldenrod}{ }}}$ 0.3%, 2.64 KB
dist/views/BrowseByFolder	${{\color{Goldenrod}{ }}}$ 0.3%, 2.61 KB
dist/views/CollectionFolders	${{\color{Goldenrod}{ }}}$ 0.2%, 2.44 KB
dist/views/ResetPassword	${{\color{Goldenrod}{ }}}$ 0.2%, 2.40 KB
dist/views/Logout	${{\color{Goldenrod}{ }}}$ 0.2%, 1.94 KB
(other)	${{\color{Goldenrod}{ ████▍ }}}$ 17.6%, 172.98 KB

Meta file: packages/payload/meta_index.json, Out file: esbuild/index.js

Path	Size
../../node_modules	${{\color{Goldenrod}{ █████████████████ }}}$ 68.1%, 946.16 KB
dist/fields/hooks	${{\color{Goldenrod}{ ▊ }}}$ 3.2%, 44.40 KB
dist/collections/operations	${{\color{Goldenrod}{ ▊ }}}$ 3.1%, 43.44 KB
dist/versions/migrations	${{\color{Goldenrod}{ ▎ }}}$ 1.3%, 18.50 KB
dist/auth/operations	${{\color{Goldenrod}{ ▎ }}}$ 1.1%, 15.63 KB
dist/fields/config	${{\color{Goldenrod}{ ▎ }}}$ 1.0%, 14.16 KB
dist/globals/operations	${{\color{Goldenrod}{ ▎ }}}$ 1.0%, 13.41 KB
dist/queues/operations	${{\color{Goldenrod}{ ▏ }}}$ 0.9%, 13.15 KB
dist/utilities/configToJSONSchema.js	${{\color{Goldenrod}{ ▏ }}}$ 0.9%, 13.13 KB
dist/fields/validations.js	${{\color{Goldenrod}{ ▏ }}}$ 0.8%, 10.54 KB
dist/bin/generateImportMap	${{\color{Goldenrod}{ ▏ }}}$ 0.7%, 9.08 KB
dist/collections/config	${{\color{Goldenrod}{ ▏ }}}$ 0.6%, 8.91 KB
dist/config/orderable	${{\color{Goldenrod}{ ▏ }}}$ 0.6%, 8.15 KB
dist/uploads/fetchAPI-multipart	${{\color{Goldenrod}{ ▏ }}}$ 0.6%, 7.83 KB
dist/index.js	${{\color{Goldenrod}{ ▏ }}}$ 0.5%, 7.59 KB
dist/database/migrations	${{\color{Goldenrod}{ ▏ }}}$ 0.5%, 7.54 KB
dist/config/sanitize.js	${{\color{Goldenrod}{ ▏ }}}$ 0.5%, 6.26 KB
dist/collections/endpoints	${{\color{Goldenrod}{ }}}$ 0.4%, 6.23 KB
dist/auth/strategies	${{\color{Goldenrod}{ }}}$ 0.4%, 5.50 KB
dist/queues/config	${{\color{Goldenrod}{ }}}$ 0.4%, 5.31 KB
(other)	${{\color{Goldenrod}{ ███████▉ }}}$ 31.9%, 443.81 KB

Meta file: packages/payload/meta_shared.json, Out file: esbuild/exports/shared.js

Path	Size
../../node_modules	${{\color{Goldenrod}{ ███████████████████▊ }}}$ 79.4%, 150.25 KB
dist/fields/validations.js	${{\color{Goldenrod}{ █▍ }}}$ 5.6%, 10.54 KB
dist/config/orderable	${{\color{Goldenrod}{ ▍ }}}$ 1.7%, 3.13 KB
dist/fields/baseFields	${{\color{Goldenrod}{ ▍ }}}$ 1.5%, 2.79 KB
dist/utilities/deepCopyObject.js	${{\color{Goldenrod}{ ▎ }}}$ 1.3%, 2.54 KB
dist/auth/cookies.js	${{\color{Goldenrod}{ ▏ }}}$ 0.8%, 1.55 KB
dist/utilities/flattenTopLevelFields.js	${{\color{Goldenrod}{ ▏ }}}$ 0.7%, 1.42 KB
dist/fields/config	${{\color{Goldenrod}{ ▏ }}}$ 0.7%, 1.28 KB
dist/utilities/getVersionsConfig.js	${{\color{Goldenrod}{ ▏ }}}$ 0.5%, 1.04 KB
dist/utilities/flattenAllFields.js	${{\color{Goldenrod}{ ▏ }}}$ 0.5%, 943 B
dist/folders/utils	${{\color{Goldenrod}{ ▏ }}}$ 0.5%, 916 B
dist/utilities/unflatten.js	${{\color{Goldenrod}{ }}}$ 0.4%, 779 B
dist/utilities/sanitizeUserDataForEmail.js	${{\color{Goldenrod}{ }}}$ 0.4%, 713 B
dist/utilities/getFieldPermissions.js	${{\color{Goldenrod}{ }}}$ 0.3%, 651 B
dist/utilities/getSafeRedirect.js	${{\color{Goldenrod}{ }}}$ 0.3%, 632 B
dist/collections/config	${{\color{Goldenrod}{ }}}$ 0.3%, 570 B
dist/bin/generateImportMap	${{\color{Goldenrod}{ }}}$ 0.3%, 561 B
dist/auth/sessions.js	${{\color{Goldenrod}{ }}}$ 0.3%, 525 B
dist/fields/getFieldPaths.js	${{\color{Goldenrod}{ }}}$ 0.3%, 485 B
dist/utilities/deepMerge.js	${{\color{Goldenrod}{ }}}$ 0.2%, 413 B
(other)	${{\color{Goldenrod}{ █████▏ }}}$ 20.6%, 38.93 KB

Meta file: packages/richtext-lexical/meta_client.json, Out file: esbuild/exports/client_optimized/index.js

Path	Size
dist/features/blocks	${{\color{Goldenrod}{ ███▏ }}}$ 12.7%, 36.44 KB
dist/lexical/plugins	${{\color{Goldenrod}{ ██▊ }}}$ 11.4%, 32.65 KB
dist/lexical/ui	${{\color{Goldenrod}{ ██▏ }}}$ 8.5%, 24.36 KB
dist/features/experimental_table	${{\color{Goldenrod}{ ██ }}}$ 8.3%, 23.70 KB
dist/packages/@lexical	${{\color{Goldenrod}{ █▋ }}}$ 6.9%, 19.80 KB
dist/features/link	${{\color{Goldenrod}{ █▋ }}}$ 6.5%, 18.63 KB
dist/features/toolbars	${{\color{Goldenrod}{ █▍ }}}$ 5.6%, 16.08 KB
dist/features/upload	${{\color{Goldenrod}{ █▏ }}}$ 4.8%, 13.77 KB
dist/features/textState	${{\color{Goldenrod}{ ▉ }}}$ 3.9%, 11.08 KB
dist/features/relationship	${{\color{Goldenrod}{ ▊ }}}$ 3.1%, 9.03 KB
dist/lexical/utils	${{\color{Goldenrod}{ ▊ }}}$ 3.1%, 8.79 KB
dist/features/converters	${{\color{Goldenrod}{ ▋ }}}$ 2.9%, 8.36 KB
dist/features/debug	${{\color{Goldenrod}{ ▋ }}}$ 2.6%, 7.40 KB
dist/utilities/fieldsDrawer	${{\color{Goldenrod}{ ▋ }}}$ 2.5%, 7.18 KB
dist/lexical/config	${{\color{Goldenrod}{ ▍ }}}$ 1.8%, 5.08 KB
dist/features/lists	${{\color{Goldenrod}{ ▍ }}}$ 1.7%, 5.00 KB
dist/features/format	${{\color{Goldenrod}{ ▎ }}}$ 1.2%, 3.46 KB
dist/lexical/LexicalEditor.js	${{\color{Goldenrod}{ ▎ }}}$ 1.1%, 3.23 KB
dist/field/Field.js	${{\color{Goldenrod}{ ▎ }}}$ 1.0%, 2.81 KB
dist/lexical/nodes	${{\color{Goldenrod}{ ▏ }}}$ 0.9%, 2.66 KB
(other)	${{\color{Goldenrod}{ █████████████████████▊ }}}$ 87.3%, 250.81 KB

Meta file: packages/ui/meta_client.json, Out file: esbuild/exports/client_optimized/index.js

Path	Size
../../node_modules	${{\color{Goldenrod}{ ████████████▎ }}}$ 49.3%, 579.15 KB
dist/elements/FolderView	${{\color{Goldenrod}{ ▋ }}}$ 2.5%, 29.40 KB
dist/elements/BulkUpload	${{\color{Goldenrod}{ ▌ }}}$ 2.4%, 28.37 KB
dist/views/Edit	${{\color{Goldenrod}{ ▍ }}}$ 1.5%, 17.39 KB
dist/elements/WhereBuilder	${{\color{Goldenrod}{ ▍ }}}$ 1.5%, 17.36 KB
dist/forms/Form	${{\color{Goldenrod}{ ▎ }}}$ 1.4%, 15.91 KB
dist/elements/Table	${{\color{Goldenrod}{ ▎ }}}$ 1.3%, 15.80 KB
dist/fields/Relationship	${{\color{Goldenrod}{ ▎ }}}$ 1.3%, 15.79 KB
dist/fields/Upload	${{\color{Goldenrod}{ ▎ }}}$ 1.2%, 14.24 KB
dist/fields/Blocks	${{\color{Goldenrod}{ ▎ }}}$ 1.2%, 13.90 KB
dist/elements/QueryPresets	${{\color{Goldenrod}{ ▏ }}}$ 0.9%, 10.36 KB
dist/elements/PublishButton	${{\color{Goldenrod}{ ▏ }}}$ 0.8%, 9.11 KB
dist/providers/Folders	${{\color{Goldenrod}{ ▏ }}}$ 0.7%, 8.46 KB
dist/elements/HTMLDiff	${{\color{Goldenrod}{ ▏ }}}$ 0.7%, 8.38 KB
dist/views/CollectionFolder	${{\color{Goldenrod}{ ▏ }}}$ 0.7%, 7.83 KB
dist/fields/Array	${{\color{Goldenrod}{ ▏ }}}$ 0.7%, 7.73 KB
dist/views/List	${{\color{Goldenrod}{ ▏ }}}$ 0.6%, 7.36 KB
dist/elements/ReactSelect	${{\color{Goldenrod}{ ▏ }}}$ 0.6%, 7.33 KB
dist/elements/ListHeader	${{\color{Goldenrod}{ ▏ }}}$ 0.6%, 7.12 KB
dist/elements/LivePreview	${{\color{Goldenrod}{ ▏ }}}$ 0.6%, 7.03 KB
(other)	${{\color{Goldenrod}{ ████████████▋ }}}$ 50.7%, 596.42 KB

Meta file: packages/ui/meta_shared.json, Out file: esbuild/exports/shared_optimized/index.js

Path	Size
dist/graphics/Logo	${{\color{Goldenrod}{ █████ }}}$ 20.0%, 3.12 KB
../../node_modules	${{\color{Goldenrod}{ ████▎ }}}$ 17.0%, 2.65 KB
dist/graphics/Icon	${{\color{Goldenrod}{ ██▍ }}}$ 9.8%, 1.52 KB
dist/utilities/formatDocTitle	${{\color{Goldenrod}{ ██▏ }}}$ 8.5%, 1.32 KB
dist/providers/TableColumns	${{\color{Goldenrod}{ █▍ }}}$ 5.5%, 862 B
dist/utilities/groupNavItems.js	${{\color{Goldenrod}{ █▎ }}}$ 5.2%, 814 B
dist/utilities/getGlobalData.js	${{\color{Goldenrod}{ █▏ }}}$ 4.9%, 762 B
dist/utilities/api.js	${{\color{Goldenrod}{ █▏ }}}$ 4.8%, 756 B
dist/elements/Translation	${{\color{Goldenrod}{ ▊ }}}$ 3.2%, 493 B
dist/utilities/handleTakeOver.js	${{\color{Goldenrod}{ ▋ }}}$ 2.8%, 440 B
dist/utilities/traverseForLocalizedFields.js	${{\color{Goldenrod}{ ▋ }}}$ 2.6%, 399 B
dist/elements/withMergedProps	${{\color{Goldenrod}{ ▌ }}}$ 2.2%, 339 B
dist/utilities/getVisibleEntities.js	${{\color{Goldenrod}{ ▌ }}}$ 2.1%, 329 B
dist/utilities/getNavGroups.js	${{\color{Goldenrod}{ ▍ }}}$ 1.9%, 301 B
dist/elements/WithServerSideProps	${{\color{Goldenrod}{ ▍ }}}$ 1.5%, 232 B
dist/utilities/handleGoBack.js	${{\color{Goldenrod}{ ▎ }}}$ 1.2%, 180 B
dist/fields/mergeFieldStyles.js	${{\color{Goldenrod}{ ▎ }}}$ 1.0%, 159 B
dist/utilities/handleBackToDashboard.js	${{\color{Goldenrod}{ ▎ }}}$ 1.0%, 152 B
dist/forms/Form	${{\color{Goldenrod}{ ▏ }}}$ 0.9%, 147 B
dist/utilities/abortAndIgnore.js	${{\color{Goldenrod}{ ▏ }}}$ 0.9%, 146 B
(other)	${{\color{Goldenrod}{ ████████████████████ }}}$ 80.0%, 12.51 KB

Details

Next to the size is how much the size has increased or decreased compared with the base branch of this PR.

• ‼️: Size increased by 20% or more. Special attention should be given to this.
• ⚠️: Size increased in acceptable range (lower than 20%).
• ✅: No change or even downsized.
• 🗑️: The out file is deleted: not found in base branch.
• 🆕: The out file is newly found: will be added to base branch.