Skip to content

fix(storage): do not apply retention or holds to parallel composite upload parts - #14894

Open
sandeepvinayak wants to merge 1 commit into
googleapis:mainfrom
sandeepvinayak:fix/storage-pcu-part-retention
Open

sandeepvinayak wants to merge 1 commit into
googleapis:mainfrom
sandeepvinayak:fix/storage-pcu-part-retention

Conversation

@sandeepvinayak

Copy link
Copy Markdown

Fixes #14892

Problem

ParallelCompositeUploadWritableByteChannel#definePart builds every part and intermediate compose object from ultimateObject.toBuilder(). It cleared only crc32c, md5, and kmsKeyName, so when the destination BlobInfo set Retention, temporaryHold, or eventBasedHold, every part inherited them. The final compose succeeded, but GCS rejects deletes of retained or held objects, so:

  • cleanupParts failed, and close() / UploadResult reported FAILED even though the destination object existed.
  • Part objects were left behind. With Retention.Mode.LOCKED, they can't be removed until the retain-until time.

Fix

definePart now clears retention, temporaryHold, and eventBasedHold on parts and intermediate compose targets. The final compose target is still ultimateObject, so the destination keeps all three.

Testing

New unit test ParallelCompositeUploadWritableByteChannelTest#partsDoNotInheritRetentionOrHolds:

  • The fake storage rejects deletes of retained or held objects, as GCS does.
  • maxElementsPerCompact = 2 forces intermediate composes, so both code paths that call definePart are covered.
  • It checks that no part or intermediate target carries retention or holds, that the final target keeps all three, and that the upload completes.

The test failed before the fix (AsynchronousCloseException from close()) and passes after it. All 26 ParallelCompositeUpload*Test tests pass.

The optional follow-up in the issue (exposing PartMetadataFieldDecorator / PartCleanupStrategy through TransferManagerConfig) is a public API addition and is left out of this PR.

…pload parts

Parts and intermediate compose objects were built from the destination
BlobInfo, so they inherited its retention, temporaryHold and
eventBasedHold. GCS then rejected the part cleanup deletes, causing the
upload to report failure after a successful final compose and leaving
undeletable part objects behind.

Fixes googleapis#14892
@sandeepvinayak
sandeepvinayak requested review from a team as code owners November 2, 2026 18:43

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request ensures that transient parts of a parallel composite upload do not inherit retention or hold settings by explicitly setting them to null, as these properties only belong on the final composed object. A unit test was added to verify this behavior. The review feedback suggests refactoring the AssertJ assertions in the new test to use allSatisfy instead of stream().allMatch(...).isTrue() to improve error reporting upon failure.

Comment on lines +926 to +929
() -> assertThat(partTargets.stream().allMatch(p -> p.getRetention() == null)).isTrue(),
() -> assertThat(partTargets.stream().allMatch(p -> p.getTemporaryHold() == null)).isTrue(),
() ->
assertThat(partTargets.stream().allMatch(p -> p.getEventBasedHold() == null)).isTrue(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using stream().allMatch(...).isTrue() in AssertJ assertions is an anti-pattern because it produces unhelpful error messages (e.g., Expected true but was false) if the assertion fails.

Using allSatisfy provides much more descriptive failure messages, showing exactly which element failed and what its actual values were.

Suggested change
() -> assertThat(partTargets.stream().allMatch(p -> p.getRetention() == null)).isTrue(),
() -> assertThat(partTargets.stream().allMatch(p -> p.getTemporaryHold() == null)).isTrue(),
() ->
assertThat(partTargets.stream().allMatch(p -> p.getEventBasedHold() == null)).isTrue(),
() -> assertThat(partTargets).allSatisfy(p -> {
assertThat(p.getRetention()).isNull();
assertThat(p.getTemporaryHold()).isNull();
assertThat(p.getEventBasedHold()).isNull();
}),

@sandeepvinayak

Copy link
Copy Markdown
Author

@ShreyasSinha I see you have been active on storage, can you please help to review it? Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant