Repository navigation
fix(storage): do not apply retention or holds to parallel composite upload parts - #14894
sandeepvinayak wants to merge 1 commit into
Conversation
…pload parts Parts and intermediate compose objects were built from the destination BlobInfo, so they inherited its retention, temporaryHold and eventBasedHold. GCS then rejected the part cleanup deletes, causing the upload to report failure after a successful final compose and leaving undeletable part objects behind. Fixes googleapis#14892
There was a problem hiding this comment.
Code Review
This pull request ensures that transient parts of a parallel composite upload do not inherit retention or hold settings by explicitly setting them to null, as these properties only belong on the final composed object. A unit test was added to verify this behavior. The review feedback suggests refactoring the AssertJ assertions in the new test to use allSatisfy instead of stream().allMatch(...).isTrue() to improve error reporting upon failure.
| () -> assertThat(partTargets.stream().allMatch(p -> p.getRetention() == null)).isTrue(), | ||
| () -> assertThat(partTargets.stream().allMatch(p -> p.getTemporaryHold() == null)).isTrue(), | ||
| () -> | ||
| assertThat(partTargets.stream().allMatch(p -> p.getEventBasedHold() == null)).isTrue(), |
There was a problem hiding this comment.
Using stream().allMatch(...).isTrue() in AssertJ assertions is an anti-pattern because it produces unhelpful error messages (e.g., Expected true but was false) if the assertion fails.
Using allSatisfy provides much more descriptive failure messages, showing exactly which element failed and what its actual values were.
| () -> assertThat(partTargets.stream().allMatch(p -> p.getRetention() == null)).isTrue(), | |
| () -> assertThat(partTargets.stream().allMatch(p -> p.getTemporaryHold() == null)).isTrue(), | |
| () -> | |
| assertThat(partTargets.stream().allMatch(p -> p.getEventBasedHold() == null)).isTrue(), | |
| () -> assertThat(partTargets).allSatisfy(p -> { | |
| assertThat(p.getRetention()).isNull(); | |
| assertThat(p.getTemporaryHold()).isNull(); | |
| assertThat(p.getEventBasedHold()).isNull(); | |
| }), |
|
@ShreyasSinha I see you have been active on storage, can you please help to review it? Thanks! |
Fixes #14892
Problem
ParallelCompositeUploadWritableByteChannel#definePartbuilds every part and intermediate compose object fromultimateObject.toBuilder(). It cleared only crc32c, md5, and kmsKeyName, so when the destinationBlobInfosetRetention,temporaryHold, oreventBasedHold, every part inherited them. The final compose succeeded, but GCS rejects deletes of retained or held objects, so:cleanupPartsfailed, andclose()/UploadResultreported FAILED even though the destination object existed.Retention.Mode.LOCKED, they can't be removed until the retain-until time.Fix
definePartnow clearsretention,temporaryHold, andeventBasedHoldon parts and intermediate compose targets. The final compose target is stillultimateObject, so the destination keeps all three.Testing
New unit test
ParallelCompositeUploadWritableByteChannelTest#partsDoNotInheritRetentionOrHolds:maxElementsPerCompact = 2forces intermediate composes, so both code paths that calldefinePartare covered.The test failed before the fix (
AsynchronousCloseExceptionfromclose()) and passes after it. All 26ParallelCompositeUpload*Testtests pass.The optional follow-up in the issue (exposing
PartMetadataFieldDecorator/PartCleanupStrategythroughTransferManagerConfig) is a public API addition and is left out of this PR.