Add presigned URL download support to S3 Transfer Manager#6951
Add presigned URL download support to S3 Transfer Manager#6951jencymaryjoseph wants to merge 2 commits into
Conversation
jencymaryjoseph
force-pushed
the
feature/master/pre-signed-url-getobject
branch
from
e037d5d to
10e45b1
Compare
jencymaryjoseph
force-pushed
the
jencyjos/pre-signed-url-getobject/transfermanager
branch
from
1e11a98 to
15a7271
Compare
jencymaryjoseph
force-pushed
the
jencyjos/pre-signed-url-getobject/transfermanager
branch
from
15a7271 to
f23f949
Compare
| private static final String SMALL_KEY = "small-key"; | ||
| private static final String LARGE_KEY = "large-key"; | ||
| private static final int SMALL_OBJ_SIZE = 5 * 1024 * 1024; | ||
| private static final int LARGE_OBJ_SIZE = 16 * 1024 * 1024; |
There was a problem hiding this comment.
Can you please run one off tests (you dont have to commit those to the PR) against larger files like 1GB and 10GB. Trying to see if size would reveal any edge cases.
There was a problem hiding this comment.
Tested for 1GB download, looks good. Will do more tests during performance tests.
|
We need to add the new methods to |
| public void methodTeardown() { | ||
| tm.close(); | ||
| } | ||
|
|
There was a problem hiding this comment.
We are missing tests that verify that the progress tracking works for presigned URL downloads.
See S3TransferManagerListenerTest.java as an example. (you need to make sure the callback hooks are firing for the provided transfer listener)
| returnFuture.completeExceptionally(throwable); | ||
| } | ||
|
|
||
| return new DefaultFileDownload(returnFuture, progressUpdater.progress(), () -> null, null); |
There was a problem hiding this comment.
The callback shouldn't just return null. We should throw an exception instead if customers call .pause();
() -> { throw new UnsupportedOperationException("pause is not supported...."); }| protected static S3Client s3; | ||
|
|
||
| protected static S3AsyncClient s3Async; | ||
| protected static S3AsyncClient s3Async;; |
| Arguments.of(tmJava)); | ||
| } | ||
|
|
||
| static Stream<Arguments> presignedUrlTransferManagers() { |
2 participants
Motivation and Context
Adds presigned URL download support to S3 Transfer Manager with
downloadFileWithPresignedUrlanddownloadWithPresignedUrlmethods. This enables customers to download objects using pre-signed URLs through the Transfer Manager with multipart download support and progress tracking.Modifications
S3TransferManager: AddeddownloadFileWithPresignedUrl(with consumer builder variant) anddownloadWithPresignedUrlpublic API methods with javadoc and code snippets.GenericS3TransferManager: Implemented presigned URL download logic with multipart support and progress tracking using wrapForNonSerialFileDownload.S3IntegrationTestBase: AddedtmNonMultipartJava(non-multipart transfer manager) andpresignedUrlTransferManagers() method source for presigned URL integration tests. CRT client is commented out with a TODO to enable when CRT presigned URL support is available.Testing
S3TransferManagerPresignedUrlDownloadTest: Unit tests covering happy path (toFile/toBytes), consumer builder, cancellation forwarding, error propagation, and null validation.S3TransferManagerPresignedUrlDownloadIntegrationTest: Integration tests parameterized by transfer manager type (nonMultipart/multipart) and file size (5MB/16MB), verifying data integrity with checksums for both downloadFileWithPresignedUrl and downloadWithPresignedUrl.Screenshots (if appropriate)
Types of changes
Checklist
mvn installsucceedsscripts/new-changescript and following the instructions. Commit the new file created by the script in.changes/next-releasewith your changes.License