AWS SDK for Java v2 — S3TransferManager + Netty-based S3AsyncClient.builder() does not perform parallel ranged GETs for a multipart-uploaded object

[pyzhou-talend]

Describe the bug

Summary

When downloading a large (multipart-uploaded) S3 object through S3TransferManager backed by the standard Netty-based S3AsyncClient.builder(), the download is executed as a single, non-ranged GET instead of multiple parallel ranged GETs. As a result the transfer runs at single-connection throughput and never uses the configured concurrency.

Control experiments confirm the problem is specific to this combination:

Client | Parallel ranged GETs? -- | -- S3AsyncClient.crtBuilder() + S3TransferManager | ✅ Yes — works as documented S3AsyncClient.builder() (Netty) + S3TransferManager | ❌ No — single GET, no Range header Manual S3AsyncClient.builder() (Netty) issuing per-part GetObjectRequest with partNumber/Range | ✅ Yes — client itself is fully capable

So the S3 object supports ranged reads, the Netty client is fine, and CRT + TransferManager works. Only S3TransferManager on top of the Netty async client fails to split the download.

Presence/absence of an HTTP proxy is irrelevant — the bug reproduces with no proxy at all.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

S3TransferManager + Netty-based S3AsyncClient.builder() should perform parallel ranged GETs for a multipart-uploaded object

Current Behavior

For a ~140 MB MPU object, S3TransferManager.downloadFile(...) should — regardless of the underlying async client — issue multiple parallel GET requests with Range: bytes=start-end (or ?partNumber=N) to saturate maxConcurrency. This is the primary value proposition of S3TransferManager.

Either:

S3TransferManager on the Netty client should perform the split itself, or
The documentation should clearly state that split-download only works with the CRT client, and S3TransferManager.builder().s3Client(nettyAsyncClient) should log a prominent WARN at construction time.

Reproduction Steps

Configure an SSO profile default in ~/.aws/config.
Run the reproducer below (essentially S3TransforManagerTest.java in this repo).
Observe the wire logs and progress output.
Reproducer (Netty S3AsyncClient.builder() + S3TransferManager)
try (S3AsyncClient s3 = S3AsyncClient.builder()
.region(Region.of("us-east-1"))
.credentialsProvider(ProfileCredentialsProvider.create("default"))
.httpClientBuilder(NettyNioAsyncHttpClient.builder()
.maxConcurrency(200)
.connectionTimeout(Duration.ofSeconds(30))
.readTimeout(Duration.ofSeconds(60)))
.build();
S3TransferManager tm = S3TransferManager.builder().s3Client(s3).build()) {

FileDownload dl = tm.downloadFile(DownloadFileRequest.builder()
        .getObjectRequest(b -> b.bucket("pyzhoutest").key("ffff"))
        .destination(Paths.get("downloads/large-file.bin"))
        .build());

dl.completionFuture().join();

}
Control 1 — CRT client (works correctly)
Same code but with S3AsyncClient.crtBuilder()...build() → multiple parallel ranged GETs are observed, throughput scales with concurrency. ✅

Control 2 — Manual parallel GETs on Netty client (works correctly)
Using the same Netty S3AsyncClient.builder() client but issuing N getObject(...) calls in parallel, each with .partNumber(i) or .range("bytes=start-end"), achieves full parallelism. ✅ This proves the Netty client and the network path have no problem serving ranged parallel GETs; only S3TransferManager's split logic is missing on this client.

Possible Solution

No response

Additional Information/Context

Only one GET is ever sent, without Range and without partNumber, and bytes arrive on a single Netty event-loop thread:

开始下载: s3://pyzhoutest/ffff
对象大小: 147297280 bytes, 按 5MB 估算分片数: 29
HeadObject: ETag="af08b66120cc11d69332976330da8273-29", partsCount=null, acceptRanges=bytes
如果并行分片生效，下面会看到多条 GET 请求且带 Range=bytes=start-end
[HTTP-OUT][main] GET https://s3.amazonaws.com/pyzhoutest/ffff Range=
进度: 0 bytes, 平均速度: 0.00 MB/s
[HTTP-IN ][aws-java-sdk-NettyEventLoop-1-2] status=200
进度: 0.01% (16715/147297280 bytes), 平均速度: 0.02 MB/s
进度: 0.02% (34123/147297280 bytes), 平均速度: 0.02 MB/s
进度: 0.04% (53261/147297280 bytes), 平均速度: 0.02 MB/s
...
Key observations:

Exactly one GET line for the whole transfer.
No Range: bytes=... header, no ?partNumber=N.
All bytes on a single aws-java-sdk-NettyEventLoop-* thread → the client-level max concurrency is unused.
DEBUG logs on software.amazon.awssdk show no “split / schedule part / range” activity from TransferManager.
What I already tried
DEBUG logging on software.amazon.awssdk — no evidence of any split-download logic being invoked when the underlying client is the Netty S3AsyncClient.
Raised Netty maxConcurrency to 200 — no change; still 1 in-flight GET.
forcePathStyle(true) — no change.
Removed the HTTP proxy entirely — no change (single non-ranged GET).
Switched to S3AsyncClient.crtBuilder() with everything else identical — parallel ranged GETs work correctly.
Manual multi-part parallel GETs on the same Netty client — work correctly.

AWS Java SDK version used

2.54.7 & 2.35.11

JDK version used

oracle-17

Operating System and version

windows 11

[debora-ito]

Hi @pyzhou-talend thank you for reaching out.

Parallel multipart transfers are supported when you set the multipartEnabled option to true:

    S3AsyncClient s3AsyncClient = S3AsyncClient.builder()
        .multipartEnabled(true)
        .build();

Multipart is an opt-in feature because it was introduced after the initial release of TransferManager downloadFile, which by default is a full-size GET request as you noticed. Adding an opt-in flag is the way to not make a breaking change of a default behavior.

You can check for more multipart configurations in the Dev Guide - Configure parallel transfer support.

Let us know if this helps.

[pyzhou-talend]

Hi @debora-ito
Thanks for the answer, actually I already check the Dev guide and try to add .multipartEnabled(true) in code.
Actually, my point is not about multipart, but about parallelism. Yes, after adding  .multipartEnabled(true) , the download request is split into 8‑MB chunks, with each chunk as a separate GET request. However, these requests run serially, which renders chunking pointless. The whole purpose of chunking is to enable parallel downloads, not just to split the data into pieces.

[debora-ito]

The download requests are expected to be performed in parallel with multipartEnabled(true), so if you're seeing differently we should investigate why. My guess is a customizable setting is forcing the downloads to be serial (for example unintentionally setting maxConcurrency to 1) or maybe the connection pool doesn't have lots of connections available in case other S3 requests are being executed at the same time.

Some questions to get more context:

• How are you checking that the downloads are being done in serial?
• Can you share a self-contained SSCCE code I can use to reproduce the issue on my side? The code sample shared in the description doesn't have the multipart enabled option
• Can you enable wire logs?
• Netty wire logs can be very very verbose, so if you can't enabled them can you at least share the TRACE-level logging on software.amazon.awssdk.services.s3.internal.multipart?