Skip to content

maxInFlightParts support for PutObject#6801

Open
alextwoods wants to merge 12 commits intomasterfrom
alexwoo/max_in_flight_s3
Open

maxInFlightParts support for PutObject#6801
alextwoods wants to merge 12 commits intomasterfrom
alexwoo/max_in_flight_s3

Conversation

@alextwoods
Copy link
Contributor

@alextwoods alextwoods commented Mar 17, 2026

Motivation and Context

When MultipartS3AsyncClient.putObject() performs a multipart upload of a large object, all UploadPart requests are dispatched eagerly with no concurrency limit. For example, a 2 GB upload with 8 MiB parts produces ~256 UploadPart requests that immediately compete for the HTTP connection pool (default maxConcurrency=50), leading to connection acquisition timeouts:

SdkClientException: Unable to execute HTTP request: Acquire operation took longer than the configured maximum time.

The existing maxInFlightParts configuration in ParallelConfiguration already limits concurrent GetObject requests for multipart downloads, but was not applied to the upload path.

Fixes #6623

Modifications

  • Extended the maxInFlightParts configuration to also apply to multipart upload (putObject) concurrency. The setting now limits concurrent part requests for both getObject (download) and putObject (upload) operations. Updated the Javadoc on ParallelConfiguration.maxInFlightParts() to reflect this broader scope.
  • In KnownContentLengthAsyncRequestBodySubscriber.onNext(): instead of unconditionally calling subscription.request(1) after dispatching each UploadPart, the subscriber now checks asyncRequestBodyInFlight < maxInFlightParts. When a part completes and in-flight count drops below the limit, subscription.request(1) resumes flow.
  • Applied the same pattern in UnknownContentLengthAsyncRequestBodySubscriber.sendUploadPartRequest().
  • Extracted UnknownContentLengthAsyncRequestBodySubscriber from an inner class of UploadWithUnknownContentLengthHelper into its own top-level class, matching the existing pattern of KnownContentLengthAsyncRequestBodySubscriber.

Testing

  • New and existing unit tests.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)

Checklist

  • I have read the CONTRIBUTING document
  • Local run of mvn install succeeds
  • My code follows the code style of this project
  • My change requires a change to the Javadoc documentation
  • I have updated the Javadoc documentation accordingly
  • I have added tests to cover my changes
  • All new and existing tests passed
  • I have added a changelog entry. Adding a new entry must be accomplished by running the scripts/new-change script and following the instructions. Commit the new file created by the script in .changes/next-release with your changes.

License

  • I confirm that this pull request can be released under the Apache 2 license

@alextwoods alextwoods changed the title Alexwoo/max in flight s3 maxInFlightParts support for PutObject Mar 17, 2026
"software.amazon.awssdk.services.s3.internal.crt.CrtResponseFileResponseTransformer"),
ArchUtils.classNameToPattern(RetryableSubAsyncRequestBody.class)));
ArchUtils.classNameToPattern(RetryableSubAsyncRequestBody.class),
ArchUtils.classNameToPattern(KnownContentLengthAsyncRequestBodySubscriber.class)));
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This warn logging was pre-existing - see the change in UploadWithUnknownContentLengthHelper that moved the existing warning into KnownContentLengthAsyncRequestBodySubscriber

@alextwoods alextwoods added the api-surface-area-approved-by-team Indicate API surface area introduced by this PR has been approved by team label Mar 18, 2026
@alextwoods alextwoods marked this pull request as ready for review March 18, 2026 19:28
@alextwoods alextwoods requested a review from a team as a code owner March 18, 2026 19:28
@sonarqubecloud
Copy link

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api-surface-area-approved-by-team Indicate API surface area introduced by this PR has been approved by team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Configurable max concurrency for MultipartS3AsyncClient multipart operations

1 participant