Skip to content

Fix S3Express utils and tests for auth scheme resolution pipeline migration #6947

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
S-Saranya1 merged 3 commits into from
May 13, 2026
Merged

Fix S3Express utils and tests for auth scheme resolution pipeline migration #6947

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
40605b3
Fix S3Express utils and tests for auth scheme resolution pipeline mig…
S-Saranya1 May 7, 2026
db13aad
Address PR feedback
S-Saranya1 May 12, 2026
d2cc6c9
Address review comments
S-Saranya1 May 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ...en/src/main/java/software/amazon/awssdk/codegen/poet/rules/EndpointResolverUtilsSpec.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -676,7 +676,7 @@ private MethodSpec setMetricValuesMethod() {
b.endControlFlow();

if (endpointRulesSpecUtils.isS3()) {
b.addStatement("$T.addS3ExpressBusinessMetricIfApplicable(executionAttributes)",
b.addStatement("$T.addS3ExpressBusinessMetricIfApplicable(endpoint, executionAttributes)",
ClassName.get("software.amazon.awssdk.services.s3.internal.s3express", "S3ExpressUtils"));
}

Expand Down
2 changes: 1 addition & 1 deletion services/s3/src/main/java/software/amazon/awssdk/services/s3/S3Utilities.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,8 @@
import software.amazon.awssdk.awscore.endpoint.AwsClientEndpointProvider;
import software.amazon.awssdk.awscore.endpoint.DualstackEnabledProvider;
import software.amazon.awssdk.awscore.endpoint.FipsEnabledProvider;
import software.amazon.awssdk.awscore.internal.defaultsmode.DefaultsModeConfiguration;
import software.amazon.awssdk.awscore.endpoints.AwsEndpointProviderUtils;
import software.amazon.awssdk.awscore.internal.defaultsmode.DefaultsModeConfiguration;
import software.amazon.awssdk.core.ClientEndpointProvider;
import software.amazon.awssdk.core.ClientType;
import software.amazon.awssdk.core.client.config.ClientOverrideConfiguration;
Expand Down
2 changes: 1 addition & 1 deletion ...rc/main/java/software/amazon/awssdk/services/s3/internal/crt/DefaultS3CrtAsyncClient.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -430,7 +430,7 @@ public void afterMarshalling(Context.AfterMarshalling context,
.put(SIGNING_REGION, executionAttributes.getAttribute(AwsSignerExecutionAttribute.SIGNING_REGION))
.put(S3InternalSdkHttpExecutionAttribute.OBJECT_FILE_PATH,
executionAttributes.getAttribute(OBJECT_FILE_PATH))
.put(USE_S3_EXPRESS_AUTH, S3ExpressUtils.useS3ExpressAuthScheme(executionAttributes))
.put(USE_S3_EXPRESS_AUTH, S3ExpressUtils.isS3ExpressAuthRequest(context.request(), executionAttributes))
Copy link
Copy Markdown
Contributor

@joviegas joviegas May 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The previous code checked SELECTED_AUTH_SCHEME, which is set by S3AuthSchemeInterceptor.beforeExecution. The new isS3ExpressAuthRequest
depends on AUTH_SCHEME_OPTIONS_RESOLVER being available. If null, it silently returns false.
Do we have tests verifying USE_S3_EXPRESS_AUTH is correctly set in the CRT path for:

  1. S3 Express bucket with default config → USE_S3_EXPRESS_AUTH is true
  2. S3 Express bucket with disableS3ExpressSessionAuth(true) → USE_S3_EXPRESS_AUTH is false

Copy link
Copy Markdown
Author

@S-Saranya1 S-Saranya1 May 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We dont have any currently, added tests for these cases.

.put(SIGNING_NAME, executionAttributes.getAttribute(SERVICE_SIGNING_NAME))
.put(REQUEST_CHECKSUM_CALCULATION,
executionAttributes.getAttribute(SdkInternalExecutionAttribute.REQUEST_CHECKSUM_CALCULATION))
Expand Down
4 changes: 2 additions & 2 deletions ...ftware/amazon/awssdk/services/s3/internal/handlers/EnableTrailingChecksumInterceptor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ public SdkRequest modifyRequest(Context.ModifyRequest context, ExecutionAttribut

SdkRequest request = context.request();
if (getObjectChecksumEnabledPerRequest(request, executionAttributes)
&& S3ExpressUtils.useS3Express(executionAttributes)) {
&& S3ExpressUtils.isS3ExpressBucket(request)) {
return ((GetObjectRequest) request).toBuilder().checksumMode(ChecksumMode.ENABLED).build();
}
return request;
Expand All @@ -63,7 +63,7 @@ public SdkHttpRequest modifyHttpRequest(Context.ModifyHttpRequest context,
ExecutionAttributes executionAttributes) {

if (getObjectChecksumEnabledPerRequest(context.request(), executionAttributes)
&& !S3ExpressUtils.useS3Express(executionAttributes)) {
&& !S3ExpressUtils.isS3ExpressBucket(context.request())) {
return context.httpRequest()
.toBuilder()
.putHeader(ENABLE_CHECKSUM_REQUEST_HEADER, ENABLE_MD5_CHECKSUM_HEADER_VALUE)
Expand Down
36 changes: 26 additions & 10 deletions ...3/src/main/java/software/amazon/awssdk/services/s3/internal/s3express/S3ExpressUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,13 @@

import static software.amazon.awssdk.core.interceptor.SdkInternalExecutionAttribute.SELECTED_AUTH_SCHEME;

import java.util.List;
import software.amazon.awssdk.annotations.SdkInternalApi;
import software.amazon.awssdk.core.SdkRequest;
import software.amazon.awssdk.core.SelectedAuthScheme;
import software.amazon.awssdk.core.interceptor.ExecutionAttributes;
import software.amazon.awssdk.core.interceptor.SdkInternalExecutionAttribute;
import software.amazon.awssdk.core.spi.identity.AuthSchemeOptionsResolver;
import software.amazon.awssdk.core.useragent.BusinessMetricFeatureId;
import software.amazon.awssdk.endpoints.Endpoint;
import software.amazon.awssdk.http.auth.spi.scheme.AuthSchemeOption;
Expand All @@ -31,26 +34,37 @@
public final class S3ExpressUtils {

public static final String S3_EXPRESS = "S3Express";
private static final String S3_EXPRESS_BUCKET_SUFFIX = "--x-s3";

private S3ExpressUtils() {
}

/**
* Returns true if the resolved endpoint contains S3Express, else false.
* Determines if this request targets an S3Express bucket by checking the bucket name suffix.
*/
public static boolean useS3Express(ExecutionAttributes executionAttributes) {
Endpoint endpoint = executionAttributes.getAttribute(SdkInternalExecutionAttribute.RESOLVED_ENDPOINT);
if (endpoint != null) {
String useS3Express = endpoint.attribute(KnownS3ExpressEndpointProperty.BACKEND);
return S3_EXPRESS.equals(useS3Express);
public static boolean isS3ExpressBucket(SdkRequest request) {
return request.getValueForField("Bucket", String.class)
.map(b -> b.endsWith(S3_EXPRESS_BUCKET_SUFFIX))
.orElse(false);
Copy link
Copy Markdown
Contributor

@joviegas joviegas May 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit/Not blocking query: The endpoint ruleset

aws-sdk-java-v2/services/s3/src/main/resources/codegen-resources/endpoint-rule-set.json

Lines 261 to 280 in 2619c3d

"fn": "substring",
"argv": [
{
"ref": "Bucket"
},
0,
6,
true
],
"assign": "bucketSuffix"
},
{
"fn": "stringEquals",
"argv": [
{
"ref": "bucketSuffix"
},
"--x-s3"
]
}
determines S3Express by checking if the bucket name ends with --x-s3. This ruleset is auto-updated without SDK code changes. isS3ExpressBucket duplicates this logic with a hardcoded endsWith("--x-s3"). If the service adds a new S3Express suffix (say, --exp-s3) in a future ruleset update, this hardcoded check would silently fail to match. Is it possible to add a test that validates the S3Express suffix in the ruleset matches what isS3ExpressBucket checks, so we catch any mismatch when the ruleset updates?

Copy link
Copy Markdown
Author

@S-Saranya1 S-Saranya1 May 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah makes sense, added tests for this.

}

/**
* Determines if this request uses S3Express auth by checking the auth scheme options.
*/
public static boolean isS3ExpressAuthRequest(SdkRequest request, ExecutionAttributes executionAttributes) {
AuthSchemeOptionsResolver resolver =
executionAttributes.getAttribute(SdkInternalExecutionAttribute.AUTH_SCHEME_OPTIONS_RESOLVER);
if (resolver != null) {
List<AuthSchemeOption> options = resolver.resolve(request);
return options.stream().anyMatch(o -> S3ExpressAuthScheme.SCHEME_ID.equals(o.schemeId()));
}
return false;
}

/**
* Whether aws.auth#sigv4-s3express is used or not
* Whether aws.auth#sigv4-s3express is the selected auth scheme.
*/
public static boolean useS3ExpressAuthScheme(ExecutionAttributes executionAttributes) {
private static boolean useS3ExpressAuthScheme(ExecutionAttributes executionAttributes) {
SelectedAuthScheme<?> selectedAuthScheme = executionAttributes.getAttribute(SELECTED_AUTH_SCHEME);
if (selectedAuthScheme != null) {
AuthSchemeOption authSchemeOption = selectedAuthScheme.authSchemeOption();
Expand All @@ -62,8 +76,10 @@ public static boolean useS3ExpressAuthScheme(ExecutionAttributes executionAttrib
/**
* Adds S3 Express business metric if applicable for the current operation.
*/
public static void addS3ExpressBusinessMetricIfApplicable(ExecutionAttributes executionAttributes) {
if (executionAttributes != null && useS3Express(executionAttributes) && useS3ExpressAuthScheme(executionAttributes)) {
public static void addS3ExpressBusinessMetricIfApplicable(Endpoint endpoint, ExecutionAttributes executionAttributes) {
if (endpoint != null && executionAttributes != null
&& S3_EXPRESS.equals(endpoint.attribute(KnownS3ExpressEndpointProperty.BACKEND))
&& useS3ExpressAuthScheme(executionAttributes)) {
executionAttributes.getOptionalAttribute(SdkInternalExecutionAttribute.BUSINESS_METRICS)
.ifPresent(businessMetrics ->
businessMetrics.addMetric(BusinessMetricFeatureId.S3_EXPRESS_BUCKET.value()));
Expand Down
2 changes: 1 addition & 1 deletion ...src/main/java/software/amazon/awssdk/services/s3/internal/signing/DefaultS3Presigner.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,10 +44,10 @@
import software.amazon.awssdk.awscore.defaultsmode.DefaultsMode;
import software.amazon.awssdk.awscore.endpoint.AwsClientEndpointProvider;
import software.amazon.awssdk.awscore.endpoints.AwsEndpointAttribute;
import software.amazon.awssdk.awscore.endpoints.AwsEndpointProviderUtils;
import software.amazon.awssdk.awscore.endpoints.authscheme.EndpointAuthScheme;
import software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder;
import software.amazon.awssdk.awscore.internal.defaultsmode.DefaultsModeConfiguration;
import software.amazon.awssdk.awscore.endpoints.AwsEndpointProviderUtils;
import software.amazon.awssdk.awscore.presigner.PresignRequest;
import software.amazon.awssdk.awscore.presigner.PresignedRequest;
import software.amazon.awssdk.core.ClientEndpointProvider;
Expand Down
6 changes: 5 additions & 1 deletion ...t/java/software/amazon/awssdk/services/s3/functionaltests/S3ExpressCreateSessionTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -316,7 +316,11 @@ private static final class PathStyleEnforcingInterceptor implements ExecutionInt
public SdkHttpRequest modifyHttpRequest(Context.ModifyHttpRequest context, ExecutionAttributes executionAttributes) {
SdkHttpRequest sdkHttpRequest = context.httpRequest();
String host = sdkHttpRequest.host();
String bucket = host.substring(0, host.indexOf(".localhost"));
int idx = host.indexOf(".localhost");
if (idx < 0) {
return sdkHttpRequest;
}
String bucket = host.substring(0, idx);

return sdkHttpRequest.toBuilder().host("localhost")
.encodedPath(SdkHttpUtils.appendUri(bucket, sdkHttpRequest.encodedPath()))
Expand Down
6 changes: 5 additions & 1 deletion ...es/s3/src/test/java/software/amazon/awssdk/services/s3/functionaltests/S3ExpressTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -411,7 +411,11 @@ private static final class PathStyleEnforcingInterceptor implements ExecutionInt
public SdkHttpRequest modifyHttpRequest(Context.ModifyHttpRequest context, ExecutionAttributes executionAttributes) {
SdkHttpRequest sdkHttpRequest = context.httpRequest();
String host = sdkHttpRequest.host();
String bucket = host.substring(0, host.indexOf(".localhost"));
int idx = host.indexOf(".localhost");
if (idx < 0) {
return sdkHttpRequest;
}
String bucket = host.substring(0, idx);

return sdkHttpRequest.toBuilder().host("localhost")
.encodedPath(SdkHttpUtils.appendUri(bucket, sdkHttpRequest.encodedPath()))
Expand Down
75 changes: 75 additions & 0 deletions ...est/java/software/amazon/awssdk/services/s3/internal/crt/DefaultS3CrtAsyncClientTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,9 @@
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.auth.signer.AwsS3V4Signer;
import software.amazon.awssdk.core.async.AsyncRequestBody;
import software.amazon.awssdk.core.async.AsyncResponseTransformer;
Expand All @@ -31,8 +33,10 @@
import software.amazon.awssdk.core.interceptor.ExecutionAttributes;
import software.amazon.awssdk.core.interceptor.ExecutionInterceptor;
import software.amazon.awssdk.core.interceptor.SdkInternalExecutionAttribute;
import software.amazon.awssdk.http.SdkHttpExecutionAttributes;
import software.amazon.awssdk.identity.spi.AwsCredentialsIdentity;
import software.amazon.awssdk.identity.spi.IdentityProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.DelegatingS3AsyncClient;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.endpoints.S3ClientContextParams;
Expand Down Expand Up @@ -161,4 +165,75 @@ void build_withAdvancedOptions() {
assertThat(client).isInstanceOf(DefaultS3CrtAsyncClient.class);
}
}

@Test
void s3ExpressBucket_defaultConfig_useS3ExpressAuthIsTrue() {
AtomicReference<Boolean> capturedUseS3ExpressAuth = new AtomicReference<>();

ExecutionInterceptor captor = new ExecutionInterceptor() {
@Override
public void beforeTransmission(Context.BeforeTransmission context, ExecutionAttributes executionAttributes) {
SdkHttpExecutionAttributes httpAttrs =
executionAttributes.getAttribute(SdkInternalExecutionAttribute.SDK_HTTP_EXECUTION_ATTRIBUTES);
if (httpAttrs != null) {
capturedUseS3ExpressAuth.set(httpAttrs.getAttribute(S3InternalSdkHttpExecutionAttribute.USE_S3_EXPRESS_AUTH));
}
throw new RuntimeException("STOP");
}
};

DefaultS3CrtAsyncClient.DefaultS3CrtClientBuilder builder =
(DefaultS3CrtAsyncClient.DefaultS3CrtClientBuilder) S3CrtAsyncClient.builder();
builder.addExecutionInterceptor(captor);

try (S3AsyncClient client = builder
.region(Region.US_EAST_1)
.credentialsProvider(StaticCredentialsProvider.create(
AwsBasicCredentials.create("key", "secret")))
.build()) {

assertThatThrownBy(() -> client.getObject(
r -> r.bucket("my-bucket--usw2-az1--x-s3").key("key"),
AsyncResponseTransformer.toBytes()).join())
.hasMessageContaining("STOP");
}

assertThat(capturedUseS3ExpressAuth.get()).isTrue();
}

@Test
void s3ExpressBucket_disableS3ExpressSessionAuth_useS3ExpressAuthIsFalse() {
AtomicReference<Boolean> capturedUseS3ExpressAuth = new AtomicReference<>();

ExecutionInterceptor captor = new ExecutionInterceptor() {
@Override
public void beforeTransmission(Context.BeforeTransmission context, ExecutionAttributes executionAttributes) {
SdkHttpExecutionAttributes httpAttrs =
executionAttributes.getAttribute(SdkInternalExecutionAttribute.SDK_HTTP_EXECUTION_ATTRIBUTES);
if (httpAttrs != null) {
capturedUseS3ExpressAuth.set(httpAttrs.getAttribute(S3InternalSdkHttpExecutionAttribute.USE_S3_EXPRESS_AUTH));
}
throw new RuntimeException("STOP");
}
};

DefaultS3CrtAsyncClient.DefaultS3CrtClientBuilder builder =
(DefaultS3CrtAsyncClient.DefaultS3CrtClientBuilder) S3CrtAsyncClient.builder();
builder.addExecutionInterceptor(captor);

try (S3AsyncClient client = builder
.region(Region.US_EAST_1)
.credentialsProvider(StaticCredentialsProvider.create(
AwsBasicCredentials.create("key", "secret")))
.disableS3ExpressSessionAuth(true)
.build()) {

assertThatThrownBy(() -> client.getObject(
r -> r.bucket("my-bucket--usw2-az1--x-s3").key("key"),
AsyncResponseTransformer.toBytes()).join())
.hasMessageContaining("STOP");
}

assertThat(capturedUseS3ExpressAuth.get()).isFalse();
}
}
27 changes: 20 additions & 7 deletions ...a/software/amazon/awssdk/services/s3/internal/s3express/S3ExpressCacheFunctionalTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.awscore.AwsRequestOverrideConfiguration;
import software.amazon.awssdk.core.SelectedAuthScheme;
import software.amazon.awssdk.core.interceptor.Context;
import software.amazon.awssdk.core.interceptor.ExecutionAttributes;
Expand Down Expand Up @@ -278,21 +279,29 @@ public List<IdentityProvider<AwsCredentialsIdentity>> apiCredentialsProviders()

@Override
public void beforeExecution(Context.BeforeExecution context, ExecutionAttributes executionAttributes) {
IdentityProviders providers = executionAttributes.getAttribute(SdkInternalExecutionAttribute.IDENTITY_PROVIDERS);
IdentityProvider<AwsCredentialsIdentity> awsCredentialsIdentityIdentityProvider =
providers.identityProvider(AwsCredentialsIdentity.class);

IdentityProvider<AwsCredentialsIdentity> credentialsProvider = context.request()
.overrideConfiguration()
.filter(c -> c instanceof AwsRequestOverrideConfiguration)
.map(c -> (AwsRequestOverrideConfiguration) c)
.flatMap(AwsRequestOverrideConfiguration::credentialsIdentityProvider)
.map(p -> (IdentityProvider<AwsCredentialsIdentity>) p)
.orElseGet(() -> {
IdentityProviders providers = executionAttributes.getAttribute(SdkInternalExecutionAttribute.IDENTITY_PROVIDERS);
return providers.identityProvider(AwsCredentialsIdentity.class);
});

String operationName = executionAttributes.getAttribute(SdkExecutionAttribute.OPERATION_NAME);
if (operationName.equalsIgnoreCase("createsession")) {
sessionRequests++;
sessionCredentialsProvider.add(awsCredentialsIdentityIdentityProvider);
sessionCredentialsProvider.add(credentialsProvider);
} else {
apiCredentialsProvider.add(awsCredentialsIdentityIdentityProvider);
apiCredentialsProvider.add(credentialsProvider);
}
}

@Override
public void beforeMarshalling(Context.BeforeMarshalling context, ExecutionAttributes executionAttributes) {
public void beforeTransmission(Context.BeforeTransmission context, ExecutionAttributes executionAttributes) {
SelectedAuthScheme<?> attribute = executionAttributes.getAttribute(SdkInternalExecutionAttribute.SELECTED_AUTH_SCHEME);
CompletableFuture<?> identity = attribute.identity();

Expand All @@ -311,7 +320,11 @@ private static final class PathStyleEnforcingInterceptor implements ExecutionInt
public SdkHttpRequest modifyHttpRequest(Context.ModifyHttpRequest context, ExecutionAttributes executionAttributes) {
SdkHttpRequest sdkHttpRequest = context.httpRequest();
String host = sdkHttpRequest.host();
String bucket = host.substring(0, host.indexOf(".localhost"));
int idx = host.indexOf(".localhost");
if (idx < 0) {
return sdkHttpRequest;
}
String bucket = host.substring(0, idx);

return sdkHttpRequest.toBuilder().host("localhost")
.encodedPath(SdkHttpUtils.appendUri(bucket, sdkHttpRequest.encodedPath()))
Expand Down
Loading
Loading