Skip to content

NettyNioAsyncHttpClient: IllegalStateException unexpected message type LastHttpContent state 0 after upgrade to 2.57.8 (AOSS / AwsSdk2Transport) #7501

Open

Description

@elijulian

Describe the bug

After upgrading the AWS SDK for Java v2 BOM from 2.57.6 → 2.57.8, all requests made through NettyNioAsyncHttpClient via OpenSearch Java's AwsSdk2Transport (SigV4 to Amazon OpenSearch Serverless, service aoss) fail immediately with:

java.lang.IllegalStateException: unexpected message type: LastHttpContent$1, state: 0
	at io.netty.handler.codec.http.HttpObjectEncoder.throwUnexpectedMessageTypeEx(HttpObjectEncoder.java:376)
	at io.netty.handler.codec.http.HttpObjectEncoder.encodeNotHttpMessageContentTypes(HttpObjectEncoder.java:275)
	at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:189)
	at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:290)
	...
	at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsHandler.completeBody(HttpStreamsHandler.java:340)
	at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsClientHandler.write(HttpStreamsClientHandler.java:59)
	...
	at org.opensearch.client.transport.aws.AwsSdk2Transport.performRequest(AwsSdk2Transport.java:243)

This is an outbound encode failure (HttpObjectEncoder state 0 receiving LastHttpContent before a request head), not an AOSS HTTP error response.

Expected behavior

Requests succeed as they did on 2.57.6 with the same application code, same AOSS collections, and same VPC endpoint networking.

Current behavior

Every OpenSearch operation we tried fails with the exception above, including:

  • index exists (indices().exists(...))
  • bulk index
  • bulk delete

Observed in both us-east-2 and eu-central-1 immediately after deploying the 2.57.8 BOM bump.

Steps to Reproduce

  1. Use AWS SDK BOM 2.57.8 + software.amazon.awssdk:netty-nio-client
  2. Build an OpenSearch Java client with:
OpenSearchTransport transport = new AwsSdk2Transport(
    NettyNioAsyncHttpClient.create(),
    URI.create(aossCollectionEndpoint).getHost(),
    "aoss",
    Region.of(region),
    AwsSdk2TransportOptions.builder()
        .setMapper(new JacksonJsonpMapper())
        .setCredentials(DefaultCredentialsProvider.builder().build())
        .build()
);
OpenSearchClient client = new OpenSearchClient(transport);
  1. Call any request that goes through AwsSdk2Transport.performRequest (e.g. client.indices().exists(...) or client.bulk(...)) against an AOSS VECTORSEARCH collection over a PrivateLink VPC endpoint.

  2. Observe the LastHttpContent$1, state: 0 encoder exception.

Context / evidence that this is client-side

  • AOSS collections remained ACTIVE
  • AOSS VPC endpoint remained ACTIVE
  • From the workload: DNS resolves to PrivateLink and TCP/443 succeeds
  • CloudWatch showed ingestion successes stop lining up with the client upgrade, with no corresponding AOSS 4xx/5xx flood
  • No AOSS data/network policy changes at onset
  • Zero occurrences of this exception before the 2.57.8 deploy; continuous failures afterward

Regression

Version Result
software.amazon.awssdk:bom:2.57.6 OK
software.amazon.awssdk:bom:2.57.8 Fails as above

Also using org.opensearch.client:opensearch-java:3.10.0 (unchanged across the regression).

Related public reports of the same exception shape

Not AOSS-specific, but the same Netty encoder failure has shown up with AWS SDK Netty async HTTP elsewhere, e.g. metabase/metabase#72532.

Notes

Happy to provide a more complete stack trace or help validate a patch/workaround (for example pinning an older Netty transitive, switching HTTP client implementation, or connection-pool settings) if useful.

Activity

  1. elijulian commented on Nov 1, 2026

    @elijulian
    Author

    Correction / root cause update

    Sorry for the noise — this report misattributed the failure to the AWS SDK BOM bump alone.

    We shipped software.amazon.awssdk:bom 2.57.6 → 2.57.8 in the same release as com.azure:azure-identity 1.18.7 → 1.19.0. Reverting only the AWS BOM left AOSS broken. Holding azure-identity at 1.18.7 and keeping AWS at 2.57.8 restored service.

    What actually broke

    azure-identity:1.19.0 pulls azure-core-http-netty:1.17.x → Netty 4.2.x onto a classpath where NettyNioAsyncHttpClient still expects Netty 4.1.x. That mixed / upgraded Netty line surfaces as:

    IllegalStateException: unexpected message type: LastHttpContent$1, state: 0
    

    in HttpObjectEncoder during outbound encode (same stack as in the original description).

    Evidence

    • With azure-identity:1.19.0: resolved netty-common → 4.2.19.Final (conflict with 4.1.138 from AWS).
    • With azure-identity:1.18.7 + AWS BOM 2.57.8: Netty stays on 4.1.x, and NettyNioAsyncHttpClient can successfully talk to OpenSearch again.

    So 2.57.8 by itself was not the regression. Happy to close this unless you still want it as a “Netty 4.2 coexistence” datapoint.

    Related open tracking

    I did not find an existing open Azure issue specifically for azure-identity 1.19+ / azure-core-http-netty 1.17+ breaking AWS NettyNioAsyncHttpClient / AOSS with LastHttpContent. Closest is the Netty line mismatch thread above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions