fix(netty-4.1): Treat Netty native client aborts as non-error responses by ygree · Pull Request #12295 · DataDog/dd-trace-java

ygree · 2026-08-25T21:28:33Z

What Does This Do

Solves #12197, a regression introduced by #11941 and released in 1.65.0.

Updates Netty 4.1 broken-pipe handling so Netty native transport write failures such as:

writevAddresses(..) failed with error(-32): Broken pipe
writevAddresses(..) failed: Connection reset by peer
syscall:writev(..) failed: Broken pipe

are treated as client-aborted responses instead of server errors.

This PR:

Handles Netty native client aborts in the Netty 4.1 server decorator, scoped to io.netty.channel.unix.Errors$NativeIoException.
Matches known Netty native write failure prefixes with Broken pipe and Connection reset by peer suffixes, while preserving the existing shared exact Broken pipe behavior.
Records native exception type/message metadata for expected client aborts without marking the server span as errored or adding an error stack trace.
Stops Netty 4.1 response tracing from replacing the application response status with a synthetic 500 when the terminal write promise fails.
Adds regression coverage for native epoll/kqueue client aborts depending on host platform.
Updates terminal async write-failure coverage and Netty HTTP/1.1 pipelining trace assertions.

Motivation

#11941 correctly moved Netty server span completion to terminal response write completion so asynchronous write failures can be observed.

However, cancelled client responses can surface through Netty native transport as NativeIoExceptions whose messages include syscall context before Broken pipe or Connection reset by peer. The existing shared suppression only handled the exact Broken pipe case, so expected client disconnects from Netty native transport started appearing as errored server spans with HTTP 500 status in Datadog.

Additional Notes

Cherry-pick to 1.65.x once merged.

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors
Once approved, use merge queue to merge the PR

Jira ticket: APMS-20399

datadog-datadog-prod-us1-2 · 2026-08-25T21:40:34Z

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 69.74% (+10.94%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: f51a702 | Docs | View more details | Give us feedback!}

dd-octo-sts · 2026-08-25T21:50:50Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.78 s	14.70 s	[-0.3%; +1.4%] (no difference)
startup:insecure-bank:tracing:Agent	13.70 s	13.64 s	[-0.4%; +1.2%] (no difference)
startup:petclinic:appsec:Agent	17.53 s	17.25 s	[+0.7%; +2.6%] (maybe worse)
startup:petclinic:iast:Agent	17.51 s	17.62 s	[-1.5%; +0.2%] (no difference)
startup:petclinic:profiling:Agent	17.58 s	17.27 s	[+0.6%; +3.1%] (maybe worse)
startup:petclinic:sca:Agent	17.44 s	17.48 s	[-1.1%; +0.7%] (no difference)
startup:petclinic:tracing:Agent	16.64 s	16.22 s	[-1.5%; +6.7%] (no difference)

Commit: f51a702c · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

…ror(-32): Broken pipe` resulting in an error span with 500 status

ygree · 2026-08-26T00:03:31Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f3e5c74504

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ygree · 2026-08-26T18:08:06Z

@codex review

chatgpt-codex-connector · 2026-08-26T18:11:26Z

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: 4b0c6577a3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Assert and drain expected traces in AppSec pipelining scenarios so spans are consumed by the tests that create them. The failure appeared now because this branch adds native Netty latest-dep dependencies/tests, changing the `latestDepTest` classpath and execution order enough for previously unasserted AppSec traces to be written after the next test’s writer reset. Add method-aware server trace matching for the HEAD response case.

from DDSpanTest.java

Replace brittle exact-message matching with a scoped matcher for Netty NativeIoException writev failures ending in known client-abort errno messages. This covers another observed Linux ECONNRESET format: writevAddresses(..) failed with error(-104): Connection reset by peer while still limiting the non-error treatment to native Netty write-side client aborts, such as Broken pipe and Connection reset by peer, instead of broadly suppressing unrelated native I/O failures.

datadog-datadog-prod-us1-2

More details

The change limits special handling to Netty native write failures with known client-abort messages. Other write failures still mark the span as an error, and the application response status stays unchanged.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit f51a702 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

amarziali

looks good

ygree · 2026-08-27T15:41:01Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-08-27T15:41:08Z

View all feedbacks in Devflow UI.

2026-08-27 15:41:08 UTC ℹ️ Start processing command /merge

2026-08-27 15:41:20 UTC ℹ️ MergeQueue: Pull request is not mergeable yet

It will be processed automatically as soon as GitHub reports it as mergeable. View in MergeQueue UI.

Run /code blockers to see what is blocking it.
Run /remove to cancel it.

2026-08-27 18:45:05 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in master is approximately 2h (p90).

2026-08-27 20:45:44 UTC ❌ MergeQueue: The build pipeline has timeout

The merge request has been interrupted because the build 6264861321344271299 took longer than expected. The current limit for the base branch 'master' is 120 minutes.

ygree · 2026-08-27T21:40:01Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-08-27T21:40:06Z

View all feedbacks in Devflow UI.

2026-08-27 21:40:05 UTC ℹ️ Start processing command /merge

2026-08-27 21:40:10 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 2h (p90).

2026-08-27 22:27:01 UTC ℹ️ MergeQueue: This merge request was merged

ygree self-assigned this Aug 25, 2026

ygree added type: bug fix Bug fix inst: netty Netty instrumentation labels Aug 25, 2026

fix(netty-4.1): Reproduce Netty's `writevAddresses(..) failed with er…

15f8f0a

…ror(-32): Broken pipe` resulting in an error span with 500 status

ygree force-pushed the ygree/treat-netty-broken-pipe-as-no-err branch from b798fda to b7970a5 Compare August 25, 2026 22:26

ygree changed the title ~~fix(netty-4.1): Reproduce Netty's writevAddresses(..) failed with error(-32): Broken pipe resulting in an error span with 500 status~~ fix(netty-4.1): Stop treating cancelled Netty responses as errors Aug 25, 2026

ygree force-pushed the ygree/treat-netty-broken-pipe-as-no-err branch from b7970a5 to f3e5c74 Compare August 25, 2026 23:37

ygree changed the title ~~fix(netty-4.1): Stop treating cancelled Netty responses as errors~~ fix(netty-4.1): Netty native broken-pipe failures being marked as server errors Aug 25, 2026

ygree added the tag: ai generated Largely based on code generated by an AI or LLM label Aug 25, 2026

chatgpt-codex-connector Bot reviewed Aug 26, 2026

View reviewed changes

Comment thread dd-trace-core/src/main/java/datadog/trace/core/DDSpan.java Outdated

ygree force-pushed the ygree/treat-netty-broken-pipe-as-no-err branch from f3e5c74 to 1a87253 Compare August 26, 2026 01:05

fix(netty-4.1): Stop treating cancelled Netty responses as errors

d8c6f8f

ygree force-pushed the ygree/treat-netty-broken-pipe-as-no-err branch from 1a87253 to d8c6f8f Compare August 26, 2026 01:42

amarziali reviewed Aug 26, 2026

View reviewed changes

Comment thread dd-trace-core/src/main/java/datadog/trace/core/DDSpan.java Outdated

amarziali reviewed Aug 26, 2026

View reviewed changes

Comment thread dd-trace-core/src/main/java/datadog/trace/core/DDSpan.java Outdated

ValentinZakharov reviewed Aug 26, 2026

View reviewed changes

Comment thread dd-trace-core/src/main/java/datadog/trace/core/DDSpan.java Outdated

ygree added 2 commits August 26, 2026 10:54

Move client abort detection into the Netty-specific decorator

925b725

Merge branch 'master' into ygree/treat-netty-broken-pipe-as-no-err

4b0c657

ygree changed the title ~~fix(netty-4.1): Netty native broken-pipe failures being marked as server errors~~ fix(netty-4.1): Treat Netty native client aborts as non-error responses Aug 26, 2026

amarziali reviewed Aug 26, 2026

View reviewed changes

Comment thread dd-trace-core/src/test/java/datadog/trace/core/DDSpanTest.java Outdated

ygree added 4 commits August 26, 2026 12:35

Removed the two stale DDSpan.addThrowable null-message hardening tests

a745262

from DDSpanTest.java

Merge branch 'master' into ygree/treat-netty-broken-pipe-as-no-err

f51a702

ygree marked this pull request as ready for review August 26, 2026 21:00

ygree requested review from a team as code owners August 26, 2026 21:00

ygree requested review from AlexeyKuznetsov-DD, ValentinZakharov and amarziali and removed request for a team August 26, 2026 21:00

datadog-datadog-prod-us1-2 Bot reviewed Aug 26, 2026

View reviewed changes

ValentinZakharov approved these changes Aug 26, 2026

View reviewed changes

amarziali approved these changes Aug 27, 2026

View reviewed changes

ygree mentioned this pull request Aug 27, 2026

Cancelled Netty responses are being treated as errors since 1.65.0 #12197

Open

gh-worker-dd-mergequeue-cf854d Bot merged commit 042bde8 into master Aug 27, 2026
615 of 618 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the ygree/treat-netty-broken-pipe-as-no-err branch August 27, 2026 22:27

github-actions Bot added this to the 1.66.0 milestone Aug 27, 2026

Uh oh!

Conversation

ygree commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-sts Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

ygree commented Aug 26, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ygree commented Aug 26, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 26, 2026

Uh oh!

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

amarziali left a comment

Choose a reason for hiding this comment

Uh oh!

ygree commented Aug 27, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ygree commented Aug 27, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ygree commented Aug 25, 2026 •

edited

Loading

datadog-datadog-prod-us1-2 Bot commented Aug 25, 2026 •

edited

Loading

dd-octo-sts Bot commented Aug 25, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Aug 27, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Aug 27, 2026 •

edited

Loading