Keep JDK Unix socket timeouts from stopping trace delivery

[bric3]

What Does This Do

Keeps trace delivery alive when an OkHttp request over a JDK Unix-domain socket times out.

This pull request makes two related but distinct changes:

1. It reports asynchronous selector closure as an I/O failure instead of letting an unchecked selector exception escape the socket boundary.
2. It serializes the compound socket lifecycle transitions that can race with Okio closing a timed-out connection.

It also adds focused selector tests and a writer-level timeout-then-reconnect test over a real Unix-domain socket.

Motivation

1. Keep selector closure on the I/O failure path

OkHttp creates the Agent connection with UnixDomainSocketFactory. Okio then wraps that socket and uses AsyncTimeout.Watchdog to enforce the read timeout. When a response stalls, the watchdog closes the socket from a different thread to interrupt the blocked read.

Closing TunnelingJdkSocket closes its selector. Depending on where the reader is, JDK NIO can then throw ClosedSelectorException or CancelledKeyException. These are unchecked exceptions, so Okio does not handle them as transport failures. They can escape the send path and stop the single dd-trace-processor thread, after which later traces are no longer sent.

flowchart LR
  subgraph DD["dd-trace-java"]
    worker["TraceProcessingWorker"]
    api["DDAgentApi"]
    udsRead["TunnelingJdkSocket.read()"]
    udsClose["TunnelingJdkSocket.close()"]
    nextPayload["next payload<br/>fresh connection"]
  end

  subgraph OkHttp["OkHttp 3.x"]
    codec["Http1Codec"]
    connection["RealConnection"]
  end

  subgraph Okio["Okio 1.x"]
    source["Okio.source(socket)"]
    watchdog["AsyncTimeout.Watchdog"]
  end

  subgraph JDK["JDK NIO"]
    select["Selector.select()"]
    lifecycleException["ClosedSelectorException<br/>or CancelledKeyException"]
  end

  worker --> api --> codec --> source --> udsRead --> select
  watchdog -->|"read timeout closes socket"| udsClose
  udsClose -->|"closes selector"| select
  select --> lifecycleException
  lifecycleException -->|"translated to SocketException"| udsRead
  udsRead -->|"IOException"| source
  source -->|"failed exchange"| codec
  codec -->|"failed send; worker survives"| worker
  worker --> nextPayload --> connection --> udsRead

Loading

The socket now translates only these selector lifecycle exceptions to SocketException. The timed-out POST is not retried because the Agent may already have received it. Instead, that payload fails normally, the processor keeps running, and the next payload uses a fresh connection.

2. Serialize socket lifecycle transitions

The exception translation makes asynchronous closure survivable, but the socket also needs a coherent lifetime while OkHttp and Okio use it from different threads.

The synchronization is deliberately narrow. It protects compound lifecycle operations rather than normal I/O:

• getInputStream() validates the state and publishes the lazily created selector before close() can snapshot it.
• shutdownInput() and shutdownOutput() keep validation, the channel operation, and the state update atomic with close().
• close() publishes the terminal state and snapshots the selector under the socket monitor, then closes the selector and channel after releasing the monitor so a blocked read can wake without extending the critical section.
• The channel is created with the socket and remains stable for its lifetime. A failed connection closes that channel and leaves the socket terminally closed.

flowchart LR
  subgraph OkHttp["OkHttp 3.x"]
    realConnection["RealConnection<br/>one socket per physical connection"]
    halfClose["connection shutdown"]
  end

  subgraph Okio["Okio 1.x"]
    sourceSetup["Okio.source(socket)"]
    timeoutClose["AsyncTimeout.Watchdog<br/>socket.close()"]
  end

  subgraph DD["dd-trace-java"]
    factory["UnixDomainSocketFactory"]
    socket["TunnelingJdkSocket"]
    inputLifecycle["getInputStream()<br/>validate + publish"]
    halfLifecycle["shutdownInput/Output()<br/>validate + update"]
    closeLifecycle["close()<br/>terminal state + snapshot"]
    monitor["synchronized (this)<br/>short lifecycle transition"]
    resourceClose["close resources<br/>outside monitor"]
  end

  subgraph JDK["JDK NIO"]
    channel["final SocketChannel"]
    selector["lazy Selector"]
  end

  factory --> socket -->|"created together"| channel
  realConnection -->|"connect"| socket
  sourceSetup --> inputLifecycle --> monitor -->|"publish"| selector
  halfClose --> halfLifecycle --> monitor -->|"native half-close"| channel
  timeoutClose --> closeLifecycle --> monitor -->|"snapshot"| resourceClose
  resourceClose -->|"wake blocked read"| selector
  resourceClose --> channel

Loading

Normal bulk reads and writes do not acquire the socket lifecycle monitor. They continue to observe volatile state and operate on the stable channel.

Additional Notes

The writer-level test uses MockWebServer's NO_RESPONSE policy over a JNR Unix-domain server adapter. It verifies that the first request times out, the second request arrives on a new connection, and both callbacks run on the same surviving writer thread.

The focused socket and writer tests pass on JDK 17, 21, and 25.

Contributor Checklist

• Format the title according to the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any other useful labels
• Avoid using close, fix, or any linking keywords when referencing an issue
  Use solves instead, and assign the PR milestone to the issue
• Update the CODEOWNERS file on source file addition, migration, or deletion
• Update public documentation with any new configuration flags or behaviors
• Once approved, use merge queue to merge the PR

Jira ticket: APMS-20292

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.91 s	14.77 s	[-0.2%; +2.0%] (no difference)
startup:insecure-bank:tracing:Agent	13.55 s	13.66 s	[-1.7%; +0.0%] (no difference)
startup:petclinic:appsec:Agent	17.44 s	17.22 s	[+0.5%; +2.1%] (maybe worse)
startup:petclinic:iast:Agent	17.37 s	17.51 s	[-1.6%; -0.0%] (maybe better)
startup:petclinic:profiling:Agent	17.21 s	17.35 s	[-2.3%; +0.6%] (no difference)
startup:petclinic:sca:Agent	17.31 s	16.45 s	[+0.7%; +9.7%] (maybe worse)
startup:petclinic:tracing:Agent	16.56 s	16.70 s	[-1.8%; +0.1%] (no difference)

Commit: 39570c8c · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.