NoSuchFieldError on Java record when Datadog Java tracer integrations are enabled (Java 21 & 25, Linux aarch64)

[cportcvent]

Tracer Version(s)

1.57.0, 1.56.3, 1.51.1

Java Version(s)

25.0.0, 21.0.1

JVM Vendor

Amazon Corretto

Bug Report

Overview of the problem

NoSuchFieldError exceptions occur on a Java record type when the Datadog Java tracer is attached with integrations enabled. The failures occur in a JVM‑based HTTP service where a record is constructed via a static parse factory method from HTTP query parameters.

The same binary, under the same test load, behaves correctly when:

• The Datadog Java tracer is disabled, or
• The tracer is enabled but all integrations are disabled via -Ddd.integrations.enabled=false, or
• The problematic record is refactored into a plain Java class.

The NoSuchFieldError complains about fields or static constants that are clearly present in the compiled record class (verified via decompilation), and it only manifests when the Datadog Java tracer integrations are active. This strongly suggests an issue with bytecode instrumentation of Java records rather than a pure application or JVM bug.

Environment

Runtime / OS

• Java versions:

  • Java 21 (LTS)
  • Java 25 (early LTS)
    The issue has been observed under both versions in CI deployments.

• OS / architecture:

  • Linux, aarch64 (Arm64), running on cloud provider Arm‑based instances.

Datadog Java tracer versions

We have observed the issue with all of the following versions in production‑like CI environments:

• 1.51.1
• 1.56.3
• 1.57.0

Across these versions, the behavior is consistent:

• With integrations enabled, the record ultimately fails with NoSuchFieldError.
• With integrations globally disabled (-Ddd.integrations.enabled=false), the same scenario passes consistently.

Failure does not occur for every HTTP request that references the record class, but occurs for a majority of such requests (at least 75% of observed cases).

Observed behavior details

When the Datadog Java tracer is attached with integrations enabled on Linux aarch64 and Java 21 or 25, the service occasionally fails requests that exercise this record type with java.lang.NoSuchFieldError.

The errors report that:

• A static Pattern field on the record is missing, and/or
• The record's canonical field (such as snackType) is missing.

These failures occur even though the compiled class file clearly defines these fields and the same binary passes unit tests and local runs.

The behavior is intermittent and tied to the presence of the tracer and its integrations:

• The same request sometimes returns HTTP 200 and sometimes fails with NoSuchFieldError on the same process and deployment.
• Disabling all agents (eg. Datadog and JaCoCo) eliminates the failures.
• Disabling other agents while leaving Datadog enabled, does not eliminate the failures.
• Disabling Datadog integrations globally with -Ddd.integrations.enabled=false eliminates the failures, even though the tracer remains attached.
• Converting the record to a regular class eliminates the failures regardless of Datadog and integrations enabled.

What we tested (matrix-style summary)

• ❌ Record + Datadog tracer + integrations ON → intermittent NoSuchFieldError.
• ❌ Record + Datadog tracer ON + JaCoCo and other agents OFF → NoSuchFieldError persists.
• ✅ Record + Datadog tracer + -Ddd.integrations.enabled=false → NoSuchFieldError never observed.
• ✅ Record + Datadog tracer + -Ddd.integrations.enabled=false + full integration list below enabled → NoSuchFieldError never observed.
• ❌ Record + Datadog tracer + -Ddd.integrations.enabled=true + full integration list below disabled → NoSuchFieldError persists.
• ✅ Class (instead of record) + Datadog tracer + integrations ON → NoSuchFieldError never observed.

Full integrations list reference

## Low risk category
-Ddd.integration.aws-sdk.enabled=true
-Ddd.integration.couchbase.enabled=true
-Ddd.integration.httpurlconnection.enabled=true
-Ddd.integration.httpclient.enabled=true
-Ddd.integration.apache-httpclient.enabled=true
-Ddd.integration.apache-http-client.enabled=true

## Highly scrutinized category
-Ddd.integration.jetty.enabled=true
-Ddd.integration.jax-rs.enabled=true
-Ddd.integration.jaxrs.enabled=true
-Ddd.integration.jax-rs-annotations.enabled=true
-Ddd.integration.jax-rs-filter.enabled=true
-Ddd.integration.servlet.enabled=true
-Ddd.integration.servlet-2.enabled=true
-Ddd.integration.servlet-3.enabled=true

Questions for maintainers

1. Are there any known issues with Datadog Java tracer instrumentation of Java records (especially on Linux aarch64 and recent Java versions) that could produce NoSuchFieldError on record fields or static constants?
2. Are there other specific integrations you would recommend disabling individually (via -Ddd.integration.<name>.enabled=false) that are known to rewrite application classes in ways that might affect records?
3. Can you offer further advice and guidance to troubleshoot this issue?

Expected Behavior

With the Datadog Java tracer attached and integrations enabled:

• Java record classes should behave identically to their compiled bytecode.
• Static fields and canonical record fields should remain present and accessible at runtime.
• Enabling integrations should not introduce NoSuchFieldError on legitimate fields, regardless of concurrency or load.

Reproduction Code

At a high level, the problematic pattern looks like this in terms of structure:

public enum SnackType {
  PIZZA,
  // ...
}

public record MySnackFilter(SnackType snackType) {
  private static final Pattern MATCH_THINGS = Pattern.compile("\"(.*)\"");

  public static MySnackFilter parse(String filterExpression) {
    // Parses a filter expression and resolves snack type details
    return new MySnackFilter(resolvedSnackType);
  }
}

The record is used in an HTTP handler that runs on every request to parse a query parameter and construct a MySnackFilter instance. When the bug manifests, we see NoSuchFieldError reported for:

• The static Pattern field (for example, MATCH_THINGS).
• The canonical record field (for example, snackType).

The same class, when converted to an ordinary Java class with equivalent fields and logic, does not reproduce under the same Datadog configuration and test load.

Steps to reproduce

1. Implement a small HTTP endpoint (JAX-RS, eg Jersey) that:

   • Accepts a GET request with a query parameter such as filter=snackType eq "PIZZA" (or filter=snackType+eq+%27PIZZA%27).
   • Uses a Java record similar to MySnackFilter with a static parse(String filter) method that:
     • Resolves an enum value.
     • Optionally uses a static Pattern field to match filter statements. This step is optional because we can reproduce with the Pattern field removed.
     • Constructs and returns a record instance.

2. Deploy this service on Linux aarch64 with Java 21 or Java 25. Attach the Datadog Java tracer using one of the versions listed in the Environment section, with integrations enabled.

3. Drive concurrent traffic through the endpoint using an integration test suite or a load tool (for example repeated HTTP calls with curl, gatling, k6, Karate, etc).

4. Under these conditions, observe that:

   • Some requests succeed with HTTP 200 and correctly construct the record.
   • Some requests, on the same host, same container image, and same process, fail with HTTP 500 and a java.lang.NoSuchFieldError, declaring the record "does not have member field" for either:
     • The static Pattern field, or
     • The record's canonical field (for example, snackType).

5. Restart the service with the same Datadog Java tracer version but with -Ddd.integrations.enabled=false, and re-run the same tests. Under this configuration, NoSuchFieldError never occurs; all tests and manual calls succeed.

6. Change the record into a plain Java class with equivalent fields and a similar static factory method, then re-enable Datadog integrations. Under the same CI tests and load, NoSuchFieldError never occurs; all tests and manual calls succeed.

[amarziali]

Hi @cportcvent thanks a lot for taking the time to report this issue and for the detailed context you shared.

We tried to reproduce the problem using a minimal Jersey endpoint that performs the same kind of record construction/parsing pattern you described, but we haven’t been able to trigger the NoSuchFieldError on our side.

At the moment we don’t have any known issues in dd-trace-java related to that, and we don’t instrument those record fields directly. That said, this kind of failure could still be caused by an interaction with other libraries or agents, or by bytecode transformations happening elsewhere in the stack.

To help us narrow it down, would you mind sharing debug logs captured when the error occurs via the Zendesk ticket you already opened? If you haven’t enabled debug logging yet, you can do so by setting DD_TRACE_DEBUG=true (or -Ddd.trace.debug=true) and then reproducing the issue. The resulting logs should give us more insight, including details about the dependencies and integrations being used.

Also, if you’re able to share a minimal reproducer (even if it still requires a subset of your dependencies to trigger the issue), that would be the best option and would greatly help us investigate further.

Best

Andrea

[jpbempel]

This problem was identified and related to a JVM bug: https://bugs.openjdk.org/browse/JDK-8376185

[dan-lind]

@jpbempel Sorry for pinging in a closed issue. We faced this issue earlier this week, was wondering if you could explain in what scenarios we would hit this issue?

We have ~250 records containing TYPE_USE annotations across 13 repos, and this week we were hit by this issue for a single one of those records. Never had any issues before. The specific service that was hit used Java 25 and dd tracer 1.58.2.

Don't know if it's relevant, but it turned at that the specific record that broke had the most TYPE_USE annotations (@NonNull) of any record across all of our repos. The fix was to remove the NonNull and just do the validation in the constructor instead.

Before we run of and rewrite 250 records, is there way to narrow down when this triggers?

[jpbempel]

@jpbempel Sorry for pinging in a closed issue. We faced this issue earlier this week, was wondering if you could explain in what scenarios we would hit this issue?

We have ~250 records containing TYPE_USE annotations across 13 repos, and this week we were hit by this issue for a single one of those records. Never had any issues before. The specific service that was hit used Java 25 and dd tracer 1.58.2.

Don't know if it's relevant, but it turned at that the specific record that broke had the most TYPE_USE annotations (@nonnull) of any record across all of our repos. The fix was to remove the NonNull and just do the validation in the constructor instead.

Before we run of and rewrite 250 records, is there way to narrow down when this triggers?

Hey @dan-lind, sorry that you are hit by that. Retransformation of classes happens either by using Live Debugger/Dynamic Instrumentation (manual process) or with Exception Replay that we start to enable by default. You can opt out by going to the Error Tracking settings page and disabling Exception Replay for applications affected by this JDK bug until the patch is released.