Skip to content

Per-component / tag cardinality limits in client-side stats #11387

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
dougqh wants to merge 59 commits into
base: dougqh/lazy-error-latencies
Choose a base branch
from
Draft

Per-component / tag cardinality limits in client-side stats #11387

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
129ebd1
Cap per-field metric tag cardinality via Property/TagCardinalityHandler
dougqh May 15, 2026
8aab88d
Canonicalize SpanSnapshot before hashing so blocked values collapse
dougqh May 15, 2026
9b70705
Defer peer-tag pair construction; capture values + canonicalize via s…
dougqh May 15, 2026
ceec2af
Rename ConflatingMetricsAggregator to ClientStatsAggregator
dougqh May 15, 2026
dd372e7
Cleanups: fix previousCounts size, drop dead code
dougqh May 15, 2026
3a1d67e
Hoist peer-tag schema sync to once per trace
dougqh May 15, 2026
fb32366
Use cached span.kind ordinal in metrics producer; drop tag-map lookup
dougqh May 15, 2026
1221b2b
Add client metrics pipeline design doc
dougqh May 15, 2026
e72fd01
Add DDAgentFeaturesDiscovery.peerTagsRevision()
dougqh May 19, 2026
dce4b2c
Move peer-tag schema cache from PeerTagSchema statics to ClientStatsA…
dougqh May 19, 2026
8f4c6f4
Merge dougqh/optimize-metric-key into dougqh/control-tag-cardinality
dougqh May 19, 2026
5a52622
Fold AggregateMetric into AggregateEntry
dougqh May 19, 2026
ece78c9
Remove accidentally-staged .claude/worktrees entries
dougqh May 19, 2026
b6c4f5f
Address review on AggregateEntry nullables + PeerTagSchema revision
dougqh May 20, 2026
14f7f58
Consolidate cardinality-handler reset behind one entry point
dougqh May 20, 2026
b953b3a
Parameterize PropertyCardinalityHandler on T extends CharSequence
dougqh May 20, 2026
40e8cbd
Add long-lived LRU cache to PropertyCardinalityHandler
dougqh May 20, 2026
d88a863
Centralize per-field cardinality limits in MetricCardinalityLimits
dougqh May 20, 2026
d01036f
Reimplement cardinality handlers as open-addressed flat arrays
dougqh May 20, 2026
7b6c5f1
Drop parallel keys array in PropertyCardinalityHandler
dougqh May 20, 2026
10ca111
Drop type parameter from PropertyCardinalityHandler
dougqh May 20, 2026
4610078
Guard cardinality-handler ctor against pathological inputs
dougqh May 20, 2026
713aa34
Make EMPTY the universal absent sentinel for AggregateEntry UTF8 fields
dougqh May 20, 2026
617bc51
Use EMPTY consistently for absent values in peer-tag canonicalization
dougqh May 20, 2026
e1dec83
Tighten handler visibility + add tests for EMPTY-on-null contract
dougqh May 20, 2026
2336bb5
Notify on peer-tag cardinality blocks
dougqh May 20, 2026
5b6c5aa
Address PR #11387 review: dual-role docs, rename, @Nullable, consumer…
dougqh May 20, 2026
3c3c8b1
Lock in cardinality-handler prior-cycle UTF8 reuse with explicit tests
dougqh May 20, 2026
d61a3c6
Merge branch 'dougqh/optimize-metric-key' into dougqh/control-tag-car…
dougqh May 20, 2026
1df8bd2
Merge branch 'dougqh/optimize-metric-key' into dougqh/control-tag-car…
dougqh May 20, 2026
57cb018
Merge branch 'dougqh/optimize-metric-key' into dougqh/control-tag-car…
dougqh May 21, 2026
517ef75
Merge branch 'dougqh/optimize-metric-key' into dougqh/control-tag-car…
dougqh May 21, 2026
9bbe2d0
Rename bootstrap test to ClientStatsAggregator + adapt PeerTagSchemaTest
dougqh May 21, 2026
0c50037
Drop unused Tags imports flagged by codenarc
dougqh May 21, 2026
078382f
Drop unused SpanKindFilter imports flagged by codenarc
dougqh May 21, 2026
7455fcc
Merge branch 'dougqh/optimize-metric-key' into dougqh/control-tag-car…
dougqh May 21, 2026
4171d15
Sync client_metrics_design doc with reconcile-on-aggregator-thread
dougqh May 21, 2026
3fb86d3
Spread input hash before masking in cardinality-handler probes
dougqh May 22, 2026
e5cfb54
Apply Spotless Javadoc reflows on metrics files
dougqh May 22, 2026
b382df5
Merge remote-tracking branch 'origin/dougqh/optimize-metric-key' into…
dougqh May 22, 2026
e4ed029
Reflow reconcilePeerTagSchema Javadoc after merge
dougqh May 26, 2026
620241e
Merge remote-tracking branch 'origin/dougqh/optimize-metric-key' into…
dougqh May 26, 2026
a6b7854
Merge remote-tracking branch 'origin/dougqh/optimize-metric-key' into…
dougqh May 26, 2026
3d8cccc
Merge remote-tracking branch 'origin/dougqh/optimize-metric-key' into…
dougqh May 26, 2026
ad01d7d
Merge remote-tracking branch 'origin/dougqh/optimize-metric-key' into…
dougqh May 26, 2026
60edaa4
Merge remote-tracking branch 'origin/dougqh/optimize-metric-key' into…
dougqh May 26, 2026
6b10345
Move AggregateEntry equality contract to test-only helper
dougqh May 26, 2026
2c3730e
Merge remote-tracking branch 'origin/dougqh/optimize-metric-key' into…
dougqh May 26, 2026
49124e5
Merge remote-tracking branch 'origin/dougqh/optimize-metric-key' into…
dougqh May 26, 2026
db08d58
Tighten client-stats cardinality plumbing
dougqh May 27, 2026
553bfbd
Merge branch 'dougqh/optimize-metric-key' into dougqh/control-tag-car…
dougqh May 27, 2026
5e72808
Encapsulate EMPTY-as-absent sentinel + align hashOf field order with …
dougqh May 27, 2026
703c9e1
Preserve warm TagCardinalityHandlers across peer-tag schema rebuilds
dougqh May 27, 2026
92a9af3
Revert "Preserve warm TagCardinalityHandlers across peer-tag schema r…
dougqh May 27, 2026
3f4c1c6
Add trace.stats.cardinality.limits.enabled flag (default off)
dougqh May 27, 2026
6dc6701
Doc the regime shift after cardinality-limits flag landed
dougqh May 27, 2026
1469811
Merge branch 'dougqh/optimize-metric-key' into dougqh/control-tag-car…
dougqh May 27, 2026
c48399d
Merge branch 'dougqh/optimize-metric-key' into dougqh/control-tag-car…
dougqh May 27, 2026
690b796
Merge branch 'dougqh/lazy-error-latencies' into dougqh/control-tag-ca…
dougqh May 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion communication/src/main/java/datadog/communication/ddagent/DDAgentFeaturesDiscovery.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,8 @@ protected long getFeaturesDiscoveryMinDelayMillis() {

private synchronized void discoverIfOutdated(final long maxElapsedMs) {
final long now = System.currentTimeMillis();
final long elapsed = now - discoveryState.lastTimeDiscovered;
Copy link
Copy Markdown
Contributor Author

@dougqh dougqh May 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude, this change is unnecessary

final State previous = discoveryState;
final long elapsed = now - previous.lastTimeDiscovered;
if (elapsed > maxElapsedMs) {
final State newState = new State();
doDiscovery(newState);
Expand Down
2 changes: 2 additions & 0 deletions dd-trace-api/src/main/java/datadog/trace/api/config/GeneralConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,8 @@ public final class GeneralConfig {
public static final String TRACER_METRICS_MAX_PENDING = "trace.tracer.metrics.max.pending";
public static final String TRACER_METRICS_IGNORED_RESOURCES =
"trace.tracer.metrics.ignored.resources";
public static final String TRACE_STATS_CARDINALITY_LIMITS_ENABLED =
"trace.stats.cardinality.limits.enabled";

public static final String AZURE_APP_SERVICES = "azure.app.services";
public static final String INTERNAL_EXIT_ON_FAILURE = "trace.internal.exit.on.failure";
Expand Down
8 changes: 4 additions & 4 deletions dd-trace-core/src/jmh/java/datadog/trace/common/metrics/AdversarialMetricsBenchmark.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@
@Fork(value = 1)
public class AdversarialMetricsBenchmark {

private ConflatingMetricsAggregator aggregator;
private ClientStatsAggregator aggregator;
private CountingHealthMetrics health;

@State(Scope.Thread)
Expand All @@ -72,13 +72,13 @@ public static class ThreadState {
public void setup() {
this.health = new CountingHealthMetrics();
this.aggregator =
new ConflatingMetricsAggregator(
new ClientStatsAggregator(
new WellKnownTags("", "", "", "", "", ""),
Collections.emptySet(),
new ConflatingMetricsAggregatorBenchmark.FixedAgentFeaturesDiscovery(
new ClientStatsAggregatorBenchmark.FixedAgentFeaturesDiscovery(
Collections.singleton("peer.hostname"), Collections.emptySet()),
this.health,
new ConflatingMetricsAggregatorBenchmark.NullSink(),
new ClientStatsAggregatorBenchmark.NullSink(),
2048,
2048,
false);
Expand Down
6 changes: 3 additions & 3 deletions ...ConflatingMetricsAggregatorBenchmark.java → ...trics/ClientStatsAggregatorBenchmark.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,12 @@
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(MICROSECONDS)
@Fork(value = 1)
public class ConflatingMetricsAggregatorBenchmark {
public class ClientStatsAggregatorBenchmark {
private final DDAgentFeaturesDiscovery featuresDiscovery =
new FixedAgentFeaturesDiscovery(
Collections.singleton("peer.hostname"), Collections.emptySet());
private final ConflatingMetricsAggregator aggregator =
new ConflatingMetricsAggregator(
private final ClientStatsAggregator aggregator =
new ClientStatsAggregator(
new WellKnownTags("", "", "", "", "", ""),
Collections.emptySet(),
featuresDiscovery,
Expand Down
14 changes: 7 additions & 7 deletions ...tingMetricsAggregatorDDSpanBenchmark.java → ...ClientStatsAggregatorDDSpanBenchmark.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@
import org.openjdk.jmh.infra.Blackhole;

/**
* Parallels {@link ConflatingMetricsAggregatorBenchmark} but uses real {@link DDSpan} instances
* instead of the lightweight {@code SimpleSpan} mock, so the JIT exercises the production {@link
* Parallels {@link ClientStatsAggregatorBenchmark} but uses real {@link DDSpan} instances instead
* of the lightweight {@code SimpleSpan} mock, so the JIT exercises the production {@link
* CoreSpan#isKind} path (cached span.kind ordinal + bit-test) rather than the groovy mock's
* dispatch.
*
Expand All @@ -50,21 +50,21 @@
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(MICROSECONDS)
@Fork(value = 1)
public class ConflatingMetricsAggregatorDDSpanBenchmark {
public class ClientStatsAggregatorDDSpanBenchmark {

private static final CoreTracer TRACER =
CoreTracer.builder().writer(new NoopWriter()).strictTraceWrites(false).build();

private final DDAgentFeaturesDiscovery featuresDiscovery =
new ConflatingMetricsAggregatorBenchmark.FixedAgentFeaturesDiscovery(
new ClientStatsAggregatorBenchmark.FixedAgentFeaturesDiscovery(
Collections.singleton("peer.hostname"), Collections.emptySet());
private final ConflatingMetricsAggregator aggregator =
new ConflatingMetricsAggregator(
private final ClientStatsAggregator aggregator =
new ClientStatsAggregator(
new WellKnownTags("", "", "", "", "", ""),
Collections.emptySet(),
featuresDiscovery,
HealthMetrics.NO_OP,
new ConflatingMetricsAggregatorBenchmark.NullSink(),
new ClientStatsAggregatorBenchmark.NullSink(),
2048,
2048,
false);
Expand Down
8 changes: 4 additions & 4 deletions ...e-core/src/jmh/java/datadog/trace/common/metrics/HighCardinalityPeerMetricsBenchmark.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@
@Fork(value = 1)
public class HighCardinalityPeerMetricsBenchmark {

private ConflatingMetricsAggregator aggregator;
private ClientStatsAggregator aggregator;
private CountingHealthMetrics health;

@State(Scope.Thread)
Expand All @@ -62,13 +62,13 @@ public static class ThreadState {
public void setup() {
this.health = new CountingHealthMetrics();
this.aggregator =
new ConflatingMetricsAggregator(
new ClientStatsAggregator(
new WellKnownTags("", "", "", "", "", ""),
Collections.emptySet(),
new ConflatingMetricsAggregatorBenchmark.FixedAgentFeaturesDiscovery(
new ClientStatsAggregatorBenchmark.FixedAgentFeaturesDiscovery(
Collections.singleton("peer.hostname"), Collections.emptySet()),
this.health,
new ConflatingMetricsAggregatorBenchmark.NullSink(),
new ClientStatsAggregatorBenchmark.NullSink(),
2048,
2048,
false);
Expand Down
8 changes: 4 additions & 4 deletions ...re/src/jmh/java/datadog/trace/common/metrics/HighCardinalityResourceMetricsBenchmark.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@
@Fork(value = 1)
public class HighCardinalityResourceMetricsBenchmark {

private ConflatingMetricsAggregator aggregator;
private ClientStatsAggregator aggregator;
private CountingHealthMetrics health;

@State(Scope.Thread)
Expand All @@ -58,13 +58,13 @@ public static class ThreadState {
public void setup() {
this.health = new CountingHealthMetrics();
this.aggregator =
new ConflatingMetricsAggregator(
new ClientStatsAggregator(
new WellKnownTags("", "", "", "", "", ""),
Collections.emptySet(),
new ConflatingMetricsAggregatorBenchmark.FixedAgentFeaturesDiscovery(
new ClientStatsAggregatorBenchmark.FixedAgentFeaturesDiscovery(
Collections.singleton("peer.hostname"), Collections.emptySet()),
this.health,
new ConflatingMetricsAggregatorBenchmark.NullSink(),
new ClientStatsAggregatorBenchmark.NullSink(),
2048,
2048,
false);
Expand Down
Loading