Skip to content

feat(otel): add Datadog fan-out collector config + local traces pipeline #374

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
NiteshDhanpal wants to merge 1 commit into
base: main
Choose a base branch
from
+86 −1
Open

feat(otel): add Datadog fan-out collector config + local traces pipeline #374

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions agentex/otel/otel-collector-config.datadog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# OpenTelemetry Collector configuration WITH Datadog fan-out.
#
# Duplicates traces and metrics to BOTH the LGTM path (Prometheus / Tempo) and
# Datadog. This is the "collector sink" phase of the ddtrace -> LGTM migration:
# once the app stops emitting to Datadog directly via ddtrace, the collector
# keeps Datadog fed, so there is no gap in Datadog coverage.
#
# Because a single OTel SDK produces the spans, the SAME trace_id is exported to
# both backends -- one trace, two destinations.
#
# Requirements / usage:
# - Uses the otel/opentelemetry-collector-contrib image (datadog exporter),
# which docker-compose already runs.
# - Set DD_API_KEY (required) and optionally DD_SITE (default datadoghq.com).
# - Optionally set TEMPO_OTLP_ENDPOINT to also fan traces out to Tempo; if
# unset, the Tempo exporter is simply not referenced (see pipelines below).
Comment on lines +15 to +16

@greptile-apps greptile-apps Bot Jul 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 TEMPO_OTLP_ENDPOINT comment implies conditional behaviour that doesn't exist

The requirements header says "Optionally set TEMPO_OTLP_ENDPOINT to also fan traces out to Tempo; if unset, the Tempo exporter is simply not referenced." However, no otlp/tempo exporter is defined in exporters: and no pipeline references it — the env var does nothing right now. An operator who reads this, sets TEMPO_OTLP_ENDPOINT, and deploys will get silent loss of the expected Tempo routing with no error from the collector. The phrase "if unset, the Tempo exporter is simply not referenced" actively suggests conditional logic that doesn't exist yet. Consider rewording to "Tempo fan-out is not yet implemented — see follow-up ticket" to avoid the false impression.

Prompt To Fix With AI 
This is a comment left during a code review.
Path: agentex/otel/otel-collector-config.datadog.yaml
Line: 15-16

Comment:
**TEMPO_OTLP_ENDPOINT comment implies conditional behaviour that doesn't exist**

The requirements header says "Optionally set `TEMPO_OTLP_ENDPOINT` to also fan traces out to Tempo; if unset, the Tempo exporter is simply not referenced." However, no `otlp/tempo` exporter is defined in `exporters:` and no pipeline references it — the env var does nothing right now. An operator who reads this, sets `TEMPO_OTLP_ENDPOINT`, and deploys will get silent loss of the expected Tempo routing with no error from the collector. The phrase "if unset, the Tempo exporter is simply not referenced" actively suggests conditional logic that doesn't exist yet. Consider rewording to "Tempo fan-out is not yet implemented — see follow-up ticket" to avoid the false impression.

How can I resolve this? If you propose a fix, please make it concise.

Fix in Cursor Fix in Claude Code Fix in Codex

# - Enable by mounting THIS file at /etc/otel-collector-config.yaml instead of
# otel-collector-config.yaml (docker-compose volume / --config).
#
# This file is OPT-IN and does not change the default (keyless) local collector.
# It is also the reference pattern for the Helm-managed production collector.
#
# IMPORTANT (migration safety): deploy + verify this Datadog fan-out in an
# environment BEFORE disabling app-side ddtrace there. Never remove the app's
# Datadog source until the collector is confirmed exporting to Datadog, or that
# environment loses traces.

receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318

processors:
batch:
timeout: 10s
send_batch_size: 1024

exporters:
debug:
verbosity: normal
Comment on lines +42 to +43

@greptile-apps greptile-apps Bot Jul 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Debug exporter lacks sampling — will log every span and metric in production

The local config (otel-collector-config.yaml) configures sampling_initial: 5 and sampling_thereafter: 200 on the debug exporter, capping console output at about 1-in-200 records after warmup. The Datadog fan-out config, described as "the reference pattern for the Helm-managed production collector," omits those settings. With verbosity: normal, every span in the traces pipeline and every metric data point in the metrics pipeline will be printed to stdout. In a staging or production deployment this generates significant log volume and cost. Either add the same sampling knobs or remove debug from the Datadog-facing pipelines if console output isn't useful there.

Prompt To Fix With AI 
This is a comment left during a code review.
Path: agentex/otel/otel-collector-config.datadog.yaml
Line: 42-43

Comment:
**Debug exporter lacks sampling — will log every span and metric in production**

The local config (`otel-collector-config.yaml`) configures `sampling_initial: 5` and `sampling_thereafter: 200` on the `debug` exporter, capping console output at about 1-in-200 records after warmup. The Datadog fan-out config, described as "the reference pattern for the Helm-managed production collector," omits those settings. With `verbosity: normal`, every span in the `traces` pipeline and every metric data point in the `metrics` pipeline will be printed to stdout. In a staging or production deployment this generates significant log volume and cost. Either add the same sampling knobs or remove `debug` from the Datadog-facing pipelines if console output isn't useful there.

How can I resolve this? If you propose a fix, please make it concise.

Fix in Cursor Fix in Claude Code Fix in Codex


# LGTM metrics path.
prometheus:
endpoint: 0.0.0.0:8889
namespace: agentex
send_timestamps: true
metric_expiration: 5m

# Datadog fan-out. Traces + metrics are duplicated here in addition to the
# LGTM path, so Datadog is fed by the collector rather than by app ddtrace.
datadog:
api:
key: ${env:DD_API_KEY}
site: ${env:DD_SITE:-datadoghq.com}

extensions:
health_check:
endpoint: 0.0.0.0:13133

service:
extensions: [health_check]
pipelines:
# Traces fan out to Datadog (and console). Add an OTLP/Tempo exporter here
# when a Tempo endpoint is available to also feed the LGTM trace backend.
traces:
receivers: [otlp]
processors: [batch]
exporters: [datadog, debug]
# Metrics fan out to both the LGTM (Prometheus) path and Datadog.
metrics:
receivers: [otlp]
processors: [batch]
exporters: [prometheus, datadog, debug]
11 changes: 10 additions & 1 deletion agentex/otel/otel-collector-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
# OpenTelemetry Collector configuration for local development
# Receives OTLP metrics and exports to console + Prometheus
# Receives OTLP metrics + traces and exports to console + Prometheus.
#
# For the Datadog fan-out variant (exports to Datadog in addition to the LGTM
# path -- used during the ddtrace -> LGTM migration so Datadog keeps receiving
# telemetry once the app stops emitting via ddtrace directly), see
# otel-collector-config.datadog.yaml.

receivers:
otlp:
Expand Down Expand Up @@ -39,3 +44,7 @@ service:
receivers: [otlp]
processors: [batch]
exporters: [debug, prometheus]
traces:
receivers: [otlp]
processors: [batch]
exporters: [debug]
Loading