Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions charts/substrate/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,11 @@ See `values.yaml` for the full set; the important keys:
| `atelet.gcpAuthForImagePulls` | `false` | Enable only when using GCP registry auth |
| `credentialProvider.namespacePolicies` | `[]` | Default-deny atespace-to-namespace grants; the chart includes get-only Secret RBAC for the provider |
| `ateApi.extraArgs` | `[]` | Additional command-line arguments appended to the ateapi defaults |
| `otel.endpoint` | `""` | Set to an OTLP endpoint to export traces, metrics and the router access log |
| `otel.endpoint` | `""` | Set to an OTLP endpoint to export traces, metrics, the actor lifecycle events and the router access log |
| `otel.traces.enabled` | `true` | Set to `false` to export no traces from the router; the Go components do not honor this yet |
| `otel.traces.endpoint` | `""` | OTLP endpoint for traces, overriding `otel.endpoint` |
| `otel.traces.samplingRatio` | `0.01` | Fraction of parentless requests that start a trace, applied to the Go components and the router |
| `otel.metrics.enabled` | `true` | Sets the OTLP metrics exporter to `none`; the Go components do not honor this yet |
| `otel.metrics.endpoint` | `""` | OTLP endpoint for metrics, overriding `otel.endpoint` |
| `otel.logs.enabled` | `true` | Set to `false` to export no logs; the router access log is the only OTLP log source today |
| `otel.logs.enabled` | `true` | Set to `false` to export no logs. Gates both OTLP log sources: ateapi's actor lifecycle events and the router access log |
| `otel.logs.endpoint` | `""` | OTLP endpoint for logs, overriding `otel.endpoint` |
8 changes: 8 additions & 0 deletions charts/substrate/templates/_helpers.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,14 @@ Usage:
value: {{ $cfg.endpoint | quote }}
{{- end }}
{{- end }}
{{- if include "substrate.otel.signalEndpoint" (list "logs" .) }}
{{- /* Only logs need turning on: serverboot defaults the component to none, so
an enabled signal exports nothing without this. Traces and metrics always
export, so they need no such branch -- keep this out of the range above.
Gated on an endpoint, since otlp without one retries localhost:4317. */}}
- name: OTEL_LOGS_EXPORTER
value: otlp
{{- end }}
{{- if include "substrate.otel.signalEndpoint" (list "traces" .) }}
- name: OTEL_TRACES_SAMPLER
value: parentbased_traceidratio
Expand Down
3 changes: 2 additions & 1 deletion charts/substrate/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,8 @@ ateApiServerEnvVarsConfigMap: ate-api-server-envvars
# traces.enabled and metrics.enabled set OTEL_<SIGNAL>_EXPORTER=none on the
# Go components, which do not read it yet and keep exporting; the setting
# takes effect on the router's agentgateway only. logs.enabled works in full,
# since the agentgateway access log is the only OTLP log source.
# and gates both OTLP log sources: ateapi's actor lifecycle events and the
# router's agentgateway access log.
otel:
endpoint: ""
traces:
Expand Down
3 changes: 2 additions & 1 deletion docs/dev/best-practices/otel-collector.md
Original file line number Diff line number Diff line change
Expand Up @@ -435,7 +435,8 @@ for a worked example.

**Substrate exports one thing over OTLP: the actor lifecycle events**, from
ateapi, through `serverboot.InitLogging`. They are off unless
`OTEL_LOGS_EXPORTER=otlp` is set, which only the kind overlay does today. See
`OTEL_LOGS_EXPORTER=otlp` is set. The kind overlay sets it, and the Helm chart
sets it from `otel.logs.enabled` once `otel.endpoint` resolves. See
[the same records over OTLP](../../observability.md#the-same-records-over-otlp).

Everything else is stdout. `serverboot.InitLogger` writes structured JSON there,
Expand Down
4 changes: 2 additions & 2 deletions docs/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,7 @@ The counter carries the same reason but no actor identity, so this record is the

#### The same records over OTLP

Both records also go out as OTLP log events, so a collector reads them without knowing substrate's stdout envelope. Set `OTEL_LOGS_EXPORTER=otlp` to turn it on; unset means `none`, which is what every environment but kind uses today. Only ateapi has a LoggerProvider — a worker pod cannot export a log record at all yet, because [the ateom relay](#the-ateom-otlp-relay) carries traces and metrics only.
Both records also go out as OTLP log events, so a collector reads them without knowing substrate's stdout envelope. Set `OTEL_LOGS_EXPORTER=otlp` to turn it on; unset means `none`. The kind overlay sets it, and a chart install gets it from `otel.logs.enabled` once `otel.endpoint` resolves. Only ateapi has a LoggerProvider — a worker pod cannot export a log record at all yet, because [the ateom relay](#the-ateom-otlp-relay) carries traces and metrics only.

Two `event.name` values, which is the OTLP LogRecord's own field rather than an attribute:

Expand Down Expand Up @@ -405,7 +405,7 @@ Telemetry is emitted the same way everywhere; only the backend differs between a
| Path | service → in-cluster `opentelemetry-collector` | service → Google Managed Prometheus (GMP) |
| Metrics | collector Prometheus exporter on `:8889` | Google Cloud Monitoring |
| Traces | Jaeger UI | Google Cloud Trace |
| Logs | pod stdout; ateapi's [actor lifecycle events](#the-same-records-over-otlp) also to the collector's `debug` exporter | pod stdout. No OTLP logs: `OTEL_LOGS_EXPORTER` is unset |
| Logs | pod stdout; ateapi's [actor lifecycle events](#the-same-records-over-otlp) also to the collector's `debug` exporter | pod stdout. No OTLP logs unless `OTEL_LOGS_EXPORTER` is set |
| Dashboards | Not supported | Google Cloud Monitoring (see [Dashboards](#5-dashboards)) |

> In Kind, `ateapi`, `atelet`, `ate-controller`, and `atenet-router` are pointed at the in-cluster collector, and the controller propagates the endpoint to the ateom worker pods it creates, so all component telemetry lands locally.
Expand Down
Loading