feat(observability): OpenTelemetry export surface for gateway operators #2507

Problem Statement

OpenShell gives gateway operators three observability surfaces today, and none of them answer "what is my gateway doing right now, and why is it slow or failing?"

• /metrics (Prometheus) exposes gRPC/HTTP request counters and duration histograms, a readiness gauge, and gateway-interceptor counters. It is scrape-only, disabled by default (--metrics-port defaults to 0), and has no Helm scrape wiring.
• OCSF logs (shorthand always on, full JSONL opt-in) describe sandbox security events for SIEM consumption. They are not gateway operational telemetry and carry no timing or causality.
• Anonymous product telemetry (openshell-core::telemetry) forwards aggregate usage to NVIDIA. It is deliberately coarse and is not readable by the operator running the gateway.

What is missing is the connective tissue operators expect from any modern control plane: distributed traces, an OTLP push path for environments that do not scrape, and a turnkey collector wiring so this works without hand-assembling a stack.

Concretely, an operator investigating a slow CreateSandbox today can see that the request took 4.2s from the request-duration histogram, and can find its log lines via the request ID from #932 — but cannot see where those 4.2 seconds went across compute-driver call, policy evaluation, supervisor session establishment, and readiness wait. That decomposition is exactly what a trace provides and no current surface does.

This issue is the tracking parent for that work. It is a sub-issue of #1055, which remains the roadmap-level umbrella covering OCSF, log export, product telemetry, and dashboards.

Audience: the operator running an OpenShell gateway. Tracing for users of the SDKs who create sandboxes (#1818) is explicitly out of scope here.

Proposed Design

Deliver an OTel export surface for the gateway process, decomposed into sub-issues. PR #1270 built a working version of most of this and was closed unmerged on 2026-05-27; it is a valuable reference, but the implementation should be re-derived rather than rebased (see Agent Investigation for why).

Proposed sub-issues

1. Gateway OTLP trace export. Append a tracing-opentelemetry layer to the existing subscriber in tracing_bus.rs, so the current tower_http::trace::TraceLayer per-request span becomes the OTLP root without rewriting handlers. Resolve configuration from standard OTEL_* environment variables with a CLI/TOML fallback, honor OTEL_TRACES_SAMPLER, and flush the batch span processor on shutdown. Default off.

2. Span coverage for gateway operations. Instrument the spans that make a trace useful rather than merely present: sandbox lifecycle transitions, compute-driver calls, policy evaluation, supervisor session establishment and relay claim, and credential-broker actions. Decide per operation whether it is a span, an event on a parent span, or neither.

3. Correlation identifiers. Carry stable fields across spans and align them with OCSF event correlation IDs so an operator can pivot from a trace to the security record for the same activity, and reuse the request ID from feat(server): add request-ID middleware for request correlation #932 rather than inventing a parallel identifier. This subsumes the emission half of Add OpenTelemetry trace correlation across gateway activity #1758.

4. OTLP metrics export. Offer OTLP push for the existing metrics families alongside the current Prometheus scrape endpoint, for operators whose collectors do not scrape. Per the discussion on feat(server): metrics instrumentation #909, scrape remains the default and push is opt-in.

5. Kubernetes monitoring surface. Helm ServiceMonitor (gated, off by default) targeting the existing named metrics port, OTEL_* projection into the StatefulSet, and values documentation. Also decide whether --metrics-port should default to enabled when the chart wires up scraping.

6. Local development stack. A mise task installing a collector plus a trace backend and Grafana into the k3d dev cluster, so contributors can validate the pipeline end to end. Mirrors the existing Keycloak dev add-on pattern.

7. Documentation. An operator-facing page under docs/observability/ covering enablement, configuration surface, what spans exist, and a reference collector setup — plus an update to architecture/gateway.md.

Sub-issues are expected to land independently; the trace export path (1–3) is the critical path and the rest can follow.

Open design questions

• Config surface. OTEL_* env vars are the ecosystem convention, but OpenShell has a TOML gateway config (RFC 0003) that should probably own this. Precedence between the two needs a decision, and docs/reference/gateway-config.mdx needs updating either way.
• Dependency cost. The OTel Rust SDK plus OTLP exporter is a non-trivial addition to the gateway's dependency graph and SBOM. Feature-gating at compile time is worth evaluating, especially alongside Conditional compilation of compute drivers #1943.
• Sandbox boundary. Whether supervisor-side spans join the gateway trace is deliberately left out of scope. It interacts with Push gateway-owned desired state to supervisors #1731's supervisor-session restructuring and with sandbox egress policy, and deserves its own design.

Alternatives Considered

• Rebase PR feat(observability): add gateway OTLP traces and initial Kube monitoring surface #1270 as-is. Fastest path, but its dependency pins were chosen for a workspace state that no longer exists, and it predates the gateway TOML config and the interceptor and middleware subsystems. The design is sound; the diff is stale.
• Metrics only, finish feat(server): metrics instrumentation #909's catalog. Cheaper and complementary, and should happen regardless — but aggregate counters cannot decompose a single slow request across subsystems, which is the specific gap here.
• Rely on OCSF JSONL for operational analysis. OCSF describes sandbox security activity for SIEM consumption. It is the wrong schema and the wrong granularity for gateway latency and failure analysis.
• Defer to a service mesh. Mesh-level traces capture the network hop but not policy evaluation, driver calls, or session establishment inside the gateway process, and would impose a mesh dependency on all deployments.

Agent Investigation

Verified against main at 0d5e5c5.

No OpenTelemetry exists in the repository. grep -rni "opentelemetry|otlp" across crates/, python/, deploy/, and docs/ returns a single hit, in deploy/sbom/resolve_licenses.py, unrelated to product code.

What is present:

• metrics facade + metrics-exporter-prometheus, /metrics route at crates/openshell-server/src/http.rs:172, builder init at crates/openshell-server/src/lib.rs:472.
• Emitted families are limited to openshell_server_grpc_requests_total, openshell_server_grpc_request_duration_seconds, openshell_server_http_requests_total, openshell_server_http_request_duration_seconds (multiplex.rs), a readiness gauge (readiness.rs), and interceptor fail-open/fail-closed/latency (openshell-gateway-interceptors/src/runtime.rs). The broader catalog proposed in feat(server): metrics instrumentation #909 is unimplemented.
• --metrics-port / OPENSHELL_METRICS_PORT defaults to 0, meaning the dedicated metrics listener is off unless configured (cli.rs:68).
• Request-ID correlation from feat(server): add request-ID middleware for request correlation #932 is in place; that issue framed itself as "the minimum unit of correlation before adopting full OpenTelemetry."

Why PR #1270 should be re-derived rather than rebased: it pinned opentelemetry 0.29 and tracing-opentelemetry 0.30, described in the PR body as the latest set compatible with the workspace's tonic 0.12 + prost 0.13. The workspace is now on tonic 0.14 / tonic-prost 0.14 / prost 0.14, so that pin rationale no longer holds and the version selection has to be redone. The PR was also authored before the gateway TOML config (#1317), the gateway interceptors (#2005), and the supervisor middleware crates landed — all of which are plausible span sources. It was closed with no review comments explaining the decision, so the reason for abandonment is not recoverable from the PR itself and may be worth confirming with a maintainer before investing.

Related issues:

• Enterprise Observability #1055 — parent roadmap issue; lists "OTEL support" as in-scope.
• Add OpenTelemetry trace correlation across gateway activity #1758 — OTel trace correlation across gateway activity. Open, state:triage-needed, no comments since 2026-06-04. Its emission and correlation requirements should be absorbed by sub-issues 1–3 and the issue closed as superseded, or retargeted as one of them.
• feat(server): metrics instrumentation #909 — metrics instrumentation. Open; complementary, and the source of the "scrape first, push as operator choice" position.
• feat(observability): add gateway OTLP traces and initial Kube monitoring surface #1270 — closed unmerged reference implementation.

Deliberately adjacent, not children:

• Add optional OpenTelemetry tracing hooks to the Python SDK #1818 — SDK tracing hooks. Different audience (sandbox users).
• feat(observability): investigate portable sandbox log collection #1922 — durable sandbox log collection. Sandbox log export path, currently state:stale with an unmerged POC branch.
• Centralized audit/event log #1933 — client-facing event stream. Real-time integration path for third-party clients.
• feat: read-only web dashboard (/ui) for developers and DevOps engineers #2192 — read-only web dashboard. A consumer of this data, not a producer.

Checklist

• I've reviewed existing issues and the architecture docs
• This is a design proposal, not a "please build this" request