Skip to content

feat: write /etc/glueops/otel.env so CDE VMs ship metrics - #515

Open
venkatamutyala wants to merge 1 commit into
mainfrom
feat/otel-env-vm-metrics
Open

feat: write /etc/glueops/otel.env so CDE VMs ship metrics#515
venkatamutyala wants to merge 1 commit into
mainfrom
feat/otel-env-vm-metrics

Conversation

@venkatamutyala

Copy link
Copy Markdown
Contributor

What

Companion to GlueOps/codespaces#580, which bakes prometheus-node-exporter + an OpenTelemetry Collector into the CDE image with its systemd unit gated on /etc/glueops/otel.env. This PR makes the bot write that file through cloud-init — the same way it already writes cde_token and tunnel_endpoint — so the collector turns on and knows which VM it is.

When OTEL_EXPORTER_OTLP_ENDPOINT is set on the bot, each new VM's write_files gains (root, 0600, b64 like codespace.env):

OTEL_EXPORTER_OTLP_ENDPOINT=https://otel-http-cde.observability.glueopshosted.com
OTEL_RESOURCE_ATTRIBUTES=cloud.region=<region>,host.type=<instance type>,host.image.name=<image>,deployment.environment.name=<APP_ENVIRONMENT>,glueops.cde.owner=<email>

plus a systemctl start otelcol-contrib || true runcmd after tailscale up (belt-and-braces for first boot; no-op on images that predate the unit). host.name is deliberately absent — the collector takes it from the hostname cloud-init already sets. deployment.environment.name is the one attribute the VM can't derive itself, which is the main reason this lives in the bot rather than the image.

Behaviour / safety

  • Unset endpoint → no file, and the generated user-data is byte-for-byte what it is today. Verified by generating both variants and parsing them as YAML (the same check the provisioner does).
  • Attribute values containing , = % or whitespace are dropped, not escaped (they'd change the meaning of the k=v,k=v string) — same policy as TUNNEL_ENDPOINT. A malformed endpoint writes nothing.
  • codespace.env is untouched. It holds user profile secrets and is fed to the dev container, which is why it isn't the carrier.
  • The endpoint is write-only with no auth, so it is not a secret and needs no redaction.

Files

  • util/get-user-data.jsotelEnvFile() (exported), write_files now emitted from a list, the runcmd start.
  • example.env, CLAUDE.md, README.md — the new optional variable and the contract.

Rollout

Either PR can land first. Old images ignore the file; new images without the file stay inert. The endpoint isn't routed yet (prod Traefik default backend), so nothing arrives until that's up — VMs queue and drop, bounded.

PR Checklist

  • README.md up to date
  • Code tested (Node 22: unset/set/bad-value/bad-endpoint cases; full user-data parsed as YAML; codespace.env unchanged)
  • AI feedback reviewed

🤖 Generated with Claude Code

https://claude.ai/code/session_013zxxz8592fPJaq2B81jsC2

The codespaces image (GlueOps/codespaces#580) carries prometheus-node-exporter
and an OpenTelemetry Collector whose systemd unit is gated on
/etc/glueops/otel.env. This writes that file through cloud-init when
OTEL_EXPORTER_OTLP_ENDPOINT is set on the bot: the OTLP/HTTP base URL plus
OTEL_RESOURCE_ATTRIBUTES built from metadata the bot already has (region,
instance type, image, owner, and APP_ENVIRONMENT as deployment.environment.name
— the one thing the VM cannot know on its own). A `systemctl start
otelcol-contrib || true` runcmd after `tailscale up` is belt-and-braces for the
first boot and a no-op on older images.

Unset endpoint = no file, and the generated user-data is byte-for-byte what it
is today. Attribute values carrying ',', '=', '%' or whitespace are dropped
rather than escaped — the same policy as TUNNEL_ENDPOINT — so one bad value
cannot cost a VM its metrics, and a malformed endpoint writes nothing.
codespace.env is untouched: it holds user secrets and is fed to the dev
container, which is why it is not the carrier here.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zxxz8592fPJaq2B81jsC2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant