feat(vm): ship node exporter + OpenTelemetry Collector on the CDE VM, inert until the bot enables it

[venkatamutyala]

What

Every CDE VM built from this image now carries prometheus-node-exporter (Debian package, bound to 127.0.0.1:9100) and otelcol-contrib (pinned, checksum-verified .deb), which scrapes it, reads per-container stats from the Docker socket (docker_stats), and ships everything as OTLP/HTTP to the GlueOps observability endpoint.

The image ships it inert. otelcol-contrib.service is gated on ConditionPathExists=/etc/glueops/otel.env and takes both the endpoint and the VM's identity (OTEL_RESOURCE_ATTRIBUTES) from that file. Nothing per-VM is baked in — the image is public (qcow2 on S3 + GitHub releases), so it could never carry a secret, and this keeps the endpoint a bot-side setting. A VM without the file (anything created before the bot writes one, or a hand-made VM) leaves the unit inactive, not failed.

Companion PR in slackbot-developer-workspaces writes /etc/glueops/otel.env through cloud-init, the same way it already writes cde_token and tunnel_endpoint. Either PR can land first; neither can break an existing VM. The file's contract is in the new README section.

Files

vm/observability.sh	Packer step (runs after developer-setup.sh). Installs both, adds otelcol-contrib to the docker group, then smoke-tests in the build VM: both units active, node_* served, both receivers produced points (via the collector's own :8888 metrics), and the unit refuses to start without otel.env. Exporter is pointed at a closed local port during the test — nothing leaves the build VM.
vm/otelcol/config.yaml	prometheus + docker_stats → memory_limiter (256 MiB) → resourcedetection → resource (service.instance.id = hostname, so instance on node dashboards is the VM name) → otlphttp with ${env:OTEL_EXPORTER_OTLP_ENDPOINT}.
vm/otelcol/glueops.conf	systemd drop-in: the gate, EnvironmentFile, ordering after cloud-config.service / docker.service.
vm/node-exporter/prometheus-node-exporter.default	loopback bind + textfile dir.
qemu.pkr.hcl	file provisioner uploads vm/; new script in the provisioner list.
hack/test-vm.sh, .github/workflows/vm-tests.yaml	Offline: syntax, shellcheck, structural contract checks, and otelcol-contrib validate against the pinned version (CI downloads it). Exists because the image build only runs post-release.
README.md	"VM metrics" section — the otel.env contract.

Cost (measured on a live host, both receivers, 30s interval)

node exporter ~22 MB RSS / ~0.2% of a core; collector ~190 MB RSS / ~0.2% of a core. ~2.6% of RAM on the smallest slot.

Verified

• hack/test-vm.sh with the pinned 0.159.0 binary: ALL PASS (config validates; refuses to start without the endpoint var).
• shellcheck -S warning clean; packer validate passes.
• Not exercised on real systemd here — that is what the build-VM smoke test is for; the first Packer build after merge is the real gate and fails loudly if the Docker API pin or the gate is wrong.

Notes

• Endpoint (otel-http-cde.observability.glueopshosted.com) is write-only, no auth, and not routed yet (prod Traefik default backend + wildcard cert). Until it is, VMs queue and drop, bounded by memory_limiter.
• docker group membership for otelcol-contrib is root-equivalent on the host — the one privilege this grants; same membership developer-setup.sh gives vscode.
• packer fmt wants to realign pre-existing lines in the HCL; left untouched.

🤖 Generated with Claude Code

https://claude.ai/code/session_013zxxz8592fPJaq2B81jsC2