Proposal: ae build --observe / ae observe — attach to a running process, opt-in per build

[paul-hammant]

Prompted by software-mansion/voyager, a FOSS desktop app from the Erlang/Elixir community that attaches to any running BEAM node and shows supervision trees, per-process state, memory, run queues and scheduler load — without installing anything on the target.

Carried in asks/live-observability-attach.md.

---

The ask, in one line

An opt-in build flag and a matching subcommand:

ae build --observe prog.ae        # compile the introspection endpoint in
ae run   --observe prog.ae        # same, for the run path
ae observe <pid|socket>           # attach to a running --observe binary

--observe is not a default capability for --emit=exe, for the same reason --trace is not: it costs a thread, a socket and a set of counters on the runtime's core loop, and it opens a read surface into a live process. Both are things an operator should say yes to explicitly.

Why Voyager is the right thing to copy, and what it gives up

Voyager's pitch is mostly not about the UI:

Nothing is installed on the target node. Voyager gathers everything through Erlang's built-in introspection functions over RPC, and all rendering and storage happens on your machine — the node you are inspecting is only ever asked for information.

That works because the BEAM always carries the introspection surface — erlang:process_info/2, erlang:system_info/1, erlang:statistics/1 — and always carries a distribution listener that will run RPC for anyone with the cookie. The zero-install property is bought with an always-on cost that every BEAM node pays, and with a security posture (the cookie is the whole auth story) the Erlang community has been apologising for since the nineties.

Aether cannot copy the zero-install part and should not want to. It compiles to C, has no VM, no always-resident reflection, and its whole capability argument is that a binary contains what you asked for and nothing else. So the port of the idea is: keep Voyager's read-only, host-side-rendering, answers-only shape; replace its always-on availability with a build flag.

	BEAM / Voyager	Aether --observe
Available on	every distributed node	binaries built with the flag
Cost when unused	always paid	zero — compiled out
Auth	shared cookie, node-wide	unix socket file perms, per-process
Blast radius	RPC = arbitrary code on the node	fixed read-only query set

The last row matters most. Erlang distribution RPC means anyone who can attach can run any code; Voyager chooses not to, but the protocol doesn't stop it. An Aether observe endpoint should be structurally incapable of it — a closed set of read queries, no eval, no send, no "call this function".

What already exists in the tree

Most of the data Voyager displays already has a counterpart being tracked. This is largely about exposing what the runtime knows, not instrumenting new things.

Voyager panel	Aether equivalent, today
Process list / process_info	ActorBase (runtime/scheduler/multicore_scheduler.h): actor_id, core_id, dead, last_activity_ns, timeout_ns, mailbox depth, spsc_queue
Registered names	std.actors registry (aether_actor_registry.c) — name → actor_ref, already process-global
Message queue length	mailbox + SPSC queue depth per actor
Scheduler / run queues	Scheduler per-core: work_count, steal_attempts, idle_cycles, parked, messages_sent, messages_processed
Memory	std.mem / aether_memory_stats.c, arena + pool accounting
System info	aether_cpu_detect.c, aether_numa.c, aether_version.h
Applications / modules	the message registry + --emit=inspect manifest
Message tracing	--trace already ships (runtime/utils/aether_trace.h) — per-core ring buffer, JSONL out

The gap is not the data. It is that every one of these is either compile-time (ae inspect) or post-mortem (--trace flushes at scheduler_shutdown()), and nothing is readable while the process is alive. Voyager's whole value is the live case: the node misbehaving right now, in production, that you do not want to restart.

Proposed surface

Build side

Parses in cmd_build / cmd_run next to --trace (tools/ae.c ~5367), sets a g_observe that forces the from-source runtime path and defines -DAETHER_OBSERVE, exactly as --trace does for -DAETHER_TRACE. Same rationale verbatim: a prebuilt libaether.a was compiled without the gate, so forcing from-source is what makes one flag sufficient.

Under #ifndef AETHER_OBSERVE every hook is ((void)0) and no field, thread or socket survives into the binary — the aether_trace.h COST WHEN OFF contract, restated.

Composition with the existing gates, stated explicitly:

• --emit=lib --observe is rejected. A library is not the host; if the host wants to be observable, the host builds with --observe. This keeps the "--emit=lib is capability-empty" invariant clean.
• --observe needs no --with=; the endpoint is served by runtime C, not by a std.net import in user code, so it does not perforate the @no_net effect tag or the import gate. Worth documenting loudly, because it looks like it should.
• The sandbox grant list (libaether_sandbox.so) needs an explicit entry for the observe socket, so a sandboxed build neither silently loses it nor silently gains a socket the grant list never approved.

Runtime side

A unix domain socket, path defaulting to $XDG_RUNTIME_DIR/aether/observe-<pid>.sock, overridable with AETHER_OBSERVE=<path> — mirroring how AETHER_TRACE=<path> arms tracing. Unset means the endpoint is not created, so even an --observe build is inert until an operator asks for it. That gives the two-key shape --trace already has: the builder allows it, the operator arms it.

Unix socket rather than TCP is deliberate — file permissions become the auth story, and the default is inherently local-only. Anyone wanting it across a network puts it behind the tunnel they already trust, which is what Voyager's SSH mode does anyway.

One low-priority thread serves it. Requests are a small closed verb set, replies are JSONL — the same format choice --trace made, for the same stated reason: a trace is read with grep and jq far more often than by a viewer.

actors              → id, name, core, state, mailbox depth, dead, idle_ns
actor <id>          → detail for one actor, incl. registered name
scheduler           → per-core work_count, steal_attempts, idle, parked, msg counters
memory              → arena/pool/heap accounting
system              → version, cores, NUMA topology, build flags, uptime
messages            → registry: declared message types and ids
trace on|off        → arm/disarm the --trace ring buffers live, if compiled in

No eval. No send. No set. The verb set is the security boundary and should be reviewed as one.

trace on|off is the item that earns the flag on its own. Today --trace answers "what happened during that run"; the pairing lets it answer "what is happening now, for the next thirty seconds, on the process that is currently wedged" — the question you actually have in production, and the one that currently requires a restart to ask.

Client side

ae observe                    # discover local --observe processes
ae observe <pid>              # attach by pid
ae observe --socket <path>    # attach explicitly
ae observe <pid> --json       # raw JSONL, for jq and for agents

Terminal-first, matching every other ae subcommand. A TUI can come later; --json from day one is what makes it scriptable and usable by a coding agent — the same reasoning behind Voyager shipping an MCP server so an agent can read a live system instead of guessing from source.

Naming: ae inspect is taken and means static inspection of a source file (it shells out to aetherc --emit=inspect). observe is both free and the term of art the Erlang community already uses for the live case (:observer), so the two names stay honestly distinguished.

Why this is the natural seam

The counters exist. The registry exists. The ring buffer, the JSONL writer and the message-name table all exist and are tested — --trace built them. What is missing is a reader that runs while the process is up, and a flag that says the binary is willing to have one.

The flag is the interesting part, not the socket. Voyager's zero-install property is genuinely good DX bought at a price a systems language should refuse to pay: every node, always, whether or not anyone is looking. Aether's version is a per-build decision with a compile-time zero when the answer is no — the same argument --emit=lib makes about std.fs, applied to introspection instead of syscalls.

Worth stating in docs/ explicitly, because "the observability tool is opt-in" reads as a limitation until you say why it isn't.