feat(telemetry): record command outcomes

.changeset/telemetry-command-outcomes.md

@@ -0,0 +1,15 @@
+---
+"@fission-ai/openspec": minor
+---
+
+Record how commands end in usage telemetry, so failures surface without someone filing an issue.
+
+A new `command_completed` event carries the outcome, a failure class from a fixed list, a bucketed exit code, and a bucketed duration. Runs that previously produced no telemetry at all now do: unknown commands, unknown flags, and a group invoked with no subcommand all exited before the tracking hook ran. Activation milestones, retry visibility, and a per-run correlation id are included.
+
+`OPENSPEC_TELEMETRY_DEBUG=1` prints every event to stderr and sends nothing, so the collected list can be verified locally rather than taken on trust. `openspec config get telemetry` now reports the enabled state, the anonymous id, and the file holding it.
+
+Command behavior, output, and exit codes are unchanged. Telemetry remains opt-out via `openspec config set telemetry.enabled false`, `OPENSPEC_TELEMETRY=0`, or `DO_NOT_TRACK=1`, and stays off in CI.
+
+**Privacy:** this collects more than earlier releases did. `SECURITY.md` previously stated that no environment was collected, and the README stated that only command names and version were collected. Both commitments end here: platform, Node major, install kind, and the invoking coding agent are now included. They are replaced by a narrower and checkable commitment — every event name, property key, and value must be a member of a fixed list, enforced by dropping anything else before the payload is built, so no field exists that could carry a name, path, or message. Tool identities are sent as standalone events with no other property attached, and durations and exit codes are bucketed, so no single event describes a machine precisely enough to single out its owner. The full property list is in the README, and a test fails if it drifts from the code. Data is described as pseudonymous rather than anonymous, and a deletion route is published — though deleting the local id severs your history without asking anyone.
+
+Every event sets `$ip: null` and `$geoip_disable: true`, so no address or derived location is recorded. The disclosure states that and the in-transit caveat, rather than asserting anything about proxy logging that this repository cannot enforce. No retention period is published until one is configured.

README.md

@@ -233,9 +233,52 @@ Open a discussion (for core design changes) or an issue before you open a PR, an
 <details>
 <summary><strong>Telemetry</strong></summary>
 
-OpenSpec collects anonymous usage stats.
+OpenSpec collects pseudonymous usage stats: a random id generated on your machine, plus the properties listed below. Automatically disabled in CI.
 
-We collect only command names and version to understand usage patterns. No arguments, paths, content, or PII. Automatically disabled in CI.
+**See exactly what would be sent, on your own machine:**
+
+```bash
+OPENSPEC_TELEMETRY_DEBUG=1 openspec list
+```
+
+That prints every event to stderr and sends nothing. It works even if you have opted out, and it does not create the id it shows you.
+
+**Everything collected:**
+
+| Property | Values |
+| --- | --- |
+| `command` | The command you ran, e.g. `archive`, `change:validate`. Never its arguments |
+| `version`, `version_code` | The OpenSpec version, and the same version as a sortable integer |
+| `outcome` | `success`, `user_error`, `internal_error`, `cancelled` |
+| `error_class` | The kind of failure, from a fixed list, e.g. `no_root`, `validation_failed`. Never the message |
+| `exit_code` | `0`, `1`, `130`, `other` |
+| `duration` | `<100`, `100-500`, `500-2000`, `2000-10000`, `10000+` milliseconds |
+| `previous_outcome`, `previous_command_same` | Whether your last run failed, and whether it was the same command |
+| `platform`, `node_major` | `darwin`/`linux`/`win32`; the Node major version |
+| `install_kind` | `global`, `npx`, `source`, `other` |
+| `invoker` | Which coding agent is running the command, from a fixed list, or `terminal`/`unknown` |
+| `stdout_tty`, `json_mode`, `prompted`, `first_run` | Booleans |
+| `profile`, `delivery` | Your install profile and delivery mode |
+| `tools_count` | How many AI tools are configured: `0`, `1`, `2-3`, `4+` |
+| `schema_source` | `package`, `project`, or `user` |
+| `store_in_use` | Whether this run resolved through a store rather than a local root. Never which one |
+| `changes` | How many active changes: `00`, `01-03`, `04-10`, `11-30`, `31+` |
+| `milestone`, `time_to_reach` | The first time you reach each of `install` (your first run), `init`, `propose` (`openspec new change`), `apply` (`openspec validate`), and `archive`, and how long it took |
+| `tool` | Each AI tool you have configured, reported once, as its own event carrying no run context and no run id |
+| `run_id`, `work_session_id` | Random ids correlating one run, and runs less than 30 minutes apart |
+| `surface` | Always `cli` |
+
+Every one of those has a fixed set of possible values. Anything else is dropped before the payload is built, so there is no field that could carry a name, path, or message.
+
+**Never collected:** command arguments, file paths, project names, change/spec/schema/artifact names, store ids or remotes, file contents, error messages, environment variable names or values, hostnames, usernames, git remotes, or IP addresses.
+
+**Stored on your machine** in the config file (`openspec config get telemetry` prints its path): the random id, the notice version, the time of your first run, the work-session id and last-activity time, which milestones and tools have been reported, and your previous run's outcome. Nothing is written at all if you have opted out.
+
+**No IP, no location.** Every event sets `$ip: null` and `$geoip_disable: true`, so the analytics backend records neither your address nor anything derived from it. Requests do reach a first-party endpoint that terminates TLS, which necessarily observes the connecting address in transit — those two flags are what the shipped code guarantees, and you can see them yourself with `OPENSPEC_TELEMETRY_DEBUG=1`.
+
+**Deletion:** open a [GitHub issue](https://github.com/Fission-AI/OpenSpec/issues/new) with the id from `openspec config get telemetry`, or send it privately through [GitHub Security Advisories](https://github.com/Fission-AI/OpenSpec/security/advisories/new) if you would rather not post it publicly. You do not have to ask us for anything, though: deleting the id from your config severs all future events from everything before it, immediately and on your own.
+
+The id identifies a configuration directory, not a person — a shared home directory means one id covers several people, so it is not a user count.
 
 **Opt-out (any one is enough):**
 - `openspec config set telemetry.enabled false` (global config; unset means on)

SECURITY.md

@@ -12,7 +12,7 @@ Fixes ship in the latest published version on npm. Older versions are not patche
 
 ## Threat model
 
-OpenSpec is a local command-line tool. It has no server, no network listener, and no privileged daemon. It reads and writes markdown under the directory you run it in, using paths you supply, with your own user permissions. It can offer to upgrade itself during `openspec update`, and only with your say-so. It sends anonymous usage telemetry, which you can disable with `OPENSPEC_TELEMETRY=0`.
+OpenSpec is a local command-line tool. It has no server, no network listener, and no privileged daemon. It reads and writes markdown under the directory you run it in, using paths you supply, with your own user permissions. It can offer to upgrade itself during `openspec update`, and only with your say-so. It sends pseudonymous usage telemetry, which you can inspect with `OPENSPEC_TELEMETRY_DEBUG=1` and disable with `OPENSPEC_TELEMETRY=0`.
 
 That shapes what is and isn't a vulnerability here:
 
@@ -45,7 +45,7 @@ ls node_modules | grep -E '^(vite|rollup|vitest|eslint|js-yaml|minimatch)$'   #
 | Install scripts | The package ships no `preinstall`, `install`, or `postinstall` script, so installing it from the npm registry runs no code from OpenSpec. (`prepare` is still declared; npm runs it only for git and local-directory installs, where it builds from source.) Shell completions are opt-in via `openspec completion install`; the CLI prints a one-line tip about them on its first run. |
 | Running other programs | Every call that goes through a shell uses a fixed literal (`which gh`, `gh auth status`). Anything carrying your input — issue text, editor paths, workset commands, the path passed to `openspec update` — uses an argument array, never string interpolation into a shell. On Windows, `.cmd` shims are launched through `cross-spawn`, which escapes arguments rather than concatenating them. |
 | Installing software | `openspec update` can run `npm install -g @fission-ai/openspec@latest` and then re-run `openspec update` with the upgraded CLI. It does this only after you answer yes to a prompt, only for the OpenSpec package itself, only when npm owns the install, and never in CI or a non-interactive shell. A global install lives outside your project, so it runs with your permissions there and executes whatever lifecycle scripts the published package ships. It then reads the installed binary's version back rather than assuming the upgrade took. Decline and it prints the command for you to run yourself. |
-| Telemetry | Command name, OpenSpec version, and a locally generated random UUID. No file paths, no file contents, no environment, no hostname, and IP capture is explicitly disabled. Opt out with `OPENSPEC_TELEMETRY=0` or `DO_NOT_TRACK=1`; it is off in CI automatically. |
+| Telemetry | The command name, how it ended (outcome, a failure class from a fixed list, a bucketed exit code and duration), and bounded run context: platform, Node major, install kind, which coding agent invoked it, a count of configured tools, and bucketed counts of changes. Plus a locally generated random UUID. **This is more than earlier releases collected — platform and Node major are environment facts, which previous versions of this document said were not collected.** No file paths, no file contents, no environment variable names or values, no hostname, no usernames. Every event sets `$ip: null` and `$geoip_disable: true`, so neither your address nor a location derived from it is recorded; the ingest endpoint terminates TLS and so observes the connecting address in transit, which those flags do not change. Every property has a fixed set of possible values and anything else is dropped before the payload is built; see the full list in the README. Verify it yourself with `OPENSPEC_TELEMETRY_DEBUG=1`, which prints the events and sends nothing. Opt out with `OPENSPEC_TELEMETRY=0` or `DO_NOT_TRACK=1`; it is off in CI automatically. |
 | Network | Telemetry when enabled, and one npm registry request during `openspec update` to check whether a newer CLI has been published. That request sends no data about you beyond what any HTTP request reveals, runs once per `openspec update` with nothing cached, and is skipped when `CI` is set to anything but an explicit off-value, under `NODE_ENV=test`, or when `OPENSPEC_NO_UPDATE_CHECK`, `DO_NOT_TRACK=1`, or `OPENSPEC_TELEMETRY=0` is set. Reading, writing, and validating specs is entirely local. |
 
 ## Automated checks

docs-lab/reference/configuration/environment-variables.md

@@ -9,6 +9,15 @@ its network-permission flag. XDG vars move the config/data directories. -->
 
 ## OPENSPEC_TELEMETRY
 
+Set to `0` to disable usage telemetry. Telemetry is on by default (opt-out) and
+off automatically when `CI` is set to anything but an explicit off-value.
+
+## OPENSPEC_TELEMETRY_DEBUG
+
+Set to `1` to print every telemetry event to stderr and send nothing. Works
+while opted out, and does not create the anonymous id it shows you. This is the
+way to verify what is collected without taking the documentation on trust.
+
 ## DO_NOT_TRACK
 
 ## XDG_CONFIG_HOME and XDG_DATA_HOME

docs/cli.md

@@ -1183,7 +1183,10 @@ openspec config profile core
 ```
 
 **Telemetry opt-out:** `telemetry.enabled` defaults to on when unset (opt-out model).
-Set it to `false` to disable anonymous usage stats and the `openspec update` version check.
+Set it to `false` to disable pseudonymous usage stats and the `openspec update` version check.
+`OPENSPEC_TELEMETRY_DEBUG=1` prints every event that would be sent to stderr and sends nothing, so you can
+see exactly what is collected. `openspec config get telemetry` reports the current state, the anonymous id,
+and the file holding it.
 Environment variables take precedence over config: `OPENSPEC_TELEMETRY=0`, `DO_NOT_TRACK=1`,
 and a truthy `CI` value (e.g. `true`/`1`/`yes`) always disable telemetry regardless of the config value.
 

docs/faq.md

@@ -132,7 +132,7 @@ OpenSpec works best with high-reasoning models. The README recommends models lik
 
 ### Does OpenSpec collect data?
 
-It collects anonymous usage stats: command names and version only. No arguments, paths, content, or personal data, and it's off automatically in CI. Opt out with `export OPENSPEC_TELEMETRY=0` or `export DO_NOT_TRACK=1`.
+It collects pseudonymous usage stats: the command you ran, how it ended, and bounded run context such as OS and Node major. No arguments, paths, content, item names, or personal data, and it's off automatically in CI. Every property has a fixed set of possible values — see the full list in the README. Print exactly what would be sent with `OPENSPEC_TELEMETRY_DEBUG=1`. Opt out with `export OPENSPEC_TELEMETRY=0` or `export DO_NOT_TRACK=1`.
 
 ### How do I upgrade?
 

openspec/changes/add-command-outcome-telemetry/proposal.md

@@ -0,0 +1,122 @@
+# Record command outcomes in anonymous telemetry
+
+## Why
+
+Telemetry today fires once, in the `preAction` hook, carrying `command`,
+`version`, and `surface` (`src/telemetry/index.ts`, `trackCommand`). It can
+answer "how often is `archive` run" and nothing else.
+
+It cannot answer any question we act on:
+
+- Did the command **succeed**? Nothing is recorded after the action runs.
+- If it failed, **how**? Every command catches its own error, prints
+  `Error: <message>`, and sets `process.exitCode = 1`. The class of failure
+  never leaves the process.
+- Was the caller a **person or an agent**? Both look identical.
+- Do users get from `init` to a first archived change? Unknown.
+
+Worse, failures are the *least* visible runs. Seventeen call sites in
+`src/cli/index.ts` end with `process.exit(1)`, which skips commander's
+`postAction` hook — the trap the code already documents at
+`src/cli/index.ts:317` and `:468`. Commander's own usage errors — unknown
+command, unknown flag, a group run with no subcommand — exit before `preAction`
+runs, so they produce no event at all. And commander chains hooks without a
+`catch`, so an error escaping a command's own handling skips the hook too.
+
+So the runs we most need to see are the ones most likely to vanish: the user
+who typed the wrong command, and our own bugs.
+
+The feedback loop is closed only in the good case. Users hitting a confusing
+failure do not run `openspec feedback` and do not open an issue; they stop using
+the tool. We ship fixes for the problems that get reported, not the problems
+that happen.
+
+This change closes that loop without collecting anything about *what* a user is
+working on.
+
+## What Changes
+
+**One new event, `command_completed`**, emitted on every exit path that reaches
+our own error handling, carrying the outcome, a bounded error class, a bucketed
+exit code, and a bucketed duration.
+
+**A hard property contract.** Every event name, property key, and property value
+must be a member of a compile-time list, a boolean, or a bucket label — and the
+allowlist is enforced at send time, not just asserted in a test. This is the
+structural reason the new data cannot describe what someone is working on: there
+is no field it could travel in, and an unrecognized field is dropped before the
+payload is serialized.
+
+**Bounded run context**: platform, Node major, install kind, invoker, TTY and
+JSON flags, profile, delivery, a *count* of configured tools, where the schema
+came from, and a bucketed change count.
+
+Deliberately excluded, each for a stated reason: schema, artifact, change, spec,
+and store names, because they are user-authored text; store remotes and paths,
+because they identify an organization; and **raw millisecond durations**, because
+they profile the machine and, at an interactive prompt, record human response
+times.
+
+**Which assistant people use, without the fingerprint.** Tool identity ships as a
+separate `tool_configured` event — once per tool per user, carrying no run
+context at all. That answers how much of the userbase runs Cursor or Claude Code
+while never assembling the configured *set* alongside platform, install kind, and
+counts in one row, which is the combination that would single out an unusual
+user. The `invoker` enum complements it by recording which agent is actually
+driving a given run.
+
+**Telemetry never interrupts.** No prompts, ever. It does not block the command,
+does not delay exit beyond the existing 1-second timeout, and never writes to
+stdout. The one-line first-run disclosure is a notice on stderr, not a question.
+
+**Outcome coverage** for all three families of exit that skip the hooks today,
+including commander's own usage errors — which are invisible now and are exactly
+the "user typed the wrong thing" signal.
+
+**Retry visibility.** Whether a user recovers from a failure is the most
+actionable signal we can have, and it is not otherwise computable. Two bounded
+properties carry it: the previous run's outcome, and whether it was the same
+command.
+
+**Correlation at two scales:** a per-invocation `run_id` whose real job is to
+reveal exit paths this spec failed to cover, and a `work_session_id` with a
+30-minute window, because a CLI work session is many invocations and command
+sequences are not computable without it.
+
+**Five milestone events** — `install`, `init`, `propose`, `apply`, `archive` —
+giving the activation funnel a real denominator.
+
+**`OPENSPEC_TELEMETRY_DEBUG=1`** prints every event that would be sent and sends
+nothing. It works when telemetry is *disabled*, since the person most likely to
+want it is someone who opted out and is deciding whether to opt back in, and it
+never creates the anonymous id it is being used to inspect.
+
+**Data subject controls**: `openspec config get telemetry` shows the state, the
+id, and the file holding it; deleting the id severs all future events from all
+prior ones; the disclosure carries a retention period and a deletion contact.
+
+**Honest disclosure.** `SECURITY.md` promises "no environment" and `README.md`
+promises "only command names and version." This change ends both. The spec
+requires the changelog to say so under a `Privacy` heading rather than quietly
+editing the promise, requires a test that fails when an allowlisted property is
+undocumented, and requires the docs to stop calling the data "anonymous"
+unqualified — a persistent id plus device characteristics is pseudonymous, and
+overstating it is what would undermine every other claim on the page.
+
+Two smaller corrections the review surfaced: cancellation must never delay exit
+to flush telemetry (Ctrl-C should stop the process, not phone home), and the
+ingest proxy must not log client IPs — it terminates TLS, so `$ip: null`
+governs what the backend records, not what our own infrastructure sees.
+
+Telemetry stays opt-out and unchanged otherwise: same `OPENSPEC_TELEMETRY=0`,
+`DO_NOT_TRACK=1`, `openspec config set telemetry.enabled false`, same automatic
+off-in-CI, same silent failure, same 1-second timeout. Users who have seen the
+old notice get a one-line notice naming what changed, once.
+
+## Impact
+
+- Affected specs: `telemetry` (ADDED: 15 requirements; MODIFIED: 3)
+- Affected code: `src/telemetry/`, `src/cli/index.ts`, `src/commands/shared-output.ts`, `src/commands/config.ts`
+- Affected docs: `README.md`, `SECURITY.md`, `CHANGELOG.md`, `docs-lab/reference/configuration/environment-variables.md`
+- Affected infrastructure: the `edge.openspec.dev` ingest proxy (IP logging, GeoIP)
+- Command behavior, output, and exit codes are unchanged for every user.