Skip to content

feat(cli): align primitive funnel measurement - #3052

Open
miguel-heygen wants to merge 3 commits into
feat/thread-message-stackfrom
feat/video-primitives-measurement-parity
Open

feat(cli): align primitive funnel measurement#3052
miguel-heygen wants to merge 3 commits into
feat/thread-message-stackfrom
feat/video-primitives-measurement-parity

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

What

Makes the primitive install funnel introduced in #3046 actually measurable: one canonical ordered event vocabulary, bounded event properties, anonymous result delivery across every catalog output mode, funnel continuity across the OAuth process boundary, and process-atomic terminal claims for preview and render.

Instrumentation correctness only. No product behavior changes.

Why

The first slice emitted events, but it did not guarantee they were the same events, in the same order, carrying the same funnel id, once the flow crossed a process boundary or ran concurrently. That leaves three failure modes that make any downstream read of the journey untrustworthy:

  • Duplicate terminals. Two concurrent renders of the same project each emit a terminal success, so the funnel appears to convert twice.
  • Split funnels. The OAuth browser round-trip restarts the process, so the post-auth half of the journey gets a fresh id and the funnel silently breaks in the middle.
  • Unbounded properties. Unclamped counts and durations blow up property cardinality and let NaN reach the transport.

Two further failures were found by running the built CLI against the real ingest endpoint rather than against mocks, and are fixed here:

  • A claimed terminal event was lost permanently. The claim is taken before the send, but delivery was the fire-and-forget exit flush: a detached child, no retry, no way to report failure. One real install claimed primitive_preview_succeeded and it never arrived. Because the claim is durable and the delivery was not, that step could never be emitted again.
  • Step order was not recoverable. primitive_auth_completed and primitive_install_started are emitted back to back inside one command and land in the same millisecond. Two identical warehouse queries returned them in opposite orders, so a funnel keyed on the sequence would intermittently see the install beginning before the auth that authorized it.

How

  • One table owns the vocabulary. PRIMITIVE_FUNNEL_SIDE_EFFECTS in telemetry/primitive-funnel.ts maps each lifecycle step to its event name, its stable event-id suffix, and its position in the canonical order. Every emitter goes through it, so a rename cannot drift between call sites.
  • Order is data, not a timestamp artifact. Each entry carries a step, emitted as funnel_step. Consumers order by it instead of by a millisecond that ties. Failure steps share the number of the step they terminate; shipped numbers are never reused.
  • Terminals are claimed, not counted. Preview and render success/failure take an exclusive claim by creating a marker file with O_EXCL (wx), named by the sha256 of the event id, under .hyperframes/primitive-funnel-claims/. First writer wins, everyone else is a no-op. Repeated 32-process preview and render races produced exactly one claimant each time.
  • A claim is only spent once the event is delivered. flush() now reports whether PostHog acknowledged the batch, the terminal path awaits it, and releasePrimitiveFunnelEvent hands the claim back when it did not. This is deliberately at-least-once: every event carries a stable event_id, so a duplicate collapses downstream, while a loss is unrecoverable. Claims are also no longer spent while telemetry is opted out.
  • Properties are clamped at the emit site. result_count is bounded to 0 to 1000, result_rank to 1 to 1000, duration_ms to 0 to 86,400,000. Non-finite input collapses to the nearest bound instead of leaking NaN.
  • Funnel context survives the process gap. primitive-funnel-state.ts persists the context through temp-file plus rename with 0600 / 0700 modes, and re-validates every field on read. A malformed state file degrades to "no funnel" rather than throwing, so telemetry can never break the command it instruments.
  • Identity binds once, and only after auth. The funnel identifies at most once, only on auth completion, and only when an account id is present. Discovery and result selection always report auth_state: anonymous.
  • All three catalog output modes report delivery. JSON, table, and interactive paths emit result delivery with no session required, and a direct add carries the same funnel id through OAuth, install, preview, and render.

Test plan

  • Full CLI suite on Linux: 187 files, 2,508 passed, 2 skipped, 0 failed.

  • Focused telemetry, funnel, auth, resume, and catalog suites: 19 files, 237 passed.

  • Concurrency: repeated 32-process preview and render races, exactly one terminal claimant per race.

  • New coverage: claim released on unacknowledged delivery and re-emitted by a later command, claim retained on acknowledgement, no claim spent while opted out, and a funnel_step ordering assertion over the delivered batches.

  • End-to-end against real ingest, three separate installs of the built CLI: anonymous discovery with no session, add with caller-supplied messages, background preview, and a real 8-frame 1920x1080 MP4. Confirmed in the warehouse that the terminal preview event now arrives, and that ordering by funnel_step puts auth completion before install start even though both events still share one millisecond.

  • Typecheck, oxfmt --check, oxlint: clean.

  • Unit tests added/updated

  • Manual testing performed (three live end-to-end installs of the built CLI, verified against real ingested events)

  • Documentation updated (if applicable) - no user-facing surface changed

Known limitation

A direct hyperframes add cannot produce steps 1 and 2. Funnel state is only persisted at install, so a preceding catalog search carries its own funnel id and does not stitch to the add that follows. Only the interactive catalog path produces the full ordered journey. That is a scoping question about what the funnel should mean, not a defect, so it is deliberately unchanged here.

Stack

Based on feat/thread-message-stack (#3046); merge that first. This PR proves instrumentation correctness only. It does not claim production adoption, which stays gated on merging and releasing the stack and then a separately authorized read of real journey data.

Terminal funnel events took a durable single-use claim before they were
sent, but delivery was the fire-and-forget exit flush: a detached child
with no retry and no way to report failure. An unacknowledged send left
the claim spent on an event PostHog never received, so that step could
never be emitted again. Observed in a real install, where
primitive_preview_succeeded was claimed and never arrived.

The terminal path now awaits delivery and hands the claim back when the
batch is not acknowledged. That trades a permanent loss for a possible
duplicate, which is the cheaper failure: every funnel event carries a
stable event_id, so duplicates collapse downstream while a loss is
unrecoverable. Claims are also no longer spent while telemetry is
opted out.

Separately, funnel order is now carried by an explicit funnel_step
rather than inferred from timestamps. Auth completion and install start
are emitted back to back inside one command and land in the same
millisecond, so ordering by time resolved them arbitrarily and could
report an install beginning before the auth that authorized it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant