fix(insights): preserve native retention evidence through publication

[izadoesdev]

Native retention investigations could lose cohort dates, denominators and identity coverage while rewriting detector prose, and could publish a saved comparison after reading contradictory counts. This change carries validated native measurements in the existing stored signal, renders the initial comparison once and blocks publication on conflicting or scope-mismatched retention reads. Additional model evidence cannot relabel native quantities. Legacy signals remain readable; completeness and the50-profile detector floor remain unchanged.

Fresh final comparison:44 synthetic model investigations plus4 deterministic saved checks, with actual output and observable-step review. In14 native cases per arm, baseline→candidate:20→17 model turns,6→3 rejected finishes,8.05→6.08s median elapsed,44.5→54 median words,12/14→14/14 common evaluator passes. All14 candidate native outputs retained the measurement facts. Revenue native checks and saved checks passed; one revenue run per arm omitted the inspected unchanged EUR control. Both arms still failed the two undersized legacy/manual cases; secondary control-period wording also remains imperfect. This is a bounded provenance/reliability improvement, not full semantic acceptance or a production benchmark. Production LoC is net+212.

Validation: root lint;33 typecheck tasks;655 Insights tests (6 live-only skips);26 shared-schema tests;210 focused flow tests;27/27 root pre-push tasks. Adversarial regressions cover hidden/sticky conflicts, scope mismatches, malformed rows and swapped quantity roles. Runtime inputs match the sealed evaluation byte-for-byte across242 captured sources; final follow-up changes tests only. No schema migration is required.

Scope: native retention evidence and its publication boundary. No dependency. Known overlap: draft#751 also touches agent.ts/SPEC.md and should rebase after this slice. AI-assisted implementation and review; maintainer contribution.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
databuddy-status	Ready	Preview	Sep 9, 2026 9:36pm UTC

2 Skipped Deployments

Project	Deployment	Actions	Updated
dashboard	Skipped		Sep 9, 2026 9:36pm UTC
documentation	Skipped		Sep 9, 2026 9:36pm UTC

[unkey-deploy]

The latest updates on your projects. Learn more about Unkey Deploy

Name	Status	Preview	Inspect	Updated (UTC)
links (preview)	Ready	Visit Preview	Inspect	Sep 9, 2026 9:36pm

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: f6ea0a76-03f9-4f16-b422-505db08002a6

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[izadoesdev]

@greptileai Please review the exact current head 43d19d3. Focus on native retention provenance, legacy signal compatibility, source alignment, the zero-or-one additional-evidence finish contract, and conflicting later measurements. Fresh paired output evaluations are still in progress; this PR remains draft until that quality gate and every review comment are resolved.

[greptile-apps]

Greptile Summary

This PR preserves validated native retention measurements through signal storage and investigation publication, renders the saved comparison deterministically, and prevents publication when later retention reads conflict with the snapshot or use mismatched scope.

• Adds a structured retention measurement to the shared investigation signal.
• Carries cohort counts, identity coverage, boundaries, and observation metadata through frozen investigations.
• Restricts model-authored retention evidence and validates later reads before publication.
• Adds regression coverage for conflicts, malformed rows, scope mismatches, and swapped quantities.
• The previous conflicting-read finding is fixed, the nested-ternary rule violation is fixed, and the unknown boundary finding was correctly conceded.

Confidence Score: 5/5

The PR appears safe to merge; no new actionable issues or outstanding previous findings remain at the reviewed head.

The current implementation deterministically preserves native retention evidence and rejects publication when successful exact-query reads conflict or cited retention evidence has mismatched scope. The nested test ternaries were removed, and the prior unknown concern was correctly conceded because the value is an untrusted runtime boundary narrowed through schema validation.

Important Files Changed

Filename	Overview
packages/shared/src/insights.ts	Adds the backward-compatible structured retention measurement schema and population invariants.
apps/insights/src/measurement-plan.ts	Validates native retention rows and attaches complete measurements to detected signals.
apps/insights/src/investigation.ts	Preserves structured retention measurements while preparing frozen investigation inputs.
apps/insights/src/agent.ts	Renders native evidence deterministically and blocks publication on conflicting or scope-mismatched reads.
apps/insights/src/investigation-flow.test.ts	Exercises publication behavior across confirmed, conflicting, malformed, and mismatched retention reads.
apps/insights/src/measurement-plan.test.ts	Verifies measurement persistence, legacy compatibility, and schema rejection of inconsistent populations.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    A[Native retention query] --> B[Validate cohort rows]
    B --> C[Store structured retention measurement]
    C --> D[Freeze investigation signal]
    D --> E[Render deterministic comparison]
    E --> F[Agent investigation]
    F --> G{Later retention reads aligned and consistent?}
    G -- Yes --> H[Allow validated publication]
    G -- No --> I[Keep investigation private]

Loading

_{Reviews (4): Last reviewed commit: "test(insights): simplify retention evide..." | Re-trigger Greptile}

[izadoesdev]

@greptileai Please review the exact final head ad8d17f, including the fixes for discussion_r3972890270. The publication boundary now enforces native retention scope/count consistency across all successful reads, cannot discard a conflict by dropping its citation or reading again, and prevents extra model prose from relabelling native quantities. Please inspect the isolated scope and swapped-field regression cases. Fresh paired model evaluations are still a separate required gate; this PR remains draft.

[izadoesdev]

@greptileai Please review exact final head 5687b08. This follow-up removes the nested test ternaries; production sources are byte-identical to your reviewed ad8d17f head. The unknown-input comment has a detailed reply citing the more specific repository requirement in .agents/skills/bun-fullstack/SKILL.md:111 for unknown + schema validation at untrusted boundaries. Please confirm that disposition and check the test cleanup. The prior publication-conflict P1 is fixed as confirmed by your last review. Fresh model evaluation remains in progress.