fix(insights): clarify measured goal and funnel populations - #737
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
2 Skipped Deployments
|
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: ASSERTIVE Plan: Team Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Greptile SummaryThis PR clarifies the visitor populations represented by native goal and funnel analytics without changing executable query behavior or schemas.
Confidence Score: 5/5The PR appears safe to merge because the revised descriptions match the implemented goal and funnel analytics semantics, and no actionable regression was identified. The changes are limited to AI-facing descriptions and evaluation guidance; the asserted goal denominator, completion population, funnel visitor counts, and date-range behavior agree with the underlying implementations. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart LR
PV[Website page-view visitors matching non-event filters] --> GE[Goal entrants]
GV[Visitors matching the complete goal definition] --> GC[Goal completions]
GE --> GR[Goal conversion: completions / entrants]
FS[Visitors reaching first funnel step] --> FE[Funnel entrants]
FE --> FO[Visitors reaching every ordered step]
FO --> FC[Funnel completions]
FE --> SR[Referrer breakdown for one date range]
FC --> SR
Reviews (1): Last reviewed commit: "fix(insights): clarify measured goal and..." | Re-trigger Greptile |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 4 files
Architecture diagram
sequenceDiagram
participant Agent as Insights Agent (agent.ts)
participant FunnelTools as Funnel Tools (funnels.ts)
participant GoalTools as Goal Tools (goals.ts)
participant Eval as Eval README (evals/)
participant Model as LLM (gpt-5.6-terra)
Note over Agent,Model: Runtime system flow with clarified population semantics
Agent->>Model: Process insights task
Agent->>Agent: Load investigation instructions (goal/funnel context)
alt Goal signal detected
Agent->>GoalTools: invoke goal analytics
GoalTools->>GoalTools: Parse native counts (entrants/completed)
Note over GoalTools: total_users_entered excludes event_name<br/>total_users_completed = visitors matching goal
GoalTools-->>Agent: Return distinct visitor counts + conversion rate
Agent->>Model: Interpret as site-to-goal conversion (not login)
else Funnel signal detected
Agent->>FunnelTools: invoke funnel analytics
FunnelTools->>FunnelTools: Count distinct visitors per ordered step
Note over FunnelTools: Entrants = first step<br/>Completions = all ordered steps
FunnelTools-->>Agent: Return visitor counts (not projects/attempts)
alt Required for ratio analysis
Agent->>FunnelTools: invoke referrer breakdown
Note over FunnelTools: Accepts single date range only
Agent->>FunnelTools: Separate call for previous period
FunnelTools-->>Agent: Source-grouped visitor counts
end
end
Agent->>Agent: Busy comparison windows (current/previous)
Agent->>Model: Draft brief with new guidance
Note over Agent,Model: Correct numbers must attach to<br/>measured population (distinct visitors)
alt Review process (eval)
Agent->>Eval: Load cohort review guidance
Note over Eval: Verify headline population, not just rubric success
Eval-->>Agent: Manual check: goal denominator vs intended audience
end
Agent->>Model: Final publication decision
Note over Agent,Model: Route audience ≠ measured cohort<br/>Prior activity ≠ current loss
Shadow auto-approve: would not auto-approve because issues were found.
Re-trigger cubic
|
The latest updates on your projects. Learn more about Unkey Deploy
|
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Shadow auto-approve: would auto-approve. Clarifies goal/funnel tool descriptions and eval guidance so counts attach to the correct visitor population; prompts and docs only, no code, schema, or runtime behavior changed, validated by evals.
Re-trigger cubic
1 participant
Goal and funnel briefs can attach valid counts to the wrong population: a workspace route's authenticated audience becomes the goal denominator, or a visitor funnel becomes a project count. Clarify the native visitor semantics and date-range behavior in the tool descriptions and investigation instructions, while trimming repeated guidance. Executable code, schemas, tools, turn limits and storage are unchanged; runtime source line count is unchanged.
Fresh synthetic comparison on production gpt-5.6-terra: 30 baseline / 30 candidate runs, plus 48 retained runs of rejected variants. Those 108 runs and 235 observable steps were reviewed. On that matched comparison (before the final review wording clarification), turns were 68 → 60, correction attempts 5 → 0, input tokens 454,522 → 390,998, and average brief length 49.2 → 46.3 words. Summed run time was 267.8 → 256.1 seconds; this small sample does not establish a durable latency gain.
The product gain is narrower, clearer cohort wording with fewer retries. Important limitations remain: all three final revenue cases still omitted an independent attribution decline, one funnel summary still described project volume, and empty category-filtered retention searches still led to overly broad capability conclusions. Both final tracking-gap briefs exceeded the 60-word target (64/65); other final briefs stayed within it. Rejected prompt/schema rewrites increased verbosity or redundant reads and are excluded from this diff.
Validation: root
bun run lint, rootbun run check-types(33 tasks), and relevant tests (361 insights, 573 AI) pass. An initial test attempt ran before the SDK build and failed to resolve its dist entry; the normal root test pipeline subsequently built dependencies and passed. No live customer data, writes, or notifications were used in evals. Local audit retains prompts, calls/results, rejected drafts, corrections, persisted outcomes, source revisions and per-run review notes; private reasoning is omitted.Scope: investigation prompt and native goal/funnel tool descriptions, with eval review guidance. No dependencies on open PRs and no overlap with #680's bulk goal analytics implementation. AI assistance disclosed; maintainer contribution under the repository's policy exemption.
Review follow-up: the funnel instruction now explicitly distinguishes first-step entrants from all-step completions. Six fresh affected runs on the final code retained the visitor comparisons and repair conditions with zero correction turns; three still exceeded the word target (61/66/66). These limitations are retained rather than counted as full quality passes. Fresh lint, types and 934 relevant tests pass.