fix(insights): ground publication decisions in cited evidence - #778
Conversation
|
The latest updates on your projects. Learn more about Unkey Deploy
|
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: ASSERTIVE Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
izadoesdev
changed the title
|
@greptileai Please review the complete current diff at 9235c19, including claim/source normalization, publication guidance, coverage formatting, and unchanged runtime guards. This is an explicit request for review of the final code head; fresh heldout semantic evaluation remains a separate merge gate. |
Greptile SummaryThis PR grounds investigation publication decisions in claim-bound citations and replaces scheduled or explicit Apply recovery checks with deterministic native analytics verification.
Confidence Score: 5/5The PR appears safe to merge with no actionable new defects identified. The deterministic verification path validates exact measurement metadata and thresholds, preserves future checks as watches, keeps inconclusive outcomes private, and prevents unsupported repairs; claim-bound citations remain compatible with the stored text outcome. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Scheduled recheck or reply] --> B{Explicit Apply verification or no human request?}
B -- No --> C[Investigation agent]
C --> D[Claim-bound evidence and sources]
D --> E[Validate publication decision]
B -- Yes --> F[Load saved goal or funnel check]
F --> G[One native analytics read]
G --> H[Validate subject, definition, population, window, sample, threshold]
H --> I{Window finished?}
I -- No --> J[Keep case open as watch]
I -- Yes --> K[Passed, failed, or inconclusive conclusion]
K --> L[Resolve without creating a new repair]
Reviews (2): Last reviewed commit: "fix(insights): verify saved recovery che..." | Re-trigger Greptile |
|
Holding merge after the final semantic audit despite green CI and code review. The candidate improved common-checker passes (48/52 vs 38/52) and reduced model turns (106 vs 127), but one population-mismatch verification invented an unsupported manual repair after dropping its rejected structured check. I am replacing model-authored saved-check conclusions with native exact-read verification and deterministic output, then rerunning the affected cases before requesting fresh review. Original runs and failures remain preserved. |
|
@greptileai Please review final head 065a527. The previous reviewed head was deliberately held after output audit found an unsupported verification repair. This revision adds deterministic scheduled/Apply checks, preserves future checks through the watch lifecycle, retains free-form human replies, records native read diagnostics, and rejects unsupported manual repairs from inconclusive checks. Please review the complete current diff, especially the agent/resume/persistence boundary and shared Apply reply template. All comments and review threads must be addressed before merge. |
1 participant
Problem and behavior
Investigations could decide to publish before organizing their evidence, lose facts while repairing citations, and turn an inconclusive saved recovery check into an unsupported new repair. This change pairs each claim with its sources and aligns the finish schema with the evidence-first publication policy.
Scheduled goal/funnel checks and explicit Apply verification replies now perform one exact native analytics read with zero model calls. Code validates the returned subject, definition, population, full UTC window, sample and threshold; it reports passed, failed or inconclusive without inventing a cause or repair. Future windows preserve an open case and the saved check until midnight UTC after the inclusive end. Free-form human replies still reach the agent with their context. Inconclusive checks alone cannot justify a new manual repair; independently inspected implementation evidence is required. Native read inputs, results and failures are correlated in production logs.
Retention evidence supplies the measured identity percentage beside its original counts. The dashboard uses “Conclusion” for a closed outcome, which can be inconclusive. Stored contracts remain compatible; no database migration.
Evaluation and review
Known limits: this is synthetic evidence, not proof the service warrants its subscription price. General investigation outputs can still omit low-identity caveats or currency controls, publish undersampled requested cohorts, and reject valid abbreviated dates during numeric grounding. These are retained findings, not reported as solved. Production source LoC increases to remove model work; no LoC reduction is claimed.
Scope and overlap
One investigation evidence/verification slice based on current staging; no PR dependency. Draft #751 overlaps agent.ts and SPEC.md by filename and will need to reconcile those surfaces independently. AI-assisted maintainer implementation, manually reviewed with observable input/tool/output audits.