fix(insights): ground publication decisions in cited evidence

SPEC.md

@@ -99,7 +99,7 @@ locks. Legacy replies without an original scope
 remain history rather than being relabeled as current business facts. Scope changes
 during execution reject the old outcome before persistence.
 
-Tools are discoverable. There is no fixed first query, query family, receipt choreography, or two-read limit. Each investigation uses one tool loop with at most eight model turns, including a reserved final turn. It ends through `finish_investigation`, which validates the outcome and returns any repair error in the same conversation; at most three finish attempts are allowed. Successful reads include exact citation references. The agent does not restart the conversation to repair output.
+Scheduled goal/funnel recovery checks and explicit Apply verification replies are deterministic: one exact native read verifies the saved subject, population, definition, full window, minimum sample and threshold without a model call. Code renders the result and keeps inconclusive checks private. An unfinished window preserves the case and saved check until midnight UTC after its inclusive end date; a completed check resolves without inventing another repair. Free-form human replies retain the investigation agent and their supplied context. Unsupported legacy population checks remain inconclusive without an aggregate read. Read inputs, results and failures remain observable. Other investigations use discoverable tools. There is no fixed first query, query family, receipt choreography, or two-read limit. Each investigation uses one tool loop with at most eight model turns, including a reserved final turn. It ends through `finish_investigation`, which validates the outcome and returns any repair error in the same conversation; at most three finish attempts are allowed. Supplied evidence and successful reads include exact citation references. Sufficient supplied evidence can finish immediately; requested reads must complete before finishing. The finish tool asks for sources and claims before the publication decision; code renders structured revenue claims and validates every claim against those sources before storing the existing text outcome. The agent does not restart the conversation to repair output.
 
 Native `revenue_overview` evidence selects a currency and metric fields from exact successful result references. Code renders labels, values, units, dates and differences for complete equal-duration comparison windows with the same website, timezone and filters, including fresh windows on a later recheck. The stored evidence remains text. This binds those numeric comparisons; other sources retain numeric grounding checks and every finding still needs semantic quality review.
 

apps/dashboard/app/(main)/insights/[id]/page.tsx

@@ -774,7 +774,7 @@ function nextCopy(
 				label: "Measuring",
 			};
 		case "resolve":
-			return { body: next.reason, label: "Verified" };
+			return { body: next.reason, label: "Conclusion" };
 		default:
 			throw new Error("Unknown investigation outcome");
 	}

apps/insights/src/evals/README.md

@@ -22,9 +22,9 @@ The checks cover signal-only evidence, a verified collection gap, a useful produ
 
 Usefulness checks require missing-access-only notices to remain private, retain a verified decline despite unavailable diagnostics, and retain its steady-arrival comparison. The source-comparison case reuses the production funnel tool input contracts with synthetic per-period and combined responses; the agent must inspect the two periods separately to locate the decline. Published briefs are measured against a 60-word budget across title, summary, cause, and evidence (plus impact for baseline/legacy outcomes); action details are excluded. Review brevity alongside retained information, not as a substitute for usefulness.
 
-Source interpretation requires manual review against the tool results. The automatic checks detect missing period reads and an omitted source cohort; a source-name or number match cannot establish a correct comparison. The runner records `reviewRequired` and prints `REVIEW REQUIRED` for a mechanically valid source case. A zero exit status means no automatic check failed; it does not complete that review. Verify direction, cohort, and period attribution, accepting equivalent measured rates. The 60-word budget is a quality target, not a runtime publication gate.
+Source interpretation requires manual review against the tool results. The automatic checks detect missing period reads and an omitted source cohort; a source-name or number match cannot establish a correct comparison. The runner records `reviewRequired` and prints `REVIEW REQUIRED` for a mechanically valid source case. Exact goal-read checks accept omitted or null cohort selectors while rejecting changed populations, websites and windows. A zero exit status means no automatic check failed; it does not complete that review. Verify direction, cohort, and period attribution, accepting equivalent measured rates. The 60-word budget is a quality target, not a runtime publication gate.
 
-The agent uses one native tool loop, ending with `finish_investigation` in a separate turn after receiving its reads. It cannot cite a read sent in the same batch, because the model has not seen that result; the validation error asks it to use the completed result next turn without repeating the read. Schema and evidence validation failures return as tool errors in the same conversation. The total budget is eight model turns, with the last reserved for finishing, and at most three finish attempts. An empty or text-only provider response fails the run without starting another conversation. Read results supply exact citation references to copy; failed reads supply none. The reported read-call count excludes the finish tool; JSONL model requests and step events record all turns and finish attempts. Compare both counts when evaluating efficiency.
+The agent uses one native tool loop and can finish immediately when supplied evidence is sufficient. Supplied evidence includes exact citation references. When a read is needed, the agent waits for its result before calling `finish_investigation`. It cannot cite a read sent in the same batch, because the model has not seen that result; the validation error asks it to use the completed result next turn without repeating the read. The finish tool accepts one or two evidence objects, each containing a `claim` and one to eight `sources`; sources precede their claim, and evidence precedes the publication decision. The model no longer aligns separate claim and citation arrays. Revenue claims retain their native `{currency, fields}` selection. Stored evidence remains text. Schema and evidence validation failures return as tool errors in the same conversation. The total budget is eight model turns, with the last reserved for finishing, and at most three finish attempts. An empty or text-only provider response fails the run without starting another conversation. Read results supply exact citation references to copy; failed reads supply none. The reported read-call count excludes the finish tool; JSONL model requests and step events record all turns and finish attempts. Compare both counts when evaluating efficiency.
 
 New model outputs omit the separate impact paragraph; the finding summary states the consequence. Stored legacy impact remains readable and new stored outcomes default it to null. Review each executed JSONL, including failed/intermediate drafts, before claiming an improvement. Keep the exact source revisions and fixture versions with the comparison; changing a fixture is not an agent improvement.
 
@@ -47,3 +47,5 @@ Cohort review must distinguish a goal's website page-view denominator from its i
 Prompt compression needs fresh repair and verification controls, not only the targeted case. Retain rejected variants: shortened instructions have produced extra definition lookups, pooled period reads, longer briefs, and dropped attribution facts. Evaluate these separately from rubric success. A useful refund finding still omits depth when an independently returned attribution decline is discarded; an empty category-filtered retention search does not establish catalog-wide absence.
 
 Native revenue cases (`revenue-native-decline`, `revenue-native-stale`, `revenue-native-unavailable`) run the real detector and signal preparation before the agent, recording detector requests, synthetic responses, and prepared inputs in `case.setup`. This catches publication failures hidden by hand-built event subjects. Compare the same fixture source in separate checkouts. Revenue detection and rechecks match canonical currency rows across both windows; an absent row is inconclusive, not zero, and legacy unscoped revenue signals cannot be remeasured safely. The standalone runner builds native detector cases once; importing qualityCases builds only static fixtures and does not execute detection. Publication requires a product_outcome with successful, field-bound evidence for the exact scoped signal currency; legacy unscoped keys are rejected. These cases do not exercise real database latency or establish refund/attribution detection when gross revenue is unchanged.
+
+Scheduled structured recovery checks and explicitly marked verification requests use one native analytics read and no model call. Free-form human replies retain the model, including new corrections; the resume boundary marks only the exact shared Apply reply template as verification intent. An identical human-authored template requests the same read-only operation; it does not establish trusted provenance. Audit `verificationRead` in the result, including failed or mismatched reads; zero model steps is expected, not missing telemetry. Assert exact requested and returned subject/population/definition/window, sample and threshold, zero token usage, and no new repair. Compare deterministic output against the original observed model outcomes. A population mismatch is an inconclusive verification, not permission to invent a repair or discard a rejected structured check. An unfinished window returns a quiet watch at midnight UTC after its inclusive end, preserving the check for the next scheduled run. Production events retain the exact read inputs and outputs or failures, correlated by call ID. Legacy reported repairs without a structured check still use the investigation agent.

apps/insights/src/evals/quality.test.ts

@@ -311,6 +311,57 @@ const holdoutOutcome: InsightAgentResult = {
 	},
 };
 
+it("does not excuse a new repair when a saved verification has population drift", () => {
+ const fixture = qualityCases.find((entry) => entry.id === "check-population-drift");
+ const previous = fixture?.input.history.find((entry) => entry.kind === "investigation");
+ if (!fixture || previous?.kind !== "investigation") throw new Error("Missing population drift evaluation");
+ expect(fixture.check({...holdoutOutcome, outcome: previous.outcome}, [])).toContain("Repeated the already-applied definition repair");
+});
+
+it.each([undefined, null])(
+	"accepts an unscoped native goal read with cohort %s while rejecting scope drift",
+	async (cohort) => {
+		const fixture = qualityCases.find(
+			(entry) => entry.id === "current-goal-unchanged"
+		);
+		const read = fixture?.tools.get_goal_analytics;
+		if (!(fixture && read?.execute && read.inputSchema instanceof z.ZodType))
+			throw new Error("Missing native goal evaluation");
+		const query = read.inputSchema.parse({
+			goalId: fixture.input.signal.entity.id,
+			startDate: fixture.input.signal.period.current.from,
+			endDate: fixture.input.signal.period.current.to,
+			...(cohort === null ? { cohort } : {}),
+		});
+		const call = {
+			name: "get_goal_analytics",
+			input: query,
+			output: await read.execute(query, { toolCallId: "goal", messages: [] }),
+		};
+		const result = {
+			...holdoutOutcome,
+			outcome: {
+				...holdoutOutcome.outcome,
+				publish: false,
+				publicationBasis: null,
+			},
+		};
+		expect(fixture.check(result, [call])).toEqual([]);
+		for (const change of [
+			{ goalId: "other-goal" },
+			{ websiteId: "other-site" },
+			{ startDate: "2026-09-01" },
+			{ cohort: { country: "US" } },
+		]) {
+			expect(
+				fixture.check(result, [{ ...call, input: { ...query, ...change } }])
+			).toEqual([
+				`Did not remeasure the exact goal for ${fixture.input.signal.period.current.from}–${fixture.input.signal.period.current.to}`,
+			]);
+		}
+	}
+);
+
 it.each([
 	false,
 	true,
@@ -345,6 +396,12 @@ it.each([
 		})
 		.parse(await read.execute(query, { toolCallId: "holdout", messages: [] }));
 	const readings = Object.values(output.results);
+	const sources = Object.keys(output.results).map((resultKey) => ({
+		source: "tool",
+		name: "get_data",
+		toolCallId: "holdout",
+		resultKey,
+	}));
 	expect(Object.keys(readings[0].data[0])[0]).toBe(
 		reordered ? "attributed_revenue" : "currency"
 	);
@@ -360,7 +417,7 @@ it.each([
 		const failures = fixture.check(
 			{ ...holdoutOutcome, outcome: { ...holdoutOutcome.outcome, evidence } },
 			[],
-			{ evidence: [selection] }
+			{ evidence: [{ claim: selection, sources }] }
 		);
 		expect(failures).toHaveLength(omitted ? 2 : 0);
 		if (omitted)
@@ -379,7 +436,7 @@ it.each([
 				outcome: { ...holdoutOutcome.outcome, evidence: [swapped] },
 			},
 			[],
-			{ evidence: [selection] }
+			{ evidence: [{ claim: selection, sources }] }
 		)
 	).toEqual(["Rendered evidence omitted Gross Revenue: 12,000 → 12,000"]);
 });

apps/insights/src/evals/quality.ts

@@ -1071,6 +1071,7 @@ for (const scenario of [
 			request: original.input.request
 				? {
 						...original.input.request,
+						kind: "verification",
 						createdAt:
 							scenario === "unfinished-window"
 								? "2026-09-04T12:00:00Z"
@@ -1139,12 +1140,7 @@ for (const scenario of [
 		},
 		reviewRequired: `Expected ${status}. Check that the customer copy agrees with the code verdict and preserves the reason, exact dates, measured count and threshold. A small sample or unfinished window cannot prove recovery.`,
 		check: (result, calls) => [
-			...original.check(result, calls).filter(
-				(failure) =>
-					// A new population mismatch can justify a different repair.
-					scenario !== "population-drift" ||
-					failure !== "Repeated the already-applied definition repair"
-			),
+			...original.check(result, calls),
 			...(result.outcome.verification?.status === status
 				? []
 				: [`Expected persisted verification status ${status}`]),
@@ -1251,16 +1247,15 @@ for (const scenario of [
 				calls.some(
 					(call) =>
 						call.name === "get_goal_analytics" &&
-						isDeepStrictEqual(call.input, {
-							startDate: window.from,
-							endDate: window.to,
-							goalId: goal.id,
-							...(call.input &&
-							typeof call.input === "object" &&
-							"websiteId" in call.input
-								? { websiteId: appContext.websiteId }
-								: {}),
-						})
+						z
+							.strictObject({
+								startDate: z.literal(window.from),
+								endDate: z.literal(window.to),
+								goalId: z.literal(goal.id),
+								websiteId: z.literal(appContext.websiteId).optional(),
+								cohort: z.null().optional(),
+							})
+							.safeParse(call.input).success
 				)
 					? []
 					: [`Did not remeasure the exact goal for ${window.from}–${window.to}`]
@@ -1821,17 +1816,19 @@ for (const reordered of [false, true]) {
 			const selection = z
 				.object({
 					evidence: z.array(
-						z.union([
-							z.string(),
-							z.object({ currency: z.string(), fields: z.array(z.string()) }),
-						])
+						z.object({
+							claim: z.union([
+								z.string(),
+								z.object({ currency: z.string(), fields: z.array(z.string()) }),
+							]),
+						})
 					),
 				})
 				.safeParse(acceptedFinish);
 			const fields = selection.success
-				? selection.data.evidence.flatMap((entry) =>
-						typeof entry !== "string" && entry.currency === "USD"
-							? entry.fields
+				? selection.data.evidence.flatMap(({ claim }) =>
+						typeof claim !== "string" && claim.currency === "USD"
+							? claim.fields
 							: []
 					)
 				: [];