feat(insights): charge $1 per completed investigation

.agents/skills/databuddy-internal/SKILL.md

@@ -30,6 +30,7 @@ Keep additions **minimal**: one bullet, a new `rg` hint, or a routing note—eno
 - Local E2E dashboard smokes that need `/api/test/e2e/*` should start the API/dashboard directly (or through Playwright's webServer command), not via `bun run dev:dashboard`; Turbo runs in strict env mode and drops `DATABUDDY_E2E_MODE`/`DATABUDDY_E2E_TEST_KEY` unless they are added to `turbo.json` `globalEnv`.
 - Dashboard Playwright public/demo analytics specs call API `/v1/query` anonymously from the browser; keep `DATABUDDY_E2E_MODE` query behavior isolated from production rate limits so CI retries do not exhaust `anon:unknown`.
 - `apps/api`: Elysia API on port `3001`
+- API tests use Vitest through `bun run test` inside `apps/api`; use Vitest test imports rather than `bun:test` in that package.
 - Public REST docs live in `apps/api/src/rpc/openapi.ts`: `/spec.json` is the generated spec, `/` is the reference UI, and hiding a router there also makes its top-level REST paths return 404 because `/*` uses the same filtered docs router.
 - `apps/slack`: Slack agent adapter; Slack installs resolve through org-scoped DB integration records, not a single env bot token/default website. Agent calls use the org-scoped internal principal synthesized from the active integration in `slack/installations.ts`, never a global internal secret.
 - Slack OAuth lives in `apps/api`, but slash commands/events require `apps/slack` to be running too; local `bun run dev:dashboard` runs dashboard + API only, so use `bun run dev:slack` when working on Slack. The Slack package scripts read the root `.env`.
@@ -45,7 +46,7 @@ Keep additions **minimal**: one bullet, a new `rg` hint, or a routing note—eno
 - `SPEC.md` is the intelligence product contract. `insight_observations` is the readable Insights history; `analytics_insights` is the durable investigation projection. The agent outcome owns brief publication and `act`/`ask` promotion; do not replace either with frontend heuristics or collapse the feed into cases. Do not add a parallel agent, evidence API, fixed query choreography, or action-specific lifecycle.
 - Insights quality reviews must compare fresh baseline/candidate outputs and lead with the product verdict and concrete examples. Score usefulness, noise, reading effort, and retained useful findings separately from code tests and contract passes; preserve interrupted attempts instead of reporting retries as an uninterrupted pass rate.
 - Insights RPC helpers that take `{ context, ...input }` must strip `context` before parsing a `.strict()` Zod input schema (same pattern as `appendInvestigationReply` / `applyInsightGoalAction`); otherwise CI fails with `Unrecognized key: "context"`.
-- `insights.history` / MCP `list_investigations` hide cases while a reply is `queued`/`running` (action-inbox verification); tests must list before reply or expect an empty list while verifying.
+- `insights.history` / MCP `list_investigations` hide cases while analysis or verification is queued/running; included clarifications use saved evidence and must not hide or mutate the case.
 - When reporting what an organization can see in Insights, follow the `insights.brief`/`history` visibility rules instead of counting `analytics_insights`; the projection can contain legacy rows without a readable or published `insight_observations` turn.
 - Production insight shadows must freeze `--reference-time`, retain a tool-name trace, and pass available GitHub context before supporting quality claims. Postgres and ClickHouse are read-only, but connector token refreshes or cache writes can still occur; never describe the whole run as zero-write.
 - Automatic investigations have one organization-wide schedule (`off`, `daily`, or `weekly`) and one organization-wide delivery set; website selection is only for manual runs. Do not reintroduce per-website overrides, hourly/custom cadence, or cron input.
@@ -167,7 +168,8 @@ Read [codebase-map.md](./references/codebase-map.md) when you need deeper routin
 - Start in `apps/api/src`
 - Shared API contracts and procedure logic live in `packages/rpc`
 - Prefer changing shared router logic in `packages/rpc` rather than duplicating validation in the dashboard
-- Investigations run in `apps/insights`; RPC only reads cases and accepts durable replies. Case identity is `websiteId|subjectKey`, where the backend owns the subject key. Persist a new observation for each turn while updating the existing insight row. The stored `changePercent` is already signed.
+- Saved investigation tool evidence must use typed, positive field allowlists; do not persist arbitrary tool outputs or rely on generic secret-pattern redaction. Preserve exact measurement scope, and record omissions instead of reconstructing missing raw evidence.
+- Investigations run in `apps/insights`; RPC only reads cases and accepts durable replies. Case identity is `websiteId|subjectKey`, where the backend owns the subject key. New analysis appends an observation; a clarification stores its answer on the reply and reads the originating observation's saved evidence without changing case state. The stored `changePercent` is already signed.
 
 ### Ingestion and analytics pipeline
 
@@ -179,8 +181,8 @@ Read [codebase-map.md](./references/codebase-map.md) when you need deeper routin
 ## Billing (Autumn)
 
 - Retried insight jobs must persist immutable external delivery effects (currently Slack) before calling providers and reuse the effect ID as the provider idempotency key. An insight observation is product memory, not a delivery checkpoint.
-- Intelligence pricing should use the existing token-cost-backed `agent_credits` and top-up flow; do not invent per-site or "monitored product" billing without explicit product selection and runtime enforcement.
-- Transactional billing email identity has three separate concepts: Autumn customer/billing owner, organization, and actual `to` recipient. Only personalize from the actual recipient record; if it is unavailable, omit the greeting rather than using the owner name. Keep `agent_credits` as an internal feature ID, but describe it to customers as investigation credits and explain that deeper investigations, replies, and rechecks can use more credits.
+- Investigations cost $1 per completed result through the separate Autumn `investigation_runs` meter. Clarifications and verification after applying a proposed repair are included. Persist the accepted price with an explicit queued analysis and bind its reservation to that price. Reserve one unit before new analysis and settle only after a readable complete result is persisted; retries reuse durable operation identity. Internal token costs are telemetry. Existing customers without the new entitlement retain legacy `agent_credits` terms; do not convert balances or point legacy credit refills at the new meter.
+- Transactional billing email identity has three separate concepts: Autumn customer/billing owner, organization, and actual `to` recipient. Only personalize from the actual recipient record; if it is unavailable, omit the greeting rather than using the owner name. Distinguish fixed-price investigations from legacy credits in billing copy.
 - `autumn-js` v1.2.2+ — import `autumnHandler` from `autumn-js/fetch` (NOT `autumn-js/elysia`, that export was removed in v1.0)
 - For Elysia, mount with `.mount(autumnHandler(...))` — NOT `.use()`
 - `identify` callback receives `(request: Request)` directly, not `({ request })`
@@ -202,6 +204,7 @@ Read [codebase-map.md](./references/codebase-map.md) when you need deeper routin
 - ClickHouse helpers and schema: `packages/db/src/clickhouse/*`
 - `ch:check` is package-scoped; run `cd packages/db && bun run ch:check`, not the root script runner.
 - After schema changes, use the repo db scripts rather than ad hoc commands
+- PostgreSQL deploys use `packages/db db:push` through `init.Dockerfile`; register new schema files in `packages/db/drizzle.config.ts`. `packages/migrate` transforms SDK source and is not a database migration runner.
 - A shipped ClickHouse table change needs a tracked forward migration alongside
   its reference DDL: bootstrap `CREATE ... IF NOT EXISTS` does not migrate
   deployed tables, and Keeper-path or sort-key changes need a shadow-table

SPEC.md

@@ -33,6 +33,37 @@ An append-only explanation of one signal at one point in time. It names the subj
 
 The durable work object for one signal. It has an `open` or `resolved` state plus observations, replies, actions, rechecks, and recurrence history.
 
+### Investigation price
+
+A completed investigation costs **$1**. The billable unit is one explicitly started
+analysis of a selected signal or new question, not the durable case that may hold
+several analyses over time. A supported measured answer, concrete inspected repair,
+or verified no-action conclusion can complete it. Failed, interrupted, inconclusive
+work and an unanswered necessary question are not completed investigations.
+
+Reserve one investigation before starting new analysis. Confirm that reservation
+only after its complete result is saved and readable; release it when the work is
+incomplete. Persist charge identity and settlement intent with the result so retries
+and recovery reuse the same unit. Uncertain payment-provider responses remain
+pending for reconciliation rather than starting a second charge.
+
+Clarifications of the same question use its saved evidence and are included.
+Verification after applying that investigation's proposed repair is also included,
+as are backend-triggered definition-change checks and deterministic continuations
+of saved verification conditions during regular scans. A new question or separate
+fresh analysis requires an explicit accepted price persisted with its queued reply;
+the reservation must match those immutable terms. The model must never decide
+whether a reply incurs a charge. Signal selection,
+preparation, model turns, and internal retries do not add customer charges.
+
+Autumn stores the new unit in a separate `investigation_runs` balance with a $1
+prepaid purchase option. Existing credit balances, credit refills, and attached
+legacy plans retain their terms until the customer adopts the new entitlement
+through an investigation purchase or a switch to a new plan version.
+An exhausted fixed-price balance does not fall back to spending legacy credits.
+Chat continues to use credits. Token usage and model costs remain internal
+telemetry for fixed-price investigations and included replies.
+
 ### Action
 
 An optional proposed change with a target and verification condition. A code action may become a patch and PR. Other actions may target tracking, a goal, a campaign, configuration, or operations.
@@ -131,6 +162,14 @@ The Insights brief reads like a short news report: headline, what happened, why
 ## Continuity
 
 - A dashboard, Slack, or MCP reply resumes the same investigation.
+- A clarification is anchored to the original observation and typed, allowlisted
+  goal/funnel measurement fields, with trusted descriptions and exact scope. Raw
+  profiles, sessions, source files, search queries, arbitrary properties and free-form
+  context are omitted with explicit limitations. Retained evidence survives history
+  truncation and later reopening of the same case.
+  The answer is stored on the reply without new data reads or case-state changes.
+  Legacy results without saved evidence receive an honest explanation of that
+  limitation; answering them never silently starts paid analysis.
 - A GitHub comment or review resumes the agent working on that PR.
 - A materially worse resolved signal reopens the same investigation with its prior outcomes.
 - Corrections such as terminology, ownership, or known infrastructure become project memory.
@@ -174,6 +213,6 @@ When business meaning is missing, inspect the definition, site, events, and conn
 
 ## Implementation constraint
 
-Use `insight_observations` as the append-only Insights source and `analytics_insights` as the current investigation projection. An `act` or `ask` creates or reopens that projection; `resolve` may update an open investigation but never creates or reopens one. Recommendations are a read projection of the latest observation for each signal: standalone setup and measurement recommendations expire at their recheck time unless renewed, while definition recommendations also verify against the current definition. Keep one agent and one evidence/tool stack. Add storage only when this model cannot represent a real use case.
+Use `insight_observations` as the append-only Insights source and `analytics_insights` as the current investigation projection. An `act` or `ask` creates or reopens that projection. A complete fixed-price result may create a resolved projection so its paid answer remains readable even when no action is needed; it does not create an interruption or reopen work. Other `resolve` outcomes may update an open investigation but never create or reopen one. Recommendations are a read projection of the latest observation for each signal: standalone setup and measurement recommendations expire at their recheck time unless renewed, while definition recommendations also verify against the current definition. Keep one agent and one evidence/tool stack. Add storage only when this model cannot represent a real use case.
 
 Exact error-customer joins run as a private, aggregate-only enrichment after the backend selects a signal. They return counts and coverage, never visitor, profile, session, payment, order, or request identifiers. Identity joins report same-window resolution explicitly; attributed completed-payment matches require the payment to predate the affected profile's first error and remain a lower bound.

apps/api/src/billing/autumn-purchase-boundary.test.ts

@@ -0,0 +1,94 @@
+import type { JSONValue } from "ai";
+import { beforeEach, describe, expect, it, vi } from "vitest";
+
+const { forward } = vi.hoisted(() => ({
+	forward: vi.fn(async (request: Request) =>
+		Response.json(await request.json())
+	),
+}));
+vi.mock("autumn-js/fetch", () => ({ autumnHandler: () => forward }));
+vi.mock("@databuddy/auth", () => ({
+	auth: { api: { getSession: vi.fn(async () => null) } },
+}));
+vi.mock("@databuddy/redis", () => ({ getRedisCache: vi.fn() }));
+vi.mock("@databuddy/rpc", () => ({
+	getBillingCustomerId: vi.fn(),
+	getMemberRole: vi.fn(),
+}));
+
+import { handleAutumnRequest } from "./autumn";
+
+function request(body: JSONValue, contentType: string | null) {
+	const value = new Request("https://synthetic.invalid/autumn/attach", {
+		method: "POST",
+		body: JSON.stringify(body),
+	});
+	if (contentType) {
+		value.headers.set("content-type", contentType);
+	} else {
+		value.headers.delete("content-type");
+	}
+	return value;
+}
+
+beforeEach(() => {
+	forward.mockClear();
+});
+
+describe.each([
+	"application/json",
+	"text/plain",
+	null,
+])("Autumn investigation boundary with %s content type", (contentType) => {
+	it("strips new-feature grants nested in another plan before native forwarding", async () => {
+		const response = await handleAutumnRequest(
+			request(
+				{
+					planId: "pro",
+					customize: {
+						addItems: [{ featureId: "investigation_runs", included: 1000 }],
+					},
+				},
+				contentType
+			)
+		);
+		expect(response.status).toBe(200);
+		expect(await response.json()).toEqual({ planId: "pro" });
+		expect(forward).toHaveBeenCalledTimes(1);
+	});
+
+	it("rejects a fixed-unit quantity override on another plan before forwarding", async () => {
+		const response = await handleAutumnRequest(
+			request(
+				{
+					planId: "pro",
+					featureQuantities: [
+						{ featureId: "investigation_runs", quantity: 1000 },
+					],
+				},
+				contentType
+			)
+		);
+		expect(response.status).toBe(422);
+		expect(forward).not.toHaveBeenCalled();
+	});
+
+	it("forwards the exact whole-unit purchase after removing client checkout URLs", async () => {
+		const purchase = {
+			planId: "investigations_topup",
+			featureQuantities: [{ featureId: "investigation_runs", quantity: 10 }],
+		};
+		const response = await handleAutumnRequest(
+			request(
+				{
+					...purchase,
+					successUrl: "https://synthetic.invalid/billing",
+				},
+				contentType
+			)
+		);
+		expect(response.status).toBe(200);
+		expect(await response.json()).toEqual(purchase);
+		expect(forward).toHaveBeenCalledTimes(1);
+	});
+});

apps/api/src/billing/autumn.ts

@@ -1,3 +1,6 @@
+import type { JSONValue } from "ai";
+import { buildHttpErrorResponse } from "@databuddy/shared/http-error-response";
+import { isInvestigationPurchaseValid } from "./investigation-purchase";
 import { auth } from "@databuddy/auth";
 import { getRedisCache } from "@databuddy/redis";
 import { getBillingCustomerId, getMemberRole } from "@databuddy/rpc";
@@ -22,19 +25,19 @@ const FORBIDDEN_BODY_KEYS = new Set([
 	"prorationBehavior",
 ]);
 
-function sanitize(value: unknown): unknown {
+function sanitize(value: JSONValue): JSONValue {
 	if (Array.isArray(value)) {
 		return value.map(sanitize);
 	}
 	if (!value || typeof value !== "object") {
 		return value;
 	}
-	const out: Record<string, unknown> = {};
+	const out: Record<string, JSONValue | undefined> = {};
 	for (const [key, val] of Object.entries(value)) {
 		if (FORBIDDEN_BODY_KEYS.has(key)) {
 			continue;
 		}
-		out[key] = sanitize(val);
+		out[key] = val === undefined ? undefined : sanitize(val);
 	}
 	return out;
 }
@@ -43,11 +46,8 @@ async function stripPrivilegedBody(request: Request): Promise<Request> {
 	if (request.method === "GET" || request.method === "HEAD") {
 		return request;
 	}
-	const contentType = request.headers.get("content-type") ?? "";
-	if (!contentType.includes("application/json")) {
-		return request;
-	}
-
+	// The native adapter parses JSON regardless of Content-Type. Apply the same
+	// restrictions to text/plain and missing-header requests before forwarding.
 	const text = await request.text();
 	let body: string | null = text || null;
 	if (text) {
@@ -133,6 +133,19 @@ async function writeAutumnCache(
 export async function handleAutumnRequest(request: Request) {
 	const sanitized = await stripPrivilegedBody(request);
 	const segment = autumnPathSegment(sanitized);
+	if (sanitized.method !== "GET" && sanitized.method !== "HEAD") {
+		const body: JSONValue = await sanitized
+			.clone()
+			.json()
+			.catch(() => null);
+		if (!isInvestigationPurchaseValid(body, segment)) {
+			const response = buildHttpErrorResponse({
+				code: "VALIDATION",
+				error: null,
+			});
+			return Response.json(response.payload, { status: response.status });
+		}
+	}
 	const ttlSec = AUTUMN_CACHE_TTL_SEC[segment];
 
 	if (ttlSec === undefined) {