feat(insights): persist complete business context and reusable briefs

[izadoesdev]

Investigations previously reused a short homepage excerpt and discarded deeper business context. Setup, commercial access and identity qualifications could disappear, and a memory-provider outage could remove the profile entirely.

Persist scoped original business sources and a reusable business brief in website_business_contexts. Supermemory indexes a derived brief; semantic matches hydrate the current Postgres record. Cold preparation selects up to seven additional pages and compiles once. A failed homepage can still use discovered pages, without retries. Warm reads reuse Postgres. Transfers/deletion invalidate the scope; native investigation page reads are retained with revision checks.

The compiler produces concise claims and selects bounded passage IDs; code attaches exact original text, eliminating quote-copy and repair loops. Every supporting passage must remain valid or the whole claim is omitted. Astra compiles the cached brief; Terra still selects pages and runs investigations. Investigators receive the original sources plus concise orientation, with no duplicate quote payload. Original sources remain necessary for deciding qualifications.

Validation:

• Root lint, all 33 workspace typechecks and the full 27-task test suite pass. Native Postgres profile flow/store checks pass (25 and 17 tests), covering citation validity, missing qualifications, concurrent refresh, indexing and homepage recovery.
• 59 fresh diagnostic model calls compare the original compiler, sourced-claim formats, passage formats and models. With identical final source/prompt/corpus, Terra scored 20/24 business decisions with 14/18 attached-claim support and 548 readable words. Two Astra builds scored 24/24 and 22/24, with every reviewed claim supported by its own excerpts (24/24 and 20/20), in 412/389 words. These are small, unblinded diagnostic samples; one answer still omitted logout guidance. This is not approval to use the brief alone.
• Fourteen native investigator runs preserve useful publication and quiet controls, but do not demonstrate a general efficiency gain. Semantic-source citation gaps and redundant discovery remain recorded in the audit.
• Final public-site/Postgres/Supermemory canary used native default routing, eight pages and two model calls: 58.7-second cold preparation, 425-word saved brief, 11.1-ms fresh-process reuse with provider keys removed and external fetch blocked. Exact completed indexing and native semantic hydration were verified. Failed homepage attempts are retained. Only public pages and isolated task scopes were used; task scopes are retired after verification.

Scope: profile preparation, persistence, source retention and lifecycle tests. Candidate selection and source-reader improvements landed independently in #753/#754. No billing-rule or customer-delivery change in this PR. No unmerged dependency.

Hold merge: apply the additive table before deploying. The verified application/root connection roles lack CREATE on public; the prior attempt failed SQLSTATE 42501 and rolled back. A migration-capable database connection is required. No worker deployment was initiated. Schema SQL, exact evaluated sources, every observable model/tool step, failures, usage and manual reviews are preserved in the local audit archive. Hidden reasoning is omitted. AI-assisted maintainer-directed implementation and independent review.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
dashboard	Ready	Preview	Sep 8, 2026 11:36am UTC
databuddy-status	Ready	Preview	Sep 8, 2026 11:36am UTC
documentation	Ready	Preview	Sep 8, 2026 11:36am UTC

[unkey-deploy]

The latest updates on your projects. Learn more about Unkey Deploy

Name	Status	Preview	Inspect	Updated (UTC)
links (preview)	Ready	Visit Preview	Inspect	Sep 8, 2026 11:35am

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 37d89649-dee2-490f-9bae-9bf9d5d28a70

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[greptile-apps]

Greptile Summary

This PR introduces durable, scope-bound PostgreSQL business profiles, compiles and indexes reusable briefs, retains pages discovered during native investigations, and invalidates persisted context during website lifecycle changes.

• Adds the website_business_contexts schema and optimistic-revision storage service.
• Adds cold profile discovery/compilation and warm PostgreSQL reuse.
• Changes Supermemory recall to hydrate canonical PostgreSQL evidence.
• Retains successful investigation page reads and adds persistence, concurrency, lifecycle, and CI coverage.
• The refresh-claim representation can temporarily expose an incomplete profile to concurrent investigations, and compilation currently bypasses the declared aggregate source budget.

Confidence Score: 4/5

The PR should not merge until concurrent refreshes stop exposing incomplete canonical profiles and the explicit repository-rule violations are corrected.

Refresh ownership is represented by overwriting the canonical profile with a brief-less claim, allowing an independent concurrent preparation to consume degraded or unavailable context; the remaining findings concern source-budget enforcement and two mandatory style rules.

Files Needing Attention: apps/insights/src/business-profile.ts, apps/insights/src/business-page-retention.test.ts, apps/insights/src/business-profile.integration.test.ts

Important Files Changed

Filename	Overview
apps/insights/src/business-profile.ts	Implements durable profile refresh, compilation, indexing, and page retention; refresh claims expose incomplete context concurrently and compilation lacks its aggregate source bound.
packages/services/src/business-profile.ts	Adds scope-validated PostgreSQL loading, optimistic writes, locking, and index-revision acknowledgement.
packages/db/src/drizzle/schema/business-context.ts	Defines the additive website-scoped business-context table with cascades and positive revision enforcement.
packages/ai/src/lib/business-context.ts	Moves shared context schemas and changes semantic recall to use Supermemory results as hints for canonical PostgreSQL evidence.
apps/insights/src/agent.ts	Captures successful native scrape results and flushes them to durable business context after the investigation.
packages/services/src/websites.ts	Atomically invalidates durable profiles on scope changes and website deletion.
packages/ai/src/ai/tools/scrape-page.ts	Adds bounded Firecrawl site-map discovery and exports the validated page-result schema.
.github/workflows/ci.yml	Runs both new PostgreSQL business-profile integration suites against the isolated CI database.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  I[Investigation preparation] --> L[Load canonical PostgreSQL profile]
  L -->|Fresh| W[Return warm sources and brief]
  L -->|Refresh required| C[Write refresh claim]
  C --> H[Read homepage and site map]
  H --> P[Select additional pages]
  P --> B[Compile quotation-based brief]
  B --> S[CAS-save complete profile]
  S --> M[Index derived brief in Supermemory]
  M --> W
  R[Semantic recall] --> M
  R --> L
  A[Native investigation page reads] --> K[Retain bounded newer pages]
  K --> L
  X[Transfer, domain change, or deletion] --> D[Invalidate durable profile]
  D --> T[Retire scoped remote memory]

Loading

_{Reviews (1): Last reviewed commit: "feat(insights): persist full business co..." | Re-trigger Greptile}