feat(conversations)!: Store conversation content in SQL with visibility-tiered retention by dcramer · Pull Request #805 · getsentry/junior

dcramer · 2026-07-09T17:41:46Z

Conversation content—the visible transcript and model execution history—now lives in the shared Junior SQL database instead of Redis. junior_conversation_messages stores visible messages and delivery marks; junior_agent_steps stores durable execution steps and context epochs. Redis remains the authority only for transient execution state such as mailbox records, leases, wake state, and thread-state scratch.

The SQL format removes structures that existed only to fit Redis:

Compaction and provider-retry rollback open a new integer context epoch with a context_epoch_started marker and ordinary pi_message rows instead of embedding a replacement transcript in projection_reset.
Advisor histories are durable child conversations linked by parent_conversation_id. They use the same step storage and conversation-tree retention as top-level conversations; reporting resolves the child from the parent's subagent steps and applies the normal privacy boundary.
Turn-session records reference the committed SQL seq, pinned to the epoch containing that boundary, instead of storing message-count cursors.
Visible messages are persisted idempotently at each boundary where the working transcript could otherwise be lost: before compaction, during agent dispatch, at thread-state persistence, and after delivery before runtime scratch is saved.

Retention is enforced by the dedicated daily /api/internal/retention cron rather than writer-owned TTLs. Content expires from the root conversation's last_activity_at using its current persisted destination visibility: 90 days for public conversations and 14 days for every other case. Purge deletes message and step rows for the full conversation tree, stamps transcript_purged_at, and scrubs private titles, channel names, and actor JSON while retaining safe metadata for reporting. Immediate erasure uses the same tree-purge primitive through purgeConversation.

Core Junior now uses drizzle-kit as its DDL generator, matching the plugin migration convention. Table definitions live under src/db/schema/ and compose into juniorSqlSchema in src/db/schema.ts; generated snapshots and future SQL migration files live in the published migrations/ directory. junior upgrade remains the sole checksum-pinned migration applier. Existing migrations 0001–0005 stay inline to preserve their recorded checksums; generated migrations start at 0006 and register through defineMigrationFromFile.

Rollout is a hard cutover with no Redis transcript fallback or dual-write authority. junior upgrade bulk-imports legacy Redis session logs, advisor sessions, visible messages, and compaction snapshots into SQL. A temporary lazy import under the conversation lease covers conversations touched by the old deployment during promotion; it is idempotent, commits visible messages and steps together, and is scheduled for removal after the legacy Redis TTL horizon on July 24, 2026.

This intentionally changes private conversation retention from roughly 30 days to 14 days; public conversation retention becomes 90 days. Reporting reads SQL only and distinguishes content expired by retention from content hidden by privacy redaction.

Review specs/conversation-storage.md and specs/terminology.md first, followed by src/db/schema/, chat/conversations/sql/migrations.ts, the storage/projection/retention/import modules under chat/conversations/, then the runtime persistence boundaries, advisor child-conversation handling, and reporting.

vercel · 2026-07-09T17:41:53Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
junior-docs	Ready	Preview, Comment	Jul 10, 2026 3:41pm

Remove Redis turn and conversation-detail reads from reporting so durable SQL conversation metadata, visible messages, and agent steps are the only reporting sources. Also preserve first reply timestamps, complete message-only legacy imports, and purge expired descendant content recursively. Refs GH-805 Co-Authored-By: OpenAI Codex <noreply@openai.com>

Add the one-time Redis-to-SQL conversation import shared by junior upgrade (bulk, bounded newest-first) and a lazy first-read straggler path for conversations the old deployment touched during promotion. The converter maps legacy sessionId markers to integer context epochs, explodes projection_reset entries into context_epoch_started markers plus per-message rows, normalizes v1 actor shapes into provenance, converts advisor transcriptRef blobs into child conversations, and copies thread-state visible messages with replied marks. Timestamps come from message-internal Pi timestamps with conversation-record fallback, never import time. Import is per-conversation all-or-nothing under an advisory lock and idempotent (skip when step rows exist). The lazy seam lives in the projection loaders, runs under the conversation lease the worker already holds, and is deletion-scoped in legacy-import.ts so the follow-up PR can remove it wholesale. Also revert a one-off test timeout in favor of fixing the layer. Refs openspec change sql-conversation-storage (tasks 4.1-4.4). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The visible conversation transcript now persists exclusively in junior_conversation_messages: thread-state stops carrying messages and piMessages entirely and becomes runtime scratch. A single serialization point (toStoredConversationMessage) folds author display facts into meta.author for both the runtime write-through and the legacy import, and record() merges meta key-wise on conflict so partial writers can never erase recorded facts. Replied marks persist as replied_at and rehydrate into the in-memory working set that turn preparation, reply policy, and channel context consume. The local runner, dispatch runner, continue runner, and OAuth callback handlers all read Pi history from the SQL step projection with no thread-state fallback. Unit tests for agent runs are hermetic: @/chat/db is mocked with in-memory step/message stores instead of reaching the real Postgres harness. The junior-qa skill documents OAuth flow QA and the SQL verification contract. Refs openspec change sql-conversation-storage (task 3.4). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… cron Conversation content now expires window(visibility) after last activity: 90 days for persisted-public destinations, 14 days for everything else (fail closed), resolved at purge time through the parent chain to the root so a public-private flip takes effect on the next pass and no expiry is ever stored. A daily /api/internal/retention cron (wired through juniorNitro like the heartbeat cron, but isolated from the repair loop) purges expired root trees in bounded oldest-first batches: one transaction per tree deletes message and step rows for the root and descendants, stamps transcript_purged_at, and for non-public conversations scrubs the raw-payload metadata (title, channel name, legacy actor JSON). The metadata row survives as the dashboard index. purgeConversation is the single-conversation erasure primitive using the same path. Refs openspec change sql-conversation-storage (tasks 5.1-5.5). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Reporting drops its last legacy Redis session-log dependency: activity timelines (tool executions, subagent start/end, nesting) now derive from current-epoch agent step rows, with tool statuses computed from the step projection. Redaction behavior is unchanged. Purged conversations present as expired under retention, distinct from privacy redaction: transcriptExpired/transcriptExpiredAt with empty content and no redaction reason, read through the ConversationStore summary's new transcriptPurgedAtMs field. Expiry takes precedence over redaction so a purged public conversation never re-exposes content. Activity attaches to the newest run only until the junior_agent_turns read model lands (documented inline); per-turn attribution is a declared follow-up. Refs openspec change sql-conversation-storage (tasks 3.6, 3.7). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Remove purgeConversation from the AgentStepStore port. It deleted and stamped but never scrubbed private metadata, making it an erasure trap next to retention's transactional tree purge. The retention module's purgeConversation is the single erasure primitive, matching the spec's assignment of purge to the retention surface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Add eval assertion helpers that read the run's durable content back through the store ports: agentSteps(session) returns validated step rows with seq/epoch/type, conversationMessages(session) returns the visible transcript, and conversationId(session) resolves the run's conversation. A new conversation-storage eval pairs a reply rubric with deterministic assertions that the turn's pi_message steps land in the current epoch in order and its visible messages land with correct roles. Advisor child-conversation assertions are enabled by the helpers but not forced into a model-dependent eval case. Refs openspec change sql-conversation-storage (task 6.1). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The legacy import's idempotence gate keys off existing agent-step rows, but steps were written before visible messages. A failure between the two phases tripped the gate on every retry and permanently skipped the visible transcript. Write messages first — record() and markReplied are idempotent — so the step write is the import's sole commit point and a partial failure retries to a complete import. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Retention purges by last_activity_at, but the message and step stores only lazy-inserted the conversation row and never bumped it, so content recorded outside a turn summary (channel skip markers, step appends) could not refresh the retention window as Redis pExpire-on-append did. The ensureConversation upserts now advance last_activity_at/updated_at via greatest(), so live appends refresh the window while backfilled or imported historical rows can never regress the clock. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…uards Two post-delivery hardening fixes from PR review. The reply executor's SQL transcript persist ran after the commit-error swallow block, so a transient SQL failure after Slack accepted the reply skipped turn completion callbacks instead of being retried and logged; it now runs first inside that block under persistWithRetry. The remaining explicit transcript persists at persistThreadStateById call sites duplicated the write-through and, in the dispatch runner, ran outside the retry-and- swallow guard where a SQL failure could fail an already-delivered dispatch into a duplicate Slack post; the write-through now covers all of them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Hydrate is a first-read boundary, but the Redis-to-SQL lazy import was wired only into step projection reads. Turn dedupe, delivered-message redelivery guards, and channel-context assembly hydrate before any projection read, so promotion-window stragglers could double-reply or rebuild context without their legacy history. hydrateConversationMessages now triggers the idempotent import first; the dynamic import avoids a static cycle with legacy-import's use of the shared serialization helper and keeps the seam deletable at the legacy TTL horizon. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Simplification pass over the sql-conversation-storage branch from a coordinated policy review (delayering, dead code, minimalism, correctness-complexity, interface design). The transcript write-through now has one owner: persistThreadState syncs patch.conversation to SQL the same way persistThreadStateById does, which deletes ten manual call-site pairings and covers five sites that persisted conversation state without flushing the transcript (latent staleness). Extract the shared conversation-row upsert used by both content stores, drop the dead messageStore injection seam, delete loadProjectionWithActor and retentionWindowFor (test-only surface duplicating live logic), un-export internal-only helpers, remove an unreachable timestamp fallback and an unpopulated schemaVersion thread, and compute the commit prefix walk once. Net -119 lines with no behavior change outside the latent-staleness fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The shared Junior database is app-wide, not chat-owned, so the SQL infrastructure boundary (executor, drivers, composed schema) moves from chat/sql to src/db with the canonical table files at src/db/schema — the standard Drizzle layout. Feature layers (chat/db.ts accessors, chat/conversations/sql query implementations) stay put. Core junior now uses drizzle-kit as its DDL generator, matching the plugin db:generate contract: drizzle.config.ts points at the composed schema, the committed drizzle/meta snapshot makes db:generate a no-op parity check against the live schema, and future migrations register through defineMigrationFromFile from kit output. The checksum-pinned junior upgrade runner remains the sole applier with its advisory-lock and expand-only deploy contract; migrations 0001-0005 stay inline and a new test pins their recorded checksums so applied DDL can never silently drift. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… entry Two PR review fixes. The message store refreshed last_activity_at from the first entry of each record() batch, but callers persist the full working set oldest-first, so greatest() never advanced the clock through this path and content could expire on a stale window; both content stores now use the newest entry in the batch. And loadConnectedMcpProviders now triggers the lazy legacy import like the other projection loaders, so a promotion-window straggler cannot lose its durable MCP connection facts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Add a two-turn eval asserting a connected MCP provider is reused on the follow-up turn without re-authorization: exactly one mcp_provider_connected step across the conversation, no authorization_requested after it, and a delivered second reply. This is the e2e guard for the provider-fact read path (the class of bug where loadConnectedMcpProviders missed durable connection history). Skipped for now: a pre-existing MCP auth-link delivery failure on main (verified against origin/main) also breaks the reference MCP-pause case in oauth-workflows.eval.ts — the private link never posts via ephemeral and falls through to conversations.open, which the Slack MSW harness rejects. Unskip with that case once delivery is fixed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Thread conversation state no longer owns Pi messages after the SQL storage cutover. Remove the obsolete fixture field so the integration test matches the durable session and SQL boundaries. Co-Authored-By: OpenAI Codex <noreply@openai.com>

Seed legacy conversation rows with the newest known activity timestamp while keeping intrinsic step timestamps unchanged. This prevents lazy steps-only imports from appearing expired immediately after promotion. Co-Authored-By: OpenAI Codex <noreply@openai.com>

Remove Redis turn and conversation-detail reads from reporting so durable SQL conversation metadata, visible messages, and agent steps are the only reporting sources. Also preserve first reply timestamps, complete message-only legacy imports, and purge expired descendant content recursively. Refs GH-805 Co-Authored-By: OpenAI Codex <noreply@openai.com>

Update remaining reporting contracts to read execution status and full transcripts from SQL instead of transient turn-session state. Refs GH-805 Co-Authored-By: OpenAI Codex <noreply@openai.com>

Update existing conversation metadata during legacy history import so newer step activity cannot be hidden by the earlier message-first write. Refs GH-805 Co-Authored-By: OpenAI Codex <noreply@openai.com>

Make SQL the sole durable authority for conversation metadata, visible history, compaction snapshots, and reporting reads. Prevent retention and legacy import races, remove fail-open SQL writes, and keep completion retries idempotent without double-counting usage. Refs GH-805 Co-Authored-By: OpenAI GPT-5.4 <noreply@openai.com>

Fail legacy transcript imports on malformed Redis payloads instead of silently dropping rows. Stop decoding unsupported turn-session compatibility shapes and remove redundant guards around typed inputs. Require SQL identity and destination foreign keys at runtime rather than falling back to legacy JSON copies. Refs GH-805 Co-Authored-By: OpenAI GPT-5.4 <noreply@openai.com>

Replace the field-by-field durable-state repair logic with one strict schema. Invalid records now fail at the storage boundary instead of being normalized, partially decoded, or treated as missing. Refs GH-805 Co-Authored-By: OpenAI GPT-5.4 <noreply@openai.com>

Exercise the completed turn path when Slack delivery succeeds but thread state persistence fails. Assert SQL-visible messages, reply markers, durable turn state, and agent-step projection remain current.\n\nCo-Authored-By: OpenAI GPT-5.4 <noreply@openai.com>

Persist the SQL-visible transcript independently from Redis thread scratch after Slack delivery. Keep retry ownership explicit and verify the Redis failure path is actually exercised.\n\nCo-Authored-By: OpenAI GPT-5.4 <noreply@openai.com>

Keep core and plugin Drizzle artifacts under migrations/. Remove injectable step-store seams so runtime and tests use the canonical SQL storage boundary. Add a dated cleanup marker for the temporary Redis-to-SQL legacy import after its TTL horizon passes. Co-Authored-By: OpenAI GPT-5.4 <noreply@openai.com>

vercel Bot deployed to Preview – junior-docs July 9, 2026 17:42 View deployment