fix(provenance): store an absence apart from a refusal

Relaxing a stored `unknown` assumed it always meant "nobody recorded this". It did not. The
sidecar collapsed two opposite claims into one value: bytes nobody recorded, and bytes a writer
refused on purpose — a child of an archive whose parent provably held secrets, a generated
asset whose safety decision came back false, a transcode whose scanner knew secrets were
present and could not locate them. Relaxing the second would have walked known-secret-bearing
content through every model boundary at once.

The decision type already told them apart: a registry that never ran is `safe: true`, while
every refusal is `safe: false`. Only storage lost it. Writers now persist `unrecorded` and
`unknown` separately, readers relax the first and refuse the second, and the constraint is
widened to accept it.

Only workspace files store this distinction, deliberately. Every other durable surface produces
a non-exact sidecar from one condition — an incomplete incoming bundle or registry — which is
always an absence. Files are the only surface that derives one stored object from another, so
they are the only one that can refuse on purpose. The shared policy is unchanged and uniform:
read an absence and audit it, refuse a taint.

Also aligns the metadata batch classifier, which answered `unknown` where the single-file reader
answers `unrecorded`, leaving two classifiers describing one policy differently.

Author: icecrasher321

apps/sim/lib/copilot/request/tools/files.test.ts

@@ -578,7 +578,12 @@ describe('maybeWriteOutputToFile', () => {
     )
   })
 
-  it('preserves legacy writes without a registry and marks their provenance unknown', async () => {
+  /**
+   * No registry means no recorder ran, so nothing was written down about these bytes — an absence,
+   * which the surface's policy may relax, and distinct from the taint a refused safety decision
+   * produces below.
+   */
+  it('preserves legacy writes without a registry and marks their provenance unrecorded', async () => {
     const result = await maybeWriteOutputToFile(
       RunFunction.id,
       { outputs: { files: [{ path: 'files/report.json', mode: 'overwrite' }] } },
@@ -589,7 +594,7 @@ describe('maybeWriteOutputToFile', () => {
     expect(result.success).toBe(true)
     expect(mockWriteWorkspaceFileByPath).toHaveBeenCalledWith(
       expect.anything(),
-      expect.objectContaining({ secretProvenance: { status: 'unknown' } })
+      expect.objectContaining({ secretProvenance: { status: 'unrecorded' } })
     )
   })
 

apps/sim/lib/execution/durable-secret-provenance-enforcement.ts

@@ -9,6 +9,19 @@ const logger = createLogger('DurableSecretProvenanceEnforcement')
  *
  * Named by the call site rather than derived, so the surface survives refactors and stays
  * greppable — the same convention the projection-refusal `site` strings use.
+ *
+ * One policy governs all of them: an **absence** — nobody recorded what these bytes carry — may be
+ * read, with an audit entry naming the surface, because a value nobody recorded says exactly what
+ * an untracked one does. A **taint** — a writer that knew secrets were present and could not map
+ * them to this output — is refused, and no policy relaxes it.
+ *
+ * Only `workspace-file` stores the difference, and that is deliberate rather than drift. On the
+ * other surfaces every non-exact sidecar comes from one condition, an incomplete incoming bundle
+ * or registry, which is always an absence; two stored statuses say everything there is to say.
+ * Files are the only surface that derives one stored object from another — an archive extracted
+ * into children, a generated asset, a transcoded output — so they are the only one that can refuse
+ * on purpose, and the only one with two claims to keep apart. Adding a third status elsewhere would
+ * encode a distinction that surface cannot make; removing it here would collapse one that matters.
  */
 export const DURABLE_SECRET_PROVENANCE_SURFACES = [
   'memory',

apps/sim/lib/uploads/contexts/workspace/workspace-file-secret-provenance.test.ts

@@ -153,23 +153,49 @@ describe('workspace file secret provenance', () => {
   })
 
   /**
-   * The union carries three states; the column's CHECK constraint accepts two. Forwarding the
-   * status verbatim would send `'unrecorded'` to the database as a value it rejects, aborting the
-   * enclosing transaction rather than writing a bad row.
+   * Absence and taint are different claims and must round-trip separately. Collapsing them is what
+   * made a file a writer deliberately refused indistinguishable from one nobody recorded, and so
+   * eligible for the same relaxation.
    */
-  it('narrows an unrecorded initialization to a status the column accepts', async () => {
+  it('persists an unrecorded initialization as its own status', async () => {
     await initializeWorkspaceFileSecretProvenanceInTx(
       dbChainMock.db as unknown as DbTransaction,
       'file-1',
       CONTENT_UPDATED_AT,
       { status: 'unrecorded' }
     )
 
+    expect(dbChainMockFns.values).toHaveBeenCalledWith(
+      expect.objectContaining({ fileId: 'file-1', status: 'unrecorded', entries: [] })
+    )
+  })
+
+  it('persists a refused write as unknown, not as an absence', async () => {
+    await replaceWorkspaceFileSecretProvenanceInTx(
+      dbChainMock.db as unknown as DbTransaction,
+      'file-1',
+      CONTENT_UPDATED_AT,
+      { status: 'unknown' }
+    )
+
     expect(dbChainMockFns.values).toHaveBeenCalledWith(
       expect.objectContaining({ fileId: 'file-1', status: 'unknown', entries: [] })
     )
   })
 
+  it('persists an unrecorded replacement as its own status', async () => {
+    await replaceWorkspaceFileSecretProvenanceInTx(
+      dbChainMock.db as unknown as DbTransaction,
+      'file-1',
+      CONTENT_UPDATED_AT,
+      { status: 'unrecorded' }
+    )
+
+    expect(dbChainMockFns.values).toHaveBeenCalledWith(
+      expect.objectContaining({ fileId: 'file-1', status: 'unrecorded', entries: [] })
+    )
+  })
+
   it('rejects a marker write that cannot bind the exact tracked content version', async () => {
     dbChainMockFns.returning.mockResolvedValueOnce([])
 
@@ -305,6 +331,17 @@ describe('workspace file secret provenance', () => {
         status: 'unknown',
         entries: [],
       },
+      {
+        id: 'unrecorded-id',
+        key: 'unrecorded-key',
+        workspaceId: 'workspace-1',
+        context: 'workspace',
+        fileContentUpdatedAt: CONTENT_UPDATED_AT,
+        secretProvenanceVersion: 1,
+        provenanceContentUpdatedAt: CONTENT_UPDATED_AT,
+        status: 'unrecorded',
+        entries: [],
+      },
       {
         id: 'other-workspace-id',
         key: 'other-workspace-key',
@@ -348,6 +385,7 @@ describe('workspace file secret provenance', () => {
       { id: 'wrong-id', key: 'safe-key' },
       { id: 'safe-id', key: 'tainted-key' },
       { id: 'unknown-id', key: 'unknown-key' },
+      { id: 'unrecorded-id', key: 'unrecorded-key' },
       { id: 'other-workspace-id', key: 'other-workspace-key' },
       { id: 'pre-marker-sidecar-id', key: 'pre-marker-sidecar-key' },
       { id: 'synthetic-execution-id', key: 'untracked-context-key' },
@@ -362,8 +400,12 @@ describe('workspace file secret provenance', () => {
       { key: 'safe-key' },
       { id: 'file-1700000000000', key: 'safe-key' },
       { id: 'wrong-id', key: 'safe-key' },
-      /** Unrecorded, so kept — the untracked keys below say the same thing and always were. */
-      { id: 'unknown-id', key: 'unknown-key' },
+      /**
+       * Unrecorded, so kept — the untracked keys below say the same thing and always were. The
+       * stored `unknown` above is dropped: a writer refused those bytes on purpose, which is a
+       * different claim from nobody having recorded them, and no policy relaxes it.
+       */
+      { id: 'unrecorded-id', key: 'unrecorded-key' },
       { id: 'pre-marker-sidecar-id', key: 'pre-marker-sidecar-key' },
       { id: 'synthetic-execution-id', key: 'untracked-context-key' },
       { id: 'legacy-id', key: 'legacy-key' },
@@ -720,7 +762,7 @@ describe('workspace file secret provenance', () => {
         fileContentUpdatedAt: CONTENT_UPDATED_AT,
         secretProvenanceVersion: 1,
         provenanceContentUpdatedAt: CONTENT_UPDATED_AT,
-        status: 'unknown',
+        status: 'unrecorded',
         entries: [],
       },
     ])
@@ -1331,7 +1373,7 @@ describe('workspace file secret provenance', () => {
         fileContentUpdatedAt: CONTENT_UPDATED_AT,
         secretProvenanceVersion: 1,
         provenanceContentUpdatedAt: CONTENT_UPDATED_AT,
-        status: 'unknown',
+        status: 'unrecorded',
         entries: [],
       },
     ])

apps/sim/lib/uploads/contexts/workspace/workspace-file-secret-provenance.ts

@@ -242,7 +242,12 @@ export async function createWorkspaceFileSecretProvenanceFromRegistry(
   representations: readonly WorkspaceFileSecretProvenanceRepresentation[] = [],
   representationsComplete = true
 ): Promise<WorkspaceFileSecretProvenanceWriteDecision> {
-  if (!registry) return { safe: true, provenance: { status: 'unknown' } }
+  /**
+   * No registry means no recorder ran, so nothing was written down about these bytes. That is an
+   * absence, and it is the one thing this surface's policy may relax — distinct from the taint
+   * every `safe: false` below produces, which a caller persists as `unknown` and no policy relaxes.
+   */
+  if (!registry) return { safe: true, provenance: { status: 'unrecorded' } }
   const sourceProvenance = registry.exportCommittedProvenanceForValue(sourceValue)
   const persistedProvenance = Object.is(sourceValue, persistedValue)
     ? sourceProvenance
@@ -422,12 +427,19 @@ export async function replaceWorkspaceFileSecretProvenanceInTx(
     return
   }
 
+  /**
+   * Absence and taint are different claims and must not share a stored value. Collapsing them here
+   * is what let a file the writer deliberately refused — a child of a secret-bearing archive, a
+   * generated asset whose safety decision was `false` — become indistinguishable from one nobody
+   * recorded, and therefore eligible for the same relaxation.
+   */
+  const status = provenance.status === 'unrecorded' ? 'unrecorded' : 'unknown'
   await tx
     .insert(workspaceFileSecretProvenance)
-    .values({ fileId, contentUpdatedAt, status: 'unknown', entries: [], updatedAt: new Date() })
+    .values({ fileId, contentUpdatedAt, status, entries: [], updatedAt: new Date() })
     .onConflictDoUpdate({
       target: workspaceFileSecretProvenance.fileId,
-      set: { contentUpdatedAt, status: 'unknown', entries: [], updatedAt: new Date() },
+      set: { contentUpdatedAt, status, entries: [], updatedAt: new Date() },
     })
   await markWorkspaceFileSecretProvenanceTrackedInTx(tx, fileId, contentUpdatedAt)
 }
@@ -449,12 +461,13 @@ export async function initializeWorkspaceFileSecretProvenanceInTx(
 ): Promise<void> {
   const isExact = provenance.status === 'exact'
   const entries = isExact ? serializeExactEntriesForStorage(provenance.entries) : []
+  const status = isExact ? 'exact' : provenance.status === 'unrecorded' ? 'unrecorded' : 'unknown'
   await tx
     .insert(workspaceFileSecretProvenance)
     .values({
       fileId,
       contentUpdatedAt,
-      status: isExact ? 'exact' : 'unknown',
+      status,
       entries,
       updatedAt: new Date(),
     })
@@ -686,10 +699,12 @@ export async function getBoundWorkspaceFileSecretProvenance(
     row.provenanceContentUpdatedAt?.getTime() === row.fileContentUpdatedAt.getTime()
   if (!bindingIsCurrent || !isValidStoredEntries(row.entries)) return { status: 'unknown' }
   /**
-   * The one shape the surface's policy may relax: a sidecar bound to this exact content that says
-   * nobody recorded what it carries — the same statement the untracked file above makes.
+   * The one shape the surface's policy may relax: a sidecar bound to this exact content recording
+   * that nobody vouched for it — the same statement the untracked file above makes. A stored
+   * `unknown` is the opposite claim, written by a writer that refused these bytes on purpose, and
+   * stays refused.
    */
-  if (row.status === 'unknown') return { status: 'unrecorded' }
+  if (row.status === 'unrecorded') return { status: 'unrecorded' }
   if (row.status !== 'exact') return { status: 'unknown' }
   return { status: 'exact', entries: deserializeExactEntriesFromStorage(row.entries) }
 }
@@ -739,12 +754,25 @@ export async function getBoundWorkspaceFileSecretProvenanceByMetadata(
       if (
         row.secretProvenanceVersion !== 1 ||
         row.provenanceContentUpdatedAt?.getTime() !== row.fileContentUpdatedAt.getTime() ||
-        row.status !== 'exact' ||
         !isValidStoredEntries(row.entries)
       ) {
         result.set(row.id, { status: 'unknown' })
         continue
       }
+      /**
+       * Same answer the single-file reader gives the same row. Collapsing a recorded absence into
+       * `unknown` here would leave two classifiers describing one policy differently, and the
+       * caller that eventually distinguishes them would get a different verdict depending on which
+       * one it happened to call.
+       */
+      if (row.status === 'unrecorded') {
+        result.set(row.id, { status: 'unrecorded' })
+        continue
+      }
+      if (row.status !== 'exact') {
+        result.set(row.id, { status: 'unknown' })
+        continue
+      }
       result.set(row.id, {
         status: 'exact',
         entries: deserializeExactEntriesFromStorage(row.entries),
@@ -909,7 +937,7 @@ function classifyModelSafeWorkspaceFileRow(
   const bindingIsCurrent =
     row.provenanceContentUpdatedAt?.getTime() === row.fileContentUpdatedAt.getTime()
   if (!bindingIsCurrent || !isValidStoredEntries(row.entries)) return 'unsafe'
-  if (row.status === 'unknown') return 'unrecorded'
+  if (row.status === 'unrecorded') return 'unrecorded'
   if (row.status !== 'exact') return 'unsafe'
   return row.entries.length === 0 ? 'safe' : 'unsafe'
 }

packages/db/migrations/0302_fat_sentry.sql

@@ -0,0 +1,4 @@
+-- migration-safe: replaces the status CHECK with a strictly wider one, adding 'unrecorded' to the existing ('exact', 'unknown'). Every stored row and every write the currently-deployed code can make already satisfies the replacement, so dropping it opens no window in which an old write is rejected, and it is re-added in the same transaction below.
+ALTER TABLE "workspace_file_secret_provenance" DROP CONSTRAINT "workspace_file_secret_provenance_status_check";--> statement-breakpoint
+ALTER TABLE "workspace_file_secret_provenance" ADD CONSTRAINT "workspace_file_secret_provenance_status_check" CHECK ("workspace_file_secret_provenance"."status" IN ('exact', 'unknown', 'unrecorded')) NOT VALID;--> statement-breakpoint
+ALTER TABLE "workspace_file_secret_provenance" VALIDATE CONSTRAINT "workspace_file_secret_provenance_status_check";