Skip to content

[codex] Add source-aware Builder database foundation#1173

Draft
3mdistal wants to merge 16 commits into
mainfrom
codex/content-source-aware-foundation
Draft

[codex] Add source-aware Builder database foundation#1173
3mdistal wants to merge 16 commits into
mainfrom
codex/content-source-aware-foundation

Conversation

@3mdistal

@3mdistal 3mdistal commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Summary

This draft PR adds the first source-aware database foundation to the Content template, with Builder CMS as the first real integration target.

It gives databases an explicit source layer: source metadata, field mappings, row identity/provenance, change sets, review records, and execution records. Builder CMS is wired as a read/diff/review target, with live writes disabled by default. The only live-write-eligible Builder target in this PR is the agent-native-blog-article-test model; blog-article and every other Builder model remain read-only.

What is completed

  • Added source/database schema and migrations for content database sources, mappings, row identity/provenance, change sets, review events, and execution/dry-run records.
  • Added source actions for attach/refresh/status, Builder model discovery, propose/review change sets, local revision staging, Builder execution prepare/validate/execute, consolidated Builder review, and source-backed property creation.
  • Added Builder CMS read/write adapter primitives for public/private read paths, model discovery, normalized row identity/source fields, dry-run write plan generation, and injectable-fetch write-client tests.
  • Added live-write engine slice M1-M4 in b4908454:
    • execution states ready, blocked, and write_disabled
    • execute-builder-source-execution with approval, push-mode, capability, safe-model, dry-run, and idempotency gates
    • post-write reconcile so returned Builder IDs are persisted and repeat pushes use PATCH
    • live write calls remain blocked unless explicitly enabled and targeting only agent-native-blog-article-test
  • Added per-source live-write enablement and Source settings UX in c8db4ef:
    • liveWritesEnabled remains false by default
    • allowed write modes are explicit per source
    • enabling is refused unless the source is Builder CMS and the model is exactly agent-native-blog-article-test
    • disabling clears live-write eligibility and mode opt-ins
    • safe-model enablement/mode settings survive Builder refresh metadata updates
    • Review Builder update lives in Database settings -> Source, with badges, modal Push, validate-only disabled state, and guarded live execution when enabled
  • Corrected unmatched fixture-row handling in 3d1de9b, with execution-path regression coverage added in 7ae60bd:
    • synthetic fixture identities shaped like builder-${documentId} are detected but are not treated as proven Builder entry IDs
    • live autosave for unmatched fixture rows is blocked before execution with: This row is not matched to a Builder entry yet. Refresh or match a Builder row before pushing.
    • the execution payload no longer presents synthetic fixture IDs as executable Builder targets
    • execute-builder-source-execution now has a regression test proving synthetic fixture rows block before executeWrite is invoked
    • import/de-dupe no longer treats fixture-wrapped IDs as already represented live entries
    • real/read-adapter Builder entry IDs still PATCH normally
  • Verified direct live-write endpoint semantics against a real agent-native-blog-article-test Builder entry:
    • readBuilderCmsContentEntries({ model: "agent-native-blog-article-test", limit: 3 }) returned state: live with three real entry IDs
    • executeBuilderCmsWrite PATCHed /api/v1/write/agent-native-blog-article-test/0bdeaf87310f4d7f9a677c6fa862764e?autoSaveOnly=true&triggerWebhooks=false
    • the probe changed data.title, then immediately reverted data.title to the original
    • both write and revert returned { ok: true, status: 200, entryId: "0bdeaf87310f4d7f9a677c6fa862764e" }
    • no write was attempted outside agent-native-blog-article-test
  • Completed full browser UI acceptance for the guarded Builder live-write flow in 6a231de4a:
    • restarted the content dev server on http://127.0.0.1:8081 with the ignored root .env loaded
    • loaded http://localhost:8081/page/oJ5v9MB3EiDZ in the in-app browser
    • cleared a stale local fixture-row acceptance artifact and fixed the source diff builder so live Builder sources no longer synthesize pushable diffs for legacy fixture rows
    • edited a real matched Builder row in the database UI
    • verified one Database settings Builder update badge appeared
    • opened Database settings -> Source -> Review Builder update
    • confirmed the modal targeted only agent-native-blog-article-test and entry 0bdeaf87310f4d7f9a677c6fa862764e
    • clicked Push and saw Pushed to Builder and reconciled locally
    • verified the change-set is applied, execution is succeeded, target model is agent-native-blog-article-test, response status is 200, and the toolbar pending badge clears
    • added a Node HTTP/HTTPS fallback for Builder write transport after the dev server action runtime hit fetch failed while direct Node fetch worked
  • Fixed the loading-loop symptom seen after failed/retry state:
    • row-selection pruning now preserves the existing array reference when selection does not change, including empty/loading states
    • browser recheck in this worker no longer produced Maximum update depth exceeded; the in-app browser itself was unauthenticated, so it could not prove the authenticated database rows
  • Moved source details toward database settings and column/property surfaces; database dev/admin link is Code Mode only.
  • Added Builder model picker/source attach test path; earlier browser verification imported 20 rows from agent-native-blog-article-test and showed 20 of 20 entries matched.

What is not completed yet

  • Builder autoSaveOnly=true visibility still needs a follow-up decision:
    • the browser UI path now reaches Builder and records a 200 write response/reconciled local success
    • the normal Builder read/content surface still returns the published/current title for the test entry after autosave, so we should decide whether the acceptance contract is "autosave revision exists" or whether the app should verify against a Builder draft/autosave-specific read path
  • Do not use the legacy first row from oJ5v9MB3EiDZ; it is an unmatched fixture row and should now be ignored by live-source diff synthesis or blocked before execution.
  • OAuth/space authorization UX is not implemented yet.
  • Full Builder schema-to-column hydration is not complete.
  • Refresh/import pruning policy, polished conflict resolution, relationships/rollups, comments/collaboration, and analytics subscriptions remain future work.

Validation

Passed:

pnpm --filter content exec vitest --run actions/execute-builder-source-execution.test.ts actions/_builder-cms-write-adapter.test.ts actions/_builder-cms-write-client.test.ts actions/_database-source-utils.test.ts app/components/editor/DocumentDatabase.test.ts

Result: 5 files passed, 126 tests passed.

Passed:

pnpm --filter content exec vitest --run actions/_database-source-utils.test.ts actions/content-database-source-actions.test.ts actions/_builder-cms-read-client.test.ts actions/_builder-cms-source-adapter.test.ts actions/_builder-cms-write-adapter.test.ts actions/_builder-cms-write-client.test.ts actions/_builder-cms-write-settings.test.ts actions/execute-builder-source-execution.test.ts app/components/editor/DocumentDatabase.test.ts

Result: 9 files passed, 161 tests passed.

Passed:

pnpm --filter content typecheck

Passed:

git diff --check
git diff --cached --check

Direct live-write proof:

  • With Builder private/public keys saved in the ignored root .env and the content dev server started with that env loaded, the orchestrator ran a direct app-client probe from templates/content against only agent-native-blog-article-test.
  • Read proof: readBuilderCmsContentEntries({ model: "agent-native-blog-article-test", limit: 3 }) returned state: live, count 3, with real entry IDs.
  • Write proof: executeBuilderCmsWrite PATCHed real test entry 0bdeaf87310f4d7f9a677c6fa862764e at /api/v1/write/agent-native-blog-article-test/0bdeaf87310f4d7f9a677c6fa862764e?autoSaveOnly=true&triggerWebhooks=false, changed data.title, then immediately reverted data.title to the original.
  • Both write and revert returned { ok: true, status: 200, entryId: "0bdeaf87310f4d7f9a677c6fa862764e" }.
  • No write was attempted outside agent-native-blog-article-test.

Browser/runtime notes:

  • Orchestrator verified cb68555 still 404ed when attempting to PATCH the legacy first row; 3d1de9b changes that case to a blocked local execution instead of a live PATCH attempt.
  • Full in-app browser acceptance is now complete on 6a231de4a:
    • source settings showed the safe Builder test source and live autosave writes enabled for the Agent Native test collection
    • the review modal summarized one real matched row title change and targeted agent-native-blog-article-test
    • Push completed with Pushed to Builder and reconciled locally
    • local DB verification showed change-set 77b018a3-3e55-4d06-8cfa-c8018ef38b44 as applied, execution as succeeded, target model agent-native-blog-article-test, entry 0bdeaf87310f4d7f9a677c6fa862764e, response status 200
    • final browser reload showed no pending Builder badge
  • The normal Builder read client still returned the published/current title for the test entry after autoSaveOnly=true, so draft/autosave visibility remains a follow-up verification question.
  • After the selection-prune fix, the worker browser recheck showed zero Maximum update depth exceeded console errors.
  • Unit tests never call live Builder and no Builder private key was printed, stored, or committed.

Latest focused validation:

pnpm --filter content exec vitest run actions/_builder-cms-write-client.test.ts actions/_database-source-utils.test.ts actions/content-database-source-actions.test.ts app/components/editor/DocumentDatabase.test.ts

Result: 4 files passed, 119 tests passed.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

@3mdistal 3mdistal force-pushed the codex/content-source-aware-foundation branch from 18ba3cd to 5f219de Compare June 15, 2026 15:38
@github-actions

github-actions Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Visual recap — skipped

The visual recap job did not run for this pull request. This is informational only and does not block the PR.

Recap skipped for 6f52977: draft PR.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

3mdistal and others added 5 commits June 15, 2026 16:58
…tch, delete-property cleanup

Found while dogfooding the content-template Builder source flow in the browser:

- Render Builder reference fields as readable labels instead of raw JSON in the source adapter (+ test)
- Remove the duplicate "Local files" sidebar nav button
- Add AGENT_NATIVE_LOCAL_BUILDER_ENV local-dev escape hatch so env Builder keys can back a signed-in user under the workspace runtime (non-prod, opt-in; default off keeps the tenant-safety gate and its test intact) + tests + DEVELOPING docs
- Clear the mapped source field when its property is deleted so it returns to the picker without a refresh
- postinstall now rebuilds better-sqlite3 to match the pinned Node ABI

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Request reference enrichment on the Builder read (content API enrich=true + MCP enrich:true) so referenced entries (e.g. blog-article -> blog-author) come back inline, and harden builderReferenceLabel to pull the referenced entry's name from value.data or the entry top level. Mapped reference columns (author, authors) now render names instead of `model:id` tokens. Verified live: Author column shows real author names.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Builder CMS date fields come back as milliseconds-since-epoch numbers, which normalizeDatePropertyValue dropped (string/object only) — so mapped Date columns rendered empty. Coerce numeric epoch values to a date before normalizing. Tests: the date normalizer and the source-field value mapper both now accept epoch numbers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant