feat(metadata-core,metadata-protocol)!: the sys_metadata family declares no organization column; indexes re-keyed (ADR-0131 D6/D7, C7b S2)

.changeset/23429-sys-metadata-family-organization-column-retired.md

@@ -0,0 +1,60 @@
+---
+'@objectstack/metadata-core': minor
+'@objectstack/metadata-protocol': minor
+'@objectstack/metadata': minor
+'@objectstack/rest': minor
+'@objectstack/spec': minor
+'@objectstack/objectql': minor
+'@objectstack/plugin-security': minor
+'@objectstack/service-datasource': minor
+'@objectstack/service-automation': minor
+'@objectstack/platform-objects': minor
+'@objectstack/driver-sql': minor
+'@objectstack/driver-turso': minor
+'@objectstack/cli': patch
+---
+
+feat(metadata-core,metadata-protocol)!: the `sys_metadata` family declares no organization column — `sys_metadata`, `sys_metadata_audit`, `sys_metadata_commit` and `sys_metadata_history` (ADR-0131 D6/D7)
+
+Clause-②: yes (narrowing)
+
+<!-- adr-0087: registered sys-metadata-organization-column-retired, sys-metadata-audit-organization-column-retired, sys-metadata-commit-organization-column-retired, sys-metadata-history-organization-column-retired -->
+
+**BREAKING**, shipped as `minor` under the repo's launch-window convention for breaking changes.
+
+Metadata is the environment's (ADR-0131 D6). The four tables of the metadata family no longer declare `organization_id`, and each sets `systemFields.tenant: false`, so the engine injects none back. The engine refuses an undeclared field, so nothing reads or writes the column any more.
+
+FROM → TO:
+
+- An authored filter, list-view column, report grouping, formula or seed key naming `organization_id` on one of the four tables → delete it; no organization owns a row. The field resolver answers it as an unknown field.
+- `new SysMetadataRepository({ engine, organizationId, orgLabel })` → `new SysMetadataRepository({ engine, orgLabel })`. The `organizationId` option is gone: every repository reads and writes the environment's rows.
+- `DatabaseLoader`'s `organizationId` option (and so `MetadataManager.setDatabaseDriver` / `setDataEngine`'s `organizationId` argument) → accepted and ignored, like `environmentId`: no write stamps it and no read filters on it. Drop the argument.
+- `@objectstack/spec`: the published `ApiError.code` / `ErrorCode` enum widens by `LEGACY_ORGANIZATION_ROWS_UNREADABLE`, registered under `@objectstack/metadata-core`.
+- `isLegacyOrganizationRow(row)` (`@objectstack/metadata-core`) → `readLegacyOrganizationRows(driver, table?)`. The row-value test read a key the engine no longer returns, so it is removed. The new read answers from the physical column, through the table's driver (`engine.getDriverForObject`): id → organization, empty on a migrated or fresh database. A seam that cannot answer is refused (`LegacyOrganizationRowsUnreadable`), never read as "none".
+- An existing database → run `os migrate organization-ownership` (review its plan), then `os migrate organization-ownership --apply --plan-file FILE` with that plan file. The ceremony promotes or reports the legacy organization rows, then drops the physical column. Until it has run, the v18 runtime refuses to start against the database.
+
+The indexes are re-keyed with the column:
+
+- The overlay unique key is (`type`, `name`, `package_id`, `bundle_member`): the declared `idx_sys_metadata_overlay_active` and the runtime partial indexes of the same names. Removing the organization TIGHTENS uniqueness. On a database the ceremony has not cleared, two legacy organizations' rows, or an organization's row and the environment's, can share a key. There the runtime index migration keeps the previous index and names the duplicates (ADR-0120 D4); it never drops a row. `os migrate duplicates` and the boot report group by the same key.
+- `sys_metadata_history` is unique on (`event_seq`) and on (`type`, `name`, `version`). Its other indexes, and those of `sys_metadata_audit` and `sys_metadata_commit`, lose their organization prefix.
+- `idx_sys_metadata_org_type` is gone.
+
+What else changes:
+
+- The boot line `[metadata_org_scoped_unserved]` is gone with the column it read. A database still holding legacy organization rows is refused at boot instead, until the ceremony has run.
+- The package uninstall no longer removes a legacy organization's row of the package's name: no row carries an organization.
+- Who reads the family through the generic data door (`/data/sys_metadata_*`), before → after:
+  - `sys_metadata_audit`, `sys_metadata_history`, `sys_metadata_commit`, **under a wall** (`isolated` / `group`): before, the tenant wall filtered each caller to its own organization's rows, and an `organization_admin` read its organization's. With the column gone the wall has nothing to filter on, so without a gate that admin would read every organization's trail, history bodies and commit ranges. After: each requires `studio.access` (`requiredPermissions`, the #22107 precedent). The capability is platform-scoped, so an `organization_admin` holding it by role does not exist, and that admin is refused.
+  - The same three, **under `single`**: before, any caller whose grant covered the table (a read wildcard) read it. After: `studio.access` is required there too. A Studio author (the platform admin among them) reads as before; a plain member is refused.
+  - Every `/meta` door serving these rows admits `studio.access`: `/meta/:type/:name/history` and `/audit` admit `studio.access`, `setup.access` or `manage_metadata`; `/packages/:id/commits` admits `studio.access` or `setup.access`. `requiredPermissions` is an AND-gate and cannot name an any-of set, and `setup.access` is held by organization admins, so it could not be the gate. A caller holding only `setup.access` or only `manage_metadata` therefore still reads these rows through `/meta`, but not through the data door.
+  - `sys_metadata` stays ungated, before and after: its rows are what `/meta` already serves to every organization.
+- On a database the ceremony has not migrated (which `os serve` refuses, but every `os migrate` boot opens, the ceremony's own among them), the legacy organization rows are still told apart, off the physical column:
+  - every read the metadata protocol makes of the family goes through one seam that leaves them out. That covers boot hydration; the `/meta` list, by-name, layered and draft-preview reads; the item lock (`_lock`) the reads report and the save and delete gates enforce; the package-less write and read address; and the package, commit, audit, history and diff reads. Two reads keep them on purpose and read the column themselves: `os migrate meta --stored` reports such a row, and the commit revert refuses a legacy commit by name. The authored hook and action binders leave them out too;
+  - the metadata repository's write never resolves to one: before, a save of an address that only a legacy organization's row held rewrote that row in place (measured on the ceremony's template promotion). Now the write targets the environment's row, and with none it creates one, which the re-keyed unique index refuses beside the organization's row until the ceremony settles it;
+  - `os migrate meta --stored` reports such a row `skipped` and never re-saves it;
+  - `os migrate security-catalog-overlays` neither lists nor deletes it;
+  - a commit recorded in a legacy organization's layer is still refused by the revert;
+  - the runtime-datasource reads, the packaged permission-set overlay reading, the kernel:ready security-catalog echo, the stored flow-credential move and a `DatabaseLoader` (engine- or driver-backed) leave them out.
+- `SqlDriver.listPhysicalColumns(object, options?)` (new, read-only): a table's physical column names, from the catalog, joining `options.transaction`. `TursoDriver`'s remote face answers it from the remote catalog. Every boot asks whether the family still carries the retired column, and it asks this way, not through `execute()`. So `os migrate plan` keeps its "a plan writes nothing" line on a migrated or fresh database. The legacy rows themselves are read with a raw statement only where the column exists.
+- `GET /forms/:slug` and `POST /forms/:slug/submit` no longer read the Default Organization's legacy layer. Every metadata read is now environment → code. The doors judge each candidate against the environment's view list, which holds one item per package of a view name. So a withdrawal in one package still closes that name's form in every package. On a deployment with no organization this closes more than before, which is the fail-closed direction. A withdrawal saved only at the organization layer is carried onto the environment by the ceremony's `metadata-promotion` step, which the runtime requires before it starts.
+- The protocol's `getMetaItems` drops its undocumented `legacyFormOrganizationId` request key. It was never in the spec's `GetMetaItemsRequest`, and only those doors passed it. A caller that still passes it now gets the environment's list.

.changeset/23429-v18-boot-refusal.md

@@ -0,0 +1,29 @@
+---
+'@objectstack/runtime': minor
+'@objectstack/cli': minor
+'@objectstack/spec': minor
+---
+
+feat(runtime,cli)!: `os serve` refuses to start against a database still in the v17 organization-ownership shape, and `os migrate apply` refuses to drop its `organization_id` columns, until `os migrate organization-ownership` has verified it (ADR-0131 D10 item 5)
+
+Clause-②: yes (narrowing)
+
+<!-- adr-0087: not-required (already-registered sys-audit-log-organization-column-retired, sys-flow-dispatch-organization-column-retired, sys-job-organization-column-retired, sys-job-queue-organization-column-retired, sys-job-run-organization-column-retired, sys-migration-journal-organization-column-retired, sys-migration-organization-column-retired, sys-presence-organization-column-retired) The step-18 D3 entries named here, registered before this change, record the retirement of each organization column this refusal reads, with the remedy that clears it; the metadata family's four entries land with its schema change, which this refusal ships beside. This change adds no authorable surface, moves no spec key and rewrites no stored row: it refuses to boot while a retired column is still physically present, and the remedy is the operator's own migration: the ceremony, whose column-drops step drops what those entries prescribe dropping. No new D3 entry and no D2 conversion. -->
+
+**BREAKING**, shipped as `minor` under the repo's launch-window convention for breaking changes.
+
+A v18 runtime no longer serves a database it was never migrated for. At `kernel:ready`, `os serve` (and so `os dev`) checks every table whose object declares no organization column (`systemFields.tenant: false`). If one still carries a physical `organization_id`, the boot is refused unless the `sys_migration` row `adr-0131-organization-ownership` is verified. The refusal names the tables and the command. It is thrown before `kernel:listening`, so no request is served. Its code is `ORGANIZATION_OWNERSHIP_MIGRATION_REQUIRED`.
+
+FROM → TO:
+
+- A 17.x database started by an 18.x `os serve` → run `os migrate organization-ownership`, review its plan, then `os migrate organization-ownership --apply --plan-file FILE` with that plan file, and start again. To keep 17.x semantics on an unmigrated database, run a 17.x runtime.
+- `os migrate apply --allow-destructive` on such a database (`@objectstack/cli`, BREAKING `minor`: the command narrows) → it now refuses to drop `organization_id` from any table in the ceremony's inventory (the D7 tables and the `sys_metadata` family) while the ceremony's marker is not verified. Before, the drift report offered each such column as an unmapped destructive column, and the apply dropped it. Dropped ahead of the ceremony, that column takes the ceremony's promotion and report steps with it, turns every legacy organization row into an environment row, and leaves the boot refusal no column to find. The refusal names `os migrate organization-ownership`, exits 1, and under `--json` answers `message: 'refused_organization_ownership_unmigrated'` with the `tables`, as the `sys_account.issuer` (#17440) and `sys_comment.reactions` (#23281) refusals do. The refusal holds the WHOLE `--allow-destructive` run, as those two do: nothing in it is created, applied or dropped, every other destructive drop and the additive work included. A run without `--allow-destructive` is unchanged. The ceremony drops these columns itself, in its column-drops step, before it records the run verified.
+- There is no escape hatch: no `OS_ALLOW_*` or `OS_SKIP_*` variable and no flag skips either check.
+
+What does not change:
+
+- A fresh database boots: no table it creates carries a retired column, so no marker is needed.
+- A database whose ceremony verified boots, whatever its columns.
+- The `os migrate` commands boot as before, the ceremony's own among them: the boot guard is composed by `os serve` only, never by the shared standalone stack (`createStandaloneStack`, which `bootSchemaStack` and every other CLI data and schema boot build on). Only an `os migrate apply --allow-destructive` run that would drop an inventory table's `organization_id` is refused, as above.
+- A driver with no physical columns to read (memory, MongoDB) is not checked.
+- A host that composes its own kernel can add the guard itself: `OrganizationOwnershipBootGuardPlugin`, or `assertOrganizationOwnershipMigrated(engine)` from `@objectstack/runtime`. `ORGANIZATION_OWNERSHIP_MIGRATION_ID` is now exported from `@objectstack/spec/system`.

content/docs/concepts/metadata-lifecycle.mdx

@@ -106,7 +106,7 @@ later ships. See [ADR-0070](https://github.com/objectstack-ai/objectstack/blob/m
 
 **Most metadata types must not be overlaid by an environment** — overlaying a packaged item would break the physical schema or the behaviour its package ships. The whitelist lives in **one** place: `MetadataTypeRegistryEntry.allowEnvironmentOverlay` in `packages/spec/src/kernel/metadata-plugin.zod.ts` (spelled `allowOrgOverride` before protocol 18; `GET /api/v1/meta/types` serves it per entry as `allowEnvironmentOverlay`).
 
-> **ADR-0131 D6 — the per-organization overlay axis is retired.** The `/meta` doors (REST and the runtime dispatcher) carry no organization into a metadata write or read: a Studio or `PUT /api/v1/meta/...` write of the five ✅ types below lands **environment-wide** (`organization_id` NULL) and is served to every organization, and an organization admin holding only the retired `manage_org_presentation` capability is refused (`403`). The protocol itself refuses every organization-scoped write, and every read — item, list, layers, cached, history, diff, audit, drafts and the commit timeline — is environment → code, whatever organization a caller names. An overlay row an earlier release stored under an organization — under the `single` posture, every Studio save of these types, filed under the Default Organization — is served by no read: boot names each such row, per type (`[metadata_org_scoped_unserved]`), and the v18 migration ceremony (ADR-0131 C7, `os migrate`) carries it to the environment layer; nothing deletes or rewrites it before then. Re-save the item in Studio to make the edit live now. Public forms are the exception: until that ceremony the anonymous form doors read a form `view` in the Default Organization and prefer its overlay for the form's body, while a withdrawal in either layer closes the form, fail-closed. So a legacy organization overlay of a public form keeps serving its body there: a Studio re-save of that body (an environment row) does not change the body the public form serves, and a Studio withdrawal (an environment row) still closes it. The table still records which types take an overlay of a shipped item.
+> **ADR-0131 D6 — the per-organization overlay axis is retired.** The `/meta` doors (REST and the runtime dispatcher) carry no organization into a metadata write or read: a Studio or `PUT /api/v1/meta/...` write of the five ✅ types below lands **environment-wide** (since ADR-0131 D7 the `sys_metadata` family declares no organization column) and is served to every organization, and an organization admin holding only the retired `manage_org_presentation` capability is refused (`403`). The protocol itself refuses every organization-scoped write, and every read — item, list, layers, cached, history, diff, audit, drafts and the commit timeline — is environment → code, whatever organization a caller names. An overlay row an earlier release stored under an organization — under the `single` posture, every Studio save of these types, filed under the Default Organization — is served by no read, and an 18.x `os serve` refuses to start on a database still holding one until the v18 migration ceremony (ADR-0131 C7, `os migrate organization-ownership --apply`) has carried it to the environment layer and dropped the column; nothing deletes or rewrites it before then. The anonymous form doors read the environment alone too: the ceremony carries every organization-layer public-form withdrawal onto the environment, fail-closed. The table still records which types take an overlay of a shipped item.
 
 | Type | `allowEnvironmentOverlay` | Rationale |
 | :--- | :---: | :--- |
@@ -234,7 +234,7 @@ The hash is `sha256:` + 64-hex of a canonical JSON serialization of the body, wi
 |:---|:---|:---|
 | `defineView(...)` / `defineFlow(...)` / any source file → compiled into `dist/objectstack.json` | ❌ Never. Loaded into the in-memory registry on boot; refreshed via HMR in dev. | ❌ The artifact's own version history *is* Git. The metadata layer does not duplicate it. |
 | Editing a `.json` under `<root>/<type>/<name>.json` (FS overlay, e.g. `<root>/view/case_grid.json`) | ❌ FS layer is independent of DB. | ✅ Appended to the change log at `<root>/.objectstack/.log/main.jsonl` by `FileSystemRepository`. |
-| Studio inline edit, or `PUT /api/v1/meta/...` (REST) on an `allowEnvironmentOverlay: true` type | ✅ Written by `SysMetadataRepository.put()` as an **overlay row**, environment-wide (`organization_id` NULL) since ADR-0131 D6. | ✅ Appended to `sys_metadata_history` in the **same transaction** as the `sys_metadata` write. No-op puts (identical hash) skip the history row entirely. |
+| Studio inline edit, or `PUT /api/v1/meta/...` (REST) on an `allowEnvironmentOverlay: true` type | ✅ Written by `SysMetadataRepository.put()` as an **overlay row**, environment-wide since ADR-0131 D6 (the table declares no organization column since D7). | ✅ Appended to `sys_metadata_history` in the **same transaction** as the `sys_metadata` write. No-op puts (identical hash) skip the history row entirely. |
 | Deploying a new build (new `dist/objectstack.json`) | ❌ The artifact is loaded into memory, not synced into `sys_metadata`. | ❌ Use Git tags / your deployment platform's release log; that's where artifact "version history" lives. |
 
 ### Why artifact never enters the database