diff --git a/.changeset/15211-ceremony-metadata-promotion.md b/.changeset/15211-ceremony-metadata-promotion.md new file mode 100644 index 00000000000..1e766e7e671 --- /dev/null +++ b/.changeset/15211-ceremony-metadata-promotion.md @@ -0,0 +1,43 @@ +--- +'@objectstack/cli': minor +--- + +feat(cli): the organization-ownership ceremony carries the `sys_metadata` organization layer into the environment (ADR-0131 D10) + +Clause-②: yes (widening) + +The plan of `os migrate organization-ownership` gains a key, `metadataConflicts` on the `sys_metadata` table. It lists every overlay key that more than one row would hold once the organization leaves the key, with each row's id. `--apply` gains a flag, `--metadata-choice ROW_ID`. Nothing is removed from the plan or the apply. + +`--apply` shipped in 18.0.0-next.1. This adds to it, and refuses no invocation that release could finish: a conflict now blocks the new step until it is chosen, but on such a database 18.0.0-next.1's `--apply` already stopped at `column-drops`, which never drops `sys_metadata`'s still-declared column. + +`--apply` gains a step, `metadata-promotion`, after the authored catalog conversion: + +- **Form withdrawals are carried first, under every posture, and fail-closed.** When an organization withdrew a public form (`sharing.enabled` or `sharing.allowAnonymous` set to `false`, link kept), every view the environment serves under that name that still serves the form open takes the withdrawal. That covers stored views and views served from code, in every package. The ceremony saves it through the metadata door, into the view's own package, and any organization's withdrawal wins. +- **Under `single`, an organization customization whose key no other row holds becomes the environment's, in place.** This covers the `sys_metadata` rows of `view`, `dashboard`, `report`, `translation` and `email_template`; the key is type, name, package and bundle member. The row takes its history with it. +- **A key held by more than one row is a conflict, listed in the plan.** Under `single` it blocks the step until the operator chooses one row with `--metadata-choice ROW_ID`, once per conflict. + - The key ends with exactly one environment row. Where the key has an environment row, that row takes the chosen body in place, and its version names the chosen row's snapshot. Where it has none, the chosen organization row is promoted in place. A withdrawal the body keeps is also saved through the metadata door. + - The in-place write does to the row what the door does. It asks the door's lock verdict first, so a locked item stays locked and the conflict blocks. It keeps the row's stored `_lock` keys. It restamps `checksum`. + - Any public-form withdrawal another row of the key holds is kept on that body. + - The environment row is read back. Only then are the key's organization rows deleted, in one statement, by id. + - A rerun after an interruption accepts the same `--metadata-choice` and finishes. +- **Under a wall (`isolated`, `group`), nothing is promoted unless the operator names the row, and nothing is ever deleted.** A named row's body becomes the environment's as above. Every organization row left, the chosen one included where the key has an environment row, is named in the report as remaining and blocking `column-drops`, with the remedy. + +**A choice deletes only rows the reviewed plan listed.** `--metadata-choice` and `--template-choice` delete the other rows of their conflict. So the apply reads the conflict groups the plan file records (`metadataConflicts` and `templateConflicts`). It refuses, before anything is written, a choice whose conflict now holds other rows than the reviewed plan listed, with a new reason, `plan-conflict-mismatch`. The refusal names the conflict, the rows added and the rows gone. Plan again, review, and pass the new file. + +`event_seq` and every `(type, name, version)` of `sys_metadata_history` become unique environment-wide. The metadata door numbers a save from history in its own scope, so each lineage is placed where no environment save can land on it: + +- **A promoted row takes its history with it.** Before the row leaves its organization, that organization's lineage of the item is moved above the environment's numbers, and above every other number that organization holds, then becomes environment history. Every later environment save of the item numbers above it, and the row's version still names its own snapshot. +- **A new step, `history-renumber`, runs just before `column-drops`, on every apply, even after a checkpoint.** It lays out every lineage still tagged to an organization above the environment's: each organization's event range by one offset, its commit ranges with it, and each item's version lineage, the organization's rows with it. + - Under `single`, that history then becomes the environment's too. + - Under a wall it stays its organization's. + +`column-drops` runs only after that same apply's `history-renumber` has verified `event_seq` and every `(type, name, version)` unique environment-wide. A duplicate it cannot lay out is named, and blocks. + +Order within an organization, and the version a rollback resolves, are unchanged. Nothing in history is deleted. + +On a 17.x database `sys_metadata` has no `bundle_member` column. The ceremony boots with schema changes held, so it reads that column as absent, the overlay key's NULL member, and plans the table rather than refusing it. + +```bash +os migrate organization-ownership --out plan.json # review plan.json: metadataConflicts lists every conflict +os migrate organization-ownership --apply --backup-taken --plan-file plan.json --metadata-choice ROW_ID # one per listed conflict +``` diff --git a/packages/cli/src/commands/migrate/organization-ownership.ts b/packages/cli/src/commands/migrate/organization-ownership.ts index 63faed9d2a6..905532e9b5e 100644 --- a/packages/cli/src/commands/migrate/organization-ownership.ts +++ b/packages/cli/src/commands/migrate/organization-ownership.ts @@ -33,6 +33,7 @@ import { import { applyOrganizationOwnership, checkpointFromDetails, + MetadataChoiceRefusal, ORGANIZATION_OWNERSHIP_MIGRATION_ID, ReviewedPlanRefusal, reviewedPlanFromFile, @@ -95,6 +96,7 @@ async function confirm(question: string): Promise { * 2): `--backup-taken`, or a yes at the prompt; a non-interactive run without * the flag is refused before the boot. It then * performs the fates in order — attribution, the authored catalog conversion, + * the promotion of organization customizations and public-form withdrawals, * the template promotion, the settings global-rung move, the verified id→name * rewrite, mirror deletion, column drops — through `../../utils/organization-ownership-apply.ts`, * checkpointing each finished step in the deployment ledger (`sys_migration`, @@ -143,7 +145,9 @@ export default class MigrateOrganizationOwnership extends Command { 'The v18 organization-ownership ceremony (ADR-0131 D10). Without --apply, the read-only plan: per table, the fate, ' + 'the row counts, the rows whose owner cannot be derived (by id), and which tables will receive NOT NULL, written ' + 'to a file — never a row or a column changed; it refuses, naming it, any table it cannot enumerate. With --apply ' + - '(the reviewed plan file in --plan-file, and a backup), only that plan: attribution, the authored catalog conversion, the template promotion, the settings move, the ' + + '(the reviewed plan file in --plan-file, and a backup), only that plan: attribution, the authored catalog ' + + 'conversion, the promotion of organization customizations and form withdrawals, the template promotion, the ' + + 'settings move, the ' + 'verified id-to-name rewrite, mirror deletion and column drops, in that order, ' + 'resumable from a recorded checkpoint, then the post-check.'; @@ -192,6 +196,15 @@ export default class MigrateOrganizationOwnership extends Command { 'deleted only after the chosen body is promoted and verified', multiple: true, }), + 'metadata-choice': Flags.string({ + description: + 'With --apply: the sys_metadata row id the environment keeps for a listed customization conflict (rows ' + + 'of one type, name and package once the organization leaves the key). Repeat once per conflict. Under ' + + 'single the key keeps one environment row defining the chosen body, and its organization rows are deleted ' + + 'only after that row reads back; under a wall nothing is deleted, and a named organization row of a key it ' + + 'alone holds is promoted knowingly', + multiple: true, + }), 'fate-supplement': Flags.string({ description: 'A JSON file of the fates the host declares for the objects it provides (OrganizationOwnershipSupplementSchema, ' + @@ -256,6 +269,10 @@ export default class MigrateOrganizationOwnership extends Command { await refuse({ reason: 'flag-without-apply', detail: '--template-choice is only meaningful with --apply' }, '--template-choice is only meaningful with --apply.'); return; } + if (flags['metadata-choice']?.length) { + await refuse({ reason: 'flag-without-apply', detail: '--metadata-choice is only meaningful with --apply' }, '--metadata-choice is only meaningful with --apply.'); + return; + } if (flags['plan-file']) { await refuse({ reason: 'flag-without-apply', detail: '--plan-file is only meaningful with --apply' }, '--plan-file is only meaningful with --apply.'); return; @@ -358,6 +375,13 @@ export default class MigrateOrganizationOwnership extends Command { ' differ. --apply blocks on it until one row is chosen with --template-choice ROW_ID.', ); } + for (const conflict of table.metadataConflicts ?? []) { + printWarning( + `Customization conflict ${conflict.type} ${conflict.name}${conflict.packageId ? ` (package ${conflict.packageId})` : ''}: ` + + conflict.candidates.map((c) => `${c.id} (${c.organizationId ? `organization ${c.organizationId}` : 'environment'})`).join(', ') + + ' hold one key. --apply blocks on it until one row is chosen with --metadata-choice ROW_ID.', + ); + } } console.log(''); const { summary } = plan; @@ -380,6 +404,7 @@ export default class MigrateOrganizationOwnership extends Command { json?: boolean; 'backup-taken'?: boolean; 'template-choice'?: string[]; + 'metadata-choice'?: string[]; 'plan-file'?: string; }, posture: string, @@ -468,7 +493,9 @@ export default class MigrateOrganizationOwnership extends Command { const engine = kernel.getService('objectql') as Parameters[0] & { registry?: unknown }; const protocol = kernel.getService('protocol') as { saveMetaItem?: (request: Record) => Promise; + getMetaItems?: (request: { type: string }) => Promise<{ items?: unknown[] } | unknown[]>; getMetaItemsForExecution?: (request: { type: string }) => Promise<{ items?: unknown[] }>; + getMetaItem?: (request: { type: string; name: string; packageId?: string }) => Promise; } | undefined; const catalog = createSecurityCatalogReader({ registry: engine.registry as never, @@ -499,15 +526,50 @@ export default class MigrateOrganizationOwnership extends Command { report = await applyOrganizationOwnership({ executor, ...(flags['template-choice']?.length ? { templateChoices: flags['template-choice'] } : {}), + ...(flags['metadata-choice']?.length ? { metadataChoices: flags['metadata-choice'] } : {}), posture, database: stack.dbLabel, catalog, door: { - save: async (type, name, body) => { + save: async (type, name, body, target) => { if (typeof protocol?.saveMetaItem !== 'function') { throw new Error('no metadata protocol is composed on this stack'); } - await protocol.saveMetaItem({ type, name, item: body, actor: CEREMONY_ACTOR }); + await protocol.saveMetaItem({ + type, + name, + item: body, + actor: CEREMONY_ACTOR, + ...(target ? { packageId: target.packageId } : {}), + }); + }, + }, + lock: { + // The save door's own verdict: the item read reports it as `editable`. + verdict: async (type, name, packageId) => { + if (typeof protocol?.getMetaItem !== 'function') { + throw new Error('no metadata protocol is composed on this stack'); + } + const read = (await protocol.getMetaItem({ type, name, ...(packageId ? { packageId } : {}) })) as + | { editable?: unknown; lock?: unknown; lockReason?: unknown } + | null + | undefined; + return { + editable: read?.editable !== false, + ...(typeof read?.lock === 'string' ? { lock: read.lock } : {}), + ...(typeof read?.lockReason === 'string' ? { lockReason: read.lockReason } : {}), + }; + }, + }, + views: { + // The env-wide view list, exactly as the anonymous form doors read it. + list: async () => { + if (typeof protocol?.getMetaItems !== 'function') { + throw new Error('no metadata protocol is composed on this stack'); + } + const result = await protocol.getMetaItems({ type: 'view' }); + const items = Array.isArray(result) ? result : (result?.items ?? []); + return items.filter((i): i is Record => !!i && typeof i === 'object'); }, }, templates: { @@ -570,6 +632,10 @@ export default class MigrateOrganizationOwnership extends Command { await refuse({ reason: error.reason, ...error.detail, detail: error.message }, `Refused: ${error.message}`); return; } + if (error instanceof MetadataChoiceRefusal) { + await refuse({ reason: 'metadata-choice-refused', detail: error.message }, `Refused: ${error.message}`); + return; + } if (error instanceof TemplateChoiceRefusal) { await refuse({ reason: 'template-choice-refused', detail: error.message }, `Refused: ${error.message}`); return; diff --git a/packages/cli/src/utils/organization-ownership-apply.served-catalog.integration.test.ts b/packages/cli/src/utils/organization-ownership-apply.served-catalog.integration.test.ts index 66243655afc..4db326677d1 100644 --- a/packages/cli/src/utils/organization-ownership-apply.served-catalog.integration.test.ts +++ b/packages/cli/src/utils/organization-ownership-apply.served-catalog.integration.test.ts @@ -130,6 +130,9 @@ async function ceremony(posture: string, db: Database.Database): Promise [] }, + lock: { verdict: async () => ({ editable: true }) }, door: { save: async (type, name) => { throw new Error(`the door was asked to save ${type} ${name}`); } }, templates: { find: async () => undefined }, columnDrops: { drop: async (columns) => columns.map((c) => ({ table: c.table, column: c.column, outcome: 'not-proposed' as const })) }, diff --git a/packages/cli/src/utils/organization-ownership-apply.test.ts b/packages/cli/src/utils/organization-ownership-apply.test.ts index 166859775b7..fbb5601837c 100644 --- a/packages/cli/src/utils/organization-ownership-apply.test.ts +++ b/packages/cli/src/utils/organization-ownership-apply.test.ts @@ -22,10 +22,13 @@ import { mkdtempSync, readFileSync, rmSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { permissionSetDefinitionFromRow } from '@objectstack/core'; -import { inventoryDigest } from './organization-ownership-plan.js'; +import { anonymousFormIntakeCandidates, anonymousFormIntakeWithdrawnIn, hashSpec } from '@objectstack/metadata-core'; +import { buildOrganizationOwnershipPlan, inventoryDigest } from './organization-ownership-plan.js'; import { applyOrganizationOwnership, + MetadataChoiceRefusal, ReviewedPlanRefusal, + reviewedConflictsOf, reviewedPlanFromFile, TemplateChoiceRefusal, type ApplyExecutor, @@ -34,11 +37,13 @@ import { type CeremonyLedgerPort, type ColumnDropPort, type MetadataDoorPort, + type MetadataLockPort, type OrganizationOwnershipApplyReport, + type ReviewedConflict, type ReviewedPlan, + type ServedViewsPort, type TemplatePort, } from './organization-ownership-apply.js'; -import { buildOrganizationOwnershipPlan } from './organization-ownership-plan.js'; import { backfillOrgAdminGrants } from '@objectstack/plugin-security'; import { assertEngineDeleteDispatch } from '@objectstack/metadata-core'; @@ -74,6 +79,8 @@ function executor(opts: { failOnceOn?: RegExp } = {}): ApplyExecutor { function catalog(): CatalogPort & MetadataDoorPort & TemplatePort & { saved: Map>; templates: Map>; + /** Environment `view` saves, in order, with the package each was saved into. */ + viewSaves: Array<{ name: string; body: Record; packageId: string | null }>; boot(): void; } { const declared = new Map>([ @@ -87,9 +94,11 @@ function catalog(): CatalogPort & MetadataDoorPort & TemplatePort & { ['auth.reset_password|en-US', { name: 'auth.reset_password', label: 'Reset', locale: 'en-US', subject: 'Reset', bodyHtml: '

code

' }], ]); let hydrated = new Map>(); + const viewSaves: Array<{ name: string; body: Record; packageId: string | null }> = []; return { saved, templates, + viewSaves, async find(name, locale) { const own = templates.get(`${name}|${locale}`); if (own) return { definition: own, environment: true }; @@ -103,7 +112,11 @@ function catalog(): CatalogPort & MetadataDoorPort & TemplatePort & { const definition = hydrated.get(`${type}|${name}`) ?? declared.get(`${type}|${name}`); return definition ? { definition } : undefined; }, - async save(type, name, body) { + async save(type, name, body, target) { + if (type === 'view') { + viewSaves.push({ name, body, packageId: target?.packageId ?? null }); + return; + } if (type === 'email_template') { if (!/^[a-z][a-z0-9_]*(\.[a-z][a-z0-9_]*)*$/.test(name)) throw new Error(`VALIDATION_ERROR: ${name} is not dotted snake_case`); templates.set(`${name}|${String(body.locale)}`, body); @@ -115,6 +128,114 @@ function catalog(): CatalogPort & MetadataDoorPort & TemplatePort & { }; } +/** A form view the code package `pkg_a` ships, open to anonymous intake at `/forms/lead-intake`. */ +const formView = (open: boolean, label = 'Lead intake'): Record => ({ + name: 'crm_lead.intake', + label, + object: 'crm_lead', + viewKind: 'form', + config: { sharing: { enabled: open, allowAnonymous: true, publicLink: '/forms/lead-intake' } }, +}); + +/** + * The env-wide view list as the anonymous doors read it: the code package's + * items, replaced per (name, package) by the active environment rows of + * `sys_metadata` and then by the door's saves this run. + */ +function servedViews(c: ReturnType, code: Array> = [{ ...formView(true), _packageId: 'pkg_a' }]): ServedViewsPort { + return { + async list() { + const slots = new Map>(); + const put = (item: Record, pkg: string | null): void => { + slots.set(`${String(item.name)}|${pkg ?? ''}`, { ...item, ...(pkg ? { _packageId: pkg } : {}) }); + }; + for (const item of code) put(item, (item._packageId as string | undefined) ?? null); + const hasTable = rows(`SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'sys_metadata'`).length > 0; + if (hasTable) { + // Once column-drops has run, every row left is the environment's. + const scoped = rows(`SELECT name FROM pragma_table_info('sys_metadata') WHERE name = 'organization_id'`).length > 0; + for (const row of rows(`SELECT * FROM sys_metadata WHERE type = 'view' AND state = 'active'${scoped ? ' AND organization_id IS NULL' : ''}`)) { + put(JSON.parse(String(row.metadata)) as Record, (row.package_id as string | null) ?? null); + } + } + for (const save of c.viewSaves) put(save.body, save.packageId); + return [...slots.values()]; + }, + }; +} + +/** + * The door as the metadata repository writes a save: onto the key's one active + * environment row of `sys_metadata` (created when the key has none), numbered + * from history in the environment's own scope while the column stands + * (`event_seq` and the item's `version`, each `MAX + 1`). A key holding two + * environment rows is refused: the repository's own pick there is a coin toss. + */ +function storingDoor(c: ReturnType): ReturnType & { envRowsAtSave: number[] } { + const has = (table: string, column?: string): boolean => + column === undefined + ? rows(`SELECT name FROM sqlite_master WHERE type = 'table' AND name = '${table}'`).length > 0 + : rows(`SELECT name FROM pragma_table_info('${table}') WHERE name = '${column}'`).length > 0; + const envRowsAtSave: number[] = []; + return { + ...c, + envRowsAtSave, + async save(type, name, body, target) { + await c.save(type, name, body, target); + // The catalog conversion and the template promotion keep the catalog double's own store. + if (!['view', 'dashboard', 'report', 'translation'].includes(type)) return; + const pkg = target?.packageId ?? null; + let version = 1; + if (has('sys_metadata_history')) { + const scoped = has('sys_metadata_history', 'organization_id'); + const env = scoped ? 'organization_id IS NULL' : '1 = 1'; + const seq = Number((db.prepare(`SELECT COALESCE(MAX(event_seq), 0) + 1 AS n FROM sys_metadata_history WHERE ${env}`).get() as { n: number }).n); + version = Number( + (db.prepare(`SELECT COALESCE(MAX(version), 0) + 1 AS n FROM sys_metadata_history WHERE ${env} AND type = ? AND name = ?`).get(type, name) as { n: number }).n, + ); + db.prepare( + `INSERT INTO sys_metadata_history (id, ${scoped ? 'organization_id, ' : ''}type, name, version, event_seq, checksum) ` + + `VALUES (?, ${scoped ? 'NULL, ' : ''}?, ?, ?, ?, ?)`, + ).run(`h_save_${seq}`, type, name, version, seq, `save_${pkg ?? ''}`); + } + if (!has('sys_metadata')) return; + const scoped = has('sys_metadata', 'organization_id'); + const where = `${scoped ? 'organization_id IS NULL AND ' : ''}type = ? AND name = ? AND COALESCE(package_id, '') = ? AND state = 'active'`; + const found = db.prepare(`SELECT id FROM sys_metadata WHERE ${where}`).all(type, name, pkg ?? '') as Array<{ id: string }>; + envRowsAtSave.push(found.length); + if (found.length > 1) throw new Error(`two environment rows hold ${type} ${name}: the write head is a coin toss`); + const versioned = has('sys_metadata', 'version'); + if (found.length === 1) { + db.prepare(`UPDATE sys_metadata SET metadata = ?${versioned ? ', version = ?' : ''} WHERE id = ?`).run( + JSON.stringify(body), + ...(versioned ? [version] : []), + found[0]!.id, + ); + return; + } + const cols = ['id', 'type', 'name', 'state', 'package_id', 'metadata', ...(versioned ? ['version'] : [])]; + db.prepare(`INSERT INTO sys_metadata (${cols.join(', ')}) VALUES (${cols.map(() => '?').join(', ')})`).run( + `door_${type}_${name}_${pkg ?? ''}`, + type, + name, + 'active', + pkg, + JSON.stringify(body), + ...(versioned ? [version] : []), + ); + }, + }; +} + +/** What the anonymous doors serve for a slug, by their own judgement over the env-wide list. */ +async function doorsServe(views: ServedViewsPort, slug: string): Promise { + const list = await views.list(); + return list + .filter((v) => anonymousFormIntakeCandidates(v).some((cand) => cand.slug === slug && !anonymousFormIntakeWithdrawnIn(list, v, cand))) + .map((v) => `${String(v.name)}|${String(v._packageId ?? '')}`) + .sort(); +} + function ledger(): CeremonyLedgerPort & { checkpoint: CeremonyCheckpoint | null; verdicts: Array<{ complete: boolean; blocking: number; advisory: number }> } { const state = { checkpoint: null as CeremonyCheckpoint | null, @@ -292,32 +413,55 @@ async function accessAfter(c: ReturnType): Promise { const apply = ( c: ReturnType, l: ReturnType, - opts: { posture?: string; executor?: ApplyExecutor; templateChoices?: string[]; reviewed?: Partial } = {}, + opts: { + posture?: string; + executor?: ApplyExecutor; + templateChoices?: string[]; + metadataChoices?: string[]; + views?: ServedViewsPort; + columnDrops?: ColumnDropPort; + reviewed?: Partial; + lock?: MetadataLockPort; + } = {}, ): Promise => { c.boot(); - return applyOrganizationOwnership({ - // The plan the operator reviewed: by default, the one this run computes. + return reviewedConflictsNow(opts.posture ?? 'single').then((conflicts) => applyOrganizationOwnership({ + views: opts.views ?? servedViews(c), + lock: opts.lock ?? { verdict: async () => ({ editable: true }) }, + // The plan the operator reviewed: by default, the one a plan run computes + // over the database as it stands, conflict groups included. reviewedPlan: { file: 'plan.json', generatedAt: '2026-10-11T00:00:00Z', database: file, inventoryDigest: inventoryDigest(), posture: opts.posture ?? 'single', + conflicts, ...opts.reviewed, }, executor: opts.executor ?? executor(), ...(opts.templateChoices ? { templateChoices: opts.templateChoices } : {}), + ...(opts.metadataChoices ? { metadataChoices: opts.metadataChoices } : {}), posture: opts.posture ?? 'single', database: file, catalog: c, door: c, templates: c, - columnDrops, + columnDrops: opts.columnDrops ?? columnDrops, ledger: l, now: () => new Date('2026-10-12T00:00:00Z'), - }); + })); }; +/** The conflict groups a plan run lists over the database as it stands (none when the plan itself refuses). */ +async function reviewedConflictsNow(posture: string): Promise { + try { + return reviewedConflictsOf(await buildOrganizationOwnershipPlan({ reader: executor(), posture, database: file })); + } catch { + return []; + } +} + /** Run `body` against a database holding nothing but the given templates, so every step can finish under any posture. */ async function onlyTemplates(values: string, body: () => Promise): Promise { const only = new Database(join(dir, 'templates.db')); @@ -368,10 +512,12 @@ describe('ADR-0131 D10 --apply — the acceptance fixture', () => { expect(report.steps.map((s) => [s.step, s.status])).toEqual([ ['attribution', 'done'], ['authored-conversion', 'done'], + ['metadata-promotion', 'done'], ['template-promotion', 'done'], ['settings-move', 'done'], ['rewrite', 'done'], ['mirror-deletion', 'done'], + ['history-renumber', 'done'], ['column-drops', 'done'], ]); expect(report.complete).toBe(true); @@ -546,10 +692,24 @@ describe('reviewedPlanFromFile', () => { it('reads the compared fields of a plan file', () => { expect(reviewedPlanFromFile('p.json', JSON.stringify({ ...plan, tables: [] }))).toEqual({ - file: 'p.json', generatedAt: 'g', database: 'd', inventoryDigest: 'x', posture: 'single', + file: 'p.json', generatedAt: 'g', database: 'd', inventoryDigest: 'x', posture: 'single', conflicts: [], }); }); + it('reads both conflict lists the plan file records, keyed and sorted', () => { + const tables = [ + { + object: 'sys_metadata', + metadataConflicts: [{ type: 'view', name: 'deal_list', packageId: 'pkg_a', bundleMember: null, candidates: [{ id: 'md_b' }, { id: 'md_a' }] }], + }, + { object: 'sys_email_template', templateConflicts: [{ name: 'crm.note', locale: 'en-US', candidates: [{ id: 'tpl_b' }, { id: 'tpl_a' }] }] }, + ]; + expect(reviewedPlanFromFile('p.json', JSON.stringify({ ...plan, tables })).conflicts).toEqual([ + { kind: 'metadata', key: 'view deal_list (package pkg_a)', candidates: ['md_a', 'md_b'] }, + { kind: 'template', key: 'crm.note (en-US)', candidates: ['tpl_a', 'tpl_b'] }, + ]); + }); + it.each([ ['not JSON', '{'], ['an apply report, not a plan', JSON.stringify({ ...plan, kind: 'organization-ownership-apply' })], @@ -577,7 +737,7 @@ describe('ADR-0131 D10 --apply — idempotent, resumable, gated', () => { const again = await apply(c, l); expect(again.resumedFrom).toEqual([ - 'attribution', 'authored-conversion', 'template-promotion', 'settings-move', 'rewrite', 'mirror-deletion', 'column-drops', + 'attribution', 'authored-conversion', 'metadata-promotion', 'template-promotion', 'settings-move', 'rewrite', 'mirror-deletion', 'history-renumber', 'column-drops', ]); expect(again.steps.every((s) => s.status === 'checkpointed')).toBe(true); expect(again.complete).toBe(true); @@ -589,14 +749,14 @@ describe('ADR-0131 D10 --apply — idempotent, resumable, gated', () => { const c = catalog(); const l = ledger(); await expect(apply(c, l, { executor: executor({ failOnceOn: /^DELETE FROM "sys_capability"/ }) })).rejects.toThrow(/killed mid-step/); - expect(Object.keys(l.checkpoint!.completed)).toEqual(['attribution', 'authored-conversion', 'template-promotion', 'settings-move', 'rewrite']); + expect(Object.keys(l.checkpoint!.completed)).toEqual(['attribution', 'authored-conversion', 'metadata-promotion', 'template-promotion', 'settings-move', 'rewrite']); expect(rows('SELECT COUNT(*) AS n FROM sys_position_permission_set')).toEqual([{ n: 3 }]); writes = []; const resumed = await apply(c, l); - expect(resumed.resumedFrom).toEqual(['attribution', 'authored-conversion', 'template-promotion', 'settings-move', 'rewrite']); - expect(resumed.steps.map((s) => s.status)).toEqual(['checkpointed', 'checkpointed', 'checkpointed', 'checkpointed', 'checkpointed', 'done', 'done']); + expect(resumed.resumedFrom).toEqual(['attribution', 'authored-conversion', 'metadata-promotion', 'template-promotion', 'settings-move', 'rewrite']); + expect(resumed.steps.map((s) => s.status)).toEqual(['checkpointed', 'checkpointed', 'checkpointed', 'checkpointed', 'checkpointed', 'checkpointed', 'done', 'done', 'done']); expect(writes.some((w) => /^UPDATE/.test(w))).toBe(false); expect(resumed.complete).toBe(true); expect(rows('SELECT COUNT(*) AS n FROM sys_position_permission_set')).toEqual([{ n: 0 }]); @@ -847,7 +1007,9 @@ describe('ADR-0131 D10 --apply — idempotent, resumable, gated', () => { // The control: the same table on the resumed run, answered `not-proposed` // (a declared object the differ proposed no drop for). const notProposed = await answering('not-proposed'); - expect(notProposed.resumedFrom).toEqual(['attribution', 'authored-conversion', 'template-promotion', 'settings-move', 'rewrite', 'mirror-deletion']); + expect(notProposed.resumedFrom).toEqual([ + 'attribution', 'authored-conversion', 'metadata-promotion', 'template-promotion', 'settings-move', 'rewrite', 'mirror-deletion', 'history-renumber', + ]); const control = notProposed.steps.find((s) => s.step === 'column-drops')!; expect(control.status).toBe('blocked'); expect(control.summary).toEqual({ dropped: 0, due: 1, notDeclared: 0 }); @@ -858,7 +1020,7 @@ describe('ADR-0131 D10 --apply — idempotent, resumable, gated', () => { it('holds a column whose rows another stage owns, and reports it', async () => { db.exec(` CREATE TABLE sys_metadata (id TEXT PRIMARY KEY, organization_id TEXT, type TEXT, name TEXT, state TEXT, package_id TEXT); - INSERT INTO sys_metadata VALUES ('md_1', 'org_b', 'view', 'deal_list', 'active', NULL); + INSERT INTO sys_metadata VALUES ('md_1', 'org_b', 'object', 'deal', 'active', NULL); `); const report = await apply(catalog(), ledger()); @@ -870,6 +1032,839 @@ describe('ADR-0131 D10 --apply — idempotent, resumable, gated', () => { }); }); +describe('ADR-0131 D10 --apply — organization customizations and form withdrawals (rulings A on #22011 and #22008)', () => { + const metadataTable = (values: string): void => { + db.exec(` + CREATE TABLE sys_metadata (id TEXT PRIMARY KEY, organization_id TEXT, type TEXT, name TEXT, state TEXT, + package_id TEXT, bundle_member TEXT, metadata TEXT); + ${values ? `INSERT INTO sys_metadata VALUES ${values};` : ''} + `); + }; + const md = (id: string, org: string | null, type: string, name: string, body: Record, pkg: string | null = null): string => + `('${id}', ${org === null ? 'NULL' : `'${org}'`}, '${type}', '${name}', 'active', ${pkg === null ? 'NULL' : `'${pkg}'`}, NULL, '${JSON.stringify(body)}')`; + const step = (report: OrganizationOwnershipApplyReport) => report.steps.find((s) => s.step === 'metadata-promotion')!; + /** Each row's owner; once column-drops has dropped the column, every row reads as the environment's (`null`). */ + const owners = (): Array<[unknown, unknown]> => { + const scoped = rows(`SELECT name FROM pragma_table_info('sys_metadata') WHERE name = 'organization_id'`).length > 0; + return rows(`SELECT id${scoped ? ', organization_id' : ''} FROM sys_metadata ORDER BY id`).map((r) => [r.id, r.organization_id ?? null]); + }; + + it('under single, an organization customization no other row holds becomes an environment row in place', async () => { + metadataTable([ + md('md_dash', 'org_default', 'dashboard', 'sales_board', { name: 'sales_board', label: 'Org board' }), + md('md_env', null, 'report', 'pipeline', { name: 'pipeline', label: 'Env report' }), + md('md_obj', 'org_default', 'object', 'deal', { name: 'deal' }), + ].join(', ')); + + const report = await apply(catalog(), ledger()); + + expect(step(report)).toMatchObject({ status: 'done', summary: { promoted: 1, conflicts: 0 } }); + // The non-presentational row is reported, never promoted; nothing is deleted. + expect(owners()).toEqual([['md_dash', null], ['md_env', null], ['md_obj', 'org_default']]); + }); + + it('a key held by an organization row and the environment row is a conflict: kept, listed by id, blocking; nothing is written', async () => { + metadataTable([ + md('md_org', 'org_default', 'view', 'deal_list', { name: 'deal_list', label: 'Org list' }), + md('md_env', null, 'view', 'deal_list', { name: 'deal_list', label: 'Env list' }), + ].join(', ')); + const c = catalog(); + + const report = await apply(c, ledger()); + + expect(step(report).status).toBe('blocked'); + expect(step(report).blockers.join('\n')).toMatch(/view deal_list is held by 2 rows .*md_env \(environment, active\), md_org \(organization org_default, active\)/); + expect(writes.filter((w) => /sys_metadata/.test(w))).toEqual([]); + expect(owners()).toEqual([['md_env', null], ['md_org', 'org_default']]); + expect(c.viewSaves).toEqual([]); + + // The operator keeps one row; the re-run promotes nothing more and goes on. + db.exec(`DELETE FROM sys_metadata WHERE id = 'md_org'`); + const again = await apply(c, ledger()); + expect(step(again)).toMatchObject({ status: 'done', summary: { promoted: 0, conflicts: 0 } }); + }); + + it('two organizations holding one key is a conflict too', async () => { + metadataTable([ + md('md_a', 'org_a', 'translation', 'fr', { name: 'fr' }), + md('md_b', 'org_b', 'translation', 'fr', { name: 'fr' }), + ].join(', ')); + const report = await apply(catalog(), ledger()); + expect(step(report).summary.conflicts).toBe(1); + expect(owners()).toEqual([['md_a', 'org_a'], ['md_b', 'org_b']]); + }); + + it('under a wall the rows are reported, never promoted, and they hold the column', async () => { + metadataTable(md('md_dash', 'org_a', 'dashboard', 'sales_board', { name: 'sales_board' })); + const report = await apply(catalog(), ledger(), { posture: 'isolated' }); + expect(step(report)).toMatchObject({ status: 'done', summary: { promoted: 0, reported: 1 } }); + expect(owners()).toEqual([['md_dash', 'org_a']]); + }); + + for (const posture of ['single', 'isolated'] as const) { + it(`⭐ a form an organization withdrew before the upgrade is refused at the anonymous doors after it, across packages (${posture})`, async () => { + // The organization withdrew the package-less copy; package pkg_a serves its own copy, open. + metadataTable(md('md_org', 'org_default', 'view', 'crm_lead.intake', formView(false, 'Org copy'))); + const c = catalog(); + const views = servedViews(c); + expect(await doorsServe(views, 'lead-intake')).toEqual(['crm_lead.intake|pkg_a']); + + const report = await apply(c, ledger(), { posture, views }); + + expect(step(report).status).toBe('done'); + expect(step(report).summary.withdrawalsCarried).toBe(1); + // Carried onto pkg_a's own copy, through the door, into its own package. + expect(c.viewSaves.map((v) => [v.name, v.packageId, (v.body.config as any).sharing.enabled])).toEqual([ + ['crm_lead.intake', 'pkg_a', false], + ]); + expect(await doorsServe(views, 'lead-intake')).toEqual([]); + + // Idempotent: a second run (no checkpoint) carries nothing more. + const again = await apply(c, ledger(), { posture, views }); + expect(step(again).summary.withdrawalsCarried).toBe(0); + expect(c.viewSaves).toHaveLength(1); + }); + } + + it('⭐ under single, a withdrawal the organization holds on the key it alone holds travels with the promoted row', async () => { + metadataTable(md('md_org', 'org_default', 'view', 'crm_lead.intake', formView(false, 'Org copy'), 'pkg_a')); + const c = catalog(); + const views = servedViews(c); + + const report = await apply(c, ledger(), { views }); + + expect(step(report).summary).toMatchObject({ promoted: 1, withdrawalsCarried: 0 }); + expect(c.viewSaves).toEqual([]); + expect(owners()).toEqual([['md_org', null]]); + expect(await doorsServe(views, 'lead-intake')).toEqual([]); + }); + + it('control: an organization row that withdraws nothing carries nothing, and the form stays served', async () => { + metadataTable(md('md_org', 'org_default', 'view', 'crm_lead.intake', formView(true, 'Org copy'))); + const c = catalog(); + const views = servedViews(c); + const report = await apply(c, ledger(), { views }); + expect(step(report).summary.withdrawalsCarried).toBe(0); + expect(c.viewSaves).toEqual([]); + expect(await doorsServe(views, 'lead-intake')).toEqual(['crm_lead.intake|', 'crm_lead.intake|pkg_a']); + }); + + it('a door that refuses the carry blocks the step, naming the form and the remedy', async () => { + metadataTable(md('md_org', 'org_default', 'view', 'crm_lead.intake', formView(false))); + const c = catalog(); + const refusing: ReturnType = { + ...c, + async save(type, name, body, target) { + if (type === 'view') throw new Error('NOT_OVERRIDABLE: the view is sealed'); + return c.save(type, name, body, target); + }, + }; + const report = await apply(refusing, ledger(), { views: servedViews(c) }); + expect(step(report).status).toBe('blocked'); + expect(step(report).blockers.join('\n')).toMatch(/lead-intake of the view crm_lead\.intake \(package pkg_a\).*NOT_OVERRIDABLE.*sharing\.enabled false/); + }); +}); + +describe('ADR-0131 D10 --apply — a customization conflict resolved by --metadata-choice (ruling 6163114098: Q1 B, Q6 A)', () => { + const metadataTable = (values: string): void => { + db.exec(` + CREATE TABLE sys_metadata (id TEXT PRIMARY KEY, organization_id TEXT, type TEXT, name TEXT, state TEXT, + package_id TEXT, bundle_member TEXT, metadata TEXT); + INSERT INTO sys_metadata VALUES ${values}; + `); + }; + const md = (id: string, org: string | null, type: string, name: string, body: Record, pkg: string | null = null): string => + `('${id}', ${org === null ? 'NULL' : `'${org}'`}, '${type}', '${name}', 'active', ${pkg === null ? 'NULL' : `'${pkg}'`}, NULL, '${JSON.stringify(body)}')`; + const step = (report: OrganizationOwnershipApplyReport) => report.steps.find((s) => s.step === 'metadata-promotion')!; + const owners = (): Array<[unknown, unknown]> => { + const scoped = rows(`SELECT name FROM pragma_table_info('sys_metadata') WHERE name = 'organization_id'`).length > 0; + return rows(`SELECT id${scoped ? ', organization_id' : ''} FROM sys_metadata ORDER BY id`).map((r) => [r.id, r.organization_id ?? null]); + }; + const labelOf = (id: string): unknown => (JSON.parse(String(rows(`SELECT metadata FROM sys_metadata WHERE id = '${id}'`)[0]!.metadata)) as { label?: unknown }).label; + const deletes = (): string[] => writes.filter((w) => /^DELETE FROM "sys_metadata"/.test(w)); + const conflictRows = [ + md('md_org', 'org_default', 'view', 'deal_list', { name: 'deal_list', label: 'Org list' }), + md('md_env', null, 'view', 'deal_list', { name: 'deal_list', label: 'Env list' }), + ].join(', '); + + it('⭐ no row of a conflict is deleted before a choice is made', async () => { + metadataTable(conflictRows); + const report = await apply(storingDoor(catalog()), ledger()); + expect(step(report).status).toBe('blocked'); + expect(step(report).blockers.join('\n')).toMatch(/choose one with --metadata-choice ROW_ID/); + expect(deletes()).toEqual([]); + expect(owners()).toEqual([['md_env', null], ['md_org', 'org_default']]); + }); + + it('⭐ choosing the organization\'s row over the environment\'s: the one environment row takes its body in place, then the organization row goes', async () => { + metadataTable(conflictRows); + const c = storingDoor(catalog()); + const report = await apply(c, ledger(), { metadataChoices: ['md_org'] }); + expect(step(report)).toMatchObject({ status: 'done', summary: { chosen: 1, organizationRowsDeleted: 1, conflicts: 0 } }); + expect(owners()).toEqual([['md_env', null]]); + expect(labelOf('md_env')).toBe('Org list'); + // Exactly one environment row at every point: the environment row took the + // body in place, by one UPDATE, and no UPDATE ever cleared an organization + // row of this key. No withdrawal to record, so the door was not asked. + expect(writes.filter((w) => /^UPDATE "sys_metadata" SET "metadata" = \?/.test(w))).toHaveLength(1); + expect(writes.some((w) => /^UPDATE "sys_metadata" SET "organization_id" = NULL/.test(w))).toBe(false); + expect(c.viewSaves).toEqual([]); + }); + + it('⭐ an item a package ships: a package-less environment row of the key takes the body, though the door writes only into the package', async () => { + // The real door writes an environment customization of a packaged item into + // that package's slot, whatever the target (measured on the showcase). + metadataTable([ + md('md_org', 'org_default', 'view', 'crm_lead.intake', formView(true, 'Org copy')), + md('md_env', null, 'view', 'crm_lead.intake', formView(false, 'Env copy')), + ].join(', ')); + const base = storingDoor(catalog()); + const packaged: ReturnType = { + ...base, + async save(type, name, body) { + return base.save(type, name, body, { packageId: 'pkg_a' }); + }, + }; + const report = await apply(packaged, ledger(), { views: servedViews(base, []), metadataChoices: ['md_org'] }); + expect(step(report)).toMatchObject({ status: 'done', summary: { chosen: 1, organizationRowsDeleted: 1 } }); + // The key's own environment row holds the chosen body, with the environment row's withdrawal kept. + expect(labelOf('md_env')).toBe('Org copy'); + expect((JSON.parse(String(rows(`SELECT metadata FROM sys_metadata WHERE id = 'md_env'`)[0]!.metadata)) as any).config.sharing.enabled).toBe(false); + expect(rows(`SELECT id FROM sys_metadata WHERE id = 'md_org'`)).toEqual([]); + }); + + it('the environment row is read back before any delete: a write that does not land deletes nothing', async () => { + metadataTable(conflictRows); + const c = catalog(); + const base = executor(); + const lost: ApplyExecutor = { + ...base, + async execute(statement, params) { + if (/^UPDATE "sys_metadata" SET "metadata"/.test(statement)) return; + return base.execute(statement, params); + }, + }; + await expect(apply(c, ledger(), { metadataChoices: ['md_org'], executor: lost })).rejects.toThrow(/did not read back as one environment row/); + expect(deletes()).toEqual([]); + expect(owners()).toEqual([['md_env', null], ['md_org', 'org_default']]); + }); + + it('⭐ a crash between the replacement and the delete: the same --metadata-choice finishes it', async () => { + metadataTable(conflictRows); + const c = storingDoor(catalog()); + const l = ledger(); + await expect(apply(c, l, { metadataChoices: ['md_org'], executor: executor({ failOnceOn: /^DELETE FROM "sys_metadata"/ }) })).rejects.toThrow( + /killed mid-step/, + ); + // The replacement landed and is recorded; the organization row is still there. + expect(labelOf('md_env')).toBe('Org list'); + expect(owners()).toEqual([['md_env', null], ['md_org', 'org_default']]); + expect(l.checkpoint!.metadataChoicesApplied).toEqual(['md_org']); + + const again = await apply(c, l, { metadataChoices: ['md_org'] }); + expect(step(again)).toMatchObject({ status: 'done', summary: { chosen: 1, organizationRowsDeleted: 1 } }); + expect(owners()).toEqual([['md_env', null]]); + + // And after the delete, before the step's checkpoint: the choice names a row + // that is gone, which the checkpoint records as applied. + l.checkpoint!.completed = {}; + const third = await apply(c, l, { metadataChoices: ['md_org'] }); + expect(step(third).status).toBe('done'); + expect(owners()).toEqual([['md_env', null]]); + }); + + it('choosing the environment\'s row: the organization row is deleted by id, and nothing is promoted', async () => { + metadataTable(conflictRows); + const report = await apply(storingDoor(catalog()), ledger(), { metadataChoices: ['md_env'] }); + expect(step(report)).toMatchObject({ status: 'done', summary: { chosen: 1, organizationRowsDeleted: 1, promoted: 0 } }); + expect(owners()).toEqual([['md_env', null]]); + expect(labelOf('md_env')).toBe('Env list'); + }); + + it('two organizations on one key and no environment row: the chosen one is promoted in place, the other deleted', async () => { + metadataTable([md('md_a', 'org_a', 'translation', 'fr', { name: 'fr', a: 1 }), md('md_b', 'org_b', 'translation', 'fr', { name: 'fr', b: 1 })].join(', ')); + const l = ledger(); + const c = storingDoor(catalog()); + const report = await apply(c, l, { metadataChoices: ['md_b'] }); + expect(step(report).summary).toMatchObject({ chosen: 1, organizationRowsDeleted: 1 }); + expect(owners()).toEqual([['md_b', null]]); + // A rerun with the same choice: the chosen row is the key's environment row now. + l.checkpoint!.completed = {}; + expect(step(await apply(c, l, { metadataChoices: ['md_b'] })).status).toBe('done'); + }); + + it('⭐ fail-closed: choosing a row that serves a form an unchosen row withdraws carries the withdrawal onto it before the delete', async () => { + metadataTable([ + md('md_org', 'org_default', 'view', 'crm_lead.intake', formView(false, 'Org copy')), + md('md_env', null, 'view', 'crm_lead.intake', formView(true, 'Env copy')), + ].join(', ')); + const c = storingDoor(catalog()); + const views = servedViews(c, []); + const report = await apply(c, ledger(), { views, metadataChoices: ['md_env'] }); + expect(step(report).status).toBe('done'); + expect(owners()).toEqual([['md_env', null]]); + expect((JSON.parse(String(rows(`SELECT metadata FROM sys_metadata WHERE id = 'md_env'`)[0]!.metadata)) as any).config.sharing.enabled).toBe(false); + expect(await doorsServe(views, 'lead-intake')).toEqual([]); + }); + + it('a door that refuses the chosen body blocks the conflict, and no organization row is deleted', async () => { + metadataTable([ + md('md_org', 'org_default', 'view', 'crm_lead.intake', formView(false, 'Org copy')), + md('md_env', null, 'view', 'crm_lead.intake', formView(true, 'Env copy')), + ].join(', ')); + const c = storingDoor(catalog()); + const refusing: ReturnType = { + ...c, + async save(type, name, body, target) { + if (type === 'view' && target?.packageId === null) throw new Error('VALIDATION_ERROR: refused'); + return c.save(type, name, body, target); + }, + }; + const report = await apply(refusing, ledger(), { views: servedViews(c, []), metadataChoices: ['md_env'] }); + expect(step(report).status).toBe('blocked'); + expect(step(report).blockers.join('\n')).toMatch(/Nothing was deleted; fix the cause, then re-run/); + expect(deletes()).toEqual([]); + expect(owners()).toEqual([['md_env', null], ['md_org', 'org_default']]); + }); + + describe('the in-place write does what the door does to that row (N2)', () => { + const withChecksum = (): void => { + db.exec(`ALTER TABLE sys_metadata ADD COLUMN checksum TEXT; UPDATE sys_metadata SET checksum = 'sha256:stale';`); + }; + + it('⭐ restamps checksum from the written body, as the door does (the version, the etag, the no-op check)', async () => { + metadataTable(conflictRows); + withChecksum(); + await apply(storingDoor(catalog()), ledger(), { metadataChoices: ['md_org'] }); + const [row] = rows(`SELECT metadata, checksum FROM sys_metadata WHERE id = 'md_env'`); + expect(row!.checksum).toBe(hashSpec(JSON.parse(String(row!.metadata)), 'view')); + expect(row!.checksum).not.toBe('sha256:stale'); + }); + + it('⭐ keeps the lock family the environment row stores, whatever the chosen body carries', async () => { + metadataTable([ + md('md_org', 'org_default', 'view', 'deal_list', { name: 'deal_list', label: 'Org list', _lock: 'none' }), + md('md_env', null, 'view', 'deal_list', { name: 'deal_list', label: 'Env list', _lock: 'customize', _lockReason: 'stamped' }), + ].join(', ')); + await apply(storingDoor(catalog()), ledger(), { metadataChoices: ['md_org'] }); + const body = JSON.parse(String(rows(`SELECT metadata FROM sys_metadata WHERE id = 'md_env'`)[0]!.metadata)) as Record; + expect(body).toMatchObject({ label: 'Org list', _lock: 'customize', _lockReason: 'stamped' }); + }); + + it('⭐ a locked item stays locked: the door\'s verdict refuses the write, and nothing is written or deleted', async () => { + metadataTable(conflictRows); + const asked: Array<[string, string, string | null]> = []; + const report = await apply(storingDoor(catalog()), ledger(), { + metadataChoices: ['md_org'], + lock: { + verdict: async (type, name, packageId) => { + asked.push([type, name, packageId]); + return { editable: false, lock: 'full', lockReason: 'shipped locked' }; + }, + }, + }); + expect(asked).toEqual([['view', 'deal_list', null]]); + expect(step(report).status).toBe('blocked'); + expect(step(report).blockers.join('\n')).toMatch(/is locked \(_lock=full: shipped locked\).*Nothing was written or deleted/s); + expect(labelOf('md_env')).toBe('Env list'); + expect(deletes()).toEqual([]); + }); + }); + + it('⭐ the lock is asked before the history fold and the promotion: under a wall, a refusal leaves history and rows as they were', async () => { + metadataTable([ + md('md_org', 'org_default', 'view', 'deal_list', { name: 'deal_list', label: 'Org list' }), + md('md_env', null, 'view', 'deal_list', { name: 'deal_list', label: 'Env list' }), + md('md_lone', 'org_default', 'dashboard', 'sales', { name: 'sales' }), + ].join(', ')); + db.exec(` + CREATE TABLE sys_metadata_history (id TEXT PRIMARY KEY, organization_id TEXT, type TEXT, name TEXT, + version INTEGER, event_seq INTEGER, checksum TEXT); + INSERT INTO sys_metadata_history VALUES + ('h_env', NULL, 'view', 'deal_list', 1, 1, 'e1'), + ('h_org1', 'org_default', 'view', 'deal_list', 1, 1, 'o1'), ('h_org2', 'org_default', 'view', 'deal_list', 2, 2, 'o2'); + `); + const history = rows('SELECT * FROM sys_metadata_history ORDER BY id'); + const report = await apply(storingDoor(catalog()), ledger(), { + posture: 'isolated', + metadataChoices: ['md_org'], + columnDrops: { drop: async (columns) => columns.map(({ table, column }) => ({ table, column, outcome: 'not-proposed' as const })) }, + lock: { verdict: async (type) => (type === 'view' ? { editable: false, lock: 'full' } : { editable: true }) }, + }); + expect(step(report).status).toBe('blocked'); + expect(step(report).blockers.join('\n')).toMatch(/view deal_list could not take the body .*Nothing was written or deleted/s); + // No fold, no promotion, no write: history and both rows of the key are as they were. + expect(writes.filter((w) => /sys_metadata/.test(w))).toEqual([]); + expect(rows('SELECT * FROM sys_metadata_history ORDER BY id')).toEqual(history); + expect(owners()).toEqual([['md_env', null], ['md_lone', 'org_default'], ['md_org', 'org_default']]); + expect(labelOf('md_env')).toBe('Env list'); + }); + + it('the door save after the in-place write sends the row as written, its stored lock family kept', async () => { + metadataTable([ + md('md_org', 'org_default', 'view', 'crm_lead.intake', formView(true, 'Org copy')), + md('md_other', 'org_b', 'view', 'crm_lead.intake', formView(false, 'Other copy')), + md('md_env', null, 'view', 'crm_lead.intake', { ...formView(true, 'Env copy'), _lock: 'customize' }), + ].join(', ')); + const c = storingDoor(catalog()); + const report = await apply(c, ledger(), { views: servedViews(c, []), metadataChoices: ['md_org'] }); + expect(step(report).status).toBe('done'); + expect(c.viewSaves.at(-1)!.body).toMatchObject({ label: 'Org copy', _lock: 'customize' }); + expect((c.viewSaves.at(-1)!.body.config as any).sharing.enabled).toBe(false); + }); + + it('two conflicts of one item that differ only by bundle member are two conflicts, each chosen for', async () => { + db.exec(` + CREATE TABLE sys_metadata (id TEXT PRIMARY KEY, organization_id TEXT, type TEXT, name TEXT, state TEXT, + package_id TEXT, bundle_member TEXT, metadata TEXT); + INSERT INTO sys_metadata VALUES + ('en_org', 'org_default', 'email_template', 'crm.note', 'active', NULL, 'en-US', '{"name":"crm.note","locale":"en-US","s":"o"}'), + ('en_env', NULL, 'email_template', 'crm.note', 'active', NULL, 'en-US', '{"name":"crm.note","locale":"en-US","s":"e"}'), + ('fr_org', 'org_default', 'email_template', 'crm.note', 'active', NULL, 'fr-FR', '{"name":"crm.note","locale":"fr-FR","s":"o"}'), + ('fr_env', NULL, 'email_template', 'crm.note', 'active', NULL, 'fr-FR', '{"name":"crm.note","locale":"fr-FR","s":"e"}'); + `); + const report = await apply(storingDoor(catalog()), ledger(), { metadataChoices: ['en_org', 'fr_env'] }); + expect(step(report)).toMatchObject({ status: 'done', summary: { chosen: 2 } }); + }); + + describe('under a wall (ruling 6163114098, Q6 A: the ceremony never deletes the unchosen rows)', () => { + it('⭐ a chosen organization row defines the environment row, and nothing is deleted', async () => { + metadataTable(conflictRows); + const report = await apply(storingDoor(catalog()), ledger(), { posture: 'isolated', metadataChoices: ['md_org'] }); + expect(step(report)).toMatchObject({ status: 'done', summary: { chosen: 1, organizationRowsDeleted: 0 } }); + expect(deletes()).toEqual([]); + expect(owners()).toEqual([['md_env', null], ['md_org', 'org_default']]); + expect(labelOf('md_env')).toBe('Org list'); + // The ruled report line: each organization row left, named as remaining and blocking column-drops, with the remedy. + expect(step(report).notes).toContainEqual( + expect.stringMatching( + /^Remaining: sys_metadata row md_org \(organization org_default, the chosen row, whose body the environment row md_env now holds\) of the view deal_list blocks column-drops on sys_metadata: .*never deletes it\. Remedy: check the environment row, then delete this row yourself/, + ), + ); + }); + + it('⭐ two organizations on one key: the chosen one is promoted, the other stays', async () => { + metadataTable([md('md_a', 'org_a', 'dashboard', 'sales', { name: 'sales', a: 1 }), md('md_b', 'org_b', 'dashboard', 'sales', { name: 'sales', b: 1 })].join(', ')); + const report = await apply(storingDoor(catalog()), ledger(), { posture: 'isolated', metadataChoices: ['md_b'] }); + expect(step(report).status).toBe('done'); + expect(deletes()).toEqual([]); + expect(owners()).toEqual([['md_a', 'org_a'], ['md_b', null]]); + }); + + it('a conflict nobody chose for is reported, not blocking the step; its rows hold the column', async () => { + metadataTable(conflictRows); + const report = await apply(storingDoor(catalog()), ledger(), { posture: 'isolated' }); + expect(step(report)).toMatchObject({ status: 'done', summary: { conflicts: 1, reported: 1 } }); + expect(step(report).notes.join('\n')).toMatch(/view deal_list is held by 2 rows/); + expect(deletes()).toEqual([]); + }); + + it('a lone row is promoted only when chosen; the rest stay reported', async () => { + metadataTable([md('md_a', 'org_a', 'dashboard', 'sales', { name: 'sales' }), md('md_b', 'org_a', 'report', 'pipe', { name: 'pipe' })].join(', ')); + const report = await apply(catalog(), ledger(), { posture: 'isolated', metadataChoices: ['md_a'] }); + expect(step(report).summary).toMatchObject({ promoted: 1, reported: 1 }); + expect(owners()).toEqual([['md_a', null], ['md_b', 'org_a']]); + }); + }); + + it('refuses, before any write, a choice naming no row awaiting one, and two choices for one conflict', async () => { + metadataTable([conflictRows, md('md_lone', 'org_default', 'dashboard', 'sales', { name: 'sales' })].join(', ')); + await expect(apply(catalog(), ledger(), { metadataChoices: ['md_nope'] })).rejects.toBeInstanceOf(MetadataChoiceRefusal); + await expect(apply(catalog(), ledger(), { metadataChoices: ['md_org', 'md_env'] })).rejects.toThrow( + /md_org and md_env both choose for view deal_list/, + ); + expect(writes).toEqual([]); + }); +}); + +describe('ADR-0131 D10 --apply — a choice deletes only rows the reviewed plan listed (plan-conflict-mismatch)', () => { + const metadataConflict = (extra = ''): void => { + db.exec(` + CREATE TABLE sys_metadata (id TEXT PRIMARY KEY, organization_id TEXT, type TEXT, name TEXT, state TEXT, + package_id TEXT, bundle_member TEXT, metadata TEXT); + INSERT INTO sys_metadata VALUES + ('md_org', 'org_default', 'view', 'deal_list', 'active', NULL, NULL, '{"name":"deal_list","label":"Org"}'), + ('md_env', NULL, 'view', 'deal_list', 'active', NULL, NULL, '{"name":"deal_list","label":"Env"}')${extra}; + `); + }; + const reviewedMetadata = (candidates: string[]): Partial => ({ + conflicts: [{ kind: 'metadata', key: 'view deal_list', candidates }], + }); + const refusalOf = async (run: Promise): Promise => { + try { + await run; + } catch (error) { + return error as ReviewedPlanRefusal; + } + throw new Error('expected the apply to be refused'); + }; + + it('⭐ --metadata-choice: a row that joined the conflict since the review refuses the apply, naming it; nothing is written', async () => { + metadataConflict(`, ('md_new', 'org_b', 'view', 'deal_list', 'active', NULL, NULL, '{"name":"deal_list","label":"New"}')`); + const err = await refusalOf(apply(storingDoor(catalog()), ledger(), { metadataChoices: ['md_org'], reviewed: reviewedMetadata(['md_env', 'md_org']) })); + expect(err).toBeInstanceOf(ReviewedPlanRefusal); + expect(err).toMatchObject({ + reason: 'plan-conflict-mismatch', + detail: { flag: '--metadata-choice', choice: 'md_org', conflict: 'view deal_list', reviewed: 'md_env,md_org', current: 'md_env,md_new,md_org' }, + }); + expect(err.message).toMatch(/added since: md_new.*Nothing was written/s); + expect(writes).toEqual([]); + expect(rows('SELECT id FROM sys_metadata ORDER BY id').map((r) => r.id)).toEqual(['md_env', 'md_new', 'md_org']); + }); + + it('--metadata-choice: a reviewed row gone since refuses too, and a plan file that lists no such conflict', async () => { + metadataConflict(); + const gone = await refusalOf(apply(storingDoor(catalog()), ledger(), { metadataChoices: ['md_org'], reviewed: reviewedMetadata(['md_env', 'md_old', 'md_org']) })); + expect(gone).toMatchObject({ reason: 'plan-conflict-mismatch' }); + expect(gone.message).toMatch(/gone since: md_old/); + const unlisted = await refusalOf(apply(storingDoor(catalog()), ledger(), { metadataChoices: ['md_org'], reviewed: { conflicts: [] } })); + expect(unlisted.message).toMatch(/lists no such conflict/); + expect(writes).toEqual([]); + }); + + it('--metadata-choice: the reviewed group, unchanged, resolves as reviewed', async () => { + metadataConflict(); + const report = await apply(storingDoor(catalog()), ledger(), { metadataChoices: ['md_org'], reviewed: reviewedMetadata(['md_env', 'md_org']) }); + expect(report.steps.find((s) => s.step === 'metadata-promotion')!.summary).toMatchObject({ chosen: 1, organizationRowsDeleted: 1 }); + }); + + it('⭐ --template-choice: a row that joined the conflict since the review refuses the apply, naming it; nothing is written', async () => { + await onlyTemplates( + `('tpl_a', 'org_a', 'crm.note', 'en-US', 'From A', 1, 'platform'), ('tpl_b', 'org_b', 'crm.note', 'en-US', 'From B', 1, 'platform'), + ('tpl_c', 'org_c', 'crm.note', 'en-US', 'From C', 1, 'platform')`, + async () => { + const c = catalog(); + const err = await refusalOf(apply(c, ledger(), { + posture: 'isolated', + templateChoices: ['tpl_b'], + reviewed: { conflicts: [{ kind: 'template', key: 'crm.note (en-US)', candidates: ['tpl_a', 'tpl_b'] }] }, + })); + expect(err).toMatchObject({ + reason: 'plan-conflict-mismatch', + detail: { flag: '--template-choice', choice: 'tpl_b', conflict: 'crm.note (en-US)', current: 'tpl_a,tpl_b,tpl_c' }, + }); + expect(err.message).toMatch(/added since: tpl_c/); + expect(writes).toEqual([]); + expect(c.templates.size).toBe(0); + expect(rows('SELECT id FROM sys_email_template ORDER BY id').map((r) => r.id)).toEqual(['tpl_a', 'tpl_b', 'tpl_c']); + }, + ); + }); + + it('--template-choice: the reviewed group, unchanged, resolves as reviewed', async () => { + await onlyTemplates( + `('tpl_a', 'org_a', 'crm.note', 'en-US', 'From A', 1, 'platform'), ('tpl_b', 'org_b', 'crm.note', 'en-US', 'From B', 1, 'platform')`, + async () => { + const report = await apply(catalog(), ledger(), { + posture: 'isolated', + templateChoices: ['tpl_b'], + reviewed: { conflicts: [{ kind: 'template', key: 'crm.note (en-US)', candidates: ['tpl_a', 'tpl_b'] }] }, + }); + expect(report.complete).toBe(true); + }, + ); + }); +}); + +describe('ADR-0131 D10 --apply — history renumbering (seat answers 6162538253, Q2 A)', () => { + /** + * The unique indexes a 17.x runtime created on history (its per-scope + * counters), checked row by row: a renumbering may never land a row on a + * sibling's number mid-statement. + */ + const HISTORY_UNIQUE = ` + CREATE UNIQUE INDEX uniq_history_org_seq ON sys_metadata_history (organization_id, event_seq); + CREATE UNIQUE INDEX uniq_history_org_version ON sys_metadata_history (organization_id, type, name, version);`; + + // A database holding only what each pin plants, so every step can finish + // under any posture and the late step is reached. + beforeEach(() => { + db.close(); + file = join(dir, 'renumber.db'); + db = new Database(file); + db.exec(`CREATE TABLE sys_organization (id TEXT PRIMARY KEY, slug TEXT); INSERT INTO sys_organization VALUES ('org_default', 'default');`); + writes = []; + }); + /** The schema-drift ceremony declining the drop: the column stays, so each pin reads whose a row is after the run. */ + const keepColumn: ColumnDropPort = { + async drop(columns) { + return columns.map(({ table, column }) => ({ table, column, outcome: 'not-proposed' as const })); + }, + }; + + const history = (): void => { + db.exec(` + CREATE TABLE sys_metadata (id TEXT PRIMARY KEY, organization_id TEXT, type TEXT, name TEXT, state TEXT, + package_id TEXT, bundle_member TEXT, version INTEGER); + INSERT INTO sys_metadata VALUES + ('row_env', NULL, 'object', 'deal', 'active', NULL, NULL, 2), + ('row_a', 'org_a', 'view', 'board', 'active', NULL, NULL, 3), + ('row_b', 'org_b', 'view', 'board', 'active', 'pkg_b', NULL, 2); + ALTER TABLE sys_metadata ADD COLUMN metadata TEXT; + CREATE TABLE sys_metadata_history (id TEXT PRIMARY KEY, organization_id TEXT, type TEXT, name TEXT, + version INTEGER, event_seq INTEGER, checksum TEXT); + ${HISTORY_UNIQUE} + INSERT INTO sys_metadata_history VALUES + ('h_env1', NULL, 'object', 'deal', 1, 1, 'e1'), + ('h_env2', NULL, 'object', 'deal', 2, 2, 'e2'), + ('h_env3', NULL, 'view', 'board', 1, 3, 'e3'), + ('h_a1', 'org_a', 'view', 'board', 1, 1, 'a1'), + ('h_a2', 'org_a', 'view', 'board', 2, 2, 'a2'), + ('h_a3', 'org_a', 'view', 'board', 3, 3, 'a3'), + ('h_b1', 'org_b', 'view', 'board', 1, 1, 'b1'), + ('h_b2', 'org_b', 'view', 'board', 2, 2, 'b2'), + ('h_c1', 'org_c', 'report', 'solo', 1, 1, 'c1'); + CREATE TABLE sys_metadata_commit (id TEXT PRIMARY KEY, organization_id TEXT, event_seq_start INTEGER, event_seq_end INTEGER); + INSERT INTO sys_metadata_commit VALUES + ('c_env', NULL, 1, 3), ('c_a', 'org_a', 1, 3), ('c_b', 'org_b', 1, 2); + `); + }; + const seqs = (): Array<[unknown, unknown, unknown, unknown]> => + rows('SELECT id, organization_id, event_seq, version FROM sys_metadata_history ORDER BY id').map((r) => [r.id, r.organization_id, r.event_seq, r.version]); + const step = (report: OrganizationOwnershipApplyReport) => report.steps.find((s) => s.step === 'history-renumber')!; + const duplicates = (): unknown[] => [ + ...rows('SELECT event_seq, COUNT(*) AS n FROM sys_metadata_history GROUP BY event_seq HAVING COUNT(*) > 1'), + ...rows('SELECT type, name, version FROM sys_metadata_history GROUP BY type, name, version HAVING COUNT(*) > 1'), + ]; + + it('makes event_seq and every (type, name, version) unique environment-wide, writing numbers only', async () => { + history(); + const before = rows('SELECT COUNT(*) AS n FROM sys_metadata_history'); + const report = await apply(catalog(), ledger(), { posture: 'isolated', columnDrops: keepColumn }); + expect(step(report)).toMatchObject({ status: 'done', summary: { organizationsShifted: 3, lineagesShifted: 2 } }); + expect(rows('SELECT COUNT(*) AS n FROM sys_metadata_history')).toEqual(before); + expect(rows('SELECT event_seq, COUNT(*) AS n FROM sys_metadata_history GROUP BY event_seq HAVING COUNT(*) > 1')).toEqual([]); + expect(rows('SELECT type, name, version FROM sys_metadata_history GROUP BY type, name, version HAVING COUNT(*) > 1')).toEqual([]); + expect(writes.some((w) => /^DELETE/.test(w))).toBe(false); + }); + + it('⭐ order within each organization and the version a rollback resolves are unchanged', async () => { + history(); + await apply(catalog(), ledger(), { posture: 'isolated', columnDrops: keepColumn }); + const ordered = (org: string): string[] => + rows(`SELECT checksum FROM sys_metadata_history WHERE organization_id = '${org}' ORDER BY event_seq`).map((r) => String(r.checksum)); + expect(ordered('org_a')).toEqual(['a1', 'a2', 'a3']); + expect(ordered('org_b')).toEqual(['b1', 'b2']); + // The overlay row's version still names its own snapshot: a rollback by + // version lands on that organization's body, never another lineage's. + for (const id of ['row_a', 'row_b']) { + const [row] = rows(`SELECT organization_id, version FROM sys_metadata WHERE id = '${id}'`); + const snap = rows( + `SELECT checksum FROM sys_metadata_history WHERE type = 'view' AND name = 'board' AND version = ${Number(row!.version)}`, + ); + expect(snap).toEqual([{ checksum: id === 'row_a' ? 'a3' : 'b2' }]); + } + // A lineage no other one shares keeps its numbers. + expect(rows(`SELECT version FROM sys_metadata_history WHERE id = 'h_c1'`)).toEqual([{ version: 1 }]); + expect(rows(`SELECT version FROM sys_metadata WHERE id = 'row_env'`)).toEqual([{ version: 2 }]); + }); + + it('commit ranges move with their organization\'s history', async () => { + history(); + await apply(catalog(), ledger(), { posture: 'isolated', columnDrops: keepColumn }); + for (const [commit, org] of [['c_a', 'org_a'], ['c_b', 'org_b']] as const) { + const [c] = rows(`SELECT event_seq_start AS s, event_seq_end AS e FROM sys_metadata_commit WHERE id = '${commit}'`); + const [range] = rows(`SELECT MIN(event_seq) AS lo, MAX(event_seq) AS hi FROM sys_metadata_history WHERE organization_id = '${org}'`); + expect([c!.s, c!.e]).toEqual([range!.lo, range!.hi]); + } + expect(rows(`SELECT event_seq_start AS s, event_seq_end AS e FROM sys_metadata_commit WHERE id = 'c_env'`)).toEqual([{ s: 1, e: 3 }]); + }); + + it('a second run moves nothing, and a run killed between the commit and history writes resumes without moving a commit twice', async () => { + history(); + await apply(catalog(), ledger(), { posture: 'isolated', columnDrops: keepColumn }); + const settled = seqs(); + const commits = rows('SELECT * FROM sys_metadata_commit ORDER BY id'); + const again = await apply(catalog(), ledger(), { posture: 'isolated', columnDrops: keepColumn }); + expect(step(again).summary).toMatchObject({ organizationsShifted: 0, lineagesShifted: 0 }); + expect(seqs()).toEqual(settled); + expect(rows('SELECT * FROM sys_metadata_commit ORDER BY id')).toEqual(commits); + + // Killed after org_a's commit range moved, before its history did. + db.close(); + db = new Database(file); + db.exec('DROP TABLE sys_metadata; DROP TABLE sys_metadata_history; DROP TABLE sys_metadata_commit;'); + history(); + await expect(apply(catalog(), ledger(), { + posture: 'isolated', + columnDrops: keepColumn, + executor: executor({ failOnceOn: /^UPDATE "sys_metadata_history" SET "event_seq"/ }), + })).rejects.toThrow(/killed mid-step/); + const l = ledger(); + await apply(catalog(), l, { posture: 'isolated', columnDrops: keepColumn }); + expect(seqs()).toEqual(settled); + expect(rows('SELECT * FROM sys_metadata_commit ORDER BY id')).toEqual(commits); + }); + + /** A view row and its history: one lineage per scope, as planted. */ + const viewWithHistory = (orgRows: string, historyRows: string, versioned = true): void => { + db.exec(` + CREATE TABLE sys_metadata (id TEXT PRIMARY KEY, organization_id TEXT, type TEXT, name TEXT, state TEXT, + package_id TEXT, bundle_member TEXT, version INTEGER, metadata TEXT); + INSERT INTO sys_metadata VALUES ${orgRows}; + CREATE TABLE sys_metadata_history (id TEXT PRIMARY KEY, organization_id TEXT, type TEXT, name TEXT, + version INTEGER, event_seq INTEGER, checksum TEXT); + ${HISTORY_UNIQUE} + ${historyRows ? `INSERT INTO sys_metadata_history VALUES ${historyRows};` : ''} + `); + void versioned; + }; + const withdrawn = JSON.stringify(formView(false, 'Org copy')); + const snapshotOf = (id: string): unknown[] => { + const [row] = rows(`SELECT type, name, version FROM sys_metadata WHERE id = '${id}'`); + return rows(`SELECT checksum FROM sys_metadata_history WHERE type = '${String(row!.type)}' AND name = '${String(row!.name)}' AND version = ${Number(row!.version)}`) + .map((r) => r.checksum); + }; + + it('⭐ the ceremony\'s own environment saves never land on a number an organization holds (the withdrawal carry)', async () => { + viewWithHistory( + `('md_org', 'org_default', 'view', 'crm_lead.intake', 'active', NULL, NULL, 2, '${withdrawn}')`, + `('h_env1', NULL, 'view', 'crm_lead.intake', 1, 1, 'e1'), ('h_env2', NULL, 'view', 'crm_lead.intake', 2, 2, 'e2'), + ('h_org1', 'org_default', 'view', 'crm_lead.intake', 1, 1, 'o1'), ('h_org2', 'org_default', 'view', 'crm_lead.intake', 2, 2, 'o2')`, + ); + const c = storingDoor(catalog()); + + const report = await apply(c, ledger(), { views: servedViews(c), columnDrops: keepColumn }); + + expect(report.steps.find((s) => s.step === 'metadata-promotion')!.summary).toMatchObject({ withdrawalsCarried: 1, promoted: 1 }); + expect(rows(`SELECT COUNT(*) AS n FROM sys_metadata_history WHERE id LIKE 'h_save_%' AND type = 'view'`)).toEqual([{ n: 1 }]); + expect(duplicates()).toEqual([]); + // The promoted row took its history with it, and still names its own snapshot. + expect(rows(`SELECT organization_id FROM sys_metadata_history WHERE id IN ('h_org1', 'h_org2')`)).toEqual([{ organization_id: null }, { organization_id: null }]); + expect(snapshotOf('md_org')).toEqual(['o2']); + expect(rows(`SELECT checksum FROM sys_metadata_history WHERE id IN ('h_org1', 'h_org2') ORDER BY event_seq`).map((r) => r.checksum)).toEqual(['o1', 'o2']); + }); + + it('⭐ an item with only an organization history: no environment save, during the run or after it, lands on its numbers', async () => { + // F2: no environment lineage at all. The package's code view is carried + // onto (another key), the package-less organization row is promoted. + viewWithHistory( + `('md_org', 'org_default', 'view', 'crm_lead.intake', 'active', NULL, NULL, 2, '${withdrawn}')`, + `('h_org1', 'org_default', 'view', 'crm_lead.intake', 1, 1, 'o1'), ('h_org2', 'org_default', 'view', 'crm_lead.intake', 2, 2, 'o2')`, + ); + const c = storingDoor(catalog()); + + await apply(c, ledger(), { views: servedViews(c), columnDrops: keepColumn }); + expect(duplicates()).toEqual([]); + expect(snapshotOf('md_org')).toEqual(['o2']); + + // A runtime save of the promoted item, and of the package's copy, after the run. + await c.save('view', 'crm_lead.intake', formView(false, 'Edited'), { packageId: null }); + await c.save('view', 'crm_lead.intake', formView(false, 'Edited'), { packageId: 'pkg_a' }); + expect(duplicates()).toEqual([]); + }); + + it('⭐ under a wall, a carry onto an item an organization row still holds is numbered apart by the late step', async () => { + viewWithHistory( + `('md_org', 'org_default', 'view', 'crm_lead.intake', 'active', NULL, NULL, 2, '${withdrawn}')`, + `('h_org1', 'org_default', 'view', 'crm_lead.intake', 1, 1, 'o1'), ('h_org2', 'org_default', 'view', 'crm_lead.intake', 2, 2, 'o2')`, + ); + const c = storingDoor(catalog()); + + const report = await apply(c, ledger(), { posture: 'isolated', views: servedViews(c), columnDrops: keepColumn }); + + expect(report.steps.find((s) => s.step === 'metadata-promotion')!.summary).toMatchObject({ withdrawalsCarried: 1, promoted: 0 }); + expect(duplicates()).toEqual([]); + // The organization row stays its organization's, and still names its own snapshot. + expect(rows(`SELECT organization_id FROM sys_metadata WHERE id = 'md_org'`)).toEqual([{ organization_id: 'org_default' }]); + expect(snapshotOf('md_org')).toEqual(['o2']); + }); + + it('⭐ two organizations on one key, no environment history, one chosen and carried onto (single)', async () => { + viewWithHistory( + `('md_a', 'org_a', 'view', 'crm_lead.intake', 'active', NULL, NULL, 2, '${withdrawn}'), + ('md_b', 'org_b', 'view', 'crm_lead.intake', 'active', NULL, NULL, 2, '${JSON.stringify(formView(true, 'B copy'))}')`, + `('h_a1', 'org_a', 'view', 'crm_lead.intake', 1, 1, 'a1'), ('h_a2', 'org_a', 'view', 'crm_lead.intake', 2, 2, 'a2'), + ('h_b1', 'org_b', 'view', 'crm_lead.intake', 1, 1, 'b1'), ('h_b2', 'org_b', 'view', 'crm_lead.intake', 2, 2, 'b2')`, + ); + const c = storingDoor(catalog()); + + const report = await apply(c, ledger(), { views: servedViews(c, []), metadataChoices: ['md_b'], columnDrops: keepColumn }); + + expect(report.steps.find((s) => s.step === 'metadata-promotion')!.summary).toMatchObject({ chosen: 1, organizationRowsDeleted: 1 }); + expect(duplicates()).toEqual([]); + // The chosen row now carries the unchosen row's withdrawal, saved above its own lineage. + expect((JSON.parse(String(rows(`SELECT metadata FROM sys_metadata WHERE id = 'md_b'`)[0]!.metadata)) as any).config.sharing.enabled).toBe(false); + await c.save('view', 'crm_lead.intake', formView(false, 'Edited'), { packageId: null }); + expect(duplicates()).toEqual([]); + }); + + it('⭐ N1: folding one item of an organization never lands on that organization\'s other events, on a rerun too', async () => { + // The review's smallest case: the environment holds event 1; the + // organization holds view A at 1-2 and view B at 3-4, both lone under single. + viewWithHistory( + `('md_a', 'org_default', 'view', 'view_a', 'active', NULL, NULL, 2, '{"name":"view_a"}'), + ('md_b', 'org_default', 'view', 'view_b', 'active', NULL, NULL, 4, '{"name":"view_b"}')`, + `('h_env', NULL, 'object', 'deal', 1, 1, 'e'), + ('h_a1', 'org_default', 'view', 'view_a', 1, 1, 'a1'), ('h_a2', 'org_default', 'view', 'view_a', 2, 2, 'a2'), + ('h_b1', 'org_default', 'view', 'view_b', 3, 3, 'b1'), ('h_b2', 'org_default', 'view', 'view_b', 4, 4, 'b2')`, + ); + const c = storingDoor(catalog()); + // A crash right after the first fold's event move, then the rerun. + await expect(apply(c, ledger(), { views: servedViews(c, []), columnDrops: keepColumn, executor: executor({ failOnceOn: /^UPDATE "sys_metadata_history" SET "organization_id" = NULL/ }) })) + .rejects.toThrow(/killed mid-step/); + const report = await apply(c, ledger(), { views: servedViews(c, []), columnDrops: keepColumn }); + expect(report.steps.find((s) => s.step === 'metadata-promotion')!.summary).toMatchObject({ promoted: 2 }); + expect(duplicates()).toEqual([]); + expect(snapshotOf('md_a')).toEqual(['a2']); + expect(snapshotOf('md_b')).toEqual(['b2']); + expect(rows(`SELECT checksum FROM sys_metadata_history WHERE id LIKE 'h_a%' OR id LIKE 'h_b%' ORDER BY event_seq`).map((r) => r.checksum)) + .toEqual(['a1', 'a2', 'b1', 'b2']); + }); + + it('⭐ column-drops never runs unless this apply\'s history-renumber verified uniqueness environment-wide', async () => { + // Two environment events share a number: the 17.x index never covered the + // environment scope (NULL is distinct), and no layout can move them. + viewWithHistory( + `('md_env', NULL, 'object', 'deal', 'active', NULL, NULL, 1, '{"name":"deal"}')`, + `('h_e1', NULL, 'object', 'deal', 1, 7, 'e1'), ('h_e2', NULL, 'object', 'deal', 2, 7, 'e2'), + ('h_v1', NULL, 'object', 'account', 3, 8, 'v1'), ('h_v2', NULL, 'object', 'account', 3, 9, 'v2')`, + ); + const drops: string[][] = []; + const report = await apply(catalog(), ledger(), { + columnDrops: { + drop: async (columns) => { + drops.push(columns.map(({ table, column }) => `${table}.${column}`)); + return columns.map(({ table, column }) => ({ table, column, outcome: 'dropped' as const })); + }, + }, + }); + const renumber = report.steps.find((s) => s.step === 'history-renumber')!; + expect(renumber.status).toBe('blocked'); + // The refusal names the table, each item holding the number, and the remedy. + expect(renumber.blockers).toContainEqual( + 'sys_metadata_history: event_seq is not unique environment-wide — event_seq 7: object deal (row h_e1), object deal ' + + '(row h_e2). The ceremony never renumbers history a database numbered twice: the organization_id column is not ' + + 'dropped until every event has its own number. Remedy: give each of these rows a number no other row holds (or ' + + 'remove the duplicate), then re-run --apply.', + ); + expect(renumber.blockers).toContainEqual(expect.stringMatching( + /^sys_metadata_history: a version of an item is held more than once — object account version 3 \(2 rows\)\. .*Remedy: give each of these rows a version no other row of the item holds/, + )); + expect(report.steps.find((s) => s.step === 'column-drops')!.status).toBe('not-reached'); + expect(drops).toEqual([]); + }); + + it('an environment save made between two runs is renumbered by the next one, the checkpoint notwithstanding', async () => { + history(); + const l = ledger(); + await apply(catalog(), l, { posture: 'isolated', columnDrops: keepColumn }); + // Environment saves since (numbered in the environment's own scope) have + // reached the numbers the first run gave org_a: the next one lands there. + const [seq] = rows(`SELECT MIN(event_seq) AS n FROM sys_metadata_history WHERE organization_id = 'org_a'`); + const [ver] = rows(`SELECT MIN(version) AS n FROM sys_metadata_history WHERE organization_id = 'org_a' AND type = 'view' AND name = 'board'`); + db.prepare(`INSERT INTO sys_metadata_history VALUES ('h_late', NULL, 'view', 'board', ?, ?, 'late')`).run(ver!.n, seq!.n); + expect(duplicates()).not.toEqual([]); + + const again = await apply(catalog(), l, { posture: 'isolated', columnDrops: keepColumn }); + + expect(step(again)).toMatchObject({ status: 'done', summary: { organizationsShifted: 3, lineagesShifted: 2 } }); + expect(duplicates()).toEqual([]); + expect(snapshotOf('row_a')).toEqual(['a3']); + expect(snapshotOf('row_b')).toEqual(['b2']); + }); +}); + describe('ADR-0131 D10 --apply — the grant\'s retired id column (C7b, seat answers 6162538253 Q5 A)', () => { const plan = () => buildOrganizationOwnershipPlan({ reader: executor(), posture: 'single', database: file }); const grantsPlan = async () => (await plan()).tables.find((t) => t.object === 'sys_user_permission_set')!.grants!; diff --git a/packages/cli/src/utils/organization-ownership-apply.ts b/packages/cli/src/utils/organization-ownership-apply.ts index 591c17fb04f..8fa3392d285 100644 --- a/packages/cli/src/utils/organization-ownership-apply.ts +++ b/packages/cli/src/utils/organization-ownership-apply.ts @@ -5,6 +5,7 @@ import { permissionSetDefinitionFromRow, positionDefinitionFromRow, } from '@objectstack/core'; +import { anonymousFormIntakeCandidates, anonymousFormIntakeWithdrawnIn, hashSpec } from '@objectstack/metadata-core'; import { inventoryEntryFor, type InventoryEntry, @@ -15,6 +16,9 @@ import { columnListSql, definitionKey, groupTemplateRows, + groupMetadataKeyRows, + readMetadataKeyRows, + METADATA_OBJECT, templatePromotionSql, SEEDED_BY, TEMPLATE_OBJECT, @@ -23,6 +27,8 @@ import { predicateSql, Sql, type AttributionSink, + type MetadataConflict, + type MetadataKeyRow, type TemplateConflict, type OrganizationOwnershipPlan, type PlanReader, @@ -54,6 +60,27 @@ import { * (`convertPositionBindingRows`). Only under `single` (ADR-0131 D3: a * Setup catalog write is an environment metadata write under `single` and * refused under a wall); under a wall the rows are reported. + * 2a. `metadata-promotion` — rulings A on #22011 and #22008 (records + * 6020163868, 6020151485): first every public-form withdrawal an + * organization saved is carried onto the environment, fail-closed (any + * organization's withdrawal wins), by writing it onto each active + * environment `view` of that name that still serves the form open, + * through the metadata door; the same predicate the anonymous doors + * judge by (`anonymousFormIntakeWithdrawnIn`) decides. Then, under + * `single`, each organization-scoped `sys_metadata` row of the five + * presentational types whose overlay key no other row holds becomes an + * environment row in place, its history with it (its `organization_id` + * cleared). A key held by more than one row is a conflict: listed, kept, + * and blocking under `single` until the operator chooses one row with + * `--metadata-choice ROW_ID`. The key then ends with exactly ONE + * environment row defining the chosen body (the key's environment row + * takes it through the door, in place; with none, the chosen row is + * promoted in place), with every withdrawal another row holds kept; it is + * read back, and only then are the key's organization rows deleted, in + * one statement, by id (ruling B on #23413 Q1). Under a wall nothing is + * promoted unless the operator names the row, and nothing is deleted + * (ruling A on #23413 Q6, ruling 6163114098): the rest are reported and + * hold the column. * 2b. `template-promotion` — ruling C on ADR-0131 §6 Q1: every * `sys_email_template` row an organization customized or authored becomes * an environment `email_template` item through the metadata door (core's @@ -78,6 +105,14 @@ import { * named categories are deleted. Templates: the promotion runs again first, * and beyond the seeded mirrors only the rows whose environment copy that * pass verified are deleted, by id. + * 4h. `history-renumber` — seat answer Q2 A: `sys_metadata_history`'s + * `event_seq` and every `(type, name, version)` unique environment-wide + * before the column goes. Every lineage still tagged to an organization + * is laid out above the environment's (each organization's event range by + * ONE offset, its commit ranges with it; each item's version lineage, the + * organization's rows with it); under `single` that history then becomes + * the environment's. It runs on every apply, even when checkpointed, so an + * environment save made since is covered. Numbers only, no delete. * 5. `column-drops` (fate 1) — `organization_id` leaves the D7 objects * through the schema-drift ceremony ({@link ColumnDropPort}), never a hand * `ALTER`; so does the retired `sys_user_permission_set.permission_set_id` @@ -108,6 +143,10 @@ import { * recorded, the posture it runs under is not the one the plan was computed * under (the posture decides what attribution and the authored conversion * write), or the file names another database ({@link ReviewedPlanRefusal}). + * A choice deletes the other rows of its conflict, so the reviewed plan also + * binds what a `--metadata-choice` or `--template-choice` may delete: the + * chosen conflict must hold exactly the rows the file listed for it + * ({@link ReviewedConflict}), or the apply refuses (`plan-conflict-mismatch`). * The remedy is always the same: plan again, review it, hand the new file in. * A checkpoint recorded under another digest is therefore one the reviewed * plan supersedes: the run starts over (each step is idempotent), and the @@ -126,10 +165,12 @@ export const ORGANIZATION_OWNERSHIP_MIGRATION_ID = 'adr-0131-organization-owners export const CEREMONY_STEPS = [ 'attribution', 'authored-conversion', + 'metadata-promotion', 'template-promotion', 'settings-move', 'rewrite', 'mirror-deletion', + 'history-renumber', 'column-drops', ] as const; export type CeremonyStep = (typeof CEREMONY_STEPS)[number]; @@ -146,7 +187,26 @@ export interface CatalogPort { /** The metadata door: an environment metadata save. Throws when the door refuses. */ export interface MetadataDoorPort { - save(type: 'position' | 'permission' | 'email_template', name: string, body: Record): Promise; + save( + type: 'position' | 'permission' | 'email_template' | 'view' | 'dashboard' | 'report' | 'translation', + name: string, + body: Record, + target?: { packageId: string | null }, + ): Promise; +} + +/** The env-wide `view` list the anonymous form doors read (every package's item of a name). */ +export interface ServedViewsPort { + list(): Promise>>>; +} + +/** + * The metadata door's own verdict on a write to an item (ADR-0010 §3.3): the + * item read's `editable`, which reports the save door's lock gate. A write the + * ceremony makes in place, not through the door, asks it first. + */ +export interface MetadataLockPort { + verdict(type: string, name: string, packageId: string | null): Promise<{ editable: boolean; lock?: string; lockReason?: string }>; } /** The `email_template` item the registry serves for one `(name, locale)` bundle member. */ @@ -191,6 +251,12 @@ export interface CeremonyCheckpoint { ceremony: 'ADR-0131 D10'; inventoryDigest: string; completed: Partial }>>; + /** + * The `--metadata-choice` rows whose conflict this ceremony resolved, recorded + * before the conflict's organization rows are deleted: a rerun after a crash + * between the two accepts the same choice and finishes. + */ + metadataChoicesApplied?: string[]; } /** Where the checkpoint lives (the command: the `sys_migration` row). */ @@ -211,6 +277,48 @@ export interface ReviewedPlan { inventoryDigest: string; /** The tenancy posture the plan was computed under. */ posture: string; + /** + * The conflict groups the plan listed, by key: the only rows a + * `--metadata-choice` or `--template-choice` may delete. Absent means the + * file listed none. + */ + conflicts?: ReviewedConflict[]; +} + +/** One conflict group a plan lists: the rows a choice there resolves. */ +export interface ReviewedConflict { + kind: 'metadata' | 'template'; + /** The group's key as the operator reads it: `view deal_list (package p)`, `crm.note (en-US)`. */ + key: string; + /** Its rows, by id, sorted. */ + candidates: string[]; +} + +/** + * The conflict groups of a plan, from the plan the apply computes or from the + * kept file (both carry `tables[].metadataConflicts` and + * `tables[].templateConflicts`), normalized so the two compare. + */ +export function reviewedConflictsOf(plan: unknown): ReviewedConflict[] { + const tables = (plan as { tables?: unknown } | null)?.tables; + if (!Array.isArray(tables)) return []; + const ids = (candidates: unknown): string[] => + Array.isArray(candidates) ? candidates.map((c) => String((c as { id?: unknown })?.id)).sort() : []; + const out: ReviewedConflict[] = []; + for (const table of tables as Array>) { + for (const c of Array.isArray(table?.metadataConflicts) ? (table.metadataConflicts as Array>) : []) { + out.push({ + kind: 'metadata', + key: `${String(c.type)} ${String(c.name)}${c.packageId ? ` (package ${String(c.packageId)})` : ''}` + + `${c.bundleMember ? ` [${String(c.bundleMember)}]` : ''}`, + candidates: ids(c.candidates), + }); + } + for (const c of Array.isArray(table?.templateConflicts) ? (table.templateConflicts as Array>) : []) { + out.push({ kind: 'template', key: `${String(c.name)} (${String(c.locale)})`, candidates: ids(c.candidates) }); + } + } + return out; } /** @@ -220,7 +328,12 @@ export interface ReviewedPlan { */ export class ReviewedPlanRefusal extends Error { constructor( - readonly reason: 'plan-file-unusable' | 'plan-digest-mismatch' | 'plan-posture-mismatch' | 'plan-database-mismatch', + readonly reason: + | 'plan-file-unusable' + | 'plan-digest-mismatch' + | 'plan-posture-mismatch' + | 'plan-database-mismatch' + | 'plan-conflict-mismatch', message: string, readonly detail: Readonly> = {}, ) { @@ -266,7 +379,14 @@ export function reviewedPlanFromFile(file: string, text: string): ReviewedPlan { { file }, ); } - return { file, generatedAt: p.generatedAt, database: p.database, inventoryDigest: p.inventoryDigest, posture: p.posture }; + return { + file, + generatedAt: p.generatedAt, + database: p.database, + inventoryDigest: p.inventoryDigest, + posture: p.posture, + conflicts: reviewedConflictsOf(p), + }; } export interface StepReport { @@ -341,6 +461,10 @@ export interface ApplyOptions { catalog: CatalogPort; door: MetadataDoorPort; templates: TemplatePort; + /** The served views, for carrying form withdrawals. */ + views: ServedViewsPort; + /** The door's lock verdict, for the one write the ceremony makes in place. */ + lock: MetadataLockPort; columnDrops: ColumnDropPort; ledger: CeremonyLedgerPort; /** The kept plan file the operator reviewed; the run refuses unless it matches (see the module doc). */ @@ -352,6 +476,13 @@ export interface ApplyOptions { * anything is written ({@link TemplateChoiceRefusal}). */ templateChoices?: readonly string[]; + /** + * The operator's choice per `sys_metadata` conflict (the row the environment + * keeps; the others are deleted once it is promoted and read back), and under + * a wall the organization rows promoted knowingly. Validated before anything + * is written ({@link MetadataChoiceRefusal}). + */ + metadataChoices?: readonly string[]; now?: () => Date; /** One line per milestone, for the command's human face. */ onProgress?: (message: string) => void; @@ -406,6 +537,18 @@ interface Run { * promotion pass mirror deletion re-runs, so a resumed run verifies afresh. */ verifiedTemplateIds: Set; + /** The checkpoint this run writes into (set once the ledger is read). */ + checkpoint?: CeremonyCheckpoint; + /** This apply's `history-renumber` verified `event_seq` and version uniqueness environment-wide. */ + historyVerified?: boolean; +} + +/** A `--metadata-choice` that names no row awaiting a choice, or two of one conflict. Nothing is written. */ +export class MetadataChoiceRefusal extends Error { + constructor(message: string) { + super(message); + this.name = 'MetadataChoiceRefusal'; + } } /** A `--template-choice` that names no candidate of a listed conflict, or two of one. Nothing is written. */ @@ -739,6 +882,817 @@ async function runAuthoredConversion(run: Run): Promise { return { step: 'authored-conversion', status: blockers.length ? 'blocked' : 'done', summary, blockers, notes }; } +// ── History numbering (seat answer Q2 A) ──────────────────────────────────── +// +// The metadata door numbers a save from history in its OWN scope: `event_seq` +// and an item's `version` are each `MAX + 1` over the rows whose +// `organization_id` is the writer's. While the column stands, the environment +// and each organization therefore count independently, and once it goes, +// `event_seq` and every `(type, name, version)` must be unique. Two rules make +// them so: +// +// - **A promoted row takes its history with it** ({@link foldLineage}). Before +// a row leaves its organization, that organization's lineage of the item is +// moved above the environment's numbers and becomes environment history. The +// door then counts it, so every later environment save of the item, the +// ceremony's or the runtime's, numbers above it, and the row's own version +// still names its own snapshot. +// - **Every other organization lineage is laid out above the environment's +// just before the column goes** ({@link runHistoryRenumber}, on every apply). +// Those lineages belong to rows that stay organization rows (reported under +// a wall) or are gone (unchosen rows), so no environment row's version points +// into them; the step moves each organization row's version with its own. + +const HISTORY = 'sys_metadata_history'; +const COMMITS = 'sys_metadata_commit'; +const OVERLAYS = 'sys_metadata'; + +const num = (value: unknown): number => { + const n = Number(value); + return Number.isFinite(n) ? n : 0; +}; + +/** + * One range of numbers, moved above `base` without a transaction and safely + * re-runnable. The offset puts the new range wholly ABOVE both `base` and the + * old range, so a row's own number says whether it moved: the dependents + * (commit ranges, overlay rows' versions) move first, each only where it still + * lies inside the old range, and the history rows last, in ONE statement. A + * re-run finds the history above `base` and moves nothing. Returns the new + * top of the range. + */ +async function shiftRange( + range: { min: number; max: number }, + base: number, + dependents: (offset: number, oldMax: number) => Promise, + history: (offset: number) => Promise, +): Promise<{ top: number; moved: boolean }> { + if (range.min > base) return { top: Math.max(base, range.max), moved: false }; + const offset = Math.max(base, range.max) - range.min + 1; + await dependents(offset, range.max); + await history(offset); + return { top: range.max + offset, moved: true }; +} + +/** The history columns the numbering reads, or `null` when the table carries no organization column. */ +async function historyShape(run: Run): Promise<{ version: boolean; seq: boolean } | null> { + const hcols = await columnsOf(run, HISTORY); + if (!hcols?.has(ORG)) return null; + return { version: hcols.has('version'), seq: hcols.has('event_seq') }; +} + +/** + * Make one organization's lineage of `(type, name)` environment history, so the + * row about to be promoted keeps it: its versions and its events each moved, by + * one offset that keeps their order, above BOTH the environment's numbers and + * every other number the same organization holds, then the rows re-tagged. The + * organization's whole history is the base because 17.x numbered `event_seq` + * per organization across all its items and holds it under a unique + * `(organization_id, event_seq)` index, checked row by row: an offset measured + * on the item alone lands on the organization's own other events. Versions take + * the same base, as ruled. The base excludes the lineage being moved, so once + * moved it sits above the base, and a rerun after a crash anywhere in this moves + * nothing twice. Commit ranges are not moved (no reader resolves history + * through them). + */ +async function foldLineage(run: Run, organizationId: string, type: string, name: string): Promise { + const shape = await historyShape(run); + if (!shape) return; + const sql = run.sql; + const h = sql.q(HISTORY, HISTORY); + const o = sql.q(ORG, HISTORY); + const ty = sql.q('type', HISTORY); + const nm = sql.q('name', HISTORY); + const mine = `${o} = ? AND ${ty} = ? AND ${nm} = ?`; + const args = [organizationId, type, name]; + if (shape.version) { + const v = sql.q('version', HISTORY); + const [own] = await sql.read(HISTORY, `SELECT MIN(${v}) AS lo, MAX(${v}) AS hi, COUNT(*) AS n FROM ${h} WHERE ${mine}`, args); + if (num(own?.n) > 0) { + const [env] = await sql.read(HISTORY, `SELECT MAX(${v}) AS m FROM ${h} WHERE ${o} IS NULL AND ${ty} = ? AND ${nm} = ?`, [type, name]); + const [others] = await sql.read( + HISTORY, + `SELECT MAX(${v}) AS m FROM ${h} WHERE ${o} = ? AND NOT (${ty} = ? AND ${nm} = ?)`, + args, + ); + const mcols = await columnsOf(run, OVERLAYS); + await shiftRange( + { min: num(own!.lo), max: num(own!.hi) }, + Math.max(num(env?.m), num(others?.m)), + async (offset, oldMax) => { + if (!mcols?.has(ORG) || !mcols.has('version')) return; + const m = sql.q(OVERLAYS, OVERLAYS); + const mv = sql.q('version', OVERLAYS); + await execute( + run, + OVERLAYS, + `UPDATE ${m} SET ${mv} = ${mv} + ? WHERE ${sql.q(ORG, OVERLAYS)} = ? AND ${sql.q('type', OVERLAYS)} = ? ` + + `AND ${sql.q('name', OVERLAYS)} = ? AND ${mv} <= ?`, + [offset, ...args, oldMax], + ); + }, + async (offset) => { + await execute(run, HISTORY, `UPDATE ${h} SET ${v} = ${v} + ? WHERE ${mine}`, [offset, ...args]); + }, + ); + } + } + if (shape.seq) { + const seq = sql.q('event_seq', HISTORY); + const [own] = await sql.read(HISTORY, `SELECT MIN(${seq}) AS lo, MAX(${seq}) AS hi, COUNT(*) AS n FROM ${h} WHERE ${mine}`, args); + const [env] = await sql.read(HISTORY, `SELECT MAX(${seq}) AS m FROM ${h} WHERE ${o} IS NULL`); + const [others] = await sql.read( + HISTORY, + `SELECT MAX(${seq}) AS m FROM ${h} WHERE ${o} = ? AND NOT (${ty} = ? AND ${nm} = ?)`, + args, + ); + const base = Math.max(num(env?.m), num(others?.m)); + if (num(own?.n) > 0 && num(own!.lo) <= base) { + // Wholly above the base and the old range: no row may land on the + // environment's numbers, the organization's other events, or a sibling's. + const offset = Math.max(base, num(own!.hi)) - num(own!.lo) + 1; + await execute(run, HISTORY, `UPDATE ${h} SET ${seq} = ${seq} + ? WHERE ${mine}`, [offset, ...args]); + } + } + await execute(run, HISTORY, `UPDATE ${h} SET ${o} = NULL WHERE ${mine}`, args); +} + +/** + * `event_seq` and every `(type, name, version)` unique environment-wide over + * the history still tagged to an organization: each organization's event range + * moved above the environment's (and the organizations' before it) by ONE + * offset, its commit ranges with it; each organization lineage of an item laid + * out above the environment's lineage of it (and the ones before it), the + * organization's overlay rows of the item with it. Order within an + * organization, and the snapshot each row's version names, are unchanged. + */ +async function runHistoryRenumber(run: Run): Promise { + run.historyVerified = false; + const report = await renumberHistory(run); + const blockers = await historyDuplicates(run); + if (blockers.length) return { ...report, status: 'blocked', blockers }; + run.historyVerified = true; + return report; +} + +/** + * `event_seq` and `(type, name, version)` duplicated anywhere in history, as + * blockers: `column-drops` runs only once this reads none (the seat's ruling on + * uniqueness under a wall). The layout above leaves none between scopes; a + * duplicate within one scope is a database's own, and is named, never guessed. + */ +async function historyDuplicates(run: Run): Promise { + const hcols = await columnsOf(run, HISTORY); + if (!hcols) return []; + const sql = run.sql; + const h = sql.q(HISTORY, HISTORY); + const out: string[] = []; + if (hcols.has('event_seq')) { + const seq = sql.q('event_seq', HISTORY); + const dup = await sql.read(HISTORY, `SELECT ${seq} AS s FROM ${h} GROUP BY ${seq} HAVING COUNT(*) > 1 ORDER BY ${seq}`); + if (dup.length) { + const shown = dup.slice(0, 10).map((d) => d.s); + const [ty, nm] = ['type', 'name'].map((c) => sql.q(c, HISTORY)); + const holders = await sql.read( + HISTORY, + `SELECT ${seq} AS s, ${sql.q('id', HISTORY)} AS id, ${ty} AS t, ${nm} AS n FROM ${h} ` + + `WHERE ${seq} IN (${shown.map(() => '?').join(', ')}) ORDER BY ${seq}, ${sql.q('id', HISTORY)}`, + shown, + ); + const byNumber = shown.map( + (n) => `event_seq ${String(n)}: ${holders.filter((r) => String(r.s) === String(n)).map((r) => `${String(r.t)} ${String(r.n)} (row ${String(r.id)})`).join(', ')}`, + ); + out.push( + `${HISTORY}: event_seq is not unique environment-wide — ${byNumber.join('; ')}` + + `${dup.length > 10 ? `; and ${dup.length - 10} more number(s)` : ''}. The ceremony never renumbers history ` + + 'a database numbered twice: the organization_id column is not dropped until every event has its own number. ' + + 'Remedy: give each of these rows a number no other row holds (or remove the duplicate), then re-run --apply.', + ); + } + } + if (hcols.has('version')) { + const [ty, nm, v] = ['type', 'name', 'version'].map((c) => sql.q(c, HISTORY)); + const dup = await sql.read( + HISTORY, + `SELECT ${ty} AS t, ${nm} AS n, ${v} AS v, COUNT(*) AS c FROM ${h} GROUP BY ${ty}, ${nm}, ${v} HAVING COUNT(*) > 1 ORDER BY ${ty}, ${nm}, ${v}`, + ); + if (dup.length) { + out.push( + `${HISTORY}: a version of an item is held more than once — ` + + `${dup.slice(0, 10).map((d) => `${String(d.t)} ${String(d.n)} version ${String(d.v)} (${String(d.c)} rows)`).join('; ')}` + + `${dup.length > 10 ? `; and ${dup.length - 10} more` : ''}. The ceremony never renumbers history a database ` + + 'numbered twice: the organization_id column is not dropped until every version names one snapshot. Remedy: give ' + + 'each of these rows a version no other row of the item holds (or remove the duplicate), then re-run --apply.', + ); + } + } + return out; +} + +async function renumberHistory(run: Run): Promise { + const summary: Record = { organizationsShifted: 0, lineagesShifted: 0 }; + const shape = await historyShape(run); + if (!shape) return { step: 'history-renumber', status: 'done', summary, blockers: [], notes: [] }; + const sql = run.sql; + const h = sql.q(HISTORY, HISTORY); + const o = sql.q(ORG, HISTORY); + + if (shape.version) { + const v = sql.q('version', HISTORY); + const ty = sql.q('type', HISTORY); + const nm = sql.q('name', HISTORY); + const mcols = await columnsOf(run, OVERLAYS); + const overlaysCarry = !!mcols?.has(ORG) && mcols.has('version'); + const lineages = await sql.read( + HISTORY, + `SELECT ${ty} AS t, ${nm} AS n, ${o} AS org, MIN(${v}) AS lo, MAX(${v}) AS hi FROM ${h} ` + + `GROUP BY ${ty}, ${nm}, ${o} ORDER BY ${ty}, ${nm}, ${o}`, + ); + const isEnv = (l: Record): boolean => l.org === null || l.org === undefined || l.org === ''; + const byItem = new Map>>(); + for (const l of lineages) { + const key = `${String(l.t)}\u0000${String(l.n)}`; + byItem.set(key, [...(byItem.get(key) ?? []), l]); + } + for (const group of byItem.values()) { + const env = group.find(isEnv); + const orgs = group.filter((l) => !isEnv(l)); + if (orgs.length === 0 || (!env && orgs.length === 1)) continue; + // Above the environment's lineage; with none, the first organization keeps its numbers. + let vbase = env ? num(env.hi) : num(orgs[0]!.hi); + for (const l of env ? orgs : orgs.slice(1)) { + const org = String(l.org); + const shifted = await shiftRange( + { min: num(l.lo), max: num(l.hi) }, + vbase, + async (offset, oldMax) => { + if (!overlaysCarry) return; + const m = sql.q(OVERLAYS, OVERLAYS); + const mv = sql.q('version', OVERLAYS); + await execute( + run, + OVERLAYS, + `UPDATE ${m} SET ${mv} = ${mv} + ? WHERE ${sql.q(ORG, OVERLAYS)} = ? AND ${sql.q('type', OVERLAYS)} = ? ` + + `AND ${sql.q('name', OVERLAYS)} = ? AND ${mv} <= ?`, + [offset, org, l.t, l.n, oldMax], + ); + }, + async (offset) => { + await execute(run, HISTORY, `UPDATE ${h} SET ${v} = ${v} + ? WHERE ${o} = ? AND ${ty} = ? AND ${nm} = ?`, [ + offset, + org, + l.t, + l.n, + ]); + }, + ); + vbase = shifted.top; + if (shifted.moved) summary.lineagesShifted++; + } + } + } + + if (shape.seq) { + const seq = sql.q('event_seq', HISTORY); + const ccols = await columnsOf(run, COMMITS); + const commitsCarry = !!ccols?.has(ORG) && ccols.has('event_seq_start') && ccols.has('event_seq_end'); + const envTop = num((await sql.read(HISTORY, `SELECT MAX(${seq}) AS m FROM ${h} WHERE ${o} IS NULL`))[0]?.m); + const ranges = await sql.read( + HISTORY, + `SELECT ${o} AS org, MIN(${seq}) AS lo, MAX(${seq}) AS hi FROM ${h} WHERE ${o} IS NOT NULL GROUP BY ${o} ORDER BY ${o}`, + ); + let base = envTop; + for (const r of ranges) { + const org = String(r.org); + const shifted = await shiftRange( + { min: num(r.lo), max: num(r.hi) }, + base, + async (offset, oldMax) => { + if (!commitsCarry) return; + const c = sql.q(COMMITS, COMMITS); + const s0 = sql.q('event_seq_start', COMMITS); + const s1 = sql.q('event_seq_end', COMMITS); + await execute( + run, + COMMITS, + `UPDATE ${c} SET ${s0} = ${s0} + ?, ${s1} = ${s1} + ? WHERE ${sql.q(ORG, COMMITS)} = ? AND ${s0} <= ?`, + [offset, offset, org, oldMax], + ); + }, + async (offset) => { + await execute(run, HISTORY, `UPDATE ${h} SET ${seq} = ${seq} + ? WHERE ${o} = ?`, [offset, org]); + }, + ); + base = shifted.top; + if (shifted.moved) summary.organizationsShifted++; + } + } + // Under `single` the environment is the one tenant: what history is left + // tagged to an organization (rows reported or deleted) becomes the + // environment's too, so the door counts it and no later save collides. + // Under a wall it stays its organization's, renumbered here on every apply. + if (run.opts.posture === 'single') { + const [left] = await sql.read(HISTORY, `SELECT COUNT(*) AS n FROM ${h} WHERE ${o} IS NOT NULL`); + if (num(left?.n) > 0) { + await execute(run, HISTORY, `UPDATE ${h} SET ${o} = NULL WHERE ${o} IS NOT NULL`, []); + summary.folded = num(left?.n); + } + const ccols = await columnsOf(run, COMMITS); + if (ccols?.has(ORG)) { + const c = sql.q(COMMITS, COMMITS); + await execute(run, COMMITS, `UPDATE ${c} SET ${sql.q(ORG, COMMITS)} = NULL WHERE ${sql.q(ORG, COMMITS)} IS NOT NULL`, []); + } + } + if (summary.organizationsShifted || summary.lineagesShifted) { + run.opts.onProgress?.( + `history renumbered: ${summary.organizationsShifted} organization event range(s), ${summary.lineagesShifted} version lineage(s)`, + ); + } + return { step: 'history-renumber', status: 'done', summary, blockers: [], notes: [] }; +} + +// ── Step 2a: organization customizations and form withdrawals (rulings A) ── + +const parseBody = (row: Record): Record | null => { + const raw = row.metadata; + try { + const body = typeof raw === 'string' ? JSON.parse(raw) : raw; + if (!body || typeof body !== 'object' || Array.isArray(body)) return null; + // A stored body names its item; the doors judge withdrawals by that name. + return { ...(body as Record), name: (body as Record).name ?? row.name }; + } catch { + return null; + } +}; + +/** The served item's own package (`_packageId`), or `null` for a package-less item. */ +const servedPackageOf = (item: Readonly>): string | null => { + const pkg = item._packageId; + return typeof pkg === 'string' && pkg !== '' && pkg !== METADATA_OBJECT ? pkg : null; +}; + +/** A served item as a body the door accepts: its runtime annotations (`_`-prefixed keys) dropped. */ +const authoredBodyOf = (item: Readonly>): Record => + JSON.parse(JSON.stringify(Object.fromEntries(Object.entries(item).filter(([k]) => !k.startsWith('_'))))) as Record; + +/** + * Carry every organization-layer withdrawal onto the environment, fail-closed: + * each view the environment serves (stored or from code, in every package) + * whose name a withdrawing organization row shares, and which still serves the + * withdrawn form open, takes the withdrawal (`sharing.enabled: false` on that + * form), saved through the metadata door into its own package. The judgement + * is the anonymous doors' own (`anonymousFormIntakeWithdrawnIn`), with the + * organization rows as the layer: a 17.x organization row replaced the served + * body whole, so its stored body is exactly what the doors judged. + * + * Under `single` a view whose own key an organization row holds is skipped: + * that row is promoted in place (its withdrawal travels with it) or is a + * conflict the operator resolves, and a door save there would make a conflict + * of a promotion. + */ +async function carryFormWithdrawals(run: Run, rows: readonly MetadataKeyRow[], summary: Record, blockers: string[]): Promise { + const t = METADATA_OBJECT; + const sql = run.sql; + const orgViews = rows.filter((r) => r.type === 'view' && r.organizationId !== null && (r.state ?? 'active') === 'active'); + if (orgViews.length === 0) return; + const layerOf = new Map[]>(); + const ids = orgViews.map((r) => r.id); + for (let i = 0; i < ids.length; i += IN_CHUNK) { + const chunk = ids.slice(i, i + IN_CHUNK); + const found = await sql.read( + t, + `SELECT ${['id', 'name', 'metadata'].map((c) => `${sql.q(c, t)} AS ${sql.q(c, t)}`).join(', ')} ` + + `FROM ${sql.q(t, t)} WHERE ${sql.q('id', t)} IN (${chunk.map(() => '?').join(', ')})`, + chunk, + ); + for (const row of found) { + const body = parseBody(row); + if (body) layerOf.set(String(row.name), [...(layerOf.get(String(row.name)) ?? []), body]); + } + } + if (layerOf.size === 0) return; + const heldByOrganization = new Set(orgViews.map((r) => `${r.name}\u0000${r.packageId ?? ''}`)); + for (const served of await run.opts.views.list()) { + const name = typeof served.name === 'string' ? served.name : undefined; + const layer = name === undefined ? undefined : layerOf.get(name); + if (!name || !layer) continue; + const packageId = servedPackageOf(served); + if (run.opts.posture === 'single' && heldByOrganization.has(`${name}\u0000${packageId ?? ''}`)) continue; + const body = authoredBodyOf(served); + const closing = anonymousFormIntakeCandidates(body).filter((c) => anonymousFormIntakeWithdrawnIn(layer, body, c)); + if (closing.length === 0) continue; + for (const candidate of closing) { + candidate.form.sharing = { ...(candidate.form.sharing as Record), enabled: false }; + } + try { + await run.opts.door.save('view', name, body, { packageId }); + summary.withdrawalsCarried += closing.length; + run.opts.onProgress?.(`carried ${closing.length} public-form withdrawal(s) onto the environment view ${name}`); + } catch (error) { + blockers.push( + `An organization withdrew the public form(s) ${closing.map((c) => c.slug).join(', ')} of the view ${name}` + + `${packageId ? ` (package ${packageId})` : ''}, and the metadata door refused to carry that withdrawal onto ` + + `the environment: ${error instanceof Error ? error.message : String(error)}. Withdraw the form there ` + + '(sharing.enabled false), then re-run. An upgrade never reopens a withdrawn form (ruling record 6020151485).', + ); + } + } +} + +/** One conflict the operator chose a row for. */ +interface MetadataResolution { + conflict: MetadataConflict; + chosen: MetadataConflict['candidates'][number]; +} + +/** The `metadata` bodies of the given `sys_metadata` rows, by id. */ +async function metadataBodies(run: Run, ids: readonly string[]): Promise | null>> { + const t = METADATA_OBJECT; + const sql = run.sql; + const out = new Map | null>(); + for (let i = 0; i < ids.length; i += IN_CHUNK) { + const chunk = ids.slice(i, i + IN_CHUNK); + const found = await sql.read( + t, + `SELECT ${['id', 'name', 'metadata'].map((c) => `${sql.q(c, t)} AS ${sql.q(c, t)}`).join(', ')} ` + + `FROM ${sql.q(t, t)} WHERE ${sql.q('id', t)} IN (${chunk.map(() => '?').join(', ')})`, + chunk, + ); + for (const row of found) out.set(String(row.id), parseBody(row)); + } + return out; +} + +/** + * Promote one organization's rows in place: each row's lineage becomes + * environment history first ({@link foldLineage}), then its `organization_id` + * is cleared. Conditioned on the owner, so a rerun promotes nothing twice. + */ +async function promoteRows(run: Run, organizationId: string, rows: ReadonlyArray<{ id: string; type: string; name: string }>): Promise { + const seen = new Set(); + for (const row of rows) { + const item = `${row.type}\u0000${row.name}`; + if (seen.has(item)) continue; + seen.add(item); + await foldLineage(run, organizationId, row.type, row.name); + } + const t = run.sql.q(METADATA_OBJECT, METADATA_OBJECT); + const org = run.sql.q(ORG, METADATA_OBJECT); + const id = run.sql.q('id', METADATA_OBJECT); + const ids = rows.map((r) => r.id); + for (let i = 0; i < ids.length; i += IN_CHUNK) { + const chunk = ids.slice(i, i + IN_CHUNK); + await execute( + run, + METADATA_OBJECT, + `UPDATE ${t} SET ${org} = NULL WHERE ${org} = ? AND ${id} IN (${chunk.map(() => '?').join(', ')})`, + [organizationId, ...chunk], + ); + } +} + +/** The environment row of one overlay key, read by key (the doors' view of it). */ +async function environmentRowOf(run: Run, conflict: MetadataConflict): Promise | null }>> { + const sql = run.sql; + const t = METADATA_OBJECT; + const cols = (await columnsOf(run, t)) ?? new Set(); + const where = [`${sql.q(ORG, t)} IS NULL`, `${sql.q('type', t)} = ?`, `${sql.q('name', t)} = ?`]; + const params: unknown[] = [conflict.type, conflict.name]; + for (const [column, value] of [['package_id', conflict.packageId], ['bundle_member', conflict.bundleMember]] as const) { + if (!cols.has(column)) continue; + if (value === null) where.push(`(${sql.q(column, t)} IS NULL OR ${sql.q(column, t)} = '')`); + else { + where.push(`${sql.q(column, t)} = ?`); + params.push(value); + } + } + if (cols.has('state')) where.push(`${sql.q('state', t)} = 'active'`); + const found = await sql.read( + t, + `SELECT ${['id', 'name', 'metadata'].map((c) => `${sql.q(c, t)} AS ${sql.q(c, t)}`).join(', ')} FROM ${sql.q(t, t)} ` + + `WHERE ${where.join(' AND ')}`, + params, + ); + return found.map((row) => ({ id: String(row.id), body: parseBody(row) })); +} + +/** + * One `sys_metadata` row takes a body in place, by ONE statement, doing to the + * row what the metadata door does on a save (its caller has asked the door's + * lock gate first, before any write of the resolution); the row's + * stored `_lock` family kept (the door never sets, changes or removes it, + * ADR-0010 §3.5), whatever the body carries; and `checksum` restamped from the + * written body (`hashSpec`, the door's own), which the repository serves as + * the row's version and etag and the view save's no-op check compares. The + * version moves too, when given and stored. Returns the body written. + */ +async function writeBody( + run: Run, + row: { id: string; stored: Readonly> | null | undefined }, + body: Record, + address: { type: string; name: string; packageId: string | null }, + version?: number, +): Promise> { + const written: Record = Object.fromEntries(Object.entries(body).filter(([k]) => !k.startsWith('_lock'))); + for (const [k, v] of Object.entries(row.stored ?? {})) if (k.startsWith('_lock')) written[k] = v; + const sql = run.sql; + const t = METADATA_OBJECT; + const cols = (await columnsOf(run, t)) ?? new Set(); + const sets = [`${sql.q('metadata', t)} = ?`]; + const params: unknown[] = [JSON.stringify(written)]; + if (cols.has('checksum')) { + sets.push(`${sql.q('checksum', t)} = ?`); + params.push(hashSpec(written, address.type)); + } + if (version !== undefined && cols.has('version')) { + sets.push(`${sql.q('version', t)} = ?`); + params.push(version); + } + await execute(run, t, `UPDATE ${sql.q(t, t)} SET ${sets.join(', ')} WHERE ${sql.q('id', t)} = ?`, [...params, row.id]); + return written; +} + +/** A `sys_metadata` row's stored version, when the table stores one. */ +async function versionOf(run: Run, rowId: string): Promise { + const cols = (await columnsOf(run, METADATA_OBJECT)) ?? new Set(); + if (!cols.has('version')) return undefined; + const [row] = await run.sql.read( + METADATA_OBJECT, + `SELECT ${run.sql.q('version', METADATA_OBJECT)} AS v FROM ${run.sql.q(METADATA_OBJECT, METADATA_OBJECT)} WHERE ${run.sql.q('id', METADATA_OBJECT)} = ?`, + [rowId], + ); + const v = Number(row?.v); + return Number.isFinite(v) ? v : undefined; +} + +/** Two bodies define the same item (key order aside). */ +const sameBody = (a: Readonly>, b: Readonly>): boolean => { + const keys = [...new Set([...Object.keys(a), ...Object.keys(b)])].sort(); + return definitionKey(a, keys) === definitionKey(b, keys); +}; + +/** + * Resolve one chosen conflict so that the key holds exactly ONE environment + * row at every point, and a rerun after a crash anywhere in it finishes it: + * + * 1. The body the environment keeps: the chosen row's, with every public-form + * withdrawal an unchosen row holds applied (fail-closed: a choice never + * reopens a form). + * 2. Where the key has an environment row, that row takes the body in place, + * by one UPDATE, even when an organization's row was chosen: the chosen + * row's lineage becomes environment history first ({@link foldLineage}) and + * the environment row takes the chosen row's version, which names that + * body's snapshot. Where the key has none, the chosen organization row is + * promoted in place, its history with it. Not through the door: for an item + * a package ships, the door writes the environment's customization into that + * package's slot, never onto a package-less row of the key. A withdrawal the + * body keeps is then saved through the door as well, which records it in + * history and, for a packaged item, closes the form in the package's slot. + * 3. The environment row is read back and must define that body. + * 4. The choice is recorded in the checkpoint. + * 5. Under `single`, the conflict's organization rows that remain are deleted, + * in ONE statement, by id (ruling B on #23413 Q1). Under a wall nothing is + * deleted: they stay organization rows, reported, and hold the column + * (ruling A on #23413 Q6). + * + * Returns a blocker, or `null` once resolved. + */ +async function resolveMetadataConflict(run: Run, r: MetadataResolution, notes: string[]): Promise { + const { conflict, chosen } = r; + const sql = run.sql; + const t = sql.q(METADATA_OBJECT, METADATA_OBJECT); + const id = sql.q('id', METADATA_OBJECT); + const what = `the ${conflict.type} ${conflict.name}${conflict.packageId ? ` (package ${conflict.packageId})` : ''}`; + const others = conflict.candidates.filter((c) => c.id !== chosen.id); + const bodies = await metadataBodies(run, conflict.candidates.map((c) => c.id)); + const chosenBody = bodies.get(chosen.id); + if (!chosenBody) { + return `The row ${chosen.id} chosen for ${what} has no readable body, so the environment cannot keep it. Choose another row, or repair it, then re-run.`; + } + const body = JSON.parse(JSON.stringify(chosenBody)) as Record; + let closing: string[] = []; + if (conflict.type === 'view') { + const layer = others.map((o) => bodies.get(o.id)).filter((b): b is Record => !!b); + const forms = anonymousFormIntakeCandidates(body).filter((c) => anonymousFormIntakeWithdrawnIn(layer, body, c)); + for (const candidate of forms) { + candidate.form.sharing = { ...(candidate.form.sharing as Record), enabled: false }; + } + closing = forms.map((c) => c.slug); + } + + const environment = await environmentRowOf(run, conflict); + if (environment.length > 1) { + return ( + `${what} has ${environment.length} active environment rows (${environment.map((e) => e.id).join(', ')}), so no ` + + 'single row can take the chosen body. Delete all but one, then re-run.' + ); + } + const env = environment[0]; + const address = { type: conflict.type, name: conflict.name, packageId: conflict.packageId }; + // The door's lock gate, asked ONCE and before any write of this resolution + // (the history fold, the promotion, the in-place write): a locked item stays + // locked, and a refusal leaves nothing written. + const writes = env ? !(env.body && sameBody(env.body, body)) : chosen.organizationId !== null; + if (writes) { + const verdict = await run.opts.lock.verdict(address.type, address.name, address.packageId); + if (!verdict.editable) { + return ( + `${what} could not take the body of the chosen row ${chosen.id}: the item is locked (_lock=${verdict.lock ?? 'unknown'}` + + `${verdict.lockReason ? `: ${verdict.lockReason}` : ''}), so the metadata door would refuse the write, and the ` + + 'ceremony does not make it. Nothing was written or deleted; unlock the item, or choose the row the environment ' + + 'already holds, then re-run.' + ); + } + } + // The body the key's environment row must read back with: the chosen body, as written. + let expected: Record = env?.body && sameBody(env.body, body) ? env.body : body; + try { + if (!env) { + if (chosen.organizationId !== null) { + await promoteRows(run, chosen.organizationId, [{ id: chosen.id, type: conflict.type, name: conflict.name }]); + if (closing.length) { + expected = await writeBody(run, { id: chosen.id, stored: chosenBody }, body, address); + } + } + } else if (!(env.body && sameBody(env.body, body))) { + let version: number | undefined; + if (chosen.organizationId !== null) { + await foldLineage(run, chosen.organizationId, conflict.type, conflict.name); + version = await versionOf(run, chosen.id); + } + expected = await writeBody(run, { id: env.id, stored: env.body }, body, address, version); + } + // The row as written (its stored lock family kept), so a rerun's door save meets the same body. + if (closing.length) await run.opts.door.save(conflict.type as never, conflict.name, expected, { packageId: conflict.packageId }); + } catch (error) { + return ( + `${what} could not take the body of the chosen row ${chosen.id}` + + `${closing.length ? ` (with the public form(s) ${closing.join(', ')} an unchosen row withdraws, withdrawn)` : ''}: ` + + `${error instanceof Error ? error.message : String(error)}. Nothing was deleted; fix the cause, then re-run.` + ); + } + const after = await environmentRowOf(run, conflict); + if (after.length !== 1 || !after[0]!.body || !sameBody(after[0]!.body, expected)) { + throw new Error( + `${what} did not read back as one environment row defining the chosen row ${chosen.id}'s body; none of the ` + + `rows ${others.map((o) => o.id).join(', ')} was deleted.`, + ); + } + if (run.checkpoint) { + run.checkpoint.metadataChoicesApplied = [...new Set([...(run.checkpoint.metadataChoicesApplied ?? []), chosen.id])]; + await run.opts.ledger.write(run.checkpoint, { complete: false, blocking: 0, advisory: 0 }); + } + + const keeper = after[0]!.id; + const leaving = conflict.candidates.filter((c) => c.organizationId !== null && c.id !== keeper).map((c) => c.id); + if (run.opts.posture !== 'single') { + for (const remaining of conflict.candidates.filter((c) => c.organizationId !== null && c.id !== keeper)) { + notes.push( + `Remaining: ${METADATA_OBJECT} row ${remaining.id} (organization ${remaining.organizationId}` + + `${remaining.id === chosen.id ? `, the chosen row, whose body the environment row ${keeper} now holds` : ''}) ` + + `of ${what} blocks column-drops on ${METADATA_OBJECT}: under the ${run.opts.posture} posture the ceremony ` + + 'never deletes it. Remedy: check the environment row, then delete this row yourself, or keep it and the ' + + 'column stays; re-run --apply.', + ); + } + run.opts.onProgress?.(`promotion: ${what} is defined by row ${chosen.id}'s body; nothing deleted under a wall`); + return null; + } + if (leaving.length) { + const where = `${id} IN (${leaving.map(() => '?').join(', ')})`; + await execute(run, METADATA_OBJECT, `DELETE FROM ${t} WHERE ${where}`, leaving); + if ((await sql.count(METADATA_OBJECT, `SELECT COUNT(*) AS n FROM ${t} WHERE ${where}`, leaving)) !== 0) { + throw new Error(`The delete of the organization rows of ${what} left rows behind.`); + } + run.deleted.set(METADATA_OBJECT, (run.deleted.get(METADATA_OBJECT) ?? 0) + leaving.length); + } + run.opts.onProgress?.(`promotion: ${what} is defined by row ${chosen.id}'s body; ${leaving.length} organization row(s) deleted`); + return null; +} + +async function runMetadataPromotion(run: Run): Promise { + const summary: Record = { withdrawalsCarried: 0, promoted: 0, conflicts: 0, chosen: 0, organizationRowsDeleted: 0, reported: 0 }; + const blockers: string[] = []; + const notes: string[] = []; + const cols = await columnsOf(run, METADATA_OBJECT); + if (!cols?.has(ORG)) return { step: 'metadata-promotion', status: 'done', summary, blockers, notes }; + const rows = await readMetadataKeyRows(run.sql, METADATA_OBJECT, cols); + const orgRows = rows.filter((r) => r.organizationId !== null); + if (orgRows.length === 0) return { step: 'metadata-promotion', status: 'done', summary, blockers, notes }; + + // Withdrawals first, under every posture: closing a form is never a leak. + await carryFormWithdrawals(run, rows, summary, blockers); + + // Under a wall nothing is promoted unless the operator names the row + // (--metadata-choice): one organization's customization is not the + // environment's, and only a knowing choice makes it so (ruling A on #23413 Q6). + const walled = run.opts.posture !== 'single'; + const choices = new Set(run.opts.metadataChoices ?? []); + const { promotable, conflicts } = groupMetadataKeyRows(rows); + + const resolutions: MetadataResolution[] = []; + let heldInConflicts = 0; + for (const conflict of conflicts) { + const chosen = conflict.candidates.find((c) => choices.has(c.id)); + if (chosen) { + resolutions.push({ conflict, chosen }); + continue; + } + summary.conflicts++; + const listing = + `The ${conflict.type} ${conflict.name}` + + `${conflict.packageId ? ` (package ${conflict.packageId})` : ''}${conflict.bundleMember ? ` [${conflict.bundleMember}]` : ''} ` + + `is held by ${conflict.candidates.length} rows once the organization leaves its key: ` + + conflict.candidates.map((c) => `${c.id} (${c.organizationId ? `organization ${c.organizationId}` : 'environment'}, ${c.state ?? 'no state'})`).join(', '); + if (walled) { + // Under a wall the organization rows stay where they are, reported; + // they hold the column, and a choice promotes one knowingly. + heldInConflicts += conflict.candidates.filter((c) => c.organizationId !== null).length; + notes.push(`${listing}. Under the ${run.opts.posture} posture they stay organization rows; choose one with --metadata-choice ROW_ID to make it the environment's.`); + continue; + } + blockers.push( + `${listing}. Which one the environment keeps is never guessed: choose one with --metadata-choice ROW_ID, then re-run. ` + + 'The organization rows are deleted only once the environment row defines the chosen body and reads back' + + (conflict.type === 'view' ? ', and a public form any of them withdraws stays withdrawn there.' : '.'), + ); + } + + const lone = promotable.filter((row) => !walled || choices.has(row.id)); + const byOrganization = new Map(); + for (const row of lone) byOrganization.set(row.organizationId!, [...(byOrganization.get(row.organizationId!) ?? []), row]); + for (const [organizationId, own] of byOrganization) { + await promoteRows(run, organizationId, own); + summary.promoted += own.length; + } + for (const resolution of resolutions) { + const deletedBefore = run.deleted.get(METADATA_OBJECT) ?? 0; + const blocker = await resolveMetadataConflict(run, resolution, notes); + if (blocker) { + blockers.push(blocker); + continue; + } + summary.chosen++; + summary.organizationRowsDeleted += (run.deleted.get(METADATA_OBJECT) ?? 0) - deletedBefore; + } + + if (walled) { + const held = promotable.length - lone.length; + summary.reported = held + heldInConflicts; + if (held > 0) { + notes.push( + `${held} organization-scoped presentational ${METADATA_OBJECT} row(s) are reported, not promoted: under the ` + + `${run.opts.posture} posture one organization's customization is not the environment's. Promote one ` + + `knowingly with --metadata-choice ROW_ID; until then they keep the ${METADATA_OBJECT} column from being dropped.`, + ); + } + } + if (summary.promoted) run.opts.onProgress?.(`promotion: ${summary.promoted} organization customization(s) are environment rows now`); + return { step: 'metadata-promotion', status: blockers.length ? 'blocked' : 'done', summary, blockers, notes }; +} + +/** + * Refuse, before anything is written, a `--metadata-choice` that names no row + * awaiting a choice, or two candidates of one conflict. A row awaits a choice + * when it is a candidate of a listed conflict or an organization's row its key + * alone holds. So that a rerun after a crash finishes with the same flags, a + * choice this ceremony already applied is accepted too: its row is now the + * key's only environment row, or it is gone and the checkpoint records it. + */ +async function validateMetadataChoices(run: Run, applied: ReadonlySet): Promise { + const choices = run.opts.metadataChoices ?? []; + if (choices.length === 0) return; + const cols = await columnsOf(run, METADATA_OBJECT); + const rows = cols?.has(ORG) ? await readMetadataKeyRows(run.sql, METADATA_OBJECT, cols) : []; + const { promotable, conflicts } = groupMetadataKeyRows(rows); + const keyOf = (c: MetadataConflict): string => + `${c.type} ${c.name}${c.packageId ? ` (package ${c.packageId})` : ''}${c.bundleMember ? ` [${c.bundleMember}]` : ''}`; + const groupOf = new Map(); + for (const c of conflicts) for (const cand of c.candidates) groupOf.set(cand.id, keyOf(c)); + for (const row of promotable) groupOf.set(row.id, `the ${row.type} ${row.name} (row ${row.id})`); + const present = new Set(rows.map((r) => r.id)); + const listed = conflicts.length + ? conflicts.map((c) => `${keyOf(c)}: ${c.candidates.map((cand) => cand.id).join(', ')}`).join('; ') + : 'none'; + const seen = new Map(); + for (const id of choices) { + const group = groupOf.get(id); + if (!group) { + const resolved = rows.some((r) => r.id === id && r.organizationId === null) || (applied.has(id) && !present.has(id)); + if (resolved) continue; + throw new MetadataChoiceRefusal( + `--metadata-choice ${id} names no row awaiting a choice. The conflicts are: ${listed}; or any organization ` + + 'row of a presentational type its key alone holds.', + ); + } + const other = seen.get(group); + if (other !== undefined && other !== id) { + throw new MetadataChoiceRefusal(`--metadata-choice ${other} and ${id} both choose for ${group}; choose exactly one row per conflict.`); + } + seen.set(group, id); + } +} + // ── Step 2b: template promotion (ruling C) ────────────────────────────────── async function runTemplatePromotion(run: Run): Promise { @@ -1234,6 +2188,18 @@ const GRANT_SET_ID_INDEXES: readonly RetiredIndex[] = [ ]; async function runColumnDrops(run: Run): Promise { + if (!run.historyVerified) { + return { + step: 'column-drops', + status: 'blocked', + summary: { dropped: 0, due: 0 }, + blockers: [ + 'column-drops runs only after this apply\'s history-renumber verified sys_metadata_history\'s event_seq and ' + + 'versions unique environment-wide, and it did not. Re-run --apply.', + ], + notes: [], + }; + } const plan = await planFor(run); const blockers: string[] = []; const due: ColumnDrop[] = []; @@ -1308,10 +2274,12 @@ async function runColumnDrops(run: Run): Promise { const RUNNERS: Record Promise> = { attribution: runAttribution, 'authored-conversion': runAuthoredConversion, + 'metadata-promotion': runMetadataPromotion, 'template-promotion': runTemplatePromotion, 'settings-move': runSettingsMove, rewrite: runRewrite, 'mirror-deletion': runMirrorDeletion, + 'history-renumber': runHistoryRenumber, 'column-drops': runColumnDrops, }; @@ -1385,6 +2353,51 @@ function refuseUnreviewedPlan(plan: OrganizationOwnershipPlan, reviewed: Reviewe } } +/** + * Refuse, before anything is written, a choice whose conflict group is not the + * one the reviewed plan listed. A `--metadata-choice` or `--template-choice` + * deletes the other rows of its group, so the apply deletes only within a group + * the operator reviewed, and only while it holds exactly the reviewed rows: a + * row that joined the group since, or one gone from it, refuses the apply with + * the group, the rows added and the rows gone. A choice in no group (a lone + * row, or a conflict an earlier run resolved) deletes nothing and is not judged. + */ +function refuseUnreviewedChoices(plan: OrganizationOwnershipPlan, opts: ApplyOptions): void { + const current = reviewedConflictsOf(plan); + const reviewed = opts.reviewedPlan.conflicts ?? []; + const flagOf = { metadata: '--metadata-choice', template: '--template-choice' } as const; + for (const [kind, choices] of [['metadata', opts.metadataChoices ?? []], ['template', opts.templateChoices ?? []]] as const) { + for (const choice of choices) { + const group = current.find((g) => g.kind === kind && g.candidates.includes(choice)); + if (!group) continue; + const listed = reviewed.find((g) => g.kind === kind && g.key === group.key); + const was = new Set(listed?.candidates ?? []); + const added = group.candidates.filter((id) => !was.has(id)); + const gone = [...was].filter((id) => !group.candidates.includes(id)); + if (listed && added.length === 0 && gone.length === 0) continue; + throw new ReviewedPlanRefusal( + 'plan-conflict-mismatch', + `${flagOf[kind]} ${choice} resolves the ${kind === 'metadata' ? 'customization' : 'template'} conflict ` + + `${group.key}, which now holds ${group.candidates.join(', ')}` + + (listed + ? `, while the plan file ${opts.reviewedPlan.file} reviewed ${listed.candidates.join(', ')}` + + `${added.length ? ` (added since: ${added.join(', ')})` : ''}${gone.length ? ` (gone since: ${gone.join(', ')})` : ''}` + : `, and the plan file ${opts.reviewedPlan.file} lists no such conflict`) + + '. A choice deletes the other rows of its conflict, so it deletes only rows the reviewed plan listed. ' + + 'Nothing was written. Run the plan again, review the conflict, and pass the new file with --plan-file.', + { + file: opts.reviewedPlan.file, + flag: flagOf[kind], + choice, + conflict: group.key, + reviewed: listed ? listed.candidates.join(',') : '', + current: group.candidates.join(','), + }, + ); + } + } +} + /** * Run the ceremony's apply: the fates in order, resuming from the checkpoint, * then the post-check. Throws only for a failure mid-write (the checkpoint @@ -1413,10 +2426,15 @@ export async function applyOrganizationOwnership(opts: ApplyOptions): Promise checkpoint.completed[s]); const steps: StepReport[] = []; const superseded = stored && stored.inventoryDigest !== before.inventoryDigest @@ -1437,6 +2455,24 @@ export async function applyOrganizationOwnership(opts: ApplyOptions): Promise n > 0)) { + steps.push(again); + checkpoint.completed[step] = { at: now().toISOString(), summary: again.summary }; + await opts.ledger.write(checkpoint, { complete: false, blocking: 0, advisory: 0 }); + run.columns.clear(); + continue; + } + } steps.push({ step, status: 'checkpointed', summary: done.summary, blockers: [], notes: [] }); continue; } @@ -1535,5 +2571,11 @@ export function checkpointFromDetails(details: unknown): CeremonyCheckpoint | nu } const cp = (parsed as { checkpoint?: unknown } | null)?.checkpoint as CeremonyCheckpoint | undefined; if (!cp || cp.ceremony !== 'ADR-0131 D10' || typeof cp.inventoryDigest !== 'string' || typeof cp.completed !== 'object') return null; - return { ceremony: cp.ceremony, inventoryDigest: cp.inventoryDigest, completed: { ...cp.completed } }; + const applied = Array.isArray(cp.metadataChoicesApplied) ? cp.metadataChoicesApplied.filter((x): x is string => typeof x === 'string') : []; + return { + ceremony: cp.ceremony, + inventoryDigest: cp.inventoryDigest, + completed: { ...cp.completed }, + ...(applied.length ? { metadataChoicesApplied: applied } : {}), + }; } diff --git a/packages/cli/src/utils/organization-ownership-inventory.test.ts b/packages/cli/src/utils/organization-ownership-inventory.test.ts index c01b37cd2b9..3afc578072d 100644 --- a/packages/cli/src/utils/organization-ownership-inventory.test.ts +++ b/packages/cli/src/utils/organization-ownership-inventory.test.ts @@ -27,6 +27,7 @@ import { readHostFateSupplement, type InventoryEntry, } from './organization-ownership-inventory.js'; +import { PRESENTATIONAL_TYPES } from './organization-ownership-plan.js'; /** * The first census's platform population — 59 carrying the column, 25 not — @@ -310,3 +311,11 @@ describe('readHostFateSupplement — a file it cannot use is refused, naming the expect(await refusalOfAsync(() => readHostFateSupplement(file))).toMatchObject({ reason: 'fate-supplement-platform-object', file, row: 0, object: 'sys_member' }); }); }); + +describe('the promotion population is the inventory\'s', () => { + it('PRESENTATIONAL_TYPES, which the apply promotes, is the list the sys_metadata promotion category names', () => { + const category = inventoryEntryFor('sys_metadata')!.categories!.find((c) => c.id === 'organization-presentational-promotion')!; + const types = category.where!.find((c) => c.column === 'type') as { in?: readonly string[] }; + expect([...PRESENTATIONAL_TYPES].sort()).toEqual([...(types.in ?? [])].sort()); + }); +}); diff --git a/packages/cli/src/utils/organization-ownership-inventory.ts b/packages/cli/src/utils/organization-ownership-inventory.ts index 042f650ac33..25bb2ac603c 100644 --- a/packages/cli/src/utils/organization-ownership-inventory.ts +++ b/packages/cli/src/utils/organization-ownership-inventory.ts @@ -318,10 +318,10 @@ export const ORGANIZATION_OWNERSHIP_INVENTORY: readonly InventoryEntry[] = [ }, { id: 'environment-overlay-duplicates', - label: 'active overlays sharing (type, name, package) once the organization leaves the key — reported, never dropped', - citation: 'triage pointer 6071418113 (the overlay index pre-flight joins the re-key stage)', + label: 'active overlays sharing (type, name, package, bundle member) once the organization leaves the key — reported, never dropped', + citation: 'triage pointer 6071418113 (the overlay index pre-flight joins the re-key stage); the bundle member is part of the overlay key (`sys_metadata`\'s overlay index)', where: [{ column: 'state', equals: 'active' }], - conflictBy: { groupBy: ['type', 'name', 'package_id'] }, + conflictBy: { groupBy: ['type', 'name', 'package_id', 'bundle_member'] }, }, ], }, diff --git a/packages/cli/src/utils/organization-ownership-plan.integration.test.ts b/packages/cli/src/utils/organization-ownership-plan.integration.test.ts index f2fe6097343..906fce74fd0 100644 --- a/packages/cli/src/utils/organization-ownership-plan.integration.test.ts +++ b/packages/cli/src/utils/organization-ownership-plan.integration.test.ts @@ -70,7 +70,8 @@ function seed(): void { -- fate 1, nothing to drop: better-auth's own table CREATE TABLE sys_user (id TEXT PRIMARY KEY, email TEXT); INSERT INTO sys_user VALUES ('usr_1', 'a@example.com'); - -- fate 1 with the ruled promotion categories + -- fate 1 with the ruled promotion categories, in the 17.x shape: no + -- bundle_member column (it first shipped in 18.0.0-next.0) CREATE TABLE sys_metadata (id TEXT PRIMARY KEY, organization_id TEXT, type TEXT, name TEXT, state TEXT, package_id TEXT); INSERT INTO sys_metadata VALUES ('md_1', NULL, 'object', 'crm_deal', 'active', NULL), @@ -118,6 +119,20 @@ afterEach(() => { }); describe('ADR-0131 D10 preflight — the per-table plan', () => { + it('plans a 17.x sys_metadata with no bundle_member column, and keys by the member where the column exists', async () => { + // The 17.x shape (the seed): planned, not refused for the missing column. + const before = table(await plan('isolated'), 'sys_metadata'); + expect(before.categories.find((c) => c.id === 'environment-overlay-duplicates')?.groups).toBe(1); + + // An 18.x shape: two locales of one item are two keys, not a duplicate. + db.exec(`ALTER TABLE sys_metadata ADD COLUMN bundle_member TEXT; + UPDATE sys_metadata SET bundle_member = 'en-US' WHERE id = 'md_2'; + UPDATE sys_metadata SET bundle_member = 'fr-FR' WHERE id = 'md_3';`); + const after = table(await plan('isolated'), 'sys_metadata'); + expect(after.categories.find((c) => c.id === 'environment-overlay-duplicates')?.groups).toBe(0); + expect(after.metadataConflicts).toEqual([]); + }); + it('names each fate with its counts, the unattributable ids, and NOT NULL readiness (isolated)', async () => { const p = await plan('isolated'); @@ -140,6 +155,19 @@ describe('ADR-0131 D10 preflight — the per-table plan', () => { 'organization-scoped-other-types': 1, 'environment-overlay-duplicates': 1, // (view, deal_list, no package) twice once the organization leaves the key }); + // The conflict list the promotion blocks on, by row id (rulings A on #22011 / #22008). + expect(metadata.metadataConflicts).toEqual([ + { + type: 'view', + name: 'deal_list', + packageId: null, + bundleMember: null, + candidates: [ + { id: 'md_2', organizationId: 'org_default', state: 'active' }, + { id: 'md_3', organizationId: 'org_b', state: 'active' }, + ], + }, + ]); expect(table(p, 'sys_position')).toMatchObject({ fate: 'mirror-deletion', diff --git a/packages/cli/src/utils/organization-ownership-plan.ts b/packages/cli/src/utils/organization-ownership-plan.ts index 435f67a0027..2e0d1b3e161 100644 --- a/packages/cli/src/utils/organization-ownership-plan.ts +++ b/packages/cli/src/utils/organization-ownership-plan.ts @@ -155,10 +155,29 @@ export interface TablePlan { * until the operator chooses one row per group; nothing is guessed. */ templateConflicts?: TemplateConflict[]; + /** + * `sys_metadata` only: the keys of the promotion population that more than + * one row holds once the organization leaves the key — two organizations, or + * an organization and the environment. Kept and blocking until the operator + * keeps one row per key; nothing is guessed. + */ + metadataConflicts?: MetadataConflict[]; /** `sys_user_permission_set` only: what the grant's unique key holds ({@link GrantKeyPlan}). */ grants?: GrantKeyPlan; } +/** + * One overlay key of `sys_metadata` (`type`, `name`, package, bundle member) + * that the promotion cannot carry alone: every row that holds it, and whose. + */ +export interface MetadataConflict { + type: string; + name: string; + packageId: string | null; + bundleMember: string | null; + candidates: Array<{ id: string; organizationId: string | null; state: string | null }>; +} + /** * `sys_user_permission_set` (C7b): the populations the unique index on * (`user_id`, `permission_set`, `organization_id`) refuses, which the ceremony @@ -211,6 +230,98 @@ export function templatePromotionSql(sql: Sql, cols: ReadonlySet): { tex }; } +/** The table whose organization-scoped customizations the ceremony promotes (ruling record 6020163868, A). */ +export const METADATA_OBJECT = 'sys_metadata'; + +/** + * The five presentational types whose organization-scoped rows are promoted to + * environment rows (ruling records 6020163868 and 6020151485, both A). The + * inventory's `organization-presentational-promotion` category names the same + * list, and a pin holds the two equal. + */ +export const PRESENTATIONAL_TYPES: readonly string[] = ['view', 'dashboard', 'report', 'translation', 'email_template']; + +/** One `sys_metadata` row of a presentational type, by its overlay key. */ +export interface MetadataKeyRow { + id: string; + type: string; + name: string; + packageId: string | null; + bundleMember: string | null; + /** `null` = an environment row. */ + organizationId: string | null; + state: string | null; +} + +/** + * Every `sys_metadata` row of a presentational type, by key, keyset-paged: + * environment rows and organization rows alike, so a key the environment + * already holds is seen as held. No body is read. + */ +export async function readMetadataKeyRows(sql: Sql, table: string, columns: ReadonlySet): Promise { + for (const column of ['type', 'name', ORG]) requireColumn(table, columns, column); + const select = ['id', 'type', 'name', ORG, ...['package_id', 'bundle_member', 'state'].filter((c) => columns.has(c))]; + const t = sql.q(table, table); + const id = sql.q(ID, table); + const types = PRESENTATIONAL_TYPES.map(() => '?').join(', '); + const out: MetadataKeyRow[] = []; + let after: unknown; + for (;;) { + const rows = await sql.read( + table, + `SELECT ${select.map((c) => `${sql.q(c, table)} AS ${sql.q(c, table)}`).join(', ')} FROM ${t} ` + + `WHERE ${sql.q('type', table)} IN (${types})${after === undefined ? '' : ` AND ${id} > ?`} ORDER BY ${id} LIMIT ${PAGE}`, + after === undefined ? [...PRESENTATIONAL_TYPES] : [...PRESENTATIONAL_TYPES, after], + ); + for (const row of rows) { + out.push({ + id: String(row.id), + type: String(row.type), + name: String(row.name), + packageId: nonEmptyId(row.package_id), + bundleMember: nonEmptyId(row.bundle_member), + organizationId: nonEmptyId(row[ORG]), + state: nonEmptyId(row.state), + }); + } + if (rows.length < PAGE) return out; + after = rows[rows.length - 1]!.id; + } +} + +/** + * Split the promotion population by overlay key: a key held by exactly one + * row, an organization's, is promotable in place; a key held by an + * organization's row and any other row (another organization's, or the + * environment's) is a conflict. Keys only the environment holds are not the + * ceremony's. + */ +export function groupMetadataKeyRows(rows: readonly MetadataKeyRow[]): { promotable: MetadataKeyRow[]; conflicts: MetadataConflict[] } { + const byKey = new Map(); + for (const row of rows) { + const key = [row.type, row.name, row.packageId ?? '', row.bundleMember ?? ''].join('\u0000'); + byKey.set(key, [...(byKey.get(key) ?? []), row]); + } + const promotable: MetadataKeyRow[] = []; + const conflicts: MetadataConflict[] = []; + for (const group of byKey.values()) { + if (!group.some((r) => r.organizationId !== null)) continue; + if (group.length === 1) { + promotable.push(group[0]!); + continue; + } + const first = group[0]!; + conflicts.push({ + type: first.type, + name: first.name, + packageId: first.packageId, + bundleMember: first.bundleMember, + candidates: group.map((r) => ({ id: r.id, organizationId: r.organizationId, state: r.state })), + }); + } + return { promotable, conflicts }; +} + /** The JSON of a definition's own keys, for comparing two bodies. */ export const definitionKey = (body: Readonly>, keys: readonly string[]): string => JSON.stringify(keys.map((k) => [k, body[k] ?? null])); @@ -637,16 +748,34 @@ async function applyAnchor( return left; } +/** + * Overlay-key columns that arrived after 17.x: `sys_metadata.bundle_member` + * first shipped in 18.0.0-next.0. Both the plan and the apply boot with schema + * changes held, so on the 17.x databases this ceremony migrates the column is + * absent, which the overlay key reads as NULL (its NULL-folded part). + */ +const FOLDED_KEY_COLUMNS: ReadonlySet = new Set(['bundle_member']); + async function planCategory(sql: Sql, table: string, columns: ReadonlySet, category: InventoryCategory): Promise<{ rows: number; groups?: number }> { + // A category that selects on the organization column has no rows left once + // the column is gone (the ceremony's post-check reads a dropped table too). + if (!columns.has(ORG) && conditionColumns(category.where).includes(ORG)) { + return category.conflictBy ? { rows: 0, groups: 0 } : { rows: 0 }; + } const t = sql.q(table, table); const where = predicateSql(sql, table, columns, category.where ?? []); if (!category.conflictBy) { return { rows: await sql.count(table, `SELECT COUNT(*) AS n FROM ${t} WHERE ${where.text}`, where.params) }; } - const groupBy = category.conflictBy.groupBy.map((c) => { - requireColumn(table, columns, c); - return sql.q(c, table); - }); + // A key column a 17.x database never had (see FOLDED_KEY_COLUMNS) reads as + // NULL on every row there, and grouping by a constant is no grouping: it is + // grouped by only where it exists. Every other column is required. + const groupBy = category.conflictBy.groupBy + .filter((c) => columns.has(c) || !FOLDED_KEY_COLUMNS.has(c)) + .map((c) => { + requireColumn(table, columns, c); + return sql.q(c, table); + }); const distinct = category.conflictBy.distinct; if (distinct) requireColumn(table, columns, distinct); const having = distinct ? `COUNT(DISTINCT ${sql.q(distinct, table)}) > 1` : 'COUNT(*) > 1'; @@ -786,6 +915,11 @@ async function planPresentTable(ctx: PlanContext, entry: InventoryEntry, source: } plan.categories.push({ id: category.id, label: category.label, citation: category.citation, rows, ...(groups !== undefined ? { groups } : {}) }); } + if (entry.object === METADATA_OBJECT && hasOrg) { + const conflicts: MetadataConflict[] = []; + for (const table of tables) conflicts.push(...groupMetadataKeyRows(await readMetadataKeyRows(sql, table, ctx.catalog.columns.get(table)!)).conflicts); + plan.metadataConflicts = conflicts; + } if (entry.object === TEMPLATE_OBJECT) { const conflicts: TemplateConflict[] = []; for (const table of tables) conflicts.push(...(await readTemplateConflicts(sql, table, ctx.catalog.columns.get(table)!)));