feat(driver-sql)!: NULL-safe organization uniques + declared-index 'organization' scope (ADR-0120 D3/D4, #5030)

Every organization-scoped unique index now materializes its organization key
part as COALESCE(organization_id, '__global__') instead of the bare column:
SQL UNIQUE is NULL-distinct, so the #3696 composite enforced nothing on
NULL-organization rows — on a single-tenant stack, every row (#5030, measured).

- uniqueIndexesFromFields / legacy replacements emit the NULL-safe key part;
  field-level unique: 'organization' accepted as the explicit synonym of true.
- normalizeDeclaredIndex / expectedIndexes / syncDeclaredIndexes learn declared
  unique: 'organization' (prepend the org key part; degrade with no tenant
  column; a listed tenant column is made NULL-safe in place). 'global' / bare
  true stays VERBATIM — the #3696 contract, now the 'global' arm (spec token
  lands separately via #4986; driver deliberately merges first).
- Drift both sides share one normalization: physical COALESCE(col, <literal>)
  attributes to col, compared literal-agnostically; the org key part is the
  sync's own vocabulary (isSyncReproducibleIndex + tenantField), scoped so the
  ADR-0048 overlay indexes keep their #4884 protection.
- D4 ceremony: the bare-composite tightening is a recreate_index gated by a
  duplicate pre-flight probe — clean → safe (dev autoMigrate applies), dirty →
  BLOCKED with a row report, old index kept; apply re-probes so even
  --allow-destructive cannot drop a constraint whose replacement fails.
- plugin-auth: '__global__' reserved as organization id/slug at the
  beforeCreateOrganization seam (ADR-0120 D3 guardrail).
- Tests per ADR-0120 D6.6: contract header rewritten, #5030 probe graduated as
  a permanent regression, NULL-bucket semantics, declared 'organization'
  pins, pre-flight both states; 'exactly as authored' retained for 'global'.
- ADR anchors for schema-drift.ts / sql-driver.ts (PD #13).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Akrzh2mHi2siSNVPPtfTw7

Author: claude

.changeset/null-safe-org-unique-driver.md

@@ -0,0 +1,59 @@
+---
+"@objectstack/driver-sql": minor
+"@objectstack/plugin-auth": patch
+---
+
+feat(driver-sql)!: organization-scoped uniques are NULL-safe — `COALESCE(organization_id, '__global__')` key part + `unique: 'organization'` on declared indexes (ADR-0120 D3/D4, #5030)
+
+SQL UNIQUE is NULL-distinct, so the `(organization_id, field)` composite #3696
+introduced enforced **nothing** on rows whose organization is NULL — which on a
+single-tenant stack (where the kernel injects the column and never fills it) is
+**every row**: field-level `unique: true` was a silent no-op there, measured in
+#5030. Per ADR-0120 D3, every organization-scoped unique now materializes its
+organization key part as `COALESCE(organization_id, '__global__')`: NULL-organization
+rows collapse into one platform bucket, unique among themselves; non-NULL rows
+are untouched. Storage stays NULL — the sentinel exists only inside the index
+key, and it is the same word the autonumber sequence table already uses
+(`GLOBAL_TENANT`), so a constraint-violation error reads as "the platform
+bucket collided", not as corrupt data.
+
+What changes, concretely:
+
+- **Field-level `unique: true`** (and the new explicit synonym
+  `'organization'`) on a tenant-scoped object → composite
+  `(COALESCE(tenantField, '__global__'), field)`. `unique: 'global'` and
+  tenant-less objects are unchanged.
+- **Declared indexes gain the ADR-0120 D1 scope vocabulary at the driver**:
+  `unique: 'organization'` prepends the NULL-safe organization key part to the
+  listed columns (degrading to the listed columns on a tenant-less object; a
+  listed tenant column is made NULL-safe in place instead — the S6 respelling).
+  `unique: true` / `'global'` on a declared index stays **verbatim** — the
+  #3696 contract, now the `'global'` arm; the nine engine dedup/idempotency
+  keys keep their exact physical shape. (The spec/lint side of the vocabulary
+  lands separately via #4986; the driver deliberately merges first.)
+- **Drift detection reads both sides through one normalization**
+  (the #4884 discipline, extended to the tenant key part): the physical
+  `COALESCE(organization_id, <literal>)` form is attributed to the column,
+  compared **literal-agnostically**, and recognised as the sync's own
+  vocabulary — a healthy database reports zero drift on every dialect.
+- **Existing bare composites migrate through the ceremony (ADR-0120 D4)**:
+  `(organization_id, X) → (COALESCE(organization_id, '__global__'), X)`
+  surfaces as a `recreate_index` drift op — a pure tightening — gated by a
+  **duplicate pre-flight probe**. Clean probe → the op grades `safe` and dev
+  `autoMigrate: 'safe'` / a plain `os migrate apply` applies it. Duplicates
+  (data the void constraint wrongly admitted) → the op is **blocked** with a
+  per-group row report, the old index stays in place, and apply re-probes so
+  even `--allow-destructive` cannot drop a constraint whose replacement is not
+  creatable. Deduplicate, re-plan, apply.
+- **`'__global__'` is reserved at the organization-minting seam**
+  (plugin-auth): an organization whose id or slug equals the sentinel is
+  rejected at creation with a prescriptive error (ADR-0120 D3 guardrail).
+
+Migration note for operators: on databases with pre-existing
+organization-composite uniques, the first `os migrate plan` after upgrading
+shows one `recreate_index` per affected index. On healthy data it auto-applies
+in dev and is a no-op content-wise; a blocked op means the #5030 defect
+admitted real duplicate rows — resolve the listed rows first. MySQL < 8.0.13 /
+MariaDB cannot express the functional key part: the driver degrades to the
+bare composite, says exactly what is not enforced at `error` level, and keeps
+reporting the tightening as drift for after the server upgrade.

packages/plugins/driver-sql/src/index.ts

@@ -34,8 +34,14 @@ export {
   parseIndexDdl,
   uniqueIndexesFromFields,
   INDEX_DRIFT_OPS,
+  // Unique-scope vocabulary + NULL-safe organization key part (ADR-0120 D1/D3)
+  GLOBAL_TENANT,
+  isUniqueScopeDeclared,
+  isOrganizationScopedUnique,
+  organizationKeyPartSql,
 } from './schema-drift.js';
 export type {
+  DeclaredIndexInput,
   ManagedDriftEntry,
   DriftOp,
   DriftCategory,

packages/plugins/driver-sql/src/schema-drift.ts

@@ -1,7 +1,13 @@
 // Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license.
 
 import { describe, it, expect, afterEach, vi } from 'vitest';
-import { SqlDriver, diffManagedIndexes, isManagedIndexName } from '../src/index.js';
+import {
+  SqlDriver,
+  classifyIndexKeyPart,
+  diffManagedIndexes,
+  isManagedIndexName,
+  parseIndexDdl,
+} from '../src/index.js';
 
 /**
  * Index-dimension managed-schema drift (#3728).
@@ -65,13 +71,37 @@ describe('SqlDriver index drift (#3728)', () => {
     },
   ];
 
+  /**
+   * Unique index name → canonical key parts, read from the index DDL so
+   * expression keys are visible (`PRAGMA index_info` reports a NULL name for
+   * them). A plain column reads as its name; the NULL-safe organization key
+   * part (ADR-0120 D3) reads as `COALESCE(<column>)` — literal elided, the
+   * same literal-agnostic identity the drift differ compares on.
+   */
   const uniqueIndexColumns = async (table: string): Promise<Record<string, string[]>> => {
     const list: any = await knexInstance.raw(`PRAGMA index_list(${table})`);
+    const master: any = await knexInstance.raw(
+      `SELECT name, sql FROM sqlite_master WHERE type = 'index' AND tbl_name = ?`,
+      [table],
+    );
+    const ddlByName = new Map<string, string>();
+    for (const r of Array.isArray(master) ? master : (master?.rows ?? [])) {
+      if (typeof r?.sql === 'string' && r.sql) ddlByName.set(r.name, r.sql);
+    }
     const out: Record<string, string[]> = {};
     for (const idx of list) {
       if (idx.origin === 'pk' || idx.unique !== 1) continue;
-      const info: any = await knexInstance.raw(`PRAGMA index_info("${idx.name}")`);
-      out[idx.name] = info.map((c: any) => c.name);
+      const parsed = parseIndexDdl(ddlByName.get(idx.name) ?? '');
+      if (parsed) {
+        out[idx.name] = parsed.keyParts.map((p) => {
+          const part = classifyIndexKeyPart(p);
+          if (part.kind === 'column') return part.column;
+          return part.column === null ? p : `COALESCE(${part.column})`;
+        });
+      } else {
+        const info: any = await knexInstance.raw(`PRAGMA index_info("${idx.name}")`);
+        out[idx.name] = info.map((c: any) => c.name);
+      }
     }
     return out;
   };
@@ -185,7 +215,7 @@ describe('SqlDriver index drift (#3728)', () => {
       // Both are current intent: the tenant composite from the field-level
       // `unique: true`, and the verbatim declared global unique.
       const uniques = await uniqueIndexColumns('hp_contact');
-      expect(uniques['uniq_hp_contact_organization_id_email']).toEqual(['organization_id', 'email']);
+      expect(uniques['uniq_hp_contact_organization_id_email']).toEqual(['COALESCE(organization_id)', 'email']);
       expect(uniques['uniq_hp_contact_email']).toEqual(['email']);
 
       // Before the fix this reported `replace_unique_index` — proposing to drop
@@ -260,7 +290,7 @@ describe('SqlDriver index drift (#3728)', () => {
 
       const uniques = await uniqueIndexColumns('product');
       expect(uniques['product_code_unique']).toBeUndefined();
-      expect(Object.values(uniques)).toContainEqual(['organization_id', 'code']);
+      expect(Object.values(uniques)).toContainEqual(['COALESCE(organization_id)', 'code']);
 
       // Existing rows survived, and the cross-tenant insert the issue is about works.
       expect(await driver.count('product', { object: 'product' })).toBe(2);
@@ -281,7 +311,7 @@ describe('SqlDriver index drift (#3728)', () => {
       const again = await driver.applyMigrationEntries(drift, { allowDestructive: false });
       expect(again.skipped).toHaveLength(0);
       expect(Object.values(await uniqueIndexColumns('product'))).toContainEqual([
-        'organization_id',
+        'COALESCE(organization_id)',
         'code',
       ]);
     });
@@ -298,7 +328,7 @@ describe('SqlDriver index drift (#3728)', () => {
 
       const uniques = await uniqueIndexColumns('product');
       expect(uniques['product_code_unique']).toBeUndefined();
-      expect(Object.values(uniques)).toContainEqual(['organization_id', 'code']);
+      expect(Object.values(uniques)).toContainEqual(['COALESCE(organization_id)', 'code']);
       expect(await driver.detectManagedDrift()).toHaveLength(0);
 
       const b = await driver.create('product', { organization_id: 'org_b', code: 'PROD-00001' });

packages/plugins/driver-sql/src/sql-driver-index-drift.test.ts

@@ -357,4 +357,91 @@ describe('overlay index drift on a fresh database (#4884)', () => {
     expect(index.expressions).toEqual(["COALESCE(package_id, '')", 'lower(name)']);
     expect(isSyncReproducibleIndex(index)).toBe(false);
   });
+
+  // ── ADR-0120 D3: the NULL-safe organization key part in the drift reader ──
+
+  describe('NULL-safe organization key part attribution (ADR-0120 D3)', () => {
+    it("attributes COALESCE(organization_id, '__global__') to organization_id in every dialect spelling", () => {
+      for (const sql of [
+        // SQLite: `sqlite_master.sql`, as the sync wrote it.
+        "COALESCE(organization_id, '__global__')",
+        'COALESCE("organization_id", \'__global__\')',
+        // Postgres: `pg_get_indexdef` casts a varchar key part.
+        "COALESCE((organization_id)::text, '__global__'::text)",
+        // MySQL: `information_schema.STATISTICS.EXPRESSION` decorates the
+        // literal with a charset introducer and backslash-escaped quotes.
+        "coalesce(`organization_id`,_utf8mb4\\'__global__\\')",
+      ]) {
+        expect(classifyIndexKeyPart(sql)).toEqual({
+          kind: 'expression',
+          sql,
+          column: 'organization_id',
+        });
+      }
+    });
+
+    it('applyIndexKeyParts records the attributed key part as a NULL-safe column', () => {
+      const index: PhysicalIndex = { name: 'i', columns: [], unique: true };
+      applyIndexKeyParts(index, ["COALESCE(organization_id, '__global__')", 'email']);
+      expect(index.columns).toEqual(['organization_id', 'email']);
+      expect(index.nullSafeColumns).toEqual(['organization_id']);
+    });
+
+    it("isSyncReproducibleIndex: the org key part is the sync's OWN vocabulary — for the tenant column only", () => {
+      const org: PhysicalIndex = { name: 'i', columns: [], unique: true };
+      applyIndexKeyParts(org, ["COALESCE(organization_id, '__global__')", 'email']);
+      expect(isSyncReproducibleIndex(org, 'organization_id')).toBe(true);
+      expect(isSyncReproducibleIndex(org, null)).toBe(false);
+      expect(isSyncReproducibleIndex(org)).toBe(false);
+      // The ADR-0048 overlay key attributes to a NON-tenant column and stays
+      // out — loosening the column scoping to "any attributable COALESCE"
+      // would resurrect the #4884 false orphan on the overlay indexes.
+      const overlay: PhysicalIndex = { name: 'o', columns: [], unique: true };
+      applyIndexKeyParts(overlay, ['type', "COALESCE(package_id, '')"]);
+      expect(isSyncReproducibleIndex(overlay, 'organization_id')).toBe(false);
+    });
+
+    it('the differ compares key-part FORM literal-agnostically: COALESCE ≡ COALESCE, bare ≠ COALESCE', () => {
+      const expected = [
+        {
+          name: 'uniq_t_organization_id_email',
+          columns: ['organization_id', 'email'],
+          unique: true,
+          nullSafeColumns: ['organization_id'],
+        },
+      ];
+      // Physical carries the COALESCE form with a DIFFERENT literal: any
+      // literal folds NULL into one bucket, so it is the same constraint —
+      // zero drift (the #4884 lesson applied to the tenant key part).
+      const physSame: PhysicalIndex = { name: 'uniq_t_organization_id_email', columns: [], unique: true };
+      applyIndexKeyParts(physSame, ["COALESCE(organization_id, '')", 'email']);
+      expect(
+        diffManagedIndexes({
+          table: 't',
+          expected,
+          legacy: [],
+          physical: [physSame],
+          tenantField: 'organization_id',
+        }),
+      ).toEqual([]);
+      // Physical is the bare NULL-distinct composite: a DIFFERENT constraint
+      // (void on NULL-organization rows, #5030) → the D4 tightening op.
+      const physBare: PhysicalIndex = {
+        name: 'uniq_t_organization_id_email',
+        columns: ['organization_id', 'email'],
+        unique: true,
+      };
+      const out = diffManagedIndexes({
+        table: 't',
+        expected,
+        legacy: [],
+        physical: [physBare],
+        tenantField: 'organization_id',
+      });
+      expect(out).toHaveLength(1);
+      expect(out[0].op).toMatchObject({ type: 'recreate_index', tightenNullSafeOnly: true });
+      expect(out[0].expected).toBe("UNIQUE (COALESCE(organization_id, '__global__'), email)");
+      expect(out[0].actual).toBe('UNIQUE (organization_id, email)');
+    });
+  });
 });