Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions .changeset/index-type-partial-removed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
---
"@objectstack/spec": major
"@objectstack/metadata-core": patch
---

refactor(spec)!: retire `indexes[].type` and `indexes[].partial` — two authorable index keys no driver ever read (#5248, #4943)

`IndexSchema` declared five keys; only three of them ever reached a `CREATE
INDEX`. `SqlDriver.syncDeclaredIndexes` builds every declared index through
knex's `table.index(fields, name)` / `table.unique(fields, { indexName })`, and
the drift differ's `DeclaredIndexInput` carries `name` / `fields` / `unique` /
`nullSafeColumns`. So:

- **`partial`** — documented as *"Partial index condition (SQL WHERE clause)"* —
produced a **full** index with the predicate silently discarded. This was the
damaging half, because it reads as a correctness control: the platform's own
`sys_metadata` declared `partial: "state = 'active'"` for overlay uniqueness,
and what the declaration alone materialized was an *unrestricted* unique index.
- **`type`** additionally carried `.default('btree')`, so it appeared in **every**
parse output of **every** index — an access-method knob that had never
influenced a single statement, rendered as live configuration. (It was pinned
as such in a `sys_presence` test, on an object that never declared it.)

Both are the ADR-0078 no-silently-inert / ADR-0049 enforce-or-remove shape.
Remove was chosen over enforce: enforcing needs per-dialect algorithm mapping
(`gin`/`gist` Postgres-only, `fulltext` MySQL-family), raw-SQL `CREATE INDEX …
WHERE` on the dialects that have partial indexes at all (MySQL does not), and a
redesign of how `isSyncReproducibleIndex` excludes partial indexes from
incremental sync — design cost for a capability with no demand. If a real need
appears it returns enforce-first.

## Migration

| FROM | TO |
| :--- | :--- |
| `indexes: [{ fields: […], type: 'gin' }]` | `indexes: [{ fields: […] }]` — create the specialised index from a database-layer migration |
| `indexes: [{ fields: […], partial: "state = 'active'" }]` | `indexes: [{ fields: […] }]` — issue `CREATE [UNIQUE] INDEX … WHERE …` from a runtime migration |

**One-line fix: delete the key.** Neither removal changes any DDL, because no
DDL ever depended on them — verified byte-for-byte against the `CREATE INDEX`
statements SQLite actually stores
(`packages/drivers/driver-sql/src/declared-index-retired-keys.test.ts`).

Both capabilities remain available where they are implementable. The index
method is the driver/dialect's choice. A partial index is issued as raw SQL from
a runtime migration — exactly what `metadata-protocol`'s `ensureOverlayIndex`
already does for `sys_metadata`, and what actually delivers that table's
active-row-scoped uniqueness today.

⚠️ **Not affected:** driver-sql's own `partial` flag (`parseIndexDdl` /
`introspectIndexes` / `isSyncReproducibleIndex`). That is a boolean parsed back
out of the *database's own* DDL for drift detection — the opposite direction —
so migration-created partial indexes stay recognized and exempt from incremental
sync, unchanged.

## The retirement kit

- `retiredKey()` tombstones at `IndexSchema` (the shape is deliberately
`.strip()`, so a plain delete would swap one silent no-op for another): writing
either key is now a `tsc` error and a parse error carrying the prescription.
They sit at the bottom of the shape per the #5606 renderer note.
- **ADR-0087 D2 conversion + D3 chain step** (`object-index-type-partial-removed`,
`toMajor: 17`, wired into the existing step-17 chain): strips both keys from
`objects[]` and `objectExtensions[]`; `os migrate meta --from 16` rewrites sources
mechanically. A pure lossless delete — there was no effect to lose.
- **Producers flipped:** `sys_metadata` (`idx_sys_metadata_overlay_active`, the
case #4943 named) and `sys_view_definition` (`idx_sys_view_def_active`), both
with their comments corrected to say what is actually materialized.
- Published skill (`objectstack-data`), `content/docs/data-modeling/objects.mdx`,
liveness ledger note and generated baselines updated.
30 changes: 22 additions & 8 deletions content/docs/data-modeling/objects.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -209,20 +209,34 @@ Optimize query performance:
{/* os:check */}
```typescript
indexes: [
{ fields: ['name'], type: 'btree', unique: false },
{ fields: ['email'], type: 'btree', unique: 'organization' },
{ fields: ['type', 'status'], type: 'btree', unique: false },
{ fields: ['name'] },
{ fields: ['email'], unique: 'organization' },
{ fields: ['type', 'status'] },
]
```

| Property | Type | Required | Description |
| :--- | :--- | :--- | :--- |
| `fields` | `string[]` | ✅ | Fields in the index |
| `type` | `enum` | optional | `'btree'`, `'hash'`, `'gin'`, `'gist'`, `'fulltext'` (default: `'btree'`) |
| `unique` | `boolean` | optional | Enforce uniqueness (default: `false`) |
| `partial` | `string` | optional | Conditional index (SQL WHERE clause) |
| `unique` | `boolean \| 'global' \| 'organization'` | optional | Enforce uniqueness, and at which scope (default: `false`) |
| `name` | `string` | optional | Index name (auto-generated if omitted) |

<Callout type="warn">
**`type` and `partial` were retired in protocol 17** (#5248, #4943). Neither
had a driver consumer: declared indexes are created through knex's
`table.index()` / `table.unique()`, so an authored `type` selected no access
method and an authored `partial` produced a **full** index with the predicate
silently discarded. Writing either now fails `tsc` and the parse with a
migration prescription — run `os migrate meta --from 16` to strip them.

Both capabilities remain available where they are actually implementable: the
index method is the driver/dialect's choice, and a partial index is issued as
raw SQL from a runtime migration (`CREATE [UNIQUE] INDEX … WHERE …`, the way
`metadata-protocol` builds `sys_metadata`'s overlay index). Drift detection
reads partiality back from the database's own DDL, so migration-created
partial indexes are recognized and left alone.
</Callout>

### Additional Properties

| Property | Type | Description |
Expand Down Expand Up @@ -347,8 +361,8 @@ export const ProjectTask = ObjectSchema.create({
},

indexes: [
{ fields: ['status'], type: 'btree', unique: false },
{ fields: ['project', 'status'], type: 'btree', unique: false },
{ fields: ['status'] },
{ fields: ['project', 'status'] },
],

enable: {
Expand Down
8 changes: 4 additions & 4 deletions content/docs/references/data/object.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -67,9 +67,9 @@ const result = ApiMethod.parse(data);
| :--- | :--- | :--- | :--- |
| **name** | `string` | optional | Index name (auto-generated if not provided) |
| **fields** | `string[]` | ✅ | Fields included in the index |
| **type** | `Enum<'btree' \| 'hash' \| 'gin' \| 'gist' \| 'fulltext'>` | ✅ | Index algorithm type |
| **unique** | `boolean \| 'global' \| 'organization'` | ✅ | Whether the index enforces uniqueness, and at which scope (ADR-0120). 'global' = materialized over exactly `fields`, no organization column injected — one holder across the whole installation; 'organization' = the driver prepends the NULL-safe organization key part (COALESCE(organization_id, '__global__')) at registration — one holder per organization; bare true = deprecated positional spelling of 'global' (warned in 17.x by lint unique/unscoped-declared-index, rejected at protocol 18, #5082) — state the scope. 'tenant'/'org' are rejected — the word is 'organization' |
| **partial** | `string` | optional | Partial index condition (SQL WHERE clause for conditional indexes) |
| **type** | `any` | optional | [REMOVED] `indexes[].type` was removed in @objectstack/spec 17.0.0 (#5248, ADR-0049) — no driver ever read it. `SqlDriver.syncDeclaredIndexes` creates every declared index through knex's `table.index()` / `table.unique()`, which cannot express an access method, so the value changed no DDL; its `.default('btree')` merely made an inert knob show up in every parse output. Delete the key. The index method is the driver/dialect's decision (Postgres defaults to B-tree; `gin`/`gist`/`fulltext` are dialect-specific and are chosen by a database-layer migration when a workload actually needs one). Run `os migrate meta --from 16` to rewrite it automatically. |
| **partial** | `any` | optional | [REMOVED] `indexes[].partial` was removed in @objectstack/spec 17.0.0 (#5248, #4943, ADR-0049) — no driver ever emitted the `WHERE` clause, so a declared partial index was materialized as a FULL index and the predicate silently did nothing. Delete the key. Partial indexes are built at the database layer, not the declaration surface: issue `CREATE [UNIQUE] INDEX … WHERE <predicate>` from a runtime migration (this is what `metadata-protocol`'s `ensureOverlayIndex` already does for `sys_metadata`). Drift detection is unaffected — it reads partiality back from the database's own DDL, never from this key. Run `os migrate meta --from 16` to rewrite it automatically. |


---
Expand Down Expand Up @@ -122,7 +122,7 @@ const result = ApiMethod.parse(data);
| **datasource** | `string` | optional | Target Datasource ID. "default" is the primary DB. |
| **external** | `{ remoteName?: string; remoteSchema?: string; writable?: boolean; columnMap?: Record<string, string>; … }` | optional | Remote table binding for federated (external) objects. |
| **fields** | `Record<string, { name?: string; label?: string; type: Enum<'text' \| 'textarea' \| 'email' \| 'url' \| 'phone' \| 'password' \| 'secret' \| 'markdown' \| 'html' \| 'richtext' \| 'number' \| 'currency' \| 'percent' \| 'date' \| 'datetime' \| 'time' \| 'boolean' \| 'toggle' \| 'select' \| 'multiselect' \| 'radio' \| 'checkboxes' \| 'lookup' \| 'master_detail' \| 'tree' \| 'user' \| 'image' \| 'file' \| 'avatar' \| 'video' \| 'audio' \| 'formula' \| 'summary' \| 'autonumber' \| 'composite' \| 'repeater' \| 'record' \| 'location' \| 'address' \| 'code' \| 'json' \| 'color' \| 'rating' \| 'slider' \| 'signature' \| 'qrcode' \| 'progress' \| 'tags' \| 'vector'>; description?: string; … }>` | ✅ | Field definitions map. Keys must be snake_case identifiers. |
| **indexes** | `{ name?: string; fields: string[]; type?: Enum<'btree' \| 'hash' \| 'gin' \| 'gist' \| 'fulltext'>; unique?: boolean \| 'global' \| 'organization'; … }[]` | optional | Database performance indexes |
| **indexes** | `{ name?: string; fields: string[]; unique?: boolean \| 'global' \| 'organization'; type?: any; … }[]` | optional | Database performance indexes |
| **fieldGroups** | `{ key: string; label: string; icon?: string; description?: string; … }[]` | optional | Ordered list of field groups (array order = display order). See ObjectFieldGroupSchema. |
| **tenancy** | `{ enabled: boolean; tenantField?: string }` | optional | Multi-tenancy configuration for SaaS applications |
| **access** | `{ default?: Enum<'public' \| 'private'> }` | optional | [ADR-0066 D2] Object exposure posture (public-by-default vs private secure-by-default). |
Expand Down Expand Up @@ -196,7 +196,7 @@ const result = ApiMethod.parse(data);
| **pluralLabel** | `string` | optional | Override plural label for the extended object |
| **description** | `string` | optional | Override description for the extended object |
| **validations** | `any[]` | optional | Additional validation rules to merge into the target object |
| **indexes** | `{ name?: string; fields: string[]; type?: Enum<'btree' \| 'hash' \| 'gin' \| 'gist' \| 'fulltext'>; unique?: boolean \| 'global' \| 'organization'; … }[]` | optional | Additional indexes to merge into the target object |
| **indexes** | `{ name?: string; fields: string[]; unique?: boolean \| 'global' \| 'organization'; type?: any; … }[]` | optional | Additional indexes to merge into the target object |
| **priority** | `integer` | optional | Merge priority (higher = applied later) |


Expand Down
3 changes: 3 additions & 0 deletions docs/protocol-upgrade-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,8 @@ The same is true of the protocol-17 retirement that closes this list, and the pa

The last enforce-or-remove entry of this step is on the RUNTIME context rather than on anything authorable: `HookContext.session.roles` (#5050). It was declared in `data/hook.zod.ts`, read by exactly two consumers — the approvals record lock and the delegation write guard, each opening with `session.roles?.includes('admin')` — and produced by nobody on the hook path: ObjectQL's `buildSession()` writes the session field by field (`userId`, `organizationId`, `accessToken`, `isSystem`, `actor`, the skip flags) and has no `roles` write, here or in `cloud`, whose hook consumers read `hookContext?.session?.userId` and nothing else (an ACTION body's `ctx.session` is a different untyped object that does carry one, tracked apart). So both branches were dead on every real engine path: an authorization decision in shape only, and — worse for a reader — a SECOND admin dialect competing with the one ADR-0095 D3 sanctions. #4839 (PR #5049) deleted the two readers on the maintainer's ruling; this step removes the declaration that outlived them, which is what ADR-0049 asks for once a key has neither end. Nothing observable changes: a key nobody wrote and nothing read cannot alter a single decision. It is tombstoned rather than deleted because `HookContextSchema` is deliberately NOT `.strict()` (strictness there would make an engine-internal enrichment a breaking change for anyone parsing a context they were handed, as `provenance` was in #3712), so a plain delete would strip the key in silence — the #3733 / ADR-0104 failure this whole pass exists to end. There is NO conversion and no source rewrite: a HookContext is built per operation by the engine and never stored, so no `sys_metadata` row, example or template can carry the key — the `openApi31` / `activationEvents` shape, one semantic TODO for hook authors. The live vocabulary is untouched and deliberately elsewhere: gate on `session.userId` / `session.isSystem` in the hook, and judge PRIVILEGE through the security service, which reads capability grants (`permissions`), placements (`positions`) and the derived posture off the execution context.

Finally it retires the two inert `IndexSchema` keys, `indexes[].type` and `indexes[].partial` (#5248, #4943). Neither ever had a DDL consumer: `SqlDriver.syncDeclaredIndexes` creates declared indexes through knex's `table.index()` / `table.unique()`, and the drift differ's `DeclaredIndexInput` carries only `name`/`fields`/`unique`/`nullSafeColumns` — so an authored `type` selected no access method and an authored `partial` produced a FULL index with its predicate discarded. `partial` was the more damaging of the two because it read as a correctness control: the platform's own `sys_metadata` declared it for overlay uniqueness, and what the declaration alone materialized was an unrestricted unique index (the active-row scoping is delivered by a runtime migration, `metadata-protocol`'s `ensureOverlayIndex`, not by the key). `type` was the louder: its `.default('btree')` put an inert knob into every parse output, so it read as live configuration — the ADR-0078 no-silently-inert shape. Remove was chosen over enforce (maintainer ruling, 2026-08-06): enforcing needs per-dialect algorithm mapping (`gin`/`gist` Postgres-only, `fulltext` MySQL-family), raw-SQL `CREATE INDEX … WHERE` on the dialects that have partial indexes at all (MySQL does not), and a redesign of how `isSyncReproducibleIndex` excludes partial indexes from incremental sync — design cost for a capability nothing has asked for. Both are lossless deletes: no DDL changes, because no DDL ever depended on them. Drift detection is untouched — the `partial` flag it consumes is parsed back out of the database's OWN `CREATE INDEX` DDL and never came from this key.

### Mechanical (applied for you)

| Conversion | Surface | Change | Load window |
Expand Down Expand Up @@ -255,6 +257,7 @@ The last enforce-or-remove entry of this step is on the RUNTIME context rather t
| `connector-rate-limit-config-removed` | `connector.rateLimitConfig` | connector key 'rateLimitConfig' removed (#4911 — no outbound rate-limiting engine exists; the runtime's only token bucket limits INBOUND requests, so every knob here was inert while reading like a configured cap. The whole ConnectorRateLimitConfig shape went with it) | retired — `migrate meta` only |
| `theme-inert-token-scales-removed` | `theme.typography.fontSize / theme.typography.fontWeight / theme.typography.lineHeight / theme.typography.letterSpacing / theme.typography.fontFamily.heading / theme.typography.fontFamily.mono / theme.animation / theme.zIndex` | theme keys 'typography.fontSize'/'fontWeight'/'lineHeight'/'letterSpacing', 'typography.fontFamily.heading'/'mono', 'animation' and 'zIndex' removed (#5021, ADR-0049 — the engine emitted --font-size-*, --font-weight-*, --line-height-*, --letter-spacing-*, --duration-*, --timing-*, --z-*, --font-heading and --font-mono faithfully, and no first-party component or stylesheet has ever read one. Re-declare any variable you actually consume under customVars, which emits it verbatim) | retired — `migrate meta` only |
| `page-header-subtitle-alias` | `page.component.page-header.description` | page-header component prop 'description' → 'subtitle' (objectui#3226 — the `subtitle ?? description` fallback retires) | live — protocol 17 loader accepts the old shape |
| `object-index-type-partial-removed` | `object.indexes[].type / object.indexes[].partial` | object index keys 'indexes[].type'/'indexes[].partial' removed (#5248, #4943 — no driver ever read either: the index method is the dialect's choice and a partial index is built by a database-layer migration, not declared) | retired — `migrate meta` only |

### Semantic (delegated to you, with acceptance criteria)

Expand Down
Loading
Loading