diff --git a/.changeset/24690-stored-expression-dropped-fields.md b/.changeset/24690-stored-expression-dropped-fields.md new file mode 100644 index 00000000000..b7cbdf75d83 --- /dev/null +++ b/.changeset/24690-stored-expression-dropped-fields.md @@ -0,0 +1,18 @@ +--- +'@objectstack/spec': minor +'@objectstack/objectql': minor +'@objectstack/service-automation': patch +--- + +feat(objectql): a caller value that a stored field's `expression` replaced is reported through `onFieldsDropped` / `droppedFields` under the new `DroppedFieldsEvent.reason` value `stored_expression` + +Clause-②: yes (widening) + +The engine computes a stored (non-`formula`) field's declared `expression` on every write, so a value the caller sends for such a field never stands. Until now nothing said so. `onFieldsDropped` stayed silent, so the PATCH and `/batch` doors answered with no `droppedFields`, and a flow's `update_record` reported success over a row that holds the computed value instead of the one sent. + +- **New `reason` value.** `DroppedFieldsEventSchema.reason` (`@objectstack/spec/data`) gains `stored_expression`. It means the field is a stored column the engine computes from its `expression`, and the caller's value was replaced by the computed one. The field IS written, so `computed` (a `formula` with no column, whose key is dropped) would be wrong for it. The four existing values are unchanged. +- **When it is reported.** On insert (single, batch, and `insertMany`'s per-row `droppedFields`), on update by id, and on a predicate update, in every context, `isSystem` included. It is reported only when the caller sent the field and the value stored differs from the caller's value. A client that sends the computed value (the console's line-item grid) or no value is told nothing. A value another strip already took, such as a stored expression field that is also `readonly`, is reported once, under that strip's reason. +- **`strictReadonlyWrites` is unchanged.** The maintainer's ruling 「甲」 says only the loud half ships in 18.x. A strict write whose value for such a field differs from the computed one is NOT refused: it completes with the computed value, and the replacement is reported through `onFieldsDropped` under strict as well. A refusal is runtime behaviour that no conversion can carry, so the strict refusal of a `stored_expression` replacement is deferred to 19.0. A strict write that also carries a refusable drop (`readonly`, `readonly_when`, `primary_key`, `computed`) is refused as before, for those drops only, and fires no listener. +- **Flow step warning.** `@objectstack/service-automation` words the new reason. Its warning says the requested value(s) were not stored, rather than that the field was not written, and its closing sentence says the write succeeded with the computed value(s). + +**For consumers that branch on `reason`.** A `Record` table fails its typecheck until it has a `stored_expression` row. That is the intended signal: add the row. A client running behind the server should treat a reason it does not recognise as "not applied", never as one of the known reasons. diff --git a/content/docs/references/api/batch.mdx b/content/docs/references/api/batch.mdx index a560df5d36a..04fc0c6f77b 100644 --- a/content/docs/references/api/batch.mdx +++ b/content/docs/references/api/batch.mdx @@ -68,7 +68,7 @@ const result = BatchConfigSchema.parse(data); | **errors** | `{ code: Enum<'VALIDATION_ERROR' \| 'INVALID_FIELD' \| 'MISSING_REQUIRED_FIELD' \| …>; declaredCode?: string; message: string; userMessage?: string; … }[]` | optional | Array of errors if operation failed. Branch on `errors[0].code` — an atomic batch that rolled back marks rows that were written then undone with code ROLLED_BACK and rows never reached with NOT_ATTEMPTED, while the causal row keeps its own error. A NON-atomic batch that stopped (the `continueOnError: false` default) marks its un-attempted tail with the same NOT_ATTEMPTED code — rows before the failure stay written and keep reporting success, since nothing was rolled back. | | **data** | `Record` | optional | Full record data (if returnRecords=true) | | **index** | `number` | optional | Index of the record in the request array | -| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed'> }[]` | optional | Write-observability: caller-supplied fields LEGALLY stripped from THIS row before it was written — static `readonly` / TRUE `readonlyWhen` on update, or the in-engine static `readonly` strip on create. Per-row because a batch can drop different fields on different rows (`readonlyWhen` is record-state-dependent). Present ONLY when ≥1 field was dropped for this row; the row still succeeded (success unchanged). A single response header cannot express per-row drops, so this body field is the canonical bulk channel — REST does not emit `X-ObjectStack-Dropped-Fields` for batches. Optional — omit-when-empty keeps the shape backward-compatible. | +| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed' \| 'stored_expression'> }[]` | optional | Write-observability: caller-supplied fields LEGALLY stripped from THIS row before it was written — static `readonly` / TRUE `readonlyWhen` on update, or the in-engine static `readonly` strip on create. Per-row because a batch can drop different fields on different rows (`readonlyWhen` is record-state-dependent). Present ONLY when ≥1 field was dropped for this row; the row still succeeded (success unchanged). A single response header cannot express per-row drops, so this body field is the canonical bulk channel — REST does not emit `X-ObjectStack-Dropped-Fields` for batches. Optional — omit-when-empty keeps the shape backward-compatible. | ### Nested Shape: `BatchOperationResult.errors[number]` @@ -86,13 +86,13 @@ const result = BatchConfigSchema.parse(data); ### Nested Shape: `BatchOperationResult.droppedFields[number]` -A write-path strip event: caller-supplied fields legally dropped from the payload +A write-path strip event: caller-supplied fields legally dropped from the payload, or written with the engine's computed value in place of the caller's | Property | Type | Required | Description | | :--- | :--- | :--- | :--- | | **object** | `string` | ✅ | Object the write targeted (resolved object name) | -| **fields** | `string[]` | ✅ | Caller-supplied field names the engine removed from the write payload | -| **reason** | `Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed'>` | ✅ | Why the fields were dropped: static readonly, a TRUE readonlyWhen predicate, the primary-key strip of a payload id the engine ruled is not an identifier, or a computed (formula) field no driver has a column for | +| **fields** | `string[]` | ✅ | Caller-supplied field names whose value the engine did not store: removed from the write payload, or (`stored_expression`) written with the computed value in place of the caller's | +| **reason** | `Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed' \| 'stored_expression'>` | ✅ | Why the fields were dropped: static readonly, a TRUE readonlyWhen predicate, the primary-key strip of a payload id the engine ruled is not an identifier, a computed (formula) field no driver has a column for, or a stored field whose declared expression the engine computes on every write, so the caller's different value was replaced by the computed one | --- @@ -213,7 +213,7 @@ A write-path strip event: caller-supplied fields legally dropped from the payloa | **errors** | `{ code: Enum<'VALIDATION_ERROR' \| 'INVALID_FIELD' \| 'MISSING_REQUIRED_FIELD' \| …>; declaredCode?: string; message: string; userMessage?: string; … }[]` | optional | Array of errors if operation failed. Branch on `errors[0].code` — an atomic batch that rolled back marks rows that were written then undone with code ROLLED_BACK and rows never reached with NOT_ATTEMPTED, while the causal row keeps its own error. A NON-atomic batch that stopped (the `continueOnError: false` default) marks its un-attempted tail with the same NOT_ATTEMPTED code — rows before the failure stay written and keep reporting success, since nothing was rolled back. | | **data** | `Record` | optional | Full record data (if returnRecords=true) | | **index** | `number` | optional | Index of the record in the request array | -| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed'> }[]` | optional | Write-observability: caller-supplied fields LEGALLY stripped from THIS row before it was written — static `readonly` / TRUE `readonlyWhen` on update, or the in-engine static `readonly` strip on create. Per-row because a batch can drop different fields on different rows (`readonlyWhen` is record-state-dependent). Present ONLY when ≥1 field was dropped for this row; the row still succeeded (success unchanged). A single response header cannot express per-row drops, so this body field is the canonical bulk channel — REST does not emit `X-ObjectStack-Dropped-Fields` for batches. Optional — omit-when-empty keeps the shape backward-compatible. | +| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed' \| 'stored_expression'> }[]` | optional | Write-observability: caller-supplied fields LEGALLY stripped from THIS row before it was written — static `readonly` / TRUE `readonlyWhen` on update, or the in-engine static `readonly` strip on create. Per-row because a batch can drop different fields on different rows (`readonlyWhen` is record-state-dependent). Present ONLY when ≥1 field was dropped for this row; the row still succeeded (success unchanged). A single response header cannot express per-row drops, so this body field is the canonical bulk channel — REST does not emit `X-ObjectStack-Dropped-Fields` for batches. Optional — omit-when-empty keeps the shape backward-compatible. | --- @@ -227,8 +227,8 @@ A cross-object batch strip event: dropped fields plus the operation index | Property | Type | Required | Description | | :--- | :--- | :--- | :--- | | **object** | `string` | ✅ | Object the write targeted (resolved object name) | -| **fields** | `string[]` | ✅ | Caller-supplied field names the engine removed from the write payload | -| **reason** | `Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed'>` | ✅ | Why the fields were dropped: static readonly, a TRUE readonlyWhen predicate, the primary-key strip of a payload id the engine ruled is not an identifier, or a computed (formula) field no driver has a column for | +| **fields** | `string[]` | ✅ | Caller-supplied field names whose value the engine did not store: removed from the write payload, or (`stored_expression`) written with the computed value in place of the caller's | +| **reason** | `Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed' \| 'stored_expression'>` | ✅ | Why the fields were dropped: static readonly, a TRUE readonlyWhen predicate, the primary-key strip of a payload id the engine ruled is not an identifier, a computed (formula) field no driver has a column for, or a stored field whose declared expression the engine computes on every write, so the caller's different value was replaced by the computed one | | **index** | `integer` | ✅ | Index of the operation in the request `operations` array | @@ -276,7 +276,7 @@ A cross-object batch strip event: dropped fields plus the operation index | Property | Type | Required | Description | | :--- | :--- | :--- | :--- | | **results** | `any[]` | ✅ | Per-operation result, index-aligned with the request operations | -| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed'>; index: integer }[]` | optional | Write-observability: caller-supplied fields the engine LEGALLY stripped from an operation before it was written — static `readonly` or a TRUE `readonlyWhen` predicate. This endpoint is the console record form's save path (master-detail writes parent + children in one transaction), so without it the ONE surface where a user edits a `readonlyWhen` field reported plain success while the value never landed. Each event carries the `index` of its operation. Present ONLY when ≥1 field was dropped; the batch still committed without them (results/success semantics unchanged). Optional — omit-when-empty keeps the shape backward-compatible. | +| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed' \| 'stored_expression'>; index: integer }[]` | optional | Write-observability: caller-supplied fields the engine LEGALLY stripped from an operation before it was written — static `readonly` or a TRUE `readonlyWhen` predicate. This endpoint is the console record form's save path (master-detail writes parent + children in one transaction), so without it the ONE surface where a user edits a `readonlyWhen` field reported plain success while the value never landed. Each event carries the `index` of its operation. Present ONLY when ≥1 field was dropped; the batch still committed without them (results/success semantics unchanged). Optional — omit-when-empty keeps the shape backward-compatible. | ### Nested Shape: `CrossObjectBatchResponse.droppedFields[number]` @@ -285,8 +285,8 @@ A cross-object batch strip event: dropped fields plus the operation index | Property | Type | Required | Description | | :--- | :--- | :--- | :--- | | **object** | `string` | ✅ | Object the write targeted (resolved object name) | -| **fields** | `string[]` | ✅ | Caller-supplied field names the engine removed from the write payload | -| **reason** | `Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed'>` | ✅ | Why the fields were dropped: static readonly, a TRUE readonlyWhen predicate, the primary-key strip of a payload id the engine ruled is not an identifier, or a computed (formula) field no driver has a column for | +| **fields** | `string[]` | ✅ | Caller-supplied field names whose value the engine did not store: removed from the write payload, or (`stored_expression`) written with the computed value in place of the caller's | +| **reason** | `Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed' \| 'stored_expression'>` | ✅ | Why the fields were dropped: static readonly, a TRUE readonlyWhen predicate, the primary-key strip of a payload id the engine ruled is not an identifier, a computed (formula) field no driver has a column for, or a stored field whose declared expression the engine computes on every write, so the caller's different value was replaced by the computed one | | **index** | `integer` | ✅ | Index of the operation in the request `operations` array | diff --git a/content/docs/references/api/export.mdx b/content/docs/references/api/export.mdx index 6f176041c86..ca8a06d964c 100644 --- a/content/docs/references/api/export.mdx +++ b/content/docs/references/api/export.mdx @@ -230,7 +230,7 @@ const result = CreateImportJobRequestSchema.parse(data); | **code** | `string` | optional | Error code (failed rows) | | **error** | `string` | optional | Human-readable error message (failed rows) | | **warnings** | `{ field: string; code: string; message: string; rule?: string; … }[]` | optional | Findings this deployment ADMITS rather than rejects, copied verbatim from the validate verdict's row `warnings`: ADR-0104 value shapes under a warn-first posture, then the hits of advisory validation rules (`severity: 'warning'` or `'info'`, each naming its `rule` and `severity`). Set only on a dry-run row the verdict accepted: the row is `ok` because the write would store it. A committed row does not carry it: the import does not copy a write answer's `warnings` onto the row. | -| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed'> }[]` | optional | Write-observability: caller-supplied fields the engine LEGALLY strips from THIS row, one event per reason, in the engine's own `droppedFields` shape and reason vocabulary. A committed row reports the strips its write made. A dry-run row reports the strips its preview runs, which for a row the import would update excludes `readonlyWhen` and primary-key strips, so that row can name fewer fields than its commit. The row still succeeds: `ok` and `action` are unchanged. Present only when at least one field was dropped. A server or write path that does not produce this report omits the key too, so an absent key alone does not prove nothing was dropped. In an async job's `results`, an ok row's drops reach the reader only if the row falls inside that capped, failures-first sample. | +| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed' \| 'stored_expression'> }[]` | optional | Write-observability: caller-supplied fields the engine LEGALLY strips from THIS row, one event per reason, in the engine's own `droppedFields` shape and reason vocabulary. A committed row reports the strips its write made. A dry-run row reports the strips its preview runs, which for a row the import would update excludes `readonlyWhen` and primary-key strips, so that row can name fewer fields than its commit. The row still succeeds: `ok` and `action` are unchanged. Present only when at least one field was dropped. A server or write path that does not produce this report omits the key too, so an absent key alone does not prove nothing was dropped. In an async job's `results`, an ok row's drops reach the reader only if the row falls inside that capped, failures-first sample. | --- @@ -358,7 +358,7 @@ Type: `{ sourceField: string; targetField: string; targetLabel?: string; transfo | **code** | `string` | optional | Error code (failed rows) | | **error** | `string` | optional | Human-readable error message (failed rows) | | **warnings** | `{ field: string; code: string; message: string; rule?: string; … }[]` | optional | Findings this deployment ADMITS rather than rejects, copied verbatim from the validate verdict's row `warnings`: ADR-0104 value shapes under a warn-first posture, then the hits of advisory validation rules (`severity: 'warning'` or `'info'`, each naming its `rule` and `severity`). Set only on a dry-run row the verdict accepted: the row is `ok` because the write would store it. A committed row does not carry it: the import does not copy a write answer's `warnings` onto the row. | -| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed'> }[]` | optional | Write-observability: caller-supplied fields the engine LEGALLY strips from THIS row, one event per reason, in the engine's own `droppedFields` shape and reason vocabulary. A committed row reports the strips its write made. A dry-run row reports the strips its preview runs, which for a row the import would update excludes `readonlyWhen` and primary-key strips, so that row can name fewer fields than its commit. The row still succeeds: `ok` and `action` are unchanged. Present only when at least one field was dropped. A server or write path that does not produce this report omits the key too, so an absent key alone does not prove nothing was dropped. In an async job's `results`, an ok row's drops reach the reader only if the row falls inside that capped, failures-first sample. | +| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed' \| 'stored_expression'> }[]` | optional | Write-observability: caller-supplied fields the engine LEGALLY strips from THIS row, one event per reason, in the engine's own `droppedFields` shape and reason vocabulary. A committed row reports the strips its write made. A dry-run row reports the strips its preview runs, which for a row the import would update excludes `readonlyWhen` and primary-key strips, so that row can name fewer fields than its commit. The row still succeeds: `ok` and `action` are unchanged. Present only when at least one field was dropped. A server or write path that does not produce this report omits the key too, so an absent key alone does not prove nothing was dropped. In an async job's `results`, an ok row's drops reach the reader only if the row falls inside that capped, failures-first sample. | --- @@ -377,7 +377,7 @@ Type: `{ sourceField: string; targetField: string; targetLabel?: string; transfo | **code** | `string` | optional | Error code (failed rows) | | **error** | `string` | optional | Human-readable error message (failed rows) | | **warnings** | `{ field: string; code: string; message: string; rule?: string; … }[]` | optional | Findings this deployment ADMITS rather than rejects, copied verbatim from the validate verdict's row `warnings`: ADR-0104 value shapes under a warn-first posture, then the hits of advisory validation rules (`severity: 'warning'` or `'info'`, each naming its `rule` and `severity`). Set only on a dry-run row the verdict accepted: the row is `ok` because the write would store it. A committed row does not carry it: the import does not copy a write answer's `warnings` onto the row. | -| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed'> }[]` | optional | Write-observability: caller-supplied fields the engine LEGALLY strips from THIS row, one event per reason, in the engine's own `droppedFields` shape and reason vocabulary. A committed row reports the strips its write made. A dry-run row reports the strips its preview runs, which for a row the import would update excludes `readonlyWhen` and primary-key strips, so that row can name fewer fields than its commit. The row still succeeds: `ok` and `action` are unchanged. Present only when at least one field was dropped. A server or write path that does not produce this report omits the key too, so an absent key alone does not prove nothing was dropped. In an async job's `results`, an ok row's drops reach the reader only if the row falls inside that capped, failures-first sample. | +| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed' \| 'stored_expression'> }[]` | optional | Write-observability: caller-supplied fields the engine LEGALLY strips from THIS row, one event per reason, in the engine's own `droppedFields` shape and reason vocabulary. A committed row reports the strips its write made. A dry-run row reports the strips its preview runs, which for a row the import would update excludes `readonlyWhen` and primary-key strips, so that row can name fewer fields than its commit. The row still succeeds: `ok` and `action` are unchanged. Present only when at least one field was dropped. A server or write path that does not produce this report omits the key too, so an absent key alone does not prove nothing was dropped. In an async job's `results`, an ok row's drops reach the reader only if the row falls inside that capped, failures-first sample. | ### Nested Shape: `ImportRowResult.warnings[number]` @@ -391,13 +391,13 @@ Type: `{ sourceField: string; targetField: string; targetLabel?: string; transfo ### Nested Shape: `ImportRowResult.droppedFields[number]` -A write-path strip event: caller-supplied fields legally dropped from the payload +A write-path strip event: caller-supplied fields legally dropped from the payload, or written with the engine's computed value in place of the caller's | Property | Type | Required | Description | | :--- | :--- | :--- | :--- | | **object** | `string` | ✅ | Object the write targeted (resolved object name) | -| **fields** | `string[]` | ✅ | Caller-supplied field names the engine removed from the write payload | -| **reason** | `Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed'>` | ✅ | Why the fields were dropped: static readonly, a TRUE readonlyWhen predicate, the primary-key strip of a payload id the engine ruled is not an identifier, or a computed (formula) field no driver has a column for | +| **fields** | `string[]` | ✅ | Caller-supplied field names whose value the engine did not store: removed from the write payload, or (`stored_expression`) written with the computed value in place of the caller's | +| **reason** | `Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed' \| 'stored_expression'>` | ✅ | Why the fields were dropped: static readonly, a TRUE readonlyWhen predicate, the primary-key strip of a payload id the engine ruled is not an identifier, a computed (formula) field no driver has a column for, or a stored field whose declared expression the engine computes on every write, so the caller's different value was replaced by the computed one | --- diff --git a/content/docs/references/api/protocol.mdx b/content/docs/references/api/protocol.mdx index 893a5685b52..c5adafbc9e5 100644 --- a/content/docs/references/api/protocol.mdx +++ b/content/docs/references/api/protocol.mdx @@ -453,7 +453,7 @@ Canonical cross-paradigm action/node descriptor (ADR-0018) | **errors** | `{ code: Enum<'VALIDATION_ERROR' \| 'INVALID_FIELD' \| 'MISSING_REQUIRED_FIELD' \| …>; declaredCode?: string; message: string; userMessage?: string; … }[]` | optional | Array of errors if operation failed. Branch on `errors[0].code` — an atomic batch that rolled back marks rows that were written then undone with code ROLLED_BACK and rows never reached with NOT_ATTEMPTED, while the causal row keeps its own error. A NON-atomic batch that stopped (the `continueOnError: false` default) marks its un-attempted tail with the same NOT_ATTEMPTED code — rows before the failure stay written and keep reporting success, since nothing was rolled back. | | **data** | `Record` | optional | Full record data (if returnRecords=true) | | **index** | `number` | optional | Index of the record in the request array | -| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed'> }[]` | optional | Write-observability: caller-supplied fields LEGALLY stripped from THIS row before it was written — static `readonly` / TRUE `readonlyWhen` on update, or the in-engine static `readonly` strip on create. Per-row because a batch can drop different fields on different rows (`readonlyWhen` is record-state-dependent). Present ONLY when ≥1 field was dropped for this row; the row still succeeded (success unchanged). A single response header cannot express per-row drops, so this body field is the canonical bulk channel — REST does not emit `X-ObjectStack-Dropped-Fields` for batches. Optional — omit-when-empty keeps the shape backward-compatible. | +| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed' \| 'stored_expression'> }[]` | optional | Write-observability: caller-supplied fields LEGALLY stripped from THIS row before it was written — static `readonly` / TRUE `readonlyWhen` on update, or the in-engine static `readonly` strip on create. Per-row because a batch can drop different fields on different rows (`readonlyWhen` is record-state-dependent). Present ONLY when ≥1 field was dropped for this row; the row still succeeded (success unchanged). A single response header cannot express per-row drops, so this body field is the canonical bulk channel — REST does not emit `X-ObjectStack-Dropped-Fields` for batches. Optional — omit-when-empty keeps the shape backward-compatible. | --- @@ -494,18 +494,18 @@ Canonical cross-paradigm action/node descriptor (ADR-0018) | **id** | `string` | ✅ | The ID of the newly created clone. | | **sourceId** | `string` | ✅ | The ID of the record the clone was copied from. | | **record** | `Record` | ✅ | The created clone, including server-generated fields. Engine-owned values (injected system/audit columns, autonumbers, computed formula/summary fields) are re-derived by the insert path rather than copied from the source; caller-supplied `overrides` win over copied values. | -| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed'> }[]` | optional | Write-observability: fields that were LEGALLY stripped before the clone was written — a non-system clone cannot seed a static `readonly` column, whether the value was COPIED from the source row or supplied through `overrides` (the strip runs inside `engine.insert`, after the `beforeInsert` hooks, `isSystem`-gated, exactly as on `createData`), so those keys are dropped and the field re-derives its default. Present ONLY when ≥1 field was dropped; the clone still succeeded without them (status/success semantics unchanged). Carried in the 201 body only — this route relays the producer verbatim and sets no `X-ObjectStack-Dropped-Fields` header. Optional — omit-when-empty keeps the shape backward-compatible for existing clients. | +| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed' \| 'stored_expression'> }[]` | optional | Write-observability: fields that were LEGALLY stripped before the clone was written — a non-system clone cannot seed a static `readonly` column, whether the value was COPIED from the source row or supplied through `overrides` (the strip runs inside `engine.insert`, after the `beforeInsert` hooks, `isSystem`-gated, exactly as on `createData`), so those keys are dropped and the field re-derives its default. Present ONLY when ≥1 field was dropped; the clone still succeeded without them (status/success semantics unchanged). Carried in the 201 body only — this route relays the producer verbatim and sets no `X-ObjectStack-Dropped-Fields` header. Optional — omit-when-empty keeps the shape backward-compatible for existing clients. | | **warnings** | `{ field: string; code: string; message: string; rule?: string; … }[]` | optional | The advisory validation-rule hits of THIS write: each `severity: 'warning'` or `'info'` rule whose verdict on the written record was "violated", with its `rule` name, `severity`, `field`, `code` and author-written `message` — the same element the validate-only preview reports in a row's `warnings`. Advisory rules never block, so the write succeeded; this is the advisory to show the person who made it. Present ONLY when at least one advisory rule hit; a write with no hit omits the key. Hits of a nested write a hook or a flow made inside this one are not included, and a rule that could not be evaluated is not a hit. | ### Nested Shape: `CloneDataResponse.droppedFields[number]` -A write-path strip event: caller-supplied fields legally dropped from the payload +A write-path strip event: caller-supplied fields legally dropped from the payload, or written with the engine's computed value in place of the caller's | Property | Type | Required | Description | | :--- | :--- | :--- | :--- | | **object** | `string` | ✅ | Object the write targeted (resolved object name) | -| **fields** | `string[]` | ✅ | Caller-supplied field names the engine removed from the write payload | -| **reason** | `Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed'>` | ✅ | Why the fields were dropped: static readonly, a TRUE readonlyWhen predicate, the primary-key strip of a payload id the engine ruled is not an identifier, or a computed (formula) field no driver has a column for | +| **fields** | `string[]` | ✅ | Caller-supplied field names whose value the engine did not store: removed from the write payload, or (`stored_expression`) written with the computed value in place of the caller's | +| **reason** | `Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed' \| 'stored_expression'>` | ✅ | Why the fields were dropped: static readonly, a TRUE readonlyWhen predicate, the primary-key strip of a payload id the engine ruled is not an identifier, a computed (formula) field no driver has a column for, or a stored field whose declared expression the engine computes on every write, so the caller's different value was replaced by the computed one | ### Nested Shape: `CloneDataResponse.warnings[number]` @@ -554,18 +554,18 @@ A write-path strip event: caller-supplied fields legally dropped from the payloa | **object** | `string` | ✅ | The object name. | | **id** | `string` | ✅ | The ID of the newly created record. | | **record** | `Record` | ✅ | The created record, including server-generated fields (created_at, owner). | -| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed'> }[]` | optional | Write-observability: caller-supplied fields that were LEGALLY stripped before the record was written — a non-system create cannot seed a static `readonly` column (the strip runs inside `engine.insert`, after the `beforeInsert` hooks, `isSystem`-gated), so those keys are dropped and the field re-derives its default. Present ONLY when ≥1 field was dropped; the create still succeeded without them (status/success semantics unchanged). REST additionally surfaces this as the `X-ObjectStack-Dropped-Fields` response header. Optional — omit-when-empty keeps the shape backward-compatible for existing clients. | +| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed' \| 'stored_expression'> }[]` | optional | Write-observability: caller-supplied fields that were LEGALLY stripped before the record was written — a non-system create cannot seed a static `readonly` column (the strip runs inside `engine.insert`, after the `beforeInsert` hooks, `isSystem`-gated), so those keys are dropped and the field re-derives its default. Present ONLY when ≥1 field was dropped; the create still succeeded without them (status/success semantics unchanged). REST additionally surfaces this as the `X-ObjectStack-Dropped-Fields` response header. Optional — omit-when-empty keeps the shape backward-compatible for existing clients. | | **warnings** | `{ field: string; code: string; message: string; rule?: string; … }[]` | optional | The advisory validation-rule hits of THIS write: each `severity: 'warning'` or `'info'` rule whose verdict on the written record was "violated", with its `rule` name, `severity`, `field`, `code` and author-written `message` — the same element the validate-only preview reports in a row's `warnings`. Advisory rules never block, so the write succeeded; this is the advisory to show the person who made it. Present ONLY when at least one advisory rule hit; a write with no hit omits the key. Hits of a nested write a hook or a flow made inside this one are not included, and a rule that could not be evaluated is not a hit. | ### Nested Shape: `CreateDataResponse.droppedFields[number]` -A write-path strip event: caller-supplied fields legally dropped from the payload +A write-path strip event: caller-supplied fields legally dropped from the payload, or written with the engine's computed value in place of the caller's | Property | Type | Required | Description | | :--- | :--- | :--- | :--- | | **object** | `string` | ✅ | Object the write targeted (resolved object name) | -| **fields** | `string[]` | ✅ | Caller-supplied field names the engine removed from the write payload | -| **reason** | `Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed'>` | ✅ | Why the fields were dropped: static readonly, a TRUE readonlyWhen predicate, the primary-key strip of a payload id the engine ruled is not an identifier, or a computed (formula) field no driver has a column for | +| **fields** | `string[]` | ✅ | Caller-supplied field names whose value the engine did not store: removed from the write payload, or (`stored_expression`) written with the computed value in place of the caller's | +| **reason** | `Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed' \| 'stored_expression'>` | ✅ | Why the fields were dropped: static readonly, a TRUE readonlyWhen predicate, the primary-key strip of a payload id the engine ruled is not an identifier, a computed (formula) field no driver has a column for, or a stored field whose declared expression the engine computes on every write, so the caller's different value was replaced by the computed one | ### Nested Shape: `CreateDataResponse.warnings[number]` @@ -601,17 +601,17 @@ A write-path strip event: caller-supplied fields legally dropped from the payloa | **object** | `string` | ✅ | Object name | | **records** | `Record[]` | ✅ | Created records | | **count** | `number` | ✅ | Number of records created | -| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed'> }[]` | optional | Write-observability: caller-supplied `readonly` fields the in-engine create-side strip (`engine.insert`, `isSystem`-gated) removed before the rows were written. AGGREGATED across the batch (one event per object/reason with the UNION of dropped field names) rather than per-row, because this response is `{ object, records, count }` and has no per-row slot to hang a drop set on — a union is the only view it can represent. So read a name here as "at least one row dropped this field", NOT "every row dropped the same set": the strip runs INSIDE `engine.insert` after the `beforeInsert` hooks and exempts keys a hook itself wrote, tracked per row: rows where a hook stamped a protected key drop a different set from rows where it did not. Present ONLY when ≥1 field was dropped; the creates still succeeded without them (count/success unchanged). Optional — omit-when-empty keeps the shape backward-compatible. (The same reading applies to the partial-success bulk create, whose batch-level `droppedFields` names no row for the same reason. The paths that DO carry per-row `droppedFields` on each result are the bulk UPDATE and the mixed batch, where each row is its own engine call — see BatchOperationResultSchema.) | +| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed' \| 'stored_expression'> }[]` | optional | Write-observability: caller-supplied `readonly` fields the in-engine create-side strip (`engine.insert`, `isSystem`-gated) removed before the rows were written. AGGREGATED across the batch (one event per object/reason with the UNION of dropped field names) rather than per-row, because this response is `{ object, records, count }` and has no per-row slot to hang a drop set on — a union is the only view it can represent. So read a name here as "at least one row dropped this field", NOT "every row dropped the same set": the strip runs INSIDE `engine.insert` after the `beforeInsert` hooks and exempts keys a hook itself wrote, tracked per row: rows where a hook stamped a protected key drop a different set from rows where it did not. Present ONLY when ≥1 field was dropped; the creates still succeeded without them (count/success unchanged). Optional — omit-when-empty keeps the shape backward-compatible. (The same reading applies to the partial-success bulk create, whose batch-level `droppedFields` names no row for the same reason. The paths that DO carry per-row `droppedFields` on each result are the bulk UPDATE and the mixed batch, where each row is its own engine call — see BatchOperationResultSchema.) | ### Nested Shape: `CreateManyDataResponse.droppedFields[number]` -A write-path strip event: caller-supplied fields legally dropped from the payload +A write-path strip event: caller-supplied fields legally dropped from the payload, or written with the engine's computed value in place of the caller's | Property | Type | Required | Description | | :--- | :--- | :--- | :--- | | **object** | `string` | ✅ | Object the write targeted (resolved object name) | -| **fields** | `string[]` | ✅ | Caller-supplied field names the engine removed from the write payload | -| **reason** | `Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed'>` | ✅ | Why the fields were dropped: static readonly, a TRUE readonlyWhen predicate, the primary-key strip of a payload id the engine ruled is not an identifier, or a computed (formula) field no driver has a column for | +| **fields** | `string[]` | ✅ | Caller-supplied field names whose value the engine did not store: removed from the write payload, or (`stored_expression`) written with the computed value in place of the caller's | +| **reason** | `Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed' \| 'stored_expression'>` | ✅ | Why the fields were dropped: static readonly, a TRUE readonlyWhen predicate, the primary-key strip of a payload id the engine ruled is not an identifier, a computed (formula) field no driver has a column for, or a stored field whose declared expression the engine computes on every write, so the caller's different value was replaced by the computed one | --- @@ -711,7 +711,7 @@ A write-path strip event: caller-supplied fields legally dropped from the payloa | **errors** | `{ code: Enum<'VALIDATION_ERROR' \| 'INVALID_FIELD' \| 'MISSING_REQUIRED_FIELD' \| …>; declaredCode?: string; message: string; userMessage?: string; … }[]` | optional | Array of errors if operation failed. Branch on `errors[0].code` — an atomic batch that rolled back marks rows that were written then undone with code ROLLED_BACK and rows never reached with NOT_ATTEMPTED, while the causal row keeps its own error. A NON-atomic batch that stopped (the `continueOnError: false` default) marks its un-attempted tail with the same NOT_ATTEMPTED code — rows before the failure stay written and keep reporting success, since nothing was rolled back. | | **data** | `Record` | optional | Full record data (if returnRecords=true) | | **index** | `number` | optional | Index of the record in the request array | -| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed'> }[]` | optional | Write-observability: caller-supplied fields LEGALLY stripped from THIS row before it was written — static `readonly` / TRUE `readonlyWhen` on update, or the in-engine static `readonly` strip on create. Per-row because a batch can drop different fields on different rows (`readonlyWhen` is record-state-dependent). Present ONLY when ≥1 field was dropped for this row; the row still succeeded (success unchanged). A single response header cannot express per-row drops, so this body field is the canonical bulk channel — REST does not emit `X-ObjectStack-Dropped-Fields` for batches. Optional — omit-when-empty keeps the shape backward-compatible. | +| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed' \| 'stored_expression'> }[]` | optional | Write-observability: caller-supplied fields LEGALLY stripped from THIS row before it was written — static `readonly` / TRUE `readonlyWhen` on update, or the in-engine static `readonly` strip on create. Per-row because a batch can drop different fields on different rows (`readonlyWhen` is record-state-dependent). Present ONLY when ≥1 field was dropped for this row; the row still succeeded (success unchanged). A single response header cannot express per-row drops, so this body field is the canonical bulk channel — REST does not emit `X-ObjectStack-Dropped-Fields` for batches. Optional — omit-when-empty keeps the shape backward-compatible. | --- @@ -3005,18 +3005,18 @@ Uninstall package response | **object** | `string` | ✅ | Object name | | **id** | `string` | ✅ | Updated record ID | | **record** | `Record` | ✅ | Updated record | -| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed'> }[]` | optional | Write-observability: caller-supplied fields the engine LEGALLY stripped from the write before persisting — static `readonly` or a TRUE `readonlyWhen` predicate. Present ONLY when ≥1 field was dropped; the update still succeeded without them (status/success semantics unchanged — stripping is legitimate, not an error). REST additionally surfaces this as the `X-ObjectStack-Dropped-Fields` response header. Optional — omit-when-empty keeps the shape backward-compatible for existing clients that only read `record`. | +| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed' \| 'stored_expression'> }[]` | optional | Write-observability: caller-supplied fields the engine LEGALLY stripped from the write before persisting — static `readonly` or a TRUE `readonlyWhen` predicate. Present ONLY when ≥1 field was dropped; the update still succeeded without them (status/success semantics unchanged — stripping is legitimate, not an error). REST additionally surfaces this as the `X-ObjectStack-Dropped-Fields` response header. Optional — omit-when-empty keeps the shape backward-compatible for existing clients that only read `record`. | | **warnings** | `{ field: string; code: string; message: string; rule?: string; … }[]` | optional | The advisory validation-rule hits of THIS write: each `severity: 'warning'` or `'info'` rule whose verdict on the written record was "violated", with its `rule` name, `severity`, `field`, `code` and author-written `message` — the same element the validate-only preview reports in a row's `warnings`. Advisory rules never block, so the write succeeded; this is the advisory to show the person who made it. Present ONLY when at least one advisory rule hit; a write with no hit omits the key. Hits of a nested write a hook or a flow made inside this one are not included, and a rule that could not be evaluated is not a hit. | ### Nested Shape: `UpdateDataResponse.droppedFields[number]` -A write-path strip event: caller-supplied fields legally dropped from the payload +A write-path strip event: caller-supplied fields legally dropped from the payload, or written with the engine's computed value in place of the caller's | Property | Type | Required | Description | | :--- | :--- | :--- | :--- | | **object** | `string` | ✅ | Object the write targeted (resolved object name) | -| **fields** | `string[]` | ✅ | Caller-supplied field names the engine removed from the write payload | -| **reason** | `Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed'>` | ✅ | Why the fields were dropped: static readonly, a TRUE readonlyWhen predicate, the primary-key strip of a payload id the engine ruled is not an identifier, or a computed (formula) field no driver has a column for | +| **fields** | `string[]` | ✅ | Caller-supplied field names whose value the engine did not store: removed from the write payload, or (`stored_expression`) written with the computed value in place of the caller's | +| **reason** | `Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed' \| 'stored_expression'>` | ✅ | Why the fields were dropped: static readonly, a TRUE readonlyWhen predicate, the primary-key strip of a payload id the engine ruled is not an identifier, a computed (formula) field no driver has a column for, or a stored field whose declared expression the engine computes on every write, so the caller's different value was replaced by the computed one | ### Nested Shape: `UpdateDataResponse.warnings[number]` @@ -3107,7 +3107,7 @@ A write-path strip event: caller-supplied fields legally dropped from the payloa | **errors** | `{ code: Enum<'VALIDATION_ERROR' \| 'INVALID_FIELD' \| 'MISSING_REQUIRED_FIELD' \| …>; declaredCode?: string; message: string; userMessage?: string; … }[]` | optional | Array of errors if operation failed. Branch on `errors[0].code` — an atomic batch that rolled back marks rows that were written then undone with code ROLLED_BACK and rows never reached with NOT_ATTEMPTED, while the causal row keeps its own error. A NON-atomic batch that stopped (the `continueOnError: false` default) marks its un-attempted tail with the same NOT_ATTEMPTED code — rows before the failure stay written and keep reporting success, since nothing was rolled back. | | **data** | `Record` | optional | Full record data (if returnRecords=true) | | **index** | `number` | optional | Index of the record in the request array | -| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed'> }[]` | optional | Write-observability: caller-supplied fields LEGALLY stripped from THIS row before it was written — static `readonly` / TRUE `readonlyWhen` on update, or the in-engine static `readonly` strip on create. Per-row because a batch can drop different fields on different rows (`readonlyWhen` is record-state-dependent). Present ONLY when ≥1 field was dropped for this row; the row still succeeded (success unchanged). A single response header cannot express per-row drops, so this body field is the canonical bulk channel — REST does not emit `X-ObjectStack-Dropped-Fields` for batches. Optional — omit-when-empty keeps the shape backward-compatible. | +| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed' \| 'stored_expression'> }[]` | optional | Write-observability: caller-supplied fields LEGALLY stripped from THIS row before it was written — static `readonly` / TRUE `readonlyWhen` on update, or the in-engine static `readonly` strip on create. Per-row because a batch can drop different fields on different rows (`readonlyWhen` is record-state-dependent). Present ONLY when ≥1 field was dropped for this row; the row still succeeded (success unchanged). A single response header cannot express per-row drops, so this body field is the canonical bulk channel — REST does not emit `X-ObjectStack-Dropped-Fields` for batches. Optional — omit-when-empty keeps the shape backward-compatible. | --- @@ -3201,7 +3201,7 @@ A write-path strip event: caller-supplied fields legally dropped from the payloa | **valid** | `boolean` | ✅ | True when this row would be accepted by the write path. | | **errors** | `{ field: string; code: string; message: string; rule?: string; … }[]` | ✅ | Findings that would REJECT this row. Empty when valid. | | **warnings** | `{ field: string; code: string; message: string; rule?: string; … }[]` | ✅ | Findings the target deployment ADMITS rather than rejects; the row is valid, and the write would store it. Two populations, in this order: ADR-0104 value shapes admitted under a warn-first posture (the write logs the same complaint), then the hits of ADVISORY validation rules (`severity: 'warning'` or `'info'`), each naming its `rule` and `severity` — the same elements the write answers as its own `warnings`. An advisory hit is reported only on a row the verdict accepts, as on the write, which answers a refused row with its error. | -| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed'> }[]` | optional | Write-observability for the preview: caller-supplied fields the write would LEGALLY strip from THIS row, one event per reason, in the shape and reason vocabulary the write itself reports. A strip is not a finding: `valid` is unaffected, and the fields appear in neither `errors` nor `warnings`. Present only when at least one field would be dropped. A server that does not produce this report omits the key too, so an absent key alone does not prove nothing would be dropped. | +| **droppedFields** | `{ object: string; fields: string[]; reason: Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed' \| 'stored_expression'> }[]` | optional | Write-observability for the preview: caller-supplied fields the write would LEGALLY strip from THIS row, one event per reason, in the shape and reason vocabulary the write itself reports. A strip is not a finding: `valid` is unaffected, and the fields appear in neither `errors` nor `warnings`. Present only when at least one field would be dropped. A server that does not produce this report omits the key too, so an absent key alone does not prove nothing would be dropped. | ### Nested Shape: `ValidateDataResponse.posture` diff --git a/content/docs/references/data/data-engine.mdx b/content/docs/references/data/data-engine.mdx index 2ba36929c9e..585f6eadba0 100644 --- a/content/docs/references/data/data-engine.mdx +++ b/content/docs/references/data/data-engine.mdx @@ -886,15 +886,15 @@ Options for DataEngine.update operations ## DroppedFieldsEvent -A write-path strip event: caller-supplied fields legally dropped from the payload +A write-path strip event: caller-supplied fields legally dropped from the payload, or written with the engine's computed value in place of the caller's ### Properties | Property | Type | Required | Description | | :--- | :--- | :--- | :--- | | **object** | `string` | ✅ | Object the write targeted (resolved object name) | -| **fields** | `string[]` | ✅ | Caller-supplied field names the engine removed from the write payload | -| **reason** | `Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed'>` | ✅ | Why the fields were dropped: static readonly, a TRUE readonlyWhen predicate, the primary-key strip of a payload id the engine ruled is not an identifier, or a computed (formula) field no driver has a column for | +| **fields** | `string[]` | ✅ | Caller-supplied field names whose value the engine did not store: removed from the write payload, or (`stored_expression`) written with the computed value in place of the caller's | +| **reason** | `Enum<'readonly' \| 'readonly_when' \| 'primary_key' \| 'computed' \| 'stored_expression'>` | ✅ | Why the fields were dropped: static readonly, a TRUE readonlyWhen predicate, the primary-key strip of a payload id the engine ruled is not an identifier, a computed (formula) field no driver has a column for, or a stored field whose declared expression the engine computes on every write, so the caller's different value was replaced by the computed one | --- diff --git a/packages/objectql/src/engine-stored-expression-dropped-fields.test.ts b/packages/objectql/src/engine-stored-expression-dropped-fields.test.ts new file mode 100644 index 00000000000..fd517cba0f1 --- /dev/null +++ b/packages/objectql/src/engine-stored-expression-dropped-fields.test.ts @@ -0,0 +1,304 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. +// +// #24690 — a caller's value for a STORED field that declares `expression` is +// overwritten by the computed value (#24619 item a), and the write now REPORTS +// that through `onFieldsDropped` under its own reason, `stored_expression`. +// +// Before: the engine overwrote the value and said nothing — `onFieldsDropped` +// stayed silent, so the `/batch` and PATCH doors' `droppedFields` did too, +// while the stored row held a value the caller never sent. No existing reason +// fit: `computed` is a `formula` field with no column, whose key is DROPPED; +// this field is written, with the engine's value in the caller's place. +// +// What this file pins, over a real-shaped driver (the showcase's invoice line, +// trimmed to what the computation needs): +// - a caller value the engine REPLACED is reported once, under +// `stored_expression`, on every write path: insert (single, batch, +// `insertMany` with its per-row `droppedFields`), update by id, and the +// predicate update — in every context, `isSystem` included; +// - a caller value EQUAL to the computed one is not reported (the console +// grid's consistent save stays silent), nor is a key the caller never sent; +// - a caller value another strip already took is reported once, under that +// strip's reason, never twice; +// - `strictReadonlyWrites` does NOT refuse a replaced value in 18.x (the +// maintainer's ruling 「甲」: only the loud half ships; the refusal waits +// for 19.0) — the write completes with the computed value and the report +// reaches the listener under strict too, while a strict write that ALSO +// carries a refusable drop is still refused, and fires no listener. + +import { describe, it, expect, beforeEach } from 'vitest'; +import type { DroppedFieldsEvent } from '@objectstack/spec/data'; +import { ObjectQL } from './engine.js'; +import { ReadonlyFieldRejectedError } from './readonly-strict-errors.js'; + +function makeDriver() { + const stores = new Map>(); + const storeFor = (o: string) => { + let s = stores.get(o); + if (!s) { s = new Map(); stores.set(o, s); } + return s; + }; + const matches = (row: any, where: any): boolean => { + if (!where || typeof where !== 'object') return true; + return Object.entries(where).every(([k, v]: [string, any]) => { + if (k === '$and') return (v as any[]).every((w) => matches(row, w)); + if (k === '$or') return (v as any[]).some((w) => matches(row, w)); + if (v !== null && typeof v === 'object' && !Array.isArray(v)) { + if ('$eq' in v) return row?.[k] === v.$eq; + if ('$in' in v) return Array.isArray(v.$in) && v.$in.includes(row?.[k]); + return true; + } + return row?.[k] === v; + }); + }; + let n = 0; + const driver: any = { + name: 'memory', version: '0.0.0', supports: {}, + async connect() {}, async disconnect() {}, async checkHealth() { return true; }, async execute() { return null; }, + async find(object: string, ast: any) { + const rows = Array.from(storeFor(object).values()).filter((r) => matches(r, ast?.where)); + return typeof ast?.limit === 'number' ? rows.slice(0, ast.limit) : rows; + }, + async findOne(object: string, ast: any) { + for (const r of storeFor(object).values()) if (matches(r, ast?.where)) return r; + return null; + }, + async create(object: string, data: Record) { + n += 1; + const id = (data.id as string) ?? `r_${n}`; + const row = { ...data, id }; + storeFor(object).set(id, row); + return row; + }, + async update(object: string, id: string, data: Record) { + const s = storeFor(object); + const row = { ...s.get(id), ...data, id }; + s.set(id, row); + return row; + }, + async updateMany(object: string, ast: any, data: Record) { + const s = storeFor(object); + let count = 0; + for (const row of [...s.values()]) { + if (!matches(row, ast?.where)) continue; + s.set(row.id, { ...row, ...data, id: row.id }); + count += 1; + } + return count; + }, + async delete(object: string, id: string) { return storeFor(object).delete(id); }, + async count() { return 0; }, + async bulkCreate(object: string, rows: Record[]) { + return Promise.all(rows.map((r) => this.create(object, r, undefined))); + }, + async bulkUpdate() { return []; }, async bulkDelete() {}, + async beginTransaction() { return { __trx: true, commit: async () => {}, rollback: async () => {} }; }, + async commit() {}, async rollback() {}, + }; + return { driver, storeFor }; +} + +const LINE = 'showcase_invoice_line'; +const AMOUNT: DroppedFieldsEvent = { object: LINE, fields: ['amount'], reason: 'stored_expression' }; + +function listener() { + const events: DroppedFieldsEvent[] = []; + return { events, onFieldsDropped: (e: DroppedFieldsEvent) => { events.push(e); } }; +} + +describe('[#24690] a caller value a stored expression replaced is reported as `stored_expression`', () => { + let engine: ObjectQL; + let storeFor: ReturnType['storeFor']; + const line = (id: string) => storeFor(LINE).get(id); + + beforeEach(async () => { + engine = new ObjectQL(); + const d = makeDriver(); + storeFor = d.storeFor; + engine.registerDriver(d.driver, true); + await engine.init(); + engine.registry.registerObject({ + name: 'showcase_invoice', + fields: { status: { type: 'select', options: [{ value: 'draft' }, { value: 'paid' }] } }, + } as any); + engine.registry.registerObject({ + name: LINE, + fields: { + invoice: { type: 'master_detail', reference: 'showcase_invoice', required: true }, + quantity: { type: 'number', readonlyWhen: "parent.status == 'paid'" }, + unit_price: { type: 'currency', readonlyWhen: "parent.status == 'paid'" }, + description: { type: 'text' }, + amount: { type: 'currency', min: 0, expression: { dialect: 'cel', source: 'record.quantity * record.unit_price' } }, + }, + } as any); + // A stored expression field that is ALSO `readonly`: the caller's value is + // the static strip's, reported once as `readonly`. + engine.registry.registerObject({ + name: 'qa_locked_line', + fields: { + quantity: { type: 'number' }, + unit_price: { type: 'currency' }, + amount: { type: 'currency', readonly: true, expression: { dialect: 'cel', source: 'record.quantity * record.unit_price' } }, + }, + } as any); + storeFor('showcase_invoice').set('INV-1003', { id: 'INV-1003', status: 'paid' }); + storeFor('showcase_invoice').set('INV-1004', { id: 'INV-1004', status: 'draft' }); + storeFor(LINE).set('line_paid', { id: 'line_paid', invoice: 'INV-1003', quantity: 6, unit_price: 49.99, amount: 299.94 }); + storeFor(LINE).set('line_draft', { id: 'line_draft', invoice: 'INV-1004', quantity: 3, unit_price: 10, amount: 30 }); + storeFor(LINE).set('line_draft_2', { id: 'line_draft_2', invoice: 'INV-1004', quantity: 2, unit_price: 7, amount: 14 }); + storeFor('qa_locked_line').set('ll_1', { id: 'll_1', quantity: 3, unit_price: 10, amount: 30 }); + }); + + describe('update by id — the PATCH door and every by-id `/batch` op', () => { + it('the card\'s /batch write on the PAID line: quantity is `readonly_when`, amount is `stored_expression`', async () => { + const l = listener(); + await engine.update(LINE, { id: 'line_paid', quantity: 60, amount: 2999.4 }, { onFieldsDropped: l.onFieldsDropped }); + expect(line('line_paid')).toMatchObject({ quantity: 6, amount: 299.94 }); + expect(l.events).toEqual([ + { object: LINE, fields: ['quantity'], reason: 'readonly_when' }, + AMOUNT, + ]); + }); + + it('a PATCH of amount alone: the stored value is computed, and the replacement is reported', async () => { + const l = listener(); + await engine.update(LINE, { id: 'line_paid', amount: 1 }, { onFieldsDropped: l.onFieldsDropped }); + expect(line('line_paid').amount).toBe(299.94); + expect(l.events).toEqual([AMOUNT]); + }); + + it('isSystem: the value is replaced in every context, so it is reported there too', async () => { + const l = listener(); + await engine.update(LINE, { id: 'line_draft', quantity: 4, amount: 5 }, { context: { isSystem: true }, onFieldsDropped: l.onFieldsDropped } as any); + expect(line('line_draft').amount).toBe(40); + expect(l.events).toEqual([AMOUNT]); + }); + + it('control: the grid\'s consistent save (amount equals the computed value) reports nothing', async () => { + const l = listener(); + await engine.update(LINE, { id: 'line_draft', quantity: 5, unit_price: 20, amount: 100 }, { onFieldsDropped: l.onFieldsDropped }); + expect(line('line_draft').amount).toBe(100); + expect(l.events).toEqual([]); + }); + + it('control: an amount the caller never sent is recomputed, and nothing is reported', async () => { + const l = listener(); + await engine.update(LINE, { id: 'line_draft', quantity: 7 }, { onFieldsDropped: l.onFieldsDropped }); + expect(line('line_draft').amount).toBe(70); + expect(l.events).toEqual([]); + }); + + it('a value the static `readonly` strip took is reported ONCE, as `readonly` — never also as `stored_expression`', async () => { + const l = listener(); + await engine.update('qa_locked_line', { id: 'll_1', quantity: 4, amount: 5 }, { onFieldsDropped: l.onFieldsDropped }); + expect(storeFor('qa_locked_line').get('ll_1')).toMatchObject({ quantity: 4, amount: 40 }); + expect(l.events).toEqual([{ object: 'qa_locked_line', fields: ['amount'], reason: 'readonly' }]); + }); + }); + + describe('insert', () => { + it('single row: a wrong amount is replaced and reported; the right amount and none are not', async () => { + const wrong = listener(); + const right = listener(); + const none = listener(); + await engine.insert(LINE, { invoice: 'INV-1004', quantity: 2, unit_price: 10, amount: 999 }, { onFieldsDropped: wrong.onFieldsDropped }); + await engine.insert(LINE, { invoice: 'INV-1004', quantity: 2, unit_price: 10, amount: 20 }, { onFieldsDropped: right.onFieldsDropped }); + await engine.insert(LINE, { invoice: 'INV-1004', quantity: 2, unit_price: 10 }, { onFieldsDropped: none.onFieldsDropped }); + expect(wrong.events).toEqual([AMOUNT]); + expect(right.events).toEqual([]); + expect(none.events).toEqual([]); + }); + + it('batch: the union is reported once', async () => { + const l = listener(); + await engine.insert(LINE, [ + { invoice: 'INV-1004', quantity: 1, unit_price: 3, amount: 0 }, + { invoice: 'INV-1004', quantity: 4, unit_price: 3, amount: 12 }, + { invoice: 'INV-1004', quantity: 5, unit_price: 3, amount: 1 }, + ], { onFieldsDropped: l.onFieldsDropped }); + expect(l.events).toEqual([AMOUNT]); + }); + + it('insertMany: only the row whose value was replaced carries the per-row `droppedFields`', async () => { + const l = listener(); + const outcomes: any[] = await engine.insertMany(LINE, [ + { invoice: 'INV-1004', quantity: 1, unit_price: 3, amount: 0 }, + { invoice: 'INV-1004', quantity: 4, unit_price: 3, amount: 12 }, + ], { onFieldsDropped: l.onFieldsDropped }); + expect(outcomes.map((o) => o.ok)).toEqual([true, true]); + expect(outcomes[0].droppedFields).toEqual([AMOUNT]); + expect(outcomes[1]).not.toHaveProperty('droppedFields'); + expect(l.events).toEqual([AMOUNT]); + }); + }); + + describe('a predicate update (`multi: true`)', () => { + it('a patch that stores one computed value on every row reports the caller\'s different value', async () => { + const l = listener(); + await engine.update(LINE, { quantity: 2, unit_price: 5, amount: 1 }, { where: { invoice: 'INV-1004' }, multi: true, onFieldsDropped: l.onFieldsDropped } as any); + expect([line('line_draft').amount, line('line_draft_2').amount]).toEqual([10, 10]); + expect(l.events).toEqual([AMOUNT]); + }); + + it('a patch that moves no input leaves the column alone and reports the caller\'s value', async () => { + const l = listener(); + await engine.update(LINE, { description: 'bulk', amount: 1 }, { where: { invoice: 'INV-1004' }, multi: true, onFieldsDropped: l.onFieldsDropped } as any); + expect([line('line_draft').amount, line('line_draft_2').amount]).toEqual([30, 14]); + expect(l.events).toEqual([AMOUNT]); + }); + + it('control: a caller value equal to what every row stores reports nothing', async () => { + const l = listener(); + await engine.update(LINE, { quantity: 2, unit_price: 5, amount: 10 }, { where: { invoice: 'INV-1004' }, multi: true, onFieldsDropped: l.onFieldsDropped } as any); + expect(l.events).toEqual([]); + }); + }); + + describe('strictReadonlyWrites — a replaced value is REPORTED, not refused, in 18.x (ruling 「甲」)', () => { + it('update by id: the write completes with the computed value, and the report reaches the listener', async () => { + const l = listener(); + await engine.update(LINE, { id: 'line_draft', quantity: 4, amount: 5 }, { strictReadonlyWrites: true, onFieldsDropped: l.onFieldsDropped } as any); + expect(line('line_draft')).toMatchObject({ quantity: 4, amount: 40 }); + expect(l.events).toEqual([AMOUNT]); + }); + + it('insert: the row is written with the computed value, and reported', async () => { + const l = listener(); + await engine.insert(LINE, { invoice: 'INV-1004', quantity: 2, unit_price: 10, amount: 999 }, { strictReadonlyWrites: true, onFieldsDropped: l.onFieldsDropped }); + const rows = [...storeFor(LINE).values()]; + expect(rows).toHaveLength(4); + expect(rows.find((r) => r.amount === 20 && r.quantity === 2)).toBeDefined(); + expect(l.events).toEqual([AMOUNT]); + }); + + it('predicate update: every matched row is written, and the replacement is reported', async () => { + const l = listener(); + await engine.update(LINE, { quantity: 2, unit_price: 5, amount: 1 }, { where: { invoice: 'INV-1004' }, multi: true, strictReadonlyWrites: true, onFieldsDropped: l.onFieldsDropped } as any); + expect([line('line_draft').amount, line('line_draft_2').amount]).toEqual([10, 10]); + expect(l.events).toEqual([AMOUNT]); + }); + + it('a strict write that ALSO carries a refusable drop is refused for THAT drop alone, writes nothing, and fires no listener', async () => { + const l = listener(); + const err: any = await engine + .update(LINE, { id: 'line_paid', quantity: 60, amount: 2999.4 }, { strictReadonlyWrites: true, onFieldsDropped: l.onFieldsDropped } as any) + .catch((e: unknown) => e); + expect(err).toBeInstanceOf(ReadonlyFieldRejectedError); + expect(err.code).toBe('ERR_READONLY_FIELD_REJECTED'); + expect(err.drops).toEqual([{ object: LINE, fields: ['quantity'], reason: 'readonly_when' }]); + expect(line('line_paid')).toMatchObject({ quantity: 6, amount: 299.94 }); + expect(l.events).toEqual([]); + }); + + it('insert: a strict write with a refusable drop is refused before the computation, carrying no stored_expression drop', async () => { + const l = listener(); + const err: any = await engine + .insert('qa_locked_line', { quantity: 4, unit_price: 2, amount: 5 }, { strictReadonlyWrites: true, onFieldsDropped: l.onFieldsDropped }) + .catch((e: unknown) => e); + expect(err).toBeInstanceOf(ReadonlyFieldRejectedError); + expect(err.drops.map((d: DroppedFieldsEvent) => d.reason)).not.toContain('stored_expression'); + expect([...storeFor('qa_locked_line').values()]).toHaveLength(1); + expect(l.events).toEqual([]); + }); + }); +}); diff --git a/packages/objectql/src/engine-stored-expression-write.test.ts b/packages/objectql/src/engine-stored-expression-write.test.ts index cea0c621529..2fb7eb114d0 100644 --- a/packages/objectql/src/engine-stored-expression-write.test.ts +++ b/packages/objectql/src/engine-stored-expression-write.test.ts @@ -148,7 +148,11 @@ describe('a stored field\'s declared expression is computed on write (#24619)', ); expect(line('line_paid')).toMatchObject({ quantity: 6, unit_price: 49.99, amount: 299.94 }); expect(total('INV-1003')).toBe(299.94); - expect(events).toEqual([{ object: 'showcase_invoice_line', fields: ['quantity'], reason: 'readonly_when' }]); + // [#24690] The caller's amount did not stand either, and is now reported. + expect(events).toEqual([ + { object: 'showcase_invoice_line', fields: ['quantity'], reason: 'readonly_when' }, + { object: 'showcase_invoice_line', fields: ['amount'], reason: 'stored_expression' }, + ]); }); it('the plain PATCH of amount alone stores the computed amount', async () => { diff --git a/packages/objectql/src/engine.ts b/packages/objectql/src/engine.ts index b857b0a3d3e..dd71823aff7 100644 --- a/packages/objectql/src/engine.ts +++ b/packages/objectql/src/engine.ts @@ -2305,8 +2305,8 @@ function stripComputedWriteFields( /** * The strip report as `DroppedFieldsEvent`s: one event per reason, `computed` - * before `readonly` — the order the strips run — and an empty list when - * nothing was taken. + * before `readonly`, then `stored_expression` (#24690) — the order the passes + * run — and an empty list when nothing was taken. * * ONE builder for both channels the create-side strips report through, so * they cannot disagree on a reason or its order: the batch-level union (the @@ -2321,10 +2321,13 @@ function droppedFieldEvents( object: string, computed: readonly string[], readonly: readonly string[], + storedExpression: readonly string[] = [], ): DroppedFieldsEvent[] { const events: DroppedFieldsEvent[] = []; if (computed.length > 0) events.push({ object, fields: [...computed], reason: 'computed' }); if (readonly.length > 0) events.push({ object, fields: [...readonly], reason: 'readonly' }); + // [#24690] Last: the stored-expression pass runs after every strip. + if (storedExpression.length > 0) events.push({ object, fields: [...storedExpression], reason: 'stored_expression' }); return events; } @@ -3022,6 +3025,45 @@ function applyStoredExpressionsToPatch( } } +/** + * [#24690] The stored computed fields whose CALLER value this write replaced — + * reported through `onFieldsDropped` under `stored_expression`, so a client + * that sent `amount: 1` is told its value did not stand, as #3407 rules for + * every other write-path strip, rather than reading success over a row that + * holds the computed value. + * + * A field is reported when all three hold: + * - the caller SENT it (`supplied`, the write's entry snapshot of the + * caller's values — never the live payload, which the pre-hook pass and the + * hooks have since written to); + * - it is still on the payload the compute pass reads (`present`): a value + * another strip took — a stored expression field also declared `readonly` + * — is that strip's to report, under its own reason, never twice; + * - the value STORED differs from the caller's on at least one row + * (`stored`: the one row of an insert or a by-id update, every matched row + * of a predicate update). A client that sends the computed value — the + * console grid's consistent save — is told nothing, because nothing it + * sent was replaced. + * + * `Object.is`, the comparison the compute passes themselves use: the caller's + * values were normalized (blank and numeric strings) before the snapshot. + */ +function replacedStoredExpressionFields( + plan: readonly FormulaPlanEntry[], + supplied: Readonly> | undefined, + present: Readonly> | null | undefined, + stored: ReadonlyArray>>, +): string[] { + if (!supplied || !present) return []; + const replaced: string[] = []; + for (const { name } of plan) { + if (!Object.prototype.hasOwnProperty.call(supplied, name)) continue; + if (!Object.prototype.hasOwnProperty.call(present, name)) continue; + if (stored.some((row) => !Object.is(row?.[name], supplied[name]))) replaced.push(name); + } + return replaced; +} + /** * Materialize ONE declared `formula` field against a record already in hand — * the read path's own evaluation, narrowed to a single field, with no round @@ -16370,27 +16412,62 @@ export class ObjectQL implements IObjectQLEngine { // so `readonly` would lie about it. One event per reason, in the order // the strips ran; a payload with no `formula` key reports exactly what // it did before. - const insertDrops = droppedFieldEvents(object, computedDropped, insertDropped); - if (insertDrops.length > 0) { - if (options?.strictReadonlyWrites === true) { - // Before the driver write and before validation — nothing is - // written, and "you sent a runtime-owned field" should not depend on - // whether some other field also failed a business rule (#5126's - // ordering on the update path, mirrored). - throw new ReadonlyFieldRejectedError( - object, [...new Set(insertDrops.flatMap((d) => d.fields))], insertDrops, 'insert', - ); - } - if (typeof options?.onFieldsDropped === 'function') { - // Under strict the listener deliberately does NOT fire (above): - // `DroppedFieldsEvent` is contracted as "dropped, and the write - // completed without them", and a refused write did not complete. - for (const drop of insertDrops) { - try { - options.onFieldsDropped(drop); - } catch (err) { - this.logger.warn('onFieldsDropped listener threw — ignored', { object, error: err }); - } + // + // [#24619] The stored computed fields, computed over the FINAL rows — + // after every hook and strip, before validation judges them and the + // driver stores them — so the stored value is the declared expression + // over the stored row, whatever the caller or a hook put in the key. + // [#24690] Computed HERE, ahead of the report, because a caller value + // the computation replaces is reported too, under `stored_expression` + // (`replacedStoredExpressionFields`). A key a strip above already took + // is that strip's report. + // + // ⛔ NOT refused under `strictReadonlyWrites`, by the maintainer's + // ruling 「甲」: in 18.x only the loud half ships — the reason and its + // report. The refusal is runtime behaviour no conversion can carry, so + // it waits for 19.0. So the strict check judges the strips' drops + // alone, BEFORE the computation, exactly where it stood, and a + // `stored_expression` report reaches the listener under strict too. + const refusableInsertDrops = droppedFieldEvents(object, computedDropped, insertDropped); + if (refusableInsertDrops.length > 0 && options?.strictReadonlyWrites === true) { + // Before the driver write and before validation — nothing is + // written, and "you sent a runtime-owned field" should not depend on + // whether some other field also failed a business rule (#5126's + // ordering on the update path, mirrored). + throw new ReadonlyFieldRejectedError( + object, [...new Set(refusableInsertDrops.flatMap((d) => d.fields))], refusableInsertDrops, 'insert', + ); + } + const storedExpressionReplaced: string[] = []; + const storedExpressionReplacedPerRow: string[][] = rows.map(() => []); + if (storedExpressionPlan.length > 0) { + const liveIndexes = rows + .map((row, i) => (rowErrors[i] === undefined && row != null && typeof row === 'object' ? i : -1)) + .filter((i) => i >= 0); + const computed = await computeStoredExpressions( + storedExpressionPlan, liveIndexes.map((i) => rows[i] as Record), opCtx.context, storedExpressionServing, + ); + liveIndexes.forEach((i, k) => { + const row = rows[i] as Record; + const taken = insertDroppedPerRow[i] ?? []; + const replaced = replacedStoredExpressionFields(storedExpressionPlan, suppliedPerRow[i], row, [computed[k]!]) + .filter((name) => !taken.includes(name)); + Object.assign(row, computed[k]); + storedExpressionReplacedPerRow[i] = replaced; + for (const name of replaced) if (!storedExpressionReplaced.includes(name)) storedExpressionReplaced.push(name); + }); + } + const insertDrops = droppedFieldEvents(object, computedDropped, insertDropped, storedExpressionReplaced); + if (insertDrops.length > 0 && typeof options?.onFieldsDropped === 'function') { + // A strict write that reaches here had no refusable drop (above), so + // every event left is a `stored_expression` report on a write that + // completes — reported under strict as well. A REFUSED write never + // fires the listener: it did not complete. + for (const drop of insertDrops) { + try { + options.onFieldsDropped(drop); + } catch (err) { + this.logger.warn('onFieldsDropped listener threw — ignored', { object, error: err }); } } } @@ -16401,15 +16478,6 @@ export class ObjectQL implements IObjectQLEngine { // has to exist here too. Gated on the schema actually declaring such a // predicate, so an object with only `record`-scoped requirements pays // nothing; batched, so N rows under M masters cost ONE header read. - // [#24619] The stored computed fields, computed over the FINAL rows — - // after every hook and strip, before validation judges them and the - // driver stores them — so the stored value is the declared expression - // over the stored row, whatever the caller or a hook put in the key. - if (storedExpressionPlan.length > 0) { - const liveForExpressions = rows.filter((row, i) => rowErrors[i] === undefined && row != null && typeof row === 'object'); - const computed = await computeStoredExpressions(storedExpressionPlan, liveForExpressions, opCtx.context, storedExpressionServing); - liveForExpressions.forEach((row, k) => Object.assign(row, computed[k])); - } const insertParentForRow = hasParentScopedRequiredWhen(schemaForValidation as any) ? await this.resolveMasterDetailParents(schemaForValidation, null, rows, opCtx.context) : undefined; @@ -16667,6 +16735,7 @@ export class ObjectQL implements IObjectQLEngine { if (rowErrors[i] !== undefined) return { ok: false as const, error: rowErrors[i] }; const droppedFields = droppedFieldEvents( object, computedStrip.droppedPerRow[i] ?? [], insertDroppedPerRow[i] ?? [], + storedExpressionReplacedPerRow[i] ?? [], ); return { ok: true as const, @@ -17026,13 +17095,28 @@ export class ObjectQL implements IObjectQLEngine { // [#8093] The address the caller wrote to is not a field it lost. (k) => !(k in afterObj) && !(idAddressesThisRow && k === 'id'), ); + emitDrop(fields, reason); + }; + /** One drop to the channel this write reports through: the strict refusal, or the listener. */ + const emitDrop = (fields: string[], reason: DroppedFieldsEvent['reason']): void => { if (fields.length === 0) return; if (strictReadonlyWrites) { strictDrops.push({ object, fields, reason }); return; } + notifyDrop(fields, reason); + }; + /** + * One drop to the listener ONLY, in every mode — never a strict refusal. + * [#24690] The `stored_expression` report goes here: by the maintainer's + * ruling 「甲」 18.x ships its report and not its refusal (runtime + * behaviour no conversion can carry waits for 19.0). Called after + * `assertNoStrictDrops()`, so a refused write never fires the listener. + */ + const notifyDrop = (fields: string[], reason: DroppedFieldsEvent['reason']): void => { + if (fields.length === 0 || !onFieldsDropped) return; try { - onFieldsDropped!({ object, fields, reason }); + onFieldsDropped({ object, fields, reason }); } catch (err) { this.logger.warn('onFieldsDropped listener threw — ignored', { object, error: err }); } @@ -18014,10 +18098,15 @@ export class ObjectQL implements IObjectQLEngine { // with the FINAL payload — after the hooks and both strips, so a // locked input the `readonlyWhen` strip took back out feeds the // stored value, never the caller's — and before the statement. + // [#24690] A caller value the computation replaces is reported + // under `stored_expression`, to the listener only (`notifyDrop`): + // never refused under strict in 18.x (ruling 「甲」). if (storedExpressionPlan.length > 0) { const patch = hookContext.input.data as Record; const [computed] = await computeStoredExpressions(storedExpressionPlan, [{ ...priorRecord, ...patch }], opCtx.context, storedExpressionServing); - applyStoredExpressionsToPatch(patch, computed, priorRecord); + const replaced = replacedStoredExpressionFields(storedExpressionPlan, suppliedValues, patch, [computed!]); + applyStoredExpressionsToPatch(patch, computed!, priorRecord); + notifyDrop(replaced, 'stored_expression'); } // [#21663] The payload is FINAL here (see the seam below), so // every readonly value on it is one the driver will store: a @@ -18295,12 +18384,21 @@ export class ObjectQL implements IObjectQLEngine { // a computed value only when it is the same on every row, and // refuses a patch that would compute different ones. The rows // are the D7 read (`readPriorRows` is a memo — no second fetch). + // [#24690] As on the by-id branch, to the listener only: a caller + // value differing from what any matched row then stores (the one + // computed value, or the column left as it is) is reported under + // `stored_expression`, and never refused under strict in 18.x. if (storedExpressionPlan.length > 0) { const matched = ((await readPriorRows()) ?? []).filter((row) => row != null && typeof row === 'object'); const patch = hookContext.input.data as Record; const before = await computeStoredExpressions(storedExpressionPlan, matched, opCtx.context, storedExpressionServing); const after = await computeStoredExpressions(storedExpressionPlan, matched.map((row) => ({ ...row, ...patch })), opCtx.context, storedExpressionServing); - hookContext.input.data = settleBulkStoredExpressions(storedExpressionPlan, patch, before, after, object) as any; + const settled = settleBulkStoredExpressions(storedExpressionPlan, patch, before, after, object); + notifyDrop( + replacedStoredExpressionFields(storedExpressionPlan, suppliedValues, patch, matched.map((row) => ({ ...row, ...settled }))), + 'stored_expression', + ); + hookContext.input.data = settled as any; } // [#21663] The predicate-path twin of the by-id second pass, at the // same point and for the same reason: the readonly values left on diff --git a/packages/objectql/src/readonly-strict-errors.ts b/packages/objectql/src/readonly-strict-errors.ts index c02452d2cab..46ba6d4688a 100644 --- a/packages/objectql/src/readonly-strict-errors.ts +++ b/packages/objectql/src/readonly-strict-errors.ts @@ -87,6 +87,14 @@ const REASON_PHRASE: Record = { computed: 'a computed `formula` field — the engine computes it on every read, so no driver has a ' + 'column for a written value', + // [#24690] Required by the exhaustive key type, and NOT refused in 18.x: + // by the maintainer's ruling 「甲」 a `stored_expression` drop is reported + // through `onFieldsDropped` only, under strict too, and its refusal waits + // for 19.0. No refusal this module builds carries one today; the row is + // the wording that refusal will use. + stored_expression: + 'a stored field the engine computes from its declared `expression` on every write — the ' + + 'value sent differs from the computed one, so it would have been replaced', }; /** diff --git a/packages/rest/src/rest-stored-expression-dropped-fields.test.ts b/packages/rest/src/rest-stored-expression-dropped-fields.test.ts new file mode 100644 index 00000000000..d0eb5f35f08 --- /dev/null +++ b/packages/rest/src/rest-stored-expression-dropped-fields.test.ts @@ -0,0 +1,137 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#24690] The PATCH door and the cross-object `/batch` door report a caller + * value a stored expression REPLACED, as `droppedFields` under + * `stored_expression`. + * + * Since #24619 (item a) the engine computes a stored field's declared + * `expression` on every write, so a client-sent `amount` never stands — and + * until this card nothing said so: the engine's `onFieldsDropped` stayed + * silent, so these doors answered `200` with no `droppedFields` over a row that + * holds a value the client never sent. + * + * Driven through the composed stack — `RestServer` → `ObjectStackProtocolImplementation` + * → `ObjectQL` → a real `SqlDriver` on better-sqlite3 — because the claim is + * about what reaches the wire, not about what a stubbed engine reports. + */ + +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import { mkdtempSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { ObjectQL } from '@objectstack/objectql'; +import { SqlDriver } from '@objectstack/driver-sql'; +import { ObjectStackProtocolImplementation } from '@objectstack/metadata-protocol'; +import { RestServer } from './rest-server'; + +const OBJECT = 'se_line'; +const AMOUNT = { object: OBJECT, fields: ['amount'], reason: 'stored_expression' }; + +function createMockServer() { + const noop = () => {}; + return { get: noop, post: noop, put: noop, delete: noop, patch: noop, use: noop, listen: async () => {}, close: async () => {} }; +} + +function makeRes() { + const res: any = { + headers: {} as Record, + write: () => true, end: () => {}, + header: (k: string, v: string) => { res.headers[k] = v; return res; }, + status: (code: number) => { res._status = code; return res; }, + json: (body: any) => { res._json = body; return res; }, + }; + return res; +} + +const dir = mkdtempSync(join(tmpdir(), 'os-24690-')); +let engine: ObjectQL; +let routes: any[]; + +beforeAll(async () => { + engine = new ObjectQL(); + engine.registerDriver( + new SqlDriver({ client: 'better-sqlite3', connection: { filename: join(dir, 'lines.sqlite') }, useNullAsDefault: true }) as any, + true, + ); + await engine.init(); + engine.registry.registerObject({ + name: OBJECT, + label: 'Line', + enable: { apiMethods: ['get', 'list', 'create', 'update', 'delete', 'bulk'] }, + fields: { + quantity: { name: 'quantity', type: 'number' }, + unit_price: { name: 'unit_price', type: 'number' }, + amount: { name: 'amount', type: 'number', expression: { dialect: 'cel', source: 'record.quantity * record.unit_price' } }, + }, + } as any); + await engine.syncSchemas(); + await engine.insert(OBJECT, [ + { id: 'l1', quantity: 3, unit_price: 10 }, + { id: 'l2', quantity: 2, unit_price: 7 }, + ] as any); + + const protocol = new ObjectStackProtocolImplementation(engine as any); + const rest = new RestServer( + createMockServer() as any, protocol as any, { api: { requireAuth: false } } as any, + undefined, undefined, undefined, undefined, + async () => engine, + ); + (rest as any).resolveExecCtx = async () => ({ userId: 'test-user' }); + rest.registerRoutes(); + routes = rest.getRoutes(); +}); + +afterAll(async () => { + try { await engine.destroy(); } catch { /* noop */ } + rmSync(dir, { recursive: true, force: true }); +}); + +const patch = async (id: string, body: Record) => { + const route = routes.find((r) => r.method === 'PATCH' && r.path === '/api/v1/data/:object/:id'); + expect(route).toBeDefined(); + const res = makeRes(); + await route.handler({ query: {}, headers: {}, params: { object: OBJECT, id }, body } as any, res); + return res; +}; + +const batch = async (operations: unknown[]) => { + const route = routes.find((r) => r.method === 'POST' && (r.metadata?.summary ?? '').startsWith('Cross-object')); + expect(route).toBeDefined(); + const res = makeRes(); + await route.handler({ method: 'POST', query: {}, headers: {}, params: {}, body: { operations } } as any, res); + return res; +}; + +const stored = async (id: string) => + (await engine.findOne(OBJECT, { where: { id }, context: { isSystem: true } })) as Record; + +describe('[#24690] PATCH /data/:object/:id reports a replaced stored-expression value', () => { + it('a PATCH of amount alone: 200, the computed amount is stored, and the replacement is on the body and the header', async () => { + const res = await patch('l1', { amount: 1 }); + expect(res._status ?? 200).toBe(200); + expect(res._json.droppedFields).toEqual([AMOUNT]); + expect(res.headers['X-ObjectStack-Dropped-Fields']).toBe('amount;reason=stored_expression'); + expect((await stored('l1')).amount).toBe(30); + }); + + it('control: a consistent save (amount equals the computed value) carries no droppedFields', async () => { + const res = await patch('l1', { quantity: 4, unit_price: 10, amount: 40 }); + expect(res._json).not.toHaveProperty('droppedFields'); + expect((await stored('l1')).amount).toBe(40); + }); +}); + +describe('[#24690] POST /batch reports a replaced stored-expression value, tagged with its op index', () => { + it('an update op whose amount the expression replaced is reported; the consistent op beside it is not', async () => { + const res = await batch([ + { object: OBJECT, action: 'update', id: 'l2', data: { quantity: 5, amount: 999 } }, + { object: OBJECT, action: 'update', id: 'l1', data: { quantity: 1, unit_price: 2, amount: 2 } }, + ]); + expect(res._status ?? 200).toBe(200); + expect(res._json.results).toHaveLength(2); + expect(res._json.droppedFields).toEqual([{ ...AMOUNT, index: 0 }]); + expect((await stored('l2')).amount).toBe(35); + expect((await stored('l1')).amount).toBe(2); + }); +}); diff --git a/packages/services/service-automation/src/builtin/crud-dropped-fields.test.ts b/packages/services/service-automation/src/builtin/crud-dropped-fields.test.ts index d440f6c9ac8..75cb1387475 100644 --- a/packages/services/service-automation/src/builtin/crud-dropped-fields.test.ts +++ b/packages/services/service-automation/src/builtin/crud-dropped-fields.test.ts @@ -125,6 +125,26 @@ describe('update_record surfaces silently-stripped write fields (#3407)', () => expect(step.warnings![1]).toContain('readonlyWhen'); }); + it('[#24690] a stored_expression report says the computed value was stored, not that the write went without it', async () => { + const engine = new AutomationEngine(makeLogger()); + const { data } = fakeDataWithDrops([ + { object: 'invoice_line', fields: ['amount'], reason: 'stored_expression' }, + ]); + registerCrudNodes(engine, ctxWith(data)); + engine.registerFlow('f_se', updateFlow('f_se')); + + await engine.execute('f_se', { userId: 'u1' }); + const runs = await engine.listRuns('f_se'); + const step = runs[0].steps.find((s) => s.nodeId === 'up')!; + expect(step.status).toBe('success'); + expect(step.warnings).toHaveLength(1); + expect(step.warnings![0]).toContain('[amount]'); + expect(step.warnings![0]).toContain('declared expression'); + expect(step.warnings![0]).toContain('with the computed value'); + expect(step.warnings![0]).not.toContain('without them'); + expect(step.warnings![0]).not.toContain('were NOT written'); + }); + it('emits NO warnings and NO droppedFields output when nothing was stripped', async () => { const engine = new AutomationEngine(makeLogger()); const { data } = fakeDataWithDrops([]); diff --git a/packages/services/service-automation/src/builtin/crud-nodes.ts b/packages/services/service-automation/src/builtin/crud-nodes.ts index 01e16fa3c44..c77866a6790 100644 --- a/packages/services/service-automation/src/builtin/crud-nodes.ts +++ b/packages/services/service-automation/src/builtin/crud-nodes.ts @@ -153,10 +153,21 @@ const DROPPED_REASON_LABEL: Record = { primary_key: "the field is the object's primary key and the value sent is not an identifier — the row(s) are identified by the id argument or the filter, so writing it would have overwritten their primary key (pass a scalar id, or put an id set in the filter)", // [#20805] computed: 'the field is a computed formula — its value is computed each time the record is read, so there is nothing to write (leave it out of the fields map)', + // [#24690] The field IS written — with the value its expression computes, + // not the one requested — so the closing sentence differs too (below). + stored_expression: 'the field is computed from its declared expression on every write, and the value requested differs from the computed one, which was stored instead (leave it out of the fields map, or change the fields its expression reads)', }; function droppedFieldsWarning(nodeType: string, e: DroppedFieldsEvent): string { - return `${nodeType}(${e.object}): requested field(s) [${e.fields.join(', ')}] were NOT written — ${DROPPED_REASON_LABEL[e.reason] ?? e.reason}. The write succeeded without them.`; + const fields = `[${e.fields.join(', ')}]`; + const why = DROPPED_REASON_LABEL[e.reason] ?? e.reason; + // A `stored_expression` field WAS written — with its computed value — so + // both "were NOT written" and "succeeded without them" would be false of + // it: what did not land is the requested VALUE, not the field. + if (e.reason === 'stored_expression') { + return `${nodeType}(${e.object}): the requested value(s) for field(s) ${fields} were NOT stored — ${why}. The write succeeded with the computed value(s).`; + } + return `${nodeType}(${e.object}): requested field(s) ${fields} were NOT written — ${why}. The write succeeded without them.`; } /** diff --git a/packages/spec/src/contracts/data-engine.ts b/packages/spec/src/contracts/data-engine.ts index 777bc807f8b..60af557a190 100644 --- a/packages/spec/src/contracts/data-engine.ts +++ b/packages/spec/src/contracts/data-engine.ts @@ -106,8 +106,8 @@ export interface WriteObservabilityOptions { * learns WHICH fields would go — but their result is discarded. * * It covers every drop `onFieldsDropped` reports — coverage DERIVED from the - * reported set, never an enumeration frozen at #5126. Today that is all four - * `DroppedFieldsEvent['reason']` arms: static `readonly: true` (#2948, which + * reported set, never an enumeration frozen at #5126. Today that is four of + * the five `DroppedFieldsEvent['reason']` arms: static `readonly: true` (#2948, which * only runs for non-system callers), a TRUE `readonlyWhen` predicate (#3042, * which runs for every API-BOUNDARY caller, `isSystem` included — but judges * only the keys the caller supplied at engine entry, so a value a @@ -122,6 +122,15 @@ export interface WriteObservabilityOptions { * static strip — still losing `readonlyWhen` fields in silence, which is the * bug this option exists to abolish. * + * ⛔ **The one arm outside that coverage, for the 18.x line: `stored_expression`** + * (#24690) — a caller's value for a stored field the engine computes from + * its `expression`, replaced by the computed one. By the maintainer's ruling + * 「甲」 only its loud half ships in 18.x: it is REPORTED through + * `onFieldsDropped` / `droppedFields`, under `strictReadonlyWrites` as well, + * and the write completes with the computed value. Its refusal is runtime + * behaviour no conversion can carry, so it waits for 19.0, where the derived + * coverage below reaches it like any other arm. + * * ⚠️ **A new `reason` therefore adds a new REFUSAL, by construction** — the * price of the derived coverage above, paid deliberately when `primary_key` * landed (#6437). Since that change a `strictReadonlyWrites` caller that puts diff --git a/packages/spec/src/data/data-engine.test.ts b/packages/spec/src/data/data-engine.test.ts index c356f0c741e..021ae7da141 100644 --- a/packages/spec/src/data/data-engine.test.ts +++ b/packages/spec/src/data/data-engine.test.ts @@ -1216,6 +1216,19 @@ describe('DroppedFieldsEventSchema.reason — why a write dropped submitted fiel expect(parsed).toEqual({ object: 'task', fields: ['id'], reason: 'primary_key' }); }); + it('stored_expression is the value the engine reports when it stores a field\'s computed value in place of the caller\'s (#24690)', () => { + // `computed` names a `formula` field whose key is DROPPED; a stored field + // that declares `expression` is WRITTEN, with the engine's value. One enum, + // two facts, so the second one has its own arm. + const parsed = DroppedFieldsEventSchema.parse({ + object: 'showcase_invoice_line', fields: ['amount'], reason: 'stored_expression', + }); + expect(parsed).toEqual({ object: 'showcase_invoice_line', fields: ['amount'], reason: 'stored_expression' }); + expect(DroppedFieldsEventSchema.shape.reason.options).toEqual( + ['readonly', 'readonly_when', 'primary_key', 'computed', 'stored_expression'], + ); + }); + it('still REJECTS a reason outside the vocabulary — widening is deliberate, not open-ended', () => { // The enum grows by decision (a new legal strip class), never by a // producer inventing a string. Rejection is asserted on the issue's own diff --git a/packages/spec/src/data/data-engine.zod.ts b/packages/spec/src/data/data-engine.zod.ts index d63e4810634..f96be389b1f 100644 --- a/packages/spec/src/data/data-engine.zod.ts +++ b/packages/spec/src/data/data-engine.zod.ts @@ -252,7 +252,8 @@ export const EngineUpdateOptionsSchema = lazySchema(() => BaseEngineOptionsSchem * (#2948) or a TRUE `readonlyWhen` predicate (#3042)), the primary-key strip * that keeps a ruled-non-key payload value out of the id column (#6437), or the * computed-field strip of a value no store has a column for (#20805) — and - * completed the write without them. The write itself still succeeds — + * completed the write without them; or it STORED a field's computed value in + * place of the caller's (`stored_expression`, #24690). The write itself still succeeds — * stripping is legitimate semantics, not an error — but callers that report * success per requested field (e.g. a flow's `update_record` step) need to know * which fields never landed (#3407). @@ -276,8 +277,12 @@ export const EngineUpdateOptionsSchema = lazySchema(() => BaseEngineOptionsSchem export const DroppedFieldsEventSchema = lazySchema(() => z.object({ /** Object the write targeted (resolved object name). */ object: z.string().describe('Object the write targeted (resolved object name)'), - /** Caller-supplied field names the engine removed from the write payload. */ - fields: z.array(z.string()).describe('Caller-supplied field names the engine removed from the write payload'), + /** + * Caller-supplied field names whose value the engine did not store: removed + * from the write payload, or — under `stored_expression` — written with the + * engine's computed value in place of the caller's. + */ + fields: z.array(z.string()).describe('Caller-supplied field names whose value the engine did not store: removed from the write payload, or (`stored_expression`) written with the computed value in place of the caller\'s'), /** * Why the fields were dropped: * - `readonly` — static `readonly: true` fields, caller-supplied writes are @@ -301,16 +306,34 @@ export const DroppedFieldsEventSchema = lazySchema(() => z.object({ * every context, `isSystem` included (#20805). NOT a read-only lock: * `isSystem` does not exempt it, because there is nowhere for the value to * land. + * - `stored_expression` — the field is STORED (not a `formula`) and declares + * an `expression`, so the engine computes its value on every write, in + * every context, `isSystem` included (#24619). The key is NOT dropped: the + * field is written, with the computed value in place of the caller's, so + * `computed` (no column, nothing written) would lie about it (#24690). + * Reported only when the caller's value DIFFERS from the value stored — a + * client that sends the computed value, or none, is told nothing — and + * only for a value no other strip took (a stored expression field also + * declared `readonly` is reported once, as `readonly`). On a predicate + * update the value stored is the one computed value, or the column left + * as it is when the patch moves no input; a caller value that differs + * from what any matched row stores is reported. The one arm + * `strictReadonlyWrites` does NOT refuse in 18.x (ruling 「甲」): the + * report reaches `onFieldsDropped` under strict too, and the refusal + * waits for 19.0 (see `WriteObservabilityOptions`). * * `primary_key` names the FIELD's role, not the offending value's shape, on * purpose: `not_a_primary_key` would describe the value and become false the * day the strip widens to the same-value truthy-scalar no-op the engine * currently leaves alone. `primary_key` stays true either way, and sits in * the same register as the two read-only arms — each answers "what about this - * FIELD caused the strip?". + * FIELD caused the strip?". `stored_expression` answers it the same way — + * the field is a stored column computed from its `expression` — rather than + * naming what happened to the value (`recomputed`, `overwritten`), so it + * stays true whatever the engine does with a caller's value for such a field. */ - reason: z.enum(['readonly', 'readonly_when', 'primary_key', 'computed']).describe('Why the fields were dropped: static readonly, a TRUE readonlyWhen predicate, the primary-key strip of a payload id the engine ruled is not an identifier, or a computed (formula) field no driver has a column for'), -}).describe('A write-path strip event: caller-supplied fields legally dropped from the payload')); + reason: z.enum(['readonly', 'readonly_when', 'primary_key', 'computed', 'stored_expression']).describe('Why the fields were dropped: static readonly, a TRUE readonlyWhen predicate, the primary-key strip of a payload id the engine ruled is not an identifier, a computed (formula) field no driver has a column for, or a stored field whose declared expression the engine computes on every write, so the caller\'s different value was replaced by the computed one'), +}).describe('A write-path strip event: caller-supplied fields legally dropped from the payload, or written with the engine\'s computed value in place of the caller\'s')); // -------------------------------------------------------------------------- // Write observability: advisory validation-rule hits (#22726)