diff --git a/.changeset/30773-dataset-dimension-formula-refused.md b/.changeset/30773-dataset-dimension-formula-refused.md new file mode 100644 index 00000000000..95364e22340 --- /dev/null +++ b/.changeset/30773-dataset-dimension-formula-refused.md @@ -0,0 +1,21 @@ +--- +'@objectstack/service-analytics': minor +'@objectstack/lint': minor +--- + +fix(service-analytics, lint)!: a dataset or cube dimension over a formula field is refused where it is authored — the dataset compile and `os validate` / `os build` / `os lint` / the runtime dataset save — instead of on the first query that groups by it + +Clause-②: no (narrowing) — the dataset compile (`AnalyticsService.registerDataset`, `queryDataset` and every door that compiles through them) refuses a dataset declaring a dimension over a virtual (`formula`) field of its base object, and the authoring rules refuse a dataset or `analyticsCubes` dimension over one with the new id `dimension-virtual-field-refused`, bodies they accepted before. Nothing is widened: the one new export (`DIMENSION_VIRTUAL_FIELD_REFUSED`, the rule id's constant beside its two siblings in `@objectstack/lint`) adds no accepted input, and no error code, status or answer member is added. +Metadata-protocol: narrowed +Skills-docs: updated here — `content/docs/deployment/validating-metadata.mdx` names `dimension-virtual-field-refused` beside `dimension-json-stored-field-refused` and in its cube paragraph; follow-up #30824 moves `skills/objectstack-data/rules/field-types.md`'s formula row (the skills lane). The prescription ships in the refusal text and below. + + + +**BREAKING** accept-set narrowing at dataset authoring (`@objectstack/service-analytics`, `@objectstack/lint`), for a dimension over a `formula` field only. + +- **Before.** Measured on `main` `2d04cd2b`. A dataset whose dimension's `field` is a `formula` (any `returnType`) compiled and registered (`registerDataset`), and a query naming only its other, stored dimensions was served; a query grouping by the formula dimension was refused at query time (`INVALID_FIELD` / 400 at `groupBy.N`), in front of whoever opened the dashboard tile. `os validate`, `os build`, `os lint` and the runtime dataset save (`PUT /api/v1/meta/dataset/NAME`, the gate Studio, REST and MCP authors share) reported nothing for it, on a dataset or an `analyticsCubes` cube. The measure side already refused a total over a formula at both doors. +- **Now.** The dataset compile refuses such a dataset with `DATASET_INVALID` / 400 naming the dataset, the dimension, its position (`dimensions.N`), the field and its type, and the remedy, whichever members a query names: `registerDataset`, `queryDataset` (native, ObjectQL and the draft preview alike) and the build probes that query through it. The authoring rules refuse the dimension with `dimension-virtual-field-refused` (error) at `datasets[N].dimensions[M].field` or `analyticsCubes[N].dimensions.KEY.sql`, judged on the leaf's own object for a dotted path, on all three CLI commands and at the runtime dataset save (`422`). The verdict is the spec's `isVirtualSearchField` (`SEARCH_VIRTUAL_TYPES`), the predicate the query-time doors use; `os explain dimension-virtual-field-refused` gives the reasoning. +- **Unchanged.** A dimension over a stored field, a date or datetime dimension with or without `dateGranularity`, and every measure verdict, a `count` over a formula and the stored-number remedy for a total included. The compile does not judge a dotted dimension (its hook answers for the base object only); the lint leg and the query-time refusal still do. +- **Who moves.** An author whose dataset or cube declares a dimension over a formula field: the dataset stops compiling until the dimension moves, and its stored dimensions stop being served with it. + +**The remedy.** Denormalise the value: store it in a field of its own on the object (written by a hook or a flow when its source changes) and point the dimension at that field, or remove the dimension. diff --git a/content/docs/deployment/validating-metadata.mdx b/content/docs/deployment/validating-metadata.mdx index 793665af36d..5274d37564b 100644 --- a/content/docs/deployment/validating-metadata.mdx +++ b/content/docs/deployment/validating-metadata.mdx @@ -247,11 +247,23 @@ above declared `multiple: true`), is refused Group by a field that stores one value instead. Like the measure check, it stays silent when the field's type cannot be resolved. +A dimension over a **virtual** field, a `formula`, is refused under a second +id. A formula is computed on read and never stored, whatever its `returnType`, +so no driver has a column to group by: the analytics service refuses a query +grouped by one with `400 INVALID_FIELD`, and refuses to compile a dataset that +declares one over its own object with `400 DATASET_INVALID`, whichever members +the query names. So such a dimension is refused +(`dimension-virtual-field-refused`) at `datasets[N].dimensions[M].field`; a +dimension through a relationship path is judged on the object the path +reaches. Store the value in a field of its own (written by a hook or a flow +when its source changes) and point the dimension at that field. + Both checks also judge the members of an analytics cube (`analyticsCubes`), because an authored cube is queried through the same analytics door. A cube dimension whose `sql` column is declared with a structured-JSON type or as a multi-value field is refused (`dimension-json-stored-field-refused`) at -`analyticsCubes[N].dimensions..sql`. A cube measure is judged exactly as a +`analyticsCubes[N].dimensions..sql`, and one whose `sql` column is a +formula is refused (`dimension-virtual-field-refused`) at the same path. A cube measure is judged exactly as a dataset measure is, its `type` being the aggregate: a pair outside the table is refused (`measure-aggregate-field-type-refused`, at `analyticsCubes[N].measures..type`). That covers `sum`, `avg`, `min` or diff --git a/packages/lint/src/index.ts b/packages/lint/src/index.ts index cf24059319f..504f4cde3b9 100644 --- a/packages/lint/src/index.ts +++ b/packages/lint/src/index.ts @@ -64,6 +64,7 @@ export { validateDatasetMeasureAggregates, MEASURE_AGGREGATE_FIELD_TYPE_REFUSED, DIMENSION_JSON_STORED_FIELD_REFUSED, + DIMENSION_VIRTUAL_FIELD_REFUSED, MEASURE_DURATION_FIELD_TYPE_REFUSED, // [#23235] The dimension leg's group-key verdict on its own, so a consumer // deriving dataset dimensions (cloud's AI build) asks the door's two diff --git a/packages/lint/src/rule-explanations.ts b/packages/lint/src/rule-explanations.ts index 61092c4a1b9..794d736b209 100644 --- a/packages/lint/src/rule-explanations.ts +++ b/packages/lint/src/rule-explanations.ts @@ -2360,6 +2360,25 @@ const DIMENSION_JSON_STORED_FIELD_REFUSED_EXPLANATION: RuleExplanation = { ], }; +const DIMENSION_VIRTUAL_FIELD_REFUSED_EXPLANATION: RuleExplanation = { + rule: 'dimension-virtual-field-refused', + covers: 'why analytics does not group by a formula field', + paragraphs: [ + 'A dimension is a GROUP KEY: a dataset dimension compiles to a cube dimension whose `sql` is its ' + + '`field`, and a query that selects it groups (or date-buckets) by that column. A virtual field, a ' + + '`formula`, is computed on read and never stored, whatever its `returnType`: no driver materialises ' + + 'a column for it, so there is nothing to group by. Every analytics face refuses such a group with ' + + '`400 INVALID_FIELD` at `groupBy.N`, and the dataset compile refuses a dataset declaring one over its ' + + 'own object with `400 DATASET_INVALID`, so no report, dashboard or query is ever served from it. ' + + 'Before those refusals the group key reached the driver as a column that does not exist, or an ' + + 'engine that aggregates in memory put every row into one null group while the answer looked served.', + 'Denormalise the value: store it in a field of its own on the object (written by a hook or a flow ' + + 'when its source changes), and point the dimension at that stored field. This rule makes the same ' + + 'refusal where the author is standing. A dotted path is judged on the leaf\'s own declaration.', + ANALYTICS_MEMBER_WALK, + ], +}; + const MEASURE_DURATION_FIELD_TYPE_REFUSED_EXPLANATION: RuleExplanation = { rule: 'measure-duration-field-type-refused', covers: 'which fields a duration measure measures between', @@ -5737,6 +5756,7 @@ export const RULE_EXPLANATIONS: Readonly> = Obje [VALIDATION_RULE_JSON_SCHEMA_UNKNOWN_FORMAT_EXPLANATION.rule]: VALIDATION_RULE_JSON_SCHEMA_UNKNOWN_FORMAT_EXPLANATION, [MEASURE_AGGREGATE_FIELD_TYPE_REFUSED_EXPLANATION.rule]: MEASURE_AGGREGATE_FIELD_TYPE_REFUSED_EXPLANATION, [DIMENSION_JSON_STORED_FIELD_REFUSED_EXPLANATION.rule]: DIMENSION_JSON_STORED_FIELD_REFUSED_EXPLANATION, + [DIMENSION_VIRTUAL_FIELD_REFUSED_EXPLANATION.rule]: DIMENSION_VIRTUAL_FIELD_REFUSED_EXPLANATION, [MEASURE_DURATION_FIELD_TYPE_REFUSED_EXPLANATION.rule]: MEASURE_DURATION_FIELD_TYPE_REFUSED_EXPLANATION, [REACT_CHART_DRILLDOWN_INVALID_EXPLANATION.rule]: REACT_CHART_DRILLDOWN_INVALID_EXPLANATION, [REACT_CHART_AGGREGATE_INVALID_EXPLANATION.rule]: REACT_CHART_AGGREGATE_INVALID_EXPLANATION, diff --git a/packages/lint/src/runtime-gate.dataset-writes.test.ts b/packages/lint/src/runtime-gate.dataset-writes.test.ts index 4704168457f..4c064e47ec0 100644 --- a/packages/lint/src/runtime-gate.dataset-writes.test.ts +++ b/packages/lint/src/runtime-gate.dataset-writes.test.ts @@ -61,6 +61,8 @@ const OBJECTS = [ issued_on: { type: 'date', label: 'Issued' }, total: { type: 'currency', label: 'Total' }, account: { type: 'lookup', label: 'Account', reference: 'acme_account' }, + // [#30773] A virtual field: computed on read, never stored. + total_label: { type: 'formula', label: 'Total label', expression: 'record.region', returnType: 'text' }, }, }, { @@ -229,6 +231,21 @@ describe('the door refuses each crossed rule s own defect (#19143)', () => { expect(result.rulesRun).toContain('validateDatasetMeasureAggregates'); }); + it('⭐ LIT — a dimension over a formula field, which no query can group by (#30773)', () => { + // The dataset compile refuses it with `400 DATASET_INVALID` and every + // analytics face refuses the group with `400 INVALID_FIELD`; this is the + // same refusal at the save, where the author is standing. + const result = expectRefusal( + cleanDataset({ dimensions: [{ name: 'status', field: 'status' }, { name: 'total_label', field: 'total_label' }] }), + 'dimension-virtual-field-refused', + ); + expect(result.errors.find((e) => e.rule === 'dimension-virtual-field-refused')?.path).toBe( + // The door rewrites the snapshot's index to the written item's name. + 'datasets.acme_invoice_metrics.dimensions[1].field', + ); + expect(result.rulesRun).toContain('validateDatasetMeasureAggregates'); + }); + it('⭐ LIT — a base object that resolves to nothing (the split-off rung)', () => { // `validateObjectReferences` owns this one rather than // `validateDatasetReferences`, because the reference is an object NAME and diff --git a/packages/lint/src/validate-dataset-measure-aggregates.test.ts b/packages/lint/src/validate-dataset-measure-aggregates.test.ts index 5a2e654b8a9..268eec9444a 100644 --- a/packages/lint/src/validate-dataset-measure-aggregates.test.ts +++ b/packages/lint/src/validate-dataset-measure-aggregates.test.ts @@ -21,6 +21,7 @@ import { MULTI_CAPABLE_TYPES, MULTI_OPTION_TYPES, NUMERIC_VALUE_TYPES, + SEARCH_VIRTUAL_TYPES, STRUCTURED_JSON_TYPES, VIRTUAL_FIELD_AGGREGATE_REMEDY, isAggregateCompatibleWithFieldType, @@ -32,6 +33,7 @@ import { runAuthoringRules } from './authoring-rules.js'; import { explainRule } from './rule-explanations.js'; import { DIMENSION_JSON_STORED_FIELD_REFUSED, + DIMENSION_VIRTUAL_FIELD_REFUSED, MEASURE_AGGREGATE_FIELD_TYPE_REFUSED, groupKeyClassOf as moduleGroupKeyClassOf, validateDatasetMeasureAggregates as validateDatasetMeasureAggregatesUnrecorded, @@ -1237,8 +1239,150 @@ describe('[#23235] groupKeyClassOf — the exported classifier and the rule agre }); }); -describe('[#22161] one-line verdicts — the two ids', () => { - const IDS = [MEASURE_AGGREGATE_FIELD_TYPE_REFUSED, DIMENSION_JSON_STORED_FIELD_REFUSED]; +// ─────────────────────────────────────────────────────────────────────────── +// [#30773] The virtual-dimension leg — `dimension-virtual-field-refused`. A +// dataset or cube dimension over a `formula` (computed on read, never stored) +// is a group key no query can serve: every analytics face refuses the group at +// query time (`400 INVALID_FIELD` at `groupBy.N`), and the dataset compile +// refuses a dataset declaring one. Measured on `origin/main` `2d04cd2b` before +// this leg: `runAuthoringRules` on validate / build / lint fired nothing for a +// dataset dimension or a cube dimension over a formula. +// ─────────────────────────────────────────────────────────────────────────── + +const VIRTUAL_RULE = DIMENSION_VIRTUAL_FIELD_REFUSED; +const FORMULA_TEXT = { type: 'formula', expression: 'record.name', returnType: 'text' }; +const FORMULA_DATE = { type: 'formula', expression: 'record.name', returnType: 'date' }; + +const virtualFindings = (stack: unknown) => + validateDatasetMeasureAggregates(stack).filter((f) => f.rule === VIRTUAL_RULE); + +describe('dimension-virtual-field-refused — a dimension over a virtual field is refused at authoring', () => { + // ⭐ The card's pin and its control, one pair. + it('refuses a dataset dimension over a formula field, and accepts one over a text field', () => { + const found = validateDatasetMeasureAggregates(dimensionStack(FORMULA_TEXT)); + expect(found).toHaveLength(1); + const issue = found[0]; + expect(issue.severity).toBe('error'); + expect(issue.rule).toBe(VIRTUAL_RULE); + expect(issue.path).toBe('datasets[0].dimensions[0].field'); + expect(issue.where).toBe('dataset "opportunity_metrics" › dimension "the_dimension"'); + expect(issue.message).toBe( + 'groups by field "grouped" (`formula` on object "crm_opportunity"), a value computed on read and never ' + + 'stored, so no query can group by it', + ); + // The remedy the query-time doors name: store the value, group by that. + expect(issue.hint).toBe( + 'Store the value in a stored field of "crm_opportunity" (written when its source changes) and point this ' + + 'dimension at that field, or remove the dimension.', + ); + expect(validateDatasetMeasureAggregates(dimensionStack({ type: 'text' }))).toEqual([]); + }); + + it('refuses it whatever the returnType — a date-typed dimension bucketed by month over a formula of date included', () => { + const stack = dimensionStack(FORMULA_DATE); + const ds = (stack.datasets as Array>)[0]; + ds.dimensions = [{ name: 'closed_month', field: 'grouped', type: 'date', dateGranularity: 'month' }]; + const [issue] = virtualFindings(stack); + expect(issue?.path).toBe('datasets[0].dimensions[0].field'); + expect(issue?.where).toBe('dataset "opportunity_metrics" › dimension "closed_month"'); + // …while a stored date dimension bucketed the same way stays clean. + expect(validateDatasetMeasureAggregates(dimensionStack({ type: 'date' }))).toEqual([]); + }); + + // The whole surface against the spec's one predicate, never a local list. + it('agrees with SEARCH_VIRTUAL_TYPES on every declared FieldType', () => { + expect(SEARCH_VIRTUAL_TYPES.size).toBeGreaterThan(0); + let refused = 0; + let accepted = 0; + for (const fieldType of FieldType.options) { + const fires = virtualFindings(dimensionStack({ type: fieldType })).length > 0; + expect(fires, fieldType).toBe(SEARCH_VIRTUAL_TYPES.has(fieldType)); + if (fires) refused++; + else accepted++; + } + expect(refused).toBe(SEARCH_VIRTUAL_TYPES.size); + expect(accepted).toBeGreaterThan(40); + }); + + it('names the later position beside a stored dimension, and never lands under the JSON-stored id', () => { + const stack = dimensionStack(FORMULA_TEXT); + const ds = (stack.datasets as Array>)[0]; + ds.dimensions = [{ name: 'by_name', field: 'name' }, { name: 'by_calc', field: 'grouped' }]; + const found = validateDatasetMeasureAggregates(stack); + expect(found.map((f) => [f.rule, f.path])).toEqual([[VIRTUAL_RULE, 'datasets[0].dimensions[1].field']]); + }); + + it('judges a dimension over a joined field on the object the leaf lives on', () => { + const joined = (leaf: Record) => ({ + name: 'analytics_probe', + objects: [ + { + name: 'crm_opportunity', + sharingModel: 'private', + fields: { name: { type: 'text' }, account: { type: 'lookup', reference: 'crm_account' } }, + }, + { name: 'crm_account', sharingModel: 'private', fields: { name: { type: 'text' }, tier: leaf } }, + ], + datasets: [ + { + name: 'opportunity_metrics', + object: 'crm_opportunity', + include: ['account'], + dimensions: [{ name: 'acct_tier', field: 'account.tier' }], + measures: [{ name: 'n', aggregate: 'count' }], + }, + ], + }); + const found = virtualFindings(joined(FORMULA_TEXT)); + expect(found).toHaveLength(1); + expect(found[0].message).toContain('"account.tier" (`formula` on joined object "crm_account")'); + expect(found[0].hint).toContain('a stored field of "crm_account"'); + expect(virtualFindings(joined({ type: 'text' }))).toEqual([]); + }); + + it('refuses a cube dimension whose sql names a formula column, and accepts a stored one', () => { + const stack = cubeStack({ dimensions: { calc: dim('calc'), stage: dim('stage') } }, {}, { calc: FORMULA_TEXT }); + const found = validateDatasetMeasureAggregates(stack); + expect(found).toHaveLength(1); + expect(found[0].rule).toBe(VIRTUAL_RULE); + expect(found[0].path).toBe('analyticsCubes[0].dimensions.calc.sql'); + expect(found[0].where).toBe('cube "fx_cube" › dimension "calc"'); + expect(found[0].message).toContain('(`formula` on object "fx_ledger")'); + expect(validateDatasetMeasureAggregates(cubeStack({ dimensions: { stage: dim('stage') } }))).toEqual([]); + }); + + it('never hands the predicate a guess — the same skips as the JSON-stored leg', () => { + expect(virtualFindings(dimensionStack(FORMULA_TEXT, { datasetObject: 'not_here' }))).toEqual([]); + expect(virtualFindings(dimensionStack(FORMULA_TEXT, { field: 'nope' }))).toEqual([]); + expect(virtualFindings(dimensionStack(FORMULA_TEXT, { field: undefined }))).toEqual([]); + }); + + it('leaves the measure leg over a formula as it was: a count accepted, a total refused under the measure id', () => { + expect(findings(stackWith('count', 'formula'))).toEqual([]); + const [total] = findings(stackWith('sum', 'formula')); + expect(total?.hint).toBe(VIRTUAL_FIELD_AGGREGATE_REMEDY); + expect(validateDatasetMeasureAggregates(stackWith('sum', 'formula')).map((f) => f.rule)).toEqual([RULE]); + }); + + it('fires through runAuthoringRules on all three commands, and only on the refused dimension', () => { + const stack = dimensionStack(FORMULA_TEXT); + for (const command of ['validate', 'build', 'lint'] as const) { + const found = runAuthoringRules(command, { normalized: stack, parsed: stack }).filter( + (f) => f.rule === VIRTUAL_RULE, + ); + expect(found, command).toHaveLength(1); + expect(found[0].severity, command).toBe('error'); + } + const control = dimensionStack({ type: 'text' }); + expect( + runAuthoringRules('lint', { normalized: control, parsed: control }).filter((f) => f.rule === VIRTUAL_RULE), + ).toEqual([]); + }); +}); + +describe('[#22161] one-line verdicts — the three ids', () => { + // [#30773] `dimension-virtual-field-refused` joined the two #22161 shaped. + const IDS = [MEASURE_AGGREGATE_FIELD_TYPE_REFUSED, DIMENSION_JSON_STORED_FIELD_REFUSED, DIMENSION_VIRTUAL_FIELD_REFUSED]; it('every verdict the cases above fired is one line of at most 200 characters', () => { // The coverage control first: each id fired at least once, so the shape @@ -1254,9 +1398,10 @@ describe('[#22161] one-line verdicts — the two ids', () => { const MOVED: Record = { [MEASURE_AGGREGATE_FIELD_TYPE_REFUSED]: ['property of the SQL dialect', 'average YEAR', 'compares values for equality', '400 DATASET_INVALID', '`measure-aggregate-incoherent`'], [DIMENSION_JSON_STORED_FIELD_REFUSED]: ['GROUP KEY', 'one groups each serialized value apart', 'before any SQL is built', 'instead of an answer'], + [DIMENSION_VIRTUAL_FIELD_REFUSED]: ['GROUP KEY', '`400 INVALID_FIELD` at `groupBy.N`', '`400 DATASET_INVALID`', 'one null group'], }; - it('covers exactly the two ids', () => { + it('covers exactly the three ids', () => { expect(Object.keys(MOVED).sort()).toEqual([...IDS].sort()); }); @@ -1283,8 +1428,8 @@ describe('[#22161] one-line verdicts — the two ids', () => { expect(text).toContain(`\`min\` and \`max\`: ${table.min.join(', ')}.`); }); - it('both explanations name the JSON-stored types the spec declares', () => { - for (const rule of IDS) { + it('both JSON-stored explanations name the JSON-stored types the spec declares', () => { + for (const rule of [MEASURE_AGGREGATE_FIELD_TYPE_REFUSED, DIMENSION_JSON_STORED_FIELD_REFUSED]) { const text = explanationOf(rule); expect(text, rule).toContain(`a structured-JSON type — ${[...STRUCTURED_JSON_TYPES].join(', ')} —`); expect(text, rule).toContain(`an inherently multi option type (${[...MULTI_OPTION_TYPES].join(', ')})`); @@ -1297,8 +1442,8 @@ describe('[#22161] one-line verdicts — the two ids', () => { // declaration accepts, and the dimension id its one remedy per class; why // `count` reaches further, the stored-numeric route, the doors that refuse the // pair later and the per-member `$contains` query are the explanations'. -describe('[#22161] one-line fixes — the two ids', () => { - const IDS = [MEASURE_AGGREGATE_FIELD_TYPE_REFUSED, DIMENSION_JSON_STORED_FIELD_REFUSED]; +describe('[#22161] one-line fixes — the three ids', () => { + const IDS = [MEASURE_AGGREGATE_FIELD_TYPE_REFUSED, DIMENSION_JSON_STORED_FIELD_REFUSED, DIMENSION_VIRTUAL_FIELD_REFUSED]; it('every `fix:` line the cases above fired is one line of at most 200 characters', () => { const converted = fired.filter((f) => IDS.includes(f.rule)); @@ -1310,6 +1455,8 @@ describe('[#22161] one-line fixes — the two ids', () => { const dimension = converted.filter((f) => f.rule === DIMENSION_JSON_STORED_FIELD_REFUSED).map((f) => f.hint); expect(dimension.some((h) => h.startsWith('Group by a field that stores one scalar value'))).toBe(true); expect(dimension.some((h) => h.includes('`$contains`'))).toBe(true); + const virtual = converted.filter((f) => f.rule === DIMENSION_VIRTUAL_FIELD_REFUSED).map((f) => f.hint); + expect(virtual.some((h) => h.startsWith('Store the value in a stored field'))).toBe(true); for (const f of converted) { expect(f.hint, f.rule).not.toContain('\n'); expect(f.hint.length, `${f.rule}: ${f.hint}`).toBeLessThanOrEqual(200); @@ -1330,6 +1477,10 @@ describe('[#22161] one-line fixes — the two ids', () => { '`where { FIELD: { $contains: VALUE } }`', 'one query per member', ], + [DIMENSION_VIRTUAL_FIELD_REFUSED]: [ + 'store it in a field of its own on the object', + 'point the dimension at that stored field', + ], }; for (const rule of IDS) { for (const fact of facts[rule]) expect(explanationOf(rule), `${rule} explanation names ${fact}`).toContain(fact); diff --git a/packages/lint/src/validate-dataset-measure-aggregates.ts b/packages/lint/src/validate-dataset-measure-aggregates.ts index 753ac03e22f..5f981455daa 100644 --- a/packages/lint/src/validate-dataset-measure-aggregates.ts +++ b/packages/lint/src/validate-dataset-measure-aggregates.ts @@ -185,6 +185,21 @@ * The same skips 1, 3 and 4 hold, plus a member that writes no `sql`; and on * a measure, skip 5 and a measure that writes no `type`. * + * ## [#30773] The virtual-dimension leg — `dimension-virtual-field-refused` + * + * A dimension over a VIRTUAL field — computed on read, never stored (the + * spec's `isVirtualSearchField` / `SEARCH_VIRTUAL_TYPES`, today `formula`), + * whatever its `returnType` — groups by a column no driver materialises. Every + * analytics face refuses that group at query time with `400 INVALID_FIELD` at + * `groupBy.N` (`assertGroupByTargetsAreStored` in `@objectstack/objectql`), and + * the dataset compile refuses a dataset declaring one over its own object with + * `400 DATASET_INVALID`. This leg makes the same refusal where the author is + * standing, on a dataset dimension and a cube dimension alike, resolved on the + * object graph exactly as the JSON-stored leg resolves it (dotted paths + * included, judged on the object the LEAF lives on) and under the same skips. + * ⛔ The verdict is the spec predicate's, never a local type list. A formula is + * never JSON-stored, so the two dimension ids never fire on one member. + * * ## [#25778] The duration leg — `measure-duration-field-type-refused` * * A dataset measure written `duration: { from, to, unit }` aggregates the time @@ -207,7 +222,8 @@ * there; an `analytics_cube` write is dispatched to no rule today. * * The function keeps its name: it is the registry's key (`authoring-rules.ts`), - * and the dimension, duration and cube legs ride on that one entry. + * and the dimension, virtual-dimension, duration and cube legs ride on that one + * entry. */ import { @@ -215,6 +231,7 @@ import { STRUCTURED_JSON_TYPES, isAggregateCompatibleWithFieldType, isMultiValueField, + isVirtualSearchField, virtualFieldAggregateRemedy, type ValueShapeFieldDef, } from '@objectstack/spec/data'; @@ -249,6 +266,14 @@ export const MEASURE_AGGREGATE_FIELD_TYPE_REFUSED = 'measure-aggregate-field-typ */ export const DIMENSION_JSON_STORED_FIELD_REFUSED = 'dimension-json-stored-field-refused'; +/** + * [#30773] Stable diagnostic id for a dataset or cube dimension whose column is + * VIRTUAL — computed on read, never stored (the spec's `isVirtualSearchField`). + * Named beside {@link DIMENSION_JSON_STORED_FIELD_REFUSED}: the same member, the + * storage axis instead of the value-shape one. + */ +export const DIMENSION_VIRTUAL_FIELD_REFUSED = 'dimension-virtual-field-refused'; + /** * [#25778] Stable diagnostic id for a dataset duration measure whose `from` or * `to` names a field declared anything but `datetime`. Named for the same axis @@ -271,6 +296,7 @@ export interface DatasetMeasureAggregateFinding { rule: | typeof MEASURE_AGGREGATE_FIELD_TYPE_REFUSED | typeof DIMENSION_JSON_STORED_FIELD_REFUSED + | typeof DIMENSION_VIRTUAL_FIELD_REFUSED | typeof MEASURE_DURATION_FIELD_TYPE_REFUSED; /** Human-readable location, e.g. `dataset "sales" › measure "avg_closed"` or `cube "sales" › dimension "meta"`. */ where: string; @@ -434,6 +460,33 @@ function jsonStoredDimensionFinding( }; } +/** + * [#30773] The finding for a dimension that groups by a VIRTUAL column, in one + * set of words for a dataset dimension and a cube dimension alike: one verdict + * sentence and one remedy line — the remedy the query-time doors name (store + * the value on a stored field and group by that). Which doors refuse it, and + * why a computed value is no group key, are `os explain + * dimension-virtual-field-refused`. + */ +function virtualDimensionFinding( + site: MemberSite, + leaf: Extract, + shape: ValueShapeFieldDef, +): DatasetMeasureAggregateFinding { + return { + severity: 'error', + rule: DIMENSION_VIRTUAL_FIELD_REFUSED, + where: site.where, + path: site.path, + message: + `groups by field "${site.column}" (${declaredAs(shape)} on ${site.declarer}), a value computed on read ` + + 'and never stored, so no query can group by it', + hint: + `Store the value in a stored field of "${leaf.object}" (written when its source changes) and point this ` + + 'dimension at that field, or remove the dimension.', + }; +} + /** * The finding for a measure whose aggregate the column's declaration cannot * carry ({@link acceptsDeclaration} said no), in one set of words for a @@ -578,23 +631,22 @@ function cubeMemberFindings(cube: AnyRec, cubePath: string, graph: ObjectGraph): if (!sql) continue; const column = columnOf(sql); if (!column) continue; + const name = strName(dimension.name) ?? path; + const site: MemberSite = { + where: `cube "${cubeName}" › dimension "${name}"`, + path: `${path}.sql`, + column: sql, + declarer: declarerOf(column.verdict, object), + }; + // [#30773] The storage axis first, as the engine's door judges it first. + if (isVirtualSearchField(column.shape)) { + findings.push(virtualDimensionFinding(site, column.verdict, column.shape)); + continue; + } const cls = groupKeyClassOf(column.shape); if (cls === null) continue; - const name = strName(dimension.name) ?? path; - findings.push( - jsonStoredDimensionFinding( - { - where: `cube "${cubeName}" › dimension "${name}"`, - path: `${path}.sql`, - column: sql, - declarer: declarerOf(column.verdict, object), - }, - column.verdict, - column.shape, - cls, - ), - ); + findings.push(jsonStoredDimensionFinding(site, column.verdict, column.shape, cls)); } for (const { rec: measure, path } of collectionEntries(cube.measures, `${cubePath}.measures`)) { @@ -635,8 +687,8 @@ function cubeMemberFindings(cube: AnyRec, cubePath: string, graph: ObjectGraph): /** * Refuse every dataset measure whose `aggregate` the field's declaration * cannot carry, [#25778] every duration column not declared `datetime`, and - * every dataset dimension whose field is JSON-stored; then - * [#21082] every cube dimension whose `sql` column is JSON-stored, and + * every dataset dimension whose field is JSON-stored or [#30773] virtual; then + * [#21082] every cube dimension whose `sql` column is JSON-stored or virtual, and * [#21419] every cube measure whose `type` the column's declaration cannot * carry. Returns findings (empty = * clean): each dataset's dimensions before its measures, then each cube's. @@ -679,23 +731,22 @@ export function validateDatasetMeasureAggregates(stack: unknown): DatasetMeasure if (!fieldType) return; const shape = shapeOf(verdict, fieldType); + const dimensionName = strName(dimension.name) ?? `#${k}`; + const site: MemberSite = { + where: `dataset "${dsName}" › dimension "${dimensionName}"`, + path: `datasets[${di}].dimensions[${k}].field`, + column: field, + declarer: declarerOf(verdict, object), + }; + // ── [#30773] The virtual-dimension leg: a group key over a column no driver stores ── + if (isVirtualSearchField(shape)) { + findings.push(virtualDimensionFinding(site, verdict, shape)); + return; + } const cls = groupKeyClassOf(shape); if (cls === null) return; - const dimensionName = strName(dimension.name) ?? `#${k}`; - findings.push( - jsonStoredDimensionFinding( - { - where: `dataset "${dsName}" › dimension "${dimensionName}"`, - path: `datasets[${di}].dimensions[${k}].field`, - column: field, - declarer: declarerOf(verdict, object), - }, - verdict, - shape, - cls, - ), - ); + findings.push(jsonStoredDimensionFinding(site, verdict, shape, cls)); }); recordsOf(ds.measures).forEach((measure, k) => { diff --git a/packages/qa/dogfood/test/analytics-formula-dimension-authoring.dogfood.test.ts b/packages/qa/dogfood/test/analytics-formula-dimension-authoring.dogfood.test.ts new file mode 100644 index 00000000000..42be5e10f7b --- /dev/null +++ b/packages/qa/dogfood/test/analytics-formula-dimension-authoring.dogfood.test.ts @@ -0,0 +1,96 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. +// +// END-TO-END gate for #30773: a dataset whose DIMENSION groups by a formula +// field is refused where it is authored — the runtime save (`PUT /meta/dataset`, +// the gate Studio, REST and MCP share) answers `422` naming +// `dimension-virtual-field-refused`, and the dataset compile behind +// `POST /analytics/dataset/query` answers `400 DATASET_INVALID` naming the +// dimension, whatever the selection — proven on the stock showcase, through the +// real stack. +// +// `showcase_project.budget_remaining` is a `formula` (`returnType: 'currency'`): +// computed on read, no column. Measured on `main` `2d04cd2b` before this card, +// such a dataset compiled and registered, a query selecting only its stored +// dimension was served, and only a query grouping by the formula was refused +// (#29304, `400 INVALID_FIELD` at `groupBy.0`) — in front of whoever opened +// the tile, not the author. +// +// Controls: the same dataset without the formula dimension saves and queries. + +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import { type VerifyStack } from '@objectstack/verify'; +import { bootShowcase } from './showcase-boot.js'; + +const OBJECT = 'showcase_project'; +const SAVED = 'os30773_project_by_formula'; +const SAVED_CONTROL = 'os30773_project_by_status'; + +const dataset = (name: string, withFormula: boolean) => ({ + name, + label: 'Projects', + object: OBJECT, + dimensions: [ + { name: 'status', label: 'Status', field: 'status', type: 'string' }, + ...(withFormula ? [{ name: 'remaining', label: 'Remaining', field: 'budget_remaining', type: 'number' }] : []), + ], + measures: [{ name: 'cnt', label: 'Count', aggregate: 'count' }], +}); + +describe('dogfood: a dataset dimension over a formula field is refused at authoring (#30773)', () => { + let stack: VerifyStack; + let token: string; + + beforeAll(async () => { + stack = await bootShowcase(); + token = await stack.signIn(); + }, 180_000); + + afterAll(async () => { + await stack?.stop?.(); + }); + + async function call(method: 'PUT' | 'POST', path: string, body: unknown) { + const res = await stack.apiAs(token, method, path, body); + return { status: res.status, body: (await res.json()) as Record }; + } + + it('PUT /meta/dataset: the save is refused 422, naming dimension-virtual-field-refused at the dimension', async () => { + const res = await call('PUT', `/meta/dataset/${SAVED}`, dataset(SAVED, true)); + expect(res.status, JSON.stringify(res.body)).toBe(422); + const text = JSON.stringify(res.body); + expect(text).toContain('dimension-virtual-field-refused'); + expect(text).toContain('dimensions[1].field'); + expect(text).toContain('budget_remaining'); + }); + + it('CONTROL PUT /meta/dataset: the same dataset over a stored dimension saves', async () => { + const res = await call('PUT', `/meta/dataset/${SAVED_CONTROL}`, dataset(SAVED_CONTROL, false)); + expect(res.status, JSON.stringify(res.body)).toBe(200); + }); + + for (const dimensions of [['remaining'], ['status']]) { + it(`POST /analytics/dataset/query (inline, selecting ${dimensions.join()}): 400 DATASET_INVALID naming the dimension and the remedy`, async () => { + const res = await call('POST', '/analytics/dataset/query', { + dataset: dataset('os30773_inline', true), + selection: { dimensions, measures: ['cnt'] }, + }); + expect(res.status, JSON.stringify(res.body)).toBe(400); + // This door builds its envelope by hand: `{ code, message }` at the top. + expect(res.body.code, JSON.stringify(res.body)).toBe('DATASET_INVALID'); + const message = String(res.body.message); + expect(message).toContain( + 'dataset "os30773_inline" dimension "remaining" (dimensions.1) groups by field "budget_remaining", ' + + `which object "${OBJECT}" declares as \`formula\``, + ); + expect(message).toContain(`Denormalise the value onto "${OBJECT}" (a stored field, written when the source changes)`); + }); + } + + it('CONTROL POST /analytics/dataset/query: the dataset without the formula dimension is served', async () => { + const res = await call('POST', '/analytics/dataset/query', { + dataset: dataset('os30773_inline_control', false), + selection: { dimensions: ['status'], measures: ['cnt'] }, + }); + expect(res.status, JSON.stringify(res.body)).toBe(200); + }); +}); diff --git a/packages/services/service-analytics/src/__tests__/aggregate-nontemporal-measure-refusal.test.ts b/packages/services/service-analytics/src/__tests__/aggregate-nontemporal-measure-refusal.test.ts index d93517eac97..4cbd72ef430 100644 --- a/packages/services/service-analytics/src/__tests__/aggregate-nontemporal-measure-refusal.test.ts +++ b/packages/services/service-analytics/src/__tests__/aggregate-nontemporal-measure-refusal.test.ts @@ -424,3 +424,117 @@ describe('#16099 — the controls: every pair the table accepts still compiles', expect(bare.length).toBe(1); }); }); + +// ───────────────────────────────────────────────────────────────────────────── +// [#30773] The DIMENSION sibling of #24426's virtual-field measure refusal +// ───────────────────────────────────────────────────────────────────────────── + +/** + * A dataset whose dimensions are `dims` (name → field), with one plain count, so + * the only thing judged is the dimension list. + */ +const datasetWithDimensions = (dims: Array>) => DatasetSchema.parse({ + name: 'task_dims', + label: 'Task dimensions', + object: 'duly_task', + include: [], + dimensions: dims, + measures: [{ name: 'task_count', aggregate: 'count' }], +}); + +/** The compile's refusal of `dimension` (at `index`) grouping by the formula `margin`. */ +const virtualDimensionRefusal = (dimension: string, index: number) => + `[dataset-compiler] dataset "task_dims" dimension "${dimension}" (dimensions.${index}) groups by field "margin", ` + + 'which object "duly_task" declares as `formula` — its value is computed on read and never stored, so no driver ' + + 'materialises a column to group by, and every query selecting this dimension is refused. Denormalise the value ' + + 'onto "duly_task" (a stored field, written when the source changes) and point the dimension at that field.'; + +describe('[#30773] a dataset DIMENSION over a virtual field is refused when the dataset is compiled', () => { + // Measured on `origin/main` `2d04cd2b` before this gate: `registerDataset` + // ACCEPTED a dataset with a dimension over a `formula`, a query selecting a + // stored dimension of it was SERVED, and only a query selecting the formula + // dimension was refused, at query time, in front of whoever opened the tile. + const DIMS = [ + { name: 'status', field: 'status', type: 'string' }, + { name: 'margin_band', field: 'margin', type: 'string' }, + ]; + + it('the request door (queryDataset): DATASET_INVALID / 400 naming the dimension, its position and the remedy, before any SQL', async () => { + for (const selected of [['margin_band'], ['status']]) { + const { svc, sqls } = makeService(); + const err = await refusalOf(() => svc.queryDataset( + datasetWithDimensions(DIMS) as never, + { dimensions: selected, measures: ['task_count'] } as never, + )); + expect(err.code, selected.join()).toBe('DATASET_INVALID'); + expect(err.status, selected.join()).toBe(400); + expect(err.message, selected.join()).toBe(virtualDimensionRefusal('margin_band', 1)); + expect(sqls, selected.join()).toEqual([]); + } + }); + + it('the configuration door (registerDataset) refuses the same dataset, in the same words', () => { + const { svc } = makeService(); + let err: (Error & { code?: string; status?: number }) | undefined; + try { + svc.registerDataset(datasetWithDimensions(DIMS) as never); + } catch (e) { + err = e as Error & { code?: string; status?: number }; + } + expect(err?.code).toBe('DATASET_INVALID'); + expect(err?.status).toBe(400); + expect(err?.message).toBe(virtualDimensionRefusal('margin_band', 1)); + }); + + it('refuses a date dimension bucketed over a formula too: the gate is the storage fact, not the returnType', async () => { + const { svc } = makeService(); + const err = await refusalOf(() => svc.queryDataset( + datasetWithDimensions([{ name: 'margin_month', field: 'margin', type: 'date', dateGranularity: 'month' }]) as never, + { measures: ['task_count'] } as never, + )); + expect(err.code).toBe('DATASET_INVALID'); + expect(err.message).toBe(virtualDimensionRefusal('margin_month', 0)); + }); + + it('controls: a stored dimension, a date dimension and a numeric one still compile and query', async () => { + const { svc, sqls } = makeService([{ status: 'open', submitted_at: '2026-03', cycle_days: 3, task_count: 1 }]); + const ds = datasetWithDimensions([ + { name: 'status', field: 'status', type: 'string' }, + { name: 'submitted_at', field: 'submitted_at', type: 'date', dateGranularity: 'month' }, + { name: 'cycle_days', field: 'cycle_days', type: 'number' }, + ]); + expect(() => svc.registerDataset(ds as never)).not.toThrow(); + const r: any = await svc.queryDataset(ds as never, { dimensions: ['status', 'cycle_days'], measures: ['task_count'] } as never); + expect(r.rows.length).toBe(1); + expect(sqls.length).toBe(1); + }); + + it('the cannot-answer tiers do not block: no hook, a field the hook cannot resolve, a relationship path', () => { + // No `sourceFieldMeta` at all (no data engine wired). + const noMeta = new AnalyticsService({ + queryCapabilities: () => ({ nativeSql: true, objectqlAggregate: false, inMemory: false }), + executeRawSql: async () => [], + } as never); + expect(() => noMeta.registerDataset(datasetWithDimensions(DIMS) as never)).not.toThrow(); + // A field the hook does not answer for. + expect(() => makeService().svc.registerDataset( + datasetWithDimensions([{ name: 'other', field: 'no_such_field', type: 'string' }]) as never, + )).not.toThrow(); + // A dotted path: the hook answers for the BASE object only, so a joined + // leaf is the query-time doors' and the lint leg's to judge. + const rel = DatasetSchema.parse({ + name: 'task_dims_rel', label: 'Task dimensions', object: 'duly_task', include: ['account'], + dimensions: [{ name: 'acct_margin', field: 'account.margin', type: 'string' }], + measures: [{ name: 'task_count', aggregate: 'count' }], + }); + expect(() => makeService().svc.registerDataset(rel as never)).not.toThrow(); + }); + + it('leaves the measure gate over the same formula as it was: a count compiles, a total keeps the stored-number remedy', async () => { + const counted = run('count', 'margin'); + await expect(counted.go()).resolves.toBeDefined(); + const err = await refusalOf(run('sum', 'margin').go); + expect(err.code).toBe('DATASET_INVALID'); + expect(err.message).toContain(VIRTUAL_FIELD_AGGREGATE_REMEDY); + }); +}); diff --git a/packages/services/service-analytics/src/__tests__/preview-date-bucket-type.test.ts b/packages/services/service-analytics/src/__tests__/preview-date-bucket-type.test.ts index 5a50582fd72..c66135efe45 100644 --- a/packages/services/service-analytics/src/__tests__/preview-date-bucket-type.test.ts +++ b/packages/services/service-analytics/src/__tests__/preview-date-bucket-type.test.ts @@ -38,6 +38,17 @@ * 500, ObjectQL the SQL driver's missing-column backstop (400, no `path`), the * preview one `null` group holding the total. Every face now answers the * engine's plain-group refusal at `groupBy.N`, field for field. + * + * [#30773] …and since then a dataset that DECLARES a dimension over a formula + * no longer compiles: the dataset compile refuses it with `DATASET_INVALID` / + * 400 before any face is chosen, whichever members the selection names. So + * the formula dimensions moved out of {@link DATASET} into + * {@link FORMULA_DATASET}, and the #29167 / #29304 cases on this harness now pin + * that earlier refusal, alike on every face. The query-time refusal itself + * stays pinned where no dataset compile stands in front of it: the engine's + * door (`@objectstack/objectql`), the `/analytics/sql` dry run + * (`objectql-echo-non-date-bucket.test.ts`) and the delegated face + * (`fallback-delegate-date-bucket-type.test.ts`). */ import { describe, it, expect, beforeAll, afterAll } from 'vitest'; @@ -67,14 +78,41 @@ const DATASET = DatasetSchema.parse({ { name: 'closed_at', field: 'closed_at', type: 'date' }, { name: 'closed_on', field: 'closed_on', type: 'date' }, { name: 'opened_time', field: 'opened_time', type: 'date' }, - { name: 'calc', field: 'calc', type: 'date' }, { name: 'region_month', field: 'region', type: 'date', dateGranularity: 'month' }, + ], + measures: [{ name: 'total', aggregate: 'sum', field: 'amount' }], +}); + +/** + * [#30773] The formula dimensions #29167 / #29304 asked through, in a dataset + * of their own behind one stored dimension: a dataset declaring any of them is + * refused when it is compiled, so they can no longer share {@link DATASET}. + */ +const FORMULA_DATASET = DatasetSchema.parse({ + name: 'os28852_formula_deals', + label: 'Deals by formula', + object: OBJECT, + dimensions: [ + { name: 'region', field: 'region', type: 'string' }, + { name: 'calc', field: 'calc', type: 'date' }, { name: 'calc_month', field: 'calc', type: 'date', dateGranularity: 'month' }, { name: 'calc_s', field: 'calc_s', type: 'string' }, ], measures: [{ name: 'total', aggregate: 'sum', field: 'amount' }], }); +/** [#30773] A dataset whose only formula dimension is `calc_s` (`returnType` text). */ +const FORMULA_TEXT_DATASET = DatasetSchema.parse({ + name: 'os28852_formula_text_deals', + label: 'Deals by text formula', + object: OBJECT, + dimensions: [ + { name: 'region', field: 'region', type: 'string' }, + { name: 'calc_s', field: 'calc_s', type: 'string' }, + ], + measures: [{ name: 'total', aggregate: 'sum', field: 'amount' }], +}); + type Face = 'native' | 'objectql' | 'preview'; const FACES: readonly Face[] = ['native', 'objectql', 'preview']; @@ -92,9 +130,9 @@ describe('[#28852] a date bucket over a column that is not a date, on every face let engine: ObjectQL; const services: Partial> = {}; - const ask = (face: Face, selection: Selection) => + const ask = (face: Face, selection: Selection, dataset: typeof DATASET = DATASET) => services[face === 'preview' ? 'objectql' : face]!.queryDataset( - DATASET, + dataset, { ...selection } as never, undefined, face === 'preview' ? { previewDrafts: true } : undefined, @@ -200,52 +238,59 @@ describe('[#28852] a date bucket over a column that is not a date, on every face expect(refusal).toMatchObject({ path: 'groupBy.0.dateGranularity', field: 'region', fields: ['region'] }); }); - it('[#29167] refuses a bucket over a formula (virtual: no stored column) with the engine\'s refusal, where the preview served one null group', async () => { - // [#28977] pinned the skip this replaces: the engine judged no formula, so - // the live faces answered the SQL driver's missing-column backstop (no - // path, a "run schema sync" prescription) and the preview served one - // `null` group holding the total. The engine's door now refuses a virtual - // field, and the service's copy gives the same refusal on the preview. - const refusal = await refusedAlike({ measures: ['total'], timeDimensions: [{ dimension: 'calc', granularity: 'month' }] }); - expect(refusal).toEqual({ - message: - `aggregate('${OBJECT}'): groupBy.0.dateGranularity buckets 'calc' by 'month', but 'calc' is a formula field — ` - + 'its value is computed on read and never stored, so no driver materialises a column to bucket. The query was ' - + `NOT run. Denormalise the value onto '${OBJECT}' (a stored date or datetime field, written when the source ` - + 'changes) and bucket that. A formula field is virtual: with no column behind it the bucket reached the driver ' - + 'as a column that does not exist, and an engine that aggregates in memory put every row into one null group ' - + 'while the answer looked served.', - code: 'INVALID_FIELD', status: 400, httpStatus: 400, - object: OBJECT, param: 'groupBy', path: 'groupBy.0.dateGranularity', field: 'calc', fields: ['calc'], - }); - }); + /** + * [#30773] Every face refuses a query on `dataset` with the compile's one + * refusal of a formula dimension, field for field: `DATASET_INVALID` / 400. + */ + const refusedAtCompileAlike = async (dataset: typeof DATASET, selection: Selection) => { + const refusals: Array> = []; + for (const face of FACES) { + const err = (await ask(face, selection, dataset).then( + () => { throw new Error(`${face}: expected a refusal, but the query resolved`); }, + (e: unknown) => e, + )) as Error & Record; + expect(err.code, `${face}: ${err.message}`).toBe('DATASET_INVALID'); + expect(err.status, face).toBe(400); + refusals.push({ message: err.message, code: err.code, status: err.status }); + } + for (const [i, refusal] of refusals.entries()) expect(refusal, `${FACES[i]} vs native`).toEqual(refusals[0]); + return refusals[0]; + }; - it("[#29167] refuses a dataset dimension's declared dateGranularity over a formula", async () => { - const refusal = await refusedAlike({ dimensions: ['calc_month'], measures: ['total'] }); - expect(refusal).toMatchObject({ path: 'groupBy.0.dateGranularity', field: 'calc', fields: ['calc'] }); + /** The compile's refusal of the formula dimension `dimension` at `index`, grouping by `field`. */ + const formulaDimensionRefusal = (dataset: string, dimension: string, index: number, field: string) => ({ + message: + `[dataset-compiler] dataset "${dataset}" dimension "${dimension}" (dimensions.${index}) groups by field ` + + `"${field}", which object "${OBJECT}" declares as \`formula\` — its value is computed on read and never ` + + 'stored, so no driver materialises a column to group by, and every query selecting this dimension is ' + + `refused. Denormalise the value onto "${OBJECT}" (a stored field, written when the source changes) and ` + + 'point the dimension at that field.', + code: 'DATASET_INVALID', status: 400, }); - it.each([ - ['date', 'calc'], - ['text', 'calc_s'], - ])('[#29304] refuses a plain dimension over a formula (returnType %s) with the engine\'s refusal on every face, where native answered 500, ObjectQL the driver backstop, and the preview one null group', async (_label, field) => { - const refusal = await refusedAlike({ dimensions: [field], measures: ['total'] }); - expect(refusal).toEqual({ - message: - `aggregate('${OBJECT}'): groupBy.0 groups by '${field}', but '${field}' is a formula field — its value is ` - + 'computed on read and never stored, so no driver materialises a column to group by. The query was NOT run. ' - + `Denormalise the value onto '${OBJECT}' (a stored field, written when the source changes) and group by that. ` - + 'A formula field is virtual: with no column behind it the group key reached the driver as a column that does ' - + 'not exist, and an engine that aggregates in memory put every row into one null group while the answer looked ' - + 'served.', - code: 'INVALID_FIELD', status: 400, httpStatus: 400, - object: OBJECT, param: 'groupBy', path: 'groupBy.0', field, fields: [field], - }); + // [#29167] pinned a bucket over a formula (a `timeDimensions` granularity and + // a declared `dateGranularity`) refused on every face with the engine's + // virtual-field refusal at `groupBy.0.dateGranularity`, where the preview + // served one null group; [#29304] a plain dimension over a formula of either + // `returnType` at `groupBy.N`, where native answered 500, ObjectQL the driver + // backstop and the preview one null group. [#30773] The dataset declaring + // them now refuses at compile, before a face is chosen — for these + // selections, and for one that names only its stored dimension. + it.each<[string, Selection]>([ + ['[#29167] a bucket over a formula', { measures: ['total'], timeDimensions: [{ dimension: 'calc', granularity: 'month' }] }], + ["[#29167] a dataset dimension's declared dateGranularity over a formula", { dimensions: ['calc_month'], measures: ['total'] }], + ['[#29304] a plain dimension over a formula (returnType date)', { dimensions: ['calc'], measures: ['total'] }], + ['[#29304] a plain dimension over a formula (returnType text)', { dimensions: ['calc_s'], measures: ['total'] }], + ['[#29304] a formula dimension after a stored one', { dimensions: ['region', 'calc_s'], measures: ['total'] }], + ['a query naming only the stored dimension', { dimensions: ['region'], measures: ['total'] }], + ])('[#30773] refuses %s at compile, alike on every face, naming the first formula dimension the dataset declares', async (_label, selection) => { + expect(await refusedAtCompileAlike(FORMULA_DATASET, selection)) + .toEqual(formulaDimensionRefusal('os28852_formula_deals', 'calc', 1, 'calc')); }); - it('[#29304] names the position of a plain formula dimension after a stored one', async () => { - const refusal = await refusedAlike({ dimensions: ['region', 'calc_s'], measures: ['total'] }); - expect(refusal).toMatchObject({ path: 'groupBy.1', field: 'calc_s', fields: ['calc_s'] }); + it('[#30773] refuses a dataset whose only formula dimension returns text, at its own position', async () => { + expect(await refusedAtCompileAlike(FORMULA_TEXT_DATASET, { dimensions: ['region'], measures: ['total'] })) + .toEqual(formulaDimensionRefusal('os28852_formula_text_deals', 'calc_s', 1, 'calc_s')); }); it('[#29304] still serves a plain dimension over a stored column alike on every face', async () => { diff --git a/packages/services/service-analytics/src/dataset-compiler.ts b/packages/services/service-analytics/src/dataset-compiler.ts index 436c1813a3a..9bf5497fcd8 100644 --- a/packages/services/service-analytics/src/dataset-compiler.ts +++ b/packages/services/service-analytics/src/dataset-compiler.ts @@ -7,6 +7,7 @@ import { AggregationFunction, isAggregateCompatibleWithFieldType, isMultiValueField, + isVirtualSearchField, virtualFieldAggregateRemedy, } from '@objectstack/spec/data'; import type { Dataset, DatasetMeasure, DatasetDimension } from '@objectstack/spec/ui'; @@ -467,6 +468,54 @@ function assertAggregateFieldTypeCompatible( ); } +/** + * [#30773] Refuse a dataset DIMENSION whose field the base object declares + * VIRTUAL — computed on read, never stored (the spec's `isVirtualSearchField` + * / `SEARCH_VIRTUAL_TYPES`, today `formula`), whatever its `returnType`. + * + * A dimension is a GROUP KEY: it compiles to a cube dimension whose `sql` is + * the field, and every query that selects it groups (or buckets) by that + * column. No driver materialises a column for a virtual field, so since + * #29304 every analytics face refuses that group at query time with + * `INVALID_FIELD` / 400 at `groupBy.N` (`aggregate-query-shape-door.ts` + * `assertGroupByTargetsAreStored`; the service's copy in + * `dataset-date-bucket-door.ts`). Before this gate the dataset itself still + * compiled and registered, so the author published a dimension no query could + * serve and the refusal reached a dashboard reader instead. This is the + * authoring half Prime Directive #12 asks for, the dimension sibling of the + * measure's virtual-field refusal (#24426, `virtualFieldAggregateRemedy`): + * `DATASET_INVALID` / 400 — a verdict about the dataset DOCUMENT, decided from + * metadata alone — naming the dimension, its position and the remedy the + * query-time doors name (store the value on a stored field and group by that). + * + * Tiering is the measure gate's ("cannot answer, do not block"): no + * `declaredValueShape` hook, a field the hook cannot resolve, or a + * RELATIONSHIP-PATH field (the hook answers for the base object only) is not + * judged here. A dotted dimension over a virtual leaf stays the query-time + * doors' and the lint leg's (`dimension-virtual-field-refused`), which resolves + * the object graph. ⛔ No local type list: the verdict is the spec predicate's. + */ +function assertDimensionFieldStored( + datasetName: string, + objectName: string, + dimension: DatasetDimension, + index: number, + declaredValueShape?: (objectName: string, field: string) => ValueShapeFieldDef | undefined, +): void { + if (!declaredValueShape) return; + const field = dimension.field; + if (!field || field.includes('.')) return; + const fieldType = declaredValueShape(objectName, field)?.type; + if (!fieldType || !isVirtualSearchField({ type: fieldType })) return; + throw datasetInvalidError( + `[dataset-compiler] dataset "${datasetName}" dimension "${dimension.name}" (dimensions.${index}) groups by ` + + `field "${field}", which object "${objectName}" declares as \`${fieldType}\` — its value is computed on read ` + + 'and never stored, so no driver materialises a column to group by, and every query selecting this dimension ' + + `is refused. Denormalise the value onto "${objectName}" (a stored field, written when the source changes) and ` + + 'point the dimension at that field.', + ); +} + /** * One column of the base object by its bare name — the spec's `duration.from` / * `to` shape (`ANALYTICS_BASE_COLUMN`, which `@objectstack/spec` does not @@ -805,8 +854,13 @@ export function compileDataset( // Compile dimensions. const dimensions: Record = {}; - for (const d of dataset.dimensions) { + for (const [index, d] of dataset.dimensions.entries()) { assertDeclared(d.field, 'dimension', d.name); + // [#30773] …and the column must be STORED: a group key over a virtual field + // is one no query can serve. Placed after the join-declaration check, as the + // measure's type gate is, so a dotted field is refused for the reason it is + // actually wrong before this gate stands down on it. + assertDimensionFieldStored(dataset.name, dataset.object, d, index, options?.declaredValueShape); // The dimension's NAME is the key it is filed under (`dimensions[d.name]` // below) — the cube member carries no inner copy: #20300 retired // `Dimension.name` (ADR-0049 enforce-or-remove), because every consumer