fix(objectql, service-analytics)!: a plain group over a formula field is refused alike on every analytics face, instead of a 500, a driver backstop, one null group or a rendered statement

.changeset/29304-formula-plain-group-refused-alike.md

@@ -0,0 +1,20 @@
+---
+'@objectstack/objectql': patch
+'@objectstack/service-analytics': patch
+---
+
+fix(objectql, service-analytics)!: a plain group over a formula field is refused alike on every analytics face, instead of a 500, a driver backstop, one null group or a rendered statement
+
+Clause-②: no (narrowing) — `engine.aggregate` now refuses `INVALID_FIELD` / 400 a `groupBy` entry of any form (a name, `{ field }`, or a bucket) naming a virtual (`formula`) field, whatever its `returnType`, before any driver is asked: at `groupBy.N` for a name, `groupBy.N.field` for the object form, and `groupBy.N.dateGranularity` for a bucket, as before. `/analytics/query` on the native composition, `/analytics/sql` (`AnalyticsService.generateSql`), the draft preview of `queryDataset` and the delegated fallback face give the same refusal, field for field. Nothing is widened: no export of either package root, public member, answer member, error code or status is added.
+Metadata-protocol: none
+
+<!-- adr-0087: not-required (no-migration-prescription) No key of any schema is added, removed, renamed or re-shaped, and no export or type of either package root changes: a query that grouped by a column the table does not store is refused before it reaches the driver, with an error code and status already on the wire. There is nothing for a migration to rewrite and no tombstone. -->
+
+**BREAKING** accept-set narrowing on `engine.aggregate` (`@objectstack/objectql`) and on every analytics face (`@objectstack/service-analytics`), for a plain group over a `formula` field only.
+
+- **Before.** Measured on `main` `447226b2` with a real `SqlDriver`, a dataset or cube dimension over a `formula` (`returnType` `date` or `text`). `/analytics/query` on the native composition answered `DATABASE_ERROR` / 500 (the native strategy ran `GROUP BY calc` itself); on the ObjectQL composition it answered the SQL driver's missing-column backstop, `INVALID_FIELD` / 400 with no `path` and a "run schema sync" prescription that cannot help. The `/analytics/sql` dry run rendered `SELECT calc AS "calc" … GROUP BY calc` on both compositions, and the draft preview served one `null` group holding the total. `engine.aggregate` on a driver with no native `aggregate` (the in-memory aggregation) served one `null` group holding every row.
+- **Now.** The engine's aggregate door gains a storage axis (`assertGroupByTargetsAreStored`, ahead of the date-bucket door), judged by the spec's `isVirtualSearchField` (`SEARCH_VIRTUAL_TYPES`), the predicate the sort, filter, search and bucket axes already refuse a formula by. The service's copy (`dataset-date-bucket-door.ts`, `assertDatasetGroupTargetsStored`) gives the same refusal on the faces that hand the engine no keys: the dry run, the preview and the delegated fallback service. The native-SQL strategy declines a query whose `dimensions` entry resolves to a virtual field, so that query reaches the engine. A dimension through a relationship (`account.calc`) over a related object's formula is judged on the keys the cross-object path hands the related object, on the query and the dry run alike, whether or not a base row exists.
+- **Unchanged.** A plain group over a stored field, and a date or datetime bucket, are served on every face, byte for byte, and stay on the native strategy where they were. A measure-only query stays native. The #29167 refusal of a bucket over a formula keeps its words and position.
+- **Who moves.** A caller grouping by a formula field: a located refusal with a remedy, instead of a 500, a backstop with the wrong advice, an empty-looking group or SQL for a statement that never runs.
+
+**The remedy.** Denormalise the value onto the object (a stored field, written when the source changes) and group by that.

packages/client/src/envelope-caller-census.test.ts

@@ -534,8 +534,8 @@ const LEDGER: readonly LedgerRow[] = [
     // ── [#28977] the /analytics/sql dry run vs the query for a date bucket over a column that is not a date: producer reads only ──
     {
         file: 'packages/services/service-analytics/src/__tests__/objectql-echo-non-date-bucket.test.ts',
-        method: 'analytics.query', receiver: 'service', count: 2, verdict: 'NOT_SDK',
-        why: 'the real AnalyticsService that AnalyticsServicePlugin registers over a live engine, called directly (no HTTP, no dispatcher envelope) to compare the query\'s refusal with the dry run\'s (a date bucket over a text, select, number, boolean or time column), and to read the `sql` the query serves for a date and a datetime bucket',
+        method: 'analytics.query', receiver: 'service', count: 4, verdict: 'NOT_SDK',
+        why: 'the real AnalyticsService that AnalyticsServicePlugin registers over a live engine, called directly (no HTTP, no dispatcher envelope) to compare the query\'s refusal with the dry run\'s (a date bucket over a text, select, number, boolean or time column), and to read the `sql` the query serves for a date and a datetime bucket; [#29304] and to read which strategy serves a stored, measure-only or cube dimension (native) and which refuses a formula dimension (the engine)',
     },
     {
         file: 'packages/client/src/analytics-automation-json-erasure.test.ts',
@@ -781,7 +781,10 @@ describe('#13079 §2 — positive controls on the matcher itself', () => {
         // bucket over a column that is not a date calls the same
         // plugin-registered AnalyticsService to compare the query's refusal
         // and served `sql` with `generateSql`'s.
-        expect(service.length, literalNote()).toBe(17);
+        // [#29304] Two more sites in that file: the route pin calls the same
+        // plugin-registered AnalyticsService to read which strategy serves a
+        // stored dimension and which refuses a formula one.
+        expect(service.length, literalNote()).toBe(19);
         expect([...new Set(service.map((s) => s.file))].sort()).toEqual([
             'packages/client/src/analytics-automation-json-erasure.test.ts',
             'packages/drivers/driver-memory/src/memory-exists-non-boolean-refusal.test.ts',
@@ -836,15 +839,16 @@ describe('#13079 §3 — every call site is classified', () => {
         expect(production, literalNote()).toEqual([]);
     });
 
-    it('records the split: 18 payload pins, 10 result-insensitive, 17 not-SDK', () => {
+    it('records the split: 18 payload pins, 10 result-insensitive, 19 not-SDK', () => {
         expect(verdictTotal('PAYLOAD_DEPENDENT')).toBe(18);
         expect(verdictTotal('RESULT_INSENSITIVE')).toBe(10);
         // [#21441] 8 -> 9: the date-bucket echo pin's producer call (§2).
         // [#21647] 9 -> 11: the bucket echo enumeration's two producer calls (§2).
         // [#28841] 11 -> 14: the dry-run parity pin's three producer calls (§2).
         // [#28852] 14 -> 15: its dateRange-precedence call.
         // [#28977] 15 -> 17: the non-date bucket dry-run parity pin's two producer calls (§2).
-        expect(verdictTotal('NOT_SDK')).toBe(17);
+        // [#29304] 17 -> 19: the same file's route pin, two producer calls (§2).
+        expect(verdictTotal('NOT_SDK')).toBe(19);
         // The three above are LEDGER sums and cannot move on a census reading;
         // this one is census-derived, so it carries the note. [#13874]
         expect(sdkSites.length, literalNote()).toBe(28);

packages/objectql/src/aggregate-query-shape-door.ts

@@ -12,6 +12,10 @@
  *    used twice, an aggregation alias equal to a group column, or two group
  *    entries projecting one column — {@link assertAggregatedColumnsAreDistinct}.
  *
+ * [#29167, #29304] …and, ahead of the first, a `groupBy` entry of any form —
+ * a name, `{ field }`, or a bucket — naming a VIRTUAL field, which no driver
+ * stores a column for — {@link assertGroupByTargetsAreStored}.
+ *
  * ## What ran before this door, measured on `origin/main` `cb0a2fcf4f`
  *
  * Through `findData` (the `POST /api/v1/data/:object/query` door) over three
@@ -48,26 +52,34 @@
  * ingress door answers an unknown one first; a system column the map does not
  * declare is bucketed as before), or a registry-less host.
  *
- * ## A virtual field (`formula`) is refused, on its storage ground [#29167]
+ * ## A virtual field (`formula`) is refused, on its storage ground [#29167, #29304]
  *
  * A `formula` names no stored value class (its value is the expression's), so
  * the class reading above cannot judge it. It is refused on the ground the
  * sort, filter, search and value-aggregate axes already refuse it on: it is
  * VIRTUAL — computed on read, no column — judged by the spec's
  * `isVirtualSearchField` (`SEARCH_VIRTUAL_TYPES`), never a local type list,
  * and whatever its `returnType` says. Measured on `origin/main` `7788eedb`
- * before this arm, a month bucket over a `formula` with `returnType: 'date'`:
+ * before the bucket arm (#29167), a month bucket over a `formula` with
+ * `returnType: 'date'`, and on `origin/main` `447226b2` before the plain arm
+ * (#29304), a plain group over a `formula` (`returnType` `date` or `text`):
  *
  * | caller | answer |
  * |:--|:--|
  * | a SQL driver | `INVALID_FIELD` / 400 from the driver's missing-column backstop: no `path`, and a "run schema sync" prescription that cannot help |
  * | a driver with no native `aggregate` (the in-memory aggregation) | `200`, one `null` group holding every row |
  *
- * Both now answer {@link virtualDateBucketMessage}, `INVALID_FIELD` / 400 at
- * `groupBy.N.dateGranularity`, with the remedy those axes give: store the
- * value on a stored field and use that. `@objectstack/service-analytics`'s
- * `dataset-date-bucket-door.ts` gives the same refusal on the two faces that
- * reach no engine (the `/analytics/sql` dry run and the draft preview).
+ * Every `groupBy` entry is judged, whatever its form, by
+ * {@link assertGroupByTargetsAreStored}, ahead of the date-bucket door: a
+ * bucket answers {@link virtualDateBucketMessage} at
+ * `groupBy.N.dateGranularity` (the #29167 words), a plain group answers
+ * {@link virtualGroupMessage} at `groupBy.N` (a name) or `groupBy.N.field`
+ * (the object form) — `INVALID_FIELD` / 400 either way, with the remedy those
+ * axes give: store the value on a stored field and use that.
+ * `@objectstack/service-analytics`'s `dataset-date-bucket-door.ts` gives the
+ * same refusals on the faces that reach no engine (the `/analytics/sql` dry
+ * run and the draft preview), and its native-SQL strategy declines a grouped
+ * dimension over a virtual field, so that face reaches this door.
  *
  * ## The column names
  *
@@ -127,11 +139,69 @@ function virtualDateBucketMessage(
     + 'that aggregates in memory put every row into one null group while the answer looked served.';
 }
 
+/**
+ * [#29304] The refusal of a plain group over a VIRTUAL field (see the module
+ * header): {@link virtualDateBucketMessage}'s words for a group key with no
+ * bucket. `@objectstack/service-analytics`'s `dataset-date-bucket-door.ts`
+ * carries the one copy, compared field for field by its tests.
+ */
+function virtualGroupMessage(object: string, path: string, field: string, type: string): string {
+  return `aggregate('${object}'): ${path} groups by '${field}', but '${field}' is a ${type} field — its value `
+    + 'is computed on read and never stored, so no driver materialises a column to group by. The query was NOT '
+    + `run. Denormalise the value onto '${object}' (a stored field, written when the source changes) and group `
+    + 'by that. A formula field is virtual: with no column behind it the group key reached the driver as a '
+    + 'column that does not exist, and an engine that aggregates in memory put every row into one null group '
+    + 'while the answer looked served.';
+}
+
+/**
+ * Refuse a `groupBy` entry of any form — a name, `{ field }`, or a
+ * `{ field, dateGranularity }` bucket — naming a field the object declares
+ * VIRTUAL (`formula`; the spec's `isVirtualSearchField`), `INVALID_FIELD` /
+ * 400 at the position it names. See the module header.
+ */
+export function assertGroupByTargetsAreStored(
+  object: string,
+  schema: unknown,
+  groupBy: unknown,
+): void {
+  if (!Array.isArray(groupBy) || groupBy.length === 0) return;
+  const fields = (schema as { fields?: unknown } | undefined)?.fields;
+  if (!fields || typeof fields !== 'object' || Array.isArray(fields)) return;
+  for (const [i, entry] of groupBy.entries()) {
+    const named = typeof entry === 'string';
+    if (!named && (entry === null || typeof entry !== 'object' || Array.isArray(entry))) continue;
+    const { field, dateGranularity } = named
+      ? { field: entry, dateGranularity: undefined }
+      : entry as { field?: unknown; dateGranularity?: unknown };
+    if (typeof field !== 'string' || !Object.prototype.hasOwnProperty.call(fields, field)) continue;
+    const declared = (fields as Record<string, { type?: unknown }>)[field];
+    if (!isVirtualSearchField(declared as never)) continue;
+    const type = String(declared?.type);
+    if (dateGranularity != null) {
+      // [#29167] The bucket keeps its own words and position.
+      const path = `groupBy.${i}.dateGranularity`;
+      throw refusal(
+        'INVALID_FIELD',
+        virtualDateBucketMessage(object, path, field, type, String(dateGranularity)),
+        { field, fields: [field], object, param: 'groupBy', path },
+      );
+    }
+    const path = named ? `groupBy.${i}` : `groupBy.${i}.field`;
+    throw refusal(
+      'INVALID_FIELD',
+      virtualGroupMessage(object, path, field, type),
+      { field, fields: [field], object, param: 'groupBy', path },
+    );
+  }
+}
+
 /**
  * Refuse a `groupBy` entry whose `dateGranularity` buckets a field the object
- * declares neither a date nor a datetime, or a virtual (`formula`) field —
- * `INVALID_FIELD` / 400, naming the position, the field and its declared type.
- * See the module header.
+ * declares neither a date nor a datetime — `INVALID_FIELD` / 400, naming the
+ * position, the field and its declared type. A virtual field is judged ahead
+ * of this door, by {@link assertGroupByTargetsAreStored}. See the module
+ * header.
  */
 export function assertDateBucketTargetsAreTemporal(
   object: string,
@@ -147,18 +217,6 @@ export function assertDateBucketTargetsAreTemporal(
     const { field, dateGranularity } = entry as { field?: unknown; dateGranularity?: unknown };
     if (dateGranularity == null || typeof field !== 'string') continue;
     const path = `groupBy.${i}.dateGranularity`;
-    const declared = Object.prototype.hasOwnProperty.call(fields, field)
-      ? (fields as Record<string, { type?: unknown }>)[field]
-      : undefined;
-    // [#29167] A virtual field first: the class reading below has no class
-    // for it, so it would otherwise be skipped unjudged.
-    if (isVirtualSearchField(declared as never)) {
-      throw refusal(
-        'INVALID_FIELD',
-        virtualDateBucketMessage(object, path, field, String(declared?.type), String(dateGranularity)),
-        { field, fields: [field], object, param: 'groupBy', path },
-      );
-    }
     classes ??= declaredFieldClasses(fields);
     const cls = classes.get(field);
     if (cls === undefined || cls === 'date' || cls === 'datetime') continue;