diff --git a/.changeset/27074-dataset-drill-window.md b/.changeset/27074-dataset-drill-window.md new file mode 100644 index 00000000000..7bc7b762276 --- /dev/null +++ b/.changeset/27074-dataset-drill-window.md @@ -0,0 +1,16 @@ +--- +'@objectstack/service-analytics': patch +'@objectstack/spec': patch +--- + +fix(service-analytics): a windowed dataset answer reports its window in `drillRanges`, so a drill list stays inside the picked period + +Clause-②: yes (widening) +Metadata-protocol: none + +A dataset query whose selection carries `timeDimensions[].dateRange` (a report's picked period) counts only that period's records, but its answer's `drillRanges` covered only the date dimensions it groups by. So a report run on "this quarter" and grouped by stage drilled to a list of every period's deals, and a month bucket cut by `last_30_days` drilled the whole month. + +- **The window rides `drillRanges`.** A windowed dimension that is not grouped gets an entry on every row: the `[gte, lt)` the query filtered by. A windowed field that a bucket already bounds narrows that bucket's entry to the part inside the window, and adds no entry of its own, so an edge bucket drills only its in-window records. Drill hosts already filter by every `drillRanges` entry, by its `field`, so they need no change. +- **One window, not a second resolver.** The bounds come from the query's own steps: the preset resolved in the reference time zone, an explicit `[a, b]` as written, the shared whole-day lowering (a bare-day end runs to the start of the next day), and the engine's temporal storage form: ISO instants for a `datetime` field, `YYYY-MM-DD` days for a `date` field. +- **No entry when there is no exact spelling.** An explicit window whose end is an instant (compared with `<=`), or whose end is the last supported day (no upper bound), gets no entry, and the drill stays as wide as before. So does a field whose type is unknown under a non-UTC time zone, as for buckets. +- **Spec.** The `drillRanges` docblock on `AnalyticsResult` and its `.describe` text on the dataset answer schema say what the key now carries. No key is added. diff --git a/content/docs/references/api/analytics.mdx b/content/docs/references/api/analytics.mdx index 8a2643a9943..30d0615ad73 100644 --- a/content/docs/references/api/analytics.mdx +++ b/content/docs/references/api/analytics.mdx @@ -158,7 +158,7 @@ const result = AnalyticsEndpoint.parse(data); | **dimensionFields** | `Record` | optional | Drill-through sidecar: each equality-drillable dimension the answer is grouped by, dimension name to that dataset dimension's own `field` (a base-object field or a `relationship.field` path). Set only on a dataset answer (`POST /analytics/dataset/query`) that has rows and groups by at least one dimension with a `field` that is not `type: 'date'`; a date dimension is never listed (see `drillRanges`). Absent on a cube query answer. | | **drillRawRows** | `Record[]` | optional | Drill-through sidecar, aligned to `rows` by index: each `dimensionFields` dimension name to the stored value that row was grouped by (a select option's value, a lookup's record id), captured before label resolution rewrites the row to its display label. An exact-match drill filter is built from these, not from the labels in `rows`. Set exactly when `dimensionFields` is. | | **drillRawTotals** | `Record[][]` | optional | Drill-through sidecar for `totals`: entry `[i][j]` aligns to `totals[i].rows[j]` and holds the stored value of the `dimensionFields` dimensions that grouping groups by, so the grand-total grouping yields an empty map per row. Set only when `dimensionFields` is and the answer carries at least one `totals` grouping. | -| **drillRanges** | `Record[]` | optional | Drill-through sidecar for time buckets, aligned to `rows` by index: date dimension name to the half-open range `[gte, lt)` that row's bucket covers. Bounds are `YYYY-MM-DD` calendar days, or ISO-8601 instants at midnight in the reference timezone when the source field is `datetime`; a row whose bucket value is null gets no entry for that dimension. Set only on a dataset answer that has rows and groups by at least one `type: 'date'` dimension with a `field` and a granularity; independent of `dimensionFields`. Absent on a cube query answer. | +| **drillRanges** | `Record[]` | optional | Drill-through sidecar for date ranges, aligned to `rows` by index: date dimension name to the half-open range `[gte, lt)` that row's records fall in on that dimension's field. A time bucket gets the span it covers; a `timeDimensions[].dateRange` window gets the window the query filtered by on every row, or narrows the bucket entry on the same field to the part inside the window. Bounds are `YYYY-MM-DD` calendar days, or ISO-8601 instants when the source field is `datetime` (a bucket's at midnight in the reference timezone); a row whose bucket value is null gets no bucket entry, and a window whose end is an instant or the last supported day gets none. Set only on a dataset answer that has rows and groups by at least one `type: 'date'` dimension with a `field` and a granularity, or carries a `dateRange` on a dimension with a `field`; independent of `dimensionFields`. Absent on a cube query answer. | --- diff --git a/packages/services/service-analytics/src/__tests__/dataset-drill-window.test.ts b/packages/services/service-analytics/src/__tests__/dataset-drill-window.test.ts new file mode 100644 index 00000000000..8db807be507 --- /dev/null +++ b/packages/services/service-analytics/src/__tests__/dataset-drill-window.test.ts @@ -0,0 +1,424 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#27074] A dataset answer whose selection carries a + * `timeDimensions[].dateRange` window reports that window in `drillRanges`, + * so a drill list filters by exactly the bounds the query filtered by. + * + * MEASURED on `origin/main` `6845035c`, before the change: a query grouped by + * `stage` with `close_date` windowed to `this_quarter` under `Asia/Shanghai` + * answered no `drillRanges` at all, so the drill list behind a stage row + * listed every period's deals; and a month bucket cut by `last_30_days` + * drilled the whole month. + * + * The unit half drives `queryDataset` over a mocked engine aggregate and reads + * the window the ObjectQL strategy handed the engine beside the answer. The + * real-engine half (SQLite through the plugin's own composition, both + * strategies) runs each row's drill filter as the drill list does, + * `{ field: { $gte, $lt } }` ANDed with the row's raw grouped value, and + * counts what it lists against what the row counted. + */ + +import { describe, it, expect, beforeAll, afterAll, beforeEach, afterEach, vi } from 'vitest'; +import { ObjectQL } from '@objectstack/objectql'; +import { SqlDriver } from '@objectstack/driver-sql'; +import { DatasetSchema } from '@objectstack/spec/ui'; +import type { ExecutionContext } from '@objectstack/spec/kernel'; +import { AnalyticsService } from '../analytics-service.js'; +import { AnalyticsServicePlugin } from '../plugin.js'; +import { installDefaultOrganization } from './default-organization.testkit.js'; + +/** Frozen: Wednesday 2026-09-09 in both UTC and Shanghai, inside Q3. */ +const NOW = new Date('2026-09-09T12:34:56.789Z'); +const CTX = { tenantId: 'org_A' } as ExecutionContext; +const SHANGHAI = 'Asia/Shanghai'; +/** Q3 2026 in Shanghai: its July and October midnights, as UTC instants. */ +const Q3_SHANGHAI = { gte: '2026-06-30T16:00:00.000Z', lt: '2026-09-30T16:00:00.000Z' }; + +const pipeline = DatasetSchema.parse({ + name: 'pipeline_27074', label: 'Pipeline', object: 'opportunity', include: [], + dimensions: [ + { name: 'stage', field: 'stage', type: 'string' }, + { name: 'close_date', field: 'close_date', type: 'date', dateGranularity: 'month' }, + // A second dimension over the SAME field, bucketed by quarter. + { name: 'close_quarter', field: 'close_date', type: 'date', dateGranularity: 'quarter' }, + ], + measures: [{ name: 'cnt', aggregate: 'count' }], +}); + +/** The ObjectQL face over `rows`, recording the filter each engine aggregate received. */ +function service(rows: Array>, closeDateType: 'datetime' | 'date' | undefined, seen: unknown[] = []) { + return new AnalyticsService({ + queryCapabilities: () => ({ nativeSql: false, objectqlAggregate: true, inMemory: false }), + executeAggregate: async (_object: string, opts: { filter?: unknown }) => { seen.push(opts.filter); return rows; }, + sourceFieldMeta: (_o, f) => (f === 'close_date' && closeDateType ? { type: closeDateType } : undefined), + }); +} + +/** The `close_date` bound pair a recorded engine filter carries, wherever it sits in the tree. */ +function closeDateBounds(filter: unknown): Record | undefined { + if (!filter || typeof filter !== 'object') return undefined; + if (Array.isArray(filter)) { + for (const f of filter) { const hit = closeDateBounds(f); if (hit) return hit; } + return undefined; + } + const obj = filter as Record; + if (obj.close_date && typeof obj.close_date === 'object') return obj.close_date as Record; + for (const v of Object.values(obj)) { const hit = closeDateBounds(v); if (hit) return hit; } + return undefined; +} + +beforeEach(() => { + vi.useFakeTimers({ toFake: ['Date'] }); + vi.setSystemTime(NOW); +}); +afterEach(() => { + vi.useRealTimers(); +}); + +describe('#27074 — a windowed dataset answer reports the window in drillRanges', () => { + it('a window on an ungrouped datetime dimension is an entry on every row, at the instants the query filtered by', async () => { + const seen: unknown[] = []; + const result = await service( + [{ stage: 'won', cnt: 3 }, { stage: 'lost', cnt: 2 }], + 'datetime', + seen, + ).queryDataset( + pipeline, + { + dimensions: ['stage'], + measures: ['cnt'], + timeDimensions: [{ dimension: 'close_date', dateRange: 'this_quarter' }], + timezone: SHANGHAI, + }, + CTX, + ); + const entry = { close_date: { field: 'close_date', ...Q3_SHANGHAI } }; + expect(result.drillRanges).toEqual([entry, entry]); + // The very bounds the query handed the engine — one window, not two. + expect(closeDateBounds(seen[0])).toEqual({ $gte: Q3_SHANGHAI.gte, $lt: Q3_SHANGHAI.lt }); + // The equality sidecars are untouched. + expect(result.dimensionFields).toEqual({ stage: 'stage' }); + expect(result.drillRawRows).toEqual([{ stage: 'won' }, { stage: 'lost' }]); + }); + + it('month buckets cut by last_30_days are narrowed to the window at both edges, the middle untouched', async () => { + // Shanghai: today 2026-09-09, so the window is [2026-08-10, 2026-09-10) there. + const result = await service( + [{ close_date: '2026-08', cnt: 4 }, { close_date: '2026-09', cnt: 1 }], + 'datetime', + ).queryDataset( + pipeline, + { + dimensions: ['close_date'], + measures: ['cnt'], + timeDimensions: [{ dimension: 'close_date', dateRange: 'last_30_days' }], + timezone: SHANGHAI, + }, + CTX, + ); + expect(result.drillRanges).toEqual([ + // August's bucket opens 2026-07-31T16:00Z; the window opens later. + { close_date: { field: 'close_date', gte: '2026-08-09T16:00:00.000Z', lt: '2026-08-31T16:00:00.000Z' } }, + // September's bucket closes 2026-09-30T16:00Z; the window closes earlier. + { close_date: { field: 'close_date', gte: '2026-08-31T16:00:00.000Z', lt: '2026-09-09T16:00:00.000Z' } }, + ]); + + const wide = await service( + [{ close_date: '2026-07', cnt: 4 }, { close_date: '2026-08', cnt: 1 }, { close_date: '2026-09', cnt: 1 }], + 'date', + ).queryDataset( + pipeline, + { + dimensions: ['close_date'], + measures: ['cnt'], + timeDimensions: [{ dimension: 'close_date', dateRange: ['2026-07-15', '2026-09-14'] }], + }, + CTX, + ); + expect(wide.drillRanges).toEqual([ + { close_date: { field: 'close_date', gte: '2026-07-15', lt: '2026-08-01' } }, + { close_date: { field: 'close_date', gte: '2026-08-01', lt: '2026-09-01' } }, + { close_date: { field: 'close_date', gte: '2026-09-01', lt: '2026-09-15' } }, + ]); + }); + + it('a window on a field a bucket of ANOTHER dimension groups narrows that bucket entry and adds none', async () => { + // A drill filter keys by field: a second `close_date` entry would replace + // the quarter bucket's bound instead of meeting it. + const result = await service([{ close_date: '2026-Q3', cnt: 2 }], 'date').queryDataset( + pipeline, + { + dimensions: ['close_quarter'], + measures: ['cnt'], + timeDimensions: [{ dimension: 'close_date', dateRange: 'last_30_days' }], + }, + CTX, + ); + expect(result.drillRanges).toEqual([ + { close_quarter: { field: 'close_date', gte: '2026-08-10', lt: '2026-09-10' } }, + ]); + }); + + it('an explicit bare-day window on a datetime field runs to the next UTC midnight, as the shared lowering compiles it', async () => { + const result = await service([{ stage: 'won', cnt: 1 }], 'datetime').queryDataset( + pipeline, + { + dimensions: ['stage'], + measures: ['cnt'], + timeDimensions: [{ dimension: 'close_date', dateRange: ['2026-07-01', '2026-09-30'] }], + timezone: SHANGHAI, + }, + CTX, + ); + expect(result.drillRanges).toEqual([ + { close_date: { field: 'close_date', gte: '2026-07-01T00:00:00.000Z', lt: '2026-10-01T00:00:00.000Z' } }, + ]); + }); + + it('a {token} window resolves as the executor resolves it', async () => { + const result = await service([{ stage: 'won', cnt: 1 }], 'date').queryDataset( + pipeline, + { + dimensions: ['stage'], + measures: ['cnt'], + timeDimensions: [{ dimension: 'close_date', dateRange: ['{month_start}', '{today}'] }], + }, + CTX, + ); + expect(result.drillRanges).toEqual([ + { close_date: { field: 'close_date', gte: '2026-09-01', lt: '2026-09-10' } }, + ]); + }); + + it('a window with no exact half-open spelling gets no entry: an instant end, the last supported day', async () => { + for (const dateRange of [ + ['2026-07-01T00:00:00Z', '2026-09-30T12:00:00Z'], + ['2026-07-01', '9999-12-31'], + ] as Array<[string, string]>) { + const result = await service([{ stage: 'won', cnt: 1 }], 'datetime').queryDataset( + pipeline, + { dimensions: ['stage'], measures: ['cnt'], timeDimensions: [{ dimension: 'close_date', dateRange }] }, + CTX, + ); + expect(result.drillRanges, JSON.stringify(dateRange)).toBeUndefined(); + } + }); + + describe('controls', () => { + it('a selection with no dateRange is unchanged: no drillRanges for an equality-only grouping', async () => { + const result = await service([{ stage: 'won', cnt: 3 }], 'datetime').queryDataset( + pipeline, + { dimensions: ['stage'], measures: ['cnt'], timezone: SHANGHAI }, + CTX, + ); + expect(result.drillRanges).toBeUndefined(); + // …nor for a timeDimensions entry that states only a bucket. + const bucketed = await service([{ close_date: '2026-09', cnt: 3 }], 'datetime').queryDataset( + pipeline, + { dimensions: ['close_date'], measures: ['cnt'], timeDimensions: [{ dimension: 'close_date', granularity: 'month' }] }, + CTX, + ); + expect(bucketed.drillRanges).toEqual([ + { close_date: { field: 'close_date', gte: '2026-09-01T00:00:00.000Z', lt: '2026-10-01T00:00:00.000Z' } }, + ]); + }); + + it('a date field serialises calendar days', async () => { + const result = await service([{ stage: 'won', cnt: 3 }], 'date').queryDataset( + pipeline, + { dimensions: ['stage'], measures: ['cnt'], timeDimensions: [{ dimension: 'close_date', dateRange: 'this_quarter' }] }, + CTX, + ); + expect(result.drillRanges).toEqual([{ close_date: { field: 'close_date', gte: '2026-07-01', lt: '2026-10-01' } }]); + }); + + it('UTC resolves a datetime window at UTC midnight', async () => { + const result = await service([{ stage: 'won', cnt: 3 }], 'datetime').queryDataset( + pipeline, + { dimensions: ['stage'], measures: ['cnt'], timeDimensions: [{ dimension: 'close_date', dateRange: 'this_quarter' }] }, + CTX, + ); + expect(result.drillRanges).toEqual([ + { close_date: { field: 'close_date', gte: '2026-07-01T00:00:00.000Z', lt: '2026-10-01T00:00:00.000Z' } }, + ]); + }); + + it('an unknown field type under a non-UTC zone gets no window entry, as it gets no bucket entry', async () => { + const result = await service([{ stage: 'won', cnt: 3 }], undefined).queryDataset( + pipeline, + { + dimensions: ['stage'], + measures: ['cnt'], + timeDimensions: [{ dimension: 'close_date', dateRange: 'this_quarter' }], + timezone: SHANGHAI, + }, + CTX, + ); + expect(result.drillRanges).toBeUndefined(); + }); + + it('compareTo is unchanged: the comparison column is computed as before, and the row drills the primary window', async () => { + const calls: unknown[] = []; + const svc = new AnalyticsService({ + queryCapabilities: () => ({ nativeSql: false, objectqlAggregate: true, inMemory: false }), + executeAggregate: async (_object: string, opts: { filter?: unknown }) => { + calls.push(opts.filter); + return calls.length === 1 ? [{ stage: 'won', cnt: 5 }] : [{ stage: 'won', cnt: 2 }]; + }, + sourceFieldMeta: (_o, f) => (f === 'close_date' ? { type: 'datetime' } : undefined), + }); + const result = await svc.queryDataset( + pipeline, + { + dimensions: ['stage'], + measures: ['cnt'], + timeDimensions: [{ dimension: 'close_date', dateRange: 'this_quarter' }], + compareTo: { kind: 'previousPeriod' }, + timezone: SHANGHAI, + }, + CTX, + ); + expect(result.rows).toEqual([{ stage: 'won', cnt: 5, cnt__compare: 2 }]); + expect(result.drillRanges).toEqual([{ close_date: { field: 'close_date', ...Q3_SHANGHAI } }]); + }); + }); +}); + +// ── The real engine: the drill list lists what the row counted ─────────────── + +const OBJECT = 'deal_27074'; +const quiet = { debug() {}, info() {}, warn() {}, error() {}, child() { return quiet; } }; + +/** + * Deals either side of each edge of Q3 2026, in Shanghai and in UTC. Shanghai + * is UTC+8 with no DST, so its July midnight is `2026-06-30T16:00Z`. + */ +const DEALS: Array<{ id: string; stage: string; closed_at: string; close_on: string }> = [ + { id: 'd1', stage: 'won', closed_at: '2026-06-30T15:59:59.000Z', close_on: '2026-06-30' }, // Shanghai Jun 30 23:59 + { id: 'd2', stage: 'won', closed_at: '2026-06-30T16:00:00.000Z', close_on: '2026-07-01' }, // Shanghai Jul 1 00:00 + { id: 'd3', stage: 'won', closed_at: '2026-07-15T08:00:00.000Z', close_on: '2026-07-15' }, + { id: 'd4', stage: 'lost', closed_at: '2026-09-30T15:59:59.000Z', close_on: '2026-09-30' }, // Shanghai Sep 30 23:59 + { id: 'd5', stage: 'lost', closed_at: '2026-09-30T16:00:00.000Z', close_on: '2026-10-01' }, // Shanghai Oct 1 00:00 + { id: 'd6', stage: 'won', closed_at: '2026-08-10T02:00:00.000Z', close_on: '2026-08-10' }, + { id: 'd7', stage: 'lost', closed_at: '2026-01-05T02:00:00.000Z', close_on: '2026-01-05' }, + { id: 'd8', stage: 'won', closed_at: '2026-08-03T02:00:00.000Z', close_on: '2026-08-03' }, // August, before last_30_days + { id: 'd9', stage: 'won', closed_at: '2026-11-20T02:00:00.000Z', close_on: '2026-11-20' }, + { id: 'd10', stage: 'lost', closed_at: '2026-09-05T02:00:00.000Z', close_on: '2026-09-05' }, +]; + +const deals = DatasetSchema.parse({ + name: 'deals_27074', label: 'Deals', object: OBJECT, include: [], + dimensions: [ + { name: 'stage', field: 'stage', type: 'string' }, + { name: 'closed', field: 'closed_at', type: 'date', dateGranularity: 'month' }, + { name: 'closed_on', field: 'close_on', type: 'date', dateGranularity: 'month' }, + ], + measures: [{ name: 'cnt', aggregate: 'count' }], +}); + +describe('#27074 — on a real engine, each row drill lists exactly the records the row counted', () => { + let engine: ObjectQL; + const faces: Record<'native' | 'objectql', AnalyticsService> = {} as never; + + beforeAll(async () => { + const driver = new SqlDriver({ client: 'better-sqlite3', connection: { filename: ':memory:' }, useNullAsDefault: true } as never); + engine = new ObjectQL({ logger: quiet } as never); + engine.registerDriver(driver as never, true); + await engine.init(); + engine.registry.registerObject({ + name: OBJECT, + label: 'Deal', + fields: { + stage: { name: 'stage', type: 'text' }, + closed_at: { name: 'closed_at', type: 'datetime' }, + close_on: { name: 'close_on', type: 'date' }, + }, + } as never); + await engine.syncSchemas(); + await installDefaultOrganization(engine); + for (const d of DEALS) await engine.insert(OBJECT, { ...d } as never); + for (const [face, caps] of [ + ['native', undefined], + ['objectql', () => ({ nativeSql: false, objectqlAggregate: true, inMemory: false })], + ] as const) { + const registered: Record = {}; + await new AnalyticsServicePlugin({ ...(caps ? { queryCapabilities: caps } : {}) } as never).init({ + getService: (name: string) => (name === 'data' ? engine : registered[name]), + registerService: (name: string, svc: unknown) => { registered[name] = svc; }, + replaceService: (name: string, svc: unknown) => { registered[name] = svc; }, + hook: () => {}, + logger: quiet, + } as never); + faces[face] = registered.analytics as AnalyticsService; + } + }); + afterAll(async () => { + try { await engine?.destroy(); } catch { /* noop */ } + }); + + /** The ids the drill list for row `i` lists: the row's raw values ANDed with every range, by field. */ + async function drilled(answer: Awaited>, i: number): Promise { + const where: Record = {}; + for (const [dim, field] of Object.entries(answer.dimensionFields ?? {})) where[field] = answer.drillRawRows![i][dim]; + for (const r of Object.values(answer.drillRanges?.[i] ?? {})) where[r.field] = { $gte: r.gte, $lt: r.lt }; + const rows = (await engine.find(OBJECT, { where, fields: ['id'] } as never)) as Array<{ id: string }>; + return rows.map((r) => r.id).sort(); + } + + /** `null`: parity only, the ids the query itself counts being another card's question. */ + const cases: Array<[string, Record, Record | null]> = [ + [ + 'stage × this_quarter on the datetime field, Shanghai', + { dimensions: ['stage'], timeDimensions: [{ dimension: 'closed', dateRange: 'this_quarter' }], timezone: SHANGHAI }, + { won: ['d2', 'd3', 'd6', 'd8'], lost: ['d10', 'd4'] }, + ], + [ + 'stage × an explicit bare-day window on the datetime field, Shanghai', + { dimensions: ['stage'], timeDimensions: [{ dimension: 'closed', dateRange: ['2026-07-01', '2026-09-30'] }], timezone: SHANGHAI }, + // The explicit arm keeps the engine's UTC calendar day (ADR-0053 D-D): + // d2 is Shanghai's July but UTC's June, d5 the other way round. + { won: ['d3', 'd6', 'd8'], lost: ['d10', 'd4', 'd5'] }, + ], + [ + 'month buckets × last_30_days on the datetime field, Shanghai', + { dimensions: ['closed'], timeDimensions: [{ dimension: 'closed', dateRange: 'last_30_days' }], timezone: SHANGHAI }, + // Both in-window months hold a deal, so the answer has the same rows + // whether or not empty buckets are filled. + { '2026-08': ['d6'], '2026-09': ['d10'] }, + ], + [ + 'stage × this_quarter on the date field, UTC', + { dimensions: ['stage'], timeDimensions: [{ dimension: 'closed_on', dateRange: 'this_quarter' }] }, + { won: ['d2', 'd3', 'd6', 'd8'], lost: ['d10', 'd4'] }, + ], + [ + // The query compares a `date` column with the preset's Shanghai-midnight + // instants read as their leading UTC day (`temporalStorageForm`), so it + // counts June 30 and not September 30 (measured, both strategies). The + // drill reports that same storage-form window, so it lists what the row + // counted; which days the query should count is not this card's. + 'stage × this_quarter on the date field, Shanghai', + { dimensions: ['stage'], timeDimensions: [{ dimension: 'closed_on', dateRange: 'this_quarter' }], timezone: SHANGHAI }, + null, + ], + ]; + + for (const face of ['native', 'objectql'] as const) { + for (const [label, selection, expected] of cases) { + it(`${label} (${face})`, async () => { + const answer = await faces[face].queryDataset(deals, { measures: ['cnt'], ...selection } as never, CTX); + const counted: Record = {}; + const listed: Record = {}; + for (const [i, row] of answer.rows.entries()) { + const key = String(row[(selection.dimensions as string[])[0]]); + counted[key] = Number(row.cnt); + listed[key] = await drilled(answer, i); + } + expect(Object.keys(counted).length).toBeGreaterThan(0); + expect(Object.fromEntries(Object.entries(listed).map(([k, ids]) => [k, ids.length]))).toEqual(counted); + if (expected) expect(listed).toEqual(expected); + }); + } + } +}); diff --git a/packages/services/service-analytics/src/analytics-service.ts b/packages/services/service-analytics/src/analytics-service.ts index a42f37de8b9..e7b6e936f75 100644 --- a/packages/services/service-analytics/src/analytics-service.ts +++ b/packages/services/service-analytics/src/analytics-service.ts @@ -98,6 +98,7 @@ import { import { findCrossFieldComparand } from './comparand-shape.js'; import { compileDataset, type CompiledDataset, type RelationshipResolver } from './dataset-compiler.js'; import { bindDatasetFiltersToObjectFields } from './dataset-filter-field-binding.js'; +import { drillWindowBounds } from './drill-window.js'; import { DatasetExecutor, declaredDefaultGranularity, @@ -2923,18 +2924,38 @@ export class AnalyticsService implements IAnalyticsService { // quarter or year got its ranges computed from the dataset's month (or, // when the dataset declared none, dropped entirely) — so drilling a bucket // opened the wrong span, or the chart lost drill-through altogether. + const instantOf = (field: string): boolean | undefined => { + const ftype = this.sourceFieldMeta?.(dataset.object, field)?.type; + if (ftype === 'datetime') return true; + if (ftype === 'date' || rangeTz === 'UTC') return false; + return undefined; // unknown field type under a non-UTC reference tz → omit (superset). + }; const rangeDims: Array<{ d: (typeof selectedDims)[number]; granularity: DateGranularityValue; instant: boolean }> = []; for (const d of selectedDims) { if (!d.field || d.type !== 'date') continue; const granularity = resolveDimensionGranularity(selection, d.name, d.dateGranularity); if (!granularity) continue; - const ftype = this.sourceFieldMeta?.(dataset.object, d.field as string)?.type; - if (ftype === 'datetime') rangeDims.push({ d, granularity, instant: true }); - else if (ftype === 'date') rangeDims.push({ d, granularity, instant: false }); - else if (rangeTz === 'UTC') rangeDims.push({ d, granularity, instant: false }); - // else: unknown field type under a non-UTC reference tz → omit (superset). + const instant = instantOf(d.field as string); + if (instant !== undefined) rangeDims.push({ d, granularity, instant }); + } + // #27074 — the WINDOW drill scope. A `timeDimensions[].dateRange` (a + // report's picked period) filters the query whether or not its dimension + // is grouped, so the drill must stay inside it too: the window rides + // `drillRanges` as the `[gte, lt)` the query filtered by, lowered by the + // query's own steps (`drill-window.ts`), never re-derived. Tokens resolve + // as the executor resolves them, in the same serialisation as the buckets. + const windowTokens = filterTokenContextFrom(context, new Date()); + const windowDims: Array<{ name: string; field: string; gte: string; lt: string }> = []; + for (const td of selection.timeDimensions ?? []) { + if (td.dateRange == null) continue; + const d = dataset.dimensions?.find((dim) => dim.name === td.dimension); + if (!d?.field) continue; + const instant = instantOf(d.field); + if (instant === undefined) continue; + const window = drillWindowBounds(resolveFilterTokens(td.dateRange, windowTokens), rangeTz, instant ? 'datetime' : 'date'); + if (window) windowDims.push({ name: d.name, field: d.field, ...window }); } - if (rangeDims.length && result.rows.length) { + if ((rangeDims.length || windowDims.length) && result.rows.length) { const bound = (ymd: string, instant: boolean): string => instant ? new Date(zonedDateStartToUtcMs(ymd, rangeTz)).toISOString() : ymd; result.drillRanges = result.rows.map((row) => { @@ -2947,6 +2968,19 @@ export class AnalyticsService implements IAnalyticsService { ranges[d.name] = { field: d.field as string, gte: bound(cal.start, instant), lt: bound(cal.end, instant) }; } } + // A window on a field a bucket above already bounds narrows that + // entry: an edge bucket can be wider than the window, and a drill + // filter keys by FIELD, so a second entry on it would replace the + // bucket's bound rather than meet it. One field, one form, so the + // bounds order as text. Any other window is an entry of its own. + for (const w of windowDims) { + const same = Object.values(ranges).filter((r) => r.field === w.field); + if (same.length === 0) ranges[w.name] = { field: w.field, gte: w.gte, lt: w.lt }; + for (const r of same) { + if (w.gte > r.gte) r.gte = w.gte; + if (w.lt < r.lt) r.lt = w.lt; + } + } return ranges; }); } diff --git a/packages/services/service-analytics/src/drill-window.ts b/packages/services/service-analytics/src/drill-window.ts new file mode 100644 index 00000000000..60da29a95b4 --- /dev/null +++ b/packages/services/service-analytics/src/drill-window.ts @@ -0,0 +1,98 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#27074] The drill bounds of a `timeDimensions[].dateRange` window: the + * half-open `[gte, lt)` the dataset answer reports in `drillRanges`, so a drill + * list filters by exactly the window the query filtered by. + * + * ## Why the answer reports it + * + * A report run inside a picked period (`selection.timeDimensions: [{ dimension, + * dateRange }]`) counts only that period's records, but `drillRanges` used to + * cover only the date dimensions the answer GROUPS by. A windowed dimension + * that is not grouped got no entry, so the drill list behind a stage bar + * listed every period's records; and an edge bucket wider than the window + * (a month cut by `last_30_days`) drilled the whole month. Re-deriving the + * window in the browser would be a second date resolver: the server resolves a + * preset at the request time zone's midnight, while a browser macro yields a + * bare day that the engine cuts at UTC midnight on a `datetime` column. + * + * ## ⛔ No second lowering + * + * Every step is the one the query itself takes: + * + * 1. {@link lowerDatasetCompareDateRange}, this package's registered face in + * the shared `dateRange` conformance kit: a preset through + * `resolveAnalyticsDateRangeString` at the reference time zone, an explicit + * `[a, b]` through `explicitDateRangeWindow`, untouched. + * 2. The `{ $gte, $lt }` / `{ $gte, $lte }` pair both strategies build from + * that window, through {@link normalizeDateRangeWindow}, the shared + * lowering. Its whole-day rule turns a bare-day `$lte` into `$lt` the next + * day. It is applied as for a `datetime` column on every column, which + * `calendar-day.ts` records is order-equivalent to `<= day` on a `date` + * column, so one `[gte, lt)` shape serves both. + * 3. `temporalStorageForm`, the rule the engine reads every temporal + * comparand through: a canonical UTC instant for `datetime`, the leading + * `YYYY-MM-DD` for `date`. + * + * A window that cannot be stated as `[gte, lt)` gets no bounds, and the drill + * stays the superset it was: an explicit end that is an instant (the query + * compares `<=` it, which no exclusive bound restates exactly), and an end on + * the last supported day, where the query compiles no upper bound. + */ + +import { temporalStorageForm } from '@objectstack/core'; +import { lowerDatasetCompareDateRange } from './dataset-executor.js'; +import { normalizeDateRangeWindow, type NormalizedFilterNode } from './strategies/filter-normalizer.js'; + +/** How a drill bound is spelled: the storage form of the window's column. */ +export type DrillBoundForm = 'datetime' | 'date'; + +/** A half-open drill bound pair, both ends in one {@link DrillBoundForm}. */ +export interface DrillWindowBounds { + gte: string; + lt: string; +} + +/** The member name the window is lowered under; the bounds do not depend on it. */ +const MEMBER = 'window'; + +/** The whole-day rule on every column — see step 2 of the module note. */ +const WHOLE_DAY = { isDatetimeColumn: () => true }; + +/** + * The `[gte, lt)` a `dateRange` window filtered by, spelled in `form`, or + * `undefined` when the window has no exact half-open spelling. + * + * @param dateRange - The entry's `dateRange`, `{token}` placeholders already + * resolved the way the executor resolves them. + * @param timezone - The reference time zone the query resolved a preset in. + */ +export function drillWindowBounds( + dateRange: string | readonly unknown[], + timezone: string, + form: DrillBoundForm, +): DrillWindowBounds | undefined { + const window = lowerDatasetCompareDateRange(dateRange, timezone); + const bounds = window.endExclusive + ? { $gte: window.start, $lt: window.end } + : { $gte: window.start, $lte: window.end }; + const node = normalizeDateRangeWindow(MEMBER, bounds, WHOLE_DAY); + const leaves: NormalizedFilterNode[] = node?.kind === 'and' ? node.children : node ? [node] : []; + let gte: unknown; + let lt: unknown; + for (const leaf of leaves) { + if (leaf.kind !== 'leaf' || leaf.member !== MEMBER || leaf.values.length !== 1) return undefined; + if (leaf.operator === 'gte' && gte === undefined) gte = leaf.values[0]; + else if (leaf.operator === 'lt' && lt === undefined) lt = leaf.values[0]; + else return undefined; + } + if (gte === undefined || lt === undefined) return undefined; + const spell = (v: unknown): string | undefined => { + const s = temporalStorageForm(v, form); + return typeof s === 'string' ? s : undefined; + }; + const gteSpelled = spell(gte); + const ltSpelled = spell(lt); + return gteSpelled !== undefined && ltSpelled !== undefined ? { gte: gteSpelled, lt: ltSpelled } : undefined; +} diff --git a/packages/spec/src/api/analytics.zod.ts b/packages/spec/src/api/analytics.zod.ts index 429d1a52c90..7e411434d54 100644 --- a/packages/spec/src/api/analytics.zod.ts +++ b/packages/spec/src/api/analytics.zod.ts @@ -180,16 +180,21 @@ export const AnalyticsResultResponseSchema = lazySchema(() => BaseResponseSchema ), drillRanges: z.array(z.record(z.string(), z.object({ field: z.string().describe('The date dimension\'s own `field`'), - gte: z.string().describe('Inclusive lower bound of the bucket'), - lt: z.string().describe('Exclusive upper bound of the bucket'), + gte: z.string().describe('Inclusive lower bound of the range'), + lt: z.string().describe('Exclusive upper bound of the range'), }))).optional().describe( - 'Drill-through sidecar for time buckets, aligned to `rows` by index: date ' - + 'dimension name to the half-open range `[gte, lt)` that row\'s bucket covers. ' - + 'Bounds are `YYYY-MM-DD` calendar days, or ISO-8601 instants at midnight in the ' - + 'reference timezone when the source field is `datetime`; a row whose bucket ' - + 'value is null gets no entry for that dimension. Set only on a dataset answer ' - + 'that has rows and groups by at least one `type: \'date\'` dimension with a ' - + '`field` and a granularity; independent of `dimensionFields`. Absent on a cube ' + 'Drill-through sidecar for date ranges, aligned to `rows` by index: date ' + + 'dimension name to the half-open range `[gte, lt)` that row\'s records fall in ' + + 'on that dimension\'s field. A time bucket gets the span it covers; a ' + + '`timeDimensions[].dateRange` window gets the window the query filtered by on ' + + 'every row, or narrows the bucket entry on the same field to the part inside ' + + 'the window. Bounds are `YYYY-MM-DD` calendar days, or ISO-8601 instants when ' + + 'the source field is `datetime` (a bucket\'s at midnight in the reference ' + + 'timezone); a row whose bucket value is null gets no bucket entry, and a window ' + + 'whose end is an instant or the last supported day gets none. Set only on a ' + + 'dataset answer that has rows and groups by at least one `type: \'date\'` ' + + 'dimension with a `field` and a granularity, or carries a `dateRange` on a ' + + 'dimension with a `field`; independent of `dimensionFields`. Absent on a cube ' + 'query answer.', ), }), diff --git a/packages/spec/src/contracts/analytics-service.ts b/packages/spec/src/contracts/analytics-service.ts index a86b2f7df33..ac281772dfb 100644 --- a/packages/spec/src/contracts/analytics-service.ts +++ b/packages/spec/src/contracts/analytics-service.ts @@ -224,22 +224,38 @@ export interface AnalyticsResult { */ drillRawTotals?: Array>>; /** - * The half-open range `[gte, lt)` each row's time bucket covers, aligned - * to `rows` by index: `drillRanges[i]` maps date dimension NAME → - * `{ field, gte, lt }`, `field` being that dimension's `field`. A bucket - * ("2026-Q2") groups a span of records, so it drills by range rather than - * by the equality `drillRawRows` carries. `gte` and `lt` are `YYYY-MM-DD` - * calendar days, except for a dimension whose source field is `datetime`: - * those are ISO-8601 instants at midnight in the reference timezone (the - * selection's `timezone`, else the request's, else UTC). A row whose - * bucket value is `null` gets no entry for that dimension. + * The half-open range `[gte, lt)` each row's records fall in on a date + * field, aligned to `rows` by index: `drillRanges[i]` maps date dimension + * NAME → `{ field, gte, lt }`, `field` being that dimension's `field`. A + * drill list filters `field` by every entry, so it lists the records the + * row counted. + * + * Two things bound a row on a date field. A time bucket ("2026-Q2") groups + * a span of records, so it drills by range rather than by the equality + * `drillRawRows` carries; its entry is the span the bucket covers. A + * `selection.timeDimensions[].dateRange` window filters every row, so its + * dimension gets an entry on every row: the window the query filtered by, + * a preset as resolved in the reference timezone, an explicit `[a, b]` + * with a bare-day `b` running to the start of the next day. When the + * windowed field is also a bucket's field, the bucket's entry is narrowed + * to the part inside the window (an edge bucket can be wider than it), + * and the window gets no entry of its own. + * + * `gte` and `lt` are `YYYY-MM-DD` calendar days, except for a dimension + * whose source field is `datetime`: those are ISO-8601 instants, a + * bucket's at midnight in the reference timezone (the selection's + * `timezone`, else the request's, else UTC). A row whose bucket value is + * `null` gets no bucket entry. A window with no exact half-open spelling + * gets no entry: an explicit end that is an instant, or one on the last + * supported day. * * Set only on a `queryDataset` answer that has rows and whose selection * groups by at least one `type: 'date'` dimension with a `field` and a - * granularity (the selection's, else the dimension's `dateGranularity`). - * Such a dimension is left out when its source field is neither `date` - * nor `datetime` (or its type is unknown) and the reference timezone is - * not UTC. Independent of `dimensionFields`: a date-only grouping carries + * granularity (the selection's, else the dimension's `dateGranularity`), + * or carries a `dateRange` on a dimension with a `field`. Such a + * dimension is left out when its source field is neither `date` nor + * `datetime` (or its type is unknown) and the reference timezone is not + * UTC. Independent of `dimensionFields`: a date-only grouping carries * `drillRanges` and none of the equality sidecars. */ drillRanges?: Array>;