fix(objectql,metadata-protocol): the data door honours a field's internal flag in its filter, sort, group-by, aggregate, $search and $expand positions (#22646)

.changeset/22646-internal-field-data-door-positions.md

@@ -0,0 +1,39 @@
+---
+'@objectstack/objectql': minor
+'@objectstack/metadata-protocol': minor
+---
+
+fix(objectql,metadata-protocol)!: the data door honours a field's `internal: true` in its filter (related-object conditions included), sort, group-by, aggregate, per-aggregation-filter, `$search` and `$expand` positions at every depth, not only the row (#22646)
+
+Clause-②: no (narrowing)
+
+<!-- adr-0087: not-required (no-migration-prescription) No metadata moves: no spec key, authorable spelling, export or stored shape is removed, renamed or re-shaped, so there is nothing for `objectstack migrate meta` to rewrite. What narrows is a runtime read door — the generic data door stops EVALUATING a field whose existing `internal: true` declaration already withholds it from every row, and the remedy is the declaration the author already wrote, never a rewrite of anyone's code or metadata. No exported declaration is removed or narrowed (not runtime-interface-only or type-surface-only); both packages publish (not unpublished); no ADR-0087 id is named or touched (not registered or already-registered). -->
+
+**BREAKING** (an accept-set narrowing), shipped as `minor` under the launch-window convention for breaking changes.
+
+A field declared `internal: true` is withheld from every generic exit (#21197): the engine omits it from every row, select-named included. The ROW position honoured that; the generic data door's EVALUATE positions did not, so a value the platform withholds from rows could still be learned through a position that answers by its stored value.
+
+**FROM.** On `origin/main`, for a member holding read and for an administrator alike, on the list and query data routes (`GET /api/v1/data/:object`, `POST /api/v1/data/:object/query`):
+
+- A **filter** naming such a field was admitted — the predicate reached the driver, so the row came back only when the guess matched: a confirmation oracle over a withheld value.
+- A **sort** by such a field was admitted — the order leaks the comparative stored value.
+- A **group-by** of, or an **aggregate operand** over, such a field was refused by the engine, but as an undeclared `500 INTERNAL_ERROR` (a bare `Error` with no `code` or `status`), not a declared refusal.
+- A **per-aggregation filter** (`aggregations[].filter`) naming it, and a **filter beside an aggregation**, were admitted — the same oracle at a second filter position.
+- A **`$search`** whose resolved set included such a field (a non-hidden `internal` text column enters the auto-default) scanned it — a cross-field substring match that confirms a guessed prefix.
+- A **nested-relation condition** (a condition on a lookup's related object) naming the related object's `internal` field was evaluated on that object — the parent row came back only when the related value matched, the same oracle one hop away.
+- An **`$expand`** entry's own `where` / `orderBy` on its TARGET object's `internal` field was admitted into the expansion sub-read, at the first level and at every deeper level — the expanded record's presence is an oracle on the related row.
+
+**TO.** Each evaluate position is refused `400 INVALID_FIELD`, a declared ADR-0112 envelope naming the object and the field, before the engine is asked:
+
+- Filter, sort, group-by, aggregate operand and per-aggregation filter are refused at the generic data door (`@objectstack/metadata-protocol`), after the field-existence gates, in the shape and envelope of the stored-metadata body/hash family one axis over. A nested-relation condition inside any filter is judged against the RELATED object's own `internal` fields, and every level of an `$expand` tree has its own filter / sort judged against that level's target object; the refusal names the object that declares the field.
+- The engine's `aggregate()` refusal (`rejectCredentialAggregation`) now carries the `INVALID_FIELD` / 400 envelope instead of the bare `Error`, so the group-by and aggregate-operand positions answer a declared `400` rather than a `500` — and the `secret` / `password` case that shared that bare `Error` is upgraded with it.
+- A `$search` never scans an `internal: true` column: the field is dropped from the set the engine's search expansion resolves over (a withhold, like the auto-default's `hidden` / credential-type exclusions), and an explicit `$searchFields` naming one is refused `400 INVALID_FIELD` at the door rather than silently widened back to the default set.
+
+**Bindings.**
+
+- **Every caller.** The refusal is at the generic data door and takes no user, so an administrator is refused exactly as a member is (#21197). There is no system carve-out at the door.
+- **The engine's privileged consumers are untouched.** The API-key verifier's `where: { key: <hash> }` lookup, the share-link / SCIM / approval-token lookups — all call the engine directly under a system context and never pass the generic data door, so authentication keeps working, exactly as #7823 kept the engine's write results whole while the generic-data-path ingress stripped them.
+- **One flag reader.** Every position judges `collectInternalReadFields` (engine) or its byte-identical `@objectstack/core` twin `collectInternalWriteResponseFields` (the door, which sits below objectql), never a second list.
+- **Unchanged.** An ordinary field in every position is served as before; an object that merely has an `internal` field is served for every other field.
+
+**Scope.** Target-OBJECT exposure through `$expand` is a separate axis (#22661); the cross-object search sweep (`searchAll`) is #22640's region and inherits the engine's search-set withholding through `engine.find` without a change to its own code.

content/docs/api/error-catalog.mdx

@@ -147,6 +147,13 @@ searchable set gets its own message, and so does a value with the wrong shape 
 an array entry that is not a string, or a value that is itself neither a
 comma-separated string nor an array of field names), `groupBy`, and
 `aggregations[].field`.
+The same code also refuses a field that DOES exist but is declared `internal: true`,
+when a read names it in a position that would evaluate its stored value: a filter
+(including a condition on a related object reached through a lookup, and an
+aggregation's own `filter`), a sort, a `groupBy`, an `aggregations[].field`, a
+`searchFields` entry, or an `expand` entry's own filter or sort at any depth. Such a
+field is never returned on a generic read, so it cannot be queried by either — the
+refusal names the object that declares it. Query by another field.
 Off the request path the same code answers `backfillSummaryNulls`'s
 `recomputeUndefinedOnEmpty` (`os migrate summary-nulls
 --recompute-undefined-on-empty object.field`) when an entry is not a roll-up

packages/metadata-protocol/src/protocol.data-door-internal-field-positions.test.ts

@@ -0,0 +1,253 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * [#22646] The generic data door refuses every EVALUATE position that names a
+ * field declared `internal: true`, before the engine is asked — the position
+ * enumeration this card's fix is pinned by.
+ *
+ * A field declared `internal: true` is withheld from every generic exit
+ * (#21197): the engine omits it from every row (a withhold — pinned on the row
+ * position in objectql's `internal-fields.test.ts`). The EVALUATE positions do
+ * NOT follow that by omission — a filter is a confirmation oracle, a sort leaks
+ * the comparative value, a group key IS the value — so this door REFUSES them,
+ * `INVALID_FIELD` / 400, the same envelope and caller-independence as the
+ * stored-metadata body/hash family one axis over.
+ *
+ * Each refusal is paired with a CONTROL on an ordinary field of the SAME object
+ * that must keep working and must reach the engine — a door that refused every
+ * query would pass every "refused" assertion and serve nothing. The door binds
+ * every caller (no context carve-out here); the member/admin end-to-end split
+ * is pinned on a real stack in `@objectstack/dogfood`.
+ *
+ * The position enumeration is the load-bearing shape: a position added later
+ * that does not pass through this door turns a case here red.
+ */
+
+import { describe, expect, it, vi } from 'vitest';
+import type { ServiceObject } from '@objectstack/spec/data';
+import { assertEngineFindOnePredicate } from '@objectstack/metadata-core';
+import { ObjectStackProtocolImplementation } from './protocol.js';
+
+const VAULT: ServiceObject = {
+    name: 'pv_vault',
+    label: 'Vault',
+    fields: {
+        id: { name: 'id', label: 'ID', type: 'text' },
+        name: { name: 'name', label: 'Name', type: 'text' },
+        rank: { name: 'rank', label: 'Rank', type: 'number' },
+        // The flagged column — a one-way digest in a `text` column, the shape
+        // the flag exists for (#7728). Non-hidden, so nothing else withholds it.
+        secret_hash: { name: 'secret_hash', label: 'Secret', type: 'text', internal: true },
+        inner_id: { name: 'inner_id', label: 'Inner', type: 'lookup', reference: 'pv_inner' },
+    },
+} as unknown as ServiceObject;
+
+/** The grand target: two relation hops from `pv_holder`, with its own flagged column. */
+const INNER: ServiceObject = {
+    name: 'pv_inner',
+    label: 'Inner',
+    fields: {
+        id: { name: 'id', label: 'ID', type: 'text' },
+        name: { name: 'name', label: 'Name', type: 'text' },
+        inner_secret: { name: 'inner_secret', label: 'Inner Secret', type: 'text', internal: true },
+    },
+} as unknown as ServiceObject;
+
+const HOLDER: ServiceObject = {
+    name: 'pv_holder',
+    label: 'Holder',
+    fields: {
+        id: { name: 'id', label: 'ID', type: 'text' },
+        name: { name: 'name', label: 'Name', type: 'text' },
+        vault_id: { name: 'vault_id', label: 'Vault', type: 'lookup', reference: 'pv_vault' },
+    },
+} as unknown as ServiceObject;
+
+const SCHEMAS: Record<string, unknown> = { pv_vault: VAULT, pv_holder: HOLDER, pv_inner: INNER };
+const ROWS: Record<string, Record<string, unknown>[]> = {
+    pv_vault: [{ id: 'v1', name: 'alpha', rank: 1, secret_hash: 'h1' }],
+    pv_holder: [{ id: 'o1', name: 'owner', vault_id: 'v1' }],
+    pv_inner: [{ id: 'i1', name: 'inner', inner_secret: 's1' }],
+};
+
+function makeProtocol() {
+    const find = vi.fn(async (object: string, _o: any) => (ROWS[object] ?? []).map((r) => ({ ...r })));
+    const findOne = vi.fn(async (object: string, o: any) => {
+        assertEngineFindOnePredicate(object, o);
+        return { ...(ROWS[object]?.[0] ?? {}) };
+    });
+    const aggregate = vi.fn(async () => [{ n: 1 }]);
+    const count = vi.fn(async () => (ROWS.pv_vault?.length ?? 0));
+    const engine: any = { registry: { getObject: (n: string) => SCHEMAS[n] }, find, findOne, count, aggregate };
+    return { p: new ObjectStackProtocolImplementation(engine), find, aggregate, count };
+}
+
+async function refusal(run: () => Promise<unknown>): Promise<any> {
+    let resolved = false;
+    let caught: any;
+    try { await run(); resolved = true; } catch (e) { caught = e; }
+    expect(resolved, 'expected a refusal, but the query ran').toBe(false);
+    return caught;
+}
+
+const F = 'secret_hash';
+
+describe('[#22646] the data door refuses an `internal: true` field in every evaluate position', () => {
+    const cases: Array<[string, Record<string, unknown>]> = [
+        ['implicit filter', { [F]: 'h1' }],
+        ['filter (OData param)', { filter: JSON.stringify({ [F]: 'h1' }) }],
+        ['filter ($eq, POST body)', { where: { [F]: { $eq: 'h1' } } }],
+        ['filter ($startsWith)', { where: { [F]: { $startsWith: 'h' } } }],
+        ['filter (under $or)', { where: { $or: [{ [F]: 'h1' }, { name: '__none__' }] } }],
+        ['filter (cross-field $field comparand)', { where: { name: { $eq: { $field: F } } } }],
+        ['sort (wire param)', { sort: F }],
+        ['sort (descending)', { sort: `-${F}` }],
+        ['sort (orderBy node)', { orderBy: [{ field: F, order: 'asc' }] }],
+        ['group-by (string)', { groupBy: [F], aggregations: [{ function: 'count', alias: 'n' }] }],
+        ['group-by (object form)', { groupBy: [{ field: F }], aggregations: [{ function: 'count', alias: 'n' }] }],
+        ['aggregate operand', { aggregations: [{ function: 'max', field: F, alias: 'm' }] }],
+        ['aggregation filter', { aggregations: [{ function: 'count', alias: 'n', filter: { [F]: 'h1' } }] }],
+        ['filter beside an aggregation (aggregate where)', { where: { [F]: 'h1' }, aggregations: [{ function: 'count', alias: 'n' }] }],
+    ];
+    for (const [label, query] of cases) {
+        it(`${label} is refused (INVALID_FIELD / 400), engine never asked`, async () => {
+            const { p, find, aggregate, count } = makeProtocol();
+            const err = await refusal(() => p.findData({ object: 'pv_vault', query }));
+            expect(err.code).toBe('INVALID_FIELD');
+            expect(err.status).toBe(400);
+            expect(err.field).toBe(F);
+            expect(err.object).toBe('pv_vault');
+            expect(find).not.toHaveBeenCalled();
+            expect(aggregate).not.toHaveBeenCalled();
+            expect(count).not.toHaveBeenCalled();
+        });
+    }
+
+    it('explicit $searchFields naming the internal field is refused (not silently widened)', async () => {
+        const { p, find } = makeProtocol();
+        const err = await refusal(() => p.findData({ object: 'pv_vault', query: { search: 'h', searchFields: F } }));
+        expect(err.code).toBe('INVALID_FIELD');
+        expect(err.status).toBe(400);
+        expect(err.field).toBe(F);
+        expect(find).not.toHaveBeenCalled();
+    });
+
+    it("a one-level expand's own filter/sort on the TARGET object's internal field is refused", async () => {
+        for (const query of [
+            { expand: { vault_id: { object: 'pv_vault', where: { [F]: 'h1' } } } },
+            { expand: { vault_id: { object: 'pv_vault', orderBy: [{ field: F, order: 'asc' }] } } },
+        ]) {
+            const { p, find } = makeProtocol();
+            const err = await refusal(() => p.findData({ object: 'pv_holder', query }));
+            expect(err.code).toBe('INVALID_FIELD');
+            expect(err.status).toBe(400);
+            expect(err.field).toBe(F);
+            expect(err.object).toBe('pv_vault');
+            expect(find).not.toHaveBeenCalled();
+        }
+    });
+});
+
+describe('[#22646] CONTROL — an ordinary field of the same object is unchanged', () => {
+    it('a filter on `name` is served and reaches the engine', async () => {
+        const { p, find } = makeProtocol();
+        const res: any = await p.findData({ object: 'pv_vault', query: { name: 'alpha' } });
+        expect(find).toHaveBeenCalledTimes(1);
+        expect((res.records ?? res.data ?? res).length).toBe(1);
+    });
+    it('a sort on `name` is served', async () => {
+        const { p, find } = makeProtocol();
+        await p.findData({ object: 'pv_vault', query: { sort: 'name' } });
+        expect(find).toHaveBeenCalledTimes(1);
+    });
+    it('a group-by on `name` is served and reaches the engine', async () => {
+        const { p, aggregate } = makeProtocol();
+        await p.findData({ object: 'pv_vault', query: { groupBy: ['name'], aggregations: [{ function: 'count', alias: 'n' }] } });
+        expect(aggregate).toHaveBeenCalledTimes(1);
+    });
+    it('an aggregate operand over the non-internal numeric `rank` is served', async () => {
+        const { p, aggregate } = makeProtocol();
+        await p.findData({ object: 'pv_vault', query: { aggregations: [{ function: 'max', field: 'rank', alias: 'm' }] } });
+        expect(aggregate).toHaveBeenCalledTimes(1);
+    });
+    it('a $searchFields on the ordinary `name` field is served', async () => {
+        const { p, find } = makeProtocol();
+        await p.findData({ object: 'pv_vault', query: { search: 'al', searchFields: 'name' } });
+        expect(find).toHaveBeenCalledTimes(1);
+    });
+});
+
+/**
+ * Rework round 1 — the positions the first pass did not walk, measured served
+ * on a real engine and SQL driver before this change (fixture objects only):
+ * a NESTED-RELATION condition evaluated on the related object, and an `expand`
+ * tree deeper than one level. Each is refused for a member AND an
+ * administrator: the door takes no caller carve-out, and the context is passed
+ * so a future carve-out would have to defeat these cases to land.
+ */
+const PERSONAS = {
+    member: { userId: 'u_member', roles: [] as string[] },
+    admin: { userId: 'u_admin', roles: ['admin'], isPlatformAdmin: true },
+} as const;
+
+describe('[#22646] relation conditions and every expand level are judged on their own object', () => {
+    const cases: Array<[string, string, Record<string, unknown>, string, string]> = [
+        ['relation filter (nested-relation form)', 'pv_holder', { where: { vault_id: { [F]: 'h1' } } }, 'pv_vault', F],
+        ['relation filter under $or', 'pv_holder', { where: { $or: [{ vault_id: { [F]: 'h1' } }, { name: '__none__' }] } }, 'pv_vault', F],
+        ['relation filter (OData filter param)', 'pv_holder', { filter: JSON.stringify({ vault_id: { [F]: 'h1' } }) }, 'pv_vault', F],
+        ['relation filter beside an aggregation', 'pv_holder', { where: { vault_id: { [F]: 'h1' } }, aggregations: [{ function: 'count', alias: 'n' }] }, 'pv_vault', F],
+        [
+            'expand depth 2: the grand target\'s filter',
+            'pv_holder',
+            { expand: { vault_id: { object: 'pv_vault', expand: { inner_id: { object: 'pv_inner', where: { inner_secret: 's1' } } } } } },
+            'pv_inner',
+            'inner_secret',
+        ],
+        [
+            'expand depth 2: the grand target\'s sort',
+            'pv_holder',
+            { expand: { vault_id: { object: 'pv_vault', expand: { inner_id: { object: 'pv_inner', orderBy: [{ field: 'inner_secret', order: 'asc' }] } } } } },
+            'pv_inner',
+            'inner_secret',
+        ],
+        [
+            'expand depth 1: a relation condition onto the grand target',
+            'pv_holder',
+            { expand: { vault_id: { object: 'pv_vault', where: { inner_id: { inner_secret: 's1' } } } } },
+            'pv_inner',
+            'inner_secret',
+        ],
+    ];
+    for (const [label, object, query, owner, field] of cases) {
+        for (const [persona, context] of Object.entries(PERSONAS)) {
+            it(`${label} — ${persona}: INVALID_FIELD / 400 on '${owner}.${field}', engine never asked`, async () => {
+                const { p, find, aggregate, count } = makeProtocol();
+                const err = await refusal(() => p.findData({ object, query, context }));
+                expect(err.code).toBe('INVALID_FIELD');
+                expect(err.status).toBe(400);
+                expect(err.object).toBe(owner);
+                expect(err.field).toBe(field);
+                expect(find).not.toHaveBeenCalled();
+                expect(aggregate).not.toHaveBeenCalled();
+                expect(count).not.toHaveBeenCalled();
+            });
+        }
+    }
+});
+
+describe('[#22646] CONTROL — ordinary fields through a relation and at depth are unchanged', () => {
+    const served: Array<[string, Record<string, unknown>]> = [
+        ['a relation condition on the related object\'s ordinary `name`', { where: { vault_id: { name: 'alpha' } } }],
+        ['an operator bag on the lookup itself (not a relation condition)', { where: { vault_id: { $eq: 'v1' } } }],
+        ['expand depth 2 with an ordinary filter', { expand: { vault_id: { object: 'pv_vault', expand: { inner_id: { object: 'pv_inner', where: { name: 'inner' } } } } } }],
+    ];
+    for (const [label, query] of served) {
+        for (const [persona, context] of Object.entries(PERSONAS)) {
+            it(`${label} — ${persona}: served, the engine is asked`, async () => {
+                const { p, find } = makeProtocol();
+                await p.findData({ object: 'pv_holder', query, context });
+                expect(find).toHaveBeenCalledTimes(1);
+            });
+        }
+    }
+});