fix(objectql)!: the engine refuses evaluating an internal field for every caller but a system context

.changeset/engine-internal-field-evaluation-gate.md

@@ -0,0 +1,30 @@
+---
+'@objectstack/spec': patch
+'@objectstack/objectql': patch
+'@objectstack/metadata-protocol': patch
+'@objectstack/runtime': patch
+'@objectstack/mcp': patch
+---
+
+fix(objectql)!: the engine refuses filtering, sorting, grouping and aggregating on an `internal: true` field for every caller but a system context
+
+Clause-②: yes (narrowing)
+Metadata-protocol: none
+
+<!-- adr-0087: not-required (no-migration-prescription) No key of any schema is added, removed, renamed or re-shaped, and nothing an author writes is refused: the `internal: true` flag is read as before. The narrowing is a runtime one, on reads made as a user: a query that evaluates a flagged field answers 400 INVALID_FIELD from the engine itself, with the data door's sentence. The published type change adds a member to the `objectql` slot contract and is satisfied by the one implementer, ObjectQL. There is nothing for `objectstack migrate meta` to rewrite and no tombstone; the remedy is the refusal's own sentence (use another field), or running the server's own read as a system context. -->
+Skills-docs: updated here — `content/docs/permissions/system-context.mdx` (row 18e: the engine's read path) and `content/docs/api/error-catalog.mdx` (`INVALID_FIELD`) state who is refused and who is served; no file under `skills/**` teaches filtering on an `internal: true` field, so no skill line changes.
+
+**BREAKING** for one shape only: server code that reads the engine as a user (not as a system context) and filters, sorts, groups or aggregates on a field declared `internal: true`.
+
+- FROM: the engine ran such a read.
+- TO: the engine refuses it with the data door's `400 INVALID_FIELD`, naming the object, the field and the position, before anything is read. A read made as a system context (`isSystem: true`) is served as before, and the flagged column is still never returned on a row.
+
+What to do: run a server-side lookup that matches on a flagged field as a system context, or match on another field.
+
+- **`@objectstack/objectql`**: the `internal: true` evaluate refusal is defined once, in the engine. `find`, `findOne`, `count` and `aggregate` apply it to every caller but a system context, in every position the data door judges: a filter (including a relation hop and a `$field` comparand), a sort, a group key, an aggregation operand and an aggregation's own filter, and the same positions inside `expand`. `ObjectQL#assertNoInternalFieldEvaluated` answers the same question for any caller, for a door that judges before it reads.
+- **`@objectstack/spec`**: new types on the `objectql` slot contract: `QueryEvaluationGate`, `InternalFieldEvaluationQuery` and `InternalFieldEvaluationOptions`. `IObjectQLEngine` extends `QueryEvaluationGate`.
+- **`@objectstack/metadata-protocol`**: the data door asks the engine's refusal instead of keeping its own copy. Its answers are unchanged.
+- **`@objectstack/runtime`**: the HTTP MCP data bridge asks the engine's refusal directly. Its answers are the data door's.
+- **`@objectstack/mcp`**: the stdio data bridge's own copy of the refusal is retired in favour of the engine's. Its answers are the data door's.
+
+The level is patch rather than the minor a new spec type would take: the changeset gate refuses `minor` beside `Metadata-protocol: none`, and nothing an author writes is refused.

content/docs/api/error-catalog.mdx

@@ -158,7 +158,10 @@ when a read names it in a position that would evaluate its stored value: a filte
 aggregation's own `filter`), a sort, a `groupBy`, an `aggregations[].field`, a
 `searchFields` entry, or an `expand` entry's own filter or sort at any depth. Such a
 field is never returned on a generic read, so it cannot be queried by either — the
-refusal names the object that declares it. Query by another field.
+refusal names the object that declares it. Query by another field. The engine
+itself gives the same refusal on its own reads for every caller but a system
+context; server code that matches on such a field runs its lookup as a system
+context.
 A non-blank `search` is refused (`param: 'search'`), not answered with every row,
 when the object's resolved search set is empty: no `searchableFields` entry names a
 stored field, and the default search scans none (`internal: true` fields never

content/docs/permissions/system-context.mdx

@@ -123,6 +123,7 @@ bugs — it is the protection or the side effect that silently does not happen.
 | 18b | Settings namespace read scope not applied | service-settings | Get: a read of `sys_setting`, `sys_setting_audit` or `sys_platform_setting` across every namespace, whatever each namespace's `readPermission` names — the settings service's own reads of its stores take this path. Lose: the per-namespace read gate the generic data API applies to every other caller, the same rule the settings routes enforce | `settingsReadDoorMiddleware` |
 | 18c | **Credential-column filter / sort refusal not applied** — a `where`, `orderBy`, per-aggregation `filter` or `expand` entry naming a `password` / `secret` column is evaluated | objectql | Get: the server's own match on a stored credential value, a lookup that never returns it. Every other caller, administrators included, is refused `400 INVALID_FIELD` for anything but a has-a-value check (ADR-0100). Lose: nothing a wire caller reaches — `isSystem` is never set on an inbound request | `rejectCredentialEvaluation` |
 | 18d | **Stored-metadata type read gate not applied** — a read of `sys_metadata` / `sys_metadata_history` is not narrowed by the capability its row's type requires | objectql | Get: every row of the family on `find` / `findOne` / `count` / `aggregate` — the metadata loader, the `/meta` doors' own store reads, boot (a read with no execution context is not narrowed either). Lose: the narrowing that serves a stored row of a capability-gated type (`datasource`, `external_catalog`) only to holders of the capability the `/meta` type gate names for it | `storedMetadataTypeReadFilter` |
+| 18e | **`internal: true` evaluate refusal not applied on the engine's read path** — a filter, sort, group key, aggregation operand, per-aggregation `filter`, nested-relation condition or `expand` entry naming a field declared `internal: true` is evaluated | objectql | Get: the server's own match on a flagged column (the API-key, share-link, SCIM-credential and approval-token lookups, the better-auth adapter), which still never returns the value. Every other caller of `find` / `findOne` / `count` / `aggregate`, administrators included, is refused `400 INVALID_FIELD`; the data door asks the same judgement for every caller, a system context included. Lose: nothing a wire caller reaches — `isSystem` is never set on an inbound request | `rejectInternalFieldEvaluation` |
 
 ### 2. Write pipeline and data integrity
 
@@ -544,6 +545,7 @@ and CI refuses one that differs from what the tree produces.
 | `packages/objectql/src/engine.ts#insert` | 3 | 21, 22 |
 | `packages/objectql/src/engine.ts#isSystem` | 1 | 64 |
 | `packages/objectql/src/engine.ts#update` | 4 | 19, 20 |
+| `packages/objectql/src/internal-field-evaluation-guard.ts#rejectInternalFieldEvaluation` | 1 | 18e |
 | `packages/objectql/src/stored-metadata-type-read-gate.ts#storedMetadataTypeReadFilter` | 1 | 18d |
 | `packages/observability/src/perf-timing.ts#isPerfDisclosurePrincipal` | 1 | 13 |
 | `packages/plugins/organizations/src/organizations-plugin.ts#start` | 1 | 62 |

packages/core/src/security/second-object-read-exposure.pin.test.ts

@@ -218,6 +218,16 @@ const CALLERS: Record<string, Classification> = {
       "serves nothing of the target — the expanded read itself is the engine's `expandRelatedRecords`. Pinned by " +
       '`packages/objectql/src/credential-evaluation-guard.test.ts` (the expand refusals assert the driver was never asked)',
   },
+  // The `internal: true` evaluate refusal's expand half (QueryEvaluationGate).
+  'packages/objectql/src/internal-field-evaluation-guard.ts': {
+    kind: 'not-a-served-read',
+    reason:
+      "names the object an `expand` entry the caller wrote targets, only to read that object's DECLARED `internal: true` " +
+      'fields from the registry and refuse (`INVALID_FIELD` / 400) a filter or sort key on one; it reads no record and ' +
+      "serves nothing of the target. A door's own exposure question runs first at each hop, through the gate's " +
+      '`beforeRelationTarget` hook (the data door hands it `refuseUnservedRelationTarget`). Pinned by ' +
+      '`packages/objectql/src/internal-field-evaluation-guard.test.ts` (the refusals assert the driver was never asked)',
+  },
   // [#25522 cut 10] The write-path reference check, moved out of engine.ts.
   'packages/objectql/src/engine-referential-integrity.ts': {
     kind: 'not-a-served-read',

packages/mcp/src/stdio-data-bridge.internal-field-evaluation.test.ts

@@ -1,20 +1,20 @@
 // Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
 
 /**
- * The stdio data bridge asks the data door's internal-field gate — the
- * `protocol` service's `assertNoInternalFieldEvaluated` — before its engine
- * reads anything, on `query` (the call's `where` and `orderBy`) and on
- * `aggregate` (`where`, `groupBy`, `aggregations`), and a refusal it throws
- * reaches the caller as thrown: the tool renders its `code`, `status` and
- * sentence, and the engine is never asked.
+ * The stdio data bridge leaves the `internal: true` evaluate refusal to the
+ * ENGINE: it hands the engine the caller's query under the caller's principal
+ * on `query` (`where`, `orderBy`) and on `aggregate` (`where`, `groupBy`,
+ * `aggregations`), never asks a `protocol` service for it (a host may register
+ * none, or a protocol other than the shipped one), and a refusal the engine
+ * throws reaches the caller as thrown: the tool renders its `code`, `status`
+ * and sentence.
  *
- * The gate itself is the protocol's and is pinned there
- * (`@objectstack/metadata-protocol`, the public-face cases beside the data
- * door's position enumeration), and the whole path on a real stack in
- * `@objectstack/dogfood`. This file pins the bridge's half: that it asks, with
- * what, before what — and that a host whose protocol has no such face (or that
- * registers no protocol) is served exactly as before. Refusals are asserted on
- * the envelope, never on a bare throw.
+ * The refusal itself is the engine's one definition and is pinned there
+ * (`@objectstack/objectql`, `internal-field-evaluation-guard.test.ts`, which
+ * also boots a kernel with no protocol service), and the whole path on a real
+ * stack, with and without a protocol service, in `@objectstack/dogfood`. This
+ * file pins the bridge's half. Refusals are asserted on the envelope, never on
+ * a bare throw.
  */
 
 import { describe, it, expect, vi } from 'vitest';
@@ -23,40 +23,41 @@ import { Client } from '@modelcontextprotocol/sdk/client/index.js';
 import { InMemoryTransport } from '@modelcontextprotocol/sdk/inMemory.js';
 import type { ExecutionContext } from '@objectstack/spec/kernel';
 import type { IDataEngine, IMetadataService } from '@objectstack/spec/contracts';
-import { createStdioDataBridge, type StdioProtocolObjectReads } from './stdio-data-bridge.js';
+import { createStdioDataBridge } from './stdio-data-bridge.js';
 import { registerObjectTools } from './mcp-http-tools.js';
 
 const OBJECT = 'ifx_vault';
 const FLAGGED = 'pin';
 const DEF = { name: OBJECT, fields: { name: { type: 'text' }, [FLAGGED]: { type: 'text' } } };
 
-/** The door's refusal as the protocol throws it (fields only; the sentence is the protocol's to pin). */
-function doorRefusal(param: string) {
-  return Object.assign(new Error(`Cannot evaluate '${OBJECT}' by '${FLAGGED}' (${param}): the query was not run.`), {
+/** The engine's refusal (fields only; the sentence is the engine's to pin). */
+function engineRefusal(param: string) {
+  return Object.assign(new Error(`Cannot filter '${OBJECT}' by '${FLAGGED}' (${param}): the query was not run.`), {
     code: 'INVALID_FIELD', status: 400, httpStatus: 400, object: OBJECT, field: FLAGGED, fields: [FLAGGED], param,
   });
 }
 
-/** A stand-in for the protocol's gate: refuses any query whose JSON names the flagged field. */
-function gate() {
-  return vi.fn((_object: string, query: Record<string, unknown>) => {
-    if (JSON.stringify(query).includes(`"${FLAGGED}"`)) throw doorRefusal('filter');
-  });
-}
+const PRINCIPAL = { userId: 'u1', isSystem: false } as unknown as ExecutionContext;
 
-function makeBridge(protocol?: StdioProtocolObjectReads) {
+/** A stand-in engine that refuses any query whose JSON names the flagged field, as the real one does. */
+function makeBridge(withProtocolFace = false) {
+  const refuses = (query: unknown) => {
+    if (JSON.stringify(query).includes(`"${FLAGGED}"`)) throw engineRefusal('filter');
+  };
   const engine = {
-    find: vi.fn(async () => [{ id: 'r1', name: 'alpha' }]),
-    aggregate: vi.fn(async () => [{ n: 1 }]),
+    find: vi.fn(async (_object: string, query: unknown) => { refuses(query); return [{ id: 'r1', name: 'alpha' }]; }),
+    aggregate: vi.fn(async (_object: string, query: unknown) => { refuses(query); return [{ n: 1 }]; }),
   };
   const metadataService = { getObject: vi.fn(async (name: string) => (name === OBJECT ? DEF : undefined)) };
+  // A protocol that still carries a face of that name: the bridge must not ask it.
+  const protocolFace = vi.fn();
   const bridge = createStdioDataBridge({
     engine: engine as unknown as IDataEngine,
     metadataService: metadataService as unknown as IMetadataService,
-    ...(protocol ? { protocol: () => protocol } : {}),
-    resolvePrincipal: async () => ({ userId: 'u1', isSystem: false }) as unknown as ExecutionContext,
+    ...(withProtocolFace ? { protocol: () => ({ assertNoInternalFieldEvaluated: protocolFace }) as never } : {}),
+    resolvePrincipal: async () => PRINCIPAL,
   });
-  return { bridge, engine };
+  return { bridge, engine, protocolFace };
 }
 
 const refusalOf = async (run: () => Promise<unknown>) => {
@@ -66,24 +67,22 @@ const refusalOf = async (run: () => Promise<unknown>) => {
 };
 const REFUSED = { code: 'INVALID_FIELD', status: 400, field: FLAGGED, object: OBJECT };
 
-describe('the stdio bridge asks the protocol\'s internal-field gate before the engine', () => {
-  it('query: the gate is asked with the call\'s where and orderBy, and its refusal stops the read', async () => {
-    const assertNoInternalFieldEvaluated = gate();
-    const { bridge, engine } = makeBridge({ assertNoInternalFieldEvaluated });
+describe('the stdio bridge leaves the internal-field refusal to the engine', () => {
+  it('query: the engine gets the call\'s where and orderBy under the principal, and its refusal reaches the caller', async () => {
+    const { bridge, engine } = makeBridge();
     const where = { [FLAGGED]: { $startsWith: 'x' } };
     const orderBy = [{ field: 'name', order: 'asc' as const }];
     expect(await refusalOf(() => bridge.query(OBJECT, { where, orderBy, limit: 5 }))).toEqual(REFUSED);
-    expect(assertNoInternalFieldEvaluated).toHaveBeenCalledWith(OBJECT, { where, orderBy });
-    expect(engine.find).not.toHaveBeenCalled();
+    expect(engine.find).toHaveBeenCalledTimes(1);
+    expect(engine.find.mock.calls[0]![1]).toMatchObject({ where, orderBy });
+    expect((engine.find.mock.calls[0] as unknown[])[2]).toEqual({ context: PRINCIPAL });
 
     const sorted = [{ field: FLAGGED, order: 'desc' as const }];
     expect(await refusalOf(() => bridge.query(OBJECT, { orderBy: sorted }))).toEqual(REFUSED);
-    expect(engine.find).not.toHaveBeenCalled();
   });
 
-  it('aggregate: the gate is asked with where, groupBy and aggregations, and its refusal stops the aggregate', async () => {
-    const assertNoInternalFieldEvaluated = gate();
-    const { bridge, engine } = makeBridge({ assertNoInternalFieldEvaluated });
+  it('aggregate: the engine gets where, groupBy and aggregations under the principal, and its refusal reaches the caller', async () => {
+    const { bridge, engine } = makeBridge();
     const bodies = [
       { where: { [FLAGGED]: 'x' }, aggregations: [{ function: 'count' as const, alias: 'n' }] },
       { groupBy: [FLAGGED], aggregations: [{ function: 'count' as const, alias: 'n' }] },
@@ -92,39 +91,31 @@ describe('the stdio bridge asks the protocol\'s internal-field gate before the e
     ];
     for (const body of bodies) {
       expect(await refusalOf(() => bridge.aggregate!(OBJECT, body))).toEqual(REFUSED);
-      expect(assertNoInternalFieldEvaluated).toHaveBeenLastCalledWith(OBJECT, {
-        where: (body as { where?: unknown }).where,
-        groupBy: (body as { groupBy?: unknown }).groupBy,
-        aggregations: body.aggregations,
-      });
+      expect(engine.aggregate.mock.lastCall![1]).toMatchObject({ ...body, context: PRINCIPAL });
     }
-    expect(engine.aggregate).not.toHaveBeenCalled();
   });
 
-  it('CONTROL — a query the gate passes reaches the engine unchanged, on both verbs', async () => {
-    const assertNoInternalFieldEvaluated = gate();
-    const { bridge, engine } = makeBridge({ assertNoInternalFieldEvaluated });
-    const where = { name: 'alpha' };
-    await bridge.query(OBJECT, { where, orderBy: [{ field: 'name', order: 'asc' }] });
+  it('never asks a protocol service for it, even one carrying a face of that name', async () => {
+    const { bridge, engine, protocolFace } = makeBridge(true);
+    expect(await refusalOf(() => bridge.query(OBJECT, { where: { [FLAGGED]: 'x' } }))).toEqual(REFUSED);
+    expect(await refusalOf(() => bridge.aggregate!(OBJECT, { groupBy: [FLAGGED], aggregations: [{ function: 'count', alias: 'n' }] }))).toEqual(REFUSED);
+    expect(protocolFace).not.toHaveBeenCalled();
     expect(engine.find).toHaveBeenCalledTimes(1);
-    expect((engine.find.mock.calls[0] as unknown[])[1]).toMatchObject({ where, orderBy: [{ field: 'name', order: 'asc' }] });
-    await bridge.aggregate!(OBJECT, { where, groupBy: ['name'], aggregations: [{ function: 'count', alias: 'n' }] });
     expect(engine.aggregate).toHaveBeenCalledTimes(1);
-    expect(assertNoInternalFieldEvaluated).toHaveBeenCalledTimes(2);
   });
 
-  it('CONTROL — no protocol, or one without the face: served as before (no door to agree with)', async () => {
-    for (const protocol of [undefined, {} as StdioProtocolObjectReads]) {
-      const { bridge, engine } = makeBridge(protocol);
-      await bridge.query(OBJECT, { where: { [FLAGGED]: 'x' } });
-      await bridge.aggregate!(OBJECT, { where: { [FLAGGED]: 'x' }, aggregations: [{ function: 'count', alias: 'n' }] });
-      expect(engine.find).toHaveBeenCalledTimes(1);
-      expect(engine.aggregate).toHaveBeenCalledTimes(1);
-    }
+  it('CONTROL — an ordinary-field query is served by the engine, on both verbs', async () => {
+    const { bridge, engine } = makeBridge();
+    const where = { name: 'alpha' };
+    const rows = await bridge.query(OBJECT, { where, orderBy: [{ field: 'name', order: 'asc' }] });
+    expect(engine.find).toHaveBeenCalledTimes(1);
+    expect(JSON.stringify(rows)).toContain('alpha');
+    await bridge.aggregate!(OBJECT, { where, groupBy: ['name'], aggregations: [{ function: 'count', alias: 'n' }] });
+    expect(engine.aggregate).toHaveBeenCalledTimes(1);
   });
 
-  it('query_records and aggregate_records render the gate\'s refusal as its envelope', async () => {
-    const { bridge, engine } = makeBridge({ assertNoInternalFieldEvaluated: gate() });
+  it('query_records and aggregate_records render the engine\'s refusal as its envelope', async () => {
+    const { bridge } = makeBridge();
     const server = new McpServer({ name: 'pin', version: '0' });
     registerObjectTools(server, bridge);
     const [clientSide, serverSide] = InMemoryTransport.createLinkedPair();
@@ -140,10 +131,8 @@ describe('the stdio bridge asks the protocol\'s internal-field gate before the e
         expect(res.isError, name).toBe(true);
         const { error } = JSON.parse(res.content[0]!.text) as { error: { code: unknown; status: unknown; message: string } };
         expect({ code: error.code, status: error.status }).toEqual({ code: 'INVALID_FIELD', status: 400 });
-        expect(error.message).toBe(doorRefusal('filter').message);
+        expect(error.message).toBe(engineRefusal('filter').message);
       }
-      expect(engine.find).not.toHaveBeenCalled();
-      expect(engine.aggregate).not.toHaveBeenCalled();
     } finally {
       await client.close();
     }