From c63a9f3b9f4568c576e49fb2215ba1cbc7a0a44d Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 5 Aug 2026 00:00:43 +0000 Subject: [PATCH 1/5] fix(driver-memory): refuse the filters the live query path cannot evaluate, compile the one it must (#5324, #5328) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `normalizeFilterCondition` had two ways of not refusing a filter it could not compile, in one `switch`: - `default: result[op] = val` handed every unrecognised `$op` to mingo, which answered with a `MingoError` carrying no `code` and no `status` — outside the ADR-0112 envelope, so a client mistake was served as a 500-shaped body (#5324); - the `$between` arm was written conditionally, so a comparand that was not a two-element array skipped it, the constraint vanished, and `find` returned `[]` (#5328). Opposite symptoms, one cause: the shape that could not be evaluated was not refused. #3948 and #4436 settled that an uncompilable filter is a loud refusal rather than a silent answer; this brings that rule to driver-memory's live query path, reusing `filter-refusal.ts`'s existing `INVALID_FILTER` / 400. `$not` goes the other way. It is a declared combinator (`LOGICAL_OPERATORS`), `cel-to-filter` emits it for a CEL `!expr` RLS scope, and driver-sql, driver-mongodb and this package's own matcher all implement it — but MongoDB has no document-level `$not`, so passing it through meant every negated scope threw. It is compiled to `$nor` with one operand (driver-mongodb's rewrite, #4405), which is NULL-safe by construction and so lands on the #5146 canon. Both filter faces now share ONE shape gate rather than a copy of one check. They had drifted: a malformed `$between` returned NO rows from the live path and EVERY row from the reference matcher. Closes the conformance gap that hid this: `FILTER_LOGIC_CASES` reached this backend through the reference matcher only, which the driver does not call, so the table's `$not` case was green while the same filter through `InMemoryDriver.find` threw. It now runs through the real driver too. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Pbu27iNUfQCHeuS551Rqo7 --- ...y-filter-refuse-what-it-cannot-evaluate.md | 48 +++ .../driver-memory/src/filter-refusal.ts | 283 +++++++++++++++++- .../src/memory-driver-document-not.test.ts | 232 ++++++++++++++ ...ry-driver-filter-logic-conformance.test.ts | 107 +++++++ .../driver-memory/src/memory-driver.ts | 123 +++++--- .../src/memory-empty-field-constraint.test.ts | 22 +- .../src/memory-filter-ast-vocabulary.test.ts | 47 +-- .../memory-filter-vocabulary-refusal.test.ts | 283 ++++++++++++++++++ .../driver-memory/src/memory-matcher.ts | 60 ++-- 9 files changed, 1107 insertions(+), 98 deletions(-) create mode 100644 .changeset/memory-filter-refuse-what-it-cannot-evaluate.md create mode 100644 packages/plugins/driver-memory/src/memory-driver-document-not.test.ts create mode 100644 packages/plugins/driver-memory/src/memory-driver-filter-logic-conformance.test.ts create mode 100644 packages/plugins/driver-memory/src/memory-filter-vocabulary-refusal.test.ts diff --git a/.changeset/memory-filter-refuse-what-it-cannot-evaluate.md b/.changeset/memory-filter-refuse-what-it-cannot-evaluate.md new file mode 100644 index 0000000000..9d2dd05e9b --- /dev/null +++ b/.changeset/memory-filter-refuse-what-it-cannot-evaluate.md @@ -0,0 +1,48 @@ +--- +"@objectstack/driver-memory": patch +--- + +fix(driver-memory): the live query path refuses the filters it cannot evaluate, and compiles the one it must (#5324, #5328) + +**This is an observable behaviour change.** Two filter shapes that used to be +answered *silently* now raise the catalogued `INVALID_FILTER` / 400 every other +filter refusal in this driver and in `driver-sql` already speaks (ADR-0112): + +| filter | before | now | +|---|---|---| +| an operator outside the Filter Protocol — `{ name: { $sounds_like: 'x' } }`, `$elemMatch`, `$size`, `$where`, field-level `$not`, … | handed to mingo, which threw a `MingoError` carrying **no `code` and no `status`** — served as a 500-shaped `{ error }` body | `INVALID_FILTER` / 400, naming the operator, the field and its position | +| a `$between` whose comparand is not `[min, max]` — `{ score: { $between: 5 } }` | the arm was skipped, the constraint **vanished**, and `find` returned `[]` | `INVALID_FILTER` / 400, wording aligned with `driver-sql`'s | + +Two more shapes join them, same cause: an undeclared `$`-combinator in a node +position (`{ $nor: … }`, `{ $where: … }` — `FilterConditionSchema` declares +`$and`/`$or`/`$not` and nothing else), and a combinator operand that is not a +filter condition (`{ $or: 'x' }`, `{ $or: [null] }`, `{ $not: 'x' }`). + +If a query of yours starts returning a 400, it was already broken — it was +returning an empty result set or an uncoded 500 for the same input, and +`driver-sql` was rejecting it. The message names the operator and the path +(`filter.$or[1].$and[0].stage`). + +**`$not` is the opposite change: it now works.** `$not` is a declared combinator +(`LOGICAL_OPERATORS`), `cel-to-filter` emits it for every CEL `!expr` in an RLS +read scope, and `driver-sql` / `driver-mongodb` / this package's own reference +matcher all implement it — but the live query path passed it to mingo, and +MongoDB has no document-level `$not`, so **every query carrying a negated scope +threw** `unknown top level operator: $not`. It is compiled to `$nor` with one +operand, the same rewrite `driver-mongodb` performs, which is NULL-safe by +construction and therefore lands on the answer #5146 ruled canonical. + +Both of this package's filter faces — the live mingo path and the reference +matcher — now share ONE shape gate, so they cannot answer one filter +differently again. They did: given a malformed `$between` the live path returned +NO rows while the matcher returned EVERY row. + +The conformance gap that hid all of this is closed too. `FILTER_LOGIC_CASES` +was run against this backend through the reference matcher only — the driver +does not call it — so the table's `$not` case had been green for as long as it +existed while the same filter through `InMemoryDriver.find` threw. The table now +runs through the real driver, as it does for the other three backends. + +Accepted operators are the spec's `FILTER_OPERATORS`, plus `$regex` (produced by +plugin-auth's ObjectQL adapter, compiled by `driver-sql`) and its `$options` +companion. diff --git a/packages/plugins/driver-memory/src/filter-refusal.ts b/packages/plugins/driver-memory/src/filter-refusal.ts index 200e6f06e0..66c3c3c2bb 100644 --- a/packages/plugins/driver-memory/src/filter-refusal.ts +++ b/packages/plugins/driver-memory/src/filter-refusal.ts @@ -1,7 +1,8 @@ // Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. /** - * The filter refusals this driver raises, in ONE place. + * The filter refusals this driver raises, in ONE place — and, since #5324/#5328, + * the ONE walk that decides which shapes are refused at all. * * Both of this package's filter surfaces refuse the same shapes with the same * wire envelope: the live query path (`memory-driver.ts` → mingo) and the @@ -9,8 +10,17 @@ * conformance suites hold against `driver-sql` and `@objectstack/formula`). They * were two independent code paths with two independent notions of what a filter * may be, which is exactly how #5240's divergence survived unnoticed in-package. + * + * #5240 gave the two faces one refusal by writing the same check twice. That was + * still two implementations of one rule, and the shapes #5324/#5328 measured + * proved how far apart two such implementations drift: given a malformed + * `$between` the live path answered "no rows" while the matcher answered "EVERY + * row" — opposite answers, inside one package, to one filter. So the rule now + * lives in exactly one function, {@link assertFilterConditionShape}, and both + * faces call it before they evaluate anything. */ +import { FILTER_OPERATORS } from '@objectstack/spec/data'; import { StandardErrorCode } from '@objectstack/spec/api'; /** @@ -59,6 +69,24 @@ export function filterArrayReachedDriverError(filters: unknown[]): Error { ); } +/** + * Is `value` a Filter Protocol NODE — the shape `FilterConditionSchema` declares + * for a `where`, for every element of `$and`/`$or`, and for the operand of + * `$not`? + * + * The prototype check is load-bearing, not pedantry, and for the same reason it + * is in `driver-sql`'s twin: a `Date`, a `RegExp`, a `Map` or a class instance + * all satisfy `typeof x === 'object' && !Array.isArray(x)` while enumerating to + * nothing, so accepting them would let a garbage operand read as the empty node + * — which means "matches every row". A filter condition always arrives as JSON + * or as the output of `compileCelToFilter`, i.e. a plain object. + */ +export function isFilterNode(value: unknown): value is Record { + if (value === null || typeof value !== 'object' || Array.isArray(value)) return false; + const proto = Object.getPrototypeOf(value); + return proto === Object.prototype || proto === null; +} + /** * [#5240] Is this field spec `{}` — a field constrained by ZERO operators? * @@ -67,10 +95,7 @@ export function filterArrayReachedDriverError(filters: unknown[]): Error { * not a constraint, and is left to the paths that already handle it. */ export function isEmptyFieldConstraint(spec: unknown): boolean { - if (spec === null || typeof spec !== 'object' || Array.isArray(spec)) return false; - const proto = Object.getPrototypeOf(spec); - if (proto !== Object.prototype && proto !== null) return false; - return Object.keys(spec as Record).length === 0; + return isFilterNode(spec) && Object.keys(spec).length === 0; } /** @@ -101,3 +126,251 @@ export function emptyFieldConstraintError(field: string, path: string): Error { `answered "matches nothing". #5240.`, ); } + +// ── [#5324 / #5328] The vocabulary, and the walk that enforces it ──────────── + +/** + * [#5324] The field-level operators this driver EVALUATES. + * + * Sourced from the spec's own `FILTER_OPERATORS` rather than hand-listed here. + * A private copy of the vocabulary is precisely what let this driver and + * driver-sql accept different operator sets (#3948, and the same note sits over + * `convertConditionToMongo`'s alias fold) — a list written out here would agree + * with the spec on the day it was typed and never again. + * + * Two additions the spec's list does not carry, both deliberate and both + * pre-existing behaviour rather than new capability: + * + * - **`$regex`** — not in `FILTER_OPERATORS`, but really produced: plugin-auth's + * ObjectQL adapter emits `{ field: { $regex: value } }` for a `contains` + * search. `driver-sql` compiles it (to a substring LIKE), `objectql`'s + * `having` allows it, and this driver's matcher implements it. Refusing it + * here would break a live producer. + * - **`$options`** — the regex-flags companion `memory-matcher` reads + * (`new RegExp(target, condition.$options)`) and `objectql`'s `having` skips + * for the same reason. It is a modifier of `$regex`, not a predicate of its + * own. + * + * Everything else is refused. That includes the mingo operators this driver used + * to hand through by accident (`$elemMatch`, `$size`, `$type`, `$mod`, `$where`, + * `$expr`, field-level `$not`) — none of them is in the Filter Protocol, none is + * implemented by the matcher, and `driver-sql` refuses every one. + */ +export const SUPPORTED_FIELD_OPERATORS: ReadonlySet = new Set([ + ...FILTER_OPERATORS, + '$regex', + '$options', +]); + +/** The vocabulary as it appears in a refusal message, in declaration order. */ +const SUPPORTED_FIELD_OPERATOR_LIST = [...SUPPORTED_FIELD_OPERATORS].join(', '); + +/** A short type name for an operand a filter refusal has to describe. */ +function describeFilterOperand(value: unknown): string { + if (value === null) return 'null'; + if (Array.isArray(value)) return 'array'; + const kind = typeof value; + if (kind !== 'object') return kind; + const ctor = (value as { constructor?: { name?: string } }).constructor; + return ctor?.name && ctor.name !== 'Object' ? ctor.name : 'object'; +} + +/** A short, non-throwing rendering of an offending value for a message. */ +function safeShapePreview(value: unknown): string { + try { + const json = JSON.stringify(value); + if (typeof json !== 'string') return typeof value; + return json.length > 80 ? `${json.slice(0, 77)}...` : json; + } catch { + return typeof value; + } +} + +/** + * [#5324] An operator this driver cannot evaluate, in a field constraint. + * + * The leading sentence is `driver-sql`'s, verbatim — one condition, one wording + * (#5240) — and the supported list is generated from + * {@link SUPPORTED_FIELD_OPERATORS} so it cannot drift from what the driver + * actually accepts. + * + * What this replaces: `normalizeFieldOperators` had a `default: result[op] = val` + * arm that handed any unrecognised `$op` to mingo unchanged. mingo then raised + * its own `MingoError` — no `code`, no `status` — which `mapDataError` served + * through its default branch as a bare 500-shaped `{ error }`, OUTSIDE the + * ADR-0112 envelope this driver's every other filter refusal speaks. The refusal + * was happening; only the envelope was lost. `$not` in a document position (the + * shape #5324 was filed on) and `{ name: { $sounds_like: 'x' } }` are the same + * hole seen from two sides. + */ +export function unknownFieldOperatorError(op: string, field: string, path: string): Error { + return unsupportedFilterError( + `Unsupported filter operator "${op}" on field "${field}". ` + + `Supported operators: ${SUPPORTED_FIELD_OPERATOR_LIST}. ` + + `Refused at ${path} rather than handed to the query engine, which answers an unknown ` + + `operator with an error carrying no code and no status — a 500-shaped body for what is a ` + + `400-class client mistake (#5324).`, + ); +} + +/** + * [#5324] A `$`-key in a NODE position that is not a declared combinator. + * + * `FilterConditionSchema` declares exactly three (`LOGICAL_OPERATORS`: + * `$and`, `$or`, `$not`); a node's other keys are field names. `$nor`, `$where`, + * `$expr` and friends were passed through to mingo verbatim by the same + * `result[key] = value` reflex the field-level arm had. + */ +export function unknownLogicalOperatorError(key: string, path: string): Error { + return unsupportedFilterError( + `Unsupported filter combinator "${key}" at ${path}. A filter node's $-prefixed keys are the ` + + `declared logical operators $and, $or and $not (@objectstack/spec LOGICAL_OPERATORS); every ` + + `other key is a field name. It is refused rather than passed through to the query engine, ` + + `which would answer with an uncoded error — or, worse, evaluate an operator the Filter ` + + `Protocol never declared (#5324).`, + ); +} + +/** + * [#5328] `$between` whose comparand is not a two-element `[min, max]` array. + * + * The leading sentence is `driver-sql`'s, verbatim, and this is the ONE wording + * every position in this package uses for the condition (#5240) — the FilterCondition + * `$between` arm, and the QueryAST `between` comparison beside it. + * + * The tail says what this driver used to do, because it is the sharpest + * illustration in the package of why a shape gate belongs in ONE place: one + * malformed filter, three silent answers that did not agree. The live path + * dropped the arm, so the field normalised to `{}` and mingo read it as "matches + * no row"; the matcher's `Array.isArray(target) && …` guard skipped the + * comparison and answered "matches EVERY row"; the AST comparison returned no + * node at all, which the caller reads as "no filter". Not one of them reported + * that the predicate had not been compiled, and `if (!rows.length)` cannot tell + * "genuinely none" from "the range never ran". + */ +export function malformedBetweenError(field: string, value: unknown, path: string): Error { + return unsupportedFilterError( + `Operator "$between" on field "${field}" requires a [min, max] value array. ` + + `Received ${describeFilterOperand(value)} (${safeShapePreview(value)}) at ${path}. ` + + `It is refused rather than skipped: a range that compiles to no predicate answers with a ` + + `row count the author never asked for, and this driver's faces did not even agree on ` + + `WHICH — the live query path returned NO rows, the reference matcher returned EVERY row, ` + + `and a dropped AST comparison would have matched every record (#5328).`, + ); +} + +/** [#5324] `$and`/`$or` take a list of nodes; anything else is refused. */ +export function filterNodeListExpectedError(key: string, value: unknown, path: string): Error { + return unsupportedFilterError( + `Filter combinator "${key}" at ${path} requires an array of filter conditions, but received a ` + + `${describeFilterOperand(value)} (${safeShapePreview(value)}). @objectstack/spec ` + + `FilterConditionSchema declares "${key}" as FilterCondition[].`, + ); +} + +/** [#5324] Every `$and`/`$or` element and every `$not` operand is a node. */ +export function filterNodeExpectedError(value: unknown, path: string): Error { + return unsupportedFilterError( + `Filter node at ${path} is a ${describeFilterOperand(value)} (${safeShapePreview(value)}), not a ` + + `filter condition object. Every element of "$and"/"$or" and the operand of "$not" must be a ` + + `plain object of field constraints (e.g. { "status": "active" }) or nested combinators — ` + + `@objectstack/spec FilterConditionSchema declares this position as a FilterCondition. It is ` + + `refused rather than skipped because skipping it would silently change which rows match.`, + ); +} + +/** + * [#5324 / #5328] The ONE shape gate, walked before either face evaluates. + * + * ## Why up front, and why exhaustive + * + * The same two reasons #5240 wrote down for the matcher, now carrying more + * weight because the walk decides more: + * + * 1. **Evaluation short-circuits, a refusal must not.** Both faces stop at the + * first failing key (`every`/`some`, mingo's own predicate composition), so a + * refusal raised mid-evaluation would fire or not fire depending on the + * RECORD being tested. A malformed filter has to be refused for every record + * or none — otherwise a permission rule is valid or invalid by luck of the + * data. + * 2. **It does not short-circuit on identities either.** `{ $or: [ { a: {} }, {} ] }` + * has a TRUE disjunct that would let an emitter return before it ever reached + * the malformed one. Same argument as `driver-sql`'s `reduceFilterNode`, whose + * doc comment calls this "a gate conditional on evaluation order". + * + * ## What it does NOT do + * + * It decides SHAPE, never MEANING: no verdict, no rewriting, no coercion. Every + * filter that evaluated before this gate existed still evaluates to the same + * rows — the gate only converts shapes that were answered *silently* (an empty + * result set, a full result set, or an uncoded `MingoError`) into a catalogued + * `INVALID_FILTER` / 400. + * + * Deliberately NOT refused, because refusing them would be this driver + * inventing a stricter contract than the backends it must agree with: + * + * - a field spec with NO `$` keys (`{ author: { name: 'x' } }`) — read as a + * deep-equality comparand here and by `driver-mongodb`, which is the reading + * that survives; + * - the MEMBER types of a `$between` array — `driver-sql` checks its arity and + * nothing else (#5041 measured the member case and deliberately left it); + * - a stringified comparand for the `LIKE` family — same, and fail-closed. + */ +export function assertFilterConditionShape(node: unknown, path: string): void { + if (!isFilterNode(node)) return; + for (const [key, value] of Object.entries(node)) { + const here = `${path}.${key}`; + if (key === '$and' || key === '$or') { + if (!Array.isArray(value)) throw filterNodeListExpectedError(key, value, here); + value.forEach((child, index) => { + const childPath = `${here}[${index}]`; + if (!isFilterNode(child)) throw filterNodeExpectedError(child, childPath); + assertFilterConditionShape(child, childPath); + }); + continue; + } + if (key === '$not') { + if (!isFilterNode(value)) throw filterNodeExpectedError(value, here); + assertFilterConditionShape(value, here); + continue; + } + if (key.startsWith('$')) throw unknownLogicalOperatorError(key, here); + assertFieldConstraintShape(key, value, here); + } +} + +/** + * [#5324 / #5328] One field constraint: `{ field: }`. + * + * A spec that is not a plain object is a COMPARAND (implicit equality) and has + * no shape to check. A plain object is an operator map when it carries at least + * one `$` key — the same test `driver-mongodb` and this package's matcher both + * make — and then EVERY key must be an operator this driver evaluates. A mixed + * `{ $eq: 1, name: 'x' }` is refused for the same reason a bare `{ $sounds_like }` + * is: the two faces silently disagreed about it (the matcher ignored the + * non-`$` key, mingo did not). + */ +function assertFieldConstraintShape(field: string, spec: unknown, path: string): void { + if (!isFilterNode(spec)) return; + const keys = Object.keys(spec); + if (keys.length === 0) throw emptyFieldConstraintError(field, path); + if (!keys.some((key) => key.startsWith('$'))) return; + for (const op of keys) { + if (!SUPPORTED_FIELD_OPERATORS.has(op)) throw unknownFieldOperatorError(op, field, path); + if (op === '$between' && !isBetweenComparand(spec[op])) { + throw malformedBetweenError(field, spec[op], `${path}.$between`); + } + } +} + +/** + * [#5328] `$between`'s comparand: a two-element `[min, max]` array. + * + * Arity only — the same condition `driver-sql`'s `$between` arm applies + * (`arr.length !== 2`). The member TYPES are left alone on purpose: ISO date + * strings are a legitimate range on every backend, and tightening beyond + * driver-sql here would replace one divergence with another. + */ +function isBetweenComparand(value: unknown): boolean { + return Array.isArray(value) && value.length === 2; +} diff --git a/packages/plugins/driver-memory/src/memory-driver-document-not.test.ts b/packages/plugins/driver-memory/src/memory-driver-document-not.test.ts new file mode 100644 index 0000000000..22471911bc --- /dev/null +++ b/packages/plugins/driver-memory/src/memory-driver-document-not.test.ts @@ -0,0 +1,232 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#5324] Document-level `$not` on the LIVE query path — the shape the issue was + * filed on. + * + * # Why this is implemented and not refused + * + * #5324 offered both directions and deliberately declined to choose. The + * evidence chooses: `$not` is a DECLARED combinator (`LOGICAL_OPERATORS` in + * `@objectstack/spec/data`, alongside `$and`/`$or`), `driver-sql` compiles it, + * `driver-mongodb` translates it, `memory-matcher` evaluates it, and + * `FILTER_LOGIC_CASES` — the standard every backend is held to — contains a case + * that requires it. Refusing it would have made this driver the only backend + * that cannot run a spec-declared operator, and would have left the conformance + * table with a case it could never pass. "Refuse what you cannot evaluate" has a + * companion clause: what the contract DECLARES, you evaluate. + * + * So the general refusal in `memory-filter-vocabulary-refusal.test.ts` covers + * every operator the Filter Protocol does not declare, and this file covers the + * one it does. + * + * # The rewrite, and why `$nor` + * + * mingo is a MongoDB-semantics engine, and MongoDB has no document-level `$not` + * — `unknown top level operator: $not`, uncoded, was the whole of #5324. The + * negation of a whole condition in MongoDB is `$nor` with a single operand, and + * that is exactly the rewrite `driver-mongodb` performs for the same reason + * (#4405). Nothing else about the condition changes. + * + * # Why the null cases are the load-bearing ones + * + * `cel-to-filter.ts` lowers a CEL `!expr` to `{ $not: {…} }`, which is the + * ordinary product of an RLS read scope — so this operator decides who sees + * which rows. #5146 ruled the JS backends' two-valued reading canonical and + * rewrote `driver-sql`'s SQL to match it, because SQL's `NOT (col = x)` is + * UNKNOWN for a NULL column and a `WHERE` drops the row. `$nor` is total by + * construction and lands on the same answer — asserted below against the exact + * fixture and expectations `memory-matcher-not-null-safe.test.ts` pins, so the + * live path is held to the ruling rather than merely to "it no longer throws". + */ + +import { describe, it, expect, beforeAll } from 'vitest'; +import type { FilterCondition } from '@objectstack/spec/data'; + +import { InMemoryDriver } from './memory-driver.js'; +import { match } from './memory-matcher.js'; + +/** Fields present but null — how a SQL NULL round-trips into a record. */ +const NULLED = [ + { id: '1', stage: 'won', owner: 'u1', amount: 10 }, + { id: '2', stage: 'lost', owner: 'u2', amount: 20 }, + { id: '3', stage: null, owner: 'u1', amount: null }, + { id: '4', stage: null, owner: null, amount: 40 }, +]; + +/** The same rows with the null fields ABSENT — the shape a partial write leaves. */ +const MISSING = [ + { id: '1', stage: 'won', owner: 'u1', amount: 10 }, + { id: '2', stage: 'lost', owner: 'u2', amount: 20 }, + { id: '3', owner: 'u1' }, + { id: '4', amount: 40 }, +]; + +const ALL = ['1', '2', '3', '4']; + +const FIELDS = { + id: { type: 'text', name: 'id' }, + stage: { type: 'text', name: 'stage' }, + owner: { type: 'text', name: 'owner' }, + amount: { type: 'number', name: 'amount' }, +}; + +describe('[#5324] InMemoryDriver.find compiles a document-level $not', () => { + let nulled: InMemoryDriver; + let missing: InMemoryDriver; + + beforeAll(async () => { + nulled = new InMemoryDriver({ persistence: false }); + await nulled.syncSchema('deal', { fields: FIELDS }); + for (const row of NULLED) await nulled.create('deal', { ...row }); + + missing = new InMemoryDriver({ persistence: false }); + await missing.syncSchema('deal', { fields: FIELDS }); + for (const row of MISSING) await missing.create('deal', { ...row }); + }); + + const idsFrom = async (driver: InMemoryDriver, where: unknown): Promise => { + const rows = await driver.find('deal', { object: 'deal', fields: ['id'], where: where as FilterCondition }); + return (rows as Array>).map((r) => String(r.id)).sort(); + }; + + /** + * Both readings of "no value" must give the same answer, and the reference + * matcher must give it too — the same contract + * `memory-matcher-not-null-safe.test.ts` states for its own face, now binding + * on the path that actually serves queries. + */ + const matched = async (where: unknown): Promise => { + const fromNulled = await idsFrom(nulled, where); + const fromMissing = await idsFrom(missing, where); + expect(fromMissing, 'a null field and an absent field must match alike').toEqual(fromNulled); + const reference = NULLED.filter((r) => match(r, where)).map((r) => r.id); + expect(fromNulled, 'the live query path and the reference matcher must agree').toEqual(reference); + return fromNulled; + }; + + describe('the shape #5324 reported — every position, not just the top level', () => { + it('at the top level', async () => { + expect(await matched({ $not: { stage: 'won' } })).toEqual(['2', '3', '4']); + }); + + it('inside a $or branch', async () => { + // The issue measured all three of these throwing `unknown top level + // operator: $not`; `normalizeFilterCondition` passed `$not` through + // wherever it sat, so nesting never helped. + expect(await matched({ $or: [{ $not: { stage: 'won' } }] })).toEqual(['2', '3', '4']); + }); + + it('inside a $and branch', async () => { + expect(await matched({ $and: [{ $not: { stage: 'won' } }] })).toEqual(['2', '3', '4']); + }); + + it('ANDs with its sibling keys', async () => { + expect(await matched({ $not: { stage: 'won' }, owner: 'u1' })).toEqual(['3']); + }); + + it('nested two combinators deep', async () => { + expect(await matched({ $and: [{ $or: [{ $not: { stage: 'won' }, owner: 'u1' }] }] })).toEqual(['3']); + }); + + it('the RLS shape a CEL `!(stage == "won")` scope lowers to', async () => { + // `cel-to-filter.ts` emits exactly this for a negated read scope. On this + // driver — the default for dev and test — it used to be an uncoded throw + // on every query the scope touched, not a wrong row count. + expect(await matched({ $not: { stage: 'won' } })).toHaveLength(3); + }); + }); + + describe('the #5146 canon, now answered by the live path too', () => { + it('$not over multiple keys matches a record missing EITHER', async () => { + expect(await matched({ $not: { stage: 'won', owner: 'u1' } })).toEqual(['2', '3', '4']); + }); + + it('$not of a $or rejects a value-less record whose OTHER branch matches', async () => { + // Record 3 has no stage but owner = 'u1', so the $or holds and the + // negation must reject it. This is the case that forced `driver-sql` to + // compile its NULL guard onto each leaf instead of beside the `NOT`. + expect(await matched({ $not: { $or: [{ stage: 'won' }, { owner: 'u1' }] } })).toEqual(['2', '4']); + }); + + it('$not of a $and matches every record failing either conjunct', async () => { + expect(await matched({ $not: { $and: [{ stage: 'won' }, { owner: 'u1' }] } })).toEqual(['2', '3', '4']); + }); + + it('a double negation is the positive filter again', async () => { + expect(await matched({ $not: { $not: { stage: 'won' } } })).toEqual(['1']); + expect(await matched({ $not: { $not: { stage: 'won' } } })).toEqual(await matched({ stage: 'won' })); + }); + + it('$not of $ne still means "the field IS that value"', async () => { + expect(await matched({ $not: { stage: { $ne: 'won' } } })).toEqual(['1']); + }); + + it('$not of $in matches the value-less records', async () => { + expect(await matched({ $not: { stage: { $in: ['won'] } } })).toEqual(['2', '3', '4']); + }); + + it('$not of an ordering comparison matches the value-less records', async () => { + expect(await matched({ $not: { amount: { $gt: 15 } } })).toEqual(['1', '3']); + }); + + it('$not of $contains matches the value-less records', async () => { + expect(await matched({ $not: { stage: { $contains: 'w' } } })).toEqual(['2', '3', '4']); + }); + + it('$not of a null predicate', async () => { + expect(await matched({ $not: { stage: { $null: true } } })).toEqual(['1', '2']); + expect(await matched({ $not: { stage: { $null: false } } })).toEqual(['3', '4']); + }); + }); + + describe('the boolean identities (#5134)', () => { + it('$not: {} matches nothing — NOT TRUE ≡ FALSE', async () => { + expect(await matched({ $not: {} })).toEqual([]); + }); + + it('$not of an empty $or matches everything', async () => { + expect(await matched({ $not: { $or: [] } })).toEqual(ALL); + }); + }); + + /** + * Measured while verifying this fix, and NOT caused by it: three operators + * answer a value-less field differently on the two faces, with or without a + * `$not` around them. mingo reads `$exists` as key presence and lets `$nin` + * match a missing key; the matcher's `value === undefined` guard and its + * `typeof value !== 'string'` test answer the opposite. + * + * This is a SEMANTIC divergence, not a shape one, so the gate this PR adds + * neither causes nor cures it — a ruling on which reading is canonical belongs + * with the identical matcher-vs-formula divergence already filed as **#5299**, + * where this measurement is recorded. Pinned as measured so the fix that lands + * there has to move these lines deliberately. + */ + describe('known two-face divergences on a value-less field — pinned, see #5299', () => { + const liveVsReference = async (where: unknown) => ({ + live: await idsFrom(nulled, where), + reference: NULLED.filter((r) => match(r, where)).map((r) => r.id), + }); + + it('$exists on a present-but-null field: mingo says "the key is there", the matcher says "no value"', async () => { + expect(await liveVsReference({ stage: { $exists: true } })).toEqual({ + live: ['1', '2', '3', '4'], + reference: ['1', '2'], + }); + }); + + it('$nin on an ABSENT field', async () => { + const live = await idsFrom(missing, { stage: { $nin: ['won'] } }); + const reference = MISSING.filter((r) => match(r, { stage: { $nin: ['won'] } })).map((r) => r.id); + expect({ live, reference }).toEqual({ live: ['2', '3', '4'], reference: ['2'] }); + }); + + it('$notContains on a null field', async () => { + expect(await liveVsReference({ $not: { stage: { $notContains: 'w' } } })).toEqual({ + live: ['1'], + reference: ['1', '3', '4'], + }); + }); + }); +}); diff --git a/packages/plugins/driver-memory/src/memory-driver-filter-logic-conformance.test.ts b/packages/plugins/driver-memory/src/memory-driver-filter-logic-conformance.test.ts new file mode 100644 index 0000000000..9690a3a745 --- /dev/null +++ b/packages/plugins/driver-memory/src/memory-driver-filter-logic-conformance.test.ts @@ -0,0 +1,107 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#5324/#5328] Filter logical-combinator conformance for the LIVE QUERY PATH — + * `InMemoryDriver.find`, through `normalizeFilterCondition` and mingo. + * + * # Why this file exists at all + * + * `FILTER_LOGIC_CASES` is the one standard five filter backends are held to + * (`@objectstack/spec/data`, #3774). Four of them ran it through the code a real + * query executes: `driver-sql` compiles it to SQL, `driver-sqlite-wasm` runs + * that SQL on sql.js, `driver-mongodb` translates and executes it, and + * `service-analytics` lowers it into its read-scope SQL. + * + * `driver-memory` ran it through `memory-matcher` ONLY + * (`memory-matcher-or-semantics.test.ts`). That file is not a driver test: the + * driver does not call `match()` — it imports exactly one symbol from that + * module, `getValueByPath`, and filters with mingo instead. So this backend's + * half of the conformance table was measured against a REFERENCE implementation + * while the half users actually run was never executed against the standard once. + * + * The cost was not hypothetical. The table's `$not ANDs with its sibling keys + * inside a branch` case was green here for as long as it has existed, while the + * same filter through `InMemoryDriver.find` threw `unknown top level operator: + * $not` — MongoDB has no document-level `$not`, so mingo has none either (#5324). + * A conformance suite that green-lights an operator the driver cannot run is + * worse than no suite: it is a gate reporting coverage it does not have, which + * is the "declared ≠ enforced" shape Prime Directive #10 names. + * + * So the gap is closed the way the other three backends close it — by running + * the table through the thing that serves queries. `memory-matcher-or-semantics` + * stays: the matcher is still the reference evaluator, and holding BOTH faces to + * the same table is what makes "this package has two filter surfaces" a + * statement someone can check. + */ + +import { describe, it, expect, beforeAll } from 'vitest'; +import { FILTER_LOGIC_CASES, FILTER_LOGIC_ROWS } from '@objectstack/spec/data'; +import type { FilterCondition } from '@objectstack/spec/data'; + +import { InMemoryDriver } from './memory-driver.js'; +import { match } from './memory-matcher.js'; + +const TABLE = 'conformance'; + +describe('[#5324] InMemoryDriver.find — filter logic conformance (the LIVE query path)', () => { + let driver: InMemoryDriver; + + beforeAll(async () => { + driver = new InMemoryDriver({ persistence: false }); + await driver.connect(); + // Every fixture column is a plain string — the shared table keeps its + // predicates boring on purpose, so nothing here is about coercion. The + // declaration is still made, because that is how a real object reaches the + // driver and how its field kinds are resolved (#4047). + await driver.syncSchema(TABLE, { + fields: { + id: { type: 'text', name: 'id' }, + a: { type: 'text', name: 'a' }, + b: { type: 'text', name: 'b' }, + c: { type: 'text', name: 'c' }, + owner: { type: 'text', name: 'owner' }, + status: { type: 'text', name: 'status' }, + parent_object: { type: 'text', name: 'parent_object' }, + parent_id: { type: 'text', name: 'parent_id' }, + }, + }); + for (const row of FILTER_LOGIC_ROWS) await driver.create(TABLE, { ...row }); + }); + + const ids = async (where: FilterCondition): Promise => { + const rows = await driver.find(TABLE, { object: TABLE, fields: ['id'], where }); + return (rows as Array>).map((r) => String(r.id)).sort((x, y) => x.localeCompare(y)); + }; + + for (const c of FILTER_LOGIC_CASES) { + it(c.name, async () => { + expect(await ids(c.filter), c.note).toEqual([...c.expected]); + }); + } + + /** + * The fixture as a whole, so a case that returns nothing because the seed + * failed cannot read as a case that correctly excluded everything. + */ + it('the fixture really is all four rows', async () => { + expect(await ids({})).toEqual(['1', '2', '3', '4']); + }); + + /** + * The two faces, on the same table, in one assertion. + * + * `memory-matcher-or-semantics.test.ts` already holds the matcher to these + * cases and this file holds the driver to them, so both being green already + * implies agreement. Asserting it directly is still worth one test: it is the + * invariant #5240 established for this package ("a backend whose two halves + * disagree about what a filter MEANS is exactly the divergence the ruling + * closes"), and stated here it survives either suite being edited. + */ + it('both filter faces answer the whole table identically', async () => { + for (const c of FILTER_LOGIC_CASES) { + const live = await ids(c.filter); + const reference = FILTER_LOGIC_ROWS.filter((r) => match(r, c.filter)).map((r) => r.id); + expect(live, `${c.name}: the live query path and the reference matcher disagree`).toEqual(reference); + } + }); +}); diff --git a/packages/plugins/driver-memory/src/memory-driver.ts b/packages/plugins/driver-memory/src/memory-driver.ts index 706b785a4e..c3f386fd36 100644 --- a/packages/plugins/driver-memory/src/memory-driver.ts +++ b/packages/plugins/driver-memory/src/memory-driver.ts @@ -7,9 +7,11 @@ import { Logger, createLogger, nextUtcCalendarDay } from '@objectstack/core'; import { Query, Aggregator } from 'mingo'; import { getValueByPath } from './memory-matcher.js'; import { - emptyFieldConstraintError, + assertFilterConditionShape, filterArrayReachedDriverError, - isEmptyFieldConstraint, + malformedBetweenError, + unknownFieldOperatorError, + unknownLogicalOperatorError, unsupportedFilterError, } from './filter-refusal.js'; import { @@ -720,6 +722,12 @@ export class InMemoryDriver implements IDataDriver { return { [op]: conditions }; } // MongoDB/FilterCondition format: { field: value } or { field: { $op: value } } + // [#5324/#5328] Shape first, then translate — the SAME gate the reference + // matcher runs (`filter-refusal.ts`), so the two faces cannot answer one + // filter differently again. It must run before `normalizeFilterCondition` + // and not inside it: the translator recurses per key and would therefore + // refuse or not refuse depending on where in the tree it gave up. + assertFilterConditionShape(filters, 'filter'); // Translate non-standard operators ($contains, $notContains, etc.) to Mingo-compatible format return this.normalizeFilterCondition(filters, object); } @@ -795,10 +803,11 @@ export class InMemoryDriver implements IDataDriver { : { $gte: store(value[0]), $lte: store(value[1]) }, }; } - throw unsupportedFilterError( - `"between" on field "${field}" needs a two-element array, got ` + - `${JSON.stringify(value)}. Returning no predicate would silently match every record.`, - ); + // [#5328] One condition, one wording — the same refusal the + // FilterCondition `$between` arm raises. They used to differ, which is + // how a caller reading two messages could believe they had hit two + // different problems. + throw malformedBetweenError(field, value, `filter.${field}.between`); default: // Was `return null`, which the caller dropped — so an operator this // driver cannot express narrowed nothing instead of erroring. driver-sql @@ -815,6 +824,15 @@ export class InMemoryDriver implements IDataDriver { * Normalize a FilterCondition object by converting non-standard $-prefixed * operators ($contains, $notContains, $startsWith, $endsWith, $between, $null) * to Mingo-compatible equivalents ($regex, $gte/$lte, null checks). + * + * [#5324/#5328] TRANSLATION ONLY. Every shape decision — the operator + * vocabulary, `$between`'s arity, what may sit in a node position — was made + * by `assertFilterConditionShape` before this ran, on the whole tree at once. + * The refusals still written here are the totality floor a translator owes + * itself (`driver-sql` keeps its emitter's `default: throw` beside + * `reduceFilterNode` for the same reason): unreachable through + * `convertToMongoQuery`, and the honest answer if this method is ever called + * from somewhere new. */ private normalizeFilterCondition(filter: Record, object?: string, path = 'filter'): Record { const result: Record = {}; @@ -823,37 +841,48 @@ export class InMemoryDriver implements IDataDriver { for (const key of Object.keys(filter)) { const value = filter[key]; const here = `${path}.${key}`; - // [#5240] `{ field: {} }` is refused in EVERY position, before mingo sees - // it. Left alone it normalises to `{ field: {} }`, which mingo reads as - // "the field deep-equals the empty document" — a filter that matches - // nothing in ordinary data and is therefore indistinguishable, from the - // outside, from the FALSE the reference matcher answered. Neither is what - // the author meant, and driver-sql read the same shape as TRUE inside a - // combinator. Refused rather than reinterpreted; see the ruling on #5240. - if (isEmptyFieldConstraint(value) && !key.startsWith('$')) { - throw emptyFieldConstraintError(key, here); - } // Recurse into logical operators if (key === '$and' || key === '$or') { - result[key] = Array.isArray(value) - ? value.map((child: any, i: number) => this.normalizeFilterCondition(child, object, `${here}[${i}]`)) - : value; + if (!Array.isArray(value)) throw unknownLogicalOperatorError(key, here); + result[key] = value.map((child: any, i: number) => this.normalizeFilterCondition(child, object, `${here}[${i}]`)); continue; } if (key === '$not') { - result[key] = value && typeof value === 'object' - ? this.normalizeFilterCondition(value, object, here) - : value; - continue; - } - // Skip $-prefixed keys that aren't field names (already handled or unknown) - if (key.startsWith('$')) { - result[key] = value; + // [#5324] The whole point of the issue. `$not` is a declared combinator + // (spec `LOGICAL_OPERATORS`), `driver-sql` compiles it, `memory-matcher` + // evaluates it, and `cel-to-filter` EMITS it — a CEL `!expr` in an RLS + // read scope lowers to `{ $not: {…} }`. Passing it through unchanged + // meant mingo received a document-level `$not`, which MongoDB does not + // have: `unknown top level operator: $not`, uncoded, on every query + // carrying a negated scope. + // + // `$nor` with one operand IS the document-level negation in MongoDB, and + // is what `driver-mongodb` rewrites to for the identical reason (#4405). + // It is also NULL-safe by construction, which is the semantics #5146 + // ruled canonical: a row whose field is null or missing does not satisfy + // the inner condition, so `$nor` admits it — the same answer this + // package's matcher and `@objectstack/formula` give, and the one + // driver-sql was rewritten to match. + // + // At most one `$not` per node (it is one object key), so this never + // overwrites a sibling `$nor`, and an input `$nor` cannot reach here — + // the shape gate refuses undeclared combinators. + if (!value || typeof value !== 'object') throw unknownLogicalOperatorError(key, here); + result.$nor = [this.normalizeFilterCondition(value, object, here)]; continue; } + if (key.startsWith('$')) throw unknownLogicalOperatorError(key, here); // Field-level: value may be primitive (implicit eq) or operator object if (value && typeof value === 'object' && !Array.isArray(value) && !(value instanceof Date) && !(value instanceof RegExp)) { - const normalized = this.normalizeFieldOperators(value, this.temporalKind(object, key)); + // A field spec with no `$` keys is a nested-object COMPARAND, not an + // operator map — mingo compares it structurally, `driver-mongodb` says + // so explicitly, and the matcher deep-equals it. Handing it to the + // operator translator would read its field names as operators. + if (!Object.keys(value).some((k) => k.startsWith('$'))) { + result[key] = value; + continue; + } + const normalized = this.normalizeFieldOperators(value, this.temporalKind(object, key), key, here); // Handle multiple regex conditions on the same field (e.g. $startsWith + $endsWith) if (normalized._multiRegex) { const regexConditions: Record[] = normalized._multiRegex; @@ -892,8 +921,11 @@ export class InMemoryDriver implements IDataDriver { * Convert non-standard field operators to Mingo-compatible format. * When multiple regex-producing operators appear on the same field * (e.g. $startsWith + $endsWith), they are combined via $and. + * + * `field` and `path` are carried only so a refusal can name the position it + * refused — the vocabulary itself is enforced one level up (#5324). */ - private normalizeFieldOperators(ops: Record, kind?: TemporalFieldKind): Record { + private normalizeFieldOperators(ops: Record, kind?: TemporalFieldKind, field = '', path = 'filter'): Record { const store = (v: any) => coerceTemporalValue(v, kind); const result: Record = {}; const regexConditions: Record[] = []; @@ -913,15 +945,20 @@ export class InMemoryDriver implements IDataDriver { case '$endsWith': regexConditions.push({ $regex: new RegExp(`${this.escapeRegex(val)}$`, 'i') }); break; - case '$between': - if (Array.isArray(val) && val.length === 2) { - result.$gte = store(val[0]); - // Bare-day max → half-open, inheriting `$lte`'s whole-day rule (#4042). - const betweenNextDay = nextUtcCalendarDay(val[1]); - if (betweenNextDay != null) result.$lt = store(betweenNextDay); - else result.$lte = store(val[1]); - } + case '$between': { + // [#5328] The arm used to be CONDITIONAL — a comparand that was not a + // two-element array skipped it and wrote nothing, so the field + // normalised to `{}` and mingo read that as "matches no row". The + // range simply vanished, and no one was told. The shape gate refuses + // it now; this throw is the totality floor. + if (!Array.isArray(val) || val.length !== 2) throw malformedBetweenError(field, val, `${path}.$between`); + result.$gte = store(val[0]); + // Bare-day max → half-open, inheriting `$lte`'s whole-day rule (#4042). + const betweenNextDay = nextUtcCalendarDay(val[1]); + if (betweenNextDay != null) result.$lt = store(betweenNextDay); + else result.$lte = store(val[1]); break; + } case '$lte': { // A bare-day upper bound means "through that whole day" (#4042; the // driver-sql twin is #3777). Order-equivalent to `<=` for plain @@ -946,9 +983,19 @@ export class InMemoryDriver implements IDataDriver { case '$in': case '$nin': result[op] = store(val); break; - default: + // Evaluated by mingo under the same name. `$exists` is a presence + // predicate, `$regex`/`$options` a pattern and its flags — none of them + // is a comparand, so none takes the field's storage form (#4047). + case '$exists': case '$regex': case '$options': result[op] = val; break; + default: + // [#5324] Was `result[op] = val` — a GENERIC passthrough that handed + // every unrecognised `$op` to mingo, which answered with a bare + // `MingoError` (no `code`, no `status`) and so escaped the ADR-0112 + // envelope as a 500-shaped body. The vocabulary gate refuses these + // before translation; this throw is the totality floor. + throw unknownFieldOperatorError(op, field, path); } } diff --git a/packages/plugins/driver-memory/src/memory-empty-field-constraint.test.ts b/packages/plugins/driver-memory/src/memory-empty-field-constraint.test.ts index 7ad35d3410..f5795dc36c 100644 --- a/packages/plugins/driver-memory/src/memory-empty-field-constraint.test.ts +++ b/packages/plugins/driver-memory/src/memory-empty-field-constraint.test.ts @@ -74,6 +74,9 @@ describe('[#5240] InMemoryDriver (live mingo path) refuses a zero-operator field return rows.map((r: any) => String(r.id)).sort(); }; + /** The same filter through the OTHER face, so an answer can be compared. */ + const matchedIds = (filter: unknown): string[] => ROWS.filter((r) => match(r, filter)).map((r) => r.id); + const refusalOf = async (where: unknown): Promise => { try { await ids(where); @@ -116,14 +119,17 @@ describe('[#5240] InMemoryDriver (live mingo path) refuses a zero-operator field it('a $-key whose value is an empty object is NOT treated as a field constraint', async () => { // `{ $not: {} }` is the #5134 identity (NOT TRUE ≡ FALSE), not this shape, - // so the #5240 gate must not claim it. On this path it still fails — but - // for a pre-existing and entirely separate reason: the live query path - // hands `$not` straight to mingo, which has no document-level `$not` at - // all (measured while verifying this issue's premise; filed as #5324). - // Pinned as the CURRENT truth, so #5324's fix is the thing that changes - // it and this suite is not silently asserting a behaviour nobody has. - await expect(ids({ $not: {} })).rejects.toThrow(/unknown top level operator/); - await expect(ids({ $not: {} })).rejects.not.toThrow(/zero operators/); + // so the #5240 gate must not claim it. + // + // [#5324] This assertion used to pin a FAILURE — `unknown top level + // operator: $not`, mingo's uncoded error, because the live query path + // handed a document-level `$not` straight to a MongoDB engine that has no + // such operator. Now that the path compiles `$not` to `$nor`, the identity + // is the answer it was always supposed to be, and it agrees with + // driver-sql (#5134: NOT of a TRUE group is FALSE → the FALSE constant) + // and with this package's own matcher. + expect(await ids({ $not: {} })).toEqual([]); + expect(matchedIds({ $not: {} })).toEqual([]); }); }); }); diff --git a/packages/plugins/driver-memory/src/memory-filter-ast-vocabulary.test.ts b/packages/plugins/driver-memory/src/memory-filter-ast-vocabulary.test.ts index 1d8fd6201e..ed87159062 100644 --- a/packages/plugins/driver-memory/src/memory-filter-ast-vocabulary.test.ts +++ b/packages/plugins/driver-memory/src/memory-filter-ast-vocabulary.test.ts @@ -62,6 +62,16 @@ describe('InMemoryDriver filter vocabulary ↔ VALID_AST_OPERATORS', () => { */ const findAuthored = (where: unknown) => find(parseFilterAST(where)); + /** The refusal itself, so its ADR-0112 envelope can be asserted (#5324). */ + const refusalOf = async (run: () => Promise): Promise => { + try { + await run(); + } catch (e) { + return e as Error & { code?: string; status?: number }; + } + throw new Error('expected the driver to refuse this filter, but it resolved'); + }; + it('reads a non-empty operator set from the spec', () => { // Guards every assertion below from passing vacuously. expect(VALID_AST_OPERATORS.size).toBeGreaterThan(0); @@ -106,26 +116,29 @@ describe('InMemoryDriver filter vocabulary ↔ VALID_AST_OPERATORS', () => { // `isFilterAST` and refuses the authored array a layer earlier. #3948's // condition holds either way: it THROWS, never a match-everything. // - // What it does NOT yet do on this path is speak the ADR-0112 envelope — - // the object path hands an unknown `$op` straight to mingo, which raises a - // bare `MingoError` with no `code`/`status`. That gap is the general form - // of #5324 (same `normalizeFilterCondition` passthrough, `$not` being the - // instance filed there) and is NOT a #5158 regression: the array path that - // used to carry the envelope for this input is what #5158 deleted, and the - // object path has always answered this way. - await expect(findAuthored([['name', 'sounds_like', 'alpha']])).rejects.toThrow(); + // [#5324] And it now throws in the ADR-0112 envelope. This assertion was + // relaxed to a bare `.rejects.toThrow()` while the object path handed an + // unknown `$op` straight to mingo, whose `MingoError` carries no `code` and + // no `status` — served as a 500-shaped body for a client mistake. Tightened + // back here, which is the assertion the relaxed one was a placeholder for. + const err = await refusalOf(() => findAuthored([['name', 'sounds_like', 'alpha']])); + expect(err.code).toBe('INVALID_FILTER'); + expect(err.status).toBe(400); + expect(err.message).toContain('$sounds_like'); }); - it('a malformed between emits no predicate on this backend — the #5328 divergence', async () => { - // driver-sql THROWS on `{ score: { $between: 5 } }`; this driver's - // `normalizeFilterCondition` skips the arm entirely and the field - // normalises to `{}`, which mingo evaluates as "matches nothing". Two - // backends, one filter, two answers — filed as #5328, not fixed here. + it('refuses a malformed between instead of emitting no predicate — #5328', async () => { + // driver-sql THROWS on `{ score: { $between: 5 } }`; this driver used to + // skip the arm entirely, leaving the field normalised to `{}` which mingo + // evaluates as "matches nothing". One filter, two backends, two answers. // - // Pinned as-is so the divergence is visible rather than folklore. It is - // pre-existing: the loud refusal this test used to observe belonged to the - // ARRAY path, which #5158 deleted; the object path never had one. - await expect(findAuthored([['score', 'between', 5]])).resolves.toEqual([]); + // This assertion was pinned at `resolves.toEqual([])` to keep the + // divergence visible rather than folklore, with a note pointing at #5328. + // #5328 is fixed, so it is pinned at the answer instead. + const err = await refusalOf(() => findAuthored([['score', 'between', 5]])); + expect(err.code).toBe('INVALID_FILTER'); + expect(err.status).toBe(400); + expect(err.message).toContain('[min, max]'); }); it('still honours a well-formed logical node', async () => { diff --git a/packages/plugins/driver-memory/src/memory-filter-vocabulary-refusal.test.ts b/packages/plugins/driver-memory/src/memory-filter-vocabulary-refusal.test.ts new file mode 100644 index 0000000000..df630dd1ab --- /dev/null +++ b/packages/plugins/driver-memory/src/memory-filter-vocabulary-refusal.test.ts @@ -0,0 +1,283 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#5324/#5328] This driver refuses what it cannot evaluate — loudly, in the + * ADR-0112 envelope, on BOTH of its filter faces. + * + * # One defect, two directions + * + * `normalizeFilterCondition` had two ways of not refusing a filter it could not + * compile, in one `switch`: + * + * | shape | before | driver-sql, same input | + * |---|---|---| + * | unknown `$op` (`default: result[op] = val`) | handed to mingo → `MingoError` with no `code`, no `status` → a 500-shaped body | `INVALID_FILTER` / 400 | + * | malformed `$between` (arm written conditionally) | the whole constraint vanished; `find` returned `[]` | `INVALID_FILTER` / 400 | + * + * One silently became a 500, the other silently became an empty result set — + * opposite directions, one cause: **the shape that could not be evaluated was + * not refused.** #3948 and #4436 already settled that an uncompilable filter is + * a loud refusal rather than a silent answer, and #5240 settled that it speaks + * one envelope; this is that rule reaching driver-memory's live query path. + * + * # Why "unknown operator" is written generically + * + * #5324 was filed on `$not`, which is a special case — a DECLARED combinator + * this driver's live path did not implement (fixed by compiling it, see + * `memory-driver-document-not.test.ts`, not by refusing it). The leak it was + * filed on is general: `default:` passed EVERY unrecognised `$op` through, so + * `{ name: { $sounds_like: 'x' } }` escaped the envelope by the identical route. + * A `$not`-shaped fix would have left that half a 500. + * + * # Both faces + * + * Each case is asserted through `InMemoryDriver.find` (mingo) AND through + * `match` (the reference matcher). They are one gate now, and this is the test + * that says so — a regression that re-forks them fails here rather than being + * discovered by a conformance table that only runs one of them (#5240). + */ + +import { describe, it, expect, beforeEach } from 'vitest'; +import type { FilterCondition } from '@objectstack/spec/data'; + +import { InMemoryDriver } from './memory-driver.js'; +import { match } from './memory-matcher.js'; + +interface WireBearingError extends Error { + code?: string; + status?: number; +} + +const ROWS = [ + { id: '1', stage: 'won', owner: 'u1', score: 10 }, + { id: '2', stage: 'lost', owner: 'u2', score: 20 }, + { id: '3', stage: 'open', owner: 'u1', score: 30 }, +]; + +/** + * The exact text `driver-sql` produces for the same two conditions, copied from + * `sql-driver.ts`. Kept as literals rather than imported: driver-memory does not + * depend on driver-sql (and must not), so the twin invariant #4436 established + * is held by pinning the other side's wording here — the same way + * `memory-filter-refusal-envelope.test.ts` pins the envelope itself. + */ +const DRIVER_SQL_WORDING = { + unknownOperator: (op: string, field: string) => `Unsupported filter operator "${op}" on field "${field}".`, + malformedBetween: (field: string) => `Operator "$between" on field "${field}" requires a [min, max] value array.`, +}; + +describe('[#5324/#5328] a filter this driver cannot evaluate is refused, not answered', () => { + let driver: InMemoryDriver; + + beforeEach(async () => { + driver = new InMemoryDriver({ persistence: false }); + await driver.syncSchema('deal', { + fields: { + id: { type: 'text', name: 'id' }, + stage: { type: 'text', name: 'stage' }, + owner: { type: 'text', name: 'owner' }, + score: { type: 'number', name: 'score' }, + }, + }); + for (const row of ROWS) await driver.create('deal', { ...row }); + }); + + const ids = async (where: unknown): Promise => { + const rows = await driver.find('deal', { object: 'deal', fields: ['id'], where: where as FilterCondition }); + return (rows as Array>).map((r) => String(r.id)).sort(); + }; + + /** The refusal raised by the LIVE query path (normalize → mingo). */ + const liveRefusal = async (where: unknown): Promise => { + try { + await ids(where); + } catch (e) { + return e as WireBearingError; + } + throw new Error('expected the live query path to refuse this filter, but it answered'); + }; + + /** The refusal raised by the REFERENCE matcher, for the same filter. */ + const matcherRefusal = (where: unknown): WireBearingError => { + try { + ROWS.filter((r) => match(r, where)); + } catch (e) { + return e as WireBearingError; + } + throw new Error('expected the reference matcher to refuse this filter, but it answered'); + }; + + /** Every refusal in this package carries the same wire identity (#4436). */ + const expectEnvelope = (err: WireBearingError): void => { + expect(err.code).toBe('INVALID_FILTER'); + expect(err.status).toBe(400); + // Driver-internal wording never ships to a client (#3867). + expect(err.message).not.toContain('[driver-memory]'); + }; + + // ── #5324: an operator outside the vocabulary ────────────────────────────── + + /** + * Deliberately more than one, and deliberately not just `$not`. The passthrough + * was generic, so the fix has to be — and each of these reached mingo + * unchanged before it. `$where` / `$expr` are the sharpest: they are real + * MongoDB operators, so an engine that implements them would have EVALUATED + * an operator the Filter Protocol never declared. + */ + const PSEUDO_OPERATORS: Array<[string, unknown]> = [ + ['$sounds_like', 'won'], + ['$elemMatch', { stage: 'won' }], + ['$size', 2], + ['$type', 'string'], + ['$mod', [2, 0]], + ['$where', 'return true'], + ['$expr', { $eq: ['$stage', 'won'] }], + // Field-level `$not` — mingo implements it, so this one was not even an + // error before: it silently evaluated an operator no other backend accepts + // (driver-sql, driver-mongodb and objectql's `having` all refuse it) and + // that this package's own matcher ignored, i.e. answered "matches every row". + ['$not', { $eq: 'won' }], + ]; + + for (const [op, comparand] of PSEUDO_OPERATORS) { + it(`refuses "${op}" in a field constraint, on both faces`, async () => { + const where = { stage: { [op]: comparand } }; + + const live = await liveRefusal(where); + expectEnvelope(live); + expect(live.message).toContain(DRIVER_SQL_WORDING.unknownOperator(op, 'stage')); + + const reference = matcherRefusal(where); + expectEnvelope(reference); + expect(reference.message).toBe(live.message); + }); + } + + it('names the position, so a refusal deep in a scope tree is actionable', async () => { + const err = await liveRefusal({ $or: [{ owner: 'u1' }, { $and: [{ stage: { $sounds_like: 'won' } }] }] }); + expectEnvelope(err); + expect(err.message).toContain('filter.$or[1].$and[0].stage'); + }); + + it('refuses an undeclared combinator in a NODE position, on both faces', async () => { + // The document-level half of the same passthrough: `normalizeFilterCondition` + // copied any `$`-key it did not recognise straight into the mingo query. + // `FilterConditionSchema` declares exactly three (LOGICAL_OPERATORS). + for (const key of ['$nor', '$where', '$comment', '$text']) { + const where = { [key]: [{ stage: 'won' }] }; + const live = await liveRefusal(where); + expectEnvelope(live); + expect(live.message).toContain(`Unsupported filter combinator "${key}"`); + expect(matcherRefusal(where).message).toBe(live.message); + } + }); + + // ── #5328: a malformed `$between` ────────────────────────────────────────── + + /** + * The four malformed comparands the issue names. Every one of them used to + * return `[]` from `find` — "the range matched nothing", indistinguishable + * from a range that genuinely matched nothing — while the reference matcher + * answered the OPPOSITE (it skipped the comparison, so every row matched). + */ + const MALFORMED_BETWEEN: Array<[string, unknown]> = [ + ['not an array', 5], + ['one element', [5]], + ['three elements', [5, 10, 15]], + ['an object instead of a [min, max] pair', { min: 5, max: 10 }], + ['a string that merely looks like a range', '5,10'], + ['null', null], + ]; + + for (const [label, comparand] of MALFORMED_BETWEEN) { + it(`refuses $between with ${label}, on both faces`, async () => { + const where = { score: { $between: comparand } }; + + const live = await liveRefusal(where); + expectEnvelope(live); + expect(live.message).toContain(DRIVER_SQL_WORDING.malformedBetween('score')); + + const reference = matcherRefusal(where); + expectEnvelope(reference); + expect(reference.message).toBe(live.message); + }); + } + + it('the two faces used to answer a malformed $between OPPOSITELY — pinned as one refusal', async () => { + // The single sharpest fact in #5328: `{ score: { $between: 5 } }` returned + // NO rows from `find` and EVERY row from `match`. One package, one filter, + // two contradictory silent answers, neither of them a range. + const where = { score: { $between: 5 } }; + expect((await liveRefusal(where)).message).toBe(matcherRefusal(where).message); + }); + + it('a well-formed $between is untouched, including the calendar-day rewrite it feeds', async () => { + expect(await ids({ score: { $between: [10, 20] } })).toEqual(['1', '2']); + expect(ROWS.filter((r) => match(r, { score: { $between: [10, 20] } })).map((r) => r.id)).toEqual(['1', '2']); + }); + + // ── malformed combinator operands, the same shape one position over ──────── + + it('refuses a combinator operand that is not a filter node', async () => { + const cases: Array<[unknown, RegExp]> = [ + [{ $or: 'nope' }, /requires an array of filter conditions/], + [{ $and: { stage: 'won' } }, /requires an array of filter conditions/], + [{ $or: [null] }, /not a filter condition object/], + [{ $and: ['stage'] }, /not a filter condition object/], + [{ $or: [[{ stage: 'won' }]] }, /not a filter condition object/], + [{ $not: 'won' }, /not a filter condition object/], + [{ $not: [{ stage: 'won' }] }, /not a filter condition object/], + ]; + for (const [where, pattern] of cases) { + const live = await liveRefusal(where); + expectEnvelope(live); + expect(live.message).toMatch(pattern); + expect(matcherRefusal(where).message).toBe(live.message); + } + }); + + // ── the gate did not catch anything it should not ────────────────────────── + + describe('every legal shape answers exactly as before', () => { + const LEGAL: Array<[unknown, string[]]> = [ + [{ stage: 'won' }, ['1']], + [{ score: { $gt: 15 } }, ['2', '3']], + [{ score: { $gte: 10, $lte: 20 } }, ['1', '2']], + [{ stage: { $in: ['won', 'lost'] } }, ['1', '2']], + [{ stage: { $nin: ['won'] } }, ['2', '3']], + [{ stage: { $ne: 'won' } }, ['2', '3']], + [{ stage: { $contains: 'o' } }, ['1', '2', '3']], + [{ stage: { $notContains: 'o' } }, []], + [{ stage: { $startsWith: 'w' } }, ['1']], + [{ stage: { $endsWith: 'n' } }, ['1', '3']], + // Two regex-producing operators on one field — the `_multiRegex` promotion. + [{ stage: { $startsWith: 'o', $endsWith: 'n' } }, ['3']], + [{ stage: { $null: false } }, ['1', '2', '3']], + [{ stage: { $null: true } }, []], + [{ stage: { $exists: true } }, ['1', '2', '3']], + [{ stage: { $regex: 'W', $options: 'i' } }, ['1']], + [{ score: { $between: [10, 20] } }, ['1', '2']], + [{ $or: [{ stage: 'won' }, { owner: 'u2' }] }, ['1', '2']], + [{ $and: [{ owner: 'u1' }, { score: { $gt: 15 } }] }, ['3']], + [{ $not: { stage: 'won' } }, ['2', '3']], + [{ $or: [{ stage: 'won' }, {}] }, ['1', '2', '3']], + [{}, ['1', '2', '3']], + ]; + + for (const [where, expected] of LEGAL) { + it(`${JSON.stringify(where)} → [${expected.join(', ')}]`, async () => { + expect(await ids(where)).toEqual(expected); + }); + } + }); + + it('the refusal does not depend on the RECORD being tested', () => { + // Same argument as #5240's: both faces short-circuit, so a gate inside + // evaluation would refuse for some rows and answer for others — a filter + // that is valid or invalid by luck of the data. The walk runs first. + for (const row of ROWS) { + expect(() => match(row, { stage: 'nothing-matches-this', score: { $between: 5 } })).toThrow(/\[min, max\]/); + expect(() => match(row, { $or: [{ stage: 'won' }, { owner: { $sounds_like: 'u1' } }] })).toThrow(/Unsupported filter operator/); + } + }); +}); diff --git a/packages/plugins/driver-memory/src/memory-matcher.ts b/packages/plugins/driver-memory/src/memory-matcher.ts index 9ae41dd68a..dc55268453 100644 --- a/packages/plugins/driver-memory/src/memory-matcher.ts +++ b/packages/plugins/driver-memory/src/memory-matcher.ts @@ -7,12 +7,16 @@ * Implements a subset of the ObjectStack Filter Protocol (MongoDB-compatible) * for evaluating conditions against in-memory JavaScript objects. * - * It answers a boolean for every filter it accepts; the one shape it REFUSES - * (throwing `INVALID_FILTER` instead of answering) is `{ field: {} }` — see - * `filter-refusal.ts` and the ruling on #5240. + * It answers a boolean for every filter it accepts. What it REFUSES — throwing + * `INVALID_FILTER` / 400 instead of answering — is decided by + * `assertFilterConditionShape` in `filter-refusal.ts`, the same gate + * `InMemoryDriver.find` runs, so this face and the live query path cannot + * disagree about which filters are evaluable: `{ field: {} }` (#5240), an + * operator outside the declared vocabulary (#5324), and a `$between` whose + * comparand is not `[min, max]` (#5328). */ -import { emptyFieldConstraintError, isEmptyFieldConstraint } from './filter-refusal.js'; +import { assertFilterConditionShape } from './filter-refusal.js'; type RecordType = Record; @@ -27,32 +31,16 @@ export function match(record: RecordType, filter: any): boolean { // short-circuits (`every`/`some`, and the loop returns on its first failing // key) — a refusal raised mid-evaluation would fire or not fire depending on // the RECORD being tested. Evaluation below is untouched. - assertFilterShape(filter, 'filter'); + // + // [#5324/#5328] The walk itself now lives in `filter-refusal.ts` and is the + // SAME function `InMemoryDriver.find` runs before handing a filter to mingo. + // It used to be a copy that refused one shape; a copy is how this face and + // the live one came to answer a malformed `$between` with EVERY row and NO + // row respectively. + assertFilterConditionShape(filter, 'filter'); return evaluate(record, filter); } -/** - * [#5240] Walk the whole condition tree and refuse any zero-operator field - * constraint. Every other malformed shape keeps whatever this matcher does with - * it today — this walk adds exactly one refusal. - */ -function assertFilterShape(node: unknown, path: string): void { - if (node == null || typeof node !== 'object' || Array.isArray(node)) return; - for (const [key, val] of Object.entries(node as Record)) { - const here = `${path}.${key}`; - if (key === '$and' || key === '$or') { - if (Array.isArray(val)) val.forEach((child, i) => assertFilterShape(child, `${here}[${i}]`)); - continue; - } - if (key === '$not') { - assertFilterShape(val, here); - continue; - } - if (key.startsWith('$')) continue; - if (isEmptyFieldConstraint(val)) throw emptyFieldConstraintError(key, here); - } -} - function evaluate(record: RecordType, filter: any): boolean { if (!filter || Object.keys(filter).length === 0) return true; @@ -167,7 +155,12 @@ function checkCondition(value: any, condition: any): boolean { if (!(value <= target)) return false; break; case '$between': - // target should be [min, max] + // [#5328] `target` is a two-element array — the shape gate refused + // anything else before evaluation started. The `Array.isArray` + // guard stays as the totality floor for a direct call, but it is + // no longer this face's ANSWER to a malformed range: it used to + // skip the comparison entirely, which meant "matches EVERY row" + // — the opposite of what the live query path silently answered. if (Array.isArray(target) && (value < target[0] || value > target[1])) return false; break; @@ -210,8 +203,15 @@ function checkCondition(value: any, condition: any): boolean { } catch (e) { return false; } break; - default: - // Unknown operator, ignore or fail. Ignoring safe for optional features. + default: + // [#5324] Unreachable through `match`: the shape gate refuses an + // operator this driver does not evaluate, with the same + // `INVALID_FILTER` / 400 the live query path raises. This arm + // read "Unknown operator, ignore or fail. Ignoring safe for + // optional features." — and ignoring a constraint WIDENS the + // result set, which on a read scope is a permission bypass, not + // a degraded optional feature (#3948, and objectql's `having` + // says the same thing over its own vocabulary). break; } } From c49907cbd66a67a3e8f857390c5ae494bd644a41 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 5 Aug 2026 00:19:48 +0000 Subject: [PATCH 2/5] refactor(driver-memory): the zero-operator check keeps its own predicate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `assertFieldConstraintShape` had inlined `keys.length === 0`, which left `isEmptyFieldConstraint` — and the #5240 reasoning in its doc comment about what does NOT count as one (a `Date` enumerates to nothing but is a comparand) — attached to nothing. Behaviour-identical: the caller has already established the spec is a filter node, which is the predicate's other half. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Pbu27iNUfQCHeuS551Rqo7 --- packages/plugins/driver-memory/src/filter-refusal.ts | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/packages/plugins/driver-memory/src/filter-refusal.ts b/packages/plugins/driver-memory/src/filter-refusal.ts index 66c3c3c2bb..12e995dd6e 100644 --- a/packages/plugins/driver-memory/src/filter-refusal.ts +++ b/packages/plugins/driver-memory/src/filter-refusal.ts @@ -352,8 +352,12 @@ export function assertFilterConditionShape(node: unknown, path: string): void { */ function assertFieldConstraintShape(field: string, spec: unknown, path: string): void { if (!isFilterNode(spec)) return; + // [#5240] The zero-operator constraint keeps its own predicate rather than an + // inlined `keys.length === 0`, so the reasoning for what does and does not + // count as one (a `Date` enumerates to nothing but is a comparand) stays + // attached to the check that applies it. + if (isEmptyFieldConstraint(spec)) throw emptyFieldConstraintError(field, path); const keys = Object.keys(spec); - if (keys.length === 0) throw emptyFieldConstraintError(field, path); if (!keys.some((key) => key.startsWith('$'))) return; for (const op of keys) { if (!SUPPORTED_FIELD_OPERATORS.has(op)) throw unknownFieldOperatorError(op, field, path); From 05a27ec7ee6c1e2f3b143a1e6ccf4a08d185ab78 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 5 Aug 2026 00:21:20 +0000 Subject: [PATCH 3/5] refactor(driver-memory): the translator's floor-throws reuse the gate's wording MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A malformed `$and`/`$or` operand and a malformed `$not` operand each had two messages: the gate's `filterNodeListExpectedError` / `filterNodeExpectedError`, and a second, differently-worded `unknownLogicalOperatorError` in the translator's unreachable floor. #5240's rule is one condition, one wording — and a floor that says something different from the gate above it is exactly what a reader would use to conclude they had hit a different problem. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Pbu27iNUfQCHeuS551Rqo7 --- packages/plugins/driver-memory/src/memory-driver.ts | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/packages/plugins/driver-memory/src/memory-driver.ts b/packages/plugins/driver-memory/src/memory-driver.ts index c3f386fd36..df8f57d61b 100644 --- a/packages/plugins/driver-memory/src/memory-driver.ts +++ b/packages/plugins/driver-memory/src/memory-driver.ts @@ -9,6 +9,8 @@ import { getValueByPath } from './memory-matcher.js'; import { assertFilterConditionShape, filterArrayReachedDriverError, + filterNodeExpectedError, + filterNodeListExpectedError, malformedBetweenError, unknownFieldOperatorError, unknownLogicalOperatorError, @@ -843,7 +845,7 @@ export class InMemoryDriver implements IDataDriver { const here = `${path}.${key}`; // Recurse into logical operators if (key === '$and' || key === '$or') { - if (!Array.isArray(value)) throw unknownLogicalOperatorError(key, here); + if (!Array.isArray(value)) throw filterNodeListExpectedError(key, value, here); result[key] = value.map((child: any, i: number) => this.normalizeFilterCondition(child, object, `${here}[${i}]`)); continue; } @@ -867,7 +869,7 @@ export class InMemoryDriver implements IDataDriver { // At most one `$not` per node (it is one object key), so this never // overwrites a sibling `$nor`, and an input `$nor` cannot reach here — // the shape gate refuses undeclared combinators. - if (!value || typeof value !== 'object') throw unknownLogicalOperatorError(key, here); + if (!value || typeof value !== 'object') throw filterNodeExpectedError(value, here); result.$nor = [this.normalizeFilterCondition(value, object, here)]; continue; } From ab2425183276973029bc51418b7702928722d9b5 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 5 Aug 2026 00:23:52 +0000 Subject: [PATCH 4/5] =?UTF-8?q?fix(driver-memory):=20`$options`=20without?= =?UTF-8?q?=20`$regex`=20is=20refused=20too=20=E2=80=94=20the=20one=20oper?= =?UTF-8?q?ator=20the=20allowlist=20would=20have=20leaked?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measured after the vocabulary landed: `{ field: { $options: 'i' } }` still escaped the ADR-0112 envelope on the live path (`unknown query operator $options`, no code, no status) while the reference matcher ignored it and matched EVERY row. #5324's exact shape, surviving for a single operator — because `$options` is in the vocabulary as a MODIFIER of `$regex`, not a predicate, so allowlisting the key without requiring its partner left the hole open for it alone. Refused when no `$regex` accompanies it, on both faces, and pinned. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Pbu27iNUfQCHeuS551Rqo7 --- .../driver-memory/src/filter-refusal.ts | 25 +++++++++++++++++++ .../memory-filter-vocabulary-refusal.test.ts | 16 ++++++++++++ 2 files changed, 41 insertions(+) diff --git a/packages/plugins/driver-memory/src/filter-refusal.ts b/packages/plugins/driver-memory/src/filter-refusal.ts index 12e995dd6e..2f1a173274 100644 --- a/packages/plugins/driver-memory/src/filter-refusal.ts +++ b/packages/plugins/driver-memory/src/filter-refusal.ts @@ -259,6 +259,28 @@ export function malformedBetweenError(field: string, value: unknown, path: strin ); } +/** + * [#5324] `$options` without the `$regex` it modifies. + * + * `$options` is in {@link SUPPORTED_FIELD_OPERATORS} as a MODIFIER, not a + * predicate — it carries the regex flags (`memory-matcher` reads it as + * `new RegExp(target, condition.$options)`, and objectql's `having` skips it for + * the same reason). On its own it is not a filter at all, and the two faces + * proved it: mingo raised `unknown query operator $options` — uncoded, the very + * escape #5324 is about — while the matcher ignored it and matched EVERY row. + * Allowlisting the key without requiring its partner would have left exactly one + * operator still leaking out of the envelope. + */ +export function danglingRegexOptionsError(field: string, path: string): Error { + return unsupportedFilterError( + `Operator "$options" on field "${field}" at ${path} has no "$regex" to modify. "$options" ` + + `carries the flags of a regex predicate (e.g. { "${field}": { "$regex": "abc", "$options": "i" } }); ` + + `it is not a predicate on its own. It is refused rather than ignored because the two ` + + `evaluation paths answered it differently — one raised an uncoded engine error, the other ` + + `matched every row (#5324).`, + ); +} + /** [#5324] `$and`/`$or` take a list of nodes; anything else is refused. */ export function filterNodeListExpectedError(key: string, value: unknown, path: string): Error { return unsupportedFilterError( @@ -365,6 +387,9 @@ function assertFieldConstraintShape(field: string, spec: unknown, path: string): throw malformedBetweenError(field, spec[op], `${path}.$between`); } } + // `$options` is the one entry in the vocabulary that is a modifier rather than + // a predicate, so it is the one that needs a companion. + if (keys.includes('$options') && !keys.includes('$regex')) throw danglingRegexOptionsError(field, path); } /** diff --git a/packages/plugins/driver-memory/src/memory-filter-vocabulary-refusal.test.ts b/packages/plugins/driver-memory/src/memory-filter-vocabulary-refusal.test.ts index df630dd1ab..36ea620fa0 100644 --- a/packages/plugins/driver-memory/src/memory-filter-vocabulary-refusal.test.ts +++ b/packages/plugins/driver-memory/src/memory-filter-vocabulary-refusal.test.ts @@ -153,6 +153,22 @@ describe('[#5324/#5328] a filter this driver cannot evaluate is refused, not ans }); } + it('refuses "$options" with no "$regex" to modify, on both faces', async () => { + // The one entry in the vocabulary that is a MODIFIER rather than a + // predicate, and therefore the one that could be allowlisted into a fresh + // leak: measured before this arm existed, `{ stage: { $options: 'i' } }` + // still escaped as an uncoded `unknown query operator $options` on the live + // path while the matcher ignored it and matched EVERY row — #5324's exact + // shape, surviving for a single operator. + const where = { stage: { $options: 'i' } }; + + const live = await liveRefusal(where); + expectEnvelope(live); + expect(live.message).toContain('has no "$regex" to modify'); + + expect(matcherRefusal(where).message).toBe(live.message); + }); + it('names the position, so a refusal deep in a scope tree is actionable', async () => { const err = await liveRefusal({ $or: [{ owner: 'u1' }, { $and: [{ stage: { $sounds_like: 'won' } }] }] }); expectEnvelope(err); From 378db38f82d69020f462f90ca5e8840ca46efb6d Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 5 Aug 2026 00:25:48 +0000 Subject: [PATCH 5/5] docs(changeset): state the $options rule the last fix added Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Pbu27iNUfQCHeuS551Rqo7 --- .changeset/memory-filter-refuse-what-it-cannot-evaluate.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.changeset/memory-filter-refuse-what-it-cannot-evaluate.md b/.changeset/memory-filter-refuse-what-it-cannot-evaluate.md index 9d2dd05e9b..07122d4afe 100644 --- a/.changeset/memory-filter-refuse-what-it-cannot-evaluate.md +++ b/.changeset/memory-filter-refuse-what-it-cannot-evaluate.md @@ -45,4 +45,7 @@ runs through the real driver, as it does for the other three backends. Accepted operators are the spec's `FILTER_OPERATORS`, plus `$regex` (produced by plugin-auth's ObjectQL adapter, compiled by `driver-sql`) and its `$options` -companion. +companion. `$options` is a modifier, not a predicate: on its own, with no +`$regex` beside it, it is refused like any other filter this driver cannot +evaluate — it used to raise the same uncoded engine error on the live path and +match every row in the matcher.