Skip to content

Fix $all heuristic to quantify over the expected values - #1737

Closed
LautaroPetaccio wants to merge 1 commit into
WebFuzzing:masterfrom
LautaroPetaccio:fix/mongo-all-quantifier-direction
Closed

Fix $all heuristic to quantify over the expected values#1737
LautaroPetaccio wants to merge 1 commit into
WebFuzzing:masterfrom
LautaroPetaccio:fix/mongo-all-quantifier-direction

Conversation

@LautaroPetaccio

Copy link
Copy Markdown
Collaborator

Problem

The truthness for {"f": {"$all": [...]}} is aggregated as an AND over the elements of the array held by "f", each element scored by an OR over the expected values. That asks "is every stored element one of the queried values" — the inverse of $all, which requires every queried value to be present in the stored array and is indifferent to extra elements.

As quantifiers over the two lists:

MongoDB:  ∀ e ∈ expected. ∃ a ∈ actual.   a == e   ⟺  expected ⊆ actual
Current:  ∀ a ∈ actual.   ∃ e ∈ expected. a == e   ⟺  actual ⊆ expected

The two predicates coincide exactly when the sets are equal — which is the case the existing testAll covers, so the suite could not observe the difference.

Impact

Verified by running the same six queries against a real MongoDB 7.0.40 and against the calculator:

query vs document MongoDB calculator ofTrue
$all:[1,5] vs [1,5,6] match NO match 0.8200
$all:[1] vs [1,2,3] match NO match 0.6850
$all:[1,2,3] vs [1,2,3] match match 1.0000
$all:[1,2,3] vs [1,2] NO match match 1.0000
$all:[1,2,3] vs [1] NO match match 1.0000
$all:[1,2,3] vs [9] NO match NO match 0.2923

Four of the six disagree. The false positives report ofTrue == 1.0, i.e. distance 0 — a condition that no data can satisfy is recorded as already covered, so the search stops generating data for it. The false-negative direction includes the common single-value shape ($all:[1] against any array holding extra elements).

With this patch all six agree with MongoDB, and the gradient is monotone: 1.0000 → 0.8425 → 0.6850 → 0.2923.

Fix

Aggregate as an AND over the expected values, each scored by an OR over the array's elements (new computeHeuristicForContainedValue). The existing $in helper could not be reused: it holds the actual value fixed and ORs across the expected ones, which is the direction that caused this.

This also restores a usable gradient. Averaging over the document's elements meant that adding an element not in the expected list appended a low-scoring conjunct and lowered the score, so the search was rewarded for shrinking the array toward the expected set rather than adding the missing values — it was guided away from satisfying the condition.

v6.1.1 had the correct direction (it summed distanceToClosestElem(actualValues) over the expected values); the quantifiers were transposed when the calculator was rewritten to return Truthness.

Tests

Three tests added: one per failure direction, plus one asserting the gradient (a document missing one expected value must score strictly higher than one missing two). All three fail on current master and pass with the fix; the other tests are unaffected.

Full Mongo suite on this branch: 275 tests, 0 failures, 0 skippedMongoHeuristicsCalculatorTest (116), QueryParserTest (122), BsonHelperTest (21), GeoJsonUtilsTest (11), GeoJsonPointTest (3), plus the Testcontainers-backed MongoHandlerTest and MongoScriptRunnerTest.

Out of scope

MongoDB also matches {f: {$all: ["a"]}} against a scalar {f: "a"} (confirmed on 7.0.40), whereas this code returns false for any non-array field. That is a separate defect and is left untouched here to keep the change focused on one claim.

🤖 Generated with Claude Code

The truthness for {"f": {"$all": [...]}} was aggregated as an AND over the
elements of the array held by "f", where each element was scored by an OR
over the expected values. That asks "is every stored element one of the
queried values", which is the inverse of $all: the operator requires every
queried value to be present in the stored array, and is indifferent to extra
elements.

The direction was inverted in both ways:

  {tags: {$all: ["a"]}}          + {tags: ["a", "b"]}  matches in MongoDB,
                                 but was scored as false;
  {tags: {$all: ["a","b","c"]}}  + {tags: ["a"]}       does not match,
                                 but was scored as fully true, ie distance 0.

The second case is the damaging one: a condition no data can satisfy is
reported as already covered, so the search stops trying to reach it.

Aggregate as an AND over the expected values instead, scoring each one with
an OR over the elements of the array, which also restores a usable gradient:
a document missing fewer of the expected values now scores closer to true.

The existing tests only compared identical lists, where both directions agree,
so they did not catch this. The three added tests cover each failure direction
and the gradient.
@LautaroPetaccio

Copy link
Copy Markdown
Collaborator Author

Superseded by #1739, which folds this fix together with three further MongoDB semantics defects in the same class ($ne across incomparable BSON types, $nin on array-valued fields, and $not on a missing field), plus E2E coverage for the shapes involved.

They touch MongoHeuristicsCalculator in overlapping places, so keeping them as separate PRs would mean rebasing one against the other. The $all commit is unchanged and still stands alone in #1739 if you prefer to take it separately.

Closing this in favour of #1739.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant