Score conditions on array-valued fields, and fix two $bits defects - #1740
Closed
LautaroPetaccio wants to merge 9 commits into
Closed
Score conditions on array-valued fields, and fix two $bits defects#1740LautaroPetaccio wants to merge 9 commits into
LautaroPetaccio wants to merge 9 commits into
Conversation
The truthness for {"f": {"$all": [...]}} was aggregated as an AND over the
elements of the array held by "f", where each element was scored by an OR
over the expected values. That asks "is every stored element one of the
queried values", which is the inverse of $all: the operator requires every
queried value to be present in the stored array, and is indifferent to extra
elements.
The direction was inverted in both ways:
{tags: {$all: ["a"]}} + {tags: ["a", "b"]} matches in MongoDB,
but was scored as false;
{tags: {$all: ["a","b","c"]}} + {tags: ["a"]} does not match,
but was scored as fully true, ie distance 0.
The second case is the damaging one: a condition no data can satisfy is
reported as already covered, so the search stops trying to reach it.
Aggregate as an AND over the expected values instead, scoring each one with
an OR over the elements of the array, which also restores a usable gradient:
a document missing fewer of the expected values now scores closer to true.
The existing tests only compared identical lists, where both directions agree,
so they did not catch this. The three added tests cover each failure direction
and the gradient.
computeHeuristicComparisonNonNullValues falls back to C_FALSE when no comparison
logic is defined for a pair of types, regardless of the operator. That is right for
equality and for the ordering operators, but wrong for $ne: MongoDB treats values of
different, mutually incomparable BSON types as different from each other, so an
inequality check on them holds.
{age: {$ne: "abc"}} against {age: 20} matches in MongoDB (verified on 7.0.40),
but was scored as definitely false, with no gradient that could ever reach true.
Return TRUE_C for NOT_EQUALS_TO in that fallback, and keep C_FALSE for the other
operators. Ordering comparisons across different BSON types do not match in MongoDB
either, so those were already correct; a test now pins that down.
$in checks each element when the queried field holds an array, but $nin passed the
whole array to the comparison instead. A List compared against a scalar has no
comparison logic defined, so it fell back to false, and the negation turned that into
a full match with distance 0.
{tags: {$nin: ["a"]}} against {tags: ["a", "b"]} does not match in MongoDB
(verified on 7.0.40), but was reported as satisfied.
$nin is the negation of $in, so both now share computeHeuristicForMembership and
differ only by that negation.
Two guards are needed by the shared path, and each fixes an IllegalArgumentException
thrown out of the empty aggregation that would otherwise be reachable:
an empty stored array holds none of the expected values, and an empty list of expected
values cannot be matched. In MongoDB {tags: {$in: []}} matches nothing and
{tags: {$nin: []}} matches everything, which is what these guards produce.
computeHeuristic for $not returned true whenever the queried field was absent from the
document. That is right only for inner operators that do not match an absent field.
Several do match one, and for those the negation must be false.
{a: {$not: {$ne: 5}}} against {} does not match in MongoDB (verified on
{a: {$not: {$nin: [1, 2]}}} 7.0.40), because the inner operator matches the
{a: {$not: {$exists: false}}} absent field. All three were reported as satisfied.
The generic path below the shortcut already produced the right answer in these cases,
since the inner operators treat an absent field as a null value, so the shortcut was
actively discarding a correct result. Removing it also keeps the complementary cases
($eq, $in, $exists:true, $gt, $size) true, as they were before.
… fixed The existing endpoints exercise these operators only in shapes that were already handled correctly: "ne" compares two numbers, "nin" is applied to a scalar field, and "not" wraps a $gt, which does not match an absent field. None of them reaches the paths that were wrong. Three endpoints are added for the shapes that were: neCrossType $ne between a string field and a number, ie two incomparable BSON types ninArrayField $nin on a field holding an array, which must be checked element-wise notExistsFalse $not wrapping an operator that does match a document lacking the field Note that "ninArrayField" is deliberately not satisfied by the two documents that saveData inserts, as both of them hold "a", so it can only be covered by data that EvoMaster generates itself.
$all, $in and $nin each decided on their own what it means for a field to hold a
value, and each got a different part of it wrong:
{f: {$all: ["a"]}} against {f: "a"} matches in MongoDB, but any
non-array field was scored false
{tags: {$in: [["a","b"]]}} against {tags: ["a","b"]} matches in MongoDB, but only
the elements were ever compared,
never the array itself
MongoDB has one rule for all of them: a field matches a value when its own value is
that value, or when it holds an array of which at least one element is. Both readings
apply at once, so {"tags": ["a","b"]} is matched both by ["a","b"] and by "a".
computeHeuristicForMatchedValue now implements that rule, and $all, $in and $nin are
expressed in terms of it: $all is an AND over the expected values, $in an OR, and
$nin the negation of $in. This drops the special cases for scalar and empty arrays,
which the rule covers on its own, and removes 11 lines overall.
Verified against MongoDB 7.0.40, including the cases where the two readings differ:
["b","a"] does not match ["a","b"], since arrays are equal only in the same order,
while an array holding ["a","b"] as an element does.
Also fixes an unrelated crash this uncovered, present since before this series:
comparing two empty arrays took the branch for lists of equal size, and aggregated
over their zero elements, throwing IllegalArgumentException instead of reporting the
two as equal. It is reachable from a plain {f: {$eq: []}}, and now from {f: {$in: [[]]}}.
$eq compared the field's value as a whole, so it missed the other half of MongoDB's
matching rule: a field holding an array is also matched by any one of its elements.
{f: {$eq: "a"}} against {f: ["a", "b"]} matches in MongoDB (verified on 7.0.40),
but was scored false.
$ne is the exact negation of $eq, and had the mirrored problem: it reported true for
the same document, where MongoDB does not match it.
Both now go through computeHeuristicForMatchedValue, which is what $all, $in and $nin
already use, so a single value is now interpreted the same way everywhere. This also
removes a live inconsistency: $in with one value already looked at array elements
while $eq with the same value did not, although MongoDB treats the two identically.
Note the whole-array reading already worked for $eq, since two lists were compared
element by element; only the reading that looks inside the array was missing. The
second added test pins the part that already held, so it passes either way.
The four $bits selectors only accepted a Long, so {"flags": {"$bitsAllSet": 1}} was
not recognised as a query. An unrecognised query parses to null, and the calculator
then throws a NullPointerException on it, which escapes MongoHandler and takes the
whole ExtraHeuristicsDto for that action with it.
MongoDB accepts any integer bitmask, and it arrives as an Integer whenever the value
fits in one, so this is an ordinary query rather than an unusual shape. Note that the
existing E2E endpoints all write their bitmask as a long literal, which is why the
suite never hit this.
QueryParserTest asserted that an Integer bitmask is invalid, which is what MongoDB
says it is not. That expectation is replaced by one covering both forms.
The bit-position array form, {"$bitsAllSet": [0, 2]}, is still not supported.
MongoDB applies the same rule to every condition expressed on a field, not only to
equality: a field holding an array satisfies the condition when the array itself does,
or when any one of its elements does. Only the equality-based operators followed it.
{a: {$gt: 2}} against {a: [1,2,3]} matches in MongoDB, but was scored false
{a: {$mod: [2,1]}} against {a: [1,2,3]} matches in MongoDB, but was scored false
{a: {$bitsAllSet:1}} against {a: [1,2,3]} matches in MongoDB, but was scored false
The rule now lives in computeHeuristicOnFieldValue, which takes the score of a single
value and applies it to the field the way MongoDB does. The helper added for equality
in the previous commits is expressed in terms of it, and $gt, $gte, $lt, $lte, $mod
and the $bits operators now use it too. This also fixes the operators that build on
them: $and, $not, and the implicit form {a: {$gte: 1, $lte: 9}}, which could report a
false positive when negating a comparison over an array.
Two further corrections are needed for that to hold:
Values that cannot be compared no longer throw. A sub-document reached the "Unsupported
type" branch, which threw out of the heuristics computation; scoring every element of an
array makes that far easier to reach, since arrays of sub-documents are common. MongoDB
does not fail on such a query, it simply does not match, so they are now treated as any
other pair of incomparable values: different from each other, hence false for equality
and ordering, and true for inequality.
A bitwise operator no longer matches a non-integral number. The value was truncated with
longValue(), so 3.5 was read as 3 and reported as matching; MongoDB matches 3.0 but not
3.5.
Verified by running 385 query and document pairs through both a real MongoDB 7.0.40 and
the calculator: disagreements drop from 123 to 69, and the exceptions thrown from 113 to
66. Everything still failing is a $type query written with a string alias, which is a
separate defect in both the parsing and the evaluation of that operator.
Collaborator
Author
Collaborator
Author
|
Closing this in favour of #1743, which carries the same findings as test cases only, with no change to the heuristics. Each defect is there as its own The implementation I had written here is still on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three further defects in
MongoHeuristicsCalculator, found by running 385 query and documentpairs through both a real MongoDB 7.0.40 and the calculator and comparing every answer.
1. Conditions on array-valued fields were not scored element by element
MongoDB applies one rule to every condition expressed on a field, not only to equality: the
field satisfies the condition when its value does, or when it holds an array of which any
element does. Only the equality-based operators followed it.
{a:{$gt:2}}vs{a:[1,2,3]}{a:{$mod:[2,1]}}vs{a:[1,2,3]}{a:{$bitsAllSet:1}}vs{a:[1,2,3]}It also propagates to everything built on those:
$and, the implicit form{a:{$gte:1,$lte:9}}, and$not, where negating a wrong comparison produces a falsepositive —
{a:{$not:{$gt:2}}}against{a:[1,2,3]}was reported as matching.The rule now lives in one helper, and
$gt,$gte,$lt,$lte,$modand the$bitsoperators are expressed in terms of it, as the equality operators already were.
2. A value that cannot be compared threw out of the heuristics computation
A sub-document reached an
"Unsupported type"branch that threwIllegalArgumentException.It escapes
MongoHandler.computeFindDistance, which has notry/catch, and takes the wholeExtraHeuristicsDtofor that action with it, SQL heuristics included.Fixing point 1 made this far easier to reach, since every element of an array is now scored and
arrays of sub-documents are a common shape. MongoDB does not fail on such a query, it simply
does not match, so these are now treated as any other pair of incomparable values: different
from each other, hence false for equality and ordering, and true for inequality.
3. Two
$bitsdefectsAn Integer bitmask was not recognised. The four selectors accepted only a
Long, so{"flags":{"$bitsAllSet":1}}parsed tonulland the calculator threw aNullPointerExceptionon it, with the same consequence as point 2. MongoDB accepts any integer bitmask, and it arrives
as an
Integerwhenever the value fits in one. The existing E2E endpoints all write theirbitmask as a long literal, which is why the suite never hit this.
A non-integral number was matched. The value was truncated with
longValue(), so3.5wasread as
3and reported as matching. MongoDB matches3.0but not3.5.QueryParserTestasserted that an Integer bitmask is invalid, which is the opposite of whatMongoDB says. That expectation is replaced by one covering both forms.
Result
Full Mongo suite: 301 tests, 0 failures, 0 skipped, including the Testcontainers-backed
MongoHandlerTestandMongoScriptRunnerTest.Still failing, and deliberately not in this PR
All 66 remaining exceptions are one defect:
$typewritten with a string alias(
{"a":{"$type":"string"}}).getTypeFromAliascallsBsonType.valueOf, which expects theenum constant name (
STRING), not MongoDB's alias ("string"), so it returns null and the queryis unparsable. Only the numeric type codes work, which is what the E2E endpoint uses.
It is left out because fixing the parsing alone would turn those crashes into silent wrong
answers: the evaluation compares
actualValue.getClass().getTypeName()against a class name fromBsonTypeClassMap, so$type:"array"would comparejava.util.ArrayListagainstjava.util.Listand never match, and$typedoes not apply the rule of point 1 either. It needsits own change.
The other three remaining disagreements are
{a:{$elemMatch:{x:1}}}matching{a:[1,2,3]},which MongoDB does not, and dotted field paths (
{"a.x": 1}) never resolving into asub-document, which MongoDB does match.