Fix sibling should-producing filters collapsing into one OR group - #1353
Open
jwils wants to merge 2 commits into
Open
Fix sibling should-producing filters collapsing into one OR group#1353jwils wants to merge 2 commits into
should-producing filters collapsing into one OR group#1353jwils wants to merge 2 commits into
Conversation
Two sibling filters that must be ANDed together can each produce `should` clauses. `FilterInterpreter` merges both sets of clauses into the same `bool` node, and `build_bool_hash` then applies one `minimum_should_match: 1` to the combined node. `(A OR B) AND (C OR D)` becomes `A OR B OR C OR D`, so the datastore returns documents that the filter excludes. The `required_matching_clause_count > 1` guard in `process_any_satisfy_filter_expression_on_scalar_list` inspects one `any_satisfy` sub-filter alone. It cannot detect a collision with a sibling filter. These specs mark the collapse with `pending`. Each one fails as soon as you remove its `pending` line. Unit specs (`any_of` sibling, two `any_satisfy` siblings, client filter next to internal filter) show the flat `should` array. Integration specs show the extra documents that the datastore returns for each case.
jwils
requested review from
BrianSigafoos-SQ,
bsorbo,
ellisandrews-toast,
jwondrusch,
marcdaniels-toast,
myronmarston and
rossroberts-toast
as code owners
August 16, 2026 00:19
`minimum_should_match: 1` applies to the entire `should` array of a bool
node, so a node can hold only one group of `should` clauses. Two sibling
filters that must be ANDed together can each produce such a group:
{any_of: [{name: {equal_to_any_of: ["x"]}},
{name: {equal_to_any_of: ["y"]}}],
ages: {any_satisfy: {any_of: [{gt: 30}, {lt: 5}]}}}
Both groups went into the same array, so `(A OR B) AND (C OR D)` became
`A OR B OR C OR D`. A document with `name = "z"` and `ages = [40]` was a
search hit, though the filter excludes it. There was no error and no log
entry.
`BooleanQuery#merge_into` now nests a `should` group in its own bool query
under `filter` when the target node already has a group. The nested group is
required on its own, which restores the AND. `BooleanQuery.filter` unwraps a
group of one clause, so a single-clause group stays cacheable.
`process_any_satisfy_filter_expression_on_scalar_list` merged its clauses
directly into the parent node, which bypassed `merge_into`. It now routes a
`should` group through `merge_into`.
The `required_matching_clause_count > 1` guard inspects one `any_satisfy`
sub-filter alone, so it could not detect a collision with a sibling filter.
Queries without a collision keep their current shape, so they stay flat.
This also fixes the same collapse between a client filter and an internal
filter, which share one bool node. An internal filter must restrict the
client's results, and a relationship `additional_filter` or a
`query_interceptor` filter can contain `any_of`.
should-producing filters collapse into one OR groupshould-producing filters collapsing into one OR group
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
minimum_should_match: 1applies to the entireshouldarray of a bool node, so a node can hold only one group ofshouldclauses. Two sibling filters that must be ANDed together can each produce such a group, and both groups went into the same array.(A OR B) AND (C OR D)becameA OR B OR C OR D.FilterInterpretermerged the clauses of anany_satisfysub-filter into the shared parent node (filter_interpreter.rb:187), andbuild_bool_hashthen stamped oneminimum_should_match: 1over the combined node.BooleanQuery#merge_intodid the same forany_of.This filter:
must mean
(name = x OR name = y) AND (age > 30 OR age < 5). It generated one flatshouldarray with all four clauses, so a document withname = "z"andages = [40]was a search hit. There was no error and no log entry.The
required_matching_clause_count > 1guard inprocess_any_satisfy_filter_expression_on_scalar_listinspects oneany_satisfysub-filter alone, so it cannot detect a collision with a sibling filter.Four legal, public filter forms collapsed:
any_ofnext toany_satisfy: {any_of: ...}— the case above.any_satisfy: {any_of: ...}filters on different scalar list fields.any_satisfy: {any_of: ...}client filter next to anany_ofinternal filter.any_ofclient filter next to anany_ofinternal filter.Cases 3 and 4 need no
any_satisfy.DatastoreQuerypassesclient_filters + internal_filtersto onebuild_querycall, so the clauses share one bool node. An internal filter must restrict the client's results, and a relationshipadditional_filteror aquery_interceptorfilter can containany_of. When it did, the restriction became one more OR branch.Fix
BooleanQuery#merge_intonow nests ashouldgroup in its own bool query underfilterwhen the target node already has a group:The nested group is required on its own, which restores the AND. It routes through
BooleanQuery.filter, so the existing unwrap keeps a single-clause group cacheable.process_any_satisfy_filter_expression_on_scalar_listmerged its clauses directly into the parent node and bypassedmerge_into, so it now routes ashouldgroup throughmerge_into. Therequired_matching_clause_count > 1guard is unchanged: when the guard passes and the sub-filter hasshouldclauses, that group is the whole sub-filter, because any other clause would raise the count above 1.The first group stays on the shared node and each later group nests, so a query without a collision keeps its current shape and stays flat. No existing spec expectation changed.
For the filter above, the query is now:
{"bool": { "minimum_should_match": 1, "should": [ {"bool": {"filter": [{"terms": {"name": ["x"]}}]}}, {"bool": {"filter": [{"terms": {"name": ["y"]}}]}} ], "filter": [ {"bool": {"minimum_should_match": 1, "should": [ {"bool": {"filter": [{"range": {"ages": {"gt": 30}}}]}}, {"bool": {"filter": [{"range": {"ages": {"lt": 5}}}]}} ]}} ] }}Specs
The specs came first and failed before the fix.
spec/unit/.../datastore_query/filtering_spec.rb— the generated query for each of the four cases, plus three clauses in one filter, the single-clause unwrap, and the equivalentall_ofform.spec/integration/.../datastore_query/filtering_spec.rb— the documents the datastore returns for cases 1 and 3. Before the fix, both returned["t1", "t2", "t3"]where onlyt1matches the filter.Verification
script/run_specs— 5263 examples, 0 failures.script/type_check— no type error.script/lint— 894 files, no offenses.