Skip to content

Fix sibling should-producing filters collapsing into one OR group - #1353

Open
jwils wants to merge 2 commits into
mainfrom
joshuaw/any-satisfy-sibling-should-repro
Open

Fix sibling should-producing filters collapsing into one OR group#1353
jwils wants to merge 2 commits into
mainfrom
joshuaw/any-satisfy-sibling-should-repro

Conversation

@jwils

@jwils jwils commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator

Problem

minimum_should_match: 1 applies to the entire should array of a bool node, so a node can hold only one group of should clauses. Two sibling filters that must be ANDed together can each produce such a group, and both groups went into the same array. (A OR B) AND (C OR D) became A OR B OR C OR D.

FilterInterpreter merged the clauses of an any_satisfy sub-filter into the shared parent node (filter_interpreter.rb:187), and build_bool_hash then stamped one minimum_should_match: 1 over the combined node. BooleanQuery#merge_into did the same for any_of.

This filter:

{
  "any_of" => [
    {"name" => {"equal_to_any_of" => ["x"]}},
    {"name" => {"equal_to_any_of" => ["y"]}}
  ],
  "ages" => {"any_satisfy" => {"any_of" => [{"gt" => 30}, {"lt" => 5}]}}
}

must mean (name = x OR name = y) AND (age > 30 OR age < 5). It generated one flat should array with all four clauses, so a document with name = "z" and ages = [40] was a search hit. There was no error and no log entry.

The required_matching_clause_count > 1 guard in process_any_satisfy_filter_expression_on_scalar_list inspects one any_satisfy sub-filter alone, so it cannot detect a collision with a sibling filter.

Four legal, public filter forms collapsed:

  1. any_of next to any_satisfy: {any_of: ...} — the case above.
  2. Two any_satisfy: {any_of: ...} filters on different scalar list fields.
  3. An any_satisfy: {any_of: ...} client filter next to an any_of internal filter.
  4. An any_of client filter next to an any_of internal filter.

Cases 3 and 4 need no any_satisfy. DatastoreQuery passes client_filters + internal_filters to one build_query call, so the clauses share one bool node. An internal filter must restrict the client's results, and a relationship additional_filter or a query_interceptor filter can contain any_of. When it did, the restriction became one more OR branch.

Fix

BooleanQuery#merge_into now nests a should group in its own bool query under filter when the target node already has a group:

def merge_into(bool_node)
  if occurrence == :should && bool_node.key?(:should)
    BooleanQuery.filter({bool: {minimum_should_match: 1, should: clauses}}).merge_into(bool_node)
  else
    bool_node[occurrence].concat(clauses)
  end
end

The nested group is required on its own, which restores the AND. It routes through BooleanQuery.filter, so the existing unwrap keeps a single-clause group cacheable.

process_any_satisfy_filter_expression_on_scalar_list merged its clauses directly into the parent node and bypassed merge_into, so it now routes a should group through merge_into. The required_matching_clause_count > 1 guard is unchanged: when the guard passes and the sub-filter has should clauses, that group is the whole sub-filter, because any other clause would raise the count above 1.

The first group stays on the shared node and each later group nests, so a query without a collision keeps its current shape and stays flat. No existing spec expectation changed.

For the filter above, the query is now:

{"bool": {
  "minimum_should_match": 1,
  "should": [
    {"bool": {"filter": [{"terms": {"name": ["x"]}}]}},
    {"bool": {"filter": [{"terms": {"name": ["y"]}}]}}
  ],
  "filter": [
    {"bool": {"minimum_should_match": 1, "should": [
      {"bool": {"filter": [{"range": {"ages": {"gt": 30}}}]}},
      {"bool": {"filter": [{"range": {"ages": {"lt": 5}}}]}}
    ]}}
  ]
}}

Specs

The specs came first and failed before the fix.

  • spec/unit/.../datastore_query/filtering_spec.rb — the generated query for each of the four cases, plus three clauses in one filter, the single-clause unwrap, and the equivalent all_of form.
  • spec/integration/.../datastore_query/filtering_spec.rb — the documents the datastore returns for cases 1 and 3. Before the fix, both returned ["t1", "t2", "t3"] where only t1 matches the filter.

Verification

  • script/run_specs — 5263 examples, 0 failures.
  • script/type_check — no type error.
  • script/lint — 894 files, no offenses.

Two sibling filters that must be ANDed together can each produce `should`
clauses. `FilterInterpreter` merges both sets of clauses into the same
`bool` node, and `build_bool_hash` then applies one `minimum_should_match: 1`
to the combined node. `(A OR B) AND (C OR D)` becomes `A OR B OR C OR D`,
so the datastore returns documents that the filter excludes.

The `required_matching_clause_count > 1` guard in
`process_any_satisfy_filter_expression_on_scalar_list` inspects one
`any_satisfy` sub-filter alone. It cannot detect a collision with a sibling
filter.

These specs mark the collapse with `pending`. Each one fails as soon as you
remove its `pending` line.

Unit specs (`any_of` sibling, two `any_satisfy` siblings, client filter next
to internal filter) show the flat `should` array. Integration specs show the
extra documents that the datastore returns for each case.
`minimum_should_match: 1` applies to the entire `should` array of a bool
node, so a node can hold only one group of `should` clauses. Two sibling
filters that must be ANDed together can each produce such a group:

    {any_of: [{name: {equal_to_any_of: ["x"]}},
              {name: {equal_to_any_of: ["y"]}}],
     ages: {any_satisfy: {any_of: [{gt: 30}, {lt: 5}]}}}

Both groups went into the same array, so `(A OR B) AND (C OR D)` became
`A OR B OR C OR D`. A document with `name = "z"` and `ages = [40]` was a
search hit, though the filter excludes it. There was no error and no log
entry.

`BooleanQuery#merge_into` now nests a `should` group in its own bool query
under `filter` when the target node already has a group. The nested group is
required on its own, which restores the AND. `BooleanQuery.filter` unwraps a
group of one clause, so a single-clause group stays cacheable.

`process_any_satisfy_filter_expression_on_scalar_list` merged its clauses
directly into the parent node, which bypassed `merge_into`. It now routes a
`should` group through `merge_into`.

The `required_matching_clause_count > 1` guard inspects one `any_satisfy`
sub-filter alone, so it could not detect a collision with a sibling filter.

Queries without a collision keep their current shape, so they stay flat.

This also fixes the same collapse between a client filter and an internal
filter, which share one bool node. An internal filter must restrict the
client's results, and a relationship `additional_filter` or a
`query_interceptor` filter can contain `any_of`.
@jwils jwils changed the title Add repro specs: sibling should-producing filters collapse into one OR group Fix sibling should-producing filters collapsing into one OR group Aug 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant