fix: keep filters with correlated subqueries above extension nodes - #25294
Open
kakiuwang-ui wants to merge 1 commit into
Open
fix: keep filters with correlated subqueries above extension nodes#25294kakiuwang-ui wants to merge 1 commit into
kakiuwang-ui wants to merge 1 commit into
Conversation
`UserDefinedLogicalNode::prevent_predicate_push_down_columns` lets an extension node name columns whose predicates must not be pushed past it. `PushDownFilter` checked that list against `Expr::column_refs`, which only collects `Expr::Column`. A subquery records the outer columns it correlates on in `Subquery::outer_ref_columns`, and `Expr`'s traversal does not descend into that field, so a predicate like `EXISTS (... WHERE outer.c = ...)` looked like it referenced no columns at all and was pushed below the node. For the node in the optimizer's own tests the result is not merely a worse plan: the pushed-down `EXISTS` lands somewhere subqueries are not allowed, and the invariant check fails the query with "In/Exist/SetComparison subquery can only be used in Projection, Filter, TableScan, Window functions, Aggregate and Join plan nodes". Collect the outer references from `Exists`, `InSubquery` and `ScalarSubquery` alongside `column_refs` when deciding what may be pushed. Closes apache#15046
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
column_refsdoes not return outer referenced columns in subqueries #15046.Rationale for this change
An extension node can name columns whose predicates must not be pushed past it, via
UserDefinedLogicalNode::prevent_predicate_push_down_columns. A predicate whose only dependency on those columns runs through a correlated subquery is pushed down anyway, and the resulting plan does not just perform worse — it fails to execute.Using the
NoopPlannode already present in the optimizer's own tests, which refuses push-down of columnc:optimizing that plan today gives
The predicate is moved below the node that asked to keep it, lands somewhere subqueries are not permitted, and the invariant check turns that into a query failure.
The cause is the one #15046 describes. The check is
and
column_refscollects onlyExpr::Column. A subquery records the outer columns it correlates on inSubquery::outer_ref_columns, whichExpr's traversal does not descend into, soEXISTS (... WHERE outer.c = ...)reports no column references at all and looks unconditionally safe to push.What changes are included in this PR?
A
references_any_columnhelper inpush_down_filter.rsthat askscolumn_refsfirst and then, forExpr::Exists,Expr::InSubqueryandExpr::ScalarSubquery, also looks at theouter_ref_columnsthose carry. The extension-node branch uses it instead ofcolumn_refsalone. Nothing else changes.I deliberately did not change
column_refsitself, which is what the issue title asks for. It has many callers whose behaviour would shift —optimize_projections, CSE, window and join handling all use it, and several of them want "columns produced by this plan's own schema", not outer references belonging to a parent. Widening it looks like a separate change with its own risk, so this PR fixes the consumer that has a demonstrable failure. Happy to look at the broader change instead if you would rather that were the fix.What is the testing strategy for this PR?
New unit test
push_down_filter::tests::user_defined_plan_outer_referenced_column, next to the existinguser_defined_plantest and reusing itsNoopPlan. It asserts theEXISTSpredicate stays above the node.The test discriminates: with the outer-reference collection short-circuited out, it fails with the invalid-plan error quoted above; with the fix it passes.
datafusion-optimizer(808) anddatafusion-expr(262) pass, as doessqllogictest.Are there any user-facing changes?
No public API changes. Plans change only where an extension node declares
prevent_predicate_push_down_columnsand a predicate correlates a subquery on one of those columns — previously an invalid plan, now a correct one. No in-tree node declares those columns outside tests, so no bundled plan snapshots move.Two adjacent things found while working on this, neither touched here:
find_out_reference_exprsdoes not see these outer references either, for the same reason, so it cannot be used as the collection mechanism. That is the same traversal gap as Expr::apply_children/map_children & LogicalPlan::map_expressions does not correctly map subqueryouter_ref_columnexpressions #16147, which @AdamGS has taken.column_refsdoes not return outer referenced columns in subqueries #15046 reports.