Skip to content

fix: keep filters with correlated subqueries above extension nodes - #25294

Open
kakiuwang-ui wants to merge 1 commit into
apache:mainfrom
kakiuwang-ui:fix/push-down-filter-subquery-outer-refs
Open

fix: keep filters with correlated subqueries above extension nodes#25294
kakiuwang-ui wants to merge 1 commit into
apache:mainfrom
kakiuwang-ui:fix/push-down-filter-subquery-outer-refs

Conversation

@kakiuwang-ui

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

An extension node can name columns whose predicates must not be pushed past it, via UserDefinedLogicalNode::prevent_predicate_push_down_columns. A predicate whose only dependency on those columns runs through a correlated subquery is pushed down anyway, and the resulting plan does not just perform worse — it fails to execute.

Using the NoopPlan node already present in the optimizer's own tests, which refuses push-down of column c:

Filter: EXISTS (SELECT sq.a FROM sq WHERE outer_ref(test.c) = sq.a)
  NoopPlan
    TableScan: test

optimizing that plan today gives

Optimizer rule 'push_down_filter' failed
  Invalid (non-executable) plan after Optimizer rule: push_down_filter
    In/Exist/SetComparison subquery can only be used in Projection, Filter, TableScan,
    Window functions, Aggregate and Join plan nodes, but was used in [NoopPlan]

The predicate is moved below the node that asked to keep it, lands somewhere subqueries are not permitted, and the invariant check turns that into a query failure.

The cause is the one #15046 describes. The check is

!expr.column_refs().iter().any(|c| prevent_cols.contains(&c.name))

and column_refs collects only Expr::Column. A subquery records the outer columns it correlates on in Subquery::outer_ref_columns, which Expr's traversal does not descend into, so EXISTS (... WHERE outer.c = ...) reports no column references at all and looks unconditionally safe to push.

What changes are included in this PR?

A references_any_column helper in push_down_filter.rs that asks column_refs first and then, for Expr::Exists, Expr::InSubquery and Expr::ScalarSubquery, also looks at the outer_ref_columns those carry. The extension-node branch uses it instead of column_refs alone. Nothing else changes.

I deliberately did not change column_refs itself, which is what the issue title asks for. It has many callers whose behaviour would shift — optimize_projections, CSE, window and join handling all use it, and several of them want "columns produced by this plan's own schema", not outer references belonging to a parent. Widening it looks like a separate change with its own risk, so this PR fixes the consumer that has a demonstrable failure. Happy to look at the broader change instead if you would rather that were the fix.

What is the testing strategy for this PR?

New unit test push_down_filter::tests::user_defined_plan_outer_referenced_column, next to the existing user_defined_plan test and reusing its NoopPlan. It asserts the EXISTS predicate stays above the node.

The test discriminates: with the outer-reference collection short-circuited out, it fails with the invalid-plan error quoted above; with the fix it passes. datafusion-optimizer (808) and datafusion-expr (262) pass, as does sqllogictest.

Are there any user-facing changes?

No public API changes. Plans change only where an extension node declares prevent_predicate_push_down_columns and a predicate correlates a subquery on one of those columns — previously an invalid plan, now a correct one. No in-tree node declares those columns outside tests, so no bundled plan snapshots move.

Two adjacent things found while working on this, neither touched here:

`UserDefinedLogicalNode::prevent_predicate_push_down_columns` lets an
extension node name columns whose predicates must not be pushed past it.
`PushDownFilter` checked that list against `Expr::column_refs`, which only
collects `Expr::Column`. A subquery records the outer columns it correlates
on in `Subquery::outer_ref_columns`, and `Expr`'s traversal does not descend
into that field, so a predicate like `EXISTS (... WHERE outer.c = ...)` looked
like it referenced no columns at all and was pushed below the node.

For the node in the optimizer's own tests the result is not merely a worse
plan: the pushed-down `EXISTS` lands somewhere subqueries are not allowed, and
the invariant check fails the query with "In/Exist/SetComparison subquery can
only be used in Projection, Filter, TableScan, Window functions, Aggregate and
Join plan nodes".

Collect the outer references from `Exists`, `InSubquery` and `ScalarSubquery`
alongside `column_refs` when deciding what may be pushed.

Closes apache#15046
@github-actions github-actions Bot added the optimizer Optimizer rules label Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Optimizer rules

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expression column_refs does not return outer referenced columns in subqueries

1 participant