Support predicate subqueries in projections

[osipovartem]

Which issue does this PR close?

• Closes Physical plan does not support logical expression Exists #23022.

Rationale for this change

Predicate subqueries such as EXISTS, IN, and NOT IN are valid scalar expressions in a SELECT list, but DataFusion currently only decorrelates them when they appear in filters. Projection queries therefore reach physical planning with an unsupported logical subquery expression.

This continues the work from the now-closed stale #23039 and covers multiple and correlated projection subqueries as well as SQL null semantics.

What changes are included in this PR?

• decorrelate EXISTS, NOT EXISTS, IN, and NOT IN expressions inside projections into mark joins
• materialize projected IN/NOT IN values with SQL three-valued logic by tracking matches, nulls, and empty subqueries
• preserve projection output names and avoid duplicate columns in correlated subquery projections

What is the testing strategy for this PR?

Optimizer plan tests cover projected IN rewriting. New SQLLogicTest cases cover uncorrelated and correlated EXISTS/IN, multiple subqueries in one projection, empty inputs, null inputs, and NOT IN three-valued logic.

Locally validated with:

• cargo test -p datafusion-optimizer decorrelate_predicate_subquery (53 passed)
• cargo test -p datafusion-sqllogictest --test sqllogictests -- subquery_projection
• cargo clippy -p datafusion-optimizer --all-targets --all-features -- -D warnings
• cargo fmt --all -- --check

Are there any user-facing changes?

Yes. Predicate subqueries can now be evaluated as projected scalar expressions. There are no public API changes.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 77.16535% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.91%. Comparing base (7b00b63) to head (a4e6878).

Files with missing lines	Patch %	Lines
...on/optimizer/src/decorrelate_predicate_subquery.rs	77.16%	6 Missing and 23 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24972      +/-   ##
==========================================
- Coverage   81.91%   81.91%   -0.01%     
==========================================
  Files        1134     1134              
  Lines      425637   425752     +115     
  Branches   425637   425752     +115     
==========================================
+ Hits       348654   348743      +89     
- Misses      56302    56308       +6     
- Partials    20681    20701      +20     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[AdamGS]

Very cool! I'll take a look

[kosiew]

@osipovartem,

Thanks for working on this. The projection rewrite and three-valued IN/NOT IN handling look good to me. I just have one non-blocking suggestion around correlated NOT IN coverage.

[kosiew]

Could we also add correlated NOT IN execution cases here? It would be useful to cover an inner NULL, an empty correlated result, and a NULL outer value. The current correlated test covers IN, while these cases would directly exercise the projection-specific not(result) path and make sure the per-row NULL and non-empty marks behave correctly.

[osipovartem]

Added correlated NOT IN execution coverage for all requested cases: an inner NULL, an empty correlated result, and a NULL outer value, along with match and miss controls. The focused subquery_projection SLT passes locally.

[kosiew]

🚀
@osipovartem
Thank you for your contribution.