POC: CTE materialization for multi-referenced CTEs#22551
Draft
nathanb9 wants to merge 6 commits into
Draft
Conversation
Add support for materializing Common Table Expressions (CTEs) that are referenced more than once in a query. When a CTE ends in an expensive operation (Aggregate, Distinct, Window, or Union), the CTE is computed once and its results are cached in memory for reuse by multiple consumers. This implements a DuckDB-inspired heuristic: only materialize CTEs that end in expensive operations, avoiding regressions where predicate pushdown through the CTE would be more beneficial. The implementation uses Extension nodes (UserDefinedLogicalNode) to avoid modifying the core LogicalPlan enum, and introduces: - MaterializedCteProducer/Reader logical nodes - MaterializedCteExec/ReaderExec physical operators - MaterializedCtePlanner extension planner - Dependency-ordered execution for nested materialized CTEs Benchmarked on TPC-DS SF1 (10 iterations): - Q47: 2.85x speedup (401ms → 141ms) - Q57: 2.67x speedup (112ms → 42ms) - Q2: 1.58x speedup (101ms → 64ms) - Q74: 1.90x speedup (311ms → 164ms) Relates to: apache#17737
github-actions
Bot
added
sql
nathanb9
changed the title
github-actions
Bot
added
documentation
Contributor
|
@nathanb9 Cool! Can you at-me when this is ready for review? |
… and return them from partition_statistics
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
This PR adds a materialized CTE path so selected CTEs can be computed once and reused.
The heuristic respects explicit
MATERIALIZED/NOT MATERIALIZEDhints, keeps single-reference CTEs inline, and generally materializes multi-reference CTEs unless they are cheap to inline or are consumed below a top-level limit. Aggregate, distinct, window, and complex multi-scan CTEs remain materialization candidates.What changes are included in this PR?
datafusion.execution.enable_materialized_ctes, defaulting totrue.SHOW ALLsqllogictest expectations.Are these changes tested?
Yes
Additional tests cover multi-partition CTE reuse, semi-join/schema cases that previously exposed CI failures, and heuristic behavior for reused table-scan CTEs, cheap literal CTEs, and top-level
LIMITconsumers.Benchmark notes
Compared materialized CTEs enabled vs disabled on the same branch/build, 10 iterations each.
TPC-DS SF1
0.913xenabled vs disabled by summed average runtime (7904.18msvs8655.19ms), about1.095xfaster.2.72x, Q572.63x, Q22.41x, Q742.37x, Q591.86x, Q641.61x, Q41.59x, Q751.58xfaster.Are there any user-facing changes?
Yes. This adds CTE materialization behavior controlled by
datafusion.execution.enable_materialized_ctes, and SQL hints can opt individual CTEs in or out where supported by the dialect.