Release DataFusion `54.0.0` (Apr 2026 / May 2026)

[alamb]

Is your feature request related to a problem or challenge?

Previous release is #19692 (Mar, 2026) so next major release would be approximately Apr / May 2026

Steps:

• Create a branch-54 branch:
• Update version and changelog:
• Test with DataFusion Python
• Test with DataFusion Comet
• Test with DataFusion Ballista
• Test with delta.rs:
• Test vortex:
• Test with iceberg-rust:
• Test with LakeSail:
• Test with parquet viewer
• Test with datafusion-materialized-views
• Voting Thread
• Publish to crates.io
• Create ticket for next release: Release DataFusion 55.0.0 (Jul 2026) #22393
• Create blog post: Blog post for the DataFusion 54.0.0 release #22392

Features to mention in the blog (if they make it)

• [EPIC] Simplify datetime predicates using "preimages" #19946 from @sdf-jkl (along with the related optimizations for date_trunc and date_part)
• Do not evaluate parquet predicates if they can be proven to be false #19028
• Migrate Avro reader to arrow-avro and remove internal conversion code #17861
• feat: add support for parquet content defined chunking options #21110 from @getChan
• Feature request: Support for lateral joins #10048 lateral joins from @neilconway
• feat: sort file groups by statistics during sort pushdown (Sort pushdown phase 2) #21182 from @zhuqi-lucas
• feat: add support for parquet content defined chunking options #21110 from @kszucs
• API to register behavior for Extension Types #18223 / Basic Extension Type Registry Implementation #20312 from @tobixdev
• perf: Implement physical execution of uncorrelated scalar subqueries #21240 from @neilconway
• feat: Optimize ORDER BY by Pruning Functionally Redundant Sort Keys #21362 from @xiedeyantu
• Add lambda support and array_transform udf #21679 from @gstvg and @rluvaton
• feat: globally reorder files and row groups by statistics for TopK queries #21956 from @zhuqi-lucas
• perf: coalesce batches before sending to distributor channels in RepartitionExec #22010 from @gabotechs

Bugs that should be fixed

• Regression in json performance for local files #21450
• PR #21743 disables Parquet pruning for = ANY([literals]) #22073
• Windows errors out when RANGE frame is decimal #22113
• Windows RANGE frame crashes on overflows #22137
• Windows AVG wrong result for NULLs #22138
• fix: preserve Inexact precision in Statistics #22146
• fix: simplifier on leaf nodes returns null #22368

Community Wishlist

• Fix: compact view buffers in ScalarValue::compact for all container t… #21934
• Optimize metric label cloning #22406
• perf: push_down_filter is pathologically slow for some plans #20002
• Streaming Aggregate operator not being used in deduplication of pre-sorted Parquet files #16919
• Support ColumnExpr in LAG/LEAD for default parameter #22082
• fix(parquet): converting parquet schema with backward compatible repeated struct/primitive with provided arrow schema arrow-rs#8496
• NTILE returns wrong results #22049
• feat: optional timezone for coerce_int96 #22318

PRs to backport to release-54

• Fix: compact view buffers in ScalarValue::compact for all container t… #21934
• Revert "Add ExecutionPlan::apply_expressions() (#20337)" #22437

[alamb]

Here is another one that would be sweet to mention:

• Migrate Avro reader to arrow-avro and remove internal conversion code #17861 from @getChan

Another would be morsel driven scans

• Introduce morsel-driven Parquet scan #20529

[tschwarzinger]

Could be interessting to mention #18223 in the blog.

[alamb]

Could be interessting to mention #18223 in the blog.

Absolutely -- added to the list

[xiedeyantu]

I'm not sure if #21362 needs to be mentioned in the blog.

[alamb]

Good call -- will add that

[2010YOUY01]

I think these are worth mentioning in the blog, and I can help write about them:

• feat: Add memory-limited execution for NestedLoopJoinExec #21448
• Perf: Window topn optimisation #21479

Additionally, we could maintain an in-progress blog draft for the next release, similar to what we’re doing for the upgrade guide:

• https://github.com/apache/datafusion/blob/main/docs/source/library-user-guide/upgrading/54.0.0.md

That way, we can add items as they’re merged and avoid forgetting them later 🤔

[alamb]

That way, we can add items as they’re merged and avoid forgetting them later 🤔

We could try for sure! I have been using the release ticket to keep track of things and then have Codex / AI make a summary draft based on the list in the ticket). Then I typically have to heavily edit what is written, but it helps as a good start.

Also I think any particular optimization that you want to explain more fully could be done as its own blog post (not the release post)