[EPIC] DataFusion 56 upgrade: regressions found by tracking DataFusion branch-56 #6410

What is the problem the feature request solves?

Draft PR #7249 (which replaced #6404) builds Comet against an unreleased DataFusion revision instead of a release. That way, upstream changes reach Comet's CI weeks before DataFusion 56.0.0 is released (targeted for Oct/Nov, apache/datafusion#24461). DataFusion has now cut branch-56 for the release, so the PR tracks that branch instead of main. It is not meant to merge before the release. It currently pins these versions:

This issue tracks the regressions that PR finds. The goal is to fix them upstream, or work around them in Comet, before the release rather than after it.

Describe the potential solution

Regressions

Check an item off once #7249 pins a revision that has the fix.

• The native Iceberg writer puts NaN bounds in manifests.

  • CometIcebergWriteActionSuite "native acceleration: NaN float/double manifest metrics match the JVM writer" fails (run). The native writer produces [1,0.0,NaN,2,0.25,NaN,3,NaN,NaN] where the JVM writer produces [1,0.0,2.5,2,0.25,1.5,3,null,null].
  • Cause: Parquet 60 now writes NaN min/max statistics for a column chunk whose values are all NaN, where 59 wrote none (get_min_max in parquet/src/column/writer/encoder.rs). iceberg-rust's MinMaxColAggregator::update copies the Parquet statistics into lower and upper bounds without filtering out NaN. So a data file with an all-NaN float or double column gets NaN bounds, where iceberg-java writes none.
  • Impact: iceberg-java readers treat a NaN bound as unreliable, so this should not cause wrong results there. But the manifests no longer match the JVM writer's.
  • The fix belongs in iceberg-rust, ideally as part of its Arrow 60 bump (Bump to Arrow 60 iceberg-rust#3257). Reported upstream as Parquet writer records NaN lower/upper bounds for all-NaN float and double columns on Parquet 60 iceberg-rust#3620. Parquet 60's NaN statistics are intended (the IEEE_754_TOTAL_ORDER column order), so nothing to report to arrow-rs.

• Nested BINARY values read as strings skip UTF-8 validation (Native Parquet scan reads invalid UTF-8 in nested BINARY values as unvalidated strings on DataFusion 56 #7469).

  • nested_arrow_binary_types_read_as_string_decode_invalid_utf8 (from fix: decode invalid UTF-8 in nested Parquet BINARY values read as strings #7106) panics in the parquet reader in debug builds. In release builds the scan returns a StringArray holding invalid UTF-8, where main decodes ill-formed sequences to U+FFFD.
  • Cause: fix: make Parquet file schema type coercion work on nested schemas datafusion#25193 made the file-schema binary-to-string coercion recurse into nested types, and the parquet reader validates UTF-8 only for columns with the UTF8 annotation. This is the nested form of Native Parquet scan returns NULL for nested BINARY values that are not valid UTF-8 when they are read as STRING #7032. Reported upstream as REGRESSION: Parquet scan returns unvalidated Utf8 for nested BINARY fields read as strings (regression from #25193) datafusion#27065, fix in draft fix: keep nested Parquet binary fields out of the binary-to-string coercion datafusion#27067 (needs a branch-56 backport). The full fix is the opt-in InvalidUtf8::Replace reader option proposed in Parquet reader: option to replace invalid UTF-8 instead of failing arrow-rs#12047, which would also cover the top-level form and Comet native scan rejects invalid UTF-8 byte sequences in STRING column (hll.sql on Spark 4.1) #4121.
  • The test is ignored on the DataFusion 56 branch until a fix lands.

• The fair pool and the spill replay pool stopped recognizing the clustered final aggregate.

  • refactor: use group clustering terminology in aggregate execution datafusion#24697 renamed OrderedFinalAggregateStream and its memory consumer to ClusteredFinalAggregateStream. Both pools match that consumer by name, so on DataFusion 56 the fair pool's exemption (limited_to_its_share) and the spill replay pool's final-aggregate check silently stopped applying.
  • Fixed in Comet on deps: upgrade to DataFusion 56.0.0 #7249 (e0446662e).

Behavior changes reviewed and accepted

• Arrow 60 accepts empty entries in unsorted Variant dictionaries (Add proptest fuzzing to parquet-variant and implement fixes for findings arrow-rs#10352). Comet's empty-key workaround used to reject two malformed dictionaries. Both now pass through unchanged, as they do in Spark.
• A failed read from a JVM input stream now reports Cannot get next batch from input stream. Error code: N. Producer error: .... This is because Arrow 60 made the C Stream callbacks private, so Comet now uses the stock RecordBatchStreamReader (named ArrowArrayStreamReader before Arrow 60.1).
• An invalid shuffle IPC schema message now returns a decode error instead of panicking (try_fb_to_schema).
• DataFusion's new floor/ceil return types and map_extract absent-key results don't reach Comet, because Comet registers its own implementations.
• DataFusion now treats a missing Parquet null count as unknown rather than zero. That is a correctness fix. It can reduce pruning on files written by parquet-rs versions before 53.1.

Upstream follow-ups

• Move the iceberg-rust pin to a main revision once Bump to Arrow 60 iceberg-rust#3257 merges.
• DataFusion 56 deprecates HashJoinExec::with_dynamic_filter_expr and SortExec::with_dynamic_filter_expr with no replacement. Comet no longer needs either (993beb81a): the runtime TopK uses the filter a SortExec with a fetch creates for itself, and DynamicFilterJoinExec has the join install its filter through handle_child_pushdown_result, as DataFusion's FilterPushdown rule does. No upstream issue needed.
• Remove the Variant workarounds that Arrow 60 makes unnecessary ([Variant] Track and remove native Variant compatibility workarounds after upstream fixes #5477).

Open Comet issues that depend on DataFusion main

Each issue below waits on a DataFusion change that is not in branch-55, so it can only be finished or rechecked once Comet moves to DataFusion 56. All of the upstream changes named below are in branch-56 (checked against 1635ed9), so each item can now be done on top of #7249. Check an item off once Comet has adopted the change and the issue is resolved.

• Remove the spill replay workaround from the memory pools once DataFusion includes apache/datafusion#25383 #6583: remove the spill replay workaround from the memory pools. Needs fix: leave memory for aggregate spill replay datafusion#25383 (merged to main on 2026-09-17, the branch-55 backport [branch-55] fix: leave memory for aggregate spill replay (#25383) datafusion#25814 was closed).
• Native PartialMerge over input sorted on a prefix of its keys fails to spill on DataFusion 55.2 #6880 (closed by fix: spill a native PartialMerge whose input is sorted on some of its keys #7222 on main): on 56, drop the switch in planner/aggregate.rs that plans a PartialMerge over input sorted on a key prefix as Final, since Single runs ClusteredSingleAggregateStream there. The consumer rename this item also listed is done (see Regressions). Note that the spill replay pool's final-aggregate check does not cover ClusteredSingleAggregateStream; that only matters if Remove the spill replay workaround from the memory pools once DataFusion includes apache/datafusion#25383 #6583 keeps the spill replay pool.
• Run SortAggregateExec as a native hash aggregate without its input sort #7146: run SortAggregateExec as a native hash aggregate. The issue relies on the ExternalSorter spill workspace fix, fix: preserve external sort workspace across spilling datafusion#24740 for External sort can lose reserved spill workspace when memory availability decreases datafusion#24739 (merged to main on 2026-09-04, not in branch-55).
• [Variant] Track and remove native Variant compatibility workarounds after upstream fixes #5477: remove the native Variant compatibility workarounds. The footer reconstruction step needs the lazy per-file schema hook from feat: Support lazy per-file Parquet Arrow schema derivation datafusion#25343 (merged to main on 2026-09-16, not in branch-55).
• Native existence join enumerates every duplicate build match (N*M candidates for M markers) #6484: native existence join enumerates every duplicate build match. Related upstream join work is feat(pwmj): support LeftSemi/LeftAnti existence joins via classic scan datafusion#23870 (merged to main on 2026-08-14, not in branch-55). It is not confirmed to fix the N*M enumeration, so recheck the issue on 56.
• Native Parquet scan assigns row groups to splits differently from Spark #6512: the native Parquet scan assigns row groups to splits differently from Spark. Recheck on 56.
• Wire stack (Support inline and stack generators in GenerateExec #5125 / feat: run inline, inline_outer, and stack generators natively #7251): needs feat: support Spark-compatible stack function datafusion#26266, which is in branch-56 (6489d15).
  Tested revisions

Additional context

Steps for triaging a red run on #7249:

1. Re-run it once and compare with main's latest nightly, since some failures are known flakes.
2. Run the failing test on the merge commit with the previous upstream revision. This tells an upstream change apart from a Comet change on main.
3. Bisect the upstream commits between the two revisions, using a local [patch] override. branch-56 only gets backports now, so the range is usually small.

#7249 has the run-iceberg-tests and run-all-spark-profiles labels, so CI runs Iceberg's Spark tests and the non-default Spark profiles once the branch is mergeable.