[andygrove]

Which issue does this PR close?

Part of #6410, which tracks the regressions this PR finds. The all-NaN Iceberg bound regression is tracked in #7242 and the nested BINARY UTF-8 regression in #7469. This is a long-lived draft that tracks DataFusion's branch-56 until DataFusion 56.0.0 is released, and it is not meant to merge before then. Related: #5477, whose Arrow 60 items it unblocks.

This replaces #6404, moving the branch from a fork to datafusion-56 in this repository.

Rationale for this change

DataFusion has cut branch-56 for the 56.0.0 release, and this PR pins its head, 1635ed9 (2026-11-03). Since 55.2.0 DataFusion has moved to arrow/parquet 60.1.0 and object_store 0.14.2 (apache/datafusion#25335, apache/datafusion#26867), and it has a number of API changes (see its 56.0.0 upgrade guide). Building Comet against the release branch now surfaces the breaks early and lets CI run the Comet suites against it, rather than finding everything at release time.

The branch has current apache/main (1ad7938) merged in.

What changes are included in this PR?

Dependencies

• datafusion, datafusion-datasource, datafusion-functions, datafusion-physical-expr-adapter and datafusion-spark in the workspace now use git = "https://github.com/apache/datafusion", rev = "1635ed9...", the head of branch-56. So do the datafusion-functions-nested dev-dependency in core, datafusion in both contrib crates and the native-executors workspace.
• arrow, arrow-data, arrow-select and parquet go from 59.2.0 to 60.1.0.
• object_store goes from 0.13.2 to 0.14.2. object_store_opendal goes from 0.58.0 to 0.59.0, the first release on object_store 0.14. That release keeps opendal on 0.58, which is what iceberg-rust uses.
• parking_lot goes from 0.12 to 0.13, the version DataFusion's LazyBatchGenerator API now uses.
• iceberg and iceberg-storage-opendal move to 1f52867, a branch on the andygrove/iceberg-rust fork.
  • iceberg-rust main is still on arrow 59, and Comet hands arrow types to iceberg-rust directly, so the two must agree. Bump to Arrow 60 iceberg-rust#3257 ("Bump to Arrow 60") is the fix, and it is still open.
  • Its head (0d5eb12) is behind the iceberg-rust main revision that Comet's main builds against (b154603), and Comet's Iceberg code uses APIs from the commits in between. So the fork branch is build: reduce overhead of fuzz testing #3257's head with b154603 merged in.
  • Move back to an upstream main revision once build: reduce overhead of fuzz testing #3257 merges. The Cargo.toml comment says so.
• Lockfiles: native/Cargo.lock and contrib/delta/native/Cargo.lock are regenerated.

Source changes (all forced by the upgrade)

• Arrow 60.1 deprecates ArrowArrayStreamReader in favor of RecordBatchStreamReader, and FixedSizeListArray::value_offset in favor of value_offset_at. Comet uses the new names.
• Arrow 60 made the FFI_ArrowArrayStream callbacks private, so the decimal-realignment test in scan.rs can no longer build a hand-rolled C stream. It now exports a struct array and imports it with from_ffi_and_data_type, the call RecordBatchStreamReader::next makes. AlignedArrowStreamReader was already gone on main.
• Parquet 60 moved the page index behind ParquetMetaData::page_index(), a PageIndexProvider with per-row-group access.
  • with_spark_arrow_schema and the rebase footer rewrite carry it over with set_page_index.
  • page_selection_bound_selectors in delta_dv.rs walks the row groups through page_index_for_row_group.
  • The tests check page_index() instead of column_index()/offset_index().
• AggregateExec::input_order_mode() is now group_clustering_mode(), returning a GroupClusteringMode. Two planner tests and a min_max_struct test use it.
• refactor: use group clustering terminology in aggregate execution datafusion#24697 renamed OrderedFinalAggregateStream and its memory consumer to ClusteredFinalAggregateStream. The fair pool's exemption and the spill replay pool's final-aggregate check match that consumer by name, so they now match the new name. Without this they silently stopped applying.
• DataFusion's coerce_int96 and coerce_int96_tz are now typed (DFTimeUnit, ConfigTimeZone) instead of strings.
• object_store 0.14 added GetResult::extensions, which ScanIoObjectStore::get_opts now forwards.
• OffsetBuffer::first()/last() now return values instead of Options.
• Field and schema metadata are now arrow's Metadata type. The datetime rebase resolution reads it directly, and the shuffle schema cache's size estimate uses len() instead of capacity().
• fb_to_schema is deprecated. The shuffle decoder uses try_fb_to_schema, so an invalid schema message now returns a decode error where it used to panic.
• FFI_ArrowSchema::with_metadata is now unsafe. The call in move_to_spark is sound because the schema was just built by TryFrom.
• HashJoinExec::with_dynamic_filter_expr and SortExec::with_dynamic_filter_expr are deprecated with no replacement, and Comet no longer calls them.
  • The runtime TopK takes the filter a SortExec with a fetch creates for itself (dynamic_expressions_produced()) and hands it to the Parquet reader below the sort.
  • DynamicFilterJoinExec reports to HashJoinExec::handle_child_pushdown_result what DataFusion's FilterPushdown rule would, namely that the probe-side self filter was pushed down. The join then installs the filter itself, after finding Comet's consumer in its probe subtree.
• Two other DataFusion APIs are deprecated, and each call stays under #[allow(deprecated)]:
  • ExecutionPlan::with_preserve_order, which DataFusion no longer calls. MetricOwningExec keeps forwarding it.
  • ParquetAccessPlan::into_overall_row_selection, in three delta_dv.rs tests that model the reader's file-level selection peak. DataFusion now keeps row-group-local entries, so the deletion-vector admission bound should be rechecked against the new reader.
• normalize_spark_empty_key_metadata_rejects_other_malformed_encodings: Arrow 60 accepts empty entries in unsorted Variant dictionaries (Add proptest fuzzing to parquet-variant and implement fixes for findings arrow-rs#10352).
  • Two of the five "malformed" dictionaries in the test were rejected only because of their empty entry: the one with a repeated entry and the one with trailing bytes.
  • Both now pass through unchanged. That matches Spark, which checks only the offsets of the keys it looks up, so the test now asserts it.
• make_map_c in comet-test-udfs: arrow 60's MapBuilder names the map children key/value by default, so the test UDF now names them keys/values explicitly to keep exercising non-canonical names.
• datafusion-spark's SparkXxhash64 now ignores the hidden children of a NULL struct (fix: xxhash64 of a NULL struct ignores the struct fields datafusion#25774) and keeps the running hash for nested dictionaries (fix: keep the running hash for nested dictionaries in Spark xxhash64 datafusion#25813). Three tests in xxhash64_diff.rs that pinned those differences now assert equality. Comet still keeps its own kernel for those types, so widening the routing is a possible follow-up.

Known issue: all-NaN float and double columns in Iceberg manifests

Parquet 60 writes NaN min and max statistics for a column chunk that holds only NaN, and iceberg-rust copies them into the manifest bounds, where iceberg-java records none. CometIcebergWriteActionSuite "native acceleration: NaN float/double manifest metrics match the JVM writer" caught it, so this PR ignores that test, links #7242 and lists the issue under the Iceberg known issues and in a new 1.3.0 section of the upgrade guide. Re-enable the test once iceberg-rust skips NaN bounds.

Known issue: invalid UTF-8 in nested BINARY values read as strings

apache/datafusion#25193 made DataFusion's Parquet file-schema coercion recurse into nested types, so a nested BINARY field that the read schema declares as a string is decoded straight into Utf8. The parquet reader validates UTF-8 only for columns with the UTF8 annotation, so an invalid value produces an unvalidated StringArray: a panic in debug builds, and unsound data in release builds. On main, #7106 decodes these values to U+FFFD. This PR ignores #7106's Rust test (nested_arrow_binary_types_read_as_string_decode_invalid_utf8) and JVM test (ParquetReadSuite "native scan decodes invalid UTF-8 in nested BINARY read as string"). The issue is tracked in #7469 and reported upstream as apache/datafusion#27065, which this PR needs fixed before it merges.

DataFusion behavior changes checked against Comet

• The floor/ceil return types and the result of map_extract for absent keys changed. Comet registers its own floor, ceil and map_extract, so neither change reaches Comet.
• Missing Parquet null counts are now treated as unknown instead of zero. This is a correctness fix. It can disable some pruning on files written by parquet-rs before 53.1.

Not included (possible follow-ups)

• [Variant] Track and remove native Variant compatibility workarounds after upstream fixes #5477: Arrow 60 contains fix(variant): accept Dictionary and RunEndEncoded metadata in VariantArray arrow-rs#10810 and Add proptest fuzzing to parquet-variant and implement fixes for findings arrow-rs#10352, so the encoded-metadata decoding and the empty-key retry (canonicalize_spark_empty_key_metadata) can now be removed.
• Move the iceberg pin back to a main revision once Bump to Arrow 60 iceberg-rust#3257 merges.
• Wire stack once feat: support Spark-compatible stack function datafusion#26266 is in the pinned revision. It is now (6489d15).

How are these changes tested?

Run locally on macOS aarch64, at the current head of this branch:

• cargo fmt --check and cargo clippy --all-targets --workspace -- -D warnings are clean, and cargo check -p datafusion-comet --all-targets --features contrib-delta,contrib-lance builds.
• cargo test --workspace in native/: 2767 passed, 0 failed, 21 ignored (including the Native Parquet scan reads invalid UTF-8 in nested BINARY values as unvalidated strings on DataFusion 56 #7469 test).
• make builds the native library and the JVM modules.
• The rest of the JVM suites were not run locally. CI runs the Comet suites on the default profile. Spark's own SQL tests, the Iceberg Spark tests and the other Spark profiles need the run-all-spark-profiles, run-spark-4.1-tests and run-iceberg-tests labels.