build: upgrade to arrow 59, DataFusion 55, and pyo3 0.29 - #8997
Draft
slachiewicz wants to merge 7 commits into
Draft
build: upgrade to arrow 59, DataFusion 55, and pyo3 0.29#8997slachiewicz wants to merge 7 commits into
slachiewicz wants to merge 7 commits into
Conversation
parquet moves with arrow because DataFusion 55 requires parquet ^59.2 and two arrow copies would split the array types. pythonize moves with pyo3 because 0.28 pinned pyo3 0.28 (links = python conflict). object_store stays at 0.13.2, which DataFusion 55 still requires.
MutableArrayData::extend is deprecated for try_extend; FixedSizeBinaryArray lost its From<Vec<_>> impls in favour of TryFrom; the record_batch! macro re-export from datafusion::common is deprecated for arrow_array::record_batch.
Their version tracks the arrow major, as in the 57 to 58 bump (lance-format#6638).
…rrow 59 geoarrow-array/geoarrow-schema 0.8 pin arrow ^58 and geodatafusion 0.5 pins datafusion ^54, so the geo feature cannot compile against arrow 59 from any published release. Revert this commit once geoarrow 0.9 and a matching geodatafusion ship.
Every ExecutionPlan node implements the new required apply_expressions and the replace_children API; with_new_children remains as a shim that forwards with ChildrenPropertiesMode::Recompute because DataFusion 55 still declares it as a required (deprecated) method. partition_statistics implementations move to child_stats_requests plus statistics_from_inputs, and callers use StatisticsContext::compute. EnforceDistribution was folded upstream into EnsureRequirements, which also enforces sorting requirements. UnionExec::new became try_new, ExecutionOptions.batch_size became ConfigNonZeroUsize, Partitioning gained a Range variant, and create_physical_expr takes a PhysicalPlanningContext.
github-actions
Bot
added
A-python
PyCapsule::new is deprecated in pyo3 0.29; the replacement takes the same arguments.
…value Fixes the previous commit, which did not compile: pyo3 0.29's replacement takes a &'static CStr rather than Option<CString>.
Contributor
Hi @slachiewicz, thanks for making a PR. In general, we wait for geodatafusion and datafusion-python to upgrade first be upgrading. If you want you can keep this PR up as a draft until that happens, and then we can rebase at that time. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves the workspace, the Python crate, and the Java JNI crate from arrow 58 / DataFusion 54 / pyo3 0.28 to arrow 59.3 / DataFusion 55.0 / pyo3 0.29, so downstream consumers (hudi-rs, Ballista) can pick up lance crates without a second arrow copy. arrow-pyarrow 59.1 is the first release that accepts pyo3 0.29, which is why the three bumps move together.
geodropped fromlancedefault features (TODO): no published geoarrow release compiles against arrow 59 yet; restore when geoarrow ships one.python/pyproject.tomlstill pinsdatafusion>=54,<55:test_table_provider.pyfails until datafusion-python 55 is on PyPI.Behaviour changes
EnforceDistributionwas folded upstream intoEnsureRequirements, which enforces sorting requirements as well as distribution. Lance's optimizer list now uses that rule.with_new_childrenstays on every node as a shim forwarding toreplace_childrenwithChildrenPropertiesMode::Recompute; DataFusion 55 still declares it as a required (deprecated) method.LanceExecutionOptions::batch_sizenow goes throughConfigNonZeroUsize::try_new, so a zero batch size is an error instead of being accepted.Verification
cargo check --workspace --tests --benches— clean, zero warnings.cargo clippy --all --tests --benches -- -D warnings— onlychunks_exact_to_as_chunksanddrain_collectfire, new in rustc 1.98, in untouched files (CI pins 1.97).cargo check --manifest-path python/Cargo.toml— clean after thePyCapsule::new_with_valuemigration.cargo test --workspace, Java JNI check, Python lint/tests,cargo deny check,make licenses.This change was created with AI assistance.