Skip to content

[python] Keep files with NaN float stats in stats-based pruning - #10877

Open
plusplusjiajia wants to merge 1 commit into
apache:masterfrom
plusplusjiajia:fix/python-nan-stats-pruning
Open

plusplusjiajia wants to merge 1 commit into
apache:masterfrom
plusplusjiajia:fix/python-nan-stats-pruning

Conversation

@plusplusjiajia

Copy link
Copy Markdown
Member

Purpose

Stats-based file pruning compared NaN min, max or literal values with IEEE semantics, where every comparison with NaN is false. For example, a file whose max is NaN was skipped for greater_than, so rows in that file were missing from the result. Java orders NaN above every other value (CompareUtils) and keeps such files, and Java's stats collectors can write NaN bounds, for example for Avro.

Stats with a NaN min, max or literal now keep the file. This only makes pruning more conservative: such files are read and filtered row by row.

PyPaimon's row-level comparisons stay IEEE, so the stats do not adopt Java's ordering yet. Full parity for NaN and signed zero is left for a separate change.

Tests

  • predicates_test.py#test_by_simple_stats_with_nan: FLOAT and DOUBLE stats with a NaN min, max or literal keep the file for every comparison, in, not_in, between and not_between, and finite stats keep their results.
  • predicates_test.py#test_append_read_with_nan_stats: an Avro append table whose file has a NaN max, as Java's stats collector writes it, returns the matching row for greater_than.
  • 17 test cases fail on master. The predicate tests pass locally: 77 tests. The tests matching predicate, pushdown, filter, stats or pruning pass: 1540 passed, and the 6 failures also fail on master because torch, vortex or the Java e2e data are missing locally. flake8 is clean.

Stats-based file pruning compared NaN min, max or literal values with
IEEE semantics, where every comparison with NaN is false, so for example
a file whose max is NaN was skipped for greater_than. Java orders NaN
above every other value (CompareUtils) and keeps such files, and its
stats collectors can write NaN bounds, for example for Avro.

Stats with a NaN min, max or literal now keep the file. PyPaimon's
row-level comparisons stay IEEE, so the stats do not adopt Java's
ordering yet; full parity for NaN and signed zero is left for a
separate change.
@plusplusjiajia
plusplusjiajia marked this pull request as ready for review October 29, 2026 13:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant