[hcrosse]

Closes #4279

Rationale for this change

Scans return deleted rows in NaN identity partitions when one position delete file covers several data files, as Java writers produce by default. Two separately decoded NaN partition values compare as unequal, so the delete index never matches the delete file to its data files. Commit validation misses conflicts in those partitions for the same reason.

I changed Record.__eq__ to compare positions in one pass, treating float NaNs at the same position as equal. The hash is unchanged because NaN records already hash equal through str. Dictionary and set lookups now find those records, so the delete index and commit validation match the NaN partitions.

The change is limited to NaN. Record(0.0) and Record(-0.0) still compare equal with different hashes, and so do Record(1) and Record(1.0). #4274 covers signed zeros on the write path. inspect.partitions() and the residual cache key on plain tuples rather than Record, so they are unaffected. The open #3285 keys equality deletes by the same partition Record, so its lookup would match NaN partitions too.

This change should land before or together with the fix for #4277. That fix lets NaN-partition rewrites reach commit validation. Without this change, a file replacement can commit over a concurrent partition-scoped position delete and permanently restore the deleted row.

I measured == on three-field records with CPython 3.12, taking the best of seven runs of one million comparisons. Equal finite records take about 442 ns, up from 101.6 ns on main. Unequal records of the same length take about 509 ns, and NaN records take about 561 ns. For unequal nested records at depth 20, the single loop reduces time from 474 ms to 8.6 µs. Dictionary and set lookups compare only after a hash match, so partition lookups rarely take that slower path.

Are these changes tested?

I added four tests.

• test_record_equality_with_nan checks separate and nested NaN records, set deduplication and unequal values.
• test_record_equality_for_nan_partition_lookup checks that the delete index finds a partition-scoped delete using a separately constructed NaN partition.
• test_validate_data_files_exist_raises_on_conflict_in_nan_partition uses mocked manifests and checks that _validate_data_files_exist rejects a concurrently deleted file in a NaN partition.
• test_record_equality_compares_nested_records_once nests a test-owned Leaf value 20 records deep and counts Leaf.__eq__ calls. It loops over cases with and without NaN alongside the nested records and checks that the call count is at most the depth.

The three NaN tests fail on main and pass with the fix. The call-count test fails on the previous PR head with 1,048,576 leaf comparisons against a bound of 20 and passes with the single loop. The issue's reproduction now returns [2, 4, 6, 8] instead of [2, 4, 5, 6, 7, 8]. The full unit suite gives 4480 passed and 5 skipped, with no failures. prek run -a passed. Integration tests and an end-to-end concurrent commit were not run.

Are there any user-facing changes?

Yes. Partition-scoped position deletes now apply in NaN partitions, and commit validation recognizes conflicts in those partitions. There is no API change. Please add the changelog label.

[Copilot]

🟡 Changes recommended

Unequal nested Record comparisons can take exponential time because the list comparison and fallback repeatedly compare the same nested pairs.

1 open issue

• Avoid exponential repeated comparisons of nested Records · New

What changed in this PR

This PR fixes NaN partition matching by treating float NaNs at corresponding positions as equal in Record.__eq__, addressing the lookup failures described in #4279.

Changes:

• Adds NaN-aware record equality while leaving hashing unchanged.
• Adds regression tests for record equality, partition-scoped delete lookup, and concurrent-delete validation.

File	Description
tests/​test_typedef.py	Tests NaN equality, nested records, set deduplication, and unequal values.
tests/​table/​test_validate.py	Tests detection of concurrent file deletion in a NaN partition.
tests/​table/​test_delete_file_index.py	Tests delete lookup with independently constructed NaN partitions.
pyiceberg/​typedef.py	Adds NaN-aware fallback comparison to record equality.

_{Review effort: Balanced}