[hcrosse]

Closes #4276

Rationale for this change

count() includes NULL rows that to_arrow() drops in the issue's example: p != 'a' gives count() == 9 versus len(to_arrow()) == 4. I made count() follow to_arrow() without changing residual evaluation or to_arrow(). The NULL convention question in #3918 stays open.

count() now reads a file instead of using record_count when one of its partition fields is NULL and the row filter references that field's source column outside IsNull or NotNull. Null checks are ignored because they are exact on a NULL partition value.

If the file's spec is missing from table metadata, count() reads the file when the filter has any such reference. Other files keep the existing behavior. Deletes are unchanged.

Are these changes tested?

I added two regression tests to tests/catalog/test_catalog_behaviors.py. Both run on the memory and SQL catalogs, and both fail on main.

• One checks that count() equals len(to_arrow()) across identity partitions, bucket partitions and an evolved spec with files written before and after removing a partition field.
• The other checks which files remain metadata-only. It deletes the files that must not be read before calling count(), so an unexpected read fails.

make test gives 4478 passed, 4 failed and 5 skipped. The same four Hive Kerberos tests fail on main because the hive-kerberos extra cannot be installed in the test environment. prek run -a passed.

The tests use locally planned scans. I did not add a REST test, because the REST planning response rejects a NULL partition value. RESTContentFile.partition accepts only PrimitiveTypeValue, which excludes None. The new check uses only the task's partition data and spec ID, so REST-planned tasks take the same path.

Are there any user-facing changes?

Yes. For these scans count() now returns the same number as to_arrow(), for example 4 instead of 9 in the issue's example. It reads the affected files instead of counting them from metadata, including some files whose count does not change. There is no API change. Please add the changelog label.

[Copilot]

🟡 Changes recommended

Omitted REST partition values can trigger an IndexError during filtered server-planned scans; missing values need a safe fallback to reading the file and regression coverage.

1 open issue

• Handle omitted REST partition values without raising IndexError · New

What changed in this PR

This PR addresses #4276 by making count() follow to_arrow() for filters involving NULL partition values, without changing residual evaluation or Arrow’s NULL semantics.

Changes:

• Reads affected files instead of relying on metadata counts, while preserving eligible metadata-only counts.
• Adds regression coverage for identity, bucket, and evolved partition specs, plus checks that unaffected files are not read.

File	Description
tests/​catalog/​test_catalog_behaviors.py	Adds count-consistency and metadata-only regression tests.
pyiceberg/​table/​__init__.py	Detects relevant NULL partition values and routes affected count tasks through Arrow.

_{Review effort: Balanced}