count() and to_arrow() disagree for != on a nullable identity partition #4276

Apache Iceberg version

0.12.0 (latest release)

Please describe the bug 🐞

For a table identity-partitioned on a nullable string column p, filtering with p != 'a' gives scan.count() == 9 but len(scan.to_arrow()) == 4. The table has 3 rows with p = 'a', 4 with p = 'b', and 5 with p = NULL. count() includes the NULL rows, while to_arrow() excludes them. to_arrow_batch_reader() also returns 4 rows.

I reproduced this on PyIceberg 0.12.0 and main at e3564a54c10960307147377c578680276865ed18, with PyArrow 26.0.1 on Linux/amd64.

Install pyiceberg[pyarrow,sql-sqlite]==0.12.0 and pyarrow==26.0.1. The reproduction uses an in-process SQLite catalog and a temporary warehouse:

import tempfile
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.expressions import NotEqualTo
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import IdentityTransform
from pyiceberg.types import LongType, NestedField, StringType

warehouse = tempfile.mkdtemp()
catalog = SqlCatalog(
    "default", uri=f"sqlite:///{warehouse}/catalog.db", warehouse=f"file://{warehouse}"
)
catalog.create_namespace("ns")
schema = Schema(
    NestedField(1, "id", LongType(), required=False),
    NestedField(2, "p", StringType(), required=False),
    NestedField(3, "v", LongType(), required=False),
)
spec = PartitionSpec(
    PartitionField(source_id=2, field_id=1000, transform=IdentityTransform(), name="p")
)
table = catalog.create_table("ns.t", schema=schema, partition_spec=spec)
table.append(
    pa.table(
        {
            "id": list(range(1, 13)),
            "p": ["a"] * 3 + ["b"] * 4 + [None] * 5,
            "v": [1, 2, None, 1, 2, None, 3, 1, None, 2, 3, None],
        },
        schema=pa.schema([("id", pa.int64()), ("p", pa.string()), ("v", pa.int64())]),
    )
)
for field, value in [("p", "a"), ("v", 1)]:
    scan = table.scan(row_filter=NotEqualTo(field, value))
    print(
        f"{field}: count()={scan.count()} to_arrow()={len(scan.to_arrow())} batch_reader={len(scan.to_arrow_batch_reader().read_all())}"
    )

Output on both 0.12.0 and main at e3564a54:

p: count()=9 to_arrow()=4 batch_reader=4
v: count()=5 to_arrow()=5 batch_reader=5

The NULL-partition task has residual=AlwaysTrue() because the residual evaluator compares None != 'a', so count() adds the file's record_count of 5 without reading the file. to_arrow() does not use the residual: it passes the scan's full row filter to ArrowScan and re-applies that filter to each file. Arrow evaluates NULL != 'a' to null, and the scanner drops those rows.

Two null conventions are involved. PyIceberg's row evaluator, Java's Evaluator.notEq, and Java's ResidualEvaluator.notEq treat NULL as matching !=, agreeing with count() here. Java uses a nulls-first string comparator. Arrow treats NULL as not matching, agreeing with to_arrow(). When count() must read a file, it follows Arrow too: for the non-partition column v, both methods return 5 for NotEqualTo("v", 1), while the row evaluator matches 9.

Not(EqualTo("p", "a")) and NotStartsWith("p", "a") also give 9 and 4. A two-value NotIn("p", {"a", "c"}) returns 9 from both methods, because Arrow's ~isin keeps NULL rows. The table in #3918 shows the same difference.

Expected: count() returns len(to_arrow()) for the same scan. Whether NULL rows match != is the choice left open in #3918. That draft PR described the two conventions and their implications for delete() and overwrite(), and it was closed as stale.

Related:

• Visitor and evaluator edge cases can over-prune files or mishandle nulls #3498 covers other residual and evaluator null cases, rather than this count() versus to_arrow() mismatch.
• The open Keep rows where a delete filter evaluates to null #4219 addresses Table.delete() with an OR filter deletes the wrong rows when a column is NULL #4184 by making delete() remove exactly the rows returned by scan(row_filter=f), including != on a column containing NULL. That ties delete() to Arrow's convention.
• count() and plan_files() return nothing for an expired snapshot id #4197 compares the same read paths for an expired snapshot ID, rather than a nullable partition filter. The merged Raise when planning a scan of an unknown snapshot #4218 made count() raise like to_arrow() for that case.

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time