Fix Iceberg read optimization returning NULLs for stats-less manifests (#1545) — antalya-26.3

[il9ue]

Changelog category (leave one):

• Bug Fix

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fix Iceberg read optimization returning NULL for every column when reading from manifests written without per-file column statistics (typical of non-Spark writers like pyiceberg with default settings). Affects icebergLocal, icebergS3, icebergAzure, icebergHDFS, and all *Cluster variants. Antalya 26.3 fix for Altinity/ClickHouse#1545.

Description

Antalya-specific bug fix on antalya-26.3. No upstream cherry-pick — this bug exists only on Antalya, introduced by Altinity/ClickHouse#1069 ("Read optimization using Iceberg metadata"). Mirror of the 25.8 fix in Altinity/ClickHouse#1688.

Why this fires

When reading an Iceberg table written by a non-Spark writer that omits per-file column statistics from the manifest's Avro schema (pyiceberg with default settings, format v1 writers, and others), the allow_experimental_iceberg_read_optimization path produces silent data loss: correct row counts, every column value NULL. The reporter confirmed it on icebergLocal; investigation showed the same code path fires for icebergS3, icebergAzure, icebergHDFS, and all *Cluster variants.

Root cause

IcebergIterator always populates file_meta_info before yielding objects, so the file_meta_data.has_value() check in the optimization passes. The issue is what's inside the populated DataFileMetaInfo: when the manifest's data_file.value_counts / column_sizes / null_value_counts Avro fields are all absent (per the Iceberg spec, all three are optional), DataFileMetaInfo::columns_info stays empty.

The optimization's second loop in StorageObjectStorageSource::createReader then iterates every requested column, finds none of them in the empty columns_info map, and adds them all to constant_columns_with_values with Field() (NULL). requested_columns_copy is cleared, need_only_count = true, the Parquet reader returns row count only, and generate() injects every column as a constant-NULL column at the correct row count.

The optimization conflates "no stats were written" with "all columns are absent." Absent stats tell us nothing about which columns are physically present in the file.

The fix

Add any_stats_field_present (bool) to DataFileMetaInfo. Populate it during manifest parsing in AvroForIcebergDeserializer.cpp — true if any of value_counts, column_sizes, or null_value_counts were emitted by the writer. Gate the optimization's absent-NULL loop on this flag: when no stats were emitted, skip the loop entirely and fall through to the Parquet reader, which correctly handles both physically-present columns (read normally) and schema-evolved-absent columns (handled upstream by IcebergMetadata::getInitialSchemaByPath setting the file's own schema as initial_header).

A per-column presence set was considered but is unnecessary because schema evolution is already handled upstream of the optimization; the boolean is sufficient.

JSON serialization (cluster reads via toJson() / JSON-ptr constructor) is updated to round-trip the new field. Missing-on-deserialization defaults to false, which matches pre-fix behavior.

Files changed

• src/Storages/ObjectStorage/DataLakes/IDataLakeMetadata.h: added any_stats_field_present field to DataFileMetaInfo; constructor signature updated.
• src/Storages/ObjectStorage/DataLakes/IDataLakeMetadata.cpp: JSON serde round-trips the new field; missing-on-deserialize defaults to false.
• src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.h: header updates for ParsedManifestFileEntry.
• src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp: tracks whether any stats Avro field was present during manifest parsing on 26.3.
• src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp: forwards the new bool when constructing DataFileMetaInfo.
• src/Storages/ObjectStorage/StorageObjectStorageSource.cpp: the absent-NULL loop now skips when any_stats_field_present is false.

Note: 26.3 uses AvroForIcebergDeserializer.cpp for manifest parsing where 25.8 / 26.1 use ManifestFile.cpp (file was split upstream). Same logic, different file.

Tested

• Integration test tests/integration/test_storage_iceberg_no_spark/test_iceberg_read_optimization_empty_stats.py ported from the 25.8 PR. Test logic and conftest fixture (started_cluster_iceberg_no_spark) compatible with 26.3 as-is. Four scenarios:
  • test_iceberg_local_returns_actual_rows_with_stats_less_manifest — reproducer, fails without the fix.
  • test_iceberg_local_returns_correct_rows_when_optimization_disabled — control.
  • test_iceberg_local_partial_stats_manifest_reads_correctly — manifest with value_counts only.
  • test_iceberg_local_full_stats_manifest_reads_correctly — full Spark-style stats regression guard.
• Local build verification: changed files passed clang -fsyntax-only against 26.3's source headers in the verification round of Backport #100607 to 25.8.16: Re-add {database} macro support in clickhouse-client prompt #1688. Full integration test execution will run on CI.

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All with Aarch64
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• Tiered Storage (2h)

[il9ue]

Heads up

CI is failing with a workflow template error:

Error: The template is not valid. .github/workflows/pull_request.yml
(Line: 5285, Col: 18): Error reading JToken from JsonReader.
Path '', line 0, position 0.

This appears to be coming from antalya-26.3's pull_request.yml — that file is identical between this PR and the base branch (verified via git diff origin/antalya-26.3 -- .github/workflows/pull_request.yml), and the failing line references fromJson(needs.config_workflow.outputs.data) which suggests config_workflow isn't producing the expected output upstream.

This looks like an infrastructure issue on antalya-26.3 affecting all PRs targeting that branch, not anything specific to this PR. The code itself builds clean locally with clang-21 + RelWithDebInfo.

Is this something the CI team is aware of, or should I open a separate issue?

[strtgbb]

This is a known issue. We have separate workflows for internal and external branches. When the author of an external branch is an org member, some jobs get confused.

[ianton-ru]

Does this break backward compatibility?