fix: nested list schema evolution by XiaoHongbo-Hope · Pull Request #446 · alibaba/paimon-cpp

XiaoHongbo-Hope · 2026-07-23T09:47:50Z

Purpose

Linked issue: close #xxx

Tests

API and Format

Documentation

Generative AI tooling

PruneDataType rejected any nested difference inside a list/map as a partial projection. A field added inside a list<struct> (schema evolution) made the read type a superset of the file type and hit the same fail-fast, so data-evolution reads of old files errored with 'partial projection inside list'. Reconcile the repeated item by field id: added read fields are allowed (null-filled downstream), dropped file fields still fail. Adds unit tests for the list/map added-field and drop-and-add cases.

After PruneDataType allows a schema-evolution added field inside a list/map, the sub-reader yields the file's structure (e.g. list<struct<a,b>>) while the output must match the read schema (list<struct<a,b,c>>). Reshape each exist sub-array to the read type, null-filling the added nested field (recursive over struct/list/map, preserving offsets and validity). NOTE: not yet compiled/tested; pending local build of paimon-cpp.

Factor AlignArrayToReadType into NestedProjectionUtils (shared by the data-evolution and field-mapping readers). In the single-file path: skip the scalar cast for differing LIST/MAP types (like STRUCT), and reshape each exist array to the read type so an evolution-added nested field is null-filled. NOTE: not yet compiled/tested; pending local build.

Verifies the reshape actually null-fills a schema-evolution added field inside list<struct> and preserves existing values.

…nner field Drives FieldMappingReader over a list<struct<a,b>> batch against a read schema whose struct gained c(id=12); asserts the output is list<struct<a,b,c>> with c null-filled.

CLAassistant · 2026-07-23T09:48:01Z

All committers have signed the CLA.

Copilot

Pull request overview

This PR improves schema evolution handling for nested repeated types by allowing LIST/MAP item STRUCTs to evolve via added fields and then reshaping the in-memory Arrow arrays to the read schema by null-filling those added nested fields. This aligns the C++ reader behavior with expected schema evolution semantics for nested lists/maps.

Changes:

Extend NestedProjectionUtils::PruneDataType to allow schema-evolution added fields within LIST/MAP item STRUCTs (while still rejecting true partial projection that drops file fields).
Add NestedProjectionUtils::AlignArrayToReadType and integrate it into readers so nested added fields are materialized as nulls at read time.
Add unit tests covering LIST/MAP pruning for added fields and end-to-end reader behavior for added fields inside LIST structs.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/paimon/core/utils/nested_projection_utils.h	Declares new `AlignArrayToReadType` helper for reshaping nested arrays to the read schema.
src/paimon/core/utils/nested_projection_utils.cpp	Implements repeated-item pruning for schema evolution and the array reshape/null-fill logic.
src/paimon/core/utils/nested_projection_utils_test.cpp	Adds UTs for LIST/MAP pruning evolution and LIST array alignment null-filling behavior.
src/paimon/core/utils/field_mapping.cpp	Updates cast-executor selection to skip scalar casts for nested STRUCT/LIST/MAP differences handled by pruning + reshape.
src/paimon/core/io/field_mapping_reader.cpp	Applies the reshape step during non-partition column casting/mapping to null-fill added nested fields.
src/paimon/core/io/field_mapping_reader_test.cpp	Adds an end-to-end test verifying added LIST-struct fields are null-filled on read.
src/paimon/common/reader/data_evolution_file_reader.cpp	Applies the same reshape/null-fill step when merging batches across evolved schemas.

Comments suppressed due to low confidence (1)

src/paimon/core/utils/nested_projection_utils.cpp:327

Same issue as above in PruneDataType MAP: arrow::map(key, item) does not preserve keys_sorted or the original key/item field properties. Construct the new MapType from the existing data_map fields and keys_sorted.

            PAIMON_ASSIGN_OR_RAISE(
                std::shared_ptr<arrow::DataType> item,
                PruneRepeatedItemType(read_map->item_type(), data_map->item_type(), "map"));
            return std::optional<std::shared_ptr<arrow::DataType>>(arrow::map(key, item));
        }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+#include "paimon/core/utils/nested_projection_utils.h"
 #include "paimon/core/utils/field_mapping.h"
 #include "paimon/core/utils/nested_projection_utils.h"


+    if (array->type()->Equals(*read_type)) {
+        return array;
+    }
+    const auto& data = array->data();


+            PAIMON_ASSIGN_OR_RAISE(
+                std::shared_ptr<arrow::DataType> item,
+                PruneRepeatedItemType(read_map->item_type(), data_map->item_type(), container));
+            return arrow::map(key, item);


- Remove duplicate nested_projection_utils.h include. - AlignArrayToReadType: fail fast on a kind/atomic-type mismatch instead of building an invalid ArrayData; only reshape a field whose data and read types differ (skips dictionary-encoded arrays of unchanged type). - Preserve MapType keys_sorted and key/value field metadata when pruning map item types.

ORC lazy decoding returns nested strings (including map keys) as Arrow dictionary arrays. AlignArrayToReadType now keeps a leaf's actual encoding (a dictionary of the read value type is kept as-is) and builds the reshaped STRUCT/LIST/MAP output type from the children's actual types, instead of forcing read_type and rejecting dictionary vs string. Add a dictionary-leaf test.

lxy-9602 · 2026-07-23T10:52:52Z

+    ASSERT_EQ(c_col->null_count(), c_col->length());  // added field is all null
+    auto a_col = std::static_pointer_cast<arrow::Int32Array>(out_struct->GetFieldByName("a"));
+    ASSERT_EQ(a_col->Value(0), 1);
+    ASSERT_EQ(a_col->Value(2), 3);


I wonder if it would be clearer to write the data in JSON format here, similar to field_mapping_reader_test.cpp.

I wonder if it would be clearer to write the data in JSON format here, similar to field_mapping_reader_test.cpp.

Sure, updated

lxy-9602 · 2026-07-23T10:54:15Z

+}
+
 TEST(NestedProjectionUtilsTest, HasNestedSubfieldProjectionNoProjection) {
    auto file_schema = arrow::schema({


Could you please add integration tests for map and list in write_and_read_inte_test.cpp to ensure the end-to-end path is covered?

Could you please add integration tests for map and list in write_and_read_inte_test.cpp to ensure the end-to-end path is covered?

Sure, added.

- Compare paimon field IDs in all no-op/substitution checks (EqualWithFieldIds), so a drop+add of a same-name/same-type field (new ID) is not silently exposed as the new field's value. - Reject a rename inside a list/map struct (matching the top-level PruneDataType behaviour). - Add field-ID-change and ORC round-trip regression tests. - clang-format 20 the touched files.

lxy-9602 · 2026-07-23T11:19:22Z

+                                                        read_schema_->field(i)->type(),
+                                                        arrow_pool_.get()));
+        target_sub_array_vec.push_back(aligned);
    }


One question I still have: if each file is already aligned via AlignArrayToReadType in field_mapping_reader.cpp, why do we need to do the same thing again in data_evolution_file_reader.cpp?

One question I still have: if each file is already aligned via AlignArrayToReadType in field_mapping_reader.cpp, why do we need to do the same thing again in data_evolution_file_reader.cpp?

Good catch, updated.

Each column-split file is already aligned to its read schema by its FieldMappingReader, so the data-evolution reader assembly no longer re-aligns. Build the dictionary-leaf test array from JSON.

The integration test surfaced that the parquet reader drops nested field-id metadata, so AlignArrayToReadType's id-based child matching errored. Match nested struct children by name (preserved by all readers), requiring agreement only when both sides carry an ID, so a drop+add of a same-name field is still not silently mapped. Add TestSchemaEvolutionAddFieldInsideListAndMap covering an added field inside a list<struct> and a map<struct> end-to-end on parquet and orc.

AlignArrayToReadType now produces exactly the read type: STRUCT/LIST/MAP are rebuilt with read-side types and nullability, and a leaf is cast to the read type. This fixes two ORC lazy-decoding failures -- a dictionary<int64, large_string> that no longer equals logical string, and a nested NOT NULL drop that left old/new files with mismatched types (arrow::ChunkedArray::Make failed). EqualWithFieldIds now compares Field::nullable(). Cover dictionary<int64,large_string>, read-side nullability, and enable ORC lazy decoding in the inte test.

lxy-9602 · 2026-07-23T14:08:30Z

-                                               array->type()->ToString(), read_type->ToString()));
+            // Leaf: cast to the read type (decodes an ORC dictionary, widens
+            // large_string to string, etc.).
+            return CastingUtils::Cast(array, read_type, arrow::compute::CastOptions::Safe(), pool);


I wonder why does AlignArrayToReadType use generic Arrow safe cast for all leaf mismatches? If this helper is only for reshaping nested arrays and decoding dictionaries, please restrict it to dictionary/format-representation cases and fail other leaf mismatches. If nested type evolution is intended, please use the same schema-evolution cast rules as FieldMappingReader instead of raw Arrow Cast.

Good point — restricted it to representation-only changes. The leaf branch now normalizes away the two physical differences that legitimately reach it under ORC lazy decoding (a dictionary wrapper and the string/binary 32/64-bit offset width) and casts only when the normalized types match; any other leaf mismatch now fails fast with AlignArrayToReadType unsupported leaf type change.

Genuine nested type evolution is intentionally not handled here: it is already rejected upstream in PruneDataType/PruneRepeatedItemType (leaf type_id must match), and top-level type promotion continues to go through FieldMappingReader cast executors. Added AlignArrayToReadTypeRejectsLeafTypeChange to cover the new guard.

I wonder why does AlignArrayToReadType use generic Arrow safe cast for all leaf mismatches? If this helper is only for reshaping nested arrays and decoding dictionaries, please restrict it to dictionary/format-representation cases and fail other leaf mismatches. If nested type evolution is intended, please use the same schema-evolution cast rules as FieldMappingReader instead of raw Arrow Cast.

Restricted the leaf cast to representation-only cases (dictionary decode + string/binary width); other type mismatches now fail fast. Added a test.

lxy-9602 · 2026-07-24T00:55:39Z

+            return arrow::binary();
+        default:
+            return t;
+    }


I’m not sure we should change large here. As far as I remember, only string has dictionary encoding while binary does not, and blob is the only type using large binary, so changing it to binary may not be right.

I’m not sure we should change large here. As far as I remember, only string has dictionary encoding while binary does not, and blob is the only type using large binary, so changing it to binary may not be right.

Thanks, removed the large_binary normalization.

SR will use large_binary converter to convert blob col to SR chunk.

Blob columns don't go through this path — their type already matches the read type, so AlignArrayToReadType returns them untouched, and it emits the read type anyway. This change is orthogonal to blob and the large_binary converter.

…inary

lxy-9602

LGTM

XiaoHongbo-Hope · 2026-07-24T07:08:20Z

LGTM

Thanks, could you please help trigger CI again.

lucasfang · 2026-07-27T02:26:08Z

Could you please add more details to the PR description?

lucasfang · 2026-07-27T02:38:33Z

+1

XiaoHongbo-Hope added 5 commits July 22, 2026 23:31

Add array-level test for AlignArrayToReadType nested null-fill

9d50b65

Verifies the reshape actually null-fills a schema-evolution added field inside list<struct> and preserves existing values.

Add end-to-end test: read list<struct> file with an evolution-added i…

bf772a8

…nner field Drives FieldMappingReader over a list<struct<a,b>> batch against a read schema whose struct gained c(id=12); asserts the output is list<struct<a,b,c>> with c null-filled.

Copilot AI review requested due to automatic review settings July 23, 2026 09:47

Copilot started reviewing on behalf of XiaoHongbo-Hope July 23, 2026 09:52 View session

Trim comments

83ee0e2

Copilot AI reviewed Jul 23, 2026

View reviewed changes

XiaoHongbo-Hope and others added 2 commits July 23, 2026 03:09

Merge branch 'main' into fix_nested_list_schema_evolution

b565f23

lxy-9602 changed the title ~~Fix nested list schema evolution~~ fix: nested list schema evolution Jul 23, 2026

lxy-9602 reviewed Jul 23, 2026

View reviewed changes

XiaoHongbo-Hope added 4 commits July 23, 2026 04:31

Address review: drop redundant DE-reader alignment; JSON dict test

9492011

Each column-split file is already aligned to its read schema by its FieldMappingReader, so the data-evolution reader assembly no longer re-aligns. Build the dictionary-leaf test array from JSON.

Trim comment

ee14e41

lxy-9602 reviewed Jul 23, 2026

View reviewed changes

XiaoHongbo-Hope added 2 commits July 23, 2026 07:12

Update expected message for map value type-change rejection

b8ad679

Restrict AlignArrayToReadType leaf cast to representation-only changes

00131a8

lxy-9602 reviewed Jul 24, 2026

View reviewed changes

XiaoHongbo-Hope and others added 3 commits July 23, 2026 19:10

Only normalize large_string in leaf representation check, not large_b…

0ffc476

…inary

Merge branch 'main' into fix_nested_list_schema_evolution

2b78370

Merge branch 'main' into fix_nested_list_schema_evolution

0b85137

lxy-9602 requested a review from lszskye July 24, 2026 03:49

lxy-9602 reviewed Jul 24, 2026

View reviewed changes

Add nested large_binary blob alignment test

2e70b7d

Merge branch 'main' into fix_nested_list_schema_evolution

def0fee

Uh oh!

Conversation

XiaoHongbo-Hope commented Jul 23, 2026

Purpose

Tests

API and Format

Documentation

Generative AI tooling

Uh oh!

CLAassistant commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lxy-9602 left a comment

Choose a reason for hiding this comment

Uh oh!

XiaoHongbo-Hope commented Jul 24, 2026

Uh oh!

lucasfang commented Jul 27, 2026

Uh oh!

lucasfang commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

CLAassistant commented Jul 23, 2026 •

edited

Loading