fix(spec): decode pre-1.0 deletion-vector metas via the writer schema - #825
jackylee-ch wants to merge 1 commit into
Conversation
|
Does this requirement come from a real production environment? |
|
Not directly from a production issue. I found this while testing compatibility with tables written by older Paimon versions. |
|
@jackylee-ch Why is it still necessary to maintain compatibility with formats from a few years ago? |
|
Agreed, 0.8/0.9 support on its own would not be worth it — that table is just how I hit it. As I read |
decode_nullable_dv_rangesread the_DELETIONS_VECTORS_RANGESitem recordpositionally and always consumed a
_CARDINALITYunion index. Java added thatfield in 1.0.0 (#4699); 0.8.0-0.9.x declared the item as
RowType.of(STRING, INT, INT), so on an older file the decoder steals the next item's bytes andIndexManifest::read_from_bytesfails withavro cursor: negative bytes length: -49. The serde reader reads the same bytes fine, and the read is alsoon the commit path (
read_prev_index_entries), not just scans.Fix: walk the writer's own nested field list, as
decode_nullable_binary_table_statsdoes, peelingarray<["null", record]>toreach the item, since array items keep their union. An item without
f0isrejected rather than defaulted,
f0being the map key.An index manifest is rewritten only by a commit that changes an index file, so
such files are still live.