Skip to content

fix(spec): decode pre-1.0 deletion-vector metas via the writer schema - #825

Open
jackylee-ch wants to merge 1 commit into
apache:mainfrom
jackylee-ch:fix/dv-ranges-writer-schema
Open

jackylee-ch wants to merge 1 commit into
apache:mainfrom
jackylee-ch:fix/dv-ranges-writer-schema

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

decode_nullable_dv_ranges read the _DELETIONS_VECTORS_RANGES item record
positionally and always consumed a _CARDINALITY union index. Java added that
field in 1.0.0 (#4699); 0.8.0-0.9.x declared the item as RowType.of(STRING, INT, INT), so on an older file the decoder steals the next item's bytes and
IndexManifest::read_from_bytes fails with avro cursor: negative bytes length: -49. The serde reader reads the same bytes fine, and the read is also
on the commit path (read_prev_index_entries), not just scans.

Fix: walk the writer's own nested field list, as
decode_nullable_binary_table_stats does, peeling array<["null", record]> to
reach the item, since array items keep their union. An item without f0 is
rejected rather than defaulted, f0 being the map key.

An index manifest is rewritten only by a commit that changes an index file, so
such files are still live.

@JingsongLi

Copy link
Copy Markdown
Contributor

Does this requirement come from a real production environment?

@jackylee-ch

Copy link
Copy Markdown
Contributor Author

Not directly from a production issue. I found this while testing compatibility with tables written by older Paimon versions.

@JingsongLi

Copy link
Copy Markdown
Contributor

@jackylee-ch Why is it still necessary to maintain compatibility with formats from a few years ago?

@jackylee-ch

Copy link
Copy Markdown
Contributor Author

Agreed, 0.8/0.9 support on its own would not be worth it — that table is just how I hit it. As I read IndexManifestEntrySerializer, this record's field list is deliberately not fixed: "Do not change when adding nullable fields. Old manifest readers skip unknown fields." Java resolves the record by name against the OCF's writer schema, so an item without _CARDINALITY reads as null; ours reads that union index by position, so the gap is on our side, not Java's. decode_nullable_global_index guards _SOURCE_META the same way (#515). Happy to drop the version framing from title and body.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants