Skip to content

[VL][Delta] Emit parquet field ids for UniForm tables on the native write path - #12957

Draft
malinjawi wants to merge 1 commit into
apache:mainfrom
malinjawi:split/delta-uniform-field-ids
Draft

[VL][Delta] Emit parquet field ids for UniForm tables on the native write path#12957
malinjawi wants to merge 1 commit into
apache:mainfrom
malinjawi:split/delta-uniform-field-ids

Conversation

@malinjawi

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

A Delta table with IcebergCompatV2 is supposed to produce parquet files whose footers carry Iceberg-compatible field ids. Vanilla Delta arranges that in prepareWrite by installing DeltaParquetWriteSupport, but that only configures parquet-mr — when the write is offloaded, Velox writes the file directly and never consults it. The ids can't leak in through Arrow either, since SparkArrowUtil.toArrowField drops StructField metadata.

So a UniForm table written natively ends up with no field ids at all, with no error and no fallback, while the IcebergConverter happily publishes Iceberg metadata over those files.

This reads the ids Delta already recorded in the schema — parquet.field.id for columns, parquet.field.nested.ids for the synthetic list-element / map-key / map-value nodes — and passes them through the write options into ParquetWriterOptions. Gated on IcebergCompatV2, so nothing changes for other tables. Mirrored into src-delta40, where the helper is identical and symlinked.

Native Delta write is still experimental and off by default, so this closes a correctness hole ahead of that flag rather than fixing shipped behaviour.

How was this patch tested?

GlutenDeltaParquetFieldIdSuite covers the encoding. DeltaUniFormIcebergSuite writes a partitioned table with nested struct/array/map columns, checks the footer with Delta's own isParquetIcebergCompatV2, and reads the table back through Iceberg.

Notes

Draft, and there's a known gap I'd rather flag than hide: the end-to-end suite drives the write with INSERT INTO, which OffloadDeltaCommand doesn't offload, so it currently exercises the vanilla path and would pass with this change reverted. I'm reworking it onto an offloaded write with an explicit offload assertion before this is ready for review.

Two open questions I'd welcome opinions on:

  • The gate could reasonably be "the schema carries parquet.field.id" rather than IcebergCompatV2. Delta enables field-id write for every session and writes the key for both name and id mapping, so the native writer is dropping ids for ordinary column-mapped tables too — the wider gate would cover IcebergCompatV1 and id mapping in the same guard.
  • Missing metadata currently yields -1. Delta's own getNestedFieldId throws instead, and Velox omits the annotation entirely for negative ids, so failing fast is probably the better behaviour.

This is the write-side counterpart to #12884, which plumbs column mapping mode through scan splits.

@github-actions github-actions Bot added CORE works for Gluten Core VELOX DOCS labels Sep 2, 2026
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@malinjawi
malinjawi force-pushed the split/delta-uniform-field-ids branch from aed54b8 to 57ba768 Compare September 2, 2026 12:27
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

…rite path

A Delta table with IcebergCompatV2 is supposed to produce parquet files whose
footers carry Iceberg-compatible field ids. Vanilla Delta arranges that in
prepareWrite by installing DeltaParquetWriteSupport, but that only configures
parquet-mr: when the write is offloaded, Velox writes the file directly and
never consults it. The ids cannot leak in through Arrow either, since
SparkArrowUtil.toArrowField drops StructField metadata.
The result is a UniForm table whose data files carry no field ids at all, with
no error and no fallback, while the IcebergConverter happily publishes Iceberg
metadata over them.
Read the ids Delta already recorded in the schema -- parquet.field.id for
columns, parquet.field.nested.ids for the synthetic list-element, map-key and
map-value nodes -- and pass them through the write options into
ParquetWriterOptions. Gated on IcebergCompatV2, so nothing changes for other
tables.
Native Delta write is still experimental and off by default, so this closes a
correctness hole ahead of that flag rather than fixing shipped behaviour.
@malinjawi
malinjawi force-pushed the split/delta-uniform-field-ids branch from 57ba768 to dfcf2eb Compare September 2, 2026 12:31
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CORE works for Gluten Core DOCS VELOX

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant