What is the problem the feature request solves?
#5407 adds direct, top-level whole-value Variant projection for ordinary native Parquet scans. Spark also permits Variant nested inside structs, arrays, and string-keyed maps, and its Parquet Variant suite covers struct<... variant>, array<variant>, and map<string,variant> shapes (Spark tests).
Comet's current scan admission only special-cases a directly top-level Variant field (CometScanRule), and native normalization runs only when the target Field itself is marked Variant (cast_column.rs). Nested Variant therefore remains a full scan fallback.
Describe the potential solution
- Recursively identify explicitly marked Variant leaf Fields while adapting ordinary Parquet batches.
- Apply the existing whole-value dictionary decode, unshredding, Binary normalization, Spark object ordering, and
[value, metadata] reordering exactly once at each Variant leaf.
- Preserve every enclosing struct field, list/map offset, container and element null bitmap, field name/nullability, and extension marker.
- Admit only Spark-supported logical shapes; keep Variant map keys and malformed/unmarked lookalike Structs rejected.
Add Spark parity and vector-layout tests for nullable struct parents, array elements, and map values; shredded and unshredded input; SQL NULL and Variant JSON null; and fields before/after each nested Variant. Verify pruning an unread nested Variant remains compatible with #5377.
Additional context
#3983 continues to own broader shredded writer, subfield-pruning, and predicate-pushdown work. This issue covers full-value ordinary-Parquet projection only.
Nested Variant expressions, C2R, shuffle/spill, Python, writes, and Iceberg remain separate.
What is the problem the feature request solves?
#5407 adds direct, top-level whole-value Variant projection for ordinary native Parquet scans. Spark also permits Variant nested inside structs, arrays, and string-keyed maps, and its Parquet Variant suite covers
struct<... variant>,array<variant>, andmap<string,variant>shapes (Spark tests).Comet's current scan admission only special-cases a directly top-level Variant field (CometScanRule), and native normalization runs only when the target Field itself is marked Variant (cast_column.rs). Nested Variant therefore remains a full scan fallback.
Describe the potential solution
[value, metadata]reordering exactly once at each Variant leaf.Add Spark parity and vector-layout tests for nullable struct parents, array elements, and map values; shredded and unshredded input; SQL NULL and Variant JSON null; and fields before/after each nested Variant. Verify pruning an unread nested Variant remains compatible with #5377.
Additional context
#3983 continues to own broader shredded writer, subfield-pruning, and predicate-pushdown work. This issue covers full-value ordinary-Parquet projection only.
Nested Variant expressions, C2R, shuffle/spill, Python, writes, and Iceberg remain separate.