What is the problem the feature request solves?
Spark 4.x provides schema_of_variant for a single value and schema_of_variant_agg for the merged schema of a Variant column. Comet does not implement either expression, so schema discovery over a natively scanned Variant column falls back:
SELECT schema_of_variant(v) FROM t;
SELECT schema_of_variant_agg(v) FROM t;
Spark's scalar implementation infers and prints the schema, using OBJECT<...> rather than STRUCT<...>, in SchemaOfVariant. The aggregate shares that inference and merges schemas across rows and partial buffers in SchemaOfVariantAgg.
Describe the potential solution
Implement one shared Spark-compatible Variant schema inference/merge helper, then expose it through the scalar expression and aggregate:
- infer scalar, decimal precision/scale, date/timestamp, binary, UUID, array, object, and Variant-null (
VOID) schemas;
- keep object fields in Spark's required order and print
OBJECT<...> names with Spark-compatible quoting;
- merge heterogeneous array elements and object fields with Spark's compatible-type rules;
- ignore SQL NULL rows in the aggregate, start/finish an empty buffer as
VOID, and support partial-buffer merge/serialization; and
- admit Variant only for these two expressions while preserving general fallback gates.
Add focused parity and native-plan tests for every scalar kind, JSON versus SQL NULL, nested arrays/objects, heterogeneous values, decimal widening, field-name quoting and ordering, empty/all-null inputs, grouping, and multi-partition partial aggregation.
Additional context
Related work: #4295, #5407, #5424, and #5425.
Non-goals: schema-driven Variant casts, subfield pruning, predicate pushdown, writing, shuffle/spill of Variant values, C2R, Python transport, and Iceberg-specific work.
What is the problem the feature request solves?
Spark 4.x provides
schema_of_variantfor a single value andschema_of_variant_aggfor the merged schema of a Variant column. Comet does not implement either expression, so schema discovery over a natively scanned Variant column falls back:Spark's scalar implementation infers and prints the schema, using
OBJECT<...>rather thanSTRUCT<...>, inSchemaOfVariant. The aggregate shares that inference and merges schemas across rows and partial buffers inSchemaOfVariantAgg.Describe the potential solution
Implement one shared Spark-compatible Variant schema inference/merge helper, then expose it through the scalar expression and aggregate:
VOID) schemas;OBJECT<...>names with Spark-compatible quoting;VOID, and support partial-buffer merge/serialization; andAdd focused parity and native-plan tests for every scalar kind, JSON versus SQL NULL, nested arrays/objects, heterogeneous values, decimal widening, field-name quoting and ordering, empty/all-null inputs, grouping, and multi-partition partial aggregation.
Additional context
Related work: #4295, #5407, #5424, and #5425.
Non-goals: schema-driven Variant casts, subfield pruning, predicate pushdown, writing, shuffle/spill of Variant values, C2R, Python transport, and Iceberg-specific work.