Support native to_variant_object

[peterxcli]

What is the problem the feature request solves?

Spark 4.x exposes to_variant_object for converting a struct, array, or string-keyed map into a Variant object/array. Comet does not serialize or evaluate it, so construction from native columns falls back:

SELECT to_variant_object(named_struct('id', id, 'tags', tags)) FROM t;
SELECT to_variant_object(array(id, id + 1)) FROM t;

The SQL function is named to_variant_object—there is no Spark 4.1.3 function named to_variant. Spark registers the name in its function registry. Scalar-to-Variant conversion is expressed as CAST(... AS VARIANT).

Spark restricts the root input to struct, array, or map and recursively validates supported children in ToVariantObject. Its encoder handles supported scalar, Variant, array, string-keyed map, and struct values recursively in VariantExpressionEvalUtils.

Describe the potential solution

Add Spark-4-specific serialization and a native recursive encoder that:

• accepts exactly Spark's root and nested datatype set;
• produces canonical Variant storage with Spark-compatible object-key ordering and metadata IDs;
• preserves nested nulls as Variant null values while a null root remains SQL NULL;
• preserves embedded Variant values without losing logical identity; and
• returns the marked [value, metadata] output Field required by Support Variant-valued native expression output and two-argument variant_get #5425.

Add focused parity and native-plan tests for structs, arrays, maps, recursively nested combinations, Variant children, SQL/nested nulls, Unicode and empty object keys, duplicate map keys, unsupported map-key types, and field ordering.

Additional context

Related work: #4295, #5407, #5425, and the dedicated Variant-cast tracker.

Non-goals: parsing JSON strings, general casts, nested Variant column transport, writing, shuffle/spill, C2R, Python transport, and Iceberg.

[peterxcli]

take