Support native is_variant_null and Spark 4.2 is_valid_variant

[peterxcli]

What is the problem the feature request solves?

Comet has no native predicate for distinguishing Variant JSON null from SQL NULL. Spark 4.0+ provides is_variant_null, and Spark 4.2 adds is_valid_variant for validating the Variant value/metadata pair.

SELECT is_variant_null(v) FROM t;
SELECT is_valid_variant(v) FROM t; -- Spark 4.2+

Spark's is_variant_null returns true only for Variant null and false for SQL NULL and all other values; malformed physical data raises an error in its evaluation helper. Spark 4.2's is_valid_variant returns true/false for a non-null Variant and SQL NULL for SQL NULL.

Describe the potential solution

Add expression-specific serializers and native evaluators over the canonical [value, metadata] representation:

• implement Spark's exact JSON-null, SQL-NULL, malformed-value, and nullability behavior;
• validate both Variant value and metadata bytes for is_valid_variant;
• expose is_valid_variant only in the Spark 4.2 shim/profile while leaving Spark 4.0/4.1 compilation and registry behavior unchanged; and
• keep Variant rejected for unrelated predicates/operators.

Add focused parity and native-plan tests for every Variant scalar/container kind, Variant null, SQL NULL, malformed value bytes, malformed metadata bytes, and the Spark 4.2 version boundary.

Additional context

Spark 4.2 registers is_valid_variant alongside the existing Variant functions in its tagged function registry. It is not present in Spark 4.0/4.1.

Related work: #4295, #5407, #5424, and #5425.

Non-goals: comparisons, hash functions, parsing, casts, Variant mutation functions, C2R, shuffle/spill, writing, and Python transport.

[peterxcli]

take