Support native parse_json and try_parse_json

[peterxcli]

What is the problem the feature request solves?

Spark 4.x provides parse_json and try_parse_json to construct VariantType values from strings, but Comet has no serializer or native evaluator for either function. A query such as the following cannot remain native:

SELECT parse_json(json_text) FROM t;
SELECT try_parse_json(json_text) FROM t;

Spark implements both forms with one ParseJson expression: parse_json throws for malformed input while try_parse_json returns SQL NULL. The runtime also distinguishes malformed input and the Variant size limit and honors spark.sql.variant.allowDuplicateKeys in VariantExpressionEvalUtils.parseJson.

Describe the potential solution

Add Spark-4-specific serialization and a native parser for the two expressions:

• parse each non-null input string into canonical Variant value and metadata bytes;
• emit the marked Variant output Field and Spark-compatible [value, metadata] children defined by Support Variant-valued native expression output and two-argument variant_get #5425;
• preserve JSON null as Variant null and SQL NULL as a null parent row;
• honor Spark's duplicate-key setting, Unicode/object-key behavior, numeric representation, and Variant size limit; and
• match strict/try malformed-input behavior and Spark-compatible errors.

Add focused Spark SQL parity and plan tests for objects, arrays, scalars, JSON null, SQL NULL, Unicode keys, duplicate keys with both configuration modes, malformed JSON, oversize input, and columns before/after the result.

Additional context

Spark registers the exact SQL names parse_json and try_parse_json in its 4.1.3 function registry.

Related work: #4295, #5407, and #5425.

Non-goals: to_variant_object, CAST(... AS VARIANT), Parquet writing, nested Variant columns, shuffle/spill, C2R, Python transport, and Iceberg.

[peterxcli]

take