[SPARK-57804][SQL] Add variant_set expression

[bojana-db]

What changes were proposed in this pull request?

Adds the SQL function variant_set(v, path, val[, create_if_missing]), which sets (inserts or replaces) a value in a Variant value at a single JSONPath location.

Details:

• Object path (e.g. $.a): replaces the field if it exists; creates it when create_if_missing is true (the default);
• Array path (e.g. $[N]): replaces the element at index N; when N is at or past the end and create_if_missing is true, the array is padded with variant nulls up to N;
• Missing intermediate keys/indices along the path are created when create_if_missing` is true;
• When create_if_missing is false, a missing leaf, a missing intermediate, or an out-of-range array index leaves v unchanged;
• Any NULL argument returns NULL;
• The value may be any expression castable to variant (primitives, arrays, or another variant; structs and maps are rejected);
• VARIANT_PATH_TYPE_MISMATCH is raised when a path segment is applied to a value of an incompatible type; the root path $ is rejected with INVALID_VARIANT_PATH, and results exceeding the size limit raise VARIANT_SIZE_LIMIT.

Why are the changes needed?

Without variant_set, updating a variant means converting it to another datatype (e.g. map), mutating, and converting back.

Does this PR introduce any user-facing change?

Yes, a new SQL function (and Scala/Python functions API).

How was this patch tested?

Unit tests.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code with Claude Opus 4.8