[SPARK-58035] Add try_variant_insert expression#57120
Conversation
try_variant_set expressiontry_variant_insert expression
harshmotw-db
left a comment
There was a problem hiding this comment.
LGTM! left minor comments
cloud-fan
left a comment
There was a problem hiding this comment.
0 blocking, 0 non-blocking, 0 nits.
Clean, idiomatic addition of try_variant_insert following the established try_variant_get / try_parse_json pattern.
No findings. Structure (flag on the shared VariantInsert case class + parameterized builder objects), nullability, error taxonomy, registration, and test layout all match the existing try_* variant expressions. The prior inline comments (catch-clause ordering, "recoverable" terminology) both appear already addressed in the current head.
Verification
Checked the try-mode error contract: the shared try in insertAtPath catches only VARIANT_PATH_TYPE_MISMATCH and VARIANT_DUPLICATE_KEY (→ NULL); VARIANT_SIZE_LIMIT always rethrows, and a malformed path throws in parseVariantPath before the try, so both strict and try modes reject it — matching try_variant_get. The catch is precise, not over-broad: checkInputDataTypes restricts the value to allowStructsAndMaps=false, so castToVariant inside the same try cannot itself raise VARIANT_DUPLICATE_KEY — the only duplicate-key reachable there is the insertion collision. Nullability is also correct: !failOnError || children.exists(_.nullable) reduces to the inherited default for variant_insert (preserving its prior behavior exactly) and is unconditionally true for the try variant.
What changes were proposed in this pull request?
Adds the SQL function
try_variant_insert(v, path, val), the error-tolerant counterpart ofvariant_insert(SPARK-57641). It inserts a value into a Variant value at a single JSONPath location, returning NULL instead of failing the query when the insertion hits a recoverable error.Details:
variant_insert: an object path (e.g.$.b) adds a new field; an array path (e.g.$[N]) inserts at indexN, shiftinglater elements right and padding gaps past the end with variant nulls; missing
intermediate keys/indices along the path are created;
(
VARIANT_DUPLICATE_KEY) and any path type mismatch (VARIANT_PATH_TYPE_MISMATCH,e.g. an object key applied to an array);
variant_insert: a malformed path isrejected with
INVALID_VARIANT_PATH(the path is parsed before the insert), and aresult exceeding the size limit raises
VARIANT_SIZE_LIMIT;Why are the changes needed?
Error-tolerant counterpart of
variant_insert, matchingtry_variant_getand othertry_*functions.Does this PR introduce any user-facing change?
Yes, a new SQL function
try_variant_insert(and the corresponding Scala/PythonfunctionsAPI).How was this patch tested?
Unit tests.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor with Claude Opus 4.8