Skip to content

[SPARK-58035] Add try_variant_insert expression#57120

Open
bojana-db wants to merge 4 commits into
apache:masterfrom
bojana-db:try-variant-insert
Open

[SPARK-58035] Add try_variant_insert expression#57120
bojana-db wants to merge 4 commits into
apache:masterfrom
bojana-db:try-variant-insert

Conversation

@bojana-db

@bojana-db bojana-db commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Adds the SQL function try_variant_insert(v, path, val), the error-tolerant counterpart of variant_insert (SPARK-57641). It inserts a value into a Variant value at a single JSONPath location, returning NULL instead of failing the query when the insertion hits a recoverable error.

Details:

  • On a valid insertion, behaves exactly like variant_insert: an object path (e.g.
    $.b) adds a new field; an array path (e.g. $[N]) inserts at index N, shifting
    later elements right and padding gaps past the end with variant nulls; missing
    intermediate keys/indices along the path are created;
  • Recoverable errors are caught and return NULL instead of throwing: a duplicate key
    (VARIANT_DUPLICATE_KEY) and any path type mismatch (VARIANT_PATH_TYPE_MISMATCH,
    e.g. an object key applied to an array);
  • Non-recoverable errors still throw, matching variant_insert: a malformed path is
    rejected with INVALID_VARIANT_PATH (the path is parsed before the insert), and a
    result exceeding the size limit raises VARIANT_SIZE_LIMIT;
  • Any NULL argument returns NULL;
  • The value may be any expression castable to variant.

Why are the changes needed?

Error-tolerant counterpart of variant_insert, matching try_variant_get and other try_* functions.

Does this PR introduce any user-facing change?

Yes, a new SQL function try_variant_insert (and the corresponding Scala/Python
functions API).

How was this patch tested?

Unit tests.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Cursor with Claude Opus 4.8

@bojana-db bojana-db changed the title Add try_variant_set expression [SPARK-58035] Add try_variant_insert expression Jul 8, 2026

@harshmotw-db harshmotw-db left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! left minor comments

Comment thread sql/core/src/test/scala/org/apache/spark/sql/VariantSuite.scala

@cloud-fan cloud-fan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 blocking, 0 non-blocking, 0 nits.
Clean, idiomatic addition of try_variant_insert following the established try_variant_get / try_parse_json pattern.

No findings. Structure (flag on the shared VariantInsert case class + parameterized builder objects), nullability, error taxonomy, registration, and test layout all match the existing try_* variant expressions. The prior inline comments (catch-clause ordering, "recoverable" terminology) both appear already addressed in the current head.

Verification

Checked the try-mode error contract: the shared try in insertAtPath catches only VARIANT_PATH_TYPE_MISMATCH and VARIANT_DUPLICATE_KEY (→ NULL); VARIANT_SIZE_LIMIT always rethrows, and a malformed path throws in parseVariantPath before the try, so both strict and try modes reject it — matching try_variant_get. The catch is precise, not over-broad: checkInputDataTypes restricts the value to allowStructsAndMaps=false, so castToVariant inside the same try cannot itself raise VARIANT_DUPLICATE_KEY — the only duplicate-key reachable there is the insertion collision. Nullability is also correct: !failOnError || children.exists(_.nullable) reduces to the inherited default for variant_insert (preserving its prior behavior exactly) and is unconditionally true for the try variant.

@srielau srielau left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants